[Bugs] [Bug 1368185] New: Directory on volume seems to be empty, but brick contains data

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1368185

            Bug ID: 1368185
           Summary: Directory on volume seems to be empty, but brick
                    contains data
           Product: GlusterFS
           Version: 3.7.14
         Component: distribute
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: sebastian.kraetzig at itonics.de
                CC: bugs at gluster.org



Created attachment 1191906
  --> https://bugzilla.redhat.com/attachment.cgi?id=1191906&action=edit
Latest GlusterFS log files after IRC chat

Description of problem:
I had a single server with three single 2 TB disks (bricks). Due of the high
CPU load at night, I had to detach and attach two of them to seperate servers.

Since this, some - not all - directories on the volume looks empty, but on the
bricks, I still can see files. If I try to list/access these files over the
volume mount point by manually typing in the file path, I can access these
files.

Version-Release number of selected component (if applicable):
3.7.14-ubuntu1~xenial1

How reproducible:


Steps to Reproduce:
1. Add multiple disks to the same server
2. Create a volume with the attached disks
3. Mount the volume
4. Upload data to this mounted volume
5. Remove one or more bricks from the volume
   gluster volume remove-brick gfsvbackup server01:/export/disk1/brick
   gluster volume remove-brick gfsvbackup server01:/export/disk2/brick

   setfattr -x trusted.glusterfs.volume-id /export/disk1/brick
   setfattr -x trusted.gfid /export/disk1/brick
   rm -rf /export/disk1/brick/.glusterfs/

   setfattr -x trusted.glusterfs.volume-id /export/disk2/brick
   setfattr -x trusted.gfid /export/disk2/brick
   rm -rf /export/disk2/brick/.glusterfs/

6. Attach the removed bricks (disks) to an other server
7. Add the bricks back to the volume
   gluster volume add-brick gfsvbackup server02:/export/disk1/brick
   gluster volume add-brick gfsvbackup server03:/export/disk2/brick

Actual results:
- Unable to delete directories
- Unable to see some directories/files

Expected results:
- Able to delete directories
- Able to see all directories/files

Additional info:
Here are some pastebin logs, after talking to nbalacha on IRC:
http://pastebin.com/E2JazkhG
http://pastebin.com/iwnV75Hy
http://pastebin.com/xskmzZPb
http://paste.fedoraproject.org/410279/71531775/

IRC history:
(16:25:25) nbalacha: which is the directory you are trying to delete?
(16:25:46) Sebbo: I think so, yes: p-v00283.itonicsit.de:/gfsvbackup on
/mnt/backup type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)

(16:26:33) Sebbo: This one (on the brick):
/export/6e221bd6-5079-4038-b607-ad2f95ba800f/brick/p-v00284/backup/; Means this
on the volume: /mnt/backup/p-v00284/backup/

(16:27:03) nbalacha: Can you get the xattrs for this dir as well
(16:27:27) Sebbo: root at p-v00367:~# getfattr -e hex -m . -d
/export/6e221bd6-5079-4038-b607-ad2f95ba800f/brick/p-v00284/backup/
(16:27:27) Sebbo: getfattr: Removing leading '/' from absolute path names
(16:27:27) Sebbo: # file:
export/6e221bd6-5079-4038-b607-ad2f95ba800f/brick/p-v00284/backup/
(16:27:27) Sebbo: trusted.gfid=0xcceec3f836b043a3a1a3fe8262e20a5c
(16:27:27) Sebbo: trusted.glusterfs.dht=0x000000010000000055555555aaaaaaa9
(16:29:15) Sebbo: root at p-v00283:~# getfattr -e hex -m . -d
/export/43e26be5-31b6-4f40-b787-c3f71422f48f/brick/p-v00284/backup/
(16:29:15) Sebbo: getfattr: Removing leading '/' from absolute path names
(16:29:15) Sebbo: # file:
export/43e26be5-31b6-4f40-b787-c3f71422f48f/brick/p-v00284/backup/
(16:29:15) Sebbo: trusted.gfid=0xcceec3f836b043a3a1a3fe8262e20a5c
(16:29:15) Sebbo: trusted.glusterfs.dht=0x00000001000000000000000055555554
(16:29:29) Sebbo: root at p-v00368:~# getfattr -e hex -m . -d
/export/c718f376-e117-4a68-be26-096a907ae04a/brick/p-v00284/backup/
(16:29:29) Sebbo: getfattr: Removing leading '/' from absolute path names
(16:29:29) Sebbo: # file:
export/c718f376-e117-4a68-be26-096a907ae04a/brick/p-v00284/backup/
(16:29:29) Sebbo: trusted.gfid=0xcceec3f836b043a3a1a3fe8262e20a5c
(16:29:29) Sebbo: trusted.glusterfs.dht=0x0000000100000000aaaaaaaaffffffff

(16:31:07) nbalacha: ok, and which brick holds the files?
(16:32:21) Sebbo: The one on p-v00367. Should be Brick1:
p-v00367.itonicsit.de:/export/6e221bd6-5079-4038-b607-ad2f95ba800f/brick

(16:49:27) nbalacha: is the file visible from the mount point?
(16:50:09) Sebbo: Yeah, seems like :D

(16:50:28) nbalacha: what happens if you try to list the dir contents now
(16:50:38) nbalacha: does this file show up?
(16:50:52) Sebbo: Nope, still empty

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.