[Gluster-users] unable to delete 'empty' dirs with 3.3qa42

[Harry Mangalam]

Just before updrading to 3.3final, we had an rsync collision on our 
gluster filesystem which left us with undeletable dirs.

The transport is IPoIB over 4 bricks as shown below.

$ gluster volume info

Volume Name: gli
Type: Distribute
Volume ID: 76cc5e88-0ac4-42ac-a4a3-31bf2ba611d4
Status: Started
Number of Bricks: 4
Transport-type: tcp,rdma
Bricks:
Brick1: pbs1ib:/bducgl
Brick2: pbs2ib:/bducgl
Brick3: pbs3ib:/bducgl
Brick4: pbs4ib:/bducgl
Options Reconfigured:
performance.io-cache: on
performance.quick-read: on
performance.io-thread-count: 64

The last 100 line of the gluster log is here: <http://pastie.org/4036243>

The problem started when I was running an rsync with --delete while 
moving a large /home to the gluster dir to prune the target of older 
cruft in a /home move..  A user directed a job to start reading from a 
directory that was simultaneously being pruned and all hell broke loose.

Now we're left with a number of dirs and files that can't be deleted.  
'du' says there are files in an apparently empty dir:

  root at bduc-login:/gl/tvanerp/adni
561 $ du -sh *
40K     dicom

altho 'ls' can't see them from the client;

root at bduc-login:/gl/tvanerp/adni
562 $ ls -lat dicom
total 40
drwxrwxr-x 3 tvanerp tvanerp 94214 Jun  5 23:40 ./
drwxrwxr-x 3 tvanerp tvanerp    72 Apr 23 13:11 ../


 From the bricks you can see that there are still files there:
Tue Jun 05 23:18:02 [3.06 2.42 1.65][457.48/606]  root at bduc-login:~
556 $ ssh pbs2 'ls -lat /bducgl/tvanerp/adni/dicom'
total 24
drwxrwxr-x 3 7335 7335 28672 2012-06-05 22:59 .
drwxr-xr-x 2 7335 7335  8192 2012-06-05 14:50 128_S_0947_20080422_MPRAGE
drwxrwxr-x 3 7335 7335    18 2012-04-23 13:11 ..

Tue Jun 05 23:18:26 [2.82 2.42 1.67][457.62/606]  root at bduc-login:~
557 $ ssh pbs3 'ls -lat /bducgl/tvanerp/adni/dicom'
total 28
drwxrwxr-x 3 7335 7335 28672 2012-06-05 23:00 .
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:52 128_S_0947_20080422_MPRAGE
drwxrwxr-x 3 7335 7335    18 2012-04-23 13:11 ..

Tue Jun 05 23:18:51 [2.73 2.43 1.69][457.7/606]  root at bduc-login:~
558 $ ssh pbs4 'ls -lat /bducgl/tvanerp/adni/dicom'
total 32
drwxrwxr-x 3 7335 7335 36864 2012-06-05 22:59 .
drwxr-xr-x 2 7335 7335 12288 2012-06-05 14:50 128_S_0947_20080422_MPRAGE
---------T 2 root root     0 2012-05-20 18:35 adni_pib_subjects.txt
drwxrwxr-x 3 7335 7335    18 2012-04-23 13:11 ..

but the client is not able to see/delete them, even as root.

Suggestions?

harry




-- 
Harry Mangalam,Research Computing, OIT,
Rm 225 MSTB, UC Irvine [mailcode 2225]
Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
Lat/Long: (33.642025,-117.844414) [paste into Google Maps]
--

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20120605/8d9fd407/attachment.html>