[Gluster-users] undeleteble directory

tetSU tetsucceed at gmail.com
Thu Dec 1 08:12:44 UTC 2016 

Hello

i have some strange problem that happens on one of the servers with
glusterfs.

Configuration:

-- gluster volume info --

Volume Name: gisdata
Type: Distribute
Volume ID: a49bcb69-fcc5-4f39-b556-e23e29da38e7
Status: Started
Number of Bricks: 7
Transport-type: tcp
Bricks:
Brick1: 10.1.0.4:/export/sdc1/brick
Brick2: 10.1.0.5:/export/sdc1/brick
Brick3: 10.1.0.6:/export/sdc1/brick
Brick4: 10.1.0.7:/export/sdc1/brick
Brick5: 10.1.0.9:/export/sdc1/brick
Brick6: 10.1.0.10:/export/sdc1/brick
Brick7: 10.1.0.12:/export/sdc1/brick
Options Reconfigured:
server.statedump-path: /tmp
performance.cache-size: 512MB
nfs.drc: off

System mount to master server as

10.1.0.4:/gisdata on /usr/local/mountpoint type fuse.glusterfs
(rw,default_permissions,allow_other,max_read=131072)

and its share by samba to the client who manage files on distribute
filesystem.

Client tryes to delete directory (Data/Projects/DELIVERY/D39) but it fails
with error: Can't delete, directory is not empty.
When i inspect that directory on the Bricks i found it totaly broken

 ls -lah /export/sdc1/brick/Data/Projects/DELIVERY/D39/
ls: cannot access
/export/sdc1/brick/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO: No such
file or directory
total 16K
drwxr-xr-x  3 root 500 8.0K Dec  1 10:52 .
drwxr-xr-x 40 root 500 4.0K Nov 30 22:43 ..
??????????  ? ?    ?      ?            ? 494-11-23-1_WD_PO

When i recreate that directory it looks fine but has doubled:

ls -lai /export/sdc1/brick/Data/Projects/DELIVERY/D39/
total 16
  30075838 drwxr-xr-x  4 root  500 8192 Dec  1 10:56 .
   9185992 drwxr-xr-x 40 root  500 4096 Nov 30 22:43 ..
2147631741 drwxr-xr-x  2 root root    6 Dec  1 10:56 494-11-23-1_WD_PO
2147631741 drwxr-xr-x  2 root root    6 Dec  1 10:56 494-11-23-1_WD_PO

Last logs:

/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.352895] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.355764] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.358448] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 10:09:53.381921] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.843840] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.846111] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.848384] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-10-28 18:36:00.857213] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.678051] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.680209] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.682785] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-02 15:34:50.697692] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.880597] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.885190] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.889059] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/4494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 04:18:51.905035] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.985591] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.990133] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:48.995346] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-09 18:15:49.012616] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.278130] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.283321] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.286403] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/052886588010_01_P001_PSH
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-15 21:37:20.305513] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO/052886588010_01/GIS_FILES
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 09:52:26.319979] E
[dht-rebalance.c:1496:gf_defrag_fix_layout] 0-gisdata-dht:
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO lookup failed
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.172294] I
[dht-layout.c:640:dht_layout_normalize] 0-gisdata-dht: found anomalies in
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO. holes=1 overlaps=0
/var/log/glusterfs/gisdata-rebalance.log:[2016-11-30 20:47:39.173616] I
[dht-common.c:2592:dht_setxattr] 0-gisdata-dht: fixing the layout of
/Data/Projects/DELIVERY/D39/494-11-23-1_WD_PO



It happens on server 10.1.0.9. Any ideas how to fix that?

-- 
_________________________
- tetsucceed
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161201/3ff04dca/attachment.html>

More information about the Gluster-users mailing list