[Gluster-users] different gfid on nodes

[Tamas Papp]

hi All,

I try to remove some directories from our cluster, but ther eis an error:

$ rm -rf _torolni
rm: cannot remove 
`_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001': 
Directory not empty
rm: cannot remove 
`_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800': 
Directory not empty




 From the log of the client:

[2014-07-23 17:25:58.532161] W [dht-common.c:419:dht_lookup_dir_cbk] 
0-w-vol-dht: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: 
gfid different on w-vol-client-1
[2014-07-23 17:25:58.532247] W [dht-common.c:419:dht_lookup_dir_cbk] 
0-w-vol-dht: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: 
gfid different on w-vol-client-3
[2014-07-23 17:25:58.532277] W [dht-common.c:419:dht_lookup_dir_cbk] 
0-w-vol-dht: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: 
gfid different on w-vol-client-4
[2014-07-23 17:25:58.532458] W [dht-common.c:419:dht_lookup_dir_cbk] 
0-w-vol-dht: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001: 
gfid different on w-vol-client-0
[2014-07-23 17:25:58.533283] W 
[client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-2: remote 
operation failed: No such file or directory. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001 
(63cf6edf-f1ba-4657-acd5-c28d225bddf1)
[2014-07-23 17:25:58.543416] W 
[client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-2: remote 
operation failed: No such file or directory. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001 
(63cf6edf-f1ba-4657-acd5-c28d225bddf1)
[2014-07-23 17:25:58.557966] W 
[client-rpc-fops.c:327:client3_3_mkdir_cbk] 0-w-vol-client-2: remote 
operation failed: File exists. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/delete/co/cp/T05_APPL_020-co_cp_v001 
(63cf6edf-f1ba-4657-acd5-c28d225bddf1)
[2014-07-23 17:25:58.653867] W [dht-common.c:419:dht_lookup_dir_cbk] 
0-w-vol-dht: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800: 
gfid different on w-vol-client-1
[2014-07-23 17:25:58.669396] W 
[client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-1: remote 
operation failed: No such file or directory. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 
(6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c)
[2014-07-23 17:25:58.676365] W 
[client-rpc-fops.c:2541:client3_3_opendir_cbk] 0-w-vol-client-1: remote 
operation failed: No such file or directory. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 
(6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c)
[2014-07-23 17:25:58.677956] W 
[client-rpc-fops.c:327:client3_3_mkdir_cbk] 0-w-vol-client-1: remote 
operation failed: File exists. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 
(6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c)
[2014-07-23 17:25:58.678121] W 
[client-rpc-fops.c:327:client3_3_mkdir_cbk] 0-w-vol-client-3: remote 
operation failed: File exists. Path: 
/_torolni/60_Elements/Prod/T06_APPL/T06_APPL_PACKSHOT/T06_APPL_PACKSHOT_010/co/cp/T06_APPL_060-co_cp_v001/f_800x800 
(6fcb6c71-348f-4e14-bdf4-a0eb5b3df66c)





I found this tool:
https://github.com/vbellur/gfid

Probably it will help.
Why did that happen? Will it happen in the future as well, is this a 
persistent bug?


Thanks,
tamas