[Gluster-users] self-healing problem on gluster-3.3.1

keith keith at science.nus.edu.sg
Mon Apr 8 09:33:08 UTC 2013 

Hi all,

I have one particular replicated volume keep getting these errors in the 
log.

glustershd.log:

[2013-04-08 17:18:30.178495] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.179174] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.180011] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.180735] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.181520] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.182282] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.183042] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.183800] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.184699] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.185371] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.186137] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.186867] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory
[2013-04-08 17:18:30.187638] W 
[client3_1-fops.c:592:client3_1_unlink_cbk] 0-CSEVOL-client-1: remote 
operation failed: No such file or directory

glusterfs/bricks/EXPORT-CSEVOL.log:

[2013-04-08 17:18:30.197321] E [posix.c:224:posix_stat] 0-CSEVOL-posix: 
lstat on /EXPORT/CSEVOL/.glusterfs/35/14/351416f3-e0c2-4a4e-952e-83
4c98ba6093 failed: No such file or directory
[2013-04-08 17:18:30.197351] I [server3_1-fops.c:1085:server_unlink_cbk] 
0-CSEVOL-server: 145425303: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6
4e4a5580495>/8d46ac2f-0b48-4080-a539-6c4ced346567 
(351416f3-e0c2-4a4e-952e-834c98ba6093) ==> -1 (No such file or directory)
[2013-04-08 17:18:30.197922] W 
[marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot 
add a new contribution node
[2013-04-08 17:18:30.198055] E [posix.c:224:posix_stat] 0-CSEVOL-posix: 
lstat on /EXPORT/CSEVOL/.glusterfs/b1/ab/b1abf4ad-b7a6-4171-af0e-47
e9809b9d7e failed: No such file or directory
[2013-04-08 17:18:30.198091] I [server3_1-fops.c:1085:server_unlink_cbk] 
0-CSEVOL-server: 145425305: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6
4e4a5580495>/135cf78e-d4ad-4c2a-b7e3-66a79322e982 
(b1abf4ad-b7a6-4171-af0e-47e9809b9d7e) ==> -1 (No such file or directory)
[2013-04-08 17:18:30.198681] W 
[marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot 
add a new contribution node
[2013-04-08 17:18:30.198810] E [posix.c:224:posix_stat] 0-CSEVOL-posix: 
lstat on /EXPORT/CSEVOL/.glusterfs/36/fe/36fe047c-dcd9-4c94-b9a1-50
fc6b4d4d24 failed: No such file or directory
[2013-04-08 17:18:30.198839] I [server3_1-fops.c:1085:server_unlink_cbk] 
0-CSEVOL-server: 145425307: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6
4e4a5580495>/34fe4e2b-3731-45f0-916c-87852c4af4b4 
(36fe047c-dcd9-4c94-b9a1-50fc6b4d4d24) ==> -1 (No such file or directory)
[2013-04-08 17:18:30.199496] W 
[marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot 
add a new contribution node
[2013-04-08 17:18:30.199627] E [posix.c:224:posix_stat] 0-CSEVOL-posix: 
lstat on /EXPORT/CSEVOL/.glusterfs/88/3e/883e86df-432c-4631-8977-9e
a5b3a46c7d failed: No such file or directory
[2013-04-08 17:18:30.199662] I [server3_1-fops.c:1085:server_unlink_cbk] 
0-CSEVOL-server: 145425309: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6
4e4a5580495>/412da580-0b2c-42f6-9dc5-b8baf18c1f81 
(883e86df-432c-4631-8977-9ea5b3a46c7d) ==> -1 (No such file or directory)
[2013-04-08 17:18:30.200213] W 
[marker-quota.c:2047:mq_inspect_directory_xattr] 0-CSEVOL-marker: cannot 
add a new contribution node
[2013-04-08 17:18:30.200345] E [posix.c:224:posix_stat] 0-CSEVOL-posix: 
lstat on /EXPORT/CSEVOL/.glusterfs/bb/af/bbaf760f-86ea-4b74-b817-c2
818823df3d failed: No such file or directory
[2013-04-08 17:18:30.200381] I [server3_1-fops.c:1085:server_unlink_cbk] 
0-CSEVOL-server: 145425311: UNLINK <gfid:8c48c031-929e-41c4-b8e4-6
4e4a5580495>/a7a313e9-3b0d-4118-9575-af50a1d9fccf 
(bbaf760f-86ea-4b74-b817-c2818823df3d) ==> -1 (No such file or directory)
[2013-04-08 17:18:38.445324] E [posix-helpers.c:701:posix_handle_pair] 
0-CSEVOL-posix: 
/EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: 
key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime 
error:No such file or directory
[2013-04-08 17:18:38.462209] E [posix-helpers.c:701:posix_handle_pair] 
0-CSEVOL-posix: 
/EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: 
key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime 
error:No such file or directory
[2013-04-08 17:18:50.625541] E [posix-helpers.c:701:posix_handle_pair] 
0-CSEVOL-posix: 
/EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: 
key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime 
error:No such file or directory
[2013-04-08 17:19:02.784643] E [posix-helpers.c:701:posix_handle_pair] 
0-CSEVOL-posix: 
/EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: 
key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime 
error:No such file or directory
[2013-04-08 17:19:02.790217] E [posix-helpers.c:701:posix_handle_pair] 
0-CSEVOL-posix: 
/EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: 
key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime 
error:No such file or directory
[2013-04-08 17:19:14.968659] E [posix-helpers.c:701:posix_handle_pair] 
0-CSEVOL-posix: 
/EXPORT/CSEVOL/.glusterfs/b3/6c/b36c3525-4416-4cfe-9583-f25646d548f4: 
key:trusted.glusterfs.0ced464e-554f-4840-82e3-1bfb50d6df36.xtime 
error:No such file or directory

My volume config:
Volume Name: CSEVOL
Type: Replicate
Volume ID: 0ced464e-554f-4840-82e3-1bfb50d6df36
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: t4-01.store:/EXPORT/CSEVOL
Brick2: t4-03.store:/EXPORT/CSEVOL
Options Reconfigured:
network.ping-timeout: 5
geo-replication.indexing: on
features.quota: on
features.limit-usage: ( i have some quotas set)

Anyone has any idea how these errors are generated?  This 2 logs are 
growing too fast that weekly logrotate is insufficient.

Appreciate if anybody can give some advise.

Thanks.

Keith

More information about the Gluster-users mailing list