<div dir="ltr">Hi Team,<div><br></div><div>We are getting the crash in glusterd after start of it. When I tried to debug in brick logs we are getting below errors:</div><div><br></div><div>[2017-12-01 14:10:14.684122] E [MSGID: 100018] [glusterfsd.c:1960:glusterfs_pidfile_update] 0-glusterfsd: pidfile /system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid lock failed [Resource temporarily unavailable]<br></div><div>:</div><div>:</div><div><div>:</div><div><div>[2017-12-01 14:10:16.862903] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=18: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:16.862985] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 92: FTRUNCATE 1 (934f08b7-e3b5-4690-84fc-742a4b1fb78b)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:16.907037] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:16.907108] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 35: FTRUNCATE 0 (109d6537-a1ec-4556-8ce1-04c365c451eb)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:16.947541] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:16.947623] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 70: FTRUNCATE 0 (8f9c8054-b0d7-4b93-a95b-cd3ab249c56d)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:16.968515] E [MSGID: 113001] [posix.c:4616:_posix_remove_xattr] 0-c_glusterfs-posix: removexattr failed on /opt/lvmdir/c2/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/configuration (for trusted.glusterfs.dht) [No space left on device]</div><div>[2017-12-01 14:10:16.968589] I [MSGID: 115058] [server-rpc-fops.c:740:server_removexattr_cbk] 0-c_glusterfs-server: 90: REMOVEXATTR <gfid:a240d2fd-869c-408d-9b95-62ee1bff074e> (a240d2fd-869c-408d-9b95-62ee1bff074e) of key ==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:17.039815] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:17.039900] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 152: FTRUNCATE 0 (d67bcfcd-ff19-4b58-9823-46d6cce9ace3)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:17.048767] E [MSGID: 113001] [posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:17.048874] I [MSGID: 115063] [server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 163: FTRUNCATE 0 (0e3ee6ad-408b-4fcf-a1a7-4262ec113316)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:17.075007] E [MSGID: 113001] [posix.c:4616:_posix_remove_xattr] 0-c_glusterfs-posix: removexattr failed on /opt/lvmdir/c2/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/java (for trusted.glusterfs.dht) [No space left on device]</div></div><div><br></div><div>Also, we are having the lack disk space.</div><div><br></div><div>Could any one please explain me what glusterd is doing in brick so that it is causing of its crash.</div><div><br></div><div>Please find the brick logs in attachment.</div><div><br></div><div>Thanks in advance!!!</div>-- <br><div class="gmail_signature"><div dir="ltr">Regards<br>
Abhishek Paliwal<br>
</div></div>
</div></div>