<div dir="auto">Any suggestion....</div><div class="gmail_extra"><br><div class="gmail_quote">On Dec 6, 2017 11:51, "ABHISHEK PALIWAL" <<a href="mailto:abhishpaliwal@gmail.com">abhishpaliwal@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Team,<div><br></div><div>We are getting the crash in glusterd after start of it. When I tried to debug in brick logs we are getting below errors:</div><div><br></div><div>[2017-12-01 14:10:14.684122] E [MSGID: 100018] [glusterfsd.c:1960:glusterfs_<wbr>pidfile_update] 0-glusterfsd: pidfile /system/glusterd/vols/c_<wbr>glusterfs/run/10.32.1.144-opt-<wbr>lvmdir-c2-brick.pid lock failed [Resource temporarily unavailable]<br></div><div>:</div><div>:</div><div><div>:</div><div><div>[2017-12-01 14:10:16.862903] E [MSGID: 113001] [posix-helpers.c:1228:posix_<wbr>fhandle_pair] 0-c_glusterfs-posix: fd=18: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:16.862985] I [MSGID: 115063] [server-rpc-fops.c:1317:<wbr>server_ftruncate_cbk] 0-c_glusterfs-server: 92: FTRUNCATE 1 (934f08b7-e3b5-4690-84fc-<wbr>742a4b1fb78b)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:16.907037] E [MSGID: 113001] [posix-helpers.c:1228:posix_<wbr>fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:16.907108] I [MSGID: 115063] [server-rpc-fops.c:1317:<wbr>server_ftruncate_cbk] 0-c_glusterfs-server: 35: FTRUNCATE 0 (109d6537-a1ec-4556-8ce1-<wbr>04c365c451eb)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:16.947541] E [MSGID: 113001] [posix-helpers.c:1228:posix_<wbr>fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:16.947623] I [MSGID: 115063] [server-rpc-fops.c:1317:<wbr>server_ftruncate_cbk] 0-c_glusterfs-server: 70: FTRUNCATE 0 (8f9c8054-b0d7-4b93-a95b-<wbr>cd3ab249c56d)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:16.968515] E [MSGID: 113001] [posix.c:4616:_posix_remove_<wbr>xattr] 0-c_glusterfs-posix: removexattr failed on /opt/lvmdir/c2/brick/.<wbr>glusterfs/00/00/00000000-0000-<wbr>0000-0000-000000000001/<wbr>configuration (for trusted.glusterfs.dht) [No space left on device]</div><div>[2017-12-01 14:10:16.968589] I [MSGID: 115058] [server-rpc-fops.c:740:server_<wbr>removexattr_cbk] 0-c_glusterfs-server: 90: REMOVEXATTR <gfid:a240d2fd-869c-408d-9b95-<wbr>62ee1bff074e> (a240d2fd-869c-408d-9b95-<wbr>62ee1bff074e) of key ==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:17.039815] E [MSGID: 113001] [posix-helpers.c:1228:posix_<wbr>fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:17.039900] I [MSGID: 115063] [server-rpc-fops.c:1317:<wbr>server_ftruncate_cbk] 0-c_glusterfs-server: 152: FTRUNCATE 0 (d67bcfcd-ff19-4b58-9823-<wbr>46d6cce9ace3)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:17.048767] E [MSGID: 113001] [posix-helpers.c:1228:posix_<wbr>fhandle_pair] 0-c_glusterfs-posix: fd=17: key:trusted.bit-rot.version [No space left on device]</div><div>[2017-12-01 14:10:17.048874] I [MSGID: 115063] [server-rpc-fops.c:1317:<wbr>server_ftruncate_cbk] 0-c_glusterfs-server: 163: FTRUNCATE 0 (0e3ee6ad-408b-4fcf-a1a7-<wbr>4262ec113316)==> (No space left on device) [No space left on device]</div><div>[2017-12-01 14:10:17.075007] E [MSGID: 113001] [posix.c:4616:_posix_remove_<wbr>xattr] 0-c_glusterfs-posix: removexattr failed on /opt/lvmdir/c2/brick/.<wbr>glusterfs/00/00/00000000-0000-<wbr>0000-0000-000000000001/java (for trusted.glusterfs.dht) [No space left on device]</div></div><div><br></div><div>Also, we are having the lack disk space.</div><div><br></div><div>Could any one please explain me what glusterd is doing in brick so that it is causing of its crash.</div><div><br></div><div>Please find the brick logs in attachment.</div><div><br></div><div>Thanks in advance!!!</div>-- <br><div class="m_3854133520433708890gmail_signature"><div dir="ltr">Regards<br>
Abhishek Paliwal<br>
</div></div>
</div></div>
</blockquote></div></div>