[Gluster-devel] Crash in glusterd!!!

[ABHISHEK PALIWAL]

Hi Team,

We are getting the crash in glusterd after start of it. When I tried to
debug in brick logs we are getting below errors:

[2017-12-01 14:10:14.684122] E [MSGID: 100018]
[glusterfsd.c:1960:glusterfs_pidfile_update] 0-glusterfsd: pidfile
/system/glusterd/vols/c_glusterfs/run/10.32.1.144-opt-lvmdir-c2-brick.pid
lock failed [Resource temporarily unavailable]
:
:
:
[2017-12-01 14:10:16.862903] E [MSGID: 113001]
[posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=18:
key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:16.862985] I [MSGID: 115063]
[server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 92:
FTRUNCATE 1 (934f08b7-e3b5-4690-84fc-742a4b1fb78b)==> (No space left on
device) [No space left on device]
[2017-12-01 14:10:16.907037] E [MSGID: 113001]
[posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17:
key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:16.907108] I [MSGID: 115063]
[server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 35:
FTRUNCATE 0 (109d6537-a1ec-4556-8ce1-04c365c451eb)==> (No space left on
device) [No space left on device]
[2017-12-01 14:10:16.947541] E [MSGID: 113001]
[posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17:
key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:16.947623] I [MSGID: 115063]
[server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 70:
FTRUNCATE 0 (8f9c8054-b0d7-4b93-a95b-cd3ab249c56d)==> (No space left on
device) [No space left on device]
[2017-12-01 14:10:16.968515] E [MSGID: 113001]
[posix.c:4616:_posix_remove_xattr] 0-c_glusterfs-posix: removexattr failed
on
/opt/lvmdir/c2/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/configuration
(for trusted.glusterfs.dht) [No space left on device]
[2017-12-01 14:10:16.968589] I [MSGID: 115058]
[server-rpc-fops.c:740:server_removexattr_cbk] 0-c_glusterfs-server: 90:
REMOVEXATTR <gfid:a240d2fd-869c-408d-9b95-62ee1bff074e>
(a240d2fd-869c-408d-9b95-62ee1bff074e) of key  ==> (No space left on
device) [No space left on device]
[2017-12-01 14:10:17.039815] E [MSGID: 113001]
[posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17:
key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:17.039900] I [MSGID: 115063]
[server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 152:
FTRUNCATE 0 (d67bcfcd-ff19-4b58-9823-46d6cce9ace3)==> (No space left on
device) [No space left on device]
[2017-12-01 14:10:17.048767] E [MSGID: 113001]
[posix-helpers.c:1228:posix_fhandle_pair] 0-c_glusterfs-posix: fd=17:
key:trusted.bit-rot.version [No space left on device]
[2017-12-01 14:10:17.048874] I [MSGID: 115063]
[server-rpc-fops.c:1317:server_ftruncate_cbk] 0-c_glusterfs-server: 163:
FTRUNCATE 0 (0e3ee6ad-408b-4fcf-a1a7-4262ec113316)==> (No space left on
device) [No space left on device]
[2017-12-01 14:10:17.075007] E [MSGID: 113001]
[posix.c:4616:_posix_remove_xattr] 0-c_glusterfs-posix: removexattr failed
on
/opt/lvmdir/c2/brick/.glusterfs/00/00/00000000-0000-0000-0000-000000000001/java
(for trusted.glusterfs.dht) [No space left on device]

Also, we are having the lack disk space.

Could any one please explain me what glusterd is doing in brick so that it
is causing of its crash.

Please find the brick logs in attachment.

Thanks in advance!!!
-- 
Regards
Abhishek Paliwal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20171206/50a53c66/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: opt-lvmdir-c2-brick.log
Type: text/x-log
Size: 3794343 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20171206/50a53c66/attachment-0001.bin>