[Gluster-users] Gluster logs filling the disk.

[Ernie Dunbar]

We had one of our gluster servers in the cluster fail on us yesterday, 
and now one (and only one) of the other servers in the cluster has 
managed to collect about 7 gigabytes of logs in the past 12 hours, 
seemingly only with lines like this:

[2016-05-20 16:08:05.119529] I [dict.c:473:dict_get] 
(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xac) 
[0x7f3ece2bd17c] 
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7) 
[0x7f3ec2ca6877] 
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac) 
[0x7f3ece2ad91c] ) 0-dict: !this || key=() [Invalid argument]
[2016-05-20 16:08:05.121478] I [dict.c:473:dict_get] 
(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xac) 
[0x7f3ece2bd17c] 
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7) 
[0x7f3ec2ca6877] 
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac) 
[0x7f3ece2ad91c] ) 0-dict: !this || key=() [Invalid argument]
[2016-05-20 16:08:05.123640] I [dict.c:473:dict_get] 
(-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_getxattr_cbk+0xac) 
[0x7f3ece2bd17c] 
-->/usr/lib/x86_64-linux-gnu/glusterfs/3.7.11/xlator/features/marker.so(marker_getxattr_cbk+0xa7) 
[0x7f3ec2ca6877] 
-->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get+0xac) 
[0x7f3ece2ad91c] ) 0-dict: !this || key=() [Invalid argument]


I've set the loglevel to INFO for the brick, and I'm still getting all 
this. It's streaming in at an insane rate. Perhaps the logger could note 
that this one message has been logged 27,000,000 times in the past 17 
seconds, instead of spewing into the log files like this? Nevermind that 
this is pure debug information that no user could ever decipher.

We're using Ubuntu and Gluster v3.7.11.