[Gluster-users] logfile flooding

Thu Nov 6 09:32:14 UTC 2014

Hi all,

I like to report an "kind of glitchy" behaviour of glusterfs I experienced today,
and would ask for a work around for it.

I'm running 
glusterfs 3.4.5 built on Aug  6 2014 19:15:07
from
deb http://ppa.launchpad.net/semiosis/ubuntu-glusterfs-3.4/ubuntu trusty main
in replica 2 mode on two up-to-date ubuntu 14.04 servers (<ip_srv_1> and <ip_srv_2>)
A client, also, ubuntu 14.04, same gluster version, fuse mounts
a volume like this:

/bin/mount -t glusterfs <ip_srv_1>:/<volume_name> <mountpoint>

I had several time stopped the client (unmounted the volume)
updated (and rebooted) first server, than the otherone, remounted the volume
in the client and all went nice and smooth as expected.

This time I indeed for the first time unmounted the volume in the client
updated/rebooted <ip_srv_2>, mounted the volume from <ip_srv_2>
in the client and then updated rebooted <ip_srv_1>.

That should work, right? and it does. All data on the volume seems to be okay
as far as I can tell.

BUT:
The logfiles on the client and server get flooded like this:

server:
/var/log/glusterfs/bricks/<volume_name>brick.log:

[2014-11-06 06:18:41.417901] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.418003] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280274: REMOVEXATTR /<path>/.<file>.qjStjE (852ed4f2-c821-4a08-9254-11ec987d1233) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.423443] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.423497] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280285: REMOVEXATTR /<path>/.<file>.qjStjE (852ed4f2-c821-4a08-9254-11ec987d1233) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.426623] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.426675] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280294: REMOVEXATTR /<path>/.<file>.qjStjE (852ed4f2-c821-4a08-9254-11ec987d1233) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.429518] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.429594] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280302: REMOVEXATTR /<path>/.<file>.qjStjE (852ed4f2-c821-4a08-9254-11ec987d1233) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.436899] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.436921] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280328: REMOVEXATTR /<path>/.<file>.zLuCBE (357ba81d-a774-462f-9e49-e3d646f3afcd) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.453168] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.453255] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280354: REMOVEXATTR /<path>/.<file>.zLuCBE (357ba81d-a774-462f-9e49-e3d646f3afcd) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.456518] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.456569] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280363: REMOVEXATTR /<path>/.<file>.zLuCBE (357ba81d-a774-462f-9e49-e3d646f3afcd) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.459100] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.459148] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280372: REMOVEXATTR /<path>/.<file>.zLuCBE (357ba81d-a774-462f-9e49-e3d646f3afcd) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.691399] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.691442] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280570: REMOVEXATTR /<path>/.<file>.Kb2vBF (69d2a022-0d12-45dd-ad5b-24c206471e8f) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.707157] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.707181] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280603: REMOVEXATTR /<path>/.<file>.Kb2vBF (69d2a022-0d12-45dd-ad5b-24c206471e8f) of key security.ima ==> (No data available)
[2014-11-06 06:18:41.708946] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:41.708972] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33280611: REMOVEXATTR /<path>/.<file>.Kb2vBF (69d2a022-0d12-45dd-ad5b-24c206471e8f) of key security.ima ==> (No data available)
[2014-11-06 06:18:43.826544] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:43.826587] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33282135: REMOVEXATTR /<path>/.<file>.rJfnCM (4dfb3cf7-80d8-4278-b3e0-5b726fa9cf0d) of key security.ima ==> (No data available)
[2014-11-06 06:18:43.832881] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:43.832933] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33282153: REMOVEXATTR /<path>/.<file>.rJfnCM (4dfb3cf7-80d8-4278-b3e0-5b726fa9cf0d) of key security.ima ==> (No data available)
[2014-11-06 06:18:43.835039] E [marker.c:2140:marker_removexattr_cbk] 0-gldata_<volume_name>-marker: No data available occurred while creating symlinks
[2014-11-06 06:18:43.835104] I [server-rpc-fops.c:745:server_removexattr_cbk] 0-gldata_<volume_name>-server: 33282162: REMOVEXATTR /<path>/.<file>.rJfnCM (4dfb3cf7-80d8-4278-b3e0-5b726fa9cf0d) of key security.ima ==> (No data available)6

client:
/var/log/glusterfs/srv-<volume_name>.log:

[2014-11-06 06:18:41.418521] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.418694] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.419664] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178626: REMOVEXATTR() /<path>/.<file>.qjStjE => -1 (No data available)
[2014-11-06 06:18:41.423948] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.423989] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.424837] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178635: REMOVEXATTR() /<path>/.<file>.qjStjE => -1 (No data available)
[2014-11-06 06:18:41.427270] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.427539] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.428103] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178641: REMOVEXATTR() /<path>/.<file>.qjStjE => -1 (No data available)
[2014-11-06 06:18:41.429955] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.430082] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.430441] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178645: REMOVEXATTR() /<path>/.<file>.qjStjE => -1 (No data available)
[2014-11-06 06:18:41.437450] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.437500] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.437854] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178663: REMOVEXATTR() /<path>/.<file>.zLuCBE => -1 (No data available)
[2014-11-06 06:18:41.453801] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.453830] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.454373] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178688: REMOVEXATTR() /<path>/.<file>.zLuCBE => -1 (No data available)
[2014-11-06 06:18:41.457040] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.457115] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.457673] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178693: REMOVEXATTR() /<path>/.<file>.zLuCBE => -1 (No data available)
[2014-11-06 06:18:41.459532] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.459647] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.460125] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178699: REMOVEXATTR() /<path>/.<file>.zLuCBE => -1 (No data available)
[2014-11-06 06:18:41.691852] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.692116] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.692711] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178894: REMOVEXATTR() /<path>/.<file>.Kb2vBF => -1 (No data available)
[2014-11-06 06:18:41.707673] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.707716] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.708004] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178923: REMOVEXATTR() /<path>/.<file>.Kb2vBF => -1 (No data available)
[2014-11-06 06:18:41.709422] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-1: remote operation failed: No data available
[2014-11-06 06:18:41.709468] W [client-rpc-fops.c:1232:client3_3_removexattr_cbk] 0-gldata_<volume_name>-client-0: remote operation failed: No data available
[2014-11-06 06:18:41.709783] W [fuse-bridge.c:1172:fuse_err_cbk] 0-glusterfs-fuse: 37178929: REMOVEXATTR() /<path>/.<file>.Kb2vBF => -1 (No data available)

As I said above, the data itself seems to be okay,
but how can I stop the log message flood?

TIA

Bernhard

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141106/f329fe7f/attachment.html>