<div dir="ltr"><div>Sam,<br></div>You might want to give glusterfs-3.12.1 a try instead.<br><br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 15, 2017 at 6:42 AM, Sam McLeod <span dir="ltr"><<a href="mailto:mailinglists@smcleod.net" target="_blank">mailinglists@smcleod.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Howdy,<br>
<br>
I'm setting up several gluster 3.12 clusters running on CentOS 7 and have having issues with glusterd.log and glustershd.log both being filled with errors relating to null client errors and client-callback functions.<br>
<br>
They seem to be related to high CPU usage across the nodes although I don't have a way of confirming that (suggestions welcomed!).<br>
<br>
<br>
in /var/log/glusterfs/glusterd.<wbr>log:<br>
<br>
csvc_request_init+0x7f) [0x7f382007b93f] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x179) [0x7f3820315e59] ) 0-client_t: null client [Invalid argument]<br>
[2017-09-15 00:54:14.454022] E [client_t.c:324:gf_client_ref] (-->/lib64/libgfrpc.so.0(<wbr>rpcsvc_request_create+0xf8) [0x7f382007e7e8] -->/lib64/libgfrpc.so.0(<wbr>rpcsvc_request_init+0x7f) [0x7f382007b93f] -->/lib64/libglusterfs.so.0(<wbr>gf_client_ref+0x179) [0x7f3820315e59] ) 0-client_t: null client [Invalid argument]<br>
<br>
<br>
This is repeated _thousands_ of times and is especially noisey when any node is running gluster volume set <volname> <option> <value>.<br>
<br>
and I'm unsure if it's related but in /var/log/glusterfs/glustershd.<wbr>log:<br>
<br>
[2017-09-15 00:36:21.654242] W [MSGID: 114010] [client-callback.c:28:client_<wbr>cbk_fetchspec] 0-my_volume_name-client-0: this function should not be called<br>
<br>
<br>
---<br>
<br>
<br>
Cluster configuration:<br>
<br>
Gluster 3.12<br>
CentOS 7.4<br>
Replica 3, Arbiter 1<br>
NFS disabled (using Kubernetes with the FUSE client)<br>
Each node is 8 Xeon E5-2660 with 16GB RAM virtualised on XenServer 7.2<br>
<br>
<br>
root@int-gluster-03:~ # gluster get<br>
glusterd state dumped to /var/run/gluster/glusterd_<wbr>state_20170915_110532<br>
<br>
[Global]<br>
MYUUID: 0b42ffb2-217a-4db6-96bf-<wbr>cf304a0fa1ae<br>
op-version: 31200<br>
<br>
[Global options]<br>
cluster.brick-multiplex: enable<br>
<br>
[Peers]<br>
Peer1.primary_hostname: int-gluster-02.fqdn.here<br>
Peer1.uuid: e614686d-0654-43c9-90ca-<wbr>42bcbeda3255<br>
Peer1.state: Peer in Cluster<br>
Peer1.connected: Connected<br>
Peer1.othernames:<br>
Peer2.primary_hostname: int-gluster-01.fqdn.here<br>
Peer2.uuid: 9b0c82ef-329d-4bd5-92fc-<wbr>95e2e90204a6<br>
Peer2.state: Peer in Cluster<br>
Peer2.connected: Connected<br>
Peer2.othernames:<br>
<br>
(Then volume options are listed)<br>
<br>
<br>
---<br>
<br>
<br>
Volume configuration:<br>
<br>
root@int-gluster-03:~ # gluster volume info my_volume_name<br>
<br>
Volume Name: my_volume_name<br>
Type: Replicate<br>
Volume ID: 6574a963-3210-404b-97e2-<wbr>bcff0fa9f4c9<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: int-gluster-01.fqdn.here:/mnt/<wbr>gluster-storage/my_volume_name<br>
Brick2: int-gluster-02.fqdn.here:/mnt/<wbr>gluster-storage/my_volume_name<br>
Brick3: int-gluster-03.fqdn.here:/mnt/<wbr>gluster-storage/my_volume_name<br>
Options Reconfigured:<br>
performance.stat-prefetch: true<br>
performance.parallel-readdir: true<br>
performance.client-io-threads: true<br>
network.ping-timeout: 5<br>
diagnostics.client-log-level: WARNING<br>
diagnostics.brick-log-level: WARNING<br>
cluster.readdir-optimize: true<br>
cluster.lookup-optimize: true<br>
transport.address-family: inet<br>
nfs.disable: on<br>
cluster.brick-multiplex: enable<br>
<br>
<br>
--<br>
Sam McLeod<br>
@s_mcleod<br>
<a href="https://smcleod.net" rel="noreferrer" target="_blank">https://smcleod.net</a><br>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Milind<br><br></div></div></div></div>
</div>