<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">FYI - I've been testing the Gluster 3.12.1 packages with the help of the SIG maintainer and I can confirm that the logs are no longer being filled with NFS or null client errors after the upgrade.<br class=""><div class="">
<div dir="auto" style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="">--<br class="">Sam McLeod <br class="">@s_mcleod<br class=""><a href="https://smcleod.net" class="">https://smcleod.net</a></div></div>
</div>
<div><br class=""><blockquote type="cite" class=""><div class="">On 18 Sep 2017, at 10:14 pm, Sam McLeod <<a href="mailto:mailinglists@smcleod.net" class="">mailinglists@smcleod.net</a>> wrote:</div><br class="Apple-interchange-newline"><div class=""><meta http-equiv="content-type" content="text/html; charset=utf-8" class=""><div dir="auto" class="">Thanks Milind,<div class=""><br class=""></div><div class="">Yes I’m hanging out for CentOS’s Storage / Gluster SIG to release the packages for 3.12.1, I can see the packages were built a week ago but they’re still not on the repo :(<br class=""><br class=""><div class="">--<div class="">Sam</div></div><div class=""><br class="">On 18 Sep 2017, at 9:57 pm, Milind Changire <<a href="mailto:mchangir@redhat.com" class="">mchangir@redhat.com</a>> wrote:<br class=""><br class=""></div><blockquote type="cite" class=""><div class=""><div dir="ltr" class=""><div class="">Sam,<br class=""></div>You might want to give glusterfs-3.12.1 a try instead.<br class=""><br class=""><br class=""></div><div class="gmail_extra"><br class=""><div class="gmail_quote">On Fri, Sep 15, 2017 at 6:42 AM, Sam McLeod <span dir="ltr" class=""><<a href="mailto:mailinglists@smcleod.net" target="_blank" class="">mailinglists@smcleod.net</a>></span> wrote:<br class=""><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Howdy,<br class="">
<br class="">
I'm setting up several gluster 3.12 clusters running on CentOS 7 and have having issues with glusterd.log and glustershd.log both being filled with errors relating to null client errors and client-callback functions.<br class="">
<br class="">
They seem to be related to high CPU usage across the nodes although I don't have a way of confirming that (suggestions welcomed!).<br class="">
<br class="">
<br class="">
in /var/log/glusterfs/glusterd.<wbr class="">log:<br class="">
<br class="">
csvc_request_init+0x7f) [0x7f382007b93f] -->/lib64/libglusterfs.so.0(<wbr class="">gf_client_ref+0x179) [0x7f3820315e59] ) 0-client_t: null client [Invalid argument]<br class="">
[2017-09-15 00:54:14.454022] E [client_t.c:324:gf_client_ref] (-->/lib64/libgfrpc.so.0(<wbr class="">rpcsvc_request_create+0xf8) [0x7f382007e7e8] -->/lib64/libgfrpc.so.0(<wbr class="">rpcsvc_request_init+0x7f) [0x7f382007b93f] -->/lib64/libglusterfs.so.0(<wbr class="">gf_client_ref+0x179) [0x7f3820315e59] ) 0-client_t: null client [Invalid argument]<br class="">
<br class="">
<br class="">
This is repeated _thousands_ of times and is especially noisey when any node is running gluster volume set <volname> <option> <value>.<br class="">
<br class="">
and I'm unsure if it's related but in /var/log/glusterfs/glustershd.<wbr class="">log:<br class="">
<br class="">
[2017-09-15 00:36:21.654242] W [MSGID: 114010] [client-callback.c:28:client_<wbr class="">cbk_fetchspec] 0-my_volume_name-client-0: this function should not be called<br class="">
<br class="">
<br class="">
---<br class="">
<br class="">
<br class="">
Cluster configuration:<br class="">
<br class="">
Gluster 3.12<br class="">
CentOS 7.4<br class="">
Replica 3, Arbiter 1<br class="">
NFS disabled (using Kubernetes with the FUSE client)<br class="">
Each node is 8 Xeon E5-2660 with 16GB RAM virtualised on XenServer 7.2<br class="">
<br class="">
<br class="">
root@int-gluster-03:~ # gluster get<br class="">
glusterd state dumped to /var/run/gluster/glusterd_<wbr class="">state_20170915_110532<br class="">
<br class="">
[Global]<br class="">
MYUUID: 0b42ffb2-217a-4db6-96bf-<wbr class="">cf304a0fa1ae<br class="">
op-version: 31200<br class="">
<br class="">
[Global options]<br class="">
cluster.brick-multiplex: enable<br class="">
<br class="">
[Peers]<br class="">
Peer1.primary_hostname: int-gluster-02.fqdn.here<br class="">
Peer1.uuid: e614686d-0654-43c9-90ca-<wbr class="">42bcbeda3255<br class="">
Peer1.state: Peer in Cluster<br class="">
Peer1.connected: Connected<br class="">
Peer1.othernames:<br class="">
Peer2.primary_hostname: int-gluster-01.fqdn.here<br class="">
Peer2.uuid: 9b0c82ef-329d-4bd5-92fc-<wbr class="">95e2e90204a6<br class="">
Peer2.state: Peer in Cluster<br class="">
Peer2.connected: Connected<br class="">
Peer2.othernames:<br class="">
<br class="">
(Then volume options are listed)<br class="">
<br class="">
<br class="">
---<br class="">
<br class="">
<br class="">
Volume configuration:<br class="">
<br class="">
root@int-gluster-03:~ # gluster volume info my_volume_name<br class="">
<br class="">
Volume Name: my_volume_name<br class="">
Type: Replicate<br class="">
Volume ID: 6574a963-3210-404b-97e2-<wbr class="">bcff0fa9f4c9<br class="">
Status: Started<br class="">
Snapshot Count: 0<br class="">
Number of Bricks: 1 x 3 = 3<br class="">
Transport-type: tcp<br class="">
Bricks:<br class="">
Brick1: int-gluster-01.fqdn.here:/mnt/<wbr class="">gluster-storage/my_volume_name<br class="">
Brick2: int-gluster-02.fqdn.here:/mnt/<wbr class="">gluster-storage/my_volume_name<br class="">
Brick3: int-gluster-03.fqdn.here:/mnt/<wbr class="">gluster-storage/my_volume_name<br class="">
Options Reconfigured:<br class="">
performance.stat-prefetch: true<br class="">
performance.parallel-readdir: true<br class="">
performance.client-io-threads: true<br class="">
network.ping-timeout: 5<br class="">
diagnostics.client-log-level: WARNING<br class="">
diagnostics.brick-log-level: WARNING<br class="">
cluster.readdir-optimize: true<br class="">
cluster.lookup-optimize: true<br class="">
transport.address-family: inet<br class="">
nfs.disable: on<br class="">
cluster.brick-multiplex: enable<br class="">
<br class="">
<br class="">
--<br class="">
Sam McLeod<br class="">
@s_mcleod<br class="">
<a href="https://smcleod.net/" rel="noreferrer" target="_blank" class="">https://smcleod.net</a><br class="">
<br class="">
______________________________<wbr class="">_________________<br class="">
Gluster-users mailing list<br class="">
<a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank" class="">http://lists.gluster.org/<wbr class="">mailman/listinfo/gluster-users</a><br class="">
</blockquote></div><br class=""><br clear="all" class=""><br class="">-- <br class=""><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr" class=""><div class=""><div dir="ltr" class="">Milind<br class=""><br class=""></div></div></div></div>
</div>
</div></blockquote></div></div>_______________________________________________<br class="">Gluster-users mailing list<br class=""><a href="mailto:Gluster-users@gluster.org" class="">Gluster-users@gluster.org</a><br class="">http://lists.gluster.org/mailman/listinfo/gluster-users</div></blockquote></div><br class=""></body></html>