<div><div><font __editorwarp__="1" style="display: inline; font-size: 14px; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-weight: 400; font-style: normal;">Thanks for your quick response</font></div><div><br></div><div><font __editorwarp__="1" style="display: inline; background-color: rgba(0, 0, 0, 0);">I have executed some commond again,</font><span style="color: rgb(51, 51, 51); font-family: Verdana, Arial;">reproduce the problem.</span></div><div><font __editorwarp__="1" style="display: inline; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-style: normal;" size="4"><b style="">Execute this command on all nodes(3 nodes)</b></font></div><div><font __editorwarp__="1" style="display: inline; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-style: normal;" size="4"><b style=""><br></b></font></div><div><b>[root@f08n25glusterfs]</b># rm -rf /var/log/glusterfs/*</div><div><br></div><div><b>[root@f08n25glusterfs]</b># rm -rf /var/lib/glusterd/*</div><div><br></div><div><b>[root@f08n25glusterfs]</b># service glusterd restart</div><div>Restarting glusterd (via systemctl): [ OK ]</div></div><div><br></div><div><b><font size="4">Execute this command on one nodes</font></b></div><div><br></div><div><div><b>[root@f08n25/]</b># ping f08n33 -c 10</div><div>PING f08n33 (10.33.0.31) 56(84) bytes of data.</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=1 ttl=64 time=0.064 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=2 ttl=64 time=0.049 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=3 ttl=64 time=0.052 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=4 ttl=64 time=0.051 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=5 ttl=64 time=0.062 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=6 ttl=64 time=0.058 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=7 ttl=64 time=0.071 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=8 ttl=64 time=0.089 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=9 ttl=64 time=0.061 ms</div><div>64 bytes from f08n33 (10.33.0.31): icmp_seq=10 ttl=64 time=0.075 ms</div><div><br></div><div>--- f08n33 ping statistics ---</div><div>10 packets transmitted, 10 received, 0% packet loss, time 9000ms</div><div>rtt min/avg/max/mdev = 0.049/0.063/0.089/0.012 ms</div><div><br></div><div><div><b>[root@f08n25/]</b># ping f08n29 -c 10</div><div>PING f08n29 (10.33.0.30) 56(84) bytes of data.</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=1 ttl=64 time=0.065 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=2 ttl=64 time=0.056 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=3 ttl=64 time=0.058 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=4 ttl=64 time=0.081 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=5 ttl=64 time=0.059 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=6 ttl=64 time=0.055 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=7 ttl=64 time=0.056 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=8 ttl=64 time=0.060 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=9 ttl=64 time=0.057 ms</div><div>64 bytes from f08n29 (10.33.0.30): icmp_seq=10 ttl=64 time=0.061 ms</div><div><br></div><div>--- f08n29 ping statistics ---</div><div>10 packets transmitted, 10 received, 0% packet loss, time 9016ms</div><div>rtt min/avg/max/mdev = 0.055/0.060/0.081/0.012 ms</div></div></div><div><br></div><div><br></div><div><div><b>[root@f08n25/]</b># time gluster peer probe f08n33</div><div>peer probe: success. </div><div><br></div><div>real<span style="white-space:pre"> </span>0m0.264s</div><div>user<span style="white-space:pre"> </span>0m0.190s</div><div>sys<span style="white-space:pre"> </span>0m0.020s</div><div><font __editorwarp__="1" style="display: inline; font-size: 14px; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-weight: 700; font-style: normal;">[root@f08n25/</font>]# time gluster peer probe f08n29</div><div>peer probe: success. </div><div><br></div><div>real<span style="white-space:pre"> </span>0m0.265s</div><div>user<span style="white-space:pre"> </span>0m0.190s</div><div>sys<span style="white-space:pre"> </span>0m0.010s</div><div><br></div><div><b>[root@f08n25/]</b># time gluster volume create test f08n33:/data/gluster/bricks{1..5} f08n29:/data/gluster/bricks{1..5} f08n25:/data/gluster/bricks{1..5} force </div><div>volume create: test: success: please start the volume to access data</div><div><br></div><div>real<span style="white-space:pre"> </span>0m38.601s</div><div>user<span style="white-space:pre"> </span>0m0.170s</div><div>sys<span style="white-space:pre"> </span>0m0.030s</div><div><b>[root@f08n25/]</b># time gluster volume start test</div><div>volume start: test: success</div><div><br></div><div>real<span style="white-space:pre"> </span>0m8.055s</div><div>user<span style="white-space:pre"> </span>0m0.140s</div><div>sys<span style="white-space:pre"> </span>0m0.060s</div><div><b>[root@f08n25/]</b># time gluster volume quota test enable</div><div>volume quota : success</div><div><br></div><div>real<span style="white-space:pre"> </span>0m17.867s</div><div>user<span style="white-space:pre"> </span>0m0.180s</div><div>sys<span style="white-space:pre"> </span>0m0.020s</div><div><br></div><div><b>[root@f08n25/]</b># time gluster volume quota test limit-usage / 10GB</div><div>volume quota : success</div><div><br></div><div>real<span style="white-space:pre"> </span>0m0.880s</div><div>user<span style="white-space:pre"> </span>0m0.180s</div><div>sys<span style="white-space:pre"> </span>0m0.020s</div><div><b>[root@f08n25/]</b># time gluster volume stop test</div><div>Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y</div><div>volume stop: test: success</div><div><br></div><div>real<span style="white-space:pre"> </span>0m11.971s</div><div>user<span style="white-space:pre"> </span>0m0.180s</div><div>sys<span style="white-space:pre"> </span>0m0.020s</div><div><b>[root@f08n25/]</b># time gluster volume start test</div><div>volume start: test: success</div><div><br></div><div>real<span style="white-space:pre"> </span>0m15.165s</div><div>user<span style="white-space:pre"> </span>0m0.190s</div><div>sys<span style="white-space:pre"> </span>0m0.010s</div><div><b>[root@f08n25/]</b># time gluster volume set test nfs.disable ON</div><div>volume set: success</div><div><br></div><div>real<span style="white-space:pre"> </span>0m25.123s</div><div>user<span style="white-space:pre"> </span>0m0.190s</div><div>sys<span style="white-space:pre"> </span>0m0.010s</div></div><div><br></div><div><includetail><div><div><b>[root@f08n25glusterfs]</b># uname -a</div><div>Linux f08n25 4.2.0 #1 SMP Tue Jun 7 01:18:20 CST 2016 aarch64 aarch64 aarch64 GNU/Linux</div><div><br></div><div><b>[root@f08n25glusterfs]</b># cat /etc/redhat-release </div><div>CentOS Linux release 7.3.1611 (AltArch) </div><div><br></div><div><b>[root@f08n25glusterfs]</b># glusterfs --version</div><div>glusterfs 3.7.20 built on Jan 30 2017 16:22:41</div><div>Repository revision: git://git.gluster.com/glusterfs.git</div><div>Copyright (c) 2006-2013 Red Hat, Inc. <http://www.redhat.com/></div><div>GlusterFS comes with ABSOLUTELY NO WARRANTY.</div><div>It is licensed to you under your choice of the GNU Lesser</div><div>General Public License, version 3 or any later version (LGPLv3</div><div>or later), or the GNU General Public License, version 2 (GPLv2),</div><div>in all cases as published by the Free Software Foundation.</div><div><br></div></div><div><br></div><div><b><font size="4"><font __editorwarp__="1" style="display: inline; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-style: normal;">attachment is the log of 3 nodes</font> </font></b></div><div><b><font size="4"><br></font></b></div><div><b><font size="4"><br></font></b></div><div><font __editorwarp__="1" style="display: inline; font-size: 14px; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-weight: 400; font-style: normal;">I can see some errors in the log,such as ,</font></div><div><font __editorwarp__="1" style="display: inline; font-size: 14px; font-family: Verdana; color: rgb(0, 0, 0); background-color: rgba(0, 0, 0, 0); font-weight: 400; font-style: normal;">in quotad.log:</font></div><div><font __editorwarp__="1" style="display: inline; background-color: rgba(0, 0, 0, 0);"><div style="">[2017-12-01 01:42:07.329344] E [MSGID: 114058] [client-handshake.c:1524:client_query_portmap_cbk] 0-test-client-3: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.</div><div style=""><br></div><div style="color: rgb(0, 0, 0); font-family: Verdana; font-size: 14px; font-style: normal; font-weight: 400;">and in etc-glusterfs-glusterd.vol.log:</div><div style="">[2017-12-01 01:31:29.196038] E [MSGID: 106243] [glusterd.c:1656:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport</div><div style="">and so on</div><div style=""><br></div><div style=""><b>Does this cause gluster to run wrong?</b></div><div style=""><br></div><div style=""><br></div><div style="">thank you very much</div><div style="">Look forward to your favourable reply</div><div style="">chenx</div></font></div><div style="font:Verdana normal 14px;color:#000;"><div style="FONT-SIZE: 12px;FONT-FAMILY: Arial Narrow;padding:2px 0 2px 0;">------------------ Original ------------------</div><div style="FONT-SIZE: 12px;background:#efefef;padding:8px;"><div id="menu_sender"><b>From: </b> "Atin Mukherjee"<amukherj@redhat.com>;</div><div><b>Date: </b> Tue, Jan 16, 2018 11:23 AM</div><div><b>To: </b> "陈曦"<chenxi@shudun.com>; <wbr></div><div><b>Cc: </b> "gluster-users"<gluster-users@gluster.org>; <wbr></div><div><b>Subject: </b> Re: [Gluster-users] Using the host name of the volume, its relatedcommands can become very slow</div></div><div> </div><div style="position:relative;"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 15, 2018 at 6:30 PM, 陈曦 <span dir="ltr"><<a href="mailto:chenxi@shudun.com" target="_blank">chenxi@shudun.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Using the host name of the volume, its related gluster commands can become very slow .For example,create,start,stop volume,nfs related commands. and some time And in some cases, the command will return Error : Request timed out</div><div>but If using ip address to create the volume. The volume all gluster commands are normal.</div><div><br></div><div>I have configured /etc/hosts correctly,Because,SSH can normally use the hostname to access another node from one node. And,I can ping other nodes use hostanme。</div><div><br></div><div>this is some logs:</div><div><div>[root@f08n25glusterfs]# tail -n 50 data-gluster-test.log </div><div>[2018-02-03 13:53:22.777184] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=test /data/gluster/test)</div><div>[2018-02-03 13:53:22.810249] I [MSGID: 101190] [event-epoll.c:632:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2018-02-03 13:53:22.811289] E [glusterfsd-mgmt.c:1590:mgmt_<wbr>getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server</div><div>[2018-02-03 13:53:22.811323] E [glusterfsd-mgmt.c:1690:mgmt_<wbr>getspec_cbk] 0-mgmt: failed to fetch volume file (key:test)</div><div>[2018-02-03 13:53:22.811847] W [glusterfsd.c:1251:cleanup_<wbr>and_exit] (-->/lib64/libgfrpc.so.0(rpc_<wbr>clnt_handle_reply+0xa0) [0xffffad0cff84] -->/usr/sbin/glusterfs(mgmt_<wbr>getspec_cbk+0x3f8) [0xaaaad6e1f734] -->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-: received signum (0), shutting down</div><div>[2018-02-03 13:53:22.811892] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/data/gluster/test'.</div><div>[2018-02-03 13:53:22.914173] W [glusterfsd.c:1251:cleanup_<wbr>and_exit] (-->/lib64/libpthread.so.0(+<wbr>0x7bb0) [0xffffaceb7bb0] -->/usr/sbin/glusterfs(<wbr>glusterfs_sigwaiter+0x11c) [0xaaaad6e19bc0] -->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-: received signum (15), shutting down</div></div><div><br></div><div><div>[root@f08n25glusterfs]# tail -n 50 etc-glusterfs-glusterd.vol.log </div><div>[2018-02-03 13:55:09.106663] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/W_W on port 49195</div><div>[2018-02-03 13:55:09.108871] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/QAQ on port 49183</div><div>[2018-02-03 13:55:09.111075] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/v_v on port 49171</div><div>[2018-02-03 13:55:09.113281] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/QAQ on port 49184</div><div>[2018-02-03 13:55:09.205465] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/OTZ on port 49210</div><div>[2018-02-03 13:55:09.207655] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/c26580f1-e879-<wbr>4225-8b3b-e3a3738130a0/A_s on port 49219</div><div>[2018-02-03 13:55:09.209824] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/v_v on port 49172</div><div>[2018-02-03 13:55:09.212030] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/W_W on port 49188</div><div>[2018-02-03 13:55:09.214255] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/W_W on port 49196</div><div>[2018-02-03 13:55:09.216464] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/QAQ on port 49185</div><div>[2018-02-03 13:55:09.218656] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-<wbr>41f1-b2ff-493485c2b449/OTZ on port 49203</div><div>[2018-02-03 13:55:09.220827] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/Q on port 49152</div><div>[2018-02-03 13:55:09.223005] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/Q on port 49159</div><div>[2018-02-03 13:55:09.225209] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/v_v on port 49173</div><div>[2018-02-03 13:55:09.227452] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/QAQ on port 49186</div><div>[2018-02-03 13:55:09.229654] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-<wbr>44a1-bb0b-594294b9c5c4/W_W on port 49189</div><div>[2018-02-03 13:55:09.231855] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/W_W on port 49197</div><div>[2018-02-03 13:55:09.234073] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/v_v on port 49174</div><div>[2018-02-03 13:55:09.236285] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/OTZ on port 49211</div><div>[2018-02-03 13:55:09.238463] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/A_s on port 49212</div><div>[2018-02-03 13:55:09.240933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-<wbr>44a1-bb0b-594294b9c5c4/Q on port 49153</div><div>[2018-02-03 13:55:09.243114] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/A_s on port 49220</div><div>[2018-02-03 13:55:09.245295] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/55e4259d-8fc9-<wbr>456b-861f-617be3683cd0/OTZ on port 49204</div><div>[2018-02-03 13:55:09.247480] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/QAQ on port 49187</div><div>[2018-02-03 13:55:09.249670] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/Q on port 49160</div><div>[2018-02-03 13:55:09.251854] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/v_v on port 49175</div><div>[2018-02-03 13:55:09.254376] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-<wbr>4fda-95e5-5d0de352966d/W_W on port 49190</div><div>[2018-02-03 13:55:09.256604] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-<wbr>44a1-bb0b-594294b9c5c4/A_s on port 49213</div><div>[2018-02-03 13:55:09.258780] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/W_W on port 49198</div><div>[2018-02-03 13:55:09.260979] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/a2a12678-7a4e-<wbr>4e0a-8e4e-a96dedc0217d/OTZ on port 49205</div><div>[2018-02-03 13:55:09.263155] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-<wbr>4fda-95e5-5d0de352966d/Q on port 49154</div><div>[2018-02-03 13:55:09.265360] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/Q on port 49161</div><div>[2018-02-03 13:55:09.267556] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/A_s on port 49221</div><div>[2018-02-03 13:55:09.269730] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-<wbr>41f1-b2ff-493485c2b449/W_W on port 49191</div><div>[2018-02-03 13:55:09.271933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/QAQ on port 49176</div><div>[2018-02-03 13:55:09.274159] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/W_W on port 49199</div><div>[2018-02-03 13:55:09.276368] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/v_v on port 49164</div><div>[2018-02-03 13:55:09.278578] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/A_s on port 49222</div><div>[2018-02-03 13:55:09.280747] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-<wbr>4fda-95e5-5d0de352966d/A_s on port 49215</div><div>[2018-02-03 13:55:09.282913] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-<wbr>41f1-b2ff-493485c2b449/Q on port 49155</div><div>[2018-02-03 13:55:09.285117] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/c26580f1-e879-<wbr>4225-8b3b-e3a3738130a0/OTZ on port 49206</div><div>[2018-02-03 13:55:09.315499] I [MSGID: 106490] [glusterd-handler.c:2600:__<wbr>glusterd_handle_incoming_<wbr>friend_req] 0-glusterd: Received probe from uuid: d8ee609f-d47e-46e4-96ac-<wbr>ef78cddf45b8</div><div>[2018-02-03 13:55:09.325999] I [MSGID: 106493] [glusterd-handler.c:3843:<wbr>glusterd_xfer_friend_add_resp] 0-glusterd: Responded to f08n29 (0), ret: 0, op_ret: 0</div><div>[2018-02-03 13:55:09.331799] I [MSGID: 106492] [glusterd-handler.c:2776:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d8ee609f-d47e-46e4-96ac-<wbr>ef78cddf45b8</div><div>[2018-02-03 13:55:09.331876] I [MSGID: 106502] [glusterd-handler.c:2821:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</div><div>[2018-02-03 13:55:09.333350] I [MSGID: 106493] [glusterd-rpc-ops.c:696:__<wbr>glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d8ee609f-d47e-46e4-96ac-<wbr>ef78cddf45b8</div><div>[2018-02-03 13:55:38.833207] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div>[2018-02-03 13:57:09.161466] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div>[2018-02-03 13:58:39.566130] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div>[2018-02-03 14:00:09.918411] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div></div></blockquote><div><br></div><div>The snippet of the glusterd log file what you pointed is during the handshaking. What we'd need to check the logs for a particular transaction. Would you be able to share the glusterd log files from all the nodes along with the cmd_history.log from the node where the commands were initiated?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div><div><br></div><div><div>[root@f08n25glusterfs]# tail -n 50 glustershd.log </div><div>2478: option remote-host 10.33.0.8</div><div>2479: option remote-subvolume /data/gluster/ec025105-5ef2-<wbr>46f2-b1cf-01a05daa237e/QAQ</div><div>2480: option transport-type socket</div><div>2481: option username 2a31c3ba-696d-4d49-9dee-<wbr>540e47791e4a</div><div>2482: option password 3f751501-3148-42bf-8c55-<wbr>bfe1b3ddc747</div><div>2483: end-volume</div><div>2484: </div><div>2485: volume QAQ-replicate-11</div><div>2486: type cluster/replicate</div><div>2487: option node-uuid 194ca8ea-df5b-4d5e-9af6-<wbr>dbf36c485334</div><div>2488: option background-self-heal-count 0</div><div>2489: option metadata-self-heal on</div><div>2490: option data-self-heal on</div><div>2491: option entry-self-heal on</div><div>2492: option self-heal-daemon enable</div><div>2493: option iam-self-heal-daemon yes</div><div>2494: subvolumes QAQ-client-33 QAQ-client-34 QAQ-client-35</div><div>2495: end-volume</div><div>2496: </div><div>2497: volume glustershd</div><div>2498: type debug/io-stats</div><div>2499: option log-level INFO</div><div>2500: subvolumes A_s-replicate-0 A_s-replicate-1 A_s-replicate-2 A_s-replicate-3 A_s-replicate-4 A_s-replicate-5 A_s-replicate-6 A_s-replicate-7 A_s-replicate-8 A_s-replicate-9 A_s-replicate-10 A_s-replicate-11 OTZ-disperse-0 OTZ-disperse-1 OTZ-disperse-2 OTZ-disperse-3 OTZ-disperse-4 OTZ-disperse-5 OTZ-disperse-6 OTZ-disperse-7 OTZ-disperse-8 OTZ-disperse-9 OTZ-disperse-10 OTZ-disperse-11 Q-disperse-0 Q-disperse-1 Q-disperse-2 Q-disperse-3 Q-disperse-4 Q-disperse-5 Q-disperse-6 Q-disperse-7 Q-disperse-8 Q-disperse-9 Q-disperse-10 Q-disperse-11 QAQ-replicate-0 QAQ-replicate-1 QAQ-replicate-2 QAQ-replicate-3 QAQ-replicate-4 QAQ-replicate-5 QAQ-replicate-6 QAQ-replicate-7 QAQ-replicate-8 QAQ-replicate-9 QAQ-replicate-10 QAQ-replicate-11</div><div>2501: end-volume</div><div>2502: </div><div>+-----------------------------<wbr>------------------------------<wbr>-------------------+</div><div>[2018-02-03 13:55:20.771993] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-0: Going UP</div><div>[2018-02-03 13:55:20.772100] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-1: Going UP</div><div>[2018-02-03 13:55:20.772157] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-2: Going UP</div><div>[2018-02-03 13:55:20.772208] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-3: Going UP</div><div>[2018-02-03 13:55:20.772260] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-4: Going UP</div><div>[2018-02-03 13:55:20.772311] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-5: Going UP</div><div>[2018-02-03 13:55:20.772364] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-6: Going UP</div><div>[2018-02-03 13:55:20.772418] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-7: Going UP</div><div>[2018-02-03 13:55:20.772468] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-8: Going UP</div><div>[2018-02-03 13:55:20.772521] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-9: Going UP</div><div>[2018-02-03 13:55:20.772572] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-10: Going UP</div><div>[2018-02-03 13:55:20.772626] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-11: Going UP</div><div>[2018-02-03 13:55:20.772679] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-0: Going UP</div><div>[2018-02-03 13:55:20.772732] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-1: Going UP</div><div>[2018-02-03 13:55:20.772786] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-2: Going UP</div><div>[2018-02-03 13:55:20.772838] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-3: Going UP</div><div>[2018-02-03 13:55:20.772891] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-4: Going UP</div><div>[2018-02-03 13:55:20.772941] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-5: Going UP</div><div>[2018-02-03 13:55:20.772994] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-6: Going UP</div><div>[2018-02-03 13:55:20.773046] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-7: Going UP</div><div>[2018-02-03 13:55:20.773099] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-8: Going UP</div><div>[2018-02-03 13:55:20.773150] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-9: Going UP</div><div>[2018-02-03 13:55:21.773401] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-10: Going UP</div><div>[2018-02-03 13:55:21.773496] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-11: Going UP</div><div>[root@f08n25glusterfs]# </div></div><div><br></div><div><br></div><div><br></div><div><u></u><u></u></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div>
</div></div><!--<![endif]--></includetail></div>