<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jan 15, 2018 at 6:30 PM, 陈曦 <span dir="ltr"><<a href="mailto:chenxi@shudun.com" target="_blank">chenxi@shudun.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div>Using the host name of the volume, its related gluster commands can become very slow .For example,create,start,stop volume,nfs related commands. and some time And in some cases, the command will return Error : Request timed out</div><div>but If using ip address to create the volume. The volume all gluster commands are normal.</div><div><br></div><div>I have configured /etc/hosts correctly,Because,SSH can normally use the hostname to access another node from one node. And,I can ping other nodes use hostanme。</div><div><br></div><div>this is some logs:</div><div><div>[root@f08n25glusterfs]# tail -n 50 data-gluster-test.log </div><div>[2018-02-03 13:53:22.777184] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=test /data/gluster/test)</div><div>[2018-02-03 13:53:22.810249] I [MSGID: 101190] [event-epoll.c:632:event_<wbr>dispatch_epoll_worker] 0-epoll: Started thread with index 1</div><div>[2018-02-03 13:53:22.811289] E [glusterfsd-mgmt.c:1590:mgmt_<wbr>getspec_cbk] 0-glusterfs: failed to get the 'volume file' from server</div><div>[2018-02-03 13:53:22.811323] E [glusterfsd-mgmt.c:1690:mgmt_<wbr>getspec_cbk] 0-mgmt: failed to fetch volume file (key:test)</div><div>[2018-02-03 13:53:22.811847] W [glusterfsd.c:1251:cleanup_<wbr>and_exit] (-->/lib64/libgfrpc.so.0(rpc_<wbr>clnt_handle_reply+0xa0) [0xffffad0cff84] -->/usr/sbin/glusterfs(mgmt_<wbr>getspec_cbk+0x3f8) [0xaaaad6e1f734] -->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-: received signum (0), shutting down</div><div>[2018-02-03 13:53:22.811892] I [fuse-bridge.c:5720:fini] 0-fuse: Unmounting '/data/gluster/test'.</div><div>[2018-02-03 13:53:22.914173] W [glusterfsd.c:1251:cleanup_<wbr>and_exit] (-->/lib64/libpthread.so.0(+<wbr>0x7bb0) [0xffffaceb7bb0] -->/usr/sbin/glusterfs(<wbr>glusterfs_sigwaiter+0x11c) [0xaaaad6e19bc0] -->/usr/sbin/glusterfs(<wbr>cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-: received signum (15), shutting down</div></div><div><br></div><div><div>[root@f08n25glusterfs]# tail -n 50 etc-glusterfs-glusterd.vol.log </div><div>[2018-02-03 13:55:09.106663] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/W_W on port 49195</div><div>[2018-02-03 13:55:09.108871] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/QAQ on port 49183</div><div>[2018-02-03 13:55:09.111075] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/v_v on port 49171</div><div>[2018-02-03 13:55:09.113281] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/QAQ on port 49184</div><div>[2018-02-03 13:55:09.205465] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/OTZ on port 49210</div><div>[2018-02-03 13:55:09.207655] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/c26580f1-e879-<wbr>4225-8b3b-e3a3738130a0/A_s on port 49219</div><div>[2018-02-03 13:55:09.209824] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/v_v on port 49172</div><div>[2018-02-03 13:55:09.212030] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/W_W on port 49188</div><div>[2018-02-03 13:55:09.214255] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/W_W on port 49196</div><div>[2018-02-03 13:55:09.216464] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/QAQ on port 49185</div><div>[2018-02-03 13:55:09.218656] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-<wbr>41f1-b2ff-493485c2b449/OTZ on port 49203</div><div>[2018-02-03 13:55:09.220827] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/Q on port 49152</div><div>[2018-02-03 13:55:09.223005] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/Q on port 49159</div><div>[2018-02-03 13:55:09.225209] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/v_v on port 49173</div><div>[2018-02-03 13:55:09.227452] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/QAQ on port 49186</div><div>[2018-02-03 13:55:09.229654] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-<wbr>44a1-bb0b-594294b9c5c4/W_W on port 49189</div><div>[2018-02-03 13:55:09.231855] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/W_W on port 49197</div><div>[2018-02-03 13:55:09.234073] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/v_v on port 49174</div><div>[2018-02-03 13:55:09.236285] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/OTZ on port 49211</div><div>[2018-02-03 13:55:09.238463] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/A_s on port 49212</div><div>[2018-02-03 13:55:09.240933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-<wbr>44a1-bb0b-594294b9c5c4/Q on port 49153</div><div>[2018-02-03 13:55:09.243114] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d5da7794-2654-<wbr>4845-b8ba-a2ed01c04b41/A_s on port 49220</div><div>[2018-02-03 13:55:09.245295] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/55e4259d-8fc9-<wbr>456b-861f-617be3683cd0/OTZ on port 49204</div><div>[2018-02-03 13:55:09.247480] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/QAQ on port 49187</div><div>[2018-02-03 13:55:09.249670] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/Q on port 49160</div><div>[2018-02-03 13:55:09.251854] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/v_v on port 49175</div><div>[2018-02-03 13:55:09.254376] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-<wbr>4fda-95e5-5d0de352966d/W_W on port 49190</div><div>[2018-02-03 13:55:09.256604] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-<wbr>44a1-bb0b-594294b9c5c4/A_s on port 49213</div><div>[2018-02-03 13:55:09.258780] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-<wbr>4b4b-b5d1-53e5652f1a8f/W_W on port 49198</div><div>[2018-02-03 13:55:09.260979] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/a2a12678-7a4e-<wbr>4e0a-8e4e-a96dedc0217d/OTZ on port 49205</div><div>[2018-02-03 13:55:09.263155] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-<wbr>4fda-95e5-5d0de352966d/Q on port 49154</div><div>[2018-02-03 13:55:09.265360] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/Q on port 49161</div><div>[2018-02-03 13:55:09.267556] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d60a681a-af2a-<wbr>40ae-9732-9902bd2be614/A_s on port 49221</div><div>[2018-02-03 13:55:09.269730] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-<wbr>41f1-b2ff-493485c2b449/W_W on port 49191</div><div>[2018-02-03 13:55:09.271933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/QAQ on port 49176</div><div>[2018-02-03 13:55:09.274159] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/f7b86039-c3e9-<wbr>425f-8d9e-d063b380058e/W_W on port 49199</div><div>[2018-02-03 13:55:09.276368] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/00812b9d-0c7a-<wbr>4960-a9b9-389a98424bce/v_v on port 49164</div><div>[2018-02-03 13:55:09.278578] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/d793e3ff-220b-<wbr>4d94-865c-99ad16953403/A_s on port 49222</div><div>[2018-02-03 13:55:09.280747] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/2f74a970-372c-<wbr>4fda-95e5-5d0de352966d/A_s on port 49215</div><div>[2018-02-03 13:55:09.282913] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/37313921-a9b2-<wbr>41f1-b2ff-493485c2b449/Q on port 49155</div><div>[2018-02-03 13:55:09.285117] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_<wbr>registry_bind] 0-pmap: adding brick /data/gluster/c26580f1-e879-<wbr>4225-8b3b-e3a3738130a0/OTZ on port 49206</div><div>[2018-02-03 13:55:09.315499] I [MSGID: 106490] [glusterd-handler.c:2600:__<wbr>glusterd_handle_incoming_<wbr>friend_req] 0-glusterd: Received probe from uuid: d8ee609f-d47e-46e4-96ac-<wbr>ef78cddf45b8</div><div>[2018-02-03 13:55:09.325999] I [MSGID: 106493] [glusterd-handler.c:3843:<wbr>glusterd_xfer_friend_add_resp] 0-glusterd: Responded to f08n29 (0), ret: 0, op_ret: 0</div><div>[2018-02-03 13:55:09.331799] I [MSGID: 106492] [glusterd-handler.c:2776:__<wbr>glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: d8ee609f-d47e-46e4-96ac-<wbr>ef78cddf45b8</div><div>[2018-02-03 13:55:09.331876] I [MSGID: 106502] [glusterd-handler.c:2821:__<wbr>glusterd_handle_friend_update] 0-management: Received my uuid as Friend</div><div>[2018-02-03 13:55:09.333350] I [MSGID: 106493] [glusterd-rpc-ops.c:696:__<wbr>glusterd_friend_update_cbk] 0-management: Received ACC from uuid: d8ee609f-d47e-46e4-96ac-<wbr>ef78cddf45b8</div><div>[2018-02-03 13:55:38.833207] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div>[2018-02-03 13:57:09.161466] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div>[2018-02-03 13:58:39.566130] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div><div>[2018-02-03 14:00:09.918411] I [MSGID: 106487] [glusterd-handler.c:1472:__<wbr>glusterd_handle_cli_list_<wbr>friends] 0-glusterd: Received cli list req</div></div></blockquote><div><br></div><div>The snippet of the glusterd log file what you pointed is during the handshaking. What we'd need to check the logs for a particular transaction. Would you be able to share the glusterd log files from all the nodes along with the cmd_history.log from the node where the commands were initiated?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><br></div><div><br></div><div><div>[root@f08n25glusterfs]# tail -n 50 glustershd.log </div><div>2478: option remote-host 10.33.0.8</div><div>2479: option remote-subvolume /data/gluster/ec025105-5ef2-<wbr>46f2-b1cf-01a05daa237e/QAQ</div><div>2480: option transport-type socket</div><div>2481: option username 2a31c3ba-696d-4d49-9dee-<wbr>540e47791e4a</div><div>2482: option password 3f751501-3148-42bf-8c55-<wbr>bfe1b3ddc747</div><div>2483: end-volume</div><div>2484: </div><div>2485: volume QAQ-replicate-11</div><div>2486: type cluster/replicate</div><div>2487: option node-uuid 194ca8ea-df5b-4d5e-9af6-<wbr>dbf36c485334</div><div>2488: option background-self-heal-count 0</div><div>2489: option metadata-self-heal on</div><div>2490: option data-self-heal on</div><div>2491: option entry-self-heal on</div><div>2492: option self-heal-daemon enable</div><div>2493: option iam-self-heal-daemon yes</div><div>2494: subvolumes QAQ-client-33 QAQ-client-34 QAQ-client-35</div><div>2495: end-volume</div><div>2496: </div><div>2497: volume glustershd</div><div>2498: type debug/io-stats</div><div>2499: option log-level INFO</div><div>2500: subvolumes A_s-replicate-0 A_s-replicate-1 A_s-replicate-2 A_s-replicate-3 A_s-replicate-4 A_s-replicate-5 A_s-replicate-6 A_s-replicate-7 A_s-replicate-8 A_s-replicate-9 A_s-replicate-10 A_s-replicate-11 OTZ-disperse-0 OTZ-disperse-1 OTZ-disperse-2 OTZ-disperse-3 OTZ-disperse-4 OTZ-disperse-5 OTZ-disperse-6 OTZ-disperse-7 OTZ-disperse-8 OTZ-disperse-9 OTZ-disperse-10 OTZ-disperse-11 Q-disperse-0 Q-disperse-1 Q-disperse-2 Q-disperse-3 Q-disperse-4 Q-disperse-5 Q-disperse-6 Q-disperse-7 Q-disperse-8 Q-disperse-9 Q-disperse-10 Q-disperse-11 QAQ-replicate-0 QAQ-replicate-1 QAQ-replicate-2 QAQ-replicate-3 QAQ-replicate-4 QAQ-replicate-5 QAQ-replicate-6 QAQ-replicate-7 QAQ-replicate-8 QAQ-replicate-9 QAQ-replicate-10 QAQ-replicate-11</div><div>2501: end-volume</div><div>2502: </div><div>+-----------------------------<wbr>------------------------------<wbr>-------------------+</div><div>[2018-02-03 13:55:20.771993] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-0: Going UP</div><div>[2018-02-03 13:55:20.772100] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-1: Going UP</div><div>[2018-02-03 13:55:20.772157] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-2: Going UP</div><div>[2018-02-03 13:55:20.772208] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-3: Going UP</div><div>[2018-02-03 13:55:20.772260] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-4: Going UP</div><div>[2018-02-03 13:55:20.772311] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-5: Going UP</div><div>[2018-02-03 13:55:20.772364] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-6: Going UP</div><div>[2018-02-03 13:55:20.772418] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-7: Going UP</div><div>[2018-02-03 13:55:20.772468] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-8: Going UP</div><div>[2018-02-03 13:55:20.772521] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-9: Going UP</div><div>[2018-02-03 13:55:20.772572] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-10: Going UP</div><div>[2018-02-03 13:55:20.772626] I [MSGID: 122061] [ec.c:313:ec_up] 0-OTZ-disperse-11: Going UP</div><div>[2018-02-03 13:55:20.772679] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-0: Going UP</div><div>[2018-02-03 13:55:20.772732] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-1: Going UP</div><div>[2018-02-03 13:55:20.772786] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-2: Going UP</div><div>[2018-02-03 13:55:20.772838] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-3: Going UP</div><div>[2018-02-03 13:55:20.772891] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-4: Going UP</div><div>[2018-02-03 13:55:20.772941] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-5: Going UP</div><div>[2018-02-03 13:55:20.772994] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-6: Going UP</div><div>[2018-02-03 13:55:20.773046] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-7: Going UP</div><div>[2018-02-03 13:55:20.773099] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-8: Going UP</div><div>[2018-02-03 13:55:20.773150] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-9: Going UP</div><div>[2018-02-03 13:55:21.773401] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-10: Going UP</div><div>[2018-02-03 13:55:21.773496] I [MSGID: 122061] [ec.c:313:ec_up] 0-Q-disperse-11: Going UP</div><div>[root@f08n25glusterfs]# </div></div><div><br></div><div><br></div><div><br></div><div><u></u><u></u></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div></div>