[Gluster-users] Using the host name of the volume, its related commands can become very slow
Atin Mukherjee
amukherj at redhat.com
Tue Jan 16 02:53:49 UTC 2018
On Mon, Jan 15, 2018 at 6:30 PM, 陈曦 <chenxi at shudun.com> wrote:
> Using the host name of the volume, its related gluster commands can become
> very slow .For example,create,start,stop volume,nfs related commands. and
> some time And in some cases, the command will return Error : Request timed
> out
> but If using ip address to create the volume. The volume all gluster
> commands are normal.
>
> I have configured /etc/hosts correctly,Because,SSH can normally use the
> hostname to access another node from one node. And,I can ping other nodes
> use hostanme。
>
> this is some logs:
> [root at f08n25glusterfs]# tail -n 50 data-gluster-test.log
> [2018-02-03 13:53:22.777184] I [MSGID: 100030] [glusterfsd.c:2338:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20
> (args: /usr/sbin/glusterfs --volfile-server=localhost --volfile-id=test
> /data/gluster/test)
> [2018-02-03 13:53:22.810249] I [MSGID: 101190] [event-epoll.c:632:event_dispatch_epoll_worker]
> 0-epoll: Started thread with index 1
> [2018-02-03 13:53:22.811289] E [glusterfsd-mgmt.c:1590:mgmt_getspec_cbk]
> 0-glusterfs: failed to get the 'volume file' from server
> [2018-02-03 13:53:22.811323] E [glusterfsd-mgmt.c:1690:mgmt_getspec_cbk]
> 0-mgmt: failed to fetch volume file (key:test)
> [2018-02-03 13:53:22.811847] W [glusterfsd.c:1251:cleanup_and_exit]
> (-->/lib64/libgfrpc.so.0(rpc_clnt_handle_reply+0xa0) [0xffffad0cff84]
> -->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x3f8) [0xaaaad6e1f734]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-:
> received signum (0), shutting down
> [2018-02-03 13:53:22.811892] I [fuse-bridge.c:5720:fini] 0-fuse:
> Unmounting '/data/gluster/test'.
> [2018-02-03 13:53:22.914173] W [glusterfsd.c:1251:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7bb0) [0xffffaceb7bb0]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0x11c) [0xaaaad6e19bc0]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x78) [0xaaaad6e199fc] ) 0-:
> received signum (15), shutting down
>
> [root at f08n25glusterfs]# tail -n 50 etc-glusterfs-glusterd.vol.log
> [2018-02-03 13:55:09.106663] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/W_W
> on port 49195
> [2018-02-03 13:55:09.108871] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/QAQ
> on port 49183
> [2018-02-03 13:55:09.111075] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/v_v
> on port 49171
> [2018-02-03 13:55:09.113281] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/QAQ
> on port 49184
> [2018-02-03 13:55:09.205465] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/OTZ
> on port 49210
> [2018-02-03 13:55:09.207655] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/c26580f1-e879-4225-8b3b-e3a3738130a0/A_s
> on port 49219
> [2018-02-03 13:55:09.209824] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/v_v
> on port 49172
> [2018-02-03 13:55:09.212030] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/W_W
> on port 49188
> [2018-02-03 13:55:09.214255] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/W_W
> on port 49196
> [2018-02-03 13:55:09.216464] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/QAQ
> on port 49185
> [2018-02-03 13:55:09.218656] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/37313921-a9b2-41f1-b2ff-493485c2b449/OTZ
> on port 49203
> [2018-02-03 13:55:09.220827] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/Q
> on port 49152
> [2018-02-03 13:55:09.223005] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/Q
> on port 49159
> [2018-02-03 13:55:09.225209] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/v_v
> on port 49173
> [2018-02-03 13:55:09.227452] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/QAQ
> on port 49186
> [2018-02-03 13:55:09.229654] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-44a1-bb0b-594294b9c5c4/W_W
> on port 49189
> [2018-02-03 13:55:09.231855] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/W_W
> on port 49197
> [2018-02-03 13:55:09.234073] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/v_v
> on port 49174
> [2018-02-03 13:55:09.236285] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/OTZ
> on port 49211
> [2018-02-03 13:55:09.238463] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/A_s
> on port 49212
> [2018-02-03 13:55:09.240933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-44a1-bb0b-594294b9c5c4/Q
> on port 49153
> [2018-02-03 13:55:09.243114] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d5da7794-2654-4845-b8ba-a2ed01c04b41/A_s
> on port 49220
> [2018-02-03 13:55:09.245295] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/55e4259d-8fc9-456b-861f-617be3683cd0/OTZ
> on port 49204
> [2018-02-03 13:55:09.247480] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/QAQ
> on port 49187
> [2018-02-03 13:55:09.249670] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/Q
> on port 49160
> [2018-02-03 13:55:09.251854] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/v_v
> on port 49175
> [2018-02-03 13:55:09.254376] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/2f74a970-372c-4fda-95e5-5d0de352966d/W_W
> on port 49190
> [2018-02-03 13:55:09.256604] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/2a5f45bd-28a8-44a1-bb0b-594294b9c5c4/A_s
> on port 49213
> [2018-02-03 13:55:09.258780] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/dfd7e810-d4f8-4b4b-b5d1-53e5652f1a8f/W_W
> on port 49198
> [2018-02-03 13:55:09.260979] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/a2a12678-7a4e-4e0a-8e4e-a96dedc0217d/OTZ
> on port 49205
> [2018-02-03 13:55:09.263155] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/2f74a970-372c-4fda-95e5-5d0de352966d/Q
> on port 49154
> [2018-02-03 13:55:09.265360] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/Q
> on port 49161
> [2018-02-03 13:55:09.267556] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d60a681a-af2a-40ae-9732-9902bd2be614/A_s
> on port 49221
> [2018-02-03 13:55:09.269730] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/37313921-a9b2-41f1-b2ff-493485c2b449/W_W
> on port 49191
> [2018-02-03 13:55:09.271933] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/QAQ
> on port 49176
> [2018-02-03 13:55:09.274159] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/f7b86039-c3e9-425f-8d9e-d063b380058e/W_W
> on port 49199
> [2018-02-03 13:55:09.276368] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/00812b9d-0c7a-4960-a9b9-389a98424bce/v_v
> on port 49164
> [2018-02-03 13:55:09.278578] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/d793e3ff-220b-4d94-865c-99ad16953403/A_s
> on port 49222
> [2018-02-03 13:55:09.280747] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/2f74a970-372c-4fda-95e5-5d0de352966d/A_s
> on port 49215
> [2018-02-03 13:55:09.282913] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/37313921-a9b2-41f1-b2ff-493485c2b449/Q
> on port 49155
> [2018-02-03 13:55:09.285117] I [MSGID: 106143] [glusterd-pmap.c:231:pmap_registry_bind]
> 0-pmap: adding brick /data/gluster/c26580f1-e879-4225-8b3b-e3a3738130a0/OTZ
> on port 49206
> [2018-02-03 13:55:09.315499] I [MSGID: 106490] [glusterd-handler.c:2600:__
> glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from
> uuid: d8ee609f-d47e-46e4-96ac-ef78cddf45b8
> [2018-02-03 13:55:09.325999] I [MSGID: 106493] [glusterd-handler.c:3843:glusterd_xfer_friend_add_resp]
> 0-glusterd: Responded to f08n29 (0), ret: 0, op_ret: 0
> [2018-02-03 13:55:09.331799] I [MSGID: 106492] [glusterd-handler.c:2776:__glusterd_handle_friend_update]
> 0-glusterd: Received friend update from uuid: d8ee609f-d47e-46e4-96ac-
> ef78cddf45b8
> [2018-02-03 13:55:09.331876] I [MSGID: 106502] [glusterd-handler.c:2821:__glusterd_handle_friend_update]
> 0-management: Received my uuid as Friend
> [2018-02-03 13:55:09.333350] I [MSGID: 106493] [glusterd-rpc-ops.c:696:__glusterd_friend_update_cbk]
> 0-management: Received ACC from uuid: d8ee609f-d47e-46e4-96ac-ef78cddf45b8
> [2018-02-03 13:55:38.833207] I [MSGID: 106487] [glusterd-handler.c:1472:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
> [2018-02-03 13:57:09.161466] I [MSGID: 106487] [glusterd-handler.c:1472:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
> [2018-02-03 13:58:39.566130] I [MSGID: 106487] [glusterd-handler.c:1472:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
> [2018-02-03 14:00:09.918411] I [MSGID: 106487] [glusterd-handler.c:1472:__
> glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
>
The snippet of the glusterd log file what you pointed is during the
handshaking. What we'd need to check the logs for a particular transaction.
Would you be able to share the glusterd log files from all the nodes along
with the cmd_history.log from the node where the commands were initiated?
>
> [root at f08n25glusterfs]# tail -n 50 glustershd.log
> 2478: option remote-host 10.33.0.8
> 2479: option remote-subvolume /data/gluster/ec025105-5ef2-
> 46f2-b1cf-01a05daa237e/QAQ
> 2480: option transport-type socket
> 2481: option username 2a31c3ba-696d-4d49-9dee-540e47791e4a
> 2482: option password 3f751501-3148-42bf-8c55-bfe1b3ddc747
> 2483: end-volume
> 2484:
> 2485: volume QAQ-replicate-11
> 2486: type cluster/replicate
> 2487: option node-uuid 194ca8ea-df5b-4d5e-9af6-dbf36c485334
> 2488: option background-self-heal-count 0
> 2489: option metadata-self-heal on
> 2490: option data-self-heal on
> 2491: option entry-self-heal on
> 2492: option self-heal-daemon enable
> 2493: option iam-self-heal-daemon yes
> 2494: subvolumes QAQ-client-33 QAQ-client-34 QAQ-client-35
> 2495: end-volume
> 2496:
> 2497: volume glustershd
> 2498: type debug/io-stats
> 2499: option log-level INFO
> 2500: subvolumes A_s-replicate-0 A_s-replicate-1 A_s-replicate-2
> A_s-replicate-3 A_s-replicate-4 A_s-replicate-5 A_s-replicate-6
> A_s-replicate-7 A_s-replicate-8 A_s-replicate-9 A_s-replicate-10
> A_s-replicate-11 OTZ-disperse-0 OTZ-disperse-1 OTZ-disperse-2
> OTZ-disperse-3 OTZ-disperse-4 OTZ-disperse-5 OTZ-disperse-6 OTZ-disperse-7
> OTZ-disperse-8 OTZ-disperse-9 OTZ-disperse-10 OTZ-disperse-11 Q-disperse-0
> Q-disperse-1 Q-disperse-2 Q-disperse-3 Q-disperse-4 Q-disperse-5
> Q-disperse-6 Q-disperse-7 Q-disperse-8 Q-disperse-9 Q-disperse-10
> Q-disperse-11 QAQ-replicate-0 QAQ-replicate-1 QAQ-replicate-2
> QAQ-replicate-3 QAQ-replicate-4 QAQ-replicate-5 QAQ-replicate-6
> QAQ-replicate-7 QAQ-replicate-8 QAQ-replicate-9 QAQ-replicate-10
> QAQ-replicate-11
> 2501: end-volume
> 2502:
> +-----------------------------------------------------------
> -------------------+
> [2018-02-03 13:55:20.771993] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-0: Going UP
> [2018-02-03 13:55:20.772100] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-1: Going UP
> [2018-02-03 13:55:20.772157] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-2: Going UP
> [2018-02-03 13:55:20.772208] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-3: Going UP
> [2018-02-03 13:55:20.772260] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-4: Going UP
> [2018-02-03 13:55:20.772311] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-5: Going UP
> [2018-02-03 13:55:20.772364] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-6: Going UP
> [2018-02-03 13:55:20.772418] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-7: Going UP
> [2018-02-03 13:55:20.772468] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-8: Going UP
> [2018-02-03 13:55:20.772521] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-9: Going UP
> [2018-02-03 13:55:20.772572] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-10: Going UP
> [2018-02-03 13:55:20.772626] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-OTZ-disperse-11: Going UP
> [2018-02-03 13:55:20.772679] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-0: Going UP
> [2018-02-03 13:55:20.772732] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-1: Going UP
> [2018-02-03 13:55:20.772786] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-2: Going UP
> [2018-02-03 13:55:20.772838] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-3: Going UP
> [2018-02-03 13:55:20.772891] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-4: Going UP
> [2018-02-03 13:55:20.772941] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-5: Going UP
> [2018-02-03 13:55:20.772994] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-6: Going UP
> [2018-02-03 13:55:20.773046] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-7: Going UP
> [2018-02-03 13:55:20.773099] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-8: Going UP
> [2018-02-03 13:55:20.773150] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-9: Going UP
> [2018-02-03 13:55:21.773401] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-10: Going UP
> [2018-02-03 13:55:21.773496] I [MSGID: 122061] [ec.c:313:ec_up]
> 0-Q-disperse-11: Going UP
> [root at f08n25glusterfs]#
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180116/7609fb09/attachment.html>
More information about the Gluster-users
mailing list