[Gluster-users] Fwd: VM freeze issue on simple gluster setup.
Ravishankar N
ravishankar at redhat.com
Thu Dec 12 12:34:54 UTC 2019
On 12/12/19 4:01 am, WK wrote:
>
> <BUMP> so I can get some sort of resolution on the issue (i.e. is it
> hardware, Gluster etc)
>
> I guess what I really need to know is
>
> 1) Node 2 complains that it cant reach node 1 and node 3. If this was
> an OS/Hardware networking issue and not internal to Gluster , then why
> didn't node1 and node3 have error message complaining about not
> reaching node2
>
[2019-12-05 22:00:43.739804] C
[rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-GL1image-client-2:
server 10.255.1.1:49153 has not responded in the last 21 seconds,
disconnecting.
[2019-12-05 22:00:43.757095] C
[rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-GL1image-client-1:
server 10.255.1.3:49152 has not responded in the last 21 seconds,
disconnecting.
[2019-12-05 22:00:43.757191] I [MSGID: 114018]
[client.c:2323:client_rpc_notify] 0-GL1image-client-2: disconnected from
GL1image-client-2. Client process will keep trying to connect to
glusterd until brick's port is available
[2019-12-05 22:00:43.757246] I [MSGID: 114018]
[client.c:2323:client_rpc_notify] 0-GL1image-client-1: disconnected from
GL1image-client-1. Client process will keep trying to connect to
glusterd until brick's port is available
[2019-12-05 22:00:43.757266] W [MSGID: 108001]
[afr-common.c:5608:afr_notify] 0-GL1image-replicate-0: Client-quorum is
not met
This seems to indicate the mount on node 2 cannot reach 2 bricks. If
quorum is not met, you will get ENOTCONN on the mount. Maybe check if
the mount is still disconnected from the bricks (either statedump or
looking at the .meta folder)?
> 2) how significant is it that the node was running 6.5 while node 1
> and node 2 were running 6.4
Minor versions should be fine but it is always a good idea to have all
nodes on the same version.
HTH,
Ravi
>
> -wk
>
> -------- Forwarded Message --------
> Subject: VM freeze issue on simple gluster setup.
> Date: Thu, 5 Dec 2019 16:23:35 -0800
> From: WK <wkmail at bneit.com>
> To: Gluster Users <gluster-users at gluster.org>
>
>
>
> I have a replica2+arbiter setup that is used for VMs.
>
> ip #.1 is the arb
>
> ip #.2 and #.3 are the kvm hosts.
>
> Two Volumes are involved and its gluster 6.5/Ubuntu 18.4/fuse The
> Gluster networking uses a two ethernet card teamd/round-robin setup
> which *should* have stayed up if one of the ports had failed.
>
> I just had a number of VMs go Read-Only due to the below communication
> failure at 22:00 but only on kvm host #2
>
> VMs on the same gluster volumes on kvm host 3 were unaffected.
>
> The logs on host #2 show the following:
>
> [2019-12-05 22:00:43.739804] C
> [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-GL1image-client-2:
> server 10.255.1.1:49153 has not responded in the last 21 seconds,
> disconnecting.
> [2019-12-05 22:00:43.757095] C
> [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-GL1image-client-1:
> server 10.255.1.3:49152 has not responded in the last 21 seconds,
> disconnecting.
> [2019-12-05 22:00:43.757191] I [MSGID: 114018]
> [client.c:2323:client_rpc_notify] 0-GL1image-client-2: disconnected
> from GL1image-client-2. Client process will keep trying to connect to
> glusterd until brick's port is available
> [2019-12-05 22:00:43.757246] I [MSGID: 114018]
> [client.c:2323:client_rpc_notify] 0-GL1image-client-1: disconnected
> from GL1image-client-1. Client process will keep trying to connect to
> glusterd until brick's port is available
> [2019-12-05 22:00:43.757266] W [MSGID: 108001]
> [afr-common.c:5608:afr_notify] 0-GL1image-replicate-0: Client-quorum
> is not met
> [2019-12-05 22:00:43.790639] E [rpc-clnt.c:346:saved_frames_unwind]
> (-->
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x139)[0x7f030d045f59]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xcbb0)[0x7f030cdf0bb0]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xccce)[0x7f030cdf0cce]
> (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x95)[0x7f030cdf1c45]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe890)[0x7f030cdf2890]
> ))))) 0-GL1image-client-2: forced unwinding frame type(GlusterFS 4.x
> v1) op(FXATTROP(34)) called at 2019-12-05 22:00:19.736456 (xid=0x825bffb)
> [2019-12-05 22:00:43.790655] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1614:client4_0_fxattrop_cbk]
> 0-GL1image-client-2: remote operation failed
> [2019-12-05 22:00:43.790686] E [rpc-clnt.c:346:saved_frames_unwind]
> (-->
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x139)[0x7f030d045f59]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xcbb0)[0x7f030cdf0bb0]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xccce)[0x7f030cdf0cce]
> (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x95)[0x7f030cdf1c45]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe890)[0x7f030cdf2890]
> ))))) 0-GL1image-client-1: forced unwinding frame type(GlusterFS 4.x
> v1) op(FXATTROP(34)) called at 2019-12-05 22:00:19.736428 (xid=0x89fee01)
> [2019-12-05 22:00:43.790703] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1614:client4_0_fxattrop_cbk]
> 0-GL1image-client-1: remote operation failed
> [2019-12-05 22:00:43.790774] E [MSGID: 114031]
> [client-rpc-fops_v2.c:1393:client4_0_finodelk_cbk]
> 0-GL1image-client-1: remote operation failed [Transport endpoint is
> not connected]
> [2019-12-05 22:00:43.790777] E [rpc-clnt.c:346:saved_frames_unwind]
> (-->
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_log_callingfn+0x139)[0x7f030d045f59]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xcbb0)[0x7f030cdf0bb0]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xccce)[0x7f030cdf0cce]
> (-->
> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x95)[0x7f030cdf1c45]
> (--> /usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xe890)[0x7f030cdf2890]
> ))))) 0-GL1image-client-2: forced unwinding frame type(GlusterFS 4.x
> v1) op(FXATTROP(34)) called at 2019-12-05 22:00:19.736542 (xid=0x825bffc)
> [2019-12-05 22:00:43.790794] W [MSGID: 114029]
> [client-rpc-fops_v2.c:4873:client4_0_finodelk] 0-GL1image-client-1:
> failed to send the fop
> [2019-12-05 22:00:43.790806] W [MSGID: 114031]
> [client-rpc-fops_v2.c:1614:client4_0_fxattrop_cbk]
> 0-GL1image-client-2: remote operation failed
> [2019-12-05 22:00:43.790825] E [MSGID: 114031]
> [client-rpc-fops_v2.c:1393:client4_0_finodelk_cbk]
> 0-GL1image-client-2: remote operation failed [Transport endpoint is
> not connected]
> [2019-12-05 22:00:43.790842] W [MSGID: 114029]
> [client-rpc-fops_v2.c:4873:client4_0_finodelk] 0-GL1image-client-2:
> failed to send the fop
>
> the fop/transport not connected errors just repeat for another 50
> lines or so until I hit 22:00:46 seconds at which point the Volumes
> appear to be fine (though the VMs were still read-only until I rebooted.
>
> [2019-12-05 22:00:46.987242] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 91701328: READ => -1
> gfid=d883b7c4-97f5-4f12-9373-7987cfc7dee4 fd=0x7f02f005b708 (Transport
> endpoint is not connected)
> [2019-12-05 22:00:47.029947] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 91701329: READ => -1
> gfid=d883b7c4-97f5-4f12-9373-7987cfc7dee4 fd=0x7f02f005b708 (Transport
> endpoint is not connected)
> [2019-12-05 22:00:49.901075] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 91701330: READ => -1
> gfid=c342dba6-a2a2-49a8-be3f-cd320e90c956 fd=0x7f02f002bee8 (Transport
> endpoint is not connected)
> [2019-12-05 22:00:49.923525] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 91701331: READ => -1
> gfid=c342dba6-a2a2-49a8-be3f-cd320e90c956 fd=0x7f02f002bee8 (Transport
> endpoint is not connected)
> [2019-12-05 22:00:49.970219] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 91701332: READ => -1
> gfid=fcec6b7a-ad23-4449-aa09-107e113877a1 fd=0x7f02f008dd58 (Transport
> endpoint is not connected)
> [2019-12-05 22:00:50.023932] W [fuse-bridge.c:2827:fuse_readv_cbk]
> 0-glusterfs-fuse: 91701333: READ => -1
> gfid=fcec6b7a-ad23-4449-aa09-107e113877a1 fd=0x7f02f008dd58 (Transport
> endpoint is not connected)
> [2019-12-05 22:00:54.807833] I [rpc-clnt.c:2028:rpc_clnt_reconfig]
> 0-GL1image-client-2: changing port to 49153 (from 0)
> [2019-12-05 22:00:54.808043] I [rpc-clnt.c:2028:rpc_clnt_reconfig]
> 0-GL1image-client-1: changing port to 49152 (from 0)
> [2019-12-05 22:00:46.115076] E [MSGID: 133014]
> [shard.c:1799:shard_common_stat_cbk] 0-GL1image-shard: stat failed:
> 7a5959d6-75fc-411d-8831-57a744776ed3 [Transport endpoint is not connected]
> [2019-12-05 22:00:54.820394] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-GL1image-client-1:
> Connected to GL1image-client-1, attached to remote volume
> '/GLUSTER/GL1image'.
> [2019-12-05 22:00:54.820447] I [MSGID: 114042]
> [client-handshake.c:930:client_post_handshake] 0-GL1image-client-1: 10
> fds open - Delaying child_up until they are re-opened
> [2019-12-05 22:00:54.820549] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-GL1image-client-2:
> Connected to GL1image-client-2, attached to remote volume
> '/GLUSTER/GL1image'.
> [2019-12-05 22:00:54.820568] I [MSGID: 114042]
> [client-handshake.c:930:client_post_handshake] 0-GL1image-client-2: 10
> fds open - Delaying child_up until they are re-opened
> [2019-12-05 22:00:54.821381] I [MSGID: 114041]
> [client-handshake.c:318:client_child_up_reopen_done]
> 0-GL1image-client-1: last fd open'd/lock-self-heal'd - notifying CHILD-UP
> [2019-12-05 22:00:54.821406] I [MSGID: 108002]
> [afr-common.c:5602:afr_notify] 0-GL1image-replicate-0: Client-quorum
> is met
> [2019-12-05 22:00:54.821446] I [MSGID: 114041]
> [client-handshake.c:318:client_child_up_reopen_done]
> 0-GL1image-client-2: last fd open'd/lock-self-heal'd - notifying CHILD-UP
>
> What is odd is that the gluster logs on the #3 and #1 show absolutely
> ZERO gluster errors around that time nor do I show any Network/teamd
> errors on any of the 3 nodes (including the problem node #2)
>
> I've checked dmesg/syslog and every other log file on the box.
>
> According to a staff member, we had this same kvm host have the same
> problem about 3 weeks ago, it was written up as a fluke possible due
> to excess disk I/O, since we have been using gluster for years and
> rarely have seen issues, especially with very basic gluster usage.
>
> In this case those VMs weren't overly busy and now we have a repeat
> problem.
>
> So I am wondering where else I can look to diagnose the problem or
> should I abandon the hardware/setup?
>
> I assume its a networking issue and not on gluster, but I am confused
> why gluster nodes #1 and #3 didn't complain about not seeing #2? If
> the networking did drop out should they have noticed?
>
> There also doesn't appear to be any visible hard disk issues (smartd
> is running)
>
> Side Note: I have reset the tcp-timeout back to 42 seconds and will
> look at upgrading to 6.6. I also see that the ARB and the unaffected
> Gluster node were running Gluster 6.4 (I don't know why #2 is on 6.5
> but I am checking on that as well, we turn off auto-upgrade)
>
> Maybe the mismatched versions are the culprit?
>
> Also, we have a large of these replica 2+1 gluster setups running
> gluster version from 5.x up and none of the others have had this issue
>
> Any advise would be appreciated.
>
> Sincerely,
>
> Wk
>
>
>
>
>
>
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191212/07c4d448/attachment.html>
More information about the Gluster-users
mailing list