[Gluster-users] client reconnect fails (was gluster heal entry reappears)

Sat Jun 17 17:31:24 UTC 2017

Hi Ravi,

back to our client-cannot-reconnect-to-gluster-brick problem ...

> Von: Ravishankar N [ravishankar at redhat.com]
> Gesendet: Montag, 29. Mai 2017 06:34
> An: Markus Stockhausen; gluster-users at gluster.org
> Betreff: Re: [Gluster-users] gluster heal entry reappears
> 
> > On 05/28/2017 10:31 PM, Markus Stockhausen wrote:
> > Hi,
> > 
> > I'm fairly new to gluster and quite happy with it. We are using it in an OVirt 
> > environment that stores its VM images in the gluster. Setup is as follows and
> > Clients mount the volume with gluster native fuse protocol. 
> > 
> > 3 storage nodes: Centos 7, Gluster 3.8.12 (managed by me), 2 bricks each
> > 5 virtualization nodes: Centos 7, Gluster 3.8.12 (managed by OVirt engine)
> > 
> > After todays reboot of one of the storage nodes the recovery did not finish
> > successfully. The state of one brick remained in:
> > 
> > [root at cfiler301 dom_md]# gluster volume heal gluster1 info
> > ...
> > Brick cfilers201:/var/data/brick1/brick
> > /b1de7818-020b-4f47-938f-f3ebb51836a3/dom_md/ids
> > Status: Connected
> > Number of entries: 1
> > ...
> > 
> > The above file is used by sanlock runing on the OVirt nodes to handle VM 
> > image locking. Issuing a manual heal with "gluster volume heal gluster1" fixed 
> > the problem but the unsynced entry reappeared a few seconds later. 
> > 
> > My question: Should this situation be recovered automatically and if yes
> > what might be the culprit?
>
> We have had this observed by our QE guys while testing too, but in all cases, there was an intermittent disconnect from the fuse mount to the bricks, leading to the 'ids' file needing heal (and being healed on reconnect) again and again.
> Perhaps you should check if and why the mount is getting disconnected from the bricks.
> 
> HTH,
> Ravi

Just finished a maintenance of one cluster brick node
and the error reappeared. Status is as follows:

- 3 gluster brick nodes: working flawlessley. logs are silent after resync
- 3 gluster fuse client nodes: working flawlessley. logs are silent after resync 
- 1 gluster fuse client node: spitting errors every 2 minutes since the maintenane

Client log beginning at the maintenance time:

[2017-06-17 09:08:26.032957] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-1: server 100.64.252.1:49154 has not responded in the last 42 seconds, disconnecting.
[2017-06-17 09:08:26.033302] C [rpc-clnt-ping.c:160:rpc_clnt_ping_timer_expired] 0-gluster1-client-2: server 100.64.252.1:49155 has not responded in the last 42 seconds, disconnecting.
[2017-06-17 09:08:26.033751] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] )))))))))) 0-gluster1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:43.578036 (xid=0x10e233)
[2017-06-17 09:08:26.033753] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] )))))))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(FINODELK(30)) called at 2017-06-17 09:07:59.914614 (xid=0xa4141)
[2017-06-17 09:08:26.033788] E [MSGID: 114031] [client-rpc-fops.c:1596:client3_3_finodelk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
[2017-06-17 09:08:26.033799] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2017-06-17 09:08:26.034008] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:43.578069 (xid=0xa413e)
[2017-06-17 09:08:26.034029] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]
[2017-06-17 09:08:26.034078] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-1: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-06-17 09:07:44.028106 (xid=0x10e234)
[2017-06-17 09:08:26.034150] W [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-gluster1-client-1: socket disconnected
[2017-06-17 09:08:26.034345] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:48.933278 (xid=0x10e235)
[2017-06-17 09:08:26.034459] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at 2017-06-17 09:07:44.028115 (xid=0xa413f)
[2017-06-17 09:08:26.034478] W [rpc-clnt-ping.c:203:rpc_clnt_ping_cbk] 0-gluster1-client-2: socket disconnected
[2017-06-17 09:08:26.034535] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-1: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:08:01.714032 (xid=0x10e236)
[2017-06-17 09:08:26.034614] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:07:48.933304 (xid=0xa4140)
[2017-06-17 09:08:26.034820] E [rpc-clnt.c:365:saved_frames_unwind] (--> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x192)[0x7f8fa739e162] (--> /lib64/libgfrpc.so.0(saved_frames_unwind+0x1de)[0x7f8fa716595e] (--> /lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f8fa7165a6e] (--> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x84)[0x7f8fa71671d4] (--> /lib64/libgfrpc.so.0(rpc_clnt_notify+0x94)[0x7f8fa7167a64] ))))) 0-gluster1-client-2: forced unwinding frame type(GlusterFS 3.3) op(LOOKUP(27)) called at 2017-06-17 09:08:01.714058 (xid=0xa4142)
[2017-06-17 09:08:26.034923] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gluster1-client-2: disconnected from gluster1-client-2. Client process will keep trying to connect to glusterd until brick's port is available
[2017-06-17 09:08:26.037127] I [socket.c:3401:socket_submit_request] 0-gluster1-client-1: not connected (priv->connected = 0)
[2017-06-17 09:08:26.037145] W [rpc-clnt.c:1657:rpc_clnt_submit] 0-gluster1-client-1: failed to submit rpc-request (XID: 0x10e237 Program: GlusterFS 3.3, ProgVers: 330, Proc: 27) to rpc-transport (gluster1-client-1)
[2017-06-17 09:08:26.037172] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: /b1de7818-020b-4f47-938f-f3ebb51836a3 (663eae40-1818-43bb-b479-dec81057b5e1) [Transport endpoint is not connected]
[2017-06-17 09:08:26.037195] I [MSGID: 114018] [client.c:2280:client_rpc_notify] 0-gluster1-client-1: disconnected from gluster1-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2017-06-17 09:08:26.038702] W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: (null) (00000000-0000-0000-0000-000000000000) [Transport endpoint is not connected]
[2017-06-17 09:08:26.121603] W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
[2017-06-17 09:08:29.040058] E [socket.c:2309:socket_connect_finish] 0-gluster1-client-1: connection to 100.64.252.1:24007 failed (No route to host)
[2017-06-17 09:08:38.142061] E [socket.c:2309:socket_connect_finish] 0-gluster1-client-2: connection to 100.64.252.1:24007 failed (No route to host)
The message "W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-1: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2017-06-17 09:08:26.033799] and [2017-06-17 09:08:26.034553]
The message "W [MSGID: 114031] [client-rpc-fops.c:2933:client3_3_lookup_cbk] 0-gluster1-client-2: remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) [Transport endpoint is not connected]" repeated 2 times between [2017-06-17 09:08:26.034029] and [2017-06-17 09:08:26.034850]
[2017-06-17 09:08:26.035008] E [MSGID: 114031] [client-rpc-fops.c:1596:client3_3_finodelk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2017-06-17 09:08:26.121603] and [2017-06-17 09:09:01.336388]
[2017-06-17 09:10:01.462178] W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2017-06-17 09:10:01.462178] and [2017-06-17 09:11:01.668207]
[2017-06-17 09:12:01.792341] W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]
The message "W [MSGID: 114031] [client-rpc-fops.c:2524:client3_3_lk_cbk] 0-gluster1-client-2: remote operation failed [Transport endpoint is not connected]" repeated 3 times between [2017-06-17 09:12:01.792341] and [2017-06-17 09:13:02.002464]
...
And so on 
...

Digging deeper into the problems with an strace of the gluster process
on the client I found the following sequence (100.64.252.1 being the
brick that was in maintenance):

4543  socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
4543  setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
4543  setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
4543  setsockopt(5, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0
4543  setsockopt(5, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
4543  setsockopt(5, SOL_TCP, TCP_USER_TIMEOUT, [0], 4) = 0
4543  open("/proc/sys/net/ipv4/ip_local_reserved_ports", O_RDONLY) = 7
4543  read(7, "\n", 4096)               = 1
4543  close(7)                          = 0
4543  bind(5, {sa_family=AF_INET, sin_port=htons(49151), sin_addr=inet_addr("100.64.251.6")}, 16) = 0
4543  fcntl(5, F_GETFL)                 = 0x2 (flags O_RDWR)
4543  fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543  connect(5, {sa_family=AF_INET, sin_port=htons(24007), sin_addr=inet_addr("100.64.252.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
4543  fcntl(5, F_GETFL)                 = 0x802 (flags O_RDWR|O_NONBLOCK)
4543  fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543  epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLONESHOT, {u32=2, u64=4952097292290}}) = 0
...
4543  socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 7
4543  setsockopt(7, SOL_TCP, TCP_NODELAY, [1], 4) = 0
4543  setsockopt(7, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
4543  setsockopt(7, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0
4543  setsockopt(7, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
4543  setsockopt(7, SOL_TCP, TCP_USER_TIMEOUT, [0], 4) = 0
4543  open("/proc/sys/net/ipv4/ip_local_reserved_ports", O_RDONLY) = 12
4543  read(12, "\n", 4096)              = 1
4543  close(12)                         = 0
4543  bind(7, {sa_family=AF_INET, sin_port=htons(49151), sin_addr=inet_addr("100.64.251.6")}, 16) = -1 EADDRINUSE (Address already in use)
4543  bind(7, {sa_family=AF_INET, sin_port=htons(49150), sin_addr=inet_addr("100.64.251.6")}, 16) = -1 EADDRINUSE (Address already in use)
4543  bind(7, {sa_family=AF_INET, sin_port=htons(49149), sin_addr=inet_addr("100.64.251.6")}, 16) = 0
4543  fcntl(7, F_GETFL)                 = 0x2 (flags O_RDWR)
4543  fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543  connect(7, {sa_family=AF_INET, sin_port=htons(24007), sin_addr=inet_addr("100.64.252.1")}, 16) = -1 EINPROGRESS (Operation now in progress)
4543  fcntl(7, F_GETFL)                 = 0x802 (flags O_RDWR|O_NONBLOCK)
4543  fcntl(7, F_SETFL, O_RDWR|O_NONBLOCK) = 0
4543  epoll_ctl(3, EPOLL_CTL_ADD, 7, {EPOLLIN|EPOLLPRI|EPOLLOUT|EPOLLONESHOT, {u32=4, u64=4810363371524}}) = 0
...
4550  <... epoll_wait resumed> {{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=2, u64=4952097292290}}}, 1, -1) = 1
4548  <... epoll_wait resumed> {{EPOLLIN|EPOLLOUT|EPOLLERR|EPOLLHUP, {u32=4, u64=4810363371524}}}, 1, -1) = 1
4550  getsockopt(5, SOL_SOCKET, SO_ERROR,  <unfinished ...>
4548  getsockopt(7, SOL_SOCKET, SO_ERROR,  <unfinished ...>
4550  <... getsockopt resumed> [110], [4]) = 0
4548  <... getsockopt resumed> [110], [4]) = 0
4550  shutdown(5, SHUT_RDWR <unfinished ...>
4548  shutdown(7, SHUT_RDWR <unfinished ...>
4550  <... shutdown resumed> )          = -1 ENOTCONN (Transport endpoint is not connected)
4548  <... shutdown resumed> )          = -1 ENOTCONN (Transport endpoint is not connected)
...

Direct ping+ssh between server and client (and vice versa) works fine.
Looking at a tcpdump on the brick side I see the TCP handshake for the 
client. Looks strange because it is repeated after 1,2,4 seconds and so 
on

18:55:51.176832 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 3453818375, win 28364, options [mss 4052,sackOK,TS val 362061385 ecr 0,nop,wscale 7], length 0
18:55:51.176884 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13791464 ecr 362061385,nop,wscale 7], length 0
18:55:51.179391 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 2219902567, win 28364, options [mss 4052,sackOK,TS val 362061388 ecr 0,nop,wscale 7], length 0
18:55:51.179433 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13791466 ecr 362061388,nop,wscale 7], length 0
18:55:52.176704 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13792464 ecr 362061385,nop,wscale 7], length 0
18:55:52.178695 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13792466 ecr 362061388,nop,wscale 7], length 0
18:55:52.178951 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 3453818375, win 28364, options [mss 4052,sackOK,TS val 362062388 ecr 0,nop,wscale 7], length 0
18:55:52.178971 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13792466 ecr 362061385,nop,wscale 7], length 0
18:55:52.180951 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 2219902567, win 28364, options [mss 4052,sackOK,TS val 362062390 ecr 0,nop,wscale 7], length 0
18:55:52.180979 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13792468 ecr 362061388,nop,wscale 7], length 0
18:55:54.178704 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13794466 ecr 362061385,nop,wscale 7], length 0
18:55:54.180704 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13794468 ecr 362061388,nop,wscale 7], length 0
18:55:54.182952 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 3453818375, win 28364, options [mss 4052,sackOK,TS val 362064392 ecr 0,nop,wscale 7], length 0
18:55:54.182982 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 1929363096, ack 3453818376, win 28280, options [mss 4052,sackOK,TS val 13794470 ecr 362061385,nop,wscale 7], length 0
18:55:54.182991 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 2219902567, win 28364, options [mss 4052,sackOK,TS val 362064392 ecr 0,nop,wscale 7], length 0
18:55:54.183004 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 2413229577, ack 2219902568, win 28280, options [mss 4052,sackOK,TS val 13794470 ecr 362061388,nop,wscale 7], length 0

Same on the client side (tcpdump at later time):

19:02:18.241460 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 911752495, win 28364, options [mss 4052,sackOK,TS val 362448449 ecr 0,nop,wscale 7], length 0
19:02:18.241642 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14178528 ecr 362448449,nop,wscale 7], length 0
19:02:18.243955 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 3972802780, win 28364, options [mss 4052,sackOK,TS val 362448452 ecr 0,nop,wscale 7], length 0
19:02:18.244132 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14178530 ecr 362448452,nop,wscale 7], length 0
19:02:19.241228 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14179528 ecr 362448449,nop,wscale 7], length 0
19:02:19.243220 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14179530 ecr 362448452,nop,wscale 7], length 0
19:02:19.244000 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 911752495, win 28364, options [mss 4052,sackOK,TS val 362449452 ecr 0,nop,wscale 7], length 0
19:02:19.244121 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14179530 ecr 362448449,nop,wscale 7], length 0
19:02:19.245995 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 3972802780, win 28364, options [mss 4052,sackOK,TS val 362449454 ecr 0,nop,wscale 7], length 0
19:02:19.246097 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14179532 ecr 362448452,nop,wscale 7], length 0
19:02:21.243229 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14181530 ecr 362448449,nop,wscale 7], length 0
19:02:21.245222 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14181532 ecr 362448452,nop,wscale 7], length 0
19:02:21.247997 IP cadmins101.49151 > cfilers201.24007: Flags [S], seq 911752495, win 28364, options [mss 4052,sackOK,TS val 362451456 ecr 0,nop,wscale 7], length 0
19:02:21.248007 IP cadmins101.49149 > cfilers201.24007: Flags [S], seq 3972802780, win 28364, options [mss 4052,sackOK,TS val 362451456 ecr 0,nop,wscale 7], length 0
19:02:21.248130 IP cfilers201.24007 > cadmins101.49151: Flags [S.], seq 3682274790, ack 911752496, win 28280, options [mss 4052,sackOK,TS val 14181534 ecr 362448449,nop,wscale 7], length 0
19:02:21.248173 IP cfilers201.24007 > cadmins101.49149: Flags [S.], seq 4166140203, ack 3972802781, win 28280, options [mss 4052,sackOK,TS val 14181534 ecr 362448452,nop,wscale 7], length 0

Out of ideas now. Maybe you see something.

Markus
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: InterScan_Disclaimer.txt
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170617/223c3d1e/attachment.txt>