<div dir="ltr">This logfile didn't have any logs about heal. Could you find the same file on the other node as well and attach it to the mail-thread? We should also see the mount logs once to confirm replication did something or not. Otherwise we should see the brick logs. Instead of checking it iteratively, could you tar /var/log/glusterfs on both the nodes and give links to these files so that we can download and check what might have happened?<br><div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 20, 2018 at 2:00 PM Johan Karlsson <<a href="mailto:Johan.Karlsson@dgc.se">Johan.Karlsson@dgc.se</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_-4521912423039131257divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif" dir="ltr">
<p style="margin-top:0px;margin-bottom:0px"></p>
<p>I understand that a 2 way replica can require some fiddling with heal, but how is it possible that all data just vanished, even from the bricks?</p>
<p>---<br>
gluster> volume info</p>
<p>Volume Name: gvol0<br>
Type: Replicate<br>
Volume ID: 17ed4d1c-2120-4fe8-abd6-dd77d7ddac59<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: gfs01:/glusterdata/brick1/gvol0<br>
Brick2: gfs02:/glusterdata/brick2/gvol0<br>
Options Reconfigured:<br>
performance.client-io-threads: off<br>
nfs.disable: on<br>
transport.address-family: inet<br>
---</p>
<p>---<br>
gfs01 - Standard upgrade:</p>
<p>Start-Date: 2018-09-12 12:51:51<br>
Commandline: apt-get dist-upgrade<br>
---</p>
<p>---<br>
gfs02 - standard upgrade:</p>
<p>Start-Date: 2018-09-12 13:28:32<br>
Commandline: apt-get dist-upgrade<br>
---</p>
<p>---<br>
gfs01 glustershd.log</p>
<p>[2018-09-12 12:52:56.211130] W [socket.c:592:__socket_rwv] 0-glusterfs: readv on <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (No data available)<br>
[2018-09-12 12:52:56.211155] I [glusterfsd-mgmt.c:2341:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost<br>
[2018-09-12 12:53:06.844040] E [socket.c:2517:socket_connect_finish] 0-glusterfs: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (Connection refused); disconnecting socket<br>
[2018-09-12 12:53:06.844066] I [glusterfsd-mgmt.c:2362:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers<br>
[2018-09-12 12:54:04.224545] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fee21cfa6ba] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x55872a03a70d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x55872a03a524<br>
] ) 0-: received signum (15), shutting down<br>
[2018-09-12 12:54:05.221508] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.4 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid<br>
-l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c7535c5e8ebaab32.socket --xlator-option *replicate*.node-uuid=5865e739-3c64-4039-8f96-5fc7a75d00fe --process-name glustershd)<br>
[2018-09-12 12:54:05.225264] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2018-09-12 12:54:06.246818] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2018-09-12 12:54:06.247109] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport<br>
[2018-09-12 12:54:06.247236] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport<br>
[2018-09-12 12:54:06.247269] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume gvol0-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host gfs01<br>
5: option remote-subvolume /glusterdata/brick1/gvol0<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
9: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
10: option transport.tcp-user-timeout 0<br>
11: option transport.socket.keepalive-time 20<br>
12: option transport.socket.keepalive-interval 2<br>
13: option transport.socket.keepalive-count 9<br>
14: end-volume<br>
15:<br>
16: volume gvol0-client-1<br>
17: type protocol/client<br>
18: option ping-timeout 42<br>
19: option remote-host gfs02<br>
20: option remote-subvolume /glusterdata/brick2/gvol0<br>
21: option transport-type socket<br>
22: option transport.address-family inet<br>
23: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
24: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
25: option transport.tcp-user-timeout 0<br>
26: option transport.socket.keepalive-time 20<br>
27: option transport.socket.keepalive-interval 2<br>
28: option transport.socket.keepalive-count 9<br>
29: end-volume<br>
30:<br>
31: volume gvol0-replicate-0<br>
32: type cluster/replicate<br>
33: option node-uuid 5865e739-3c64-4039-8f96-5fc7a75d00fe<br>
34: option afr-pending-xattr gvol0-client-0,gvol0-client-1<br>
35: option background-self-heal-count 0<br>
36: option metadata-self-heal on<br>
37: option data-self-heal on<br>
38: option entry-self-heal on<br>
39: option self-heal-daemon enable<br>
40: option use-compound-fops off<br>
41: option iam-self-heal-daemon yes<br>
42: subvolumes gvol0-client-0 gvol0-client-1<br>
43: end-volume<br>
44:<br>
45: volume glustershd<br>
[2018-09-12 12:54:06.247484] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
46: type debug/io-stats<br>
47: option log-level INFO<br>
48: subvolumes gvol0-replicate-0<br>
49: end-volume<br>
50:<br>
+------------------------------------------------------------------------------+<br>
[2018-09-12 12:54:06.249099] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 12:54:06.249561] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 12:54:06.249790] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 12:54:06.250309] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 12:54:06.250889] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
[2018-09-12 12:54:06.250904] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-1' came back up; going online.<br>
[2018-09-12 12:54:06.260091] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 12:54:06.269981] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 12:54:06.270175] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 12:54:06.270309] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 12:54:06.270698] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:57:40.616257] W [socket.c:592:__socket_rwv] 0-glusterfs: readv on <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (No data available)<br>
[2018-09-12 13:57:40.616312] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost<br>
[2018-09-12 13:57:50.942555] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7fb690a156ba] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x561b24e0d70d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x561b24e0d524<br>
] ) 0-: received signum (15), shutting down<br>
[2018-09-12 13:58:06.192019] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.4 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid<br>
-l /var/log/glusterfs/glustershd.log -S /var/run/gluster/c7535c5e8ebaab32.socket --xlator-option *replicate*.node-uuid=5865e739-3c64-4039-8f96-5fc7a75d00fe --process-name glustershd)<br>
[2018-09-12 13:58:06.196996] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2018-09-12 13:58:07.322458] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2018-09-12 13:58:07.322772] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:58:07.323166] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:58:07.323196] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.323327] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.323420] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-gvol0-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br>
[2018-09-12 13:58:07.323459] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-0: disconnected from gvol0-client-0. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:58:07.323486] E [MSGID: 108006] [afr-common.c:5317:__afr_handle_child_down_event] 0-gvol0-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume gvol0-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host gfs01<br>
5: option remote-subvolume /glusterdata/brick1/gvol0<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
9: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
10: option transport.tcp-user-timeout 0<br>
11: option transport.socket.keepalive-time 20<br>
12: option transport.socket.keepalive-interval 2<br>
13: option transport.socket.keepalive-count 9<br>
14: end-volume<br>
15:<br>
16: volume gvol0-client-1<br>
17: type protocol/client<br>
18: option ping-timeout 42<br>
19: option remote-host gfs02<br>
20: option remote-subvolume /glusterdata/brick2/gvol0<br>
21: option transport-type socket<br>
22: option transport.address-family inet<br>
23: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
24: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
25: option transport.tcp-user-timeout 0<br>
26: option transport.socket.keepalive-time 20<br>
27: option transport.socket.keepalive-interval 2<br>
28: option transport.socket.keepalive-count 9<br>
29: end-volume<br>
30:<br>
31: volume gvol0-replicate-0<br>
32: type cluster/replicate<br>
33: option node-uuid 5865e739-3c64-4039-8f96-5fc7a75d00fe<br>
34: option afr-pending-xattr gvol0-client-0,gvol0-client-1<br>
35: option background-self-heal-count 0<br>
36: option metadata-self-heal on<br>
37: option data-self-heal on<br>
38: option entry-self-heal on<br>
39: option self-heal-daemon enable<br>
40: option use-compound-fops off<br>
41: option iam-self-heal-daemon yes<br>
42: subvolumes gvol0-client-0 gvol0-client-1<br>
43: end-volume<br>
44:<br>
45: volume glustershd<br>
46: type debug/io-stats<br>
47: option log-level INFO<br>
48: subvolumes gvol0-replicate-0<br>
49: end-volume<br>
50:<br>
+------------------------------------------------------------------------------+<br>
[2018-09-12 13:58:07.323808] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.324101] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.324288] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:58:07.324737] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.325066] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.337185] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
[2018-09-12 13:58:07.337202] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-1' came back up; going online.<br>
[2018-09-12 13:58:11.193402] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:11.193575] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:11.193661] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 13:58:11.193975] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:11.194217] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:11.194773] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:59:05.215057] W [socket.c:592:__socket_rwv] 0-gvol0-client-1: readv on <a href="http://192.168.4.85:49152" target="_blank">192.168.4.85:49152</a> failed (No data available)<br>
[2018-09-12 13:59:05.215112] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-1: disconnected from gvol0-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:59:18.521991] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:19.504398] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:19.505038] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-gvol0-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br>
[2018-09-12 13:59:19.505088] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-1: disconnected from gvol0-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:59:21.519674] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.519929] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.520103] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:59:21.520531] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.520754] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.521890] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
---</p>
<p>---<br>
gfs01 mountpoint log:</p>
<p>[2018-09-12 13:58:06.497145] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.4 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=gfs01 --volfile-id=/gvol0 /tss/filestore)<br>
[2018-09-12 13:58:06.534575] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2018-09-12 13:58:07.381591] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2018-09-12 13:58:07.386730] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:58:07.387087] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:58:07.387129] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.387268] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume gvol0-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host gfs01<br>
5: option remote-subvolume /glusterdata/brick1/gvol0<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
9: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
10: option transport.tcp-user-timeout 0<br>
11: option transport.socket.keepalive-time 20<br>
12: option transport.socket.keepalive-interval 2<br>
13: option transport.socket.keepalive-count 9<br>
14: option send-gids true<br>
15: end-volume<br>
16:<br>
17: volume gvol0-client-1<br>
18: type protocol/client<br>
19: option ping-timeout 42<br>
20: option remote-host gfs02<br>
21: option remote-subvolume /glusterdata/brick2/gvol0<br>
22: option transport-type socket<br>
23: option transport.address-family inet<br>
24: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
25: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
26: option transport.tcp-user-timeout 0<br>
27: option transport.socket.keepalive-time 20<br>
28: option transport.socket.keepalive-interval 2<br>
29: option transport.socket.keepalive-count 9<br>
[2018-09-12 13:58:07.387367] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-gvol0-client-0: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br>
30: option send-gids true<br>
31: end-volume<br>
32:<br>
33: volume gvol0-replicate-0<br>
34: type cluster/replicate<br>
35: option afr-pending-xattr gvol0-client-0,gvol0-client-1<br>
36: option use-compound-fops off<br>
37: subvolumes gvol0-client-0 gvol0-client-1<br>
38: end-volume<br>
39:<br>
40: volume gvol0-dht<br>
41: type cluster/distribute<br>
42: option lock-migration off<br>
43: option force-migration off<br>
[2018-09-12 13:58:07.387461] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-0: disconnected from gvol0-client-0. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:58:07.387490] E [MSGID: 108006] [afr-common.c:5317:__afr_handle_child_down_event] 0-gvol0-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.<br>
44: subvolumes gvol0-replicate-0<br>
45: end-volume<br>
46:<br>
47: volume gvol0-write-behind<br>
48: type performance/write-behind<br>
49: subvolumes gvol0-dht<br>
50: end-volume<br>
51:<br>
52: volume gvol0-read-ahead<br>
53: type performance/read-ahead<br>
54: subvolumes gvol0-write-behind<br>
55: end-volume<br>
56:<br>
57: volume gvol0-readdir-ahead<br>
58: type performance/readdir-ahead<br>
59: option parallel-readdir off<br>
60: option rda-request-size 131072<br>
61: option rda-cache-limit 10MB<br>
62: subvolumes gvol0-read-ahead<br>
63: end-volume<br>
64:<br>
65: volume gvol0-io-cache<br>
66: type performance/io-cache<br>
67: subvolumes gvol0-readdir-ahead<br>
68: end-volume<br>
69:<br>
70: volume gvol0-quick-read<br>
71: type performance/quick-read<br>
72: subvolumes gvol0-io-cache<br>
73: end-volume<br>
74:<br>
75: volume gvol0-open-behind<br>
76: type performance/open-behind<br>
77: subvolumes gvol0-quick-read<br>
78: end-volume<br>
79:<br>
80: volume gvol0-md-cache<br>
[2018-09-12 13:58:07.387621] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
81: type performance/md-cache<br>
82: subvolumes gvol0-open-behind<br>
83: end-volume<br>
84:<br>
85: volume gvol0<br>
86: type debug/io-stats<br>
87: option log-level INFO<br>
88: option latency-measurement off<br>
89: option count-fop-hits off<br>
90: subvolumes gvol0-md-cache<br>
91: end-volume<br>
92:<br>
93: volume meta-autoload<br>
94: type meta<br>
95: subvolumes gvol0<br>
96: end-volume<br>
97:<br>
+------------------------------------------------------------------------------+<br>
[2018-09-12 13:58:07.387891] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.388118] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:58:07.388701] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.389814] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:07.390371] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
[2018-09-12 13:58:07.390390] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-1' came back up; going online.<br>
[2018-09-12 13:58:07.391330] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23<br>
[2018-09-12 13:58:07.391346] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0<br>
[2018-09-12 13:58:07.393037] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-gvol0-dht: Directory selfheal failed: Unable to form layout for directory /<br>
[2018-09-12 13:58:10.534498] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:10.534637] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:10.534727] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 13:58:10.535015] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:10.535155] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:10.536297] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:59:05.215073] W [socket.c:592:__socket_rwv] 0-gvol0-client-1: readv on <a href="http://192.168.4.85:49152" target="_blank">192.168.4.85:49152</a> failed (No data available)<br>
[2018-09-12 13:59:05.215112] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-1: disconnected from gvol0-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:59:18.861826] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:19.505060] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:19.517843] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-gvol0-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br>
[2018-09-12 13:59:19.517934] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-1: disconnected from gvol0-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:59:21.860457] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.860727] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.860903] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:59:21.861333] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.861588] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:21.862134] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
---</p>
<p>---<br>
gfs02 glustershd.log</p>
<p>[2018-09-12 13:29:24.440044] W [socket.c:592:__socket_rwv] 0-glusterfs: readv on <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (No data available)<br>
[2018-09-12 13:29:24.440066] I [glusterfsd-mgmt.c:2341:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost<br>
[2018-09-12 13:29:35.300684] E [socket.c:2517:socket_connect_finish] 0-glusterfs: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (Connection refused); disconnecting socket<br>
[2018-09-12 13:29:35.300719] I [glusterfsd-mgmt.c:2362:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers<br>
[2018-09-12 13:30:28.718734] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f671aa8f6ba] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x55d18aa3670d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x55d18aa36524<br>
] ) 0-: received signum (15), shutting down<br>
[2018-09-12 13:30:29.721210] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.4 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid<br>
-l /var/log/glusterfs/glustershd.log -S /var/run/gluster/3c69308176cfc594.socket --xlator-option *replicate*.node-uuid=44192eee-3f26-4e14-84d5-be847d66df7b --process-name glustershd)<br>
[2018-09-12 13:30:29.724100] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2018-09-12 13:30:30.748354] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2018-09-12 13:30:30.752656] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:30:30.752794] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:30:30.753009] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume gvol0-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host gfs01<br>
5: option remote-subvolume /glusterdata/brick1/gvol0<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
[2018-09-12 13:30:30.754060] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
9: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
10: option transport.tcp-user-timeout 0<br>
11: option transport.socket.keepalive-time 20<br>
12: option transport.socket.keepalive-interval 2<br>
13: option transport.socket.keepalive-count 9<br>
14: end-volume<br>
15:<br>
16: volume gvol0-client-1<br>
17: type protocol/client<br>
18: option ping-timeout 42<br>
19: option remote-host gfs02<br>
20: option remote-subvolume /glusterdata/brick2/gvol0<br>
21: option transport-type socket<br>
22: option transport.address-family inet<br>
23: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
24: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
25: option transport.tcp-user-timeout 0<br>
26: option transport.socket.keepalive-time 20<br>
27: option transport.socket.keepalive-interval 2<br>
28: option transport.socket.keepalive-count 9<br>
29: end-volume<br>
30:<br>
31: volume gvol0-replicate-0<br>
32: type cluster/replicate<br>
33: option node-uuid 44192eee-3f26-4e14-84d5-be847d66df7b<br>
34: option afr-pending-xattr gvol0-client-0,gvol0-client-1<br>
35: option background-self-heal-count 0<br>
36: option metadata-self-heal on<br>
37: option data-self-heal on<br>
38: option entry-self-heal on<br>
39: option self-heal-daemon enable<br>
40: option use-compound-fops off<br>
41: option iam-self-heal-daemon yes<br>
42: subvolumes gvol0-client-0 gvol0-client-1<br>
43: end-volume<br>
44:<br>
45: volume glustershd<br>
46: type debug/io-stats<br>
47: option log-level INFO<br>
48: subvolumes gvol0-replicate-0<br>
49: end-volume<br>
50:<br>
+------------------------------------------------------------------------------+<br>
[2018-09-12 13:30:30.763395] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:30:30.765518] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:30:30.765727] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 13:30:30.766021] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:30:30.766308] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:30:30.767339] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:30:30.767362] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-0' came back up; going online.<br>
[2018-09-12 13:30:30.772846] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:30:30.773011] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:30:30.773125] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:30:30.773472] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
[2018-09-12 13:58:05.409172] W [socket.c:592:__socket_rwv] 0-gvol0-client-0: readv on <a href="http://192.168.4.84:49152" target="_blank">192.168.4.84:49152</a> failed (Connection reset by peer)<br>
[2018-09-12 13:58:05.409219] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-0: disconnected from gvol0-client-0. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:58:15.871815] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:15.872066] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:15.872229] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 13:58:15.872457] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:15.872704] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:58:15.873272] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:58:54.575838] W [socket.c:592:__socket_rwv] 0-glusterfs: readv on <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (No data available)<br>
[2018-09-12 13:58:54.575873] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify] 0-glusterfsd-mgmt: disconnected from remote-host: localhost<br>
[2018-09-12 13:59:04.876731] E [socket.c:2517:socket_connect_finish] 0-glusterfs: connection to <a href="http://127.0.0.1:24007" target="_blank">127.0.0.1:24007</a> failed (Connection refused); disconnecting socket<br>
[2018-09-12 13:59:04.876764] I [glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all volfile servers<br>
[2018-09-12 13:59:05.213422] W [glusterfsd.c:1514:cleanup_and_exit] (-->/lib/x86_64-linux-gnu/libpthread.so.0(+0x76ba) [0x7f995004b6ba] -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xed) [0x55c76d21470d] -->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x55c76d214524<br>
] ) 0-: received signum (15), shutting down<br>
[2018-09-12 13:59:25.843013] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.4 (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid<br>
-l /var/log/glusterfs/glustershd.log -S /var/run/gluster/3c69308176cfc594.socket --xlator-option *replicate*.node-uuid=44192eee-3f26-4e14-84d5-be847d66df7b --process-name glustershd)<br>
[2018-09-12 13:59:25.847197] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2018-09-12 13:59:26.945403] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2018-09-12 13:59:26.945824] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:59:26.946110] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:59:26.946384] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume gvol0-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host gfs01<br>
5: option remote-subvolume /glusterdata/brick1/gvol0<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
9: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
10: option transport.tcp-user-timeout 0<br>
11: option transport.socket.keepalive-time 20<br>
12: option transport.socket.keepalive-interval 2<br>
13: option transport.socket.keepalive-count 9<br>
14: end-volume<br>
15:<br>
16: volume gvol0-client-1<br>
17: type protocol/client<br>
18: option ping-timeout 42<br>
19: option remote-host gfs02<br>
20: option remote-subvolume /glusterdata/brick2/gvol0<br>
21: option transport-type socket<br>
22: option transport.address-family inet<br>
23: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
24: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
25: option transport.tcp-user-timeout 0<br>
26: option transport.socket.keepalive-time 20<br>
27: option transport.socket.keepalive-interval 2<br>
28: option transport.socket.keepalive-count 9<br>
29: end-volume<br>
30:<br>
31: volume gvol0-replicate-0<br>
32: type cluster/replicate<br>
33: option node-uuid 44192eee-3f26-4e14-84d5-be847d66df7b<br>
34: option afr-pending-xattr gvol0-client-0,gvol0-client-1<br>
35: option background-self-heal-count 0<br>
36: option metadata-self-heal on<br>
37: option data-self-heal on<br>
38: option entry-self-heal on<br>
39: option self-heal-daemon enable<br>
40: option use-compound-fops off<br>
41: option iam-self-heal-daemon yes<br>
42: subvolumes gvol0-client-0 gvol0-client-1<br>
43: end-volume<br>
44:<br>
45: volume glustershd<br>
46: type debug/io-stats<br>
47: option log-level INFO<br>
48: subvolumes gvol0-replicate-0<br>
49: end-volume<br>
50:<br>
+------------------------------------------------------------------------------+<br>
[2018-09-12 13:59:26.946860] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:26.946961] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:26.946966] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:26.947054] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-gvol0-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br>
[2018-09-12 13:59:26.947165] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 13:59:26.947213] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-1: disconnected from gvol0-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:59:26.947233] E [MSGID: 108006] [afr-common.c:5317:__afr_handle_child_down_event] 0-gvol0-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.<br>
[2018-09-12 13:59:26.947557] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:26.947796] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:26.948355] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:59:26.948368] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-0' came back up; going online.<br>
[2018-09-12 13:59:30.845313] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.845467] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.845537] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:59:30.845785] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.845953] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.846293] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.<br>
---</p>
<p>---<br>
gfs02 mountpoint log:</p>
<p>[2018-09-12 13:59:26.116762] I [MSGID: 100030] [glusterfsd.c:2741:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.1.4 (args: /usr/sbin/glusterfs --process-name fuse --volfile-server=gfs02 --volfile-id=/gvol0 /tss/filestore)<br>
[2018-09-12 13:59:26.142136] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br>
[2018-09-12 13:59:27.029834] I [MSGID: 101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<br>
[2018-09-12 13:59:27.034636] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-0: parent translators are ready, attempting connect on transport<br>
[2018-09-12 13:59:27.034977] I [MSGID: 114020] [client.c:2328:notify] 0-gvol0-client-1: parent translators are ready, attempting connect on transport<br>
Final graph:<br>
+------------------------------------------------------------------------------+<br>
1: volume gvol0-client-0<br>
[2018-09-12 13:59:27.035277] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:27.035328] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
2: type protocol/client<br>
3: option ping-timeout 42<br>
4: option remote-host gfs01<br>
5: option remote-subvolume /glusterdata/brick1/gvol0<br>
6: option transport-type socket<br>
7: option transport.address-family inet<br>
8: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
9: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
10: option transport.tcp-user-timeout 0<br>
11: option transport.socket.keepalive-time 20<br>
12: option transport.socket.keepalive-interval 2<br>
13: option transport.socket.keepalive-count 9<br>
14: option send-gids true<br>
15: end-volume<br>
16:<br>
17: volume gvol0-client-1<br>
18: type protocol/client<br>
19: option ping-timeout 42<br>
20: option remote-host gfs02<br>
21: option remote-subvolume /glusterdata/brick2/gvol0<br>
22: option transport-type socket<br>
23: option transport.address-family inet<br>
24: option username d5e3e173-156f-46c1-9eb7-a35b201fc311<br>
25: option password 8d3c3564-cef3-4261-90bd-c64e85c6d267<br>
26: option transport.tcp-user-timeout 0<br>
27: option transport.socket.keepalive-time 20<br>
28: option transport.socket.keepalive-interval 2<br>
29: option transport.socket.keepalive-count 9<br>
30: option send-gids true<br>
31: end-volume<br>
32:<br>
33: volume gvol0-replicate-0<br>
34: type cluster/replicate<br>
35: option afr-pending-xattr gvol0-client-0,gvol0-client-1<br>
36: option use-compound-fops off<br>
37: subvolumes gvol0-client-0 gvol0-client-1<br>
38: end-volume<br>
39:<br>
40: volume gvol0-dht<br>
41: type cluster/distribute<br>
42: option lock-migration off<br>
43: option force-migration off<br>
44: subvolumes gvol0-replicate-0<br>
45: end-volume<br>
46:<br>
47: volume gvol0-write-behind<br>
48: type performance/write-behind<br>
49: subvolumes gvol0-dht<br>
50: end-volume<br>
51:<br>
52: volume gvol0-read-ahead<br>
53: type performance/read-ahead<br>
54: subvolumes gvol0-write-behind<br>
55: end-volume<br>
56:<br>
57: volume gvol0-readdir-ahead<br>
58: type performance/readdir-ahead<br>
59: option parallel-readdir off<br>
60: option rda-request-size 131072<br>
61: option rda-cache-limit 10MB<br>
62: subvolumes gvol0-read-ahead<br>
63: end-volume<br>
64:<br>
65: volume gvol0-io-cache<br>
66: type performance/io-cache<br>
67: subvolumes gvol0-readdir-ahead<br>
68: end-volume<br>
69:<br>
70: volume gvol0-quick-read<br>
71: type performance/quick-read<br>
72: subvolumes gvol0-io-cache<br>
73: end-volume<br>
74:<br>
75: volume gvol0-open-behind<br>
76: type performance/open-behind<br>
77: subvolumes gvol0-quick-read<br>
78: end-volume<br>
[2018-09-12 13:59:27.035568] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
79:<br>
[2018-09-12 13:59:27.035672] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
80: volume gvol0-md-cache<br>
81: type performance/md-cache<br>
82: subvolumes gvol0-open-behind<br>
83: end-volume<br>
84:<br>
85: volume gvol0<br>
86: type debug/io-stats<br>
87: option log-level INFO<br>
88: option latency-measurement off<br>
89: option count-fop-hits off<br>
90: subvolumes gvol0-md-cache<br>
91: end-volume<br>
92:<br>
93: volume meta-autoload<br>
94: type meta<br>
95: subvolumes gvol0<br>
96: end-volume<br>
97:<br>
+------------------------------------------------------------------------------+<br>
[2018-09-12 13:59:27.035769] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-0: changing port to 49152 (from 0)<br>
[2018-09-12 13:59:27.036156] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:27.036187] E [MSGID: 114058] [client-handshake.c:1523:client_query_portmap_cbk] 0-gvol0-client-1: failed to get the port number for remote subvolume. Please run 'gluster volume status' on server to see if brick process is running.<br>
[2018-09-12 13:59:27.036230] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-gvol0-client-1: disconnected from gvol0-client-1. Client process will keep trying to connect to glusterd until brick's port is available<br>
[2018-09-12 13:59:27.036240] E [MSGID: 108006] [afr-common.c:5317:__afr_handle_child_down_event] 0-gvol0-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.<br>
[2018-09-12 13:59:27.036411] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-0: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:27.036967] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-0: Connected to gvol0-client-0, attached to remote volume '/glusterdata/brick1/gvol0'.<br>
[2018-09-12 13:59:27.036979] I [MSGID: 108005] [afr-common.c:5240:__afr_handle_child_up_event] 0-gvol0-replicate-0: Subvolume 'gvol0-client-0' came back up; going online.<br>
[2018-09-12 13:59:27.037684] I [fuse-bridge.c:4294:fuse_init] 0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel 7.23<br>
[2018-09-12 13:59:27.037696] I [fuse-bridge.c:4927:fuse_graph_sync] 0-fuse: switched to graph 0<br>
[2018-09-12 13:59:27.038866] I [MSGID: 109005] [dht-selfheal.c:2342:dht_selfheal_directory] 0-gvol0-dht: Directory selfheal failed: Unable to form layout for directory /<br>
[2018-09-12 13:59:30.139072] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.139208] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.139282] I [rpc-clnt.c:2105:rpc_clnt_reconfig] 0-gvol0-client-1: changing port to 49152 (from 0)<br>
[2018-09-12 13:59:30.139537] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.139650] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-gvol0-client-1: error returned while attempting to connect to host:(null), port:0<br>
[2018-09-12 13:59:30.139981] I [MSGID: 114046] [client-handshake.c:1176:client_setvolume_cbk] 0-gvol0-client-1: Connected to gvol0-client-1, attached to remote volume '/glusterdata/brick2/gvol0'.</p>
<p><br>
</p>
<p></p>
<p>Regards,</p>
<p><br>
</p>
<p>Johan Karlsson</p>
<p><br>
</p>
</div>
<hr style="display:inline-block;width:98%">
<div id="m_-4521912423039131257divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>><br>
<b>Sent:</b> Thursday, September 20, 2018 8:13:47 AM<br>
<b>To:</b> Gowdappa, Raghavendra<br>
<b>Cc:</b> Johan Karlsson; gluster-users; Ravishankar Narayanankutty<br>
<b>Subject:</b> Re: [Gluster-users] Data on gluster volume gone</font>
<div> </div>
</div>
<div>
<div dir="ltr">Please also attach the logs for the mount points and the glustershd.logs<br>
</div>
<br>
<div class="m_-4521912423039131257x_gmail_quote">
<div dir="ltr">On Thu, Sep 20, 2018 at 11:41 AM Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br>
</div>
<blockquote class="m_-4521912423039131257x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">How did you do the upgrade?<br>
<div><br>
<div class="m_-4521912423039131257x_gmail_quote">
<div dir="ltr">On Thu, Sep 20, 2018 at 11:01 AM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br>
</div>
<blockquote class="m_-4521912423039131257x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><br>
<div class="m_-4521912423039131257x_gmail_extra"><br>
<div class="m_-4521912423039131257x_gmail_quote">On Thu, Sep 20, 2018 at 1:29 AM, Raghavendra Gowdappa <span dir="ltr">
<<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br>
<blockquote class="m_-4521912423039131257x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Can you give volume info? Looks like you are using 2 way replica.<br>
</div>
</blockquote>
<div><br>
</div>
<div>Yes indeed.<br>
gluster volume create gvol0 replica 2 gfs01:/glusterdata/brick1/gvol0 gfs02:/glusterdata/brick2/gvol0<br>
<br>
</div>
<div>+Pranith. +Ravi.<br>
<br>
</div>
<div>Not sure whether 2 way replication has caused this. From what I understand we need either 3 way replication or arbiter for correct resolution of heals.<br>
<br>
</div>
<blockquote class="m_-4521912423039131257x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"></div>
<div class="m_-4521912423039131257x_m_7256587473638831700m_-5372146261027786726HOEnZb">
<div class="m_-4521912423039131257x_m_7256587473638831700m_-5372146261027786726h5">
<div class="m_-4521912423039131257x_gmail_extra"><br>
<div class="m_-4521912423039131257x_gmail_quote">On Wed, Sep 19, 2018 at 9:39 AM, Johan Karlsson <span dir="ltr">
<<a href="mailto:Johan.Karlsson@dgc.se" target="_blank">Johan.Karlsson@dgc.se</a>></span> wrote:<br>
<blockquote class="m_-4521912423039131257x_gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I have two servers setup with glusterFS in replica mode, a single volume exposed via a mountpoint. The servers are running Ubuntu 16.04 LTS<br>
<br>
After a package upgrade + reboot of both servers, it was discovered that the data was completely gone. New data written on the volume via the mountpoint is replicated correctly, and gluster status/info commands states that everything is ok (no split brain scenario
or any healing needed etc). But the previous data is completely gone, not even present on any of the bricks.<br>
<br>
The following upgrade was done:<br>
<br>
glusterfs-server:amd64 (4.1.0-ubuntu1~xenial3 -> 4.1.4-ubuntu1~xenial1)<br>
glusterfs-client:amd64 (4.1.0-ubuntu1~xenial3 -> 4.1.4-ubuntu1~xenial1)<br>
glusterfs-common:amd64 (4.1.0-ubuntu1~xenial3 -> 4.1.4-ubuntu1~xenial1)<br>
<br>
The logs only show that connection between the servers was lost, which is expected.<br>
<br>
I can't even determine if it was the package upgrade or the reboot that caused this issue, but I've tried to recreate the issue without success.<br>
<br>
Any idea what could have gone wrong, or if I have done some wrong during the setup. For reference, this is how I've done the setup:<br>
<br>
---<br>
Add a separate disk with a single partition on both servers (/dev/sdb1)<br>
<br>
Add gfs hostnames for direct communication without DNS, on both servers:<br>
<br>
/etc/hosts<br>
<br>
192.168.4.45 gfs01<br>
192.168.4.46 gfs02<br>
<br>
On gfs01, create a new LVM Volume Group:<br>
vgcreate gfs01-vg /dev/sdb1<br>
<br>
And on the gfs02:<br>
vgcreate gfs02-vg /dev/sdb1<br>
<br>
Create logical volumes named "brick" on the servers:<br>
<br>
gfs01:<br>
lvcreate -l 100%VG -n brick1 gfs01-vg<br>
gfs02:<br>
lvcreate -l 100%VG -n brick2 gfs02-vg<br>
<br>
Format the volumes with ext4 filesystem:<br>
<br>
gfs01:<br>
mkfs.ext4 /dev/gfs01-vg/brick1<br>
gfs02:<br>
mkfs.ext4 /dev/gfs02-vg/brick2<br>
<br>
Create a mountpoint for the bricks on the servers:<br>
<br>
gfs01:<br>
mkdir -p /glusterdata/brick1<br>
gds02:<br>
mkdir -p /glusterdata/brick2<br>
<br>
Make a permanent mount on the servers:<br>
<br>
gfs01:<br>
/dev/gfs01-vg/brick1 /glusterdata/brick1 ext4 defaults 0 0<br>
gfs02:<br>
/dev/gfs02-vg/brick2 /glusterdata/brick2 ext4 defaults 0 0<br>
<br>
Mount it:<br>
mount -a<br>
<br>
Create a gluster volume mount point on the bricks on the servers:<br>
<br>
gfs01:<br>
mkdir -p /glusterdata/brick1/gvol0<br>
gfs02:<br>
mkdir -p /glusterdata/brick2/gvol0<br>
<br>
>From each server, peer probe the other one:<br>
<br>
gluster peer probe gfs01<br>
peer probe: success<br>
<br>
gluster peer probe gfs02<br>
peer probe: success<br>
<br>
>From any single server, create the gluster volume as a "replica" with two nodes; gfs01 and gfs02:<br>
<br>
gluster volume create gvol0 replica 2 gfs01:/glusterdata/brick1/gvol0 gfs02:/glusterdata/brick2/gvol0<br>
<br>
Start the volume:<br>
<br>
gluster volume start gvol0<br>
<br>
On each server, mount the gluster filesystem on the /filestore mount point:<br>
<br>
gfs01:<br>
mount -t glusterfs gfs01:/gvol0 /filestore<br>
gfs02:<br>
mount -t glusterfs gfs02:/gvol0 /filestore<br>
<br>
Make the mount permanent on the servers:<br>
<br>
/etc/fstab<br>
<br>
gfs01:<br>
gfs01:/gvol0 /filestore glusterfs defaults,_netdev 0 0<br>
gfs02:<br>
gfs02:/gvol0 /filestore glusterfs defaults,_netdev 0 0<br>
---<br>
<br>
Regards,<br>
<br>
Johan Karlsson<br>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="m_-4521912423039131257x_m_7256587473638831700gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<br>
-- <br>
<div dir="ltr" class="m_-4521912423039131257x_gmail_signature">
<div dir="ltr">Pranith<br>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div></div></div>