[Gluster-users] Problem with SSL/TLS encryption on Gluster 4.0 & 4.1

Thu Jul 12 08:45:50 UTC 2018

Hello,

I am doing some tests with GlusterFS 4.0 and I can't seem to solve some 
SSL/TLS issues. I am trying to set up a 2 node replicated gluster volume 
with SSL/TLS. For this setup, I use 3 KVM VMs (2 storage nodes + 1 
client node). For the networking part, I use a dedicated private LAN for 
the KVM VMs. Each VM is able to ping the other, so there's no problem 
with the connectivity.

To try to make the procedure I used as clear as possible, I will put all 
commands in chronological order:

=====================================================

1. First, I update the systems, install ntp and then reboot:

yum update

yum install ntp

systemctl status ntpd
systemctl start ntpd
systemctl enable ntpd
systemctl status ntpd

=====================================================

2. I use a separate partition within the two VM storage nodes. Each of 
the two nodes see the separate partition as /dev/sdb. After creating a 
thinly provisioned LV on each of the two nodes, I create an XFS 
filesystem on them:

pvcreate /dev/sdb1

vgcreate -s 32M vg_glusterfs /dev/sdb1

lvcreate -L 20G --thinpool glusterfs_thin_pool vg_glusterfs

lvcreate -V 15G --thin -n glusterfs_thin_vol1 
vg_glusterfs/glusterfs_thin_pool

mkfs.xfs -i size=512 /dev/vg_glusterfs/glusterfs_thin_vol1

=====================================================

3. On each node, I configured the brick data partition and added the 
following to /etc/fstab:

mkdir /data

echo "/dev/vg_glusterfs/glusterfs_thin_vol1 /data xfs defaults 1 2" >> 
/etc/fstab

mount -a

=====================================================

4. After mounting the volume, I see the following in df -Th, which is 
correct:

[root at gluster1 brick1]# df -Th
Filesystem                                   Type      Size  Used Avail 
Use% Mounted on
/dev/sda1                                    ext4       46G  1.4G 42G   4% /
devtmpfs                                     devtmpfs  3.9G     0 3.9G   
0% /dev
tmpfs                                        tmpfs     3.9G     0 3.9G   
0% /dev/shm
tmpfs                                        tmpfs     3.9G  8.6M 3.9G   
1% /run
tmpfs                                        tmpfs     3.9G     0 3.9G   
0% /sys/fs/cgroup
tmpfs                                        tmpfs     783M     0 783M   
0% /run/user/0
/dev/mapper/vg_glusterfs-glusterfs_thin_vol1 xfs        15G   34M 15G   
1% /data

=====================================================

5. Create specific volume dirs on all storage nodes:

mkdir -pv /data/glusterfs/${HOSTNAME%%.*}/vol01

=====================================================

6. Add entries in /etc/hosts:

vim /etc/hosts

192.168.10.233    gluster1
192.168.10.234    gluster2
192.168.10.237    gluster-client

=====================================================

7. Install gluster from the CentOS SIG:

yum search centos-release-gluster
yum install centos-release-gluster40

yum install glusterfs-server

=====================================================

8. Set up TLS/SSL encryption on all nodes and clients (gluster1, 
gluster2, gluster-client):

openssl genrsa -out /etc/ssl/glusterfs.key 2048

In gluster1 node:
openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=gluster1" 
-out /etc/ssl/glusterfs.pem
In gluster2 node:
openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj "/CN=gluster2" 
-out /etc/ssl/glusterfs.pem
In gluster-client node:
openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj 
"/CN=gluster-client" -out /etc/ssl/glusterfs.pem

=====================================================

9. On another box, I concatenate all of the .pem certificates into a .ca 
file.

Bring all .pem files locally:

scp gluster1:/etc/ssl/glusterfs.pem gluster01.pem
scp gluster2:/etc/ssl/glusterfs.pem gluster02.pem
scp gluster-client:/etc/ssl/glusterfs.pem gluster-client.pem

For storage nodes, I concatenate all .pem certificates (including the 
client's .pem):

cat gluster01.pem gluster02.pem gluster-client.pem > glusterfs-nodes.ca

For server clients:

cat gluster01.pem gluster02.pem > glusterfs-client.ca

=====================================================

10. Put glusterfs-nodes.ca file on all the storage nodes (this includes 
storage nodes .pem + client's .pem):

scp glusterfs-nodes.ca gluster1:/etc/ssl/glusterfs.ca
scp glusterfs-nodes.ca gluster2:/etc/ssl/glusterfs.ca

Put glusterfs-client.ca file on all the storage nodes (this includes 
only storage nodes .pem):

scp glusterfs-client.ca gluster-client:/etc/ssl/glusterfs.ca

=====================================================

11. Enable management encryption on each node (command ran on gluster1, 
gluster2, gluster-client):

touch /var/lib/glusterd/secure-access

=====================================================

12. Start, enable and check status of glusterd on gluster1 and gluster2:

systemctl start glusterd
systemctl enable glusterd
systemctl status glusterd

=====================================================

13. Configure trusted storage pool (TSP):

 From gluster1:

gluster peer status

gluster peer probe gluster2

=====================================================

14. Create the replicated gluster volume but don't start it:

gluster volume create vol01 replica 2 transport tcp 
gluster1:/data/glusterfs/gluster1/vol01/brick1 
gluster2:/data/glusterfs/gluster2/vol01/brick1

=====================================================

15. Setup SSL/TLS access to the volume:

gluster volume set vol01 auth.ssl-allow 'gluster01,gluster02,gluster-client'

gluster volume set vol01 client.ssl on
gluster volume set vol01 server.ssl on

gluster volume set vol01 network.ping-timeout "5"

gluster volume start vol01

=====================================================

16. Mount the volume on gluster-client:

mount -t glusterfs gluster1:/vol01 /mnt

After mounting the volume, df -h has the correct output:

[root at gluster-client mnt]# df -h
Filesystem       Size  Used Avail Use% Mounted on
/dev/sda1         48G   36G  9.3G  80% /
devtmpfs         1.9G     0  1.9G   0% /dev
tmpfs            1.9G     0  1.9G   0% /dev/shm
tmpfs            1.9G  8.5M  1.9G   1% /run
tmpfs            1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs            379M     0  379M   0% /run/user/0
gluster1:/vol01   15G  187M   15G   2% /mnt

=====================================================

17. If I try to copy an archive, it throws an error after transferring 
part of the file and the /mnt folder becomes inaccessible for a while:

[root at gluster-client mnt]# cp /root/mybee.tar.gz /mnt/
cp: error writing ‘/mnt/mybee.tar.gz’: Transport endpoint is not connected
cp: failed to extend ‘/mnt/mybee.tar.gz’: Transport endpoint is not 
connected
cp: failed to close ‘/mnt/mybee.tar.gz’: Transport endpoint is not connected
[root at gluster-client mnt]# ll /mnt
ls: cannot open directory .: Transport endpoint is not connected

=====================================================

This is the log from gluster-client for the mnt mountpoint (mnt.log):

[2018-06-29 07:52:03.869382] I [MSGID: 100030] [glusterfsd.c:2625:main] 
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 4.0.2 
(args: /usr/sbin/glusterfs --process-name fuse --volfile-server=gluster1 
--volfile-id=/vol01 /mnt)
[2018-06-29 07:52:03.878709] I [socket.c:4470:socket_init] 0-glusterfs: 
SSL support on the I/O path is ENABLED
[2018-06-29 07:52:03.878782] I [socket.c:4473:socket_init] 0-glusterfs: 
SSL support for glusterd is ENABLED
[2018-06-29 07:52:03.878795] I [socket.c:4490:socket_init] 0-glusterfs: 
using private polling thread
[2018-06-29 07:52:03.879108] E [socket.c:4541:socket_init] 0-glusterfs: 
failed to open /etc/ssl/dhparam.pem, DH ciphers are disabled
[2018-06-29 07:52:03.885718] I [MSGID: 101190] 
[event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 1
[2018-06-29 07:52:03.901983] I [MSGID: 101190] 
[event-epoll.c:609:event_dispatch_epoll_worker] 0-epoll: Started thread 
with index 2
[2018-06-29 07:52:03.902092] I [socket.c:4470:socket_init] 
0-vol01-client-1: SSL support on the I/O path is ENABLED
[2018-06-29 07:52:03.902117] I [socket.c:4473:socket_init] 
0-vol01-client-1: SSL support for glusterd is ENABLED
[2018-06-29 07:52:03.902142] I [socket.c:4490:socket_init] 
0-vol01-client-1: using private polling thread
[2018-06-29 07:52:03.902266] E [socket.c:4541:socket_init] 
0-vol01-client-1: failed to open /etc/ssl/dhparam.pem, DH ciphers are 
disabled
[2018-06-29 07:52:03.902977] I [socket.c:4470:socket_init] 
0-vol01-client-0: SSL support on the I/O path is ENABLED
[2018-06-29 07:52:03.903018] I [socket.c:4473:socket_init] 
0-vol01-client-0: SSL support for glusterd is ENABLED
[2018-06-29 07:52:03.903028] I [socket.c:4490:socket_init] 
0-vol01-client-0: using private polling thread
[2018-06-29 07:52:03.903161] E [socket.c:4541:socket_init] 
0-vol01-client-0: failed to open /etc/ssl/dhparam.pem, DH ciphers are 
disabled
[2018-06-29 07:52:03.903756] I [MSGID: 114020] [client.c:2300:notify] 
0-vol01-client-0: parent translators are ready, attempting connect on 
transport
[2018-06-29 07:52:03.908432] I [MSGID: 114020] [client.c:2300:notify] 
0-vol01-client-1: parent translators are ready, attempting connect on 
transport
Final graph:
+------------------------------------------------------------------------------+
   1: volume vol01-client-0
   2:     type protocol/client
   3:     option ping-timeout 5
   4:     option remote-host gluster1
   5:     option remote-subvolume /data/glusterfs/gluster1/vol01/brick1
   6:     option transport-type socket
   7:     option transport.address-family inet
   8:     option transport.socket.ssl-enabled on
   9:     option transport.tcp-user-timeout 0
  10:     option transport.socket.keepalive-time 20
  11:     option transport.socket.keepalive-interval 2
  12:     option transport.socket.keepalive-count 9
  13:     option send-gids true
  14: end-volume
  15:
  16: volume vol01-client-1
  17:     type protocol/client
  18:     option ping-timeout 5
  19:     option remote-host gluster2
  20:     option remote-subvolume /data/glusterfs/gluster2/vol01/brick1
  21:     option transport-type socket
  22:     option transport.address-family inet
  23:     option transport.socket.ssl-enabled on
  24:     option transport.tcp-user-timeout 0
  25:     option transport.socket.keepalive-time 20
  26:     option transport.socket.keepalive-interval 2
  27:     option transport.socket.keepalive-count 9
  28:     option send-gids true
  29: end-volume
  30:
  31: volume vol01-replicate-0
  32:     type cluster/replicate
  33:     option afr-pending-xattr vol01-client-0,vol01-client-1
  34:     option use-compound-fops off
  35:     subvolumes vol01-client-0 vol01-client-1
  36: end-volume
  37:
  38: volume vol01-dht
  39:     type cluster/distribute
  40:     option lock-migration off
  41:     option force-migration off
  42:     subvolumes vol01-replicate-0
  43: end-volume
  44:
  45: volume vol01-write-behind
  46:     type performance/write-behind
  47:     subvolumes vol01-dht
  48: end-volume
  49:
  50: volume vol01-read-ahead
  51:     type performance/read-ahead
  52:     subvolumes vol01-write-behind
  53: end-volume
  54:
  55: volume vol01-readdir-ahead
  56:     type performance/readdir-ahead
  57:     option parallel-readdir off
  58:     option rda-request-size 131072
  59:     option rda-cache-limit 10MB
  60:     subvolumes vol01-read-ahead
  61: end-volume
  62:
  63: volume vol01-io-cache
  64:     type performance/io-cache
  65:     subvolumes vol01-readdir-ahead
  66: end-volume
  67:
  68: volume vol01-quick-read
  69:     type performance/quick-read
  70:     subvolumes vol01-io-cache
  71: end-volume
  72:
  73: volume vol01-open-behind
  74:     type performance/open-behind
  75:     subvolumes vol01-quick-read
  76: end-volume
  77:
  78: volume vol01-md-cache
  79:     type performance/md-cache
  80:     subvolumes vol01-open-behind
  81: end-volume
  82:
  83: volume vol01
  84:     type debug/io-stats
  85:     option log-level INFO
  86:     option latency-measurement off
  87:     option count-fop-hits off
  88:     subvolumes vol01-md-cache
  89: end-volume
  90:
  91: volume meta-autoload
  92:     type meta
  93:     subvolumes vol01
  94: end-volume
  95:
+------------------------------------------------------------------------------+
[2018-06-29 07:52:03.917518] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.918459] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.919289] I [rpc-clnt.c:2071:rpc_clnt_reconfig] 
0-vol01-client-0: changing port to 49152 (from 0)
[2018-06-29 07:52:03.922903] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.923922] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.925115] I [rpc-clnt.c:2071:rpc_clnt_reconfig] 
0-vol01-client-1: changing port to 49152 (from 0)
[2018-06-29 07:52:03.930023] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.930756] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.931859] I [MSGID: 114046] 
[client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0: 
Connected to vol01-client-0, attached to remote volume 
'/data/glusterfs/gluster1/vol01/brick1'.
[2018-06-29 07:52:03.931894] I [MSGID: 108005] 
[afr-common.c:5081:__afr_handle_child_up_event] 0-vol01-replicate-0: 
Subvolume 'vol01-client-0' came back up; going online.
[2018-06-29 07:52:03.938903] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.939803] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:03.941300] I [MSGID: 114046] 
[client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-1: 
Connected to vol01-client-1, attached to remote volume 
'/data/glusterfs/gluster2/vol01/brick1'.
[2018-06-29 07:52:03.942884] I [fuse-bridge.c:4234:fuse_init] 
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 
kernel 7.22
[2018-06-29 07:52:03.942907] I [fuse-bridge.c:4864:fuse_graph_sync] 
0-fuse: switched to graph 0
[2018-06-29 07:52:03.946767] W [socket.c:592:__socket_rwv] 
0-vol01-client-0: readv on 192.168.10.233:49152 failed (Input/output error)
[2018-06-29 07:52:03.946791] E [socket.c:2785:socket_poller] 
0-vol01-client-0: socket_poller 192.168.10.233:49152 failed 
(Input/output error)
[2018-06-29 07:52:03.946837] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:03.947264] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(LOOKUP(27)) called at 2018-06-29 07:52:03.945958 (xid=0xb)
[2018-06-29 07:52:03.947297] W [MSGID: 114031] 
[client-rpc-fops_v2.c:2540:client4_0_lookup_cbk] 0-vol01-client-0: 
remote operation failed. Path: / (00000000-0000-0000-0000-000000000001) 
[Transport endpoint is not connected]
[2018-06-29 07:52:03.947455] I [MSGID: 109005] 
[dht-selfheal.c:2328:dht_selfheal_directory] 0-vol01-dht: Directory 
selfheal failed: Unable to form layout for directory /
[2018-06-29 07:52:07.885389] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:07.886457] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:07.887452] I [rpc-clnt.c:2071:rpc_clnt_reconfig] 
0-vol01-client-0: changing port to 49152 (from 0)
[2018-06-29 07:52:07.900627] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:07.901594] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:07.903220] I [MSGID: 114046] 
[client-handshake.c:1176:client_setvolume_cbk] 0-vol01-client-0: 
Connected to vol01-client-0, attached to remote volume 
'/data/glusterfs/gluster1/vol01/brick1'.
[2018-06-29 07:52:13.062404] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:13.062780] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.048761 (xid=0x22)
[2018-06-29 07:52:13.062826] W [MSGID: 114031] 
[client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0: remote 
operation failed [Transport endpoint is not connected]
[2018-06-29 07:52:13.063017] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.049162 (xid=0x23)
[2018-06-29 07:52:13.063175] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.049865 (xid=0x24)
[2018-06-29 07:52:13.063319] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.050199 (xid=0x25)
[2018-06-29 07:52:13.063473] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.050489 (xid=0x26)
[2018-06-29 07:52:13.063600] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.050757 (xid=0x27)
[2018-06-29 07:52:13.063745] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.051031 (xid=0x28)
[2018-06-29 07:52:13.063883] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.053035 (xid=0x29)
The message "W [MSGID: 114031] 
[client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-0: remote 
operation failed [Transport endpoint is not connected]" repeated 7 times 
between [2018-06-29 07:52:13.062826] and [2018-06-29 07:52:13.063903]
[2018-06-29 07:52:13.104580] W [MSGID: 114031] 
[client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-0: 
remote operation failed
[2018-06-29 07:52:13.109230] W [socket.c:592:__socket_rwv] 
0-vol01-client-1: writev on 192.168.10.234:49152 failed (No data available)
[2018-06-29 07:52:13.109282] E [socket.c:2777:socket_poller] 
0-vol01-client-1: poll error on socket
[2018-06-29 07:52:13.109340] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:13.109357] E [MSGID: 108006] 
[afr-common.c:5158:__afr_handle_child_down_event] 0-vol01-replicate-0: 
All subvolumes are down. Going offline until atleast one of them comes 
back up.
[2018-06-29 07:52:13.109461] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:13.109679] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FSTAT(25)) called at 2018-06-29 07:52:13.104513 (xid=0x42)
[2018-06-29 07:52:13.109700] W [MSGID: 114031] 
[client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: remote 
operation failed [Transport endpoint is not connected]
[2018-06-29 07:52:13.109854] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FXATTROP(34)) called at 2018-06-29 07:52:13.104729 (xid=0x43)
[2018-06-29 07:52:13.109877] W [MSGID: 114031] 
[client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-1: 
remote operation failed
[2018-06-29 07:52:13.109897] E [MSGID: 114031] 
[client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 0-vol01-client-0: 
remote operation failed [Transport endpoint is not connected]
[2018-06-29 07:52:13.110036] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(WRITE(13)) called at 2018-06-29 07:52:13.105623 (xid=0x44)
[2018-06-29 07:52:13.110053] W [MSGID: 114031] 
[client-rpc-fops_v2.c:658:client4_0_writev_cbk] 0-vol01-client-1: remote 
operation failed [Transport endpoint is not connected]
[2018-06-29 07:52:13.110232] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FSTAT(25)) called at 2018-06-29 07:52:13.105658 (xid=0x45)
[2018-06-29 07:52:13.110250] W [MSGID: 114031] 
[client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: remote 
operation failed [Transport endpoint is not connected]
[2018-06-29 07:52:13.110386] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FSTAT(25)) called at 2018-06-29 07:52:13.106862 (xid=0x46)
[2018-06-29 07:52:13.110530] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FSTAT(25)) called at 2018-06-29 07:52:13.108103 (xid=0x47)
[2018-06-29 07:52:13.110660] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FSTAT(25)) called at 2018-06-29 07:52:13.108969 (xid=0x48)
[2018-06-29 07:52:13.110793] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FXATTROP(34)) called at 2018-06-29 07:52:13.109256 (xid=0x49)
The message "W [MSGID: 114031] 
[client-rpc-fops_v2.c:1260:client4_0_fstat_cbk] 0-vol01-client-1: remote 
operation failed [Transport endpoint is not connected]" repeated 3 times 
between [2018-06-29 07:52:13.110250] and [2018-06-29 07:52:13.110675]
[2018-06-29 07:52:13.110808] W [MSGID: 114031] 
[client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-1: 
remote operation failed
[2018-06-29 07:52:13.110933] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GlusterFS 4.x v1) 
op(FXATTROP(34)) called at 2018-06-29 07:52:13.109284 (xid=0x4a)
The message "W [MSGID: 114031] 
[client-rpc-fops_v2.c:1570:client4_0_fxattrop_cbk] 0-vol01-client-1: 
remote operation failed" repeated 3 times between [2018-06-29 
07:52:13.110808] and [2018-06-29 07:52:14.878874]
[2018-06-29 07:52:14.878895] E [MSGID: 114031] 
[client-rpc-fops_v2.c:1352:client4_0_finodelk_cbk] 0-vol01-client-1: 
remote operation failed [Transport endpoint is not connected]
[2018-06-29 07:52:14.879095] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:14.879217] W [fuse-bridge.c:2427:fuse_writev_cbk] 
0-glusterfs-fuse: 76: WRITE => -1 
gfid=a8848be6-1c05-4a72-88a7-5209eeef7c2c fd=0x7f7c50023a68 (Transport 
endpoint is not connected)
[2018-06-29 07:52:14.880509] W [fuse-bridge.c:1406:fuse_err_cbk] 
0-glusterfs-fuse: 77: FLUSH() ERR => -1 (Transport endpoint is not 
connected)
The message "I [MSGID: 108006] [afr-common.c:5444:afr_local_init] 
0-vol01-replicate-0: no subvolumes up" repeated 11 times between 
[2018-06-29 07:52:14.879095] and [2018-06-29 07:52:17.908056]
[2018-06-29 07:52:17.908066] E [MSGID: 101046] 
[dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
[2018-06-29 07:52:17.908175] W [fuse-bridge.c:896:fuse_attr_cbk] 
0-glusterfs-fuse: 79: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:18.758975] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:18.759155] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:18.759165] E [MSGID: 101046] 
[dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
[2018-06-29 07:52:18.759214] W [fuse-bridge.c:896:fuse_attr_cbk] 
0-glusterfs-fuse: 80: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:21.662626] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:21.662756] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:21.662765] E [MSGID: 101046] 
[dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
[2018-06-29 07:52:21.662811] W [fuse-bridge.c:896:fuse_attr_cbk] 
0-glusterfs-fuse: 81: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:21.663567] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:21.663630] W [fuse-bridge.c:3205:fuse_statfs_cbk] 
0-glusterfs-fuse: 82: ERR => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:23.896362] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:23.896687] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:23.897321] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:23.896427 (xid=0x2a)
[2018-06-29 07:52:23.897353] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:23.903241] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:23.903566] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:23.904166] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:23.903281 (xid=0x4b)
[2018-06-29 07:52:23.904199] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:26.909819] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:26.910175] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:26.910858] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:26.909871 (xid=0x2b)
[2018-06-29 07:52:26.910888] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:26.913921] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:26.914163] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:26.914417] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:26.913961 (xid=0x4c)
[2018-06-29 07:52:26.914443] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:29.921628] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:29.921867] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:29.922215] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:29.921674 (xid=0x2c)
[2018-06-29 07:52:29.922257] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:29.926462] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:29.926774] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:29.927415] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:29.926499 (xid=0x4d)
[2018-06-29 07:52:29.927447] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:32.935218] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:32.935553] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:32.936196] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:32.935271 (xid=0x2d)
[2018-06-29 07:52:32.936230] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:32.938940] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:32.939232] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:32.939497] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:32.938982 (xid=0x4e)
[2018-06-29 07:52:32.939526] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:35.948789] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:35.949172] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:35.949860] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:35.948839 (xid=0x2e)
[2018-06-29 07:52:35.949901] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:35.952154] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:35.952408] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:35.952670] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:35.952199 (xid=0x4f)
[2018-06-29 07:52:35.952699] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:38.959369] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:38.959733] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:38.960536] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:38.959426 (xid=0x2f)
[2018-06-29 07:52:38.960573] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:38.963381] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:38.963618] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:38.963877] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:38.963428 (xid=0x50)
[2018-06-29 07:52:38.963906] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:41.969152] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:41.969397] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:41.969731] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:41.969204 (xid=0x30)
[2018-06-29 07:52:41.969769] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:41.974130] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:41.974381] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:41.974669] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:41.974198 (xid=0x51)
[2018-06-29 07:52:41.974698] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:44.984465] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:44.984781] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:44.985444] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:44.984514 (xid=0x31)
[2018-06-29 07:52:44.985477] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:44.990039] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:44.990352] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:44.990729] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:44.990094 (xid=0x52)
[2018-06-29 07:52:44.990771] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:47.994448] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:47.994684] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:47.994947] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:47.994499 (xid=0x32)
[2018-06-29 07:52:47.994984] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:48.002544] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:48.002809] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:48.003388] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:48.002577 (xid=0x53)
[2018-06-29 07:52:48.003416] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:49.440101] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:49.440257] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:49.440266] E [MSGID: 101046] 
[dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
[2018-06-29 07:52:49.440310] W [fuse-bridge.c:896:fuse_attr_cbk] 
0-glusterfs-fuse: 83: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:50.182948] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:50.183108] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:50.183116] E [MSGID: 101046] 
[dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
[2018-06-29 07:52:50.197572] W [fuse-bridge.c:896:fuse_attr_cbk] 
0-glusterfs-fuse: 84: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:50.575887] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:50.576030] I [MSGID: 108006] 
[afr-common.c:5444:afr_local_init] 0-vol01-replicate-0: no subvolumes up
[2018-06-29 07:52:50.576039] E [MSGID: 101046] 
[dht-common.c:1501:dht_lookup_dir_cbk] 0-vol01-dht: dict is null
[2018-06-29 07:52:50.576087] W [fuse-bridge.c:896:fuse_attr_cbk] 
0-glusterfs-fuse: 85: LOOKUP() / => -1 (Transport endpoint is not connected)
[2018-06-29 07:52:51.012626] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-0: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:51.012924] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-0: disconnected from 
vol01-client-0. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:51.013615] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-0: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:51.012667 (xid=0x33)
[2018-06-29 07:52:51.013647] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-0: 
received RPC status error [Transport endpoint is not connected]
[2018-06-29 07:52:51.018855] W [rpc-clnt.c:1739:rpc_clnt_submit] 
0-vol01-client-1: error returned while attempting to connect to 
host:(null), port:0
[2018-06-29 07:52:51.019093] I [MSGID: 114018] 
[client.c:2227:client_rpc_notify] 0-vol01-client-1: disconnected from 
vol01-client-1. Client process will keep trying to connect to glusterd 
until brick's port is available
[2018-06-29 07:52:51.019378] E [rpc-clnt.c:350:saved_frames_unwind] (--> 
/lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f7c73b8ae5b] (--> 
/lib64/libgfrpc.so.0(+0xce4e)[0x7f7c73955e4e] (--> 
/lib64/libgfrpc.so.0(+0xcf6e)[0x7f7c73955f6e] (--> 
/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x8d)[0x7f7c7395760d] 
(--> /lib64/libgfrpc.so.0(+0xf178)[0x7f7c73958178] ))))) 
0-vol01-client-1: forced unwinding frame type(GF-DUMP) op(DUMP(1)) 
called at 2018-06-29 07:52:51.018893 (xid=0x54)
[2018-06-29 07:52:51.019408] W [MSGID: 114032] 
[client-handshake.c:1620:client_dump_version_cbk] 0-vol01-client-1: 
received RPC status error [Transport endpoint is not connected]

=====================================================

After trying to copy the archive, I end up with different sized files on 
the two bricks:

[root at gluster1 brick1]# pwd
/data/glusterfs/gluster1/vol01/brick1
[root at gluster1 brick1]# ll
total 384
-rw------- 2 root root 393216 Jun 29 10:52 mybee.tar.gz

[root at gluster2 brick1]# pwd
/data/glusterfs/gluster2/vol01/brick1
[root at gluster2 brick1]# ll
total 1280
-rw------- 2 root root 1310720 Jun 29 10:52 mybee.tar.gz

=====================================================

These are the informations regarding the gluster volume:

[root at gluster1 glusterfs]# gluster volume info vol01

Volume Name: vol01
Type: Replicate
Volume ID: d3b39520-3fbb-4cfc-b736-338d7726b5a1
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: gluster1:/data/glusterfs/gluster1/vol01/brick1
Brick2: gluster2:/data/glusterfs/gluster2/vol01/brick1
Options Reconfigured:
network.ping-timeout: 5
server.ssl: on
client.ssl: on
auth.ssl-allow: gluster01,gluster02,gluster-client
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

=====================================================

Here is the peers information:

[root at gluster1 brick1]# gluster peer status
Number of Peers: 1

Hostname: gluster2
Uuid: 9ccac8e3-71c1-43e4-b723-fb7374be8a1d
State: Peer in Cluster (Connected)

=====================================================

Here is the volume status:

[root at gluster1 brick1]# gluster volume status vol01
Status of volume: vol01
Gluster process                             TCP Port  RDMA Port Online  Pid
------------------------------------------------------------------------------
Brick gluster1:/data/glusterfs/gluster1/vol
01/brick1                                   49152     0 Y       11108
Brick gluster2:/data/glusterfs/gluster2/vol
01/brick1                                   49152     0 Y       10942
Self-heal Daemon on localhost               N/A       N/A Y       11133
Self-heal Daemon on gluster2                N/A       N/A Y       10968

Task Status of Volume vol01
------------------------------------------------------------------------------
There are no active volume tasks

=====================================================

These are the installed packages on gluster-client:

[root at gluster-client glusterfs]# rpm -qa | grep "gluster\|fuse"
centos-release-gluster40-1.0-2.el7.centos.x86_64
glusterfs-client-xlators-4.0.2-1.el7.x86_64
fuse-2.9.2-10.el7.x86_64
glusterfs-4.0.2-1.el7.x86_64
glusterfs-fuse-4.0.2-1.el7.x86_64
fuse-libs-2.9.2-10.el7.x86_64
glusterfs-libs-4.0.2-1.el7.x86_64

And these are the installed packages on gluster1 and gluster2 storage nodes:

[root at gluster1 glusterfs]# rpm -qa | grep "gluster\|fuse"
glusterfs-api-4.0.2-1.el7.x86_64
centos-release-gluster40-1.0-2.el7.centos.x86_64
glusterfs-libs-4.0.2-1.el7.x86_64
glusterfs-client-xlators-4.0.2-1.el7.x86_64
glusterfs-cli-4.0.2-1.el7.x86_64
glusterfs-fuse-4.0.2-1.el7.x86_64
glusterfs-server-4.0.2-1.el7.x86_64
glusterfs-4.0.2-1.el7.x86_64

Can anyone point me in the right direction? What am I doing wrong? The 
mount works fine until I try to copy an archive, multiple smaller files 
or a bigger file on it (meaning it shows correctly in df -Th and I can 
create several files with "touch file1 file2..."). Basically, after any 
data transfer, I get these errors.

I followed the indications from the redhat page:

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption

UPDATE 1:
I tried doing the exact same steps in Gluster 3.12 and had no problem. 
The steps worked and SSL/TLS was enabled. There was no transport error 
or anything and I also checked if SSL/TLS was enabled. Afterwards, I 
also tried with the new release 4.1 and the problem persists (same error 
with "Transport endpoint is not connected").
Let me know if you need any other info. Any help is much appreciated.

Thank you.