<div dir="ltr">Thanks David Spisla for pitching in.<div><br></div><div>Havriliuc, if you can file the bug as requested, we would like to check this out.</div><div><br></div><div>-Amar</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jul 13, 2018 at 3:44 PM, David Spisla <span dir="ltr">&lt;<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>Hello Andrei,</div><div><br></div><div>I did some research. But I didn&#39;t find a bug report for this issue. If you want you can create such a report here:</div><div><a href="https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS" target="_blank">https://bugzilla.redhat.com/<wbr>enter_bug.cgi?product=<wbr>GlusterFS</a></div><div><br></div><div>It is the recommended way in GlusterFS to report a bug, github issues are for new features<br></div><div><br></div><div>Regards</div><span class="HOEnZb"><font color="#888888"><div>David<br></div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">2018-07-13 11:44 GMT+02:00 Havriliuc Andrei <span dir="ltr">&lt;<a href="mailto:andrei@hostvision.ro" target="_blank">andrei@hostvision.ro</a>&gt;</span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
  
    
  
  <div text="#000000" bgcolor="#FFFFFF">
    <p>Hello David,</p>
    <p><br>
    </p>
    <p>Yeah. I also disabled SSL/TLS for management and communication
      for Gluster. The problem is that I will have traffic over public
      WAN for a geo replication instance and I cannot go forward until
      this is solved.</p>
    <p><br>
    </p>
    <p>Was there a bug filed about this? I&#39;ve looked at the issues on
      the Glusterfs github page but I didn&#39;t find anything related to
      this:</p>
    <p><br>
    </p>
    <p><a href="https://github.com/gluster/glusterfs/issues" target="_blank">https://github.com/gluster/glu<wbr>sterfs/issues</a><br>
    </p>
    <p><br>
    </p>
    <p>Regards,</p>
    <p>Andrei<br>
    </p><div><div>
    <br>
    <div>On 7/12/18 4:03 PM, David Spisla wrote:<br>
    </div>
    <blockquote type="cite">
      <div dir="ltr">
        <div>Hello Andrei,</div>
        <div><br>
        </div>
        <div>I am also using Gluster 4.1 on CentOS and I have the same
          problem. I tested it with a volume which had no network
          encryption and one with network encryption. You are not the
          only one.</div>
        <div>It seems to be a bug. At the moment there is no other
          choice to disable client.ssl and sever.ssl on a volume to have
          stable I/O.</div>
        <div><br>
        </div>
        <div>Regards</div>
        <div>David</div>
        <div><br>
        </div>
      </div>
      <div class="gmail_extra"><br>
        <div class="gmail_quote">2018-07-12 10:45 GMT+02:00 Havriliuc
          Andrei <span dir="ltr">&lt;<a href="mailto:andrei@hostvision.ro" target="_blank">andrei@hostvision.ro</a>&gt;</span>:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hello,<br>
            <br>
            I am doing some tests with GlusterFS 4.0 and I can&#39;t seem to
            solve some SSL/TLS issues. I am trying to set up a 2 node
            replicated gluster volume with SSL/TLS. For this setup, I
            use 3 KVM VMs (2 storage nodes + 1 client node). For the
            networking part, I use a dedicated private LAN for the KVM
            VMs. Each VM is able to ping the other, so there&#39;s no
            problem with the connectivity.<br>
            <br>
            To try to make the procedure I used as clear as possible, I
            will put all commands in chronological order:<br>
            <br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            1. First, I update the systems, install ntp and then reboot:<br>
            <br>
            yum update<br>
            <br>
            yum install ntp<br>
            <br>
            systemctl status ntpd<br>
            systemctl start ntpd<br>
            systemctl enable ntpd<br>
            systemctl status ntpd<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            2. I use a separate partition within the two VM storage
            nodes. Each of the two nodes see the separate partition as
            /dev/sdb. After creating a thinly provisioned LV on each of
            the two nodes, I create an XFS filesystem on them:<br>
            <br>
            <br>
            pvcreate /dev/sdb1<br>
            <br>
            vgcreate -s 32M vg_glusterfs /dev/sdb1<br>
            <br>
            lvcreate -L 20G --thinpool glusterfs_thin_pool vg_glusterfs<br>
            <br>
            lvcreate -V 15G --thin -n glusterfs_thin_vol1
            vg_glusterfs/glusterfs_thin_po<wbr>ol<br>
            <br>
            mkfs.xfs -i size=512 /dev/vg_glusterfs/glusterfs_th<wbr>in_vol1<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            3. On each node, I configured the brick data partition and
            added the following to /etc/fstab:<br>
            <br>
            mkdir /data<br>
            <br>
            echo &quot;/dev/vg_glusterfs/glusterfs_t<wbr>hin_vol1 /data xfs
            defaults 1 2&quot; &gt;&gt; /etc/fstab<br>
            <br>
            mount -a<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            4. After mounting the volume, I see the following in df -Th,
            which is correct:<br>
            <br>
            [root@gluster1 brick1]# df -Th<br>
            Filesystem                                   Type      Size 
            Used Avail Use% Mounted on<br>
            /dev/sda1                                    ext4       46G 
            1.4G 42G   4% /<br>
            devtmpfs                                     devtmpfs  3.9G 
               0 3.9G   0% /dev<br>
            tmpfs                                        tmpfs     3.9G 
               0 3.9G   0% /dev/shm<br>
            tmpfs                                        tmpfs     3.9G 
            8.6M 3.9G   1% /run<br>
            tmpfs                                        tmpfs     3.9G 
               0 3.9G   0% /sys/fs/cgroup<br>
            tmpfs                                        tmpfs     783M 
               0 783M   0% /run/user/0<br>
            /dev/mapper/vg_glusterfs-glust<wbr>erfs_thin_vol1 xfs       
            15G   34M 15G   1% /data<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            5. Create specific volume dirs on all storage nodes:<br>
            <br>
            <br>
            mkdir -pv /data/glusterfs/${HOSTNAME%%.*<wbr>}/vol01<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            6. Add entries in /etc/hosts:<br>
            <br>
            vim /etc/hosts<br>
            <br>
            192.168.10.233    gluster1<br>
            192.168.10.234    gluster2<br>
            192.168.10.237    gluster-client<br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            7. Install gluster from the CentOS SIG:<br>
            <br>
            <br>
            yum search centos-release-gluster<br>
            yum install centos-release-gluster40<br>
            <br>
            <br>
            yum install glusterfs-server<br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            8. Set up TLS/SSL encryption on all nodes and clients
            (gluster1, gluster2, gluster-client):<br>
            <br>
            openssl genrsa -out /etc/ssl/glusterfs.key 2048<br>
            <br>
            In gluster1 node:<br>
            openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
            &quot;/CN=gluster1&quot; -out /etc/ssl/glusterfs.pem<br>
            In gluster2 node:<br>
            openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
            &quot;/CN=gluster2&quot; -out /etc/ssl/glusterfs.pem<br>
            In gluster-client node:<br>
            openssl req -new -x509 -key /etc/ssl/glusterfs.key -subj
            &quot;/CN=gluster-client&quot; -out /etc/ssl/glusterfs.pem<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            9. On another box, I concatenate all of the .pem
            certificates into a .ca file.<br>
            <br>
            Bring all .pem files locally:<br>
            <br>
            scp gluster1:/etc/ssl/<a href="http://glusterfs.pe" target="_blank">glusterfs.pe</a><wbr>m
            gluster01.pem<br>
            scp gluster2:/etc/ssl/<a href="http://glusterfs.pe" target="_blank">glusterfs.pe</a><wbr>m
            gluster02.pem<br>
            scp gluster-client:/etc/ssl/gluste<wbr>rfs.pem
            gluster-client.pem<br>
            <br>
            For storage nodes, I concatenate all .pem certificates
            (including the client&#39;s .pem):<br>
            <br>
            cat gluster01.pem gluster02.pem gluster-client.pem &gt; <a href="http://glusterfs-nodes.ca" rel="noreferrer" target="_blank">glusterfs-nodes.ca</a><br>
            <br>
            For server clients:<br>
            <br>
            cat gluster01.pem gluster02.pem &gt; <a href="http://glusterfs-client.ca" rel="noreferrer" target="_blank">glusterfs-client.ca</a><br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            10. Put <a href="http://glusterfs-nodes.ca" rel="noreferrer" target="_blank">glusterfs-nodes.ca</a>
            file on all the storage nodes (this includes storage nodes
            .pem + client&#39;s .pem):<br>
            <br>
            scp <a href="http://glusterfs-nodes.ca" rel="noreferrer" target="_blank">glusterfs-nodes.ca</a>
            gluster1:/etc/ssl/<a href="http://glusterfs.ca" rel="noreferrer" target="_blank">glusterfs.ca</a><br>
            scp <a href="http://glusterfs-nodes.ca" rel="noreferrer" target="_blank">glusterfs-nodes.ca</a>
            gluster2:/etc/ssl/<a href="http://glusterfs.ca" rel="noreferrer" target="_blank">glusterfs.ca</a><br>
            <br>
            Put <a href="http://glusterfs-client.ca" rel="noreferrer" target="_blank">glusterfs-client.ca</a>
            file on all the storage nodes (this includes only storage
            nodes .pem):<br>
            <br>
            scp <a href="http://glusterfs-client.ca" rel="noreferrer" target="_blank">glusterfs-client.ca</a>
            gluster-client:/etc/ssl/<a href="http://glusterfs.ca" rel="noreferrer" target="_blank">gluste<wbr>rfs.ca</a><br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            11. Enable management encryption on each node (command ran
            on gluster1, gluster2, gluster-client):<br>
            <br>
            <br>
            touch /var/lib/glusterd/secure-acces<wbr>s<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            12. Start, enable and check status of glusterd on gluster1
            and gluster2:<br>
            <br>
            systemctl start glusterd<br>
            systemctl enable glusterd<br>
            systemctl status glusterd<br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            13. Configure trusted storage pool (TSP):<br>
            <br>
            &gt;From gluster1:<br>
            <br>
            <br>
            gluster peer status<br>
            <br>
            <br>
            gluster peer probe gluster2<br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            14. Create the replicated gluster volume but don&#39;t start it:<br>
            <br>
            <br>
            gluster volume create vol01 replica 2 transport tcp
            gluster1:/data/glusterfs/glust<wbr>er1/vol01/brick1
            gluster2:/data/glusterfs/glust<wbr>er2/vol01/brick1<br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            15. Setup SSL/TLS access to the volume:<br>
            <br>
            gluster volume set vol01 auth.ssl-allow
            &#39;gluster01,gluster02,gluster-c<wbr>lient&#39;<br>
            <br>
            <br>
            gluster volume set vol01 client.ssl on<br>
            gluster volume set vol01 server.ssl on<br>
            <br>
            gluster volume set vol01 network.ping-timeout &quot;5&quot;<br>
            <br>
            gluster volume start vol01<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            16. Mount the volume on gluster-client:<br>
            <br>
            mount -t glusterfs gluster1:/vol01 /mnt<br>
            <br>
            <br>
            After mounting the volume, df -h has the correct output:<br>
            <br>
            [root@gluster-client mnt]# df -h<br>
            Filesystem       Size  Used Avail Use% Mounted on<br>
            /dev/sda1         48G   36G  9.3G  80% /<br>
            devtmpfs         1.9G     0  1.9G   0% /dev<br>
            tmpfs            1.9G     0  1.9G   0% /dev/shm<br>
            tmpfs            1.9G  8.5M  1.9G   1% /run<br>
            tmpfs            1.9G     0  1.9G   0% /sys/fs/cgroup<br>
            tmpfs            379M     0  379M   0% /run/user/0<br>
            gluster1:/vol01   15G  187M   15G   2% /mnt<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            17. If I try to copy an archive, it throws an error after
            transferring part of the file and the /mnt folder becomes
            inaccessible for a while:<br>
            <br>
            [root@gluster-client mnt]# cp /root/mybee.tar.gz /mnt/<br>
            cp: error writing ‘/mnt/mybee.tar.gz’: Transport endpoint is
            not connected<br>
            cp: failed to extend ‘/mnt/mybee.tar.gz’: Transport endpoint
            is not connected<br>
            cp: failed to close ‘/mnt/mybee.tar.gz’: Transport endpoint
            is not connected<br>
            [root@gluster-client mnt]# ll /mnt<br>
            ls: cannot open directory .: Transport endpoint is not
            connected<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            This is the log from gluster-client for the mnt mountpoint
            (mnt.log):<br>
            <br>
            <br>
            [2018-06-29 07:52:03.869382] I [MSGID: 100030]
            [glusterfsd.c:2625:main] 0-/usr/sbin/glusterfs: Started
            running /usr/sbin/glusterfs version 4.0.2 (args:
            /usr/sbin/glusterfs --process-name fuse
            --volfile-server=gluster1 --volfile-id=/vol01 /mnt)<br>
            [2018-06-29 07:52:03.878709] I [socket.c:4470:socket_init]
            0-glusterfs: SSL support on the I/O path is ENABLED<br>
            [2018-06-29 07:52:03.878782] I [socket.c:4473:socket_init]
            0-glusterfs: SSL support for glusterd is ENABLED<br>
            [2018-06-29 07:52:03.878795] I [socket.c:4490:socket_init]
            0-glusterfs: using private polling thread<br>
            [2018-06-29 07:52:03.879108] E [socket.c:4541:socket_init]
            0-glusterfs: failed to open /etc/ssl/dhparam.pem, DH ciphers
            are disabled<br>
            [2018-06-29 07:52:03.885718] I [MSGID: 101190]
            [event-epoll.c:609:event_dispa<wbr>tch_epoll_worker]
            0-epoll: Started thread with index 1<br>
            [2018-06-29 07:52:03.901983] I [MSGID: 101190]
            [event-epoll.c:609:event_dispa<wbr>tch_epoll_worker]
            0-epoll: Started thread with index 2<br>
            [2018-06-29 07:52:03.902092] I [socket.c:4470:socket_init]
            0-vol01-client-1: SSL support on the I/O path is ENABLED<br>
            [2018-06-29 07:52:03.902117] I [socket.c:4473:socket_init]
            0-vol01-client-1: SSL support for glusterd is ENABLED<br>
            [2018-06-29 07:52:03.902142] I [socket.c:4490:socket_init]
            0-vol01-client-1: using private polling thread<br>
            [2018-06-29 07:52:03.902266] E [socket.c:4541:socket_init]
            0-vol01-client-1: failed to open /etc/ssl/dhparam.pem, DH
            ciphers are disabled<br>
            [2018-06-29 07:52:03.902977] I [socket.c:4470:socket_init]
            0-vol01-client-0: SSL support on the I/O path is ENABLED<br>
            [2018-06-29 07:52:03.903018] I [socket.c:4473:socket_init]
            0-vol01-client-0: SSL support for glusterd is ENABLED<br>
            [2018-06-29 07:52:03.903028] I [socket.c:4490:socket_init]
            0-vol01-client-0: using private polling thread<br>
            [2018-06-29 07:52:03.903161] E [socket.c:4541:socket_init]
            0-vol01-client-0: failed to open /etc/ssl/dhparam.pem, DH
            ciphers are disabled<br>
            [2018-06-29 07:52:03.903756] I [MSGID: 114020]
            [client.c:2300:notify] 0-vol01-client-0: parent translators
            are ready, attempting connect on transport<br>
            [2018-06-29 07:52:03.908432] I [MSGID: 114020]
            [client.c:2300:notify] 0-vol01-client-1: parent translators
            are ready, attempting connect on transport<br>
            Final graph:<br>
            +-----------------------------<wbr>------------------------------<wbr>-------------------+<br>
              1: volume vol01-client-0<br>
              2:     type protocol/client<br>
              3:     option ping-timeout 5<br>
              4:     option remote-host gluster1<br>
              5:     option remote-subvolume
            /data/glusterfs/gluster1/vol01<wbr>/brick1<br>
              6:     option transport-type socket<br>
              7:     option transport.address-family inet<br>
              8:     option transport.socket.ssl-enabled on<br>
              9:     option transport.tcp-user-timeout 0<br>
             10:     option transport.socket.keepalive-tim<wbr>e 20<br>
             11:     option transport.socket.keepalive-int<wbr>erval 2<br>
             12:     option transport.socket.keepalive-cou<wbr>nt 9<br>
             13:     option send-gids true<br>
             14: end-volume<br>
             15:<br>
             16: volume vol01-client-1<br>
             17:     type protocol/client<br>
             18:     option ping-timeout 5<br>
             19:     option remote-host gluster2<br>
             20:     option remote-subvolume
            /data/glusterfs/gluster2/vol01<wbr>/brick1<br>
             21:     option transport-type socket<br>
             22:     option transport.address-family inet<br>
             23:     option transport.socket.ssl-enabled on<br>
             24:     option transport.tcp-user-timeout 0<br>
             25:     option transport.socket.keepalive-tim<wbr>e 20<br>
             26:     option transport.socket.keepalive-int<wbr>erval 2<br>
             27:     option transport.socket.keepalive-cou<wbr>nt 9<br>
             28:     option send-gids true<br>
             29: end-volume<br>
             30:<br>
             31: volume vol01-replicate-0<br>
             32:     type cluster/replicate<br>
             33:     option afr-pending-xattr
            vol01-client-0,vol01-client-1<br>
             34:     option use-compound-fops off<br>
             35:     subvolumes vol01-client-0 vol01-client-1<br>
             36: end-volume<br>
             37:<br>
             38: volume vol01-dht<br>
             39:     type cluster/distribute<br>
             40:     option lock-migration off<br>
             41:     option force-migration off<br>
             42:     subvolumes vol01-replicate-0<br>
             43: end-volume<br>
             44:<br>
             45: volume vol01-write-behind<br>
             46:     type performance/write-behind<br>
             47:     subvolumes vol01-dht<br>
             48: end-volume<br>
             49:<br>
             50: volume vol01-read-ahead<br>
             51:     type performance/read-ahead<br>
             52:     subvolumes vol01-write-behind<br>
             53: end-volume<br>
             54:<br>
             55: volume vol01-readdir-ahead<br>
             56:     type performance/readdir-ahead<br>
             57:     option parallel-readdir off<br>
             58:     option rda-request-size 131072<br>
             59:     option rda-cache-limit 10MB<br>
             60:     subvolumes vol01-read-ahead<br>
             61: end-volume<br>
             62:<br>
             63: volume vol01-io-cache<br>
             64:     type performance/io-cache<br>
             65:     subvolumes vol01-readdir-ahead<br>
             66: end-volume<br>
             67:<br>
             68: volume vol01-quick-read<br>
             69:     type performance/quick-read<br>
             70:     subvolumes vol01-io-cache<br>
             71: end-volume<br>
             72:<br>
             73: volume vol01-open-behind<br>
             74:     type performance/open-behind<br>
             75:     subvolumes vol01-quick-read<br>
             76: end-volume<br>
             77:<br>
             78: volume vol01-md-cache<br>
             79:     type performance/md-cache<br>
             80:     subvolumes vol01-open-behind<br>
             81: end-volume<br>
             82:<br>
             83: volume vol01<br>
             84:     type debug/io-stats<br>
             85:     option log-level INFO<br>
             86:     option latency-measurement off<br>
             87:     option count-fop-hits off<br>
             88:     subvolumes vol01-md-cache<br>
             89: end-volume<br>
             90:<br>
             91: volume meta-autoload<br>
             92:     type meta<br>
             93:     subvolumes vol01<br>
             94: end-volume<br>
             95:<br>
            +-----------------------------<wbr>------------------------------<wbr>-------------------+<br>
            [2018-06-29 07:52:03.917518] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.918459] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.919289] I
            [rpc-clnt.c:2071:rpc_clnt_reco<wbr>nfig] 0-vol01-client-0:
            changing port to 49152 (from 0)<br>
            [2018-06-29 07:52:03.922903] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.923922] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.925115] I
            [rpc-clnt.c:2071:rpc_clnt_reco<wbr>nfig] 0-vol01-client-1:
            changing port to 49152 (from 0)<br>
            [2018-06-29 07:52:03.930023] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.930756] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.931859] I [MSGID: 114046]
            [client-handshake.c:1176:clien<wbr>t_setvolume_cbk]
            0-vol01-client-0: Connected to vol01-client-0, attached to
            remote volume &#39;/data/glusterfs/gluster1/vol0<wbr>1/brick1&#39;.<br>
            [2018-06-29 07:52:03.931894] I [MSGID: 108005]
            [afr-common.c:5081:__afr_handl<wbr>e_child_up_event]
            0-vol01-replicate-0: Subvolume &#39;vol01-client-0&#39; came back
            up; going online.<br>
            [2018-06-29 07:52:03.938903] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.939803] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:03.941300] I [MSGID: 114046]
            [client-handshake.c:1176:clien<wbr>t_setvolume_cbk]
            0-vol01-client-1: Connected to vol01-client-1, attached to
            remote volume &#39;/data/glusterfs/gluster2/vol0<wbr>1/brick1&#39;.<br>
            [2018-06-29 07:52:03.942884] I
            [fuse-bridge.c:4234:fuse_init] 0-glusterfs-fuse: FUSE inited
            with protocol versions: glusterfs 7.24 kernel 7.22<br>
            [2018-06-29 07:52:03.942907] I
            [fuse-bridge.c:4864:fuse_graph<wbr>_sync] 0-fuse: switched
            to graph 0<br>
            [2018-06-29 07:52:03.946767] W [socket.c:592:__socket_rwv]
            0-vol01-client-0: readv on <a href="http://192.168.10.233:49152" rel="noreferrer" target="_blank">192.168.10.233:49152</a>
            failed (Input/output error)<br>
            [2018-06-29 07:52:03.946791] E [socket.c:2785:socket_poller]
            0-vol01-client-0: socket_poller <a href="http://192.168.10.233:49152" rel="noreferrer" target="_blank">192.168.10.233:49152</a>
            failed (Input/output error)<br>
            [2018-06-29 07:52:03.946837] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:03.947264] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(LOOKUP(27)) called at 2018-06-29
            07:52:03.945958 (xid=0xb)<br>
            [2018-06-29 07:52:03.947297] W [MSGID: 114031]
            [client-rpc-fops_v2.c:2540:cli<wbr>ent4_0_lookup_cbk]
            0-vol01-client-0: remote operation failed. Path: /
            (00000000-0000-0000-0000-00000<wbr>0000001) [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:03.947455] I [MSGID: 109005]
            [dht-selfheal.c:2328:dht_selfh<wbr>eal_directory]
            0-vol01-dht: Directory selfheal failed: Unable to form
            layout for directory /<br>
            [2018-06-29 07:52:07.885389] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:07.886457] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:07.887452] I
            [rpc-clnt.c:2071:rpc_clnt_reco<wbr>nfig] 0-vol01-client-0:
            changing port to 49152 (from 0)<br>
            [2018-06-29 07:52:07.900627] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:07.901594] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:07.903220] I [MSGID: 114046]
            [client-handshake.c:1176:clien<wbr>t_setvolume_cbk]
            0-vol01-client-0: Connected to vol01-client-0, attached to
            remote volume &#39;/data/glusterfs/gluster1/vol0<wbr>1/brick1&#39;.<br>
            [2018-06-29 07:52:13.062404] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:13.062780] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.048761 (xid=0x22)<br>
            [2018-06-29 07:52:13.062826] W [MSGID: 114031]
            [client-rpc-fops_v2.c:658:clie<wbr>nt4_0_writev_cbk]
            0-vol01-client-0: remote operation failed [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:13.063017] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.049162 (xid=0x23)<br>
            [2018-06-29 07:52:13.063175] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.049865 (xid=0x24)<br>
            [2018-06-29 07:52:13.063319] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.050199 (xid=0x25)<br>
            [2018-06-29 07:52:13.063473] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.050489 (xid=0x26)<br>
            [2018-06-29 07:52:13.063600] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.050757 (xid=0x27)<br>
            [2018-06-29 07:52:13.063745] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.051031 (xid=0x28)<br>
            [2018-06-29 07:52:13.063883] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.053035 (xid=0x29)<br>
            The message &quot;W [MSGID: 114031]
            [client-rpc-fops_v2.c:658:clie<wbr>nt4_0_writev_cbk]
            0-vol01-client-0: remote operation failed [Transport
            endpoint is not connected]&quot; repeated 7 times between
            [2018-06-29 07:52:13.062826] and [2018-06-29
            07:52:13.063903]<br>
            [2018-06-29 07:52:13.104580] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
            0-vol01-client-0: remote operation failed<br>
            [2018-06-29 07:52:13.109230] W [socket.c:592:__socket_rwv]
            0-vol01-client-1: writev on <a href="http://192.168.10.234:49152" rel="noreferrer" target="_blank">192.168.10.234:49152</a>
            failed (No data available)<br>
            [2018-06-29 07:52:13.109282] E [socket.c:2777:socket_poller]
            0-vol01-client-1: poll error on socket<br>
            [2018-06-29 07:52:13.109340] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:13.109357] E [MSGID: 108006]
            [afr-common.c:5158:__afr_handl<wbr>e_child_down_event]
            0-vol01-replicate-0: All subvolumes are down. Going offline
            until atleast one of them comes back up.<br>
            [2018-06-29 07:52:13.109461] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:13.109679] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.104513 (xid=0x42)<br>
            [2018-06-29 07:52:13.109700] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1260:cli<wbr>ent4_0_fstat_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:13.109854] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FXATTROP(34)) called at 2018-06-29
            07:52:13.104729 (xid=0x43)<br>
            [2018-06-29 07:52:13.109877] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
            0-vol01-client-1: remote operation failed<br>
            [2018-06-29 07:52:13.109897] E [MSGID: 114031]
            [client-rpc-fops_v2.c:1352:cli<wbr>ent4_0_finodelk_cbk]
            0-vol01-client-0: remote operation failed [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:13.110036] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(WRITE(13)) called at 2018-06-29
            07:52:13.105623 (xid=0x44)<br>
            [2018-06-29 07:52:13.110053] W [MSGID: 114031]
            [client-rpc-fops_v2.c:658:clie<wbr>nt4_0_writev_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:13.110232] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.105658 (xid=0x45)<br>
            [2018-06-29 07:52:13.110250] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1260:cli<wbr>ent4_0_fstat_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:13.110386] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.106862 (xid=0x46)<br>
            [2018-06-29 07:52:13.110530] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.108103 (xid=0x47)<br>
            [2018-06-29 07:52:13.110660] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FSTAT(25)) called at 2018-06-29
            07:52:13.108969 (xid=0x48)<br>
            [2018-06-29 07:52:13.110793] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FXATTROP(34)) called at 2018-06-29
            07:52:13.109256 (xid=0x49)<br>
            The message &quot;W [MSGID: 114031]
            [client-rpc-fops_v2.c:1260:cli<wbr>ent4_0_fstat_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]&quot; repeated 3 times between
            [2018-06-29 07:52:13.110250] and [2018-06-29
            07:52:13.110675]<br>
            [2018-06-29 07:52:13.110808] W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
            0-vol01-client-1: remote operation failed<br>
            [2018-06-29 07:52:13.110933] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame
            type(GlusterFS 4.x v1) op(FXATTROP(34)) called at 2018-06-29
            07:52:13.109284 (xid=0x4a)<br>
            The message &quot;W [MSGID: 114031]
            [client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
            0-vol01-client-1: remote operation failed&quot; repeated 3 times
            between [2018-06-29 07:52:13.110808] and [2018-06-29
            07:52:14.878874]<br>
            [2018-06-29 07:52:14.878895] E [MSGID: 114031]
            [client-rpc-fops_v2.c:1352:cli<wbr>ent4_0_finodelk_cbk]
            0-vol01-client-1: remote operation failed [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:14.879095] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:14.879217] W
            [fuse-bridge.c:2427:fuse_write<wbr>v_cbk] 0-glusterfs-fuse:
            76: WRITE =&gt; -1 gfid=a8848be6-1c05-4a72-88a7-5<wbr>209eeef7c2c
            fd=0x7f7c50023a68 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:14.880509] W
            [fuse-bridge.c:1406:fuse_err_c<wbr>bk] 0-glusterfs-fuse: 77:
            FLUSH() ERR =&gt; -1 (Transport endpoint is not connected)<br>
            The message &quot;I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up&quot; repeated 11 times between [2018-06-29
            07:52:14.879095] and [2018-06-29 07:52:17.908056]<br>
            [2018-06-29 07:52:17.908066] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_<wbr>dir_cbk] 0-vol01-dht:
            dict is null<br>
            [2018-06-29 07:52:17.908175] W
            [fuse-bridge.c:896:fuse_attr_c<wbr>bk] 0-glusterfs-fuse: 79:
            LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:18.758975] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:18.759155] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:18.759165] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_<wbr>dir_cbk] 0-vol01-dht:
            dict is null<br>
            [2018-06-29 07:52:18.759214] W
            [fuse-bridge.c:896:fuse_attr_c<wbr>bk] 0-glusterfs-fuse: 80:
            LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:21.662626] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:21.662756] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:21.662765] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_<wbr>dir_cbk] 0-vol01-dht:
            dict is null<br>
            [2018-06-29 07:52:21.662811] W
            [fuse-bridge.c:896:fuse_attr_c<wbr>bk] 0-glusterfs-fuse: 81:
            LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:21.663567] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:21.663630] W
            [fuse-bridge.c:3205:fuse_statf<wbr>s_cbk] 0-glusterfs-fuse:
            82: ERR =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:23.896362] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:23.896687] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:23.897321] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:23.896427 (xid=0x2a)<br>
            [2018-06-29 07:52:23.897353] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:23.903241] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:23.903566] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:23.904166] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:23.903281 (xid=0x4b)<br>
            [2018-06-29 07:52:23.904199] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:26.909819] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:26.910175] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:26.910858] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:26.909871 (xid=0x2b)<br>
            [2018-06-29 07:52:26.910888] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:26.913921] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:26.914163] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:26.914417] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:26.913961 (xid=0x4c)<br>
            [2018-06-29 07:52:26.914443] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:29.921628] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:29.921867] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:29.922215] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:29.921674 (xid=0x2c)<br>
            [2018-06-29 07:52:29.922257] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:29.926462] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:29.926774] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:29.927415] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:29.926499 (xid=0x4d)<br>
            [2018-06-29 07:52:29.927447] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:32.935218] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:32.935553] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:32.936196] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:32.935271 (xid=0x2d)<br>
            [2018-06-29 07:52:32.936230] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:32.938940] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:32.939232] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:32.939497] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:32.938982 (xid=0x4e)<br>
            [2018-06-29 07:52:32.939526] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:35.948789] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:35.949172] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:35.949860] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:35.948839 (xid=0x2e)<br>
            [2018-06-29 07:52:35.949901] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:35.952154] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:35.952408] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:35.952670] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:35.952199 (xid=0x4f)<br>
            [2018-06-29 07:52:35.952699] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:38.959369] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:38.959733] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:38.960536] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:38.959426 (xid=0x2f)<br>
            [2018-06-29 07:52:38.960573] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:38.963381] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:38.963618] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:38.963877] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:38.963428 (xid=0x50)<br>
            [2018-06-29 07:52:38.963906] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:41.969152] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:41.969397] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:41.969731] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:41.969204 (xid=0x30)<br>
            [2018-06-29 07:52:41.969769] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:41.974130] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:41.974381] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:41.974669] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:41.974198 (xid=0x51)<br>
            [2018-06-29 07:52:41.974698] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:44.984465] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:44.984781] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:44.985444] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:44.984514 (xid=0x31)<br>
            [2018-06-29 07:52:44.985477] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:44.990039] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:44.990352] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:44.990729] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:44.990094 (xid=0x52)<br>
            [2018-06-29 07:52:44.990771] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:47.994448] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:47.994684] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:47.994947] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:47.994499 (xid=0x32)<br>
            [2018-06-29 07:52:47.994984] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:48.002544] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:48.002809] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:48.003388] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:48.002577 (xid=0x53)<br>
            [2018-06-29 07:52:48.003416] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:49.440101] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:49.440257] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:49.440266] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_<wbr>dir_cbk] 0-vol01-dht:
            dict is null<br>
            [2018-06-29 07:52:49.440310] W
            [fuse-bridge.c:896:fuse_attr_c<wbr>bk] 0-glusterfs-fuse: 83:
            LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:50.182948] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:50.183108] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:50.183116] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_<wbr>dir_cbk] 0-vol01-dht:
            dict is null<br>
            [2018-06-29 07:52:50.197572] W
            [fuse-bridge.c:896:fuse_attr_c<wbr>bk] 0-glusterfs-fuse: 84:
            LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:50.575887] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:50.576030] I [MSGID: 108006]
            [afr-common.c:5444:afr_local_i<wbr>nit] 0-vol01-replicate-0:
            no subvolumes up<br>
            [2018-06-29 07:52:50.576039] E [MSGID: 101046]
            [dht-common.c:1501:dht_lookup_<wbr>dir_cbk] 0-vol01-dht:
            dict is null<br>
            [2018-06-29 07:52:50.576087] W
            [fuse-bridge.c:896:fuse_attr_c<wbr>bk] 0-glusterfs-fuse: 85:
            LOOKUP() / =&gt; -1 (Transport endpoint is not connected)<br>
            [2018-06-29 07:52:51.012626] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-0:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:51.012924] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-0:
            disconnected from vol01-client-0. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:51.013615] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-0: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:51.012667 (xid=0x33)<br>
            [2018-06-29 07:52:51.013647] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-0: received RPC status error [Transport
            endpoint is not connected]<br>
            [2018-06-29 07:52:51.018855] W
            [rpc-clnt.c:1739:rpc_clnt_subm<wbr>it] 0-vol01-client-1:
            error returned while attempting to connect to host:(null),
            port:0<br>
            [2018-06-29 07:52:51.019093] I [MSGID: 114018]
            [client.c:2227:client_rpc_noti<wbr>fy] 0-vol01-client-1:
            disconnected from vol01-client-1. Client process will keep
            trying to connect to glusterd until brick&#39;s port is
            available<br>
            [2018-06-29 07:52:51.019378] E
            [rpc-clnt.c:350:saved_frames_u<wbr>nwind] (--&gt;
            /lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
            (--&gt; /lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
            (--&gt; /lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
            (--&gt; /lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
            (--&gt; /lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
            ))))) 0-vol01-client-1: forced unwinding frame type(GF-DUMP)
            op(DUMP(1)) called at 2018-06-29 07:52:51.018893 (xid=0x54)<br>
            [2018-06-29 07:52:51.019408] W [MSGID: 114032]
            [client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
            0-vol01-client-1: received RPC status error [Transport
            endpoint is not connected]<br>
            <br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            After trying to copy the archive, I end up with different
            sized files on the two bricks:<br>
            <br>
            [root@gluster1 brick1]# pwd<br>
            /data/glusterfs/gluster1/vol01<wbr>/brick1<br>
            [root@gluster1 brick1]# ll<br>
            total 384<br>
            -rw------- 2 root root 393216 Jun 29 10:52 mybee.tar.gz<br>
            <br>
            <br>
            <br>
            [root@gluster2 brick1]# pwd<br>
            /data/glusterfs/gluster2/vol01<wbr>/brick1<br>
            [root@gluster2 brick1]# ll<br>
            total 1280<br>
            -rw------- 2 root root 1310720 Jun 29 10:52 mybee.tar.gz<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            These are the informations regarding the gluster volume:<br>
            <br>
            [root@gluster1 glusterfs]# gluster volume info vol01<br>
            <br>
            Volume Name: vol01<br>
            Type: Replicate<br>
            Volume ID: d3b39520-3fbb-4cfc-b736-338d77<wbr>26b5a1<br>
            Status: Started<br>
            Snapshot Count: 0<br>
            Number of Bricks: 1 x 2 = 2<br>
            Transport-type: tcp<br>
            Bricks:<br>
            Brick1: gluster1:/data/glusterfs/glust<wbr>er1/vol01/brick1<br>
            Brick2: gluster2:/data/glusterfs/glust<wbr>er2/vol01/brick1<br>
            Options Reconfigured:<br>
            network.ping-timeout: 5<br>
            server.ssl: on<br>
            client.ssl: on<br>
            auth.ssl-allow: gluster01,gluster02,gluster-cl<wbr>ient<br>
            transport.address-family: inet<br>
            nfs.disable: on<br>
            performance.client-io-threads: off<br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            Here is the peers information:<br>
            <br>
            [root@gluster1 brick1]# gluster peer status<br>
            Number of Peers: 1<br>
            <br>
            Hostname: gluster2<br>
            Uuid: 9ccac8e3-71c1-43e4-b723-fb7374<wbr>be8a1d<br>
            State: Peer in Cluster (Connected)<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            Here is the volume status:<br>
            <br>
            [root@gluster1 brick1]# gluster volume status vol01<br>
            Status of volume: vol01<br>
            Gluster process                             TCP Port  RDMA
            Port Online  Pid<br>
            ------------------------------<wbr>------------------------------<wbr>------------------<br>
            Brick gluster1:/data/glusterfs/glust<wbr>er1/vol<br>
            01/brick1                                   49152     0 Y   
               11108<br>
            Brick gluster2:/data/glusterfs/glust<wbr>er2/vol<br>
            01/brick1                                   49152     0 Y   
               10942<br>
            Self-heal Daemon on localhost               N/A       N/A Y 
                 11133<br>
            Self-heal Daemon on gluster2                N/A       N/A Y 
                 10968<br>
            <br>
            Task Status of Volume vol01<br>
            ------------------------------<wbr>------------------------------<wbr>------------------<br>
            There are no active volume tasks<br>
            <br>
            <br>
            ==============================<wbr>=======================<br>
            <br>
            <br>
            These are the installed packages on gluster-client:<br>
            <br>
            [root@gluster-client glusterfs]# rpm -qa | grep
            &quot;gluster\|fuse&quot;<br>
            centos-release-gluster40-1.0-2<wbr>.el7.centos.x86_64<br>
            glusterfs-client-xlators-4.0.2<wbr>-1.el7.x86_64<br>
            fuse-2.9.2-10.el7.x86_64<br>
            glusterfs-4.0.2-1.el7.x86_64<br>
            glusterfs-fuse-4.0.2-1.el7.x86<wbr>_64<br>
            fuse-libs-2.9.2-10.el7.x86_64<br>
            glusterfs-libs-4.0.2-1.el7.x86<wbr>_64<br>
            <br>
            <br>
            <br>
            And these are the installed packages on gluster1 and
            gluster2 storage nodes:<br>
            <br>
            <br>
            [root@gluster1 glusterfs]# rpm -qa | grep &quot;gluster\|fuse&quot;<br>
            glusterfs-api-4.0.2-1.el7.x86_<wbr>64<br>
            centos-release-gluster40-1.0-2<wbr>.el7.centos.x86_64<br>
            glusterfs-libs-4.0.2-1.el7.x86<wbr>_64<br>
            glusterfs-client-xlators-4.0.2<wbr>-1.el7.x86_64<br>
            glusterfs-cli-4.0.2-1.el7.x86_<wbr>64<br>
            glusterfs-fuse-4.0.2-1.el7.x86<wbr>_64<br>
            glusterfs-server-4.0.2-1.el7.x<wbr>86_64<br>
            glusterfs-4.0.2-1.el7.x86_64<br>
            <br>
            <br>
            <br>
            <br>
            Can anyone point me in the right direction? What am I doing
            wrong? The mount works fine until I try to copy an archive,
            multiple smaller files or a bigger file on it (meaning it
            shows correctly in df -Th and I can create several files
            with &quot;touch file1 file2...&quot;). Basically, after any data
            transfer, I get these errors.<br>
            <br>
            <br>
            I followed the indications from the redhat page:<br>
            <br>
            <a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption" rel="noreferrer" target="_blank">https://access.redhat.com/docu<wbr>mentation/en-us/red_hat_gluste<wbr>r_storage/3.1/html/administrat<wbr>ion_guide/chap-network_encrypt<wbr>ion</a><br>
            <br>
            UPDATE 1:<br>
            I tried doing the exact same steps in Gluster 3.12 and had
            no problem. The steps worked and SSL/TLS was enabled. There
            was no transport error or anything and I also checked if
            SSL/TLS was enabled. Afterwards, I also tried with the new
            release 4.1 and the problem persists (same error with
            &quot;Transport endpoint is not connected&quot;).<br>
            Let me know if you need any other info. Any help is much
            appreciated.<br>
            <br>
            Thank you.<br>
            ______________________________<wbr>_________________<br>
            Gluster-users mailing list<br>
            <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
            <a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a></blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </div></div></div>

</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</div>