<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello,</p>
<p>I just filed the bug. The thread can be seen here:</p>
<p><br>
</p>
<p><a class="moz-txt-link-freetext" href="https://bugzilla.redhat.com/show_bug.cgi?id=1601356">https://bugzilla.redhat.com/show_bug.cgi?id=1601356</a></p>
<p><br>
</p>
<p>The information is mostly the same. Let me know if you need any
other info.</p>
<p><br>
</p>
<p>Regards.</p>
<p>Andrei H.<br>
</p>
<br>
<div class="moz-cite-prefix">On 7/13/18 1:27 PM, Amar Tumballi
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAHxyDdPb-yUCz2eo=3dJREoY6eh4+i43og4EM6mH1KDZKHcn=A@mail.gmail.com">
<div dir="ltr">Thanks David Spisla for pitching in.
<div><br>
</div>
<div>Havriliuc, if you can file the bug as requested, we would
like to check this out.</div>
<div><br>
</div>
<div>-Amar</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Jul 13, 2018 at 3:44 PM, David
Spisla <span dir="ltr"><<a
href="mailto:spisla80@gmail.com" target="_blank"
moz-do-not-send="true">spisla80@gmail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Hello Andrei,</div>
<div><br>
</div>
<div>I did some research. But I didn't find a bug report
for this issue. If you want you can create such a report
here:</div>
<div><a
href="https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS"
target="_blank" moz-do-not-send="true">https://bugzilla.redhat.com/<wbr>enter_bug.cgi?product=<wbr>GlusterFS</a></div>
<div><br>
</div>
<div>It is the recommended way in GlusterFS to report a
bug, github issues are for new features<br>
</div>
<div><br>
</div>
<div>Regards</div>
<span class="HOEnZb"><font color="#888888">
<div>David<br>
</div>
</font></span></div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-07-13 11:44 GMT+02:00
Havriliuc Andrei <span dir="ltr"><<a
href="mailto:andrei@hostvision.ro"
target="_blank" moz-do-not-send="true">andrei@hostvision.ro</a>></span>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<p>Hello David,</p>
<p><br>
</p>
<p>Yeah. I also disabled SSL/TLS for management
and communication for Gluster. The problem is
that I will have traffic over public WAN for a
geo replication instance and I cannot go
forward until this is solved.</p>
<p><br>
</p>
<p>Was there a bug filed about this? I've looked
at the issues on the Glusterfs github page but
I didn't find anything related to this:</p>
<p><br>
</p>
<p><a
href="https://github.com/gluster/glusterfs/issues"
target="_blank" moz-do-not-send="true">https://github.com/gluster/glu<wbr>sterfs/issues</a><br>
</p>
<p><br>
</p>
<p>Regards,</p>
<p>Andrei<br>
</p>
<div>
<div> <br>
<div>On 7/12/18 4:03 PM, David Spisla wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Hello Andrei,</div>
<div><br>
</div>
<div>I am also using Gluster 4.1 on
CentOS and I have the same problem. I
tested it with a volume which had no
network encryption and one with
network encryption. You are not the
only one.</div>
<div>It seems to be a bug. At the moment
there is no other choice to disable
client.ssl and sever.ssl on a volume
to have stable I/O.</div>
<div><br>
</div>
<div>Regards</div>
<div>David</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">2018-07-12
10:45 GMT+02:00 Havriliuc Andrei <span
dir="ltr"><<a
href="mailto:andrei@hostvision.ro"
target="_blank"
moz-do-not-send="true">andrei@hostvision.ro</a>></span>:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">Hello,<br>
<br>
I am doing some tests with GlusterFS
4.0 and I can't seem to solve some
SSL/TLS issues. I am trying to set
up a 2 node replicated gluster
volume with SSL/TLS. For this setup,
I use 3 KVM VMs (2 storage nodes + 1
client node). For the networking
part, I use a dedicated private LAN
for the KVM VMs. Each VM is able to
ping the other, so there's no
problem with the connectivity.<br>
<br>
To try to make the procedure I used
as clear as possible, I will put all
commands in chronological order:<br>
<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
1. First, I update the systems,
install ntp and then reboot:<br>
<br>
yum update<br>
<br>
yum install ntp<br>
<br>
systemctl status ntpd<br>
systemctl start ntpd<br>
systemctl enable ntpd<br>
systemctl status ntpd<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
2. I use a separate partition within
the two VM storage nodes. Each of
the two nodes see the separate
partition as /dev/sdb. After
creating a thinly provisioned LV on
each of the two nodes, I create an
XFS filesystem on them:<br>
<br>
<br>
pvcreate /dev/sdb1<br>
<br>
vgcreate -s 32M vg_glusterfs
/dev/sdb1<br>
<br>
lvcreate -L 20G --thinpool
glusterfs_thin_pool vg_glusterfs<br>
<br>
lvcreate -V 15G --thin -n
glusterfs_thin_vol1
vg_glusterfs/glusterfs_thin_po<wbr>ol<br>
<br>
mkfs.xfs -i size=512
/dev/vg_glusterfs/glusterfs_th<wbr>in_vol1<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
3. On each node, I configured the
brick data partition and added the
following to /etc/fstab:<br>
<br>
mkdir /data<br>
<br>
echo "/dev/vg_glusterfs/glusterfs_t<wbr>hin_vol1
/data xfs defaults 1 2" >>
/etc/fstab<br>
<br>
mount -a<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
4. After mounting the volume, I see
the following in df -Th, which is
correct:<br>
<br>
[root@gluster1 brick1]# df -Th<br>
Filesystem
Type Size Used Avail
Use% Mounted on<br>
/dev/sda1
ext4 46G 1.4G 42G
4% /<br>
devtmpfs
devtmpfs 3.9G 0 3.9G
0% /dev<br>
tmpfs
tmpfs 3.9G 0 3.9G
0% /dev/shm<br>
tmpfs
tmpfs 3.9G 8.6M 3.9G
1% /run<br>
tmpfs
tmpfs 3.9G 0 3.9G
0% /sys/fs/cgroup<br>
tmpfs
tmpfs 783M 0 783M
0% /run/user/0<br>
/dev/mapper/vg_glusterfs-glust<wbr>erfs_thin_vol1
xfs 15G 34M 15G 1% /data<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
5. Create specific volume dirs on
all storage nodes:<br>
<br>
<br>
mkdir -pv
/data/glusterfs/${HOSTNAME%%.*<wbr>}/vol01<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
6. Add entries in /etc/hosts:<br>
<br>
vim /etc/hosts<br>
<br>
192.168.10.233 gluster1<br>
192.168.10.234 gluster2<br>
192.168.10.237 gluster-client<br>
<br>
==============================<wbr>=======================<br>
<br>
7. Install gluster from the CentOS
SIG:<br>
<br>
<br>
yum search centos-release-gluster<br>
yum install centos-release-gluster40<br>
<br>
<br>
yum install glusterfs-server<br>
<br>
==============================<wbr>=======================<br>
<br>
8. Set up TLS/SSL encryption on all
nodes and clients (gluster1,
gluster2, gluster-client):<br>
<br>
openssl genrsa -out
/etc/ssl/glusterfs.key 2048<br>
<br>
In gluster1 node:<br>
openssl req -new -x509 -key
/etc/ssl/glusterfs.key -subj
"/CN=gluster1" -out
/etc/ssl/glusterfs.pem<br>
In gluster2 node:<br>
openssl req -new -x509 -key
/etc/ssl/glusterfs.key -subj
"/CN=gluster2" -out
/etc/ssl/glusterfs.pem<br>
In gluster-client node:<br>
openssl req -new -x509 -key
/etc/ssl/glusterfs.key -subj
"/CN=gluster-client" -out
/etc/ssl/glusterfs.pem<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
9. On another box, I concatenate all
of the .pem certificates into a .ca
file.<br>
<br>
Bring all .pem files locally:<br>
<br>
scp gluster1:/etc/ssl/<a
href="http://glusterfs.pe"
target="_blank"
moz-do-not-send="true">glusterfs.pe</a><wbr>m
gluster01.pem<br>
scp gluster2:/etc/ssl/<a
href="http://glusterfs.pe"
target="_blank"
moz-do-not-send="true">glusterfs.pe</a><wbr>m
gluster02.pem<br>
scp gluster-client:/etc/ssl/gluste<wbr>rfs.pem
gluster-client.pem<br>
<br>
For storage nodes, I concatenate all
.pem certificates (including the
client's .pem):<br>
<br>
cat gluster01.pem gluster02.pem
gluster-client.pem > <a
href="http://glusterfs-nodes.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-nodes.ca</a><br>
<br>
For server clients:<br>
<br>
cat gluster01.pem gluster02.pem >
<a href="http://glusterfs-client.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-client.ca</a><br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
10. Put <a
href="http://glusterfs-nodes.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-nodes.ca</a>
file on all the storage nodes (this
includes storage nodes .pem +
client's .pem):<br>
<br>
scp <a
href="http://glusterfs-nodes.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-nodes.ca</a>
gluster1:/etc/ssl/<a
href="http://glusterfs.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs.ca</a><br>
scp <a
href="http://glusterfs-nodes.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-nodes.ca</a>
gluster2:/etc/ssl/<a
href="http://glusterfs.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs.ca</a><br>
<br>
Put <a
href="http://glusterfs-client.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-client.ca</a>
file on all the storage nodes (this
includes only storage nodes .pem):<br>
<br>
scp <a
href="http://glusterfs-client.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">glusterfs-client.ca</a>
gluster-client:/etc/ssl/<a
href="http://glusterfs.ca"
rel="noreferrer" target="_blank"
moz-do-not-send="true">gluste<wbr>rfs.ca</a><br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
11. Enable management encryption on
each node (command ran on gluster1,
gluster2, gluster-client):<br>
<br>
<br>
touch /var/lib/glusterd/secure-acces<wbr>s<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
12. Start, enable and check status
of glusterd on gluster1 and
gluster2:<br>
<br>
systemctl start glusterd<br>
systemctl enable glusterd<br>
systemctl status glusterd<br>
<br>
==============================<wbr>=======================<br>
<br>
13. Configure trusted storage pool
(TSP):<br>
<br>
>From gluster1:<br>
<br>
<br>
gluster peer status<br>
<br>
<br>
gluster peer probe gluster2<br>
<br>
==============================<wbr>=======================<br>
<br>
14. Create the replicated gluster
volume but don't start it:<br>
<br>
<br>
gluster volume create vol01 replica
2 transport tcp
gluster1:/data/glusterfs/glust<wbr>er1/vol01/brick1
gluster2:/data/glusterfs/glust<wbr>er2/vol01/brick1<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
15. Setup SSL/TLS access to the
volume:<br>
<br>
gluster volume set vol01
auth.ssl-allow
'gluster01,gluster02,gluster-c<wbr>lient'<br>
<br>
<br>
gluster volume set vol01 client.ssl
on<br>
gluster volume set vol01 server.ssl
on<br>
<br>
gluster volume set vol01
network.ping-timeout "5"<br>
<br>
gluster volume start vol01<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
16. Mount the volume on
gluster-client:<br>
<br>
mount -t glusterfs gluster1:/vol01
/mnt<br>
<br>
<br>
After mounting the volume, df -h has
the correct output:<br>
<br>
[root@gluster-client mnt]# df -h<br>
Filesystem Size Used Avail
Use% Mounted on<br>
/dev/sda1 48G 36G 9.3G
80% /<br>
devtmpfs 1.9G 0 1.9G
0% /dev<br>
tmpfs 1.9G 0 1.9G
0% /dev/shm<br>
tmpfs 1.9G 8.5M 1.9G
1% /run<br>
tmpfs 1.9G 0 1.9G
0% /sys/fs/cgroup<br>
tmpfs 379M 0 379M
0% /run/user/0<br>
gluster1:/vol01 15G 187M 15G
2% /mnt<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
17. If I try to copy an archive, it
throws an error after transferring
part of the file and the /mnt folder
becomes inaccessible for a while:<br>
<br>
[root@gluster-client mnt]# cp
/root/mybee.tar.gz /mnt/<br>
cp: error writing
‘/mnt/mybee.tar.gz’: Transport
endpoint is not connected<br>
cp: failed to extend
‘/mnt/mybee.tar.gz’: Transport
endpoint is not connected<br>
cp: failed to close
‘/mnt/mybee.tar.gz’: Transport
endpoint is not connected<br>
[root@gluster-client mnt]# ll /mnt<br>
ls: cannot open directory .:
Transport endpoint is not connected<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
This is the log from gluster-client
for the mnt mountpoint (mnt.log):<br>
<br>
<br>
[2018-06-29 07:52:03.869382] I
[MSGID: 100030]
[glusterfsd.c:2625:main]
0-/usr/sbin/glusterfs: Started
running /usr/sbin/glusterfs version
4.0.2 (args: /usr/sbin/glusterfs
--process-name fuse
--volfile-server=gluster1
--volfile-id=/vol01 /mnt)<br>
[2018-06-29 07:52:03.878709] I
[socket.c:4470:socket_init]
0-glusterfs: SSL support on the I/O
path is ENABLED<br>
[2018-06-29 07:52:03.878782] I
[socket.c:4473:socket_init]
0-glusterfs: SSL support for
glusterd is ENABLED<br>
[2018-06-29 07:52:03.878795] I
[socket.c:4490:socket_init]
0-glusterfs: using private polling
thread<br>
[2018-06-29 07:52:03.879108] E
[socket.c:4541:socket_init]
0-glusterfs: failed to open
/etc/ssl/dhparam.pem, DH ciphers are
disabled<br>
[2018-06-29 07:52:03.885718] I
[MSGID: 101190]
[event-epoll.c:609:event_dispa<wbr>tch_epoll_worker]
0-epoll: Started thread with index 1<br>
[2018-06-29 07:52:03.901983] I
[MSGID: 101190]
[event-epoll.c:609:event_dispa<wbr>tch_epoll_worker]
0-epoll: Started thread with index 2<br>
[2018-06-29 07:52:03.902092] I
[socket.c:4470:socket_init]
0-vol01-client-1: SSL support on the
I/O path is ENABLED<br>
[2018-06-29 07:52:03.902117] I
[socket.c:4473:socket_init]
0-vol01-client-1: SSL support for
glusterd is ENABLED<br>
[2018-06-29 07:52:03.902142] I
[socket.c:4490:socket_init]
0-vol01-client-1: using private
polling thread<br>
[2018-06-29 07:52:03.902266] E
[socket.c:4541:socket_init]
0-vol01-client-1: failed to open
/etc/ssl/dhparam.pem, DH ciphers are
disabled<br>
[2018-06-29 07:52:03.902977] I
[socket.c:4470:socket_init]
0-vol01-client-0: SSL support on the
I/O path is ENABLED<br>
[2018-06-29 07:52:03.903018] I
[socket.c:4473:socket_init]
0-vol01-client-0: SSL support for
glusterd is ENABLED<br>
[2018-06-29 07:52:03.903028] I
[socket.c:4490:socket_init]
0-vol01-client-0: using private
polling thread<br>
[2018-06-29 07:52:03.903161] E
[socket.c:4541:socket_init]
0-vol01-client-0: failed to open
/etc/ssl/dhparam.pem, DH ciphers are
disabled<br>
[2018-06-29 07:52:03.903756] I
[MSGID: 114020]
[client.c:2300:notify]
0-vol01-client-0: parent translators
are ready, attempting connect on
transport<br>
[2018-06-29 07:52:03.908432] I
[MSGID: 114020]
[client.c:2300:notify]
0-vol01-client-1: parent translators
are ready, attempting connect on
transport<br>
Final graph:<br>
+-----------------------------<wbr>------------------------------<wbr>-------------------+<br>
1: volume vol01-client-0<br>
2: type protocol/client<br>
3: option ping-timeout 5<br>
4: option remote-host gluster1<br>
5: option remote-subvolume
/data/glusterfs/gluster1/vol01<wbr>/brick1<br>
6: option transport-type
socket<br>
7: option
transport.address-family inet<br>
8: option
transport.socket.ssl-enabled on<br>
9: option
transport.tcp-user-timeout 0<br>
10: option
transport.socket.keepalive-tim<wbr>e
20<br>
11: option
transport.socket.keepalive-int<wbr>erval
2<br>
12: option
transport.socket.keepalive-cou<wbr>nt
9<br>
13: option send-gids true<br>
14: end-volume<br>
15:<br>
16: volume vol01-client-1<br>
17: type protocol/client<br>
18: option ping-timeout 5<br>
19: option remote-host gluster2<br>
20: option remote-subvolume
/data/glusterfs/gluster2/vol01<wbr>/brick1<br>
21: option transport-type
socket<br>
22: option
transport.address-family inet<br>
23: option
transport.socket.ssl-enabled on<br>
24: option
transport.tcp-user-timeout 0<br>
25: option
transport.socket.keepalive-tim<wbr>e
20<br>
26: option
transport.socket.keepalive-int<wbr>erval
2<br>
27: option
transport.socket.keepalive-cou<wbr>nt
9<br>
28: option send-gids true<br>
29: end-volume<br>
30:<br>
31: volume vol01-replicate-0<br>
32: type cluster/replicate<br>
33: option afr-pending-xattr
vol01-client-0,vol01-client-1<br>
34: option use-compound-fops
off<br>
35: subvolumes vol01-client-0
vol01-client-1<br>
36: end-volume<br>
37:<br>
38: volume vol01-dht<br>
39: type cluster/distribute<br>
40: option lock-migration off<br>
41: option force-migration off<br>
42: subvolumes
vol01-replicate-0<br>
43: end-volume<br>
44:<br>
45: volume vol01-write-behind<br>
46: type
performance/write-behind<br>
47: subvolumes vol01-dht<br>
48: end-volume<br>
49:<br>
50: volume vol01-read-ahead<br>
51: type performance/read-ahead<br>
52: subvolumes
vol01-write-behind<br>
53: end-volume<br>
54:<br>
55: volume vol01-readdir-ahead<br>
56: type
performance/readdir-ahead<br>
57: option parallel-readdir off<br>
58: option rda-request-size
131072<br>
59: option rda-cache-limit 10MB<br>
60: subvolumes vol01-read-ahead<br>
61: end-volume<br>
62:<br>
63: volume vol01-io-cache<br>
64: type performance/io-cache<br>
65: subvolumes
vol01-readdir-ahead<br>
66: end-volume<br>
67:<br>
68: volume vol01-quick-read<br>
69: type performance/quick-read<br>
70: subvolumes vol01-io-cache<br>
71: end-volume<br>
72:<br>
73: volume vol01-open-behind<br>
74: type
performance/open-behind<br>
75: subvolumes vol01-quick-read<br>
76: end-volume<br>
77:<br>
78: volume vol01-md-cache<br>
79: type performance/md-cache<br>
80: subvolumes
vol01-open-behind<br>
81: end-volume<br>
82:<br>
83: volume vol01<br>
84: type debug/io-stats<br>
85: option log-level INFO<br>
86: option latency-measurement
off<br>
87: option count-fop-hits off<br>
88: subvolumes vol01-md-cache<br>
89: end-volume<br>
90:<br>
91: volume meta-autoload<br>
92: type meta<br>
93: subvolumes vol01<br>
94: end-volume<br>
95:<br>
+-----------------------------<wbr>------------------------------<wbr>-------------------+<br>
[2018-06-29 07:52:03.917518] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.918459] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.919289] I
[rpc-clnt.c:2071:rpc_clnt_reco<wbr>nfig]
0-vol01-client-0: changing port to
49152 (from 0)<br>
[2018-06-29 07:52:03.922903] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.923922] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.925115] I
[rpc-clnt.c:2071:rpc_clnt_reco<wbr>nfig]
0-vol01-client-1: changing port to
49152 (from 0)<br>
[2018-06-29 07:52:03.930023] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.930756] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.931859] I
[MSGID: 114046]
[client-handshake.c:1176:clien<wbr>t_setvolume_cbk]
0-vol01-client-0: Connected to
vol01-client-0, attached to remote
volume
'/data/glusterfs/gluster1/vol0<wbr>1/brick1'.<br>
[2018-06-29 07:52:03.931894] I
[MSGID: 108005]
[afr-common.c:5081:__afr_handl<wbr>e_child_up_event]
0-vol01-replicate-0: Subvolume
'vol01-client-0' came back up; going
online.<br>
[2018-06-29 07:52:03.938903] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.939803] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:03.941300] I
[MSGID: 114046]
[client-handshake.c:1176:clien<wbr>t_setvolume_cbk]
0-vol01-client-1: Connected to
vol01-client-1, attached to remote
volume
'/data/glusterfs/gluster2/vol0<wbr>1/brick1'.<br>
[2018-06-29 07:52:03.942884] I
[fuse-bridge.c:4234:fuse_init]
0-glusterfs-fuse: FUSE inited with
protocol versions: glusterfs 7.24
kernel 7.22<br>
[2018-06-29 07:52:03.942907] I
[fuse-bridge.c:4864:fuse_graph<wbr>_sync]
0-fuse: switched to graph 0<br>
[2018-06-29 07:52:03.946767] W
[socket.c:592:__socket_rwv]
0-vol01-client-0: readv on <a
href="http://192.168.10.233:49152"
rel="noreferrer" target="_blank"
moz-do-not-send="true">192.168.10.233:49152</a>
failed (Input/output error)<br>
[2018-06-29 07:52:03.946791] E
[socket.c:2785:socket_poller]
0-vol01-client-0: socket_poller <a
href="http://192.168.10.233:49152"
rel="noreferrer" target="_blank"
moz-do-not-send="true">192.168.10.233:49152</a>
failed (Input/output error)<br>
[2018-06-29 07:52:03.946837] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:03.947264] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(LOOKUP(27)) called at
2018-06-29 07:52:03.945958 (xid=0xb)<br>
[2018-06-29 07:52:03.947297] W
[MSGID: 114031]
[client-rpc-fops_v2.c:2540:cli<wbr>ent4_0_lookup_cbk]
0-vol01-client-0: remote operation
failed. Path: /
(00000000-0000-0000-0000-00000<wbr>0000001)
[Transport endpoint is not
connected]<br>
[2018-06-29 07:52:03.947455] I
[MSGID: 109005]
[dht-selfheal.c:2328:dht_selfh<wbr>eal_directory]
0-vol01-dht: Directory selfheal
failed: Unable to form layout for
directory /<br>
[2018-06-29 07:52:07.885389] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:07.886457] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:07.887452] I
[rpc-clnt.c:2071:rpc_clnt_reco<wbr>nfig]
0-vol01-client-0: changing port to
49152 (from 0)<br>
[2018-06-29 07:52:07.900627] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:07.901594] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:07.903220] I
[MSGID: 114046]
[client-handshake.c:1176:clien<wbr>t_setvolume_cbk]
0-vol01-client-0: Connected to
vol01-client-0, attached to remote
volume
'/data/glusterfs/gluster1/vol0<wbr>1/brick1'.<br>
[2018-06-29 07:52:13.062404] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:13.062780] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.048761
(xid=0x22)<br>
[2018-06-29 07:52:13.062826] W
[MSGID: 114031]
[client-rpc-fops_v2.c:658:clie<wbr>nt4_0_writev_cbk]
0-vol01-client-0: remote operation
failed [Transport endpoint is not
connected]<br>
[2018-06-29 07:52:13.063017] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.049162
(xid=0x23)<br>
[2018-06-29 07:52:13.063175] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.049865
(xid=0x24)<br>
[2018-06-29 07:52:13.063319] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.050199
(xid=0x25)<br>
[2018-06-29 07:52:13.063473] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.050489
(xid=0x26)<br>
[2018-06-29 07:52:13.063600] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.050757
(xid=0x27)<br>
[2018-06-29 07:52:13.063745] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.051031
(xid=0x28)<br>
[2018-06-29 07:52:13.063883] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.053035
(xid=0x29)<br>
The message "W [MSGID: 114031]
[client-rpc-fops_v2.c:658:clie<wbr>nt4_0_writev_cbk]
0-vol01-client-0: remote operation
failed [Transport endpoint is not
connected]" repeated 7 times between
[2018-06-29 07:52:13.062826] and
[2018-06-29 07:52:13.063903]<br>
[2018-06-29 07:52:13.104580] W
[MSGID: 114031]
[client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
0-vol01-client-0: remote operation
failed<br>
[2018-06-29 07:52:13.109230] W
[socket.c:592:__socket_rwv]
0-vol01-client-1: writev on <a
href="http://192.168.10.234:49152"
rel="noreferrer" target="_blank"
moz-do-not-send="true">192.168.10.234:49152</a>
failed (No data available)<br>
[2018-06-29 07:52:13.109282] E
[socket.c:2777:socket_poller]
0-vol01-client-1: poll error on
socket<br>
[2018-06-29 07:52:13.109340] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:13.109357] E
[MSGID: 108006]
[afr-common.c:5158:__afr_handl<wbr>e_child_down_event]
0-vol01-replicate-0: All subvolumes
are down. Going offline until
atleast one of them comes back up.<br>
[2018-06-29 07:52:13.109461] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:13.109679] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FSTAT(25)) called at
2018-06-29 07:52:13.104513
(xid=0x42)<br>
[2018-06-29 07:52:13.109700] W
[MSGID: 114031]
[client-rpc-fops_v2.c:1260:cli<wbr>ent4_0_fstat_cbk]
0-vol01-client-1: remote operation
failed [Transport endpoint is not
connected]<br>
[2018-06-29 07:52:13.109854] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FXATTROP(34)) called at
2018-06-29 07:52:13.104729
(xid=0x43)<br>
[2018-06-29 07:52:13.109877] W
[MSGID: 114031]
[client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
0-vol01-client-1: remote operation
failed<br>
[2018-06-29 07:52:13.109897] E
[MSGID: 114031]
[client-rpc-fops_v2.c:1352:cli<wbr>ent4_0_finodelk_cbk]
0-vol01-client-0: remote operation
failed [Transport endpoint is not
connected]<br>
[2018-06-29 07:52:13.110036] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(WRITE(13)) called at
2018-06-29 07:52:13.105623
(xid=0x44)<br>
[2018-06-29 07:52:13.110053] W
[MSGID: 114031]
[client-rpc-fops_v2.c:658:clie<wbr>nt4_0_writev_cbk]
0-vol01-client-1: remote operation
failed [Transport endpoint is not
connected]<br>
[2018-06-29 07:52:13.110232] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FSTAT(25)) called at
2018-06-29 07:52:13.105658
(xid=0x45)<br>
[2018-06-29 07:52:13.110250] W
[MSGID: 114031]
[client-rpc-fops_v2.c:1260:cli<wbr>ent4_0_fstat_cbk]
0-vol01-client-1: remote operation
failed [Transport endpoint is not
connected]<br>
[2018-06-29 07:52:13.110386] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FSTAT(25)) called at
2018-06-29 07:52:13.106862
(xid=0x46)<br>
[2018-06-29 07:52:13.110530] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FSTAT(25)) called at
2018-06-29 07:52:13.108103
(xid=0x47)<br>
[2018-06-29 07:52:13.110660] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FSTAT(25)) called at
2018-06-29 07:52:13.108969
(xid=0x48)<br>
[2018-06-29 07:52:13.110793] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FXATTROP(34)) called at
2018-06-29 07:52:13.109256
(xid=0x49)<br>
The message "W [MSGID: 114031]
[client-rpc-fops_v2.c:1260:cli<wbr>ent4_0_fstat_cbk]
0-vol01-client-1: remote operation
failed [Transport endpoint is not
connected]" repeated 3 times between
[2018-06-29 07:52:13.110250] and
[2018-06-29 07:52:13.110675]<br>
[2018-06-29 07:52:13.110808] W
[MSGID: 114031]
[client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
0-vol01-client-1: remote operation
failed<br>
[2018-06-29 07:52:13.110933] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GlusterFS 4.x
v1) op(FXATTROP(34)) called at
2018-06-29 07:52:13.109284
(xid=0x4a)<br>
The message "W [MSGID: 114031]
[client-rpc-fops_v2.c:1570:cli<wbr>ent4_0_fxattrop_cbk]
0-vol01-client-1: remote operation
failed" repeated 3 times between
[2018-06-29 07:52:13.110808] and
[2018-06-29 07:52:14.878874]<br>
[2018-06-29 07:52:14.878895] E
[MSGID: 114031]
[client-rpc-fops_v2.c:1352:cli<wbr>ent4_0_finodelk_cbk]
0-vol01-client-1: remote operation
failed [Transport endpoint is not
connected]<br>
[2018-06-29 07:52:14.879095] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:14.879217] W
[fuse-bridge.c:2427:fuse_write<wbr>v_cbk]
0-glusterfs-fuse: 76: WRITE => -1
gfid=a8848be6-1c05-4a72-88a7-5<wbr>209eeef7c2c
fd=0x7f7c50023a68 (Transport
endpoint is not connected)<br>
[2018-06-29 07:52:14.880509] W
[fuse-bridge.c:1406:fuse_err_c<wbr>bk]
0-glusterfs-fuse: 77: FLUSH() ERR
=> -1 (Transport endpoint is not
connected)<br>
The message "I [MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up" repeated 11 times between
[2018-06-29 07:52:14.879095] and
[2018-06-29 07:52:17.908056]<br>
[2018-06-29 07:52:17.908066] E
[MSGID: 101046]
[dht-common.c:1501:dht_lookup_<wbr>dir_cbk]
0-vol01-dht: dict is null<br>
[2018-06-29 07:52:17.908175] W
[fuse-bridge.c:896:fuse_attr_c<wbr>bk]
0-glusterfs-fuse: 79: LOOKUP() /
=> -1 (Transport endpoint is not
connected)<br>
[2018-06-29 07:52:18.758975] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:18.759155] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:18.759165] E
[MSGID: 101046]
[dht-common.c:1501:dht_lookup_<wbr>dir_cbk]
0-vol01-dht: dict is null<br>
[2018-06-29 07:52:18.759214] W
[fuse-bridge.c:896:fuse_attr_c<wbr>bk]
0-glusterfs-fuse: 80: LOOKUP() /
=> -1 (Transport endpoint is not
connected)<br>
[2018-06-29 07:52:21.662626] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:21.662756] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:21.662765] E
[MSGID: 101046]
[dht-common.c:1501:dht_lookup_<wbr>dir_cbk]
0-vol01-dht: dict is null<br>
[2018-06-29 07:52:21.662811] W
[fuse-bridge.c:896:fuse_attr_c<wbr>bk]
0-glusterfs-fuse: 81: LOOKUP() /
=> -1 (Transport endpoint is not
connected)<br>
[2018-06-29 07:52:21.663567] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:21.663630] W
[fuse-bridge.c:3205:fuse_statf<wbr>s_cbk]
0-glusterfs-fuse: 82: ERR => -1
(Transport endpoint is not
connected)<br>
[2018-06-29 07:52:23.896362] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:23.896687] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:23.897321] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:23.896427 (xid=0x2a)<br>
[2018-06-29 07:52:23.897353] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:23.903241] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:23.903566] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:23.904166] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:23.903281 (xid=0x4b)<br>
[2018-06-29 07:52:23.904199] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:26.909819] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:26.910175] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:26.910858] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:26.909871 (xid=0x2b)<br>
[2018-06-29 07:52:26.910888] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:26.913921] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:26.914163] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:26.914417] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:26.913961 (xid=0x4c)<br>
[2018-06-29 07:52:26.914443] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:29.921628] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:29.921867] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:29.922215] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:29.921674 (xid=0x2c)<br>
[2018-06-29 07:52:29.922257] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:29.926462] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:29.926774] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:29.927415] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:29.926499 (xid=0x4d)<br>
[2018-06-29 07:52:29.927447] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:32.935218] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:32.935553] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:32.936196] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:32.935271 (xid=0x2d)<br>
[2018-06-29 07:52:32.936230] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:32.938940] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:32.939232] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:32.939497] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:32.938982 (xid=0x4e)<br>
[2018-06-29 07:52:32.939526] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:35.948789] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:35.949172] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:35.949860] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:35.948839 (xid=0x2e)<br>
[2018-06-29 07:52:35.949901] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:35.952154] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:35.952408] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:35.952670] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:35.952199 (xid=0x4f)<br>
[2018-06-29 07:52:35.952699] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:38.959369] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:38.959733] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:38.960536] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:38.959426 (xid=0x2f)<br>
[2018-06-29 07:52:38.960573] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:38.963381] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:38.963618] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:38.963877] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:38.963428 (xid=0x50)<br>
[2018-06-29 07:52:38.963906] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:41.969152] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:41.969397] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:41.969731] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:41.969204 (xid=0x30)<br>
[2018-06-29 07:52:41.969769] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:41.974130] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:41.974381] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:41.974669] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:41.974198 (xid=0x51)<br>
[2018-06-29 07:52:41.974698] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:44.984465] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:44.984781] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:44.985444] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:44.984514 (xid=0x31)<br>
[2018-06-29 07:52:44.985477] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:44.990039] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:44.990352] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:44.990729] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:44.990094 (xid=0x52)<br>
[2018-06-29 07:52:44.990771] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:47.994448] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:47.994684] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:47.994947] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:47.994499 (xid=0x32)<br>
[2018-06-29 07:52:47.994984] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:48.002544] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:48.002809] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:48.003388] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:48.002577 (xid=0x53)<br>
[2018-06-29 07:52:48.003416] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:49.440101] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:49.440257] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:49.440266] E
[MSGID: 101046]
[dht-common.c:1501:dht_lookup_<wbr>dir_cbk]
0-vol01-dht: dict is null<br>
[2018-06-29 07:52:49.440310] W
[fuse-bridge.c:896:fuse_attr_c<wbr>bk]
0-glusterfs-fuse: 83: LOOKUP() /
=> -1 (Transport endpoint is not
connected)<br>
[2018-06-29 07:52:50.182948] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:50.183108] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:50.183116] E
[MSGID: 101046]
[dht-common.c:1501:dht_lookup_<wbr>dir_cbk]
0-vol01-dht: dict is null<br>
[2018-06-29 07:52:50.197572] W
[fuse-bridge.c:896:fuse_attr_c<wbr>bk]
0-glusterfs-fuse: 84: LOOKUP() /
=> -1 (Transport endpoint is not
connected)<br>
[2018-06-29 07:52:50.575887] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:50.576030] I
[MSGID: 108006]
[afr-common.c:5444:afr_local_i<wbr>nit]
0-vol01-replicate-0: no subvolumes
up<br>
[2018-06-29 07:52:50.576039] E
[MSGID: 101046]
[dht-common.c:1501:dht_lookup_<wbr>dir_cbk]
0-vol01-dht: dict is null<br>
[2018-06-29 07:52:50.576087] W
[fuse-bridge.c:896:fuse_attr_c<wbr>bk]
0-glusterfs-fuse: 85: LOOKUP() /
=> -1 (Transport endpoint is not
connected)<br>
[2018-06-29 07:52:51.012626] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-0: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:51.012924] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-0: disconnected from
vol01-client-0. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:51.013615] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-0: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:51.012667 (xid=0x33)<br>
[2018-06-29 07:52:51.013647] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-0: received RPC
status error [Transport endpoint is
not connected]<br>
[2018-06-29 07:52:51.018855] W
[rpc-clnt.c:1739:rpc_clnt_subm<wbr>it]
0-vol01-client-1: error returned
while attempting to connect to
host:(null), port:0<br>
[2018-06-29 07:52:51.019093] I
[MSGID: 114018]
[client.c:2227:client_rpc_noti<wbr>fy]
0-vol01-client-1: disconnected from
vol01-client-1. Client process will
keep trying to connect to glusterd
until brick's port is available<br>
[2018-06-29 07:52:51.019378] E
[rpc-clnt.c:350:saved_frames_u<wbr>nwind]
(-->
/lib64/libglusterfs.so.0(_gf_l<wbr>og_callingfn+0x13b)[0x7f7c73b8<wbr>ae5b]
(-->
/lib64/libgfrpc.so.0(+0xce4e)[<wbr>0x7f7c73955e4e]
(-->
/lib64/libgfrpc.so.0(+0xcf6e)[<wbr>0x7f7c73955f6e]
(-->
/lib64/libgfrpc.so.0(rpc_clnt_<wbr>connection_cleanup+0x8d)[0x7f7<wbr>c7395760d]
(-->
/lib64/libgfrpc.so.0(+0xf178)[<wbr>0x7f7c73958178]
))))) 0-vol01-client-1: forced
unwinding frame type(GF-DUMP)
op(DUMP(1)) called at 2018-06-29
07:52:51.018893 (xid=0x54)<br>
[2018-06-29 07:52:51.019408] W
[MSGID: 114032]
[client-handshake.c:1620:clien<wbr>t_dump_version_cbk]
0-vol01-client-1: received RPC
status error [Transport endpoint is
not connected]<br>
<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
After trying to copy the archive, I
end up with different sized files on
the two bricks:<br>
<br>
[root@gluster1 brick1]# pwd<br>
/data/glusterfs/gluster1/vol01<wbr>/brick1<br>
[root@gluster1 brick1]# ll<br>
total 384<br>
-rw------- 2 root root 393216 Jun 29
10:52 mybee.tar.gz<br>
<br>
<br>
<br>
[root@gluster2 brick1]# pwd<br>
/data/glusterfs/gluster2/vol01<wbr>/brick1<br>
[root@gluster2 brick1]# ll<br>
total 1280<br>
-rw------- 2 root root 1310720 Jun
29 10:52 mybee.tar.gz<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
These are the informations regarding
the gluster volume:<br>
<br>
[root@gluster1 glusterfs]# gluster
volume info vol01<br>
<br>
Volume Name: vol01<br>
Type: Replicate<br>
Volume ID:
d3b39520-3fbb-4cfc-b736-338d77<wbr>26b5a1<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1:
gluster1:/data/glusterfs/glust<wbr>er1/vol01/brick1<br>
Brick2:
gluster2:/data/glusterfs/glust<wbr>er2/vol01/brick1<br>
Options Reconfigured:<br>
network.ping-timeout: 5<br>
server.ssl: on<br>
client.ssl: on<br>
auth.ssl-allow:
gluster01,gluster02,gluster-cl<wbr>ient<br>
transport.address-family: inet<br>
nfs.disable: on<br>
performance.client-io-threads: off<br>
<br>
==============================<wbr>=======================<br>
<br>
Here is the peers information:<br>
<br>
[root@gluster1 brick1]# gluster peer
status<br>
Number of Peers: 1<br>
<br>
Hostname: gluster2<br>
Uuid: 9ccac8e3-71c1-43e4-b723-fb7374<wbr>be8a1d<br>
State: Peer in Cluster (Connected)<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
Here is the volume status:<br>
<br>
[root@gluster1 brick1]# gluster
volume status vol01<br>
Status of volume: vol01<br>
Gluster process
TCP Port RDMA Port Online
Pid<br>
------------------------------<wbr>------------------------------<wbr>------------------<br>
Brick gluster1:/data/glusterfs/glust<wbr>er1/vol<br>
01/brick1
49152 0 Y 11108<br>
Brick gluster2:/data/glusterfs/glust<wbr>er2/vol<br>
01/brick1
49152 0 Y 10942<br>
Self-heal Daemon on localhost
N/A N/A Y 11133<br>
Self-heal Daemon on gluster2
N/A N/A Y 10968<br>
<br>
Task Status of Volume vol01<br>
------------------------------<wbr>------------------------------<wbr>------------------<br>
There are no active volume tasks<br>
<br>
<br>
==============================<wbr>=======================<br>
<br>
<br>
These are the installed packages on
gluster-client:<br>
<br>
[root@gluster-client glusterfs]# rpm
-qa | grep "gluster\|fuse"<br>
centos-release-gluster40-1.0-2<wbr>.el7.centos.x86_64<br>
glusterfs-client-xlators-4.0.2<wbr>-1.el7.x86_64<br>
fuse-2.9.2-10.el7.x86_64<br>
glusterfs-4.0.2-1.el7.x86_64<br>
glusterfs-fuse-4.0.2-1.el7.x86<wbr>_64<br>
fuse-libs-2.9.2-10.el7.x86_64<br>
glusterfs-libs-4.0.2-1.el7.x86<wbr>_64<br>
<br>
<br>
<br>
And these are the installed packages
on gluster1 and gluster2 storage
nodes:<br>
<br>
<br>
[root@gluster1 glusterfs]# rpm -qa |
grep "gluster\|fuse"<br>
glusterfs-api-4.0.2-1.el7.x86_<wbr>64<br>
centos-release-gluster40-1.0-2<wbr>.el7.centos.x86_64<br>
glusterfs-libs-4.0.2-1.el7.x86<wbr>_64<br>
glusterfs-client-xlators-4.0.2<wbr>-1.el7.x86_64<br>
glusterfs-cli-4.0.2-1.el7.x86_<wbr>64<br>
glusterfs-fuse-4.0.2-1.el7.x86<wbr>_64<br>
glusterfs-server-4.0.2-1.el7.x<wbr>86_64<br>
glusterfs-4.0.2-1.el7.x86_64<br>
<br>
<br>
<br>
<br>
Can anyone point me in the right
direction? What am I doing wrong?
The mount works fine until I try to
copy an archive, multiple smaller
files or a bigger file on it
(meaning it shows correctly in df
-Th and I can create several files
with "touch file1 file2...").
Basically, after any data transfer,
I get these errors.<br>
<br>
<br>
I followed the indications from the
redhat page:<br>
<br>
<a
href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/chap-network_encryption"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://access.redhat.com/docu<wbr>mentation/en-us/red_hat_gluste<wbr>r_storage/3.1/html/administrat<wbr>ion_guide/chap-network_encrypt<wbr>ion</a><br>
<br>
UPDATE 1:<br>
I tried doing the exact same steps
in Gluster 3.12 and had no problem.
The steps worked and SSL/TLS was
enabled. There was no transport
error or anything and I also checked
if SSL/TLS was enabled. Afterwards,
I also tried with the new release
4.1 and the problem persists (same
error with "Transport endpoint is
not connected").<br>
Let me know if you need any other
info. Any help is much appreciated.<br>
<br>
Thank you.<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a
href="mailto:Gluster-users@gluster.org"
target="_blank"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank"
moz-do-not-send="true">https://lists.gluster.org/mail<wbr>man/listinfo/gluster-users</a></blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<div class="gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr">
<div>Amar Tumballi (amarts)<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>