<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Dear Gluster-Users,</p>
<p>I am experiencing RDMA problems.<br>
</p>
<p>I have installed Ubuntu 16.04.4 running with 4.15.0-13-generic
kernel, MLNX_OFED_LINUX-4.3-1.0.1.0-ubuntu16.04-x86_64 to 4
different servers. All of them has Mellanox ConnectX-4 LX dual
port NICs. These four servers are connected via Mellanox SN2100
Switch. <br>
</p>
<p>I have installed GlusterFS Server v3.10 (from Ubuntu PPA) to 3
servers. These 3 boxes are running as gluster cluster.
Additionally, I have installed Glusterfs Client to the last one.</p>
<p>I have created Gluster Volume with this command:</p>
<p># gluster volume create db transport rdma replica 3 arbiter 1
gluster1:/storage/db/ gluster2:/storage/db/ cinder:/storage/db
force</p>
<p>(network.ping-timeout is 3)<br>
</p>
<p>Then I have mounted this volume using mount command below.<br>
</p>
<p>mount -t glusterfs -o transport=rdma gluster1:/db /db</p>
<p>After mountings "/db", I can access the files. <br>
</p>
<p>The problem is, when I reboot one of the cluster nodes, fuse
client gives this error below and hangs. <br>
</p>
<p>[2018-04-17 07:42:55.506422] W [MSGID: 103070]
[rdma.c:4284:gf_rdma_handle_failed_send_completion]
0-rpc-transport/rdma: <b>send work request on `mlx5_0' returned
error wc.status = 5, wc.vendor_err = 245, post->buf =
0x7f8b92016000, wc.byte_len = 0, post->reused = 135</b><br>
</p>
<p>When I change transport mode from rdma to tcp, fuse client works
well. No hangs.<br>
</p>
<p>I also tried Gluster 3.8, 3.10, 4.0.0 and 4.0.1 (from Ubuntu
PPAs) on Ubuntu 16.04.4 and Centos 7.4. But results were the same.<br>
</p>
<p>Thank you.</p>
Necati. <br>
</body>
</html>