[Gluster-users] gluster fails under heavy array job load load

[harry mangalam]

On Thursday, December 12, 2013 11:46:03 PM Anand Avati wrote:
> - I see RDMA is enabled on the volume. Are you mounting clients through
> RDMA? If so, for the purpose of diagnostics can you mount through TCP and
> check the stability improves? If you are using RDMA with such a high
> write-behind-window-size, spurious ping-timeouts are an almost certainty
> during heavy writes. The RDMA driver has limited flow control, and setting
> such a high window-size can easily congest all the RDMA buffers resulting
> in spurious ping-timeouts and disconnections. 

Is there a way to remove the RDMA transport option once it is enabled.  I was 
under the impression that our system was NOT using RDMA, but from the logs, I 
see the following that implies that they /are/ using RDMA now.

==> 10.2.7.11 <==
  4:     option transport-type socket,rdma
[2013-12-10 17:42:12.498076] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid1.rdma on port 49153
[2013-12-10 17:42:15.571287] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid2.rdma on port 49155

==> 10.2.7.12 <==
  4:     option transport-type socket,rdma
[2013-12-10 17:42:17.974841] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid1.rdma on port 49153
[2013-12-10 17:42:21.266486] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid2.rdma on port 49155

==> 10.2.7.13 <==
  4:     option transport-type socket,rdma
[2013-12-10 17:42:17.929753] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid1.rdma on port 49153
[2013-12-10 17:42:21.646482] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid2.rdma on port 49155

==> 10.2.7.14 <==
  4:     option transport-type socket,rdma
[2013-12-10 17:42:15.791176] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid1.rdma on port 49153
[2013-12-10 17:42:15.941182] I [glusterd-pmap.c:227:pmap_registry_bind] 0-
pmap: adding brick /raid2.rdma on port 49155


---
Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine
[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487
415 South Circle View Dr, Irvine, CA, 92697 [shipping]
MSTB Lat/Long: (33.642025,-117.844414) (paste into Google Maps)
---
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131213/145bcfc9/attachment.html>