[Gluster-users] Self healing of 3.3.0 cause our 2 bricks replicated cluster freeze (client read/write timeout)
driver at megahappy.net
Thu Nov 29 06:48:30 UTC 2012
when you mount xfs, also use the inode64 option. That will help with xfs
My offhand guess is you are likely running into limited network bandwidth
for the 2 bricks to sync. As the network gets flooded nfs response gets
poor. Make sure you are getting full-duplex connections - or upgrade your
network to 10G or (even better) Infiniband.
On Mon, Nov 26, 2012 at 1:46 AM, ZHANG Cheng <czhang.oss at gmail.com> wrote:
> Early this morning our 2 bricks replicated cluster had an outage. The
> disk space for one of the brick server (brick02) was used up. When we
> responded to the disk full alert, the issue already lasted for a few
> hours. We reclaimed some disk space, and reboot the brick02 server,
> expecting once it come back it will go self healing.
> It did go self healing, but just after couple minutes, access to
> gluster filesystem freeze. Tons of "nfs: server brick not responding,
> still trying" popped up in dmesg. The load average on app server went
> up to 200 something from usual 0.10. We had to shutdown brick02 server
> or stop gluster server process on it, to get the gluster cluster back
> How could we deal with this issue? Thanks in advance.
> Our gluster setup is followed the official doc.
> gluster> volume info
> Volume Name: staticvol
> Type: Replicate
> Volume ID: fdcbf635-5faf-45d6-ab4e-be97c74d7715
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Brick1: brick01:/exports/static
> Brick2: brick02:/exports/static
> Underlying filesystem is xfs (on a lvm volume), as:
> /dev/mapper/vg_node-brick on /exports/static type xfs
> The brick servers don't act as gluster client.
> Our app servers are the gluster client, mount via nfs.
> brick:/staticvol on /mnt/gfs-static type nfs
> brick is a DNS round-robin record for brick01 and brick02.
> Gluster-users mailing list
> Gluster-users at gluster.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Gluster-users