[Gluster-users] Self healing of 3.3.0 cause our 2 bricks replicated cluster freeze (client read/write timeout)

[ZHANG Cheng]

We had a planned outage yesterday, which requires us to shutdown one
of replicated brick server (brick02) for 30 minutes. The maintenance
went smooth. But after about couple minutes I brought brick02 back
online, our app server's load raise to a very high number (200~300+),
as it came to a halt as a REST API backend. The same problem we had
that described in previous post.

If I shutdown brick02, restart the jboss-as powered our REST API, the
app server's load will keep at normal level. So looks like our app
server's file access pattern will lead to file operation freeze during
the gluster server's self healing process.

Our app is a REST API backend for mobile forum/community, so the main
content is threads and posts, which contains pictures, showing in
pinterest style in our iOS app. For each picture URL in the JSON
response, our API server's java code does a check as:

  public static boolean checkImage(String path){
      File file=new File(path);
      if(null!=file&&file.exists()&&file.length()>0){
          return true;
      }
      return false;
  }

Usually for each of such response, there are about 10 to 20 pictures
in it, which means this checkImage() will be called that many times.
Because most of request are asking for recent uploaded pictures, these
picture files almost certain are the kind of files requiring self
healing. Even during our off-peak hours there are 0-3 such thread/post
api request per second, sooner or later we will run into the same
freezing problem, if the glusterfs servers are doing self healing.

I think now I have more concrete info to file a bug report.

On Mon, Nov 26, 2012 at 5:46 PM, ZHANG Cheng <czhang.oss at gmail.com> wrote:
> Early this morning our 2 bricks replicated cluster had an outage. The
> disk space for one of the brick server (brick02) was used up. When we
> responded to the disk full alert, the issue already lasted for a few
> hours. We reclaimed some disk space, and reboot the brick02 server,
> expecting once it come back it will go self healing.
>
> It did go self healing, but just after couple minutes, access to
> gluster filesystem freeze. Tons of "nfs: server brick not responding,
> still trying" popped up in dmesg. The load average on app server went
> up to 200 something from usual 0.10. We had to shutdown brick02 server
> or stop gluster server process on it, to get the gluster cluster back
> working.
>
> How could we deal with this issue? Thanks in advance.
>
> Our gluster setup is followed the official doc.
>
> gluster> volume info
>
> Volume Name: staticvol
> Type: Replicate
> Volume ID: fdcbf635-5faf-45d6-ab4e-be97c74d7715
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: brick01:/exports/static
> Brick2: brick02:/exports/static
>
> Underlying filesystem is xfs (on a lvm volume), as:
> /dev/mapper/vg_node-brick on /exports/static type xfs
> (rw,noatime,nodiratime,nobarrier,logbufs=8)
>
> The brick servers don't act as gluster client.
>
> Our app servers are the gluster client, mount via nfs.
> brick:/staticvol on /mnt/gfs-static type nfs
> (rw,noatime,nodiratime,vers=3,rsize=8192,wsize=8192,addr=10.10.10.51)
>
> brick is a DNS round-robin record for brick01 and brick02.