[Gluster-users] strange error hangs hangs any access to gluster mount

[Amar Tumballi]

Hi James,

To fix this, you can go to *any one pair* backend and run below commands on
the directories where the layout has issues:

bash# setfattr -x trusted.glusterfs.dht <directory>

[ pair backend means, in replica set's volumes ]

and then from the client machine (ie, where you have mount point), run below
commands,

 bash# echo 3 > /proc/sys/vm/drop_caches
 bash# stat <directory> # through the mount point.

In this step, the layout will get fixed again automatically, which should
solve this issue.

Regards,
Amar


On Tue, Mar 29, 2011 at 12:45 AM, Burnash, James <jburnash at knight.com>wrote:

> Thanks Jeff. That at least gives me shot at figuring out some similar
> problems.
>
> It's possible that in the course of bringing up the mirrors initially I
> futzed something up. I'll have to check the read-write servers as well.
>
> James Burnash, Unix Engineering
>
> -----Original Message-----
> From: Jeff Darcy [mailto:jdarcy at redhat.com]
> Sent: Monday, March 28, 2011 3:09 PM
> To: Burnash, James
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] strange error hangs hangs any access to
> gluster mount
>
> On 03/28/2011 02:29 PM, Burnash, James wrote:
> > Sorry - paste went awry.
> >
> > Updated here:
> >
> > http://pastebin.com/M74LAYej
>
> OK, that definitely shows a problem.  Here's the whole map of which
> nodes are claiming which ranges:
>
> 00000000 0ccccccb: g07 on gfs17/gfs18
> 0ccccccc 19999997: g08 on gfs17/gfs18
> 19999998 26666663: g09 on gfs17/gfs18
> 26666664 3333332f: g10 on gfs17/gfs18
> 33333330 3ffffffb: g01 on gfs17/gfs18
> 3ffffffc 4cccccc7: g02 on gfs17/gfs18
> 4cccccc8 59999993: g01 on gfs14/gfs14
> 59999994 6666665f: g02 on gfs14/gfs14
> 66666660 7333332b: g03 on gfs14/gfs14
> 7333332c 7ffffff7: g04 on gfs14/gfs14
> 7ffffff8 8cccccc3: g05 on gfs14/gfs14
> 8cccccc4 9999998f: g06 on gfs14/gfs14
> 99999990 a666665b: g07 on gfs14/gfs14
> a666665c b3333327: g08 on gfs14/gfs14
> b3333328 b333332e: g09 on gfs14/gfs14
> b333332f bffffff3: g09 on gfs14/gfs14
>                   *** AND g04 on gfs17/18
> bffffff4 ccccccbf: g10 on gfs14/gfs14
>                   *** AND g04 on gfs17/18
> ccccccc0 ccccccc7: g03 on gfs17/gfs18
>                   *** AND g04 on gfs17/18
> ccccccc8 d999998b: g03 on gfs17/gfs18
> d999998c e6666657: *** GAP ***
> e6666658 f3333323: g05 on gfs17/gfs18
> f3333324 ffffffff: g06 on gfs17/gfs18
>
> I know this all seems like numerology, but bear with me.  Note that all
> of the problems seem to involve g04 on gfs17/gfs18 claiming the wrong
> range, and that the range it's claiming is almost exactly twice the size
> of all the other ranges.  In fact, it's the range it would have been
> assigned if there had been ten nodes instead of twenty.  For example, if
> that filesystem had been restored to an earlier state on gfs17/gfs18,
> and then self-healed in the wrong direction (self-mangled?) you would
> get exactly this set of symptoms.  I'm not saying that's what happened;
> it's just a way to illustrate what these values mean and the
> consequences of their being out of sync with each other.
>
> So, why only one client?  Since you're reporting values on the servers,
> I'd guess it's because only that client has remounted.  The others are
> probably still operating from cached (and apparently correct) layout
> information.  This is a very precarious state, I'd have to say.  You
> *might* be able to fix this by fixing the xattr values on that one
> filesystem, but I really can't recommend trying that without some input
> from Gluster themselves.
>
>
> DISCLAIMER:
> This e-mail, and any attachments thereto, is intended only for use by the
> addressee(s) named herein and may contain legally privileged and/or
> confidential information. If you are not the intended recipient of this
> e-mail, you are hereby notified that any dissemination, distribution or
> copying of this e-mail, and any attachments thereto, is strictly prohibited.
> If you have received this in error, please immediately notify me and
> permanently delete the original and any copy of any e-mail and any printout
> thereof. E-mail transmission cannot be guaranteed to be secure or
> error-free. The sender therefore does not accept liability for any errors or
> omissions in the contents of this message which arise as a result of e-mail
> transmission.
> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at
> its discretion, monitor and review the content of all e-mail communications.
> http://www.knight.com
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>