[Gluster-users] strange error hangs any access to gluster mount

[Burnash, James]

Hi Jeff - thanks for responding so quickly - MUCH appreciated.

The fix I made was this:

1) From Amar:

go to any one pair backend and run below commands on the directories where the layout has issues:

bash# setfattr -x trusted.glusterfs.dht <directory>

[ pair backend means, in replica set's volumes ]

and then from the client machine (ie, where you have mount point), run below commands,

 bash# echo 3 > /proc/sys/vm/drop_caches
 bash# stat <directory> # through the mount point.

In this step, the layout will get fixed again automatically, which should solve this issue.

2) By examining the logs on every client (grepping on the keyword "Given" with the -B flag grabbing the line directly before, I was able to find out the last time the client had been restarted, and identify 7 clients that had the wrong layout. I believe that once they had been restarted, the self heal fixed the errant attributes.

I consider this problem fixed now.

Many thanks to Jeff, Amar, and everyone else who helped.

James Burnash, Unix Engineering

-----Original Message-----
From: Jeff Darcy [mailto:jdarcy at redhat.com]
Sent: Monday, April 18, 2011 9:39 AM
To: Burnash, James
Cc: 'Amar Tumballi'; gluster-users at gluster.org
Subject: Re: [Gluster-users] strange error hangs any access to gluster mount

On 04/18/2011 09:23 AM, Burnash, James wrote:
> After taking the actions suggested by Jeff and confirmed by Amar, it
> looks like all of my maps are clean (thought I am still having problems,
> which may or may not be related).
>
>
>
> Here are the maps after all the corrections were done:
>
>  ...
>
> How do these look to you? I don’t see a gap/hole, and since mirrored
> pairs are claiming (only) the same range for each mirrored brick, all
> looks ok to me.  Can  one / both of you just confirm that my layout now
> looks clean?

Those look much healthier to me, and I'm glad to hear that at least some
issues seem to have been resolved.  Just for future reference, did the
suggestion to modify the layout xattr directly work, or did you have to
do a full manual migration of the files on the "weird" subvolume?  It
seems like a couple of other users on IRC have run into layout issues in
3.1.3 that look disturbingly similar, so it would be nice to know which
fixes work and which don't.  Thanks!


DISCLAIMER:
This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com