[Gluster-users] Files present on the backend but have become invisible from clients

[Joe Landman]

Interestingly, one of our customers is running into this, though with 
DHT only, and on a 3.0.5 system.  Just filed a report #3646 in the 
ticketing system.

Files which are invisible have no xattrs (I do not believe they went 
poking around the backend on their own).  This includes many of the 
directories that are missing ... no xattr.

Is there any curative measures we can take?  Copy out of backend and 
copy back in?




On 06/23/2011 02:05 PM, Burnash, James wrote:
> Sorry – volfile:
>
> http://pastebin.com/2hYZw2V9
>
> James Burnash
>
> Unix Engineer
>
> Knight Capital Group
>
> *From:*gluster-users-bounces at gluster.org
> [mailto:gluster-users-bounces at gluster.org] *On Behalf Of *Burnash, James
> *Sent:* Thursday, June 23, 2011 2:03 PM
> *To:* 'Anand Avati'; Jeff Darcy
> *Cc:* gluster-users at gluster.org
> *Subject:* Re: [Gluster-users] Files present on the backend but have
> become invisible from clients
>
> Hi Anand.
>
> Here is my script (no heckling – I know I do not produce beautiful code
> J), and below is the invocation I used:
>
> #!/bin/bash
>
> #
>
> # description: # Gluster server script to check brick extended attributes
>
> #
>
> # J. Burnash 2011
>
> #
>
> # $Id$
>
> ECHO_CMD=""
>
> SORT_CMD="cat"
>
> brick_parent="read-only"
>
> while getopts "a:b:ehsw" flag
>
> do
>
> case $flag in
>
> a) attribute=$OPTARG;
>
> OPTARG="";
>
> ;;
>
> b) brick=$OPTARG;
>
> OPTARG="";
>
> ;;
>
> e) ECHO_CMD="-n";
>
> ;;
>
> s) SORT_CMD="sort -k 2";
>
> ;;
>
> w) brick_parent="read-write"
>
> OPTARG="";
>
> ;;
>
> h) progname=$(basename $0);
>
> echo "$progname: Usage:";
>
> echo "$progname <-a attribute> <-b brick> hostname1 <hostname2> ...
> <hostnameN>";
>
> echo "where -e puts the hostname at the beginning of the output from the
> requested comand,";
>
> echo " -a <attribute name> sets the extended attribute to be displayed,";
>
> echo " -b <brick name> sets the brick for which extended attributes are
> to be displayed,";
>
> echo "and -h gives this help message";
>
> echo "the brick and attribute arguments may be enclosed in quotes to
> allow multiple matches for each host";
>
> echo "Example: $progname 'service nitemond restart' hostname1 <hostname2>"
>
> exit;
>
> ;;
>
> *) echo "Unknown argument $flag";
>
> ;;
>
> esac
>
> # Debug
>
> #echo "Flag=$flag OPTIND=$OPTIND OPTARG=$OPTARG"
>
> done
>
> # Dispose of processed option arguments:
>
> shift $((OPTIND-1)); OPTIND=1
>
> loop_check -f $ECHO_CMD "cd /export/$brick_parent; for brick_root in
> $brick; do echo -n \$HOSTNAME; getfattr -d -e hex -m $attribute
> \$brick_root | xargs echo | sed -e \"s/# file:/ /\" ; done" $@ | awk
> '{print $2,$3,$1}' | $SORT_CMD
>
> Invocation: check_brick_attrs_ro -e -s -a trusted.afr -b 'g0{1,2}'
> jc1letgfs{14,15,17,18}
>
> Thanks,
>
> James Burnash
>
> Unix Engineer
>
> Knight Capital Group
>
> *From:*Anand Avati [mailto:anand.avati at gmail.com]
> *Sent:* Thursday, June 23, 2011 6:31 AM
> *To:* Jeff Darcy
> *Cc:* Burnash, James; gluster-users at gluster.org
> *Subject:* Re: [Gluster-users] Files present on the backend but have
> become invisible from clients
>
> On Thu, Jun 23, 2011 at 2:10 AM, Jeff Darcy <jdarcy at redhat.com
> <mailto:jdarcy at redhat.com>> wrote:
>
> On 06/22/2011 02:44 PM, Burnash, James wrote:
>  > g01/pfs-ro1-client-0=0x000000000000000000000000 jc1letgfs17
>  > g01/pfs-ro1-client-0=0x000000000600000800000000 jc1letgfs18
>  > g01/pfs-ro1-client-20=0x000000000000000000000000 jc1letgfs14
>  > g01/pfs-ro1-client-20=0x000000000200000000000000 jc1letgfs15
>  > g02/pfs-ro1-client-2=0x000000000000000000000000 jc1letgfs17
>  > g02/pfs-ro1-client-2=0x000000004500000400000000 jc1letgfs18
>  > g02/pfs-ro1-client-22=0x000000000000000000000000 jc1letgfs14
>  > g02/pfs-ro1-client-22=0x000000000200000000000000 jc1letgfs15
>
>  >
>  > Would anybody have any insights as to what is going on here? I'm
>  > seeing attributes in my sleep these days ... that cannot be good!
>
> Can you give your script or explain what each of those fields mean and
> how they fit into your volume configuration? Also can you post your
> client volfile?
>
> Avati
>
> DISCLAIMER:
> This e-mail, and any attachments thereto, is intended only for use by
> the addressee(s)named herein and
> may contain legally privileged and/or confidential information. If you
> are not the intended recipient of this
> e-mail, you are hereby notified that any dissemination, distribution or
> copying of this e-mail and any attachments
> thereto, is strictly prohibited. If you have received this in error,
> please immediately notify me and permanently
> delete the original and any printout thereof.E-mail transmission cannot
> be guaranteed to be secure or error-free.
> The sender therefore does not accept liability for any errors or
> omissions in the contents of this message which
> arise as a result of e-mail transmission.
> NOTICE REGARDING PRIVACY AND CONFIDENTIALITY
> Knight Capital Group may, at its discretion, monitor and review the
> content of all e-mail communications.
>
> http://www.knight.com <http://www.knight.com/>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users


-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics, Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615