[Gluster-users] Input/output error when trying to access a file on client

[Krutika Dhananjay]

Hi, 

Could you provide the xattrs in hex format? 

You can execute `getfattr -d -m . -e hex <path-to-the-directory/file-on-the-brick(s)>` 

-Krutika 

----- Original Message -----

> From: "Alessandro Ipe" <Alessandro.Ipe at meteo.be>
> To: "Krutika Dhananjay" <kdhananj at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Thursday, March 12, 2015 5:15:08 PM
> Subject: Re: [Gluster-users] Input/output error when trying to access a file
> on client

> Hi,

> Actually, my gluster volume is distribute-replicate so I should provide the
> attributes on all the bricks. Here they are:

> 1. # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA

> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==

> 2. # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA

> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==

> 3. # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA

> trusted.afr.md1-client-3=0sAAAAAAAAAAEAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA==

> 4. # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-2=0sAAAAAAAAAAEAAAAA

> trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAAAAAAAAVVVVVA==

> 5. # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-4=0sAAAAAAAAAAAAAAAA

> trusted.afr.md1-client-5=0sAAAAAAAAAAEAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ==

> 6. # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-4=0sAAAAAAAAAAEAAAAA

> trusted.afr.md1-client-5=0sAAAAAAAAAAAAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAABVVVVVqqqqqQ==

> so it seems in fact that there are discrepancies between 3-4 and 5-6
> (replicate pairs).

> A.

> On Thursday 12 March 2015 11:33:00 Alessandro Ipe wrote:

> Hi,

> "gluster volume heal md1 info split-brain" returns approximatively 2000 files
> (already divided by 2

> due to replicate volume). So manually repairing each split-brain is
> unfeasable. Before scripting some

> procedure, I need to be sure that I will not harm further the gluster system.

> Moreover, I noticed that the messages printed in the logs are all about
> directories,

> e.g.

> [2015-03-12 10:06:53.423856] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-1:
> Unable to self-heal contents of '/root' (possible split-brain). Please
> delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0
> 1 ] [ 1 0 ] ]

> [2015-03-12 10:06:53.424005] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-2:
> Unable to self-heal contents of '/root' (possible split-brain). Please
> delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0
> 1 ] [ 1 0 ] ]

> [2015-03-12 10:06:53.424110] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-md1-replicate-1: metadata self heal failed, on /root

> [2015-03-12 10:06:53.424290] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-md1-replicate-2: metadata self heal failed, on /root

> Getting the attributes of that directory on each brick gives me for the first

> # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA

> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==

> and for the second

> # file: data/glusterfs/md1/brick1/root

> trusted.afr.md1-client-0=0sAAAAAAAAAAAAAAAA

> trusted.afr.md1-client-1=0sAAAAAAAAAAAAAAAA

> trusted.gfid=0s3DmMvSq0QOyf7T1ZN2VPSw==

> trusted.glusterfs.dht=0sAAAAAQAAAACqqqqq/////w==

> so it seems that there are both rigorously identical. However, according to
> your split -brain tutorial,

> none of them has 0x000000000000000000000000. What 0sAAAAAAAAAAAAAAAA means

> in fact ?

> Should I change both attributes on each directory to
> 0x000000000000000000000000 ?

> Many thanks,

> A.

> On Wednesday 11 March 2015 08:02:56 Krutika Dhananjay wrote:

> Hi,

> Have you gone through
> https://github.com/gluster/glusterfs/blob/master/doc/debugging/split-brain.md
> ?

> If not, could you go through that once and try the steps given there? Do let
> us know if something is not clear in the doc.

> -Krutika

> From: "Alessandro Ipe" <Alessandro.Ipe at meteo.be>
> To: gluster-users at gluster.org
> Sent: Wednesday, March 11, 2015 4:54:09 PM
> Subject: Re: [Gluster-users] Input/output error when trying to access a file
> on client

> Well, it is even worse. Now when doing a "ls -R" on the volume results in a
> lot of

> [2015-03-11 11:18:31.957505] E
> [afr-self-heal-common.c:233:afr_sh_print_split_brain_log] 0-md1-replicate-2:
> Unable to self-heal contents of '/library' (possible split-brain). Please
> delete the file from all but the preferred subvolume.- Pending matrix: [ [ 0
> 2 ] [ 1 0 ] ]
> [2015-03-11 11:18:31.957692] E
> [afr-self-heal-common.c:2868:afr_log_self_heal_completion_status]
> 0-md1-replicate-2: metadata self heal failed, on /library

> I am desperate...

> A.

> On Wednesday 11 March 2015 12:05:33 you wrote:
> > Hi,
> >
> >
> > When trying to access a file on a gluster client (through fuse), I get an
> > "Input/output error" message.
> >
> > Getting the attributes for the file gives me for the first brick
> > # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2
> > trusted.afr.md1-client-2=0sAAAAAAAAAAAAAAAA
> > trusted.afr.md1-client-3=0sAAABdAAAAAAAAAAA
> > trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ==
> >
> > while for the second (replicate) brick
> > # file: data/glusterfs/md1/brick1/kvm/hail/hail_home.qcow2
> > trusted.afr.md1-client-2=0sAAABJAAAAAAAAAAA
> > trusted.afr.md1-client-3=0sAAAAAAAAAAAAAAAA
> > trusted.gfid=0sOCFPGCdrQ9uyq2yTTPCKqQ==
> >
> > It seems that I have a split-brain. How can I solve this issue by resetting
> > the attributes, please ?
> >
> >
> > Thanks,
> >
> >
> > Alessandro.
> >
> > ==================
> > gluster volume info md1
> >
> > Volume Name: md1
> > Type: Distributed-Replicate
> > Volume ID: 6da4b915-1def-4df4-a41c-2f3300ebf16b
> > Status: Started
> > Number of Bricks: 3 x 2 = 6
> > Transport-type: tcp
> > Bricks:
> > Brick1: tsunami1:/data/glusterfs/md1/brick1
> > Brick2: tsunami2:/data/glusterfs/md1/brick1
> > Brick3: tsunami3:/data/glusterfs/md1/brick1
> > Brick4: tsunami4:/data/glusterfs/md1/brick1
> > Brick5: tsunami5:/data/glusterfs/md1/brick1
> > Brick6: tsunami6:/data/glusterfs/md1/brick1
> > Options Reconfigured:
> > server.allow-insecure: on
> > cluster.read-hash-mode: 2
> > features.quota: off
> > performance.write-behind: on
> > performance.write-behind-window-size: 4MB
> > performance.flush-behind: off
> > performance.io-thread-count: 64
> > performance.cache-size: 512MB
> > nfs.disable: on
> > cluster.lookup-unhashed: off

> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

> --

> Dr. Ir. Alessandro Ipe

> Department of Observations Tel. +32 2 373 06 31

> Remote Sensing from Space Fax. +32 2 374 67 88

> Royal Meteorological Institute

> Avenue Circulaire 3 Email:

> B-1180 Brussels Belgium Alessandro.Ipe at meteo.be

> Web: http://gerb.oma.be
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150312/5624c62c/attachment.html>