[Gluster-users] Client and server file "view", different results?! Client can't see the right file.
Martin Schenker
martin.schenker at profitbricks.com
Tue May 17 10:19:30 UTC 2011
It version 3.1.3 (we tried 3.2.0 for about 10h and rolled back)
Unfortunateltly the file view was "repaired" already by brutally copying
manually from the correct /mnt (server) mountpoint to the /opt (client)
mount which fixed the situation for now. We needed the files accessible
ASAP.
Best, Martin
> -----Original Message-----
> From: Pranith Kumar. Karampuri [mailto:pranithk at gluster.com]
> Sent: Tuesday, May 17, 2011 10:41 AM
> To: Martin Schenker
> Cc: gluster-users at gluster.org
> Subject: Re: [Gluster-users] Client and server file "view",
> different results?! Client can't see the right file.
>
>
> hi Martin,
> Could you please gather the following outputs so that we
> can debug as to what is happening:
> 1) whats the version of the gluster.
> 2) backend "ls -l" of files in question on all bricks that
> file is replicated on.
> 3) 'ls -l" o/p from mnt point for that file.
>
> Thanks
> Pranith
> ----- Original Message -----
> From: "Martin Schenker" <martin.schenker at profitbricks.com>
> To: "Pranith Kumar. Karampuri" <pranithk at gluster.com>
> Cc: gluster-users at gluster.org
> Sent: Tuesday, May 17, 2011 11:13:32 AM
> Subject: RE: [Gluster-users] Client and server file "view",
> different results?! Client can't see the right file.
>
> Yes, it is!
>
> Here's the volfile:
>
> cat /mnt/gluster/brick0/config/vols/storage0/storage0-fuse.vol:
>
> volume storage0-client-0
> type protocol/client
> option remote-host de-dc1-c1-pserver3
> option remote-subvolume /mnt/gluster/brick0/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-1
> type protocol/client
> option remote-host de-dc1-c1-pserver5
> option remote-subvolume /mnt/gluster/brick0/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-2
> type protocol/client
> option remote-host de-dc1-c1-pserver3
> option remote-subvolume /mnt/gluster/brick1/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-3
> type protocol/client
> option remote-host de-dc1-c1-pserver5
> option remote-subvolume /mnt/gluster/brick1/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-4
> type protocol/client
> option remote-host de-dc1-c1-pserver12
> option remote-subvolume /mnt/gluster/brick0/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-5
> type protocol/client
> option remote-host de-dc1-c1-pserver13
> option remote-subvolume /mnt/gluster/brick0/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-6
> type protocol/client
> option remote-host de-dc1-c1-pserver12
> option remote-subvolume /mnt/gluster/brick1/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-client-7
> type protocol/client
> option remote-host de-dc1-c1-pserver13
> option remote-subvolume /mnt/gluster/brick1/storage
> option transport-type rdma
> option ping-timeout 5
> end-volume
>
> volume storage0-replicate-0
> type cluster/replicate
> subvolumes storage0-client-0 storage0-client-1
> end-volume
>
> volume storage0-replicate-1
> type cluster/replicate
> subvolumes storage0-client-2 storage0-client-3
> end-volume
>
> volume storage0-replicate-2
> type cluster/replicate
> subvolumes storage0-client-4 storage0-client-5
> end-volume
>
> volume storage0-replicate-3
> type cluster/replicate
> subvolumes storage0-client-6 storage0-client-7
> end-volume
>
> volume storage0-dht
> type cluster/distribute
> subvolumes storage0-replicate-0 storage0-replicate-1
> storage0-replicate-2 storage0-replicate-3 end-volume
>
> volume storage0-write-behind
> type performance/write-behind
> subvolumes storage0-dht
> end-volume
>
> volume storage0-read-ahead
> type performance/read-ahead
> subvolumes storage0-write-behind
> end-volume
>
> volume storage0-io-cache
> type performance/io-cache
> option cache-size 4096MB
> subvolumes storage0-read-ahead
> end-volume
>
> volume storage0-quick-read
> type performance/quick-read
> option cache-size 4096MB
> subvolumes storage0-io-cache
> end-volume
>
> volume storage0-stat-prefetch
> type performance/stat-prefetch
> subvolumes storage0-quick-read
> end-volume
>
> volume storage0
> type debug/io-stats
> subvolumes storage0-stat-prefetch
> end-volume
>
>
> > -----Original Message-----
> > From: Pranith Kumar. Karampuri [mailto:pranithk at gluster.com]
> > Sent: Tuesday, May 17, 2011 7:16 AM
> > To: Martin Schenker
> > Cc: gluster-users at gluster.org
> > Subject: Re: [Gluster-users] Client and server file "view",
> > different results?! Client can't see the right file.
> >
> >
> > Martin,
> > Is this a distributed-replicate setup?. Could you
> > attach the vol-file of the client.
> >
> > Pranith
> > ----- Original Message -----
> > From: "Martin Schenker" <martin.schenker at profitbricks.com>
> > To: gluster-users at gluster.org
> > Sent: Monday, May 16, 2011 2:49:29 PM
> > Subject: [Gluster-users] Client and server file "view",
> > different results?! Client can't see the right file.
> >
> >
> > Client and server file "view", different results?! Client
> > can't see the right file.
> >
> > Hi all!
> >
> > Here we have another mismatch between the client "view" and
> > the server mounts:
> >
> > From the server site everything seems well, the 20G file is
> > visible and the attributes seem to match:
> >
> > 0 root at pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
> > /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8
> > f-8542864da6ef/hdd-images/
> >
> > # file:
> > mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f
> > -8542864da6ef/hdd-images//20964
> > trusted.afr.storage0-client-2=0x000000000000000000000000
> > trusted.afr.storage0-client-3=0x000000000000000000000000
> >
> > 0 root at pserver5:~ # find /mnt/gluster/ -name 20964 | xargs -i
> > ls -al {}
> > -rwxrwx--- 1 libvirt-qemu vcb 21474836480 May 13 11:21
> > /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8
> > f-8542864da6ef/hdd-images/20964
> >
> > But the client view shows 2!! files with 0 byte size!! And
> > these aren't any link files created by Gluster. ( with the T
> > on the end)
> >
> > 0 root at pserver5:~ # find /opt/profitbricks/storage/ -name
> > 20964 | xargs -i ls -al {}
> > -rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
> > /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> > 8542864da6ef/hdd-images/20964
> >
> > -rwxrwx--- 1 libvirt-qemu kvm 0 May 13 11:24
> > /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> > 8542864da6ef/hdd-images/20964
> >
> > I'm a bit stumped that we seem to have so many weird errors
> > cropping up. Any ideas? I've checked the ext4 filesystem on
> > all boxes, no real problems. We run a distributed cluster
> > with 4 servers offering 2 bricks each.
> >
> > Best, Martin
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Mohit Anchlia [ mailto:mohitanchlia at gmail.com ]
> > > Sent: Monday, May 16, 2011 2:24 AM
> > > To: Martin Schenker
> > > Cc: gluster-users at gluster.org
> > > Subject: Re: [Gluster-users] Brick pair file mismatch,
> > > self-heal problems?
> > >
> > >
> > > Try this to trigger self heal:
> > >
> > > find <gluster-mount> -noleaf -print0 -name <file name>|
> xargs --null
> > > stat >/dev/null
> > >
> > >
> > >
> > > On Sun, May 15, 2011 at 11:20 AM, Martin Schenker
> > > <martin.schenker at profitbricks.com> wrote:
> > > > Can someone enlighten me what's going on here? We have a
> > two peers,
> > > > the file 21313 is shown through the client mountpoint as
> > > "1Jan1970",
> > > > attribs on server pserver3 don't match but NO self-heal or
> > > repair can
> > > > be triggered through "ls -alR"?!?
> > > >
> > > > Checking the files through the server mounts show that
> > two versions
> > > > are on the system. But the wrong one (as with the
> > > "1Jan1970") seems to
> > > > be the preferred one by the client?!?
> > > >
> > > > Do I need to use setattr or what in order to get the client
> > > to see the
> > > > RIGHT version?!? This is not the ONLY file displaying this
> > > problematic
> > > > behaviour!
> > > >
> > > > Thanks for any feedback.
> > > >
> > > > Martin
> > > >
> > > > pserver5:
> > > >
> > > > 0 root at pserver5:~ # ls -al
> > > >
> > >
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > > > 4da6ef
> > > > /hdd-images
> > > >
> > > > -rwxrwx--- 1 libvirt-qemu vcb 483183820800 May 13 13:41 21313
> > > >
> > > > 0 root at pserver5:~ # getfattr -R -d -e hex -m "trusted.afr."
> > > >
> > >
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > > > 4da6ef
> > > > /hdd-images/21313
> > > > getfattr: Removing leading '/' from absolute path names
> > > > # file:
> > > >
> > > mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f
> > > -8542864da6ef/
> > > > hdd-images/21313
> > > > trusted.afr.storage0-client-2=0x000000000000000000000000
> > > > trusted.afr.storage0-client-3=0x000000000000000000000000
> > > >
> > > > 0 root at pserver5:~ # ls -alR
> > > >
> > >
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
> > > > a6ef/h
> > > > dd-images/21313
> > > > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
> > > >
> > > /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> > > 8542864da6ef/h
> > > > dd-images/21313
> > > >
> > > > pserver3:
> > > >
> > > > 0 root at pserver3:~ # ls -al
> > > >
> > >
> >
> /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-854286
> > > > 4da6ef
> > > > /hdd-images
> > > >
> > > > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970 21313
> > > >
> > > > 0 root at pserver3:~ # ls -alR
> > > >
> > >
> >
> /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864d
> > > > a6ef/h
> > > > dd-images/21313
> > > > -rwxrwx--- 1 libvirt-qemu kvm 483183820800 Jan 1 1970
> > > >
> > > /opt/profitbricks/storage/images/2078/ebb83b05-3a83-9d18-ad8f-
> > > 8542864da6ef/h
> > > > dd-images/21313
> > > >
> > > > 0 root at pserver3:~ # getfattr -R -d -e hex -m "trusted.afr."
> > > > /mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-
> > > > ad8f-8542864da6ef/hdd-images/21313
> > > > getfattr: Removing leading '/' from absolute path names
> > > > # file:
> > > >
> > >
> >
> mnt/gluster/brick1/storage/images/2078/ebb83b05-3a83-9d18-ad8f-8542864
> > > > da6ef/
> > > > hdd-images/21313
> > > > trusted.afr.storage0-client-2=0x000000000000000000000000
> > > > trusted.afr.storage0-client-3=0x0b0000090900000000000000
> > > <- mismatch,
> > > > should be targeted for self-heal/repair? Why is there a
> > > difference in the
> > > > views?
> > > >
> > > >
> > > > From the volfile:
> > > >
> > > > volume storage0-client-2
> > > > type protocol/client
> > > > option remote-host de-dc1-c1-pserver3
> > > > option remote-subvolume /mnt/gluster/brick1/storage
> > > > option transport-type rdma
> > > > option ping-timeout 5
> > > > end-volume
> > > >
> > > > volume storage0-client-3
> > > > type protocol/client
> > > > option remote-host de-dc1-c1-pserver5
> > > > option remote-subvolume /mnt/gluster/brick1/storage
> > > > option transport-type rdma
> > > > option ping-timeout 5
> > > > end-volume
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> > > >
> > >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >
>
More information about the Gluster-users
mailing list