[Gluster-devel] fstest for fuse mount gluster
Raghavendra Gowdappa
rgowdapp at redhat.com
Mon Apr 24 07:13:45 UTC 2017
----- Original Message -----
> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
> Cc: "qingwei wei" <tchengwee at gmail.com>, "Gluster Devel" <gluster-devel at gluster.org>, "Nithya Balachandran"
> <nbalacha at redhat.com>
> Sent: Monday, April 24, 2017 11:45:43 AM
> Subject: Re: [Gluster-devel] fstest for fuse mount gluster
>
> On Mon, Apr 24, 2017 at 11:28 AM, Raghavendra Gowdappa <rgowdapp at redhat.com>
> wrote:
>
> > Its a known issue in DHT. Since DHT relies on some bits (S+T) in the stat
> > for identifying that a file is being migrated by a rebalance process, it
> > strips them off. But the problem is if the same bits are being used by
> > application (as in this case). The fix (if we can come up with one) might
> > require some time for validation as impact can be large. Possible solutions
> > are:
> >
> > 1. Pranith pointed out that rebalance process always holds an inodelk
> > during file migration. Probably we can leverage this information and decide
> > whether to strip the flags or not.
> > 2. There is also a linkto xattr stored on the data-file during migration.
> > Instead of unconditionally striping Phase1(2) flags, lookup/stat codepaths
> > can check for the presence of this xattr too and strip the flags only if it
> > is present. Note that with this solution application won't see the (S+T)
> > bits in stat during rebalance.
> >
>
> Innocent question. If we have a linkto-xattr and the lock, do we need
> anything more to detect that migration is in progress? I am wondering if we
> can remove the reliance on S+T bits. Worst case we will have to store the
> info on the xattr.
We can use locks, but the lock information is not persistent. If a brick goes down and comes back up that information is lost. Also, if we are using locks as metadata for identifying migration, it shouldn't be an explicit call as all write fops check for migration status. Instead, bricks can return lock information in various fops like writev, setattr, setxattr etc. Note that the change of rebalance process acquiring a lock on a file before migration was introduced much later. So, that information was not always available. We cannot just rely on linkto xattrs as there are two phases in migration:
1. phase 1 - migration is in progress (clients have to replay write operations on dst subvol too after they do on src)
2. phase 2 - migration is complete (clients have to switch to dst subvol)
So, rebalance need two metadata - (S+T) and linkto xattr.
Also, not all write fops (writev, setattr, setxattr etc) don't return xattrs like linkto etc. I assume the initial designers wanted to use as less xattrs as possible.
However, using xattr to store migration metadata looks simple and doesn't affect anything visible to application. But of course, there is an added getxattr overhead on brick during _all_ write (writev, setxattr, setattr, fxattrop etc) operations.
>
>
> >
> > regards,
> > Raghavendra.
> >
> > ----- Original Message -----
> > > From: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > > To: "qingwei wei" <tchengwee at gmail.com>
> > > Cc: "Gluster Devel" <gluster-devel at gluster.org>, "Raghavendra Gowdappa"
> > <rgowdapp at redhat.com>, "Nithya Balachandran"
> > > <nbalacha at redhat.com>
> > > Sent: Monday, April 24, 2017 11:03:29 AM
> > > Subject: Re: [Gluster-devel] fstest for fuse mount gluster
> > >
> > > This is a bug in dht it seems like. It is stripping PHASE1 flags
> > > unconditionally.
> > >
> > > (gdb)
> > > 1212 DHT_STRIP_PHASE1_FLAGS (&local->stbuf);
> > > (gdb) p local->stbuf.ia_prot
> > > $18 = {
> > > suid = 1 '\001',
> > > sgid = 1 '\001', <-------
> > > sticky = 1 '\001', <-------
> > > .........
> > > }
> > > (gdb) n
> > > 1213 dht_set_fixed_dir_stat (&local->postparent);
> > > (gdb) p local->stbuf.ia_prot
> > > $19 = {
> > > suid = 1 '\001',
> > > sgid = 0 '\000', <-------
> > > sticky = 0 '\000', <-------
> > > ...
> > >
> > > This is leading to 7777-->4777
> > >
> > > Will update bug with same info
> > >
> > >
> > > On Thu, Apr 20, 2017 at 8:58 PM, qingwei wei <tchengwee at gmail.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > Posted this in gluster-user mailing list but got no response so far,
> > so i
> > > > post in gluster-devel.
> > > >
> > > > I found this test suite (https://github.com/Hnasar/pjdfstest) for me
> > to
> > > > test fuse mount gluster and there is some reported issue from the
> > test. One
> > > > of the error is as follow.
> > > >
> > > > When i chmod 7777 to a file in fuse mounted gluster volume. the return
> > > > stat value for the file is not 7777 instead of 4777.
> > > >
> > > > root at ubuntu16d:/mnt/g310mp# touch test
> > > > root at ubuntu16d:/mnt/g310mp# chmod 7777 test
> > > > root at ubuntu16d:/mnt/g310mp# stat test
> > > > File: 'test'
> > > > Size: 0 Blocks: 0 IO Block: 131072 regular
> > empty
> > > > file
> > > > Device: 29h/41d Inode: 9618589997017543511 Links: 1
> > > > Access: (4777/-rwsrwxrwx) Uid: ( 0/ root) Gid: ( 0/
> > root)
> > > > Access: 2017-11-30 14:21:23.374871207 +0800
> > > > Modify: 2017-11-30 14:21:16.974871000 +0800
> > > > Change: 2017-11-30 14:21:23.374871207 +0800
> > > > Birth: -
> > > >
> > > > Performing this operation in normal ext4 system produce correct result.
> > > >
> > > > root at ubuntu16d:/mnt/g310mp# touch ~/testfile
> > > > root at ubuntu16d:/mnt/g310mp# chmod 7777 ~/testfile
> > > > root at ubuntu16d:/mnt/g310mp# stat ~/testfile
> > > > File: '/home/ubuntu/testfile'
> > > > Size: 0 Blocks: 0 IO Block: 4096 regular
> > empty
> > > > file
> > > > Device: fc00h/64512d Inode: 662649 Links: 1
> > > > Access: (7777/-rwsrwsrwt) Uid: ( 0/ root) Gid: ( 0/
> > root)
> > > > Access: 2017-11-30 14:23:00.518867795 +0800
> > > > Modify: 2017-11-30 14:23:00.518867795 +0800
> > > > Change: 2017-11-30 14:23:08.742867507 +0800
> > > > Birth: -
> > > >
> > > > Besides 7777, 3777 also an issue. The stat return is 0777.
> > > >
> > > > My OS is Ubuntu 16.04 and my gluster version is 3.10.1 and the it is a
> > > > simple volume with default parameter.
> > > >
> > > > root at ubuntu16d:/mnt/g310mp# gluster volume info
> > > >
> > > > Volume Name: g310
> > > > Type: Distribute
> > > > Volume ID: 114666c6-4884-436a-81a8-2deb3c0923ba
> > > > Status: Started
> > > > Snapshot Count: 0
> > > > Number of Bricks: 1
> > > > Transport-type: tcp
> > > > Bricks:
> > > > Brick1: 192.168.36.130:/mnt/g310brick
> > > > Options Reconfigured:
> > > > transport.address-family: inet
> > > > nfs.disable: on
> > > >
> > > > for me, i seldom use those mode (3777 & 7777) but i cannot say for sure
> > > > for others. So is this something i should be concerned about?
> > > >
> > > > Cw
> > > >
> > > > _______________________________________________
> > > > Gluster-devel mailing list
> > > > Gluster-devel at gluster.org
> > > > http://lists.gluster.org/mailman/listinfo/gluster-devel
> > > >
> > >
> > >
> > >
> > > --
> > > Pranith
> > >
> >
>
>
>
> --
> Pranith
>
More information about the Gluster-devel
mailing list