[Gluster-devel] dht mkdir preop check, afr and (non-)readable afr subvols
Raghavendra G
raghavendra at gluster.com
Mon Jun 6 08:54:17 UTC 2016
On Wed, Jun 1, 2016 at 12:50 PM, Xavier Hernandez <xhernandez at datalab.es>
wrote:
> Hi,
>
> On 01/06/16 08:53, Raghavendra Gowdappa wrote:
>
>>
>>
>> ----- Original Message -----
>>
>>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>>> To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>, "Raghavendra G" <
>>> raghavendra at gluster.com>
>>> Cc: "Gluster Devel" <gluster-devel at gluster.org>
>>> Sent: Wednesday, June 1, 2016 11:57:12 AM
>>> Subject: Re: [Gluster-devel] dht mkdir preop check, afr and
>>> (non-)readable afr subvols
>>>
>>> Oops, you are right. For entry operations the current version of the
>>> parent directory is not checked, just to avoid this problem.
>>>
>>> This means that mkdir will be sent to all alive subvolumes. However it
>>> still selects the group of answers that have a minimum quorum equal or
>>> greater than #bricks - redundancy. So it should be still valid.
>>>
>>
>> What if the quorum is met on "bad" subvolumes? and mkdir was successful
>> on bad subvolumes? Do we consider mkdir as successful? If yes, even EC
>> suffers from the problem described in bz
>> https://bugzilla.redhat.com/show_bug.cgi?id=1341429.
>>
>
> I don't understand the real problem. How a subvolume of EC could be in bad
> state from the point of view of DHT ?
>
> If you use xattrs to configure something in the parent directories, you
> should have needed to use setxattr or xattrop to do that. These operations
> do consider good/bad bricks because they touch inode metadata. This will
> only succeed if enough (quorum) bricks have successfully processed it. If
> quorum is met but for an error answer, an error will be reported to DHT and
> the majority of bricks will be left in the old state (these should be
> considered the good subvolumes). If some brick has succeeded, it will be
> considered bad and will be healed. If no quorum is met (even for an error
> answer), EIO will be returned and the state of the directory should be
> considered unknown/damaged.
>
Yes. Ideally, dht should use a getxattr for the layout xattr. But, for
performance reasons we thought of overloading mkdir by introducing
pre-operations (done by bricks). With plain dht it is a simple comparison
of xattrs passed as argument and xattrs stored on disk. But, I failed to
include afr and EC in the picture. Hence this issue. How difficult for EC
and AFR to bring this kind of check? Is it even possible for afr and EC to
implement this kind of pre-op checks with reasonable complexity?
> If a later mkdir checks this value in storage/posix and succeeds in enough
> bricks, it necessarily means that is has succeeded in good bricks, because
> there cannot be enough bricks with the bad xattr value.
>
> Note that quorum is always > #bricks/2 so we cannot have a quorum with
> good and bad bricks at the same time.
>
> Xavi
>
>
>
>>
>>> Xavi
>>>
>>> On 01/06/16 06:51, Pranith Kumar Karampuri wrote:
>>>
>>>> Xavi,
>>>> But if we keep winding only to good subvolumes, there is a case
>>>> where bad subvolumes will never catch up right? i.e. if we keep creating
>>>> files in same directory and everytime self-heal completes there are more
>>>> entries mounts would have created on the good subvolumes alone. I think
>>>> I must have missed this in the reviews if this is the current behavior.
>>>> It was not in the earlier releases. Right?
>>>>
>>>> Pranith
>>>>
>>>> On Tue, May 31, 2016 at 2:17 PM, Raghavendra G <raghavendra at gluster.com
>>>> <mailto:raghavendra at gluster.com>> wrote:
>>>>
>>>>
>>>>
>>>> On Tue, May 31, 2016 at 12:37 PM, Xavier Hernandez
>>>> <xhernandez at datalab.es <mailto:xhernandez at datalab.es>> wrote:
>>>>
>>>> Hi,
>>>>
>>>> On 31/05/16 07:05, Raghavendra Gowdappa wrote:
>>>>
>>>> +gluster-devel, +Xavi
>>>>
>>>> Hi all,
>>>>
>>>> The context is [1], where bricks do pre-operation checks
>>>> before doing a fop and proceed with fop only if pre-op check
>>>> is successful.
>>>>
>>>> @Xavi,
>>>>
>>>> We need your inputs on behavior of EC subvolumes as well.
>>>>
>>>>
>>>> If I understand correctly, EC shouldn't have any problems here.
>>>>
>>>> EC sends the mkdir request to all subvolumes that are currently
>>>> considered "good" and tries to combine the answers. Answers that
>>>> match in return code, errno (if necessary) and xdata contents
>>>> (except for some special xattrs that are ignored for combination
>>>> purposes), are grouped.
>>>>
>>>> Then it takes the group with more members/answers. If that group
>>>> has a minimum size of #bricks - redundancy, it is considered the
>>>> good answer. Otherwise EIO is returned because bricks are in an
>>>> inconsistent state.
>>>>
>>>> If there's any answer in another group, it's considered bad and
>>>> gets marked so that self-heal will repair it using the good
>>>> information from the majority of bricks.
>>>>
>>>> xdata is combined and returned even if return code is -1.
>>>>
>>>> Is that enough to cover the needed behavior ?
>>>>
>>>>
>>>> Thanks Xavi. That's sufficient for the feature in question. One of
>>>> the main cases I was interested in was what would be the behaviour
>>>> if mkdir succeeds on "bad" subvolume and fails on "good" subvolume.
>>>> Since you never wind mkdir to "bad" subvolume(s), this situation
>>>> never arises.
>>>>
>>>>
>>>>
>>>>
>>>> Xavi
>>>>
>>>>
>>>>
>>>> [1] http://review.gluster.org/13885
>>>>
>>>> regards,
>>>> Raghavendra
>>>>
>>>> ----- Original Message -----
>>>>
>>>> From: "Pranith Kumar Karampuri" <pkarampu at redhat.com
>>>> <mailto:pkarampu at redhat.com>>
>>>> To: "Raghavendra Gowdappa" <rgowdapp at redhat.com
>>>> <mailto:rgowdapp at redhat.com>>
>>>> Cc: "team-quine-afr" <team-quine-afr at redhat.com
>>>> <mailto:team-quine-afr at redhat.com>>, "rhs-zteam"
>>>> <rhs-zteam at redhat.com <mailto:rhs-zteam at redhat.com>>
>>>> Sent: Tuesday, May 31, 2016 10:22:49 AM
>>>> Subject: Re: dht mkdir preop check, afr and
>>>> (non-)readable afr subvols
>>>>
>>>> I think you should start a discussion on gluster-devel
>>>> so that Xavi gets a
>>>> chance to respond on the mails as well.
>>>>
>>>> On Tue, May 31, 2016 at 10:21 AM, Raghavendra Gowdappa
>>>> <rgowdapp at redhat.com <mailto:rgowdapp at redhat.com>>
>>>> wrote:
>>>>
>>>> Also note that we've plans to extend this pre-op
>>>> check to all dentry
>>>> operations which also depend parent layout. So, the
>>>> discussion need to
>>>> cover all dentry operations like:
>>>>
>>>> 1. create
>>>> 2. mkdir
>>>> 3. rmdir
>>>> 4. mknod
>>>> 5. symlink
>>>> 6. unlink
>>>> 7. rename
>>>>
>>>> We also plan to have similar checks in lock codepath
>>>> for directories too
>>>> (planning to use hashed-subvolume as lock-subvolume
>>>> for directories). So,
>>>> more fops :)
>>>> 8. lk (posix locks)
>>>> 9. inodelk
>>>> 10. entrylk
>>>>
>>>> regards,
>>>> Raghavendra
>>>>
>>>> ----- Original Message -----
>>>>
>>>> From: "Raghavendra Gowdappa"
>>>> <rgowdapp at redhat.com <mailto:
>>>> rgowdapp at redhat.com>>
>>>> To: "team-quine-afr" <team-quine-afr at redhat.com
>>>> <mailto:team-quine-afr at redhat.com>>
>>>> Cc: "rhs-zteam" <rhs-zteam at redhat.com
>>>> <mailto:rhs-zteam at redhat.com>>
>>>> Sent: Tuesday, May 31, 2016 10:15:04 AM
>>>> Subject: dht mkdir preop check, afr and
>>>> (non-)readable afr subvols
>>>>
>>>> Hi all,
>>>>
>>>> I have some queries related to the behavior of
>>>> afr_mkdir with respect to
>>>> readable subvols.
>>>>
>>>> 1. While winding mkdir to subvols does afr check
>>>> whether the subvolume is
>>>> good/readable? Or does it wind to all subvols
>>>> irrespective of whether a
>>>> subvol is good/bad? In the latter case, what if
>>>> a. mkdir succeeds on non-readable subvolume
>>>> b. fails on readable subvolume
>>>>
>>>> What is the result reported to higher layers
>>>> in the above scenario? If
>>>> mkdir is failed, is it cleaned up on
>>>> non-readable subvolume where it
>>>> failed?
>>>>
>>>> I am interested in this case as dht-preop check
>>>> relies on layout xattrs
>>>>
>>>> and I
>>>>
>>>> assume layout xattrs in particular (and all
>>>> xattrs in general) are
>>>> guaranteed to be correct only on a readable
>>>> subvolume of afr. So, in
>>>>
>>>> essence
>>>>
>>>> we shouldn't be winding down mkdir on
>>>> non-readable subvols as whatever
>>>>
>>>> the
>>>>
>>>> decision brick makes as part of pre-op check is
>>>> inherently flawed.
>>>>
>>>> regards,
>>>> Raghavendra
>>>>
>>>> --
>>>> Pranith
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Raghavendra G
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Pranith
>>>>
>>>
>>> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
--
Raghavendra G
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160606/5b139fda/attachment-0001.html>
More information about the Gluster-devel
mailing list