[Gluster-devel] sub-directory geo-replication, snapshot features

[Shyam]

On 03/09/2016 09:05 AM, Kaushal M wrote:
> On Wed, Mar 9, 2016 at 6:24 PM, Shyam <srangana at redhat.com> wrote:
>> On 03/09/2016 02:25 AM, Pranith Kumar Karampuri wrote:
>>>
>>>
>>>
>>> On 03/09/2016 10:40 AM, Kaushal M wrote:
>>>>
>>>> On Tue, Mar 8, 2016 at 11:58 PM, Atin Mukherjee <amukherj at redhat.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> On 03/08/2016 07:32 PM, Pranith Kumar Karampuri wrote:
>>>>>>
>>>>>> hi,
>>>>>>            Late last week I sent a solution for how to achieve
>>>>>> subdirectory-mount support with access-controls
>>>>>>
>>>>>> (http://www.gluster.org/pipermail/gluster-devel/2016-March/048537.html).
>>>>>>
>>>>>> What follows here is a short description of how other features of
>>>>>> gluster volumes are implemented for sub-directories.
>>>>>>
>>>>>> Please note that the sub-directories are not allowed to be accessed by
>>>>>> normal mounts i.e. top-level volume mounts. All access to the
>>>>>> sub-directories goes only through sub-directory mounts.
>>>>>
>>>>> Is this acceptable? If I have a,b,c sub directories in the volume and if
>>>>> I mount the same volume in /mnt then do you mean to say I won't be able
>>>>> to access /mnt/a or /mnt/b and I can only access them using sub
>>>>> directory mounts? Or you are talking about some specific case here?
>>>>>>
>>>>>> 1) Geo-replication:
>>>>>> The direction in which we are going is to allow geo-replicating just
>>>>>> some sub-directories and not all of the volume based on options. When
>>>>>> these options are set, server xlators populate extra information in the
>>>>>> frames/xdata to write changelog for the fops coming from their
>>>>>> sub-directory mounts. changelog xlator on seeing this will only
>>>>>> geo-replicate the files/directories that are in the changelog. Thus
>>>>>> only
>>>>>> the sub-directories are geo-replicated. There is also a suggestion from
>>>>>> Vijay and Aravinda to have separate domains for operations inside
>>>>>> sub-directories for changelogs.
>>>>>>
>>>>>> 2) Sub-directory snapshots using lvm
>>>>>> Every time a sub-directory needs to be created, Our idea is that the
>>>>>> admin needs to execute subvolume creation command which creates a mount
>>>>>> to an empty snapshot at the given sub-directory name. All these
>>>>>> directories can be modified in parallel and we can take individual
>>>>>> snapshots of each of the directories. We will be providing a detailed
>>>>>> list commands to do the same once they are fleshed out. At the moment
>>>>>> these are the directions we are going to increase granularity from
>>>>>> volume to subdirectory for the main features.
>>>>
>>>> We use hardlinks to the `.glusterfs` directory on bricks. So wouldn't
>>>> having multiple filesystems inside a brick break the brick?
>>>
>>>
>>> You are right. I think we will have to do full separation, where it will
>>> be more like multiple tenants :-/.
>>
>>
>> Ah! I had a half baked response to this yesterday, asking if subdirs were
>> their own FS instances and LVM instances etc. looks like that was the case.
>>
>> Anyway, apart form the .glusterfs this idea merits some thinking, like a
>> virtual volume that chains up separate mounts on the local FS, consider it a
>> virtual namespace that links subvolumes together. So that we do not need to
>> create separate volumes and have that heavy weight process in place, at the
>> same time provide snapshots as well.
>
> If we are thinking about subdirs as virtual volumes and want to avoid
> a process per volume, why not just consider brick-multiplexing and
> proper full fledged volumes.
> We already have multiplexing on the agenda for 4.0, and it would seem
> easier to just leverage the facilities provided by that to get similar
> behavior, instead of attempting to add-in sub-directory support for
> all sort of operations/features.

+100, I simply agree. This (daemon multiplexing) and volume management 
scalability with GD2.0 (i.e ability to manage larger number of volumes 
per trusted pool) would help a good deal. (I rather subscribe to this 
view myself anyway). Also, things like network isolation etc. but those 
can be worked at a volume level from there on.

>
> One of the benefits of supporting sub directory mounts is that it
> would make it easier to create and manage multiple shares for many
> tenants. Trying to re-implement all volume operations for
> sub-directories only makes it harder for us and makes the solution
> really complex.

Yes agreed again.

The one potential (external?) issue is dm-thinp scalability in this 
scenario needs to be tested (i.e ability to carve out multiple LVs from 
a single thin-p, and their management).

>
>>
>>
>>>
>>>>
>>>> Also, I'd prefer if sub-directory mounts and sub-directory snapshots
>>>> remained separate, and not tied with each other. This mail gives the
>>>> feeling that they will be tied together.
>>>
>>>
>>> With the above point, I don't think it will be.
>>>
>>>>
>>>>>> Pranith
>>>>>> _______________________________________________
>>>>>> Gluster-devel mailing list
>>>>>> Gluster-devel at gluster.org
>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-devel mailing list
>>>>> Gluster-devel at gluster.org
>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-devel