[Gluster-users] Question on replicated volumes with bricks on the same server

[Vijay Bellur]

On 02/12/2014 11:10 PM, Antonio Messina wrote:
> On Wed, Feb 12, 2014 at 11:21 AM, Vijay Bellur <vbellur at redhat.com> wrote:
>> On 02/11/2014 03:21 PM, Antonio Messina wrote:
>>> So, my guess is that if I create a "replica N" replicated+distributed
>>> volumes using the bricks:
>>>
>>>     gluster-1:/srv/gluster
>>>     ...
>>>     gluster-[N*M]:/srv/gluster
>>>
>>> gluster internally creates a distributed volumes made of the following
>>> replicated "volumes":
>>>
>>>     replicated volume 1: gluster-[1..N]:/srv/gluster
>>>     replicated volume 2: gluster-[N+1..2N]:/srv/gluster
>>>     ...
>>>     replicated volume M: gluster-[N*(M-1)+1..N*M]:/srv/gluster
>>>
>>> Is that correct or there is a more complex algorithm involved?
>>>
>>
>> The interpretation is correct. The way replica sets are chosen is related to
>> the order in which bricks are defined at the time of volume creation.
>
> Thank you for your clarification.
>
> However, I am also trying to play with the vol files. It's not clear to me yet
> *when* and *how* these files are read, so I don't know if I am doing
> something very stupid... however, here is my question:
>
> I create a distributed+replicated volume with 4 bricks with:
>
>      gluster volume create default replica 2
> gluster-data00{1,2,3,4}:/srv/gluster/brick force
>
> mount the filesystem, created multiple files on it and I have verified
> that the replicas are stored as we discussed before.
>
> The `default-fuse.vol` file looks like:
>
>      [...]
>      volume default-replicate-0
>          type cluster/replicate
>          subvolumes default-client-0 default-client-1
>      end-volume
>
>      volume default-replicate-1
>          type cluster/replicate
>          subvolumes default-client-2 default-client-3
>      end-volume
>
>      volume default-dht
>          type cluster/distribute
>          subvolumes default-replicate-0 default-replicate-1
>      end-volume
>      [...]
>
> Then I have stopped the volume and modified the vol file, in order to
> swap two servers (default-client-3 and default-client-1):


Modifying a volume file manually is not a recommended step and the 
mechanism employed here is quite disruptive to GlusterFS and your data :).

A lot of GlusterFS translators store state information in the form of 
extended attributes. Some of the keys used in these extended attributes 
are a function of the subvolume names and with this manual swap, it can 
cause self-heals to happen in a random fashion. Re-using a brick in a 
different form is not normally recommended and a fair degree of state 
cleanup has to be done before it can be used.


>
>      [...]
>      volume default-replicate-0
>          type cluster/replicate
>          subvolumes default-client-0 default-client-3
>      end-volume
>
>      volume default-replicate-1
>          type cluster/replicate
>          subvolumes default-client-2 default-client-1
>      end-volume
>
>      volume default-dht
>          type cluster/distribute
>          subvolumes default-replicate-0 default-replicate-1
>      end-volume
>      [...]
>
> re-started and re-mounted the volume.
>
> I would expect that new files are replicated now on client-0 and
> client-3 (called gluster-data00{1,4} in my cluster), but if I create a
> file they are again replicated on client-0 and client-1 instead.

If you happen to mount the client in the trusted storage pool, the 
volume file that gets used upon a mount is trusted-<volname>-fuse.vol.

>
> Am I missing some basic step?

If you are testing functionality by altering the volume file manually, 
it would be advisable to change something more simpler - like altering a 
translator option to see the change being operational rather than 
changing the volume topology manually. volume topology modifications are 
recommended to be performed from the gluster CLI.

Regards,
Vijay