[Gluster-users] xattr and ZFS

[Vijay Bellur]

On 01/14/2014 05:34 AM, Franco Broi wrote:
> On Mon, 2014-01-13 at 23:48 +0530, Vijay Bellur wrote:
>> On 01/13/2014 01:54 PM, Franco Broi wrote:
>>> Hi all
>>>
>>> I have about 400TB of storage running dht on ZFS and want to change the
>>> way the extended attributes are stored, assuming it's safe to do this in
>>> ZFS version 0.6.2? I discovered by experimentation that if I remove the
>>> attributes on the server, Gluster will remake them when a file is first
>>> accessed. So the question is, would it be safe to simply delete all the
>>> attributes from all files and just let Gluster remake them?
>>>
>>
>> GlusterFS does maintain state in extended attributes and manually
>> altering this state information has to be done very carefully.Removing
>> extended attributes for directories is not recommended as this can
>> potentially result in gfid split brains.
>>
>> As far as files go in a distributed cluster (with no replication), the
>> gfid attribute can be removed and it gets re-created on access. What
>> other attributes do you intend removing?
>
> I thought I might need to remove trusted.glusterfs.dht as well.
>
> Which attributes get access when reading a directory? Does a stat on a
> file need to read attributes?

Reading a directory does result in extended attributes used by md-cache 
translator and others being read.A lookup operation that can precede a 
stat does read extended attributes.

>
> I just did a test where I made 1000 directories over 8 bricks. I removed
> trusted.gfid and trusted.glusterfs.dht and gluster remade them when I
> did an ls -l from the client.

A lookup from a client can heal these extended attributes. In the past 
(primarily in 3.2.x versions), we had problems when directories missed 
extended attributes and multiple clients tried to access & set these 
attributes, we would end up with different gfids for the same 
directories in different bricks. We have added resiliency to prevent 
such occurrences beyond 3.2 but it is best to attempt this operation 
when your production storage system is not heavily loaded.

Is this change to make use of SA based extended attributes in zfs?

-Vijay