[Gluster-users] vfs_fruit and extended attributes

Terry McGuire tmcguire at ualberta.ca
Thu Oct 19 21:30:27 UTC 2017 

It seems not many folks have much to say about this.

Here’s an update for anyone in a similar situation:

We narrowed down the issue to being with dispersed volumes.  When we created a replicated volume, vfs_fruit worked without issue.

Ultimately, what we were seeking was performance.  Even for our Windows and Linux users, things could be slow.  The good news is that we discovered an SMB config that helped immensely:  vfs_glusterfs.  In retrospect, I suppose it’s a bit obvious, but we somehow failed to discover this module.  Using it, instead of pointing Samba at the Gluster volume mounted with FUSE, we’re seeing 3x performance improvement with small files.

The bad news is that it looks like these 2 vfs modules can’t be used together.  I can’t make ‘em work anyway.

The good news about the bad news is that, with the vfs_glusterfs speedup, things are fast enough on our dispersed volume, even for Mac users, that we’re ok without vfs_fruit.  That said, it’d still be nice to get both vfs_fruit and vfs_glusterfs working at the same time.  And, of course, to get vfs_fruit to play nice with dispersed volumes.

Hope that helps anyone else out there dealing with something like this.

Regards,
Terry



> On Sep 22, 2017, at 15:38, Terry McGuire <tmcguire at ualberta.ca> wrote:
> 
> Hello Anoop.  Thanks for helping with this!
> 
>> On Sep 22, 2017, at 00:11, Anoop C S <anoopcs at autistici.org> wrote:
>> 
>> On Thu, 2017-09-21 at 10:35 -0600, Terry McGuire wrote:
>>> Hello list.  I’m attempting to improve how Samba shares directories on our Gluster volume to Mac
>>> users by using the vfs_fruit module.
>> 
>> What versions of GlusterFS and Samba have you installed? And which platform/distro are you using as
>> Samba server?
> 
> glusterfs-3.10.5-1.el7.x86_64
> samba-4.4.4-14.el7_3.x86_64
> 
> CentOS Linux release 7.3.1611 (Core)
> 
> This is a brand new service.  We’re running gluster clients, and samba, on the servers themselves.
> 
>> 
>> Please paste the output of `gluster volume info <VOLNAME>`.
> 
> (Apologies for the length of this…)
> 
> root at mfs-01 ~]#gluster volume info mfs1
> 
> Volume Name: mfs1
> Type: Distributed-Disperse
> Volume ID: 2fa02e5d-95b4-4aaa-b16c-5de90e0b11b2
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 6 x (8 + 4) = 72
> Transport-type: tcp
> Bricks:
> Brick1: mfs-b01:/mnt/gfs001/data
> Brick2: mfs-b01:/mnt/gfs002/data
> Brick3: mfs-b01:/mnt/gfs003/data
> Brick4: mfs-b02:/mnt/gfs019/data
> Brick5: mfs-b02:/mnt/gfs020/data
> Brick6: mfs-b02:/mnt/gfs021/data
> Brick7: mfs-b03:/mnt/gfs037/data
> Brick8: mfs-b03:/mnt/gfs038/data
> Brick9: mfs-b03:/mnt/gfs039/data
> Brick10: mfs-b04:/mnt/gfs055/data
> Brick11: mfs-b04:/mnt/gfs056/data
> Brick12: mfs-b04:/mnt/gfs057/data
> Brick13: mfs-b01:/mnt/gfs004/data
> Brick14: mfs-b01:/mnt/gfs005/data
> Brick15: mfs-b01:/mnt/gfs006/data
> Brick16: mfs-b02:/mnt/gfs022/data
> Brick17: mfs-b02:/mnt/gfs023/data
> Brick18: mfs-b02:/mnt/gfs024/data
> Brick19: mfs-b03:/mnt/gfs040/data
> Brick20: mfs-b03:/mnt/gfs041/data
> Brick21: mfs-b03:/mnt/gfs042/data
> Brick22: mfs-b04:/mnt/gfs058/data
> Brick23: mfs-b04:/mnt/gfs059/data
> Brick24: mfs-b04:/mnt/gfs060/data
> Brick25: mfs-b01:/mnt/gfs007/data
> Brick26: mfs-b01:/mnt/gfs008/data
> Brick27: mfs-b01:/mnt/gfs009/data
> Brick28: mfs-b02:/mnt/gfs025/data
> Brick29: mfs-b02:/mnt/gfs026/data
> Brick30: mfs-b02:/mnt/gfs027/data
> Brick31: mfs-b03:/mnt/gfs043/data
> Brick32: mfs-b03:/mnt/gfs044/data
> Brick33: mfs-b03:/mnt/gfs045/data
> Brick34: mfs-b04:/mnt/gfs061/data
> Brick35: mfs-b04:/mnt/gfs062/data
> Brick36: mfs-b04:/mnt/gfs063/data
> Brick37: mfs-b01:/mnt/gfs010/data
> Brick38: mfs-b01:/mnt/gfs011/data
> Brick39: mfs-b01:/mnt/gfs012/data
> Brick40: mfs-b02:/mnt/gfs028/data
> Brick41: mfs-b02:/mnt/gfs029/data
> Brick42: mfs-b02:/mnt/gfs030/data
> Brick43: mfs-b03:/mnt/gfs046/data
> Brick44: mfs-b03:/mnt/gfs047/data
> Brick45: mfs-b03:/mnt/gfs048/data
> Brick46: mfs-b04:/mnt/gfs064/data
> Brick47: mfs-b04:/mnt/gfs065/data
> Brick48: mfs-b04:/mnt/gfs066/data
> Brick49: mfs-b01:/mnt/gfs013/data
> Brick50: mfs-b01:/mnt/gfs014/data
> Brick51: mfs-b01:/mnt/gfs015/data
> Brick52: mfs-b02:/mnt/gfs031/data
> Brick53: mfs-b02:/mnt/gfs032/data
> Brick54: mfs-b02:/mnt/gfs033/data
> Brick55: mfs-b03:/mnt/gfs049/data
> Brick56: mfs-b03:/mnt/gfs050/data
> Brick57: mfs-b03:/mnt/gfs051/data
> Brick58: mfs-b04:/mnt/gfs067/data
> Brick59: mfs-b04:/mnt/gfs068/data
> Brick60: mfs-b04:/mnt/gfs069/data
> Brick61: mfs-b01:/mnt/gfs016/data
> Brick62: mfs-b01:/mnt/gfs017/data
> Brick63: mfs-b01:/mnt/gfs018/data
> Brick64: mfs-b02:/mnt/gfs034/data
> Brick65: mfs-b02:/mnt/gfs035/data
> Brick66: mfs-b02:/mnt/gfs036/data
> Brick67: mfs-b03:/mnt/gfs052/data
> Brick68: mfs-b03:/mnt/gfs053/data
> Brick69: mfs-b03:/mnt/gfs054/data
> Brick70: mfs-b04:/mnt/gfs070/data
> Brick71: mfs-b04:/mnt/gfs071/data
> Brick72: mfs-b04:/mnt/gfs072/data
> Options Reconfigured:
> features.quota-deem-statfs: on
> features.inode-quota: on
> features.quota: on
> nfs.disable: on
> transport.address-family: inet
> 
>> 
>>> This module does wonders for speeding listings and downloads of directories with large numbers of
>>> files in the Finder, but it kills uploads dead.  Finder gives an error:
>>> 
>>> The Finder can’t complete the operation because some data in “[filename]” can’t be read or
>>> written.
>>> (Error code -36)
>> 
>> Can you please check for any errors in brick logs under /var/log/glusterfs/bricks/ while you face
>> this Finder error?
> 
> There’s stuff in those logs, but nothing is added when the error occurs.
> 
>> 
>>> The man page for the module indicates that the vfs_streams_xattr module must also be loaded, like
>>> this:
>>> 
>>> vfs objects = fruit streams_xattr
>> 
>> Can you please share the output of `testparm -s`?
> 
> root at mfs-01 ~]#testparm -s
> Load smb config files from /etc/samba/smb.conf
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
> Processing section "[sa1]"
> Processing section "[nongluster]"
> Processing section "[share1]"
> Processing section "[share2]"
> Loaded services file OK.
> Server role: ROLE_DOMAIN_MEMBER
> 
> # Global parameters
> [global]
> 	realm = XXXX.UALBERTA.CA
> 	workgroup = STS
> 	log file = /var/log/samba/log.%m
> 	max log size = 50
> 	server min protocol = SMB2
> 	map to guest = Bad User
> 	security = ADS
> 	idmap config * : backend = tdb
> 	access based share enum = Yes
> 	force create mode = 0777
> 	force directory mode = 0777
> 	smb encrypt = desired
> 	vfs objects = fruit streams_xattr
> 
> 
> [sa1]
> 	path = /mfsmount/sa1
> 	read only = No
> 	valid users = @mfs-sa1 at XXXX.ualberta.ca
> 
> 
> [nongluster]
> 	path = /nongluster
> 	read only = No
> 	valid users = @mfs-sa1 at XXXX.ualberta.ca
> 
> 
> [share1]
> 	path = /mfsmount/share1
> 	read only = No
> 	valid users = @mfs-ga at XXXX.ualberta.ca
> 
> 
> [share2]
> 	path = /mfsmount/share2
> 	read only = No
> 	valid users = @mfs-gb at XXXX.ualberta.ca
> 
> 
>>> I’ve done this, and I’ve futzed with various combinations of options, but haven’t got writes
>>> working.
>> 
>> Is that a simple copy operation? Did you verify whether the file content itself is written
>> completely? We have experienced similar error while copying large files into GlusterFS volumes where
>> the real copy operation was completed successfully and the remaining extra extended attribute
>> setting failed which resulted in a Finder error. Just wanted to check whether it is the same case or
>> not?
> 
> Yup, just a simple drag and drop of a file into the share’s window.  No content is written, though a file of zero size is created.
> 
> Woah.  I’ve just discovered something.  While *files* cause the error, *folders* sometimes do not.  I can drag and drop certain folders of files and it works fine, but if I drag the files individually, it fails.  Other folders fail, but some of the files in the folder are successful.  Perhaps the Finder attaches problem metadata to files but only under certain circumstances.
> 
>> 
>>> This issue seems specific to Gluster, as I can share a directory that’s not in the Gluster volume
>>> and it works fine.  I’m guessing the problem has something to do with extended attributes, but,
>>> near as I can tell, the Gluster volume ought to do xattrs just fine, as its bricks are XFS.
>>> 
>>> Anyone have any experience with this?  Anyone have vfs_fruit working with Gluster?
>> 
>> Apart from this write issue, are you facing any other issues while accessing GlusterFS volumes from
>> Mac via Samba using vfs_fruit module? We would like to really fix any obvious errors which stops Mac
>> users working with GlusterFS volumes.
> 
> No other issues.  We really want to use this module, as it dramatically speeds up certain things, like opening folders with many items, or copying large numbers of small files.
> 
> Regards,
> Terry
>

More information about the Gluster-users mailing list