[Gluster-users] vfs_fruit and extended attributes

Terry McGuire tmcguire at ualberta.ca
Fri Sep 22 21:38:08 UTC 2017

Hello Anoop.  Thanks for helping with this!

> On Sep 22, 2017, at 00:11, Anoop C S <anoopcs at autistici.org> wrote:
> On Thu, 2017-09-21 at 10:35 -0600, Terry McGuire wrote:
>> Hello list.  I’m attempting to improve how Samba shares directories on our Gluster volume to Mac
>> users by using the vfs_fruit module.
> What versions of GlusterFS and Samba have you installed? And which platform/distro are you using as
> Samba server?


CentOS Linux release 7.3.1611 (Core)

This is a brand new service.  We’re running gluster clients, and samba, on the servers themselves.

> Please paste the output of `gluster volume info <VOLNAME>`.

(Apologies for the length of this…)

root at mfs-01 ~]#gluster volume info mfs1
Volume Name: mfs1
Type: Distributed-Disperse
Volume ID: 2fa02e5d-95b4-4aaa-b16c-5de90e0b11b2
Status: Started
Snapshot Count: 0
Number of Bricks: 6 x (8 + 4) = 72
Transport-type: tcp
Brick1: mfs-b01:/mnt/gfs001/data
Brick2: mfs-b01:/mnt/gfs002/data
Brick3: mfs-b01:/mnt/gfs003/data
Brick4: mfs-b02:/mnt/gfs019/data
Brick5: mfs-b02:/mnt/gfs020/data
Brick6: mfs-b02:/mnt/gfs021/data
Brick7: mfs-b03:/mnt/gfs037/data
Brick8: mfs-b03:/mnt/gfs038/data
Brick9: mfs-b03:/mnt/gfs039/data
Brick10: mfs-b04:/mnt/gfs055/data
Brick11: mfs-b04:/mnt/gfs056/data
Brick12: mfs-b04:/mnt/gfs057/data
Brick13: mfs-b01:/mnt/gfs004/data
Brick14: mfs-b01:/mnt/gfs005/data
Brick15: mfs-b01:/mnt/gfs006/data
Brick16: mfs-b02:/mnt/gfs022/data
Brick17: mfs-b02:/mnt/gfs023/data
Brick18: mfs-b02:/mnt/gfs024/data
Brick19: mfs-b03:/mnt/gfs040/data
Brick20: mfs-b03:/mnt/gfs041/data
Brick21: mfs-b03:/mnt/gfs042/data
Brick22: mfs-b04:/mnt/gfs058/data
Brick23: mfs-b04:/mnt/gfs059/data
Brick24: mfs-b04:/mnt/gfs060/data
Brick25: mfs-b01:/mnt/gfs007/data
Brick26: mfs-b01:/mnt/gfs008/data
Brick27: mfs-b01:/mnt/gfs009/data
Brick28: mfs-b02:/mnt/gfs025/data
Brick29: mfs-b02:/mnt/gfs026/data
Brick30: mfs-b02:/mnt/gfs027/data
Brick31: mfs-b03:/mnt/gfs043/data
Brick32: mfs-b03:/mnt/gfs044/data
Brick33: mfs-b03:/mnt/gfs045/data
Brick34: mfs-b04:/mnt/gfs061/data
Brick35: mfs-b04:/mnt/gfs062/data
Brick36: mfs-b04:/mnt/gfs063/data
Brick37: mfs-b01:/mnt/gfs010/data
Brick38: mfs-b01:/mnt/gfs011/data
Brick39: mfs-b01:/mnt/gfs012/data
Brick40: mfs-b02:/mnt/gfs028/data
Brick41: mfs-b02:/mnt/gfs029/data
Brick42: mfs-b02:/mnt/gfs030/data
Brick43: mfs-b03:/mnt/gfs046/data
Brick44: mfs-b03:/mnt/gfs047/data
Brick45: mfs-b03:/mnt/gfs048/data
Brick46: mfs-b04:/mnt/gfs064/data
Brick47: mfs-b04:/mnt/gfs065/data
Brick48: mfs-b04:/mnt/gfs066/data
Brick49: mfs-b01:/mnt/gfs013/data
Brick50: mfs-b01:/mnt/gfs014/data
Brick51: mfs-b01:/mnt/gfs015/data
Brick52: mfs-b02:/mnt/gfs031/data
Brick53: mfs-b02:/mnt/gfs032/data
Brick54: mfs-b02:/mnt/gfs033/data
Brick55: mfs-b03:/mnt/gfs049/data
Brick56: mfs-b03:/mnt/gfs050/data
Brick57: mfs-b03:/mnt/gfs051/data
Brick58: mfs-b04:/mnt/gfs067/data
Brick59: mfs-b04:/mnt/gfs068/data
Brick60: mfs-b04:/mnt/gfs069/data
Brick61: mfs-b01:/mnt/gfs016/data
Brick62: mfs-b01:/mnt/gfs017/data
Brick63: mfs-b01:/mnt/gfs018/data
Brick64: mfs-b02:/mnt/gfs034/data
Brick65: mfs-b02:/mnt/gfs035/data
Brick66: mfs-b02:/mnt/gfs036/data
Brick67: mfs-b03:/mnt/gfs052/data
Brick68: mfs-b03:/mnt/gfs053/data
Brick69: mfs-b03:/mnt/gfs054/data
Brick70: mfs-b04:/mnt/gfs070/data
Brick71: mfs-b04:/mnt/gfs071/data
Brick72: mfs-b04:/mnt/gfs072/data
Options Reconfigured:
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
nfs.disable: on
transport.address-family: inet

>> This module does wonders for speeding listings and downloads of directories with large numbers of
>> files in the Finder, but it kills uploads dead.  Finder gives an error:
>> The Finder can’t complete the operation because some data in “[filename]” can’t be read or
>> written.
>> (Error code -36)
> Can you please check for any errors in brick logs under /var/log/glusterfs/bricks/ while you face
> this Finder error?

There’s stuff in those logs, but nothing is added when the error occurs.

>> The man page for the module indicates that the vfs_streams_xattr module must also be loaded, like
>> this:
>> vfs objects = fruit streams_xattr
> Can you please share the output of `testparm -s`?

root at mfs-01 ~]#testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[sa1]"
Processing section "[nongluster]"
Processing section "[share1]"
Processing section "[share2]"
Loaded services file OK.

# Global parameters
	workgroup = STS
	log file = /var/log/samba/log.%m
	max log size = 50
	server min protocol = SMB2
	map to guest = Bad User
	security = ADS
	idmap config * : backend = tdb
	access based share enum = Yes
	force create mode = 0777
	force directory mode = 0777
	smb encrypt = desired
	vfs objects = fruit streams_xattr

	path = /mfsmount/sa1
	read only = No
	valid users = @mfs-sa1 at XXXX.ualberta.ca

	path = /nongluster
	read only = No
	valid users = @mfs-sa1 at XXXX.ualberta.ca

	path = /mfsmount/share1
	read only = No
	valid users = @mfs-ga at XXXX.ualberta.ca

	path = /mfsmount/share2
	read only = No
	valid users = @mfs-gb at XXXX.ualberta.ca

>> I’ve done this, and I’ve futzed with various combinations of options, but haven’t got writes
>> working.
> Is that a simple copy operation? Did you verify whether the file content itself is written
> completely? We have experienced similar error while copying large files into GlusterFS volumes where
> the real copy operation was completed successfully and the remaining extra extended attribute
> setting failed which resulted in a Finder error. Just wanted to check whether it is the same case or
> not?

Yup, just a simple drag and drop of a file into the share’s window.  No content is written, though a file of zero size is created.

Woah.  I’ve just discovered something.  While *files* cause the error, *folders* sometimes do not.  I can drag and drop certain folders of files and it works fine, but if I drag the files individually, it fails.  Other folders fail, but some of the files in the folder are successful.  Perhaps the Finder attaches problem metadata to files but only under certain circumstances.

>> This issue seems specific to Gluster, as I can share a directory that’s not in the Gluster volume
>> and it works fine.  I’m guessing the problem has something to do with extended attributes, but,
>> near as I can tell, the Gluster volume ought to do xattrs just fine, as its bricks are XFS.
>> Anyone have any experience with this?  Anyone have vfs_fruit working with Gluster?
> Apart from this write issue, are you facing any other issues while accessing GlusterFS volumes from
> Mac via Samba using vfs_fruit module? We would like to really fix any obvious errors which stops Mac
> users working with GlusterFS volumes.

No other issues.  We really want to use this module, as it dramatically speeds up certain things, like opening folders with many items, or copying large numbers of small files.


More information about the Gluster-users mailing list