[Gluster-users] vfs_gluster broken

[Anoop C S]

On Wed, 2018-09-12 at 10:37 -0600, Terry McGuire wrote:
> Hey Anoop.  Thanks for looking into this.  Responses inline:
> 
> > On Sep 11, 2018, at 23:42, Anoop C S <anoopcs at autistici.org> wrote:
> > 
> > On Tue, 2018-09-11 at 15:10 -0600, Terry McGuire wrote:
> > > Hello list.  I had happily been sharing a Gluster volume with Samba using vfs_gluster, but it
> > > has
> > > recently stopped working right.  I think it might have been after updating Samba from 4.6.2 to
> > > 4.7.1 (as part of updating CentOS 7.4 to 7.5). The shares suffer a variety of weird issues,
> > > including:
> > > 
> > > - sporadic connection refusals (credentials are accepted as valid, but volume is unavailable)
> > 
> > Does that mean after authentication share is not at all listed?
> 
> On a Mac, after the auth dialog disappears (suggesting the auth was valid), a dialog appears
> saying the volume is unavailable.  Can’t recall what the behaviour on Windows is, but it would be
> equivalent.  (This error doesn’t happen often, so I can’t quickly reproduce it, and I can’t even
> quite be sure it’s related to this problem, but I suspect it is.)
> 
> > 
> > > - on Mac, when attempting to write a file: "The operation can’t be completed because an
> > > unexpected
> > > error occurred (error code -50)."
> > 
> > How is this write performed? via Finder or via command-line?
> 
> That error appears when using the Finder.  It happens pretty much all the time, and is the
> clearest symptom of this problem.  Using the command-line, anything that attempts I/O with the
> share gives an “Invalid argument” error:
> 
> Mac:~ root# ls /Volumes/
> Macintosh HD	module
> 
> Mac:~ root# ls -l /Volumes/
> ls: module: Invalid argument
> total 8
> lrwxr-xr-x  1 root  wheel  1 24 Aug 15:48 Macintosh HD -> /
> 
> Mac:~ root# touch /Volumes/module/test
> touch: /Volumes/module/test: Invalid argument
> Mac:~ root# 
> 
> > 
> > > - on Windows, sometimes when writing and sometimes when reading: "Z:\ is not accessible. The
> > > parameter is incorrect"
> > 
> > How is this write performed? via Explorer or Powershell?
> 
> Explorer.
> > 
> > > -on Mac and Windows, the contents of the volume in Finder/Explorer windows sometimes
> > > disappears,
> > > sometimes reappearing later, sometimes not.
> > > - on Mac (and similar on Windows), volume icon sometimes disappears - maybe the volume
> > > unmounts,
> > > but it's unclear.
> > 
> > Is this a clustered Samba setup i.e, with CTDB for high availability?
> 
> Not clustered.  Just a plain vanilla Samba.
> > 
> > > All these issues vanish when I switch to sharing the FUSE-mounted volume, but, of course, I
> > > lose
> > > the advantages of vfs_gluster.
> > 
> > Can you please attach the output of `testparm -s` so as to look through how Samba is setup?
> 
> From our test server (“nomodule-nofruit” is currently the only well-behaved share):
> 
> root at mfsuat-01 ~]#testparm -s
> Load smb config files from /etc/samba/smb.conf
> rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
> Processing section "[share1]"
> Processing section "[share2]"
> Processing section "[nomodule]"
> Processing section "[nomodule-nofruit]"
> Processing section "[module]"
> Processing section "[IPC$]"
> WARNING: No path in service IPC$ - making it unavailable!
> NOTE: Service IPC$ is flagged unavailable.
> Loaded services file OK.
> idmap range not specified for domain '*'
> ERROR: Invalid idmap range for domain *!
> 
> WARNING: You have some share names that are longer than 12 characters.
> These may not be accessible to some older clients.
> (Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.)
> WARNING: some services use vfs_fruit, others don't. Mounting them in conjunction on OS X clients
> results in undefined behaviour.
> 
> Server role: ROLE_DOMAIN_MEMBER
> 
> # Global parameters
> [global]
> 	log file = /var/log/samba/log.%m
> 	map to guest = Bad User
> 	max log size = 50
> 	realm = XXXX.AD.UALBERTA.CA
> 	security = ADS
> 	workgroup = STS
> 	glusterfs:volume = mfs1
> 	idmap config * : backend = tdb
> 	access based share enum = Yes
> 	force create mode = 0777
> 	force directory mode = 0777
> 	include = /mfsmount/admin/etc/mfs/smb_shares.conf
> 	kernel share modes = No
> 	read only = No
> 	smb encrypt = desired
> 	vfs objects = glusterfs
> 
> 
> [share1]
> 	path = /share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> 
> 
> [share2]
> 	path = /share2
> 	valid users = @mfs-test-group at xxxx.ad.ualberta.ca
> 
> 
> [nomodule]
> 	kernel share modes = Yes
> 	path = /mfsmount/share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> 	vfs objects = fruit streams_xattr
> 
> 
> [nomodule-nofruit]
> 	kernel share modes = Yes
> 	path = /mfsmount/share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> 	vfs objects = 
> 
> 
> [module]
> 	path = /share1
> 	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
> 
> 
> [IPC$]
> 	available = No
> 	vfs objects = 
> 
> 
> > 
> > > My gluster version initially was 3.10.12.  I’ve since updated to gluster 3.12.13, but the
> > > symptoms
> > > are the same.
> > > 

Did you mean 3.10.12 or 3.12.12?(your initial GlusterFS version)?

If GlusterFS was at v3.10.12, are you sure that you were experiencing issues with samba v4.7.1?

> > > Does this sound familiar to anyone?
> > 
> > All mentioned symptoms point towards a disconnection. We need to find out the origin of this
> > disconnection. What do we have in logs under /var/log/samba/? Any errors?
> 
> Actually, yes.  Large numbers of:
> 
> [2018/09/12 09:37:17.873711,  0] ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat)
>   glfs_stat(.) failed: Invalid argument

Might be related to https://bugzilla.samba.org/show_bug.cgi?id=13585.

Can you try installing a version of GlusterFS < 3.12.13 and see if it works normally?

> There appears to be some sort of connection remaining, as I can continue to cause these errors in
> the server log by attempting I/O with the share.
> 
> This seems like the most promising lead to find the root cause.  Hopefully you (or someone) can
> interpret what it means, and what I might do about it (besides not using vfs_gluster anymore).
> 
> Regards,
> Terry
>