[Gluster-users] vfs_gluster broken

[Terry McGuire]

Hey Anoop.  Thanks for looking into this.  Responses inline:

> On Sep 11, 2018, at 23:42, Anoop C S <anoopcs at autistici.org> wrote:
> 
> On Tue, 2018-09-11 at 15:10 -0600, Terry McGuire wrote:
>> Hello list.  I had happily been sharing a Gluster volume with Samba using vfs_gluster, but it has
>> recently stopped working right.  I think it might have been after updating Samba from 4.6.2 to
>> 4.7.1 (as part of updating CentOS 7.4 to 7.5). The shares suffer a variety of weird issues,
>> including:
>> 
>> - sporadic connection refusals (credentials are accepted as valid, but volume is unavailable)
> 
> Does that mean after authentication share is not at all listed?

On a Mac, after the auth dialog disappears (suggesting the auth was valid), a dialog appears saying the volume is unavailable.  Can’t recall what the behaviour on Windows is, but it would be equivalent.  (This error doesn’t happen often, so I can’t quickly reproduce it, and I can’t even quite be sure it’s related to this problem, but I suspect it is.)

> 
>> - on Mac, when attempting to write a file: "The operation can’t be completed because an unexpected
>> error occurred (error code -50)."
> 
> How is this write performed? via Finder or via command-line?

That error appears when using the Finder.  It happens pretty much all the time, and is the clearest symptom of this problem.  Using the command-line, anything that attempts I/O with the share gives an “Invalid argument” error:

Mac:~ root# ls /Volumes/
Macintosh HD	module

Mac:~ root# ls -l /Volumes/
ls: module: Invalid argument
total 8
lrwxr-xr-x  1 root  wheel  1 24 Aug 15:48 Macintosh HD -> /

Mac:~ root# touch /Volumes/module/test
touch: /Volumes/module/test: Invalid argument
Mac:~ root# 

> 
>> - on Windows, sometimes when writing and sometimes when reading: "Z:\ is not accessible. The
>> parameter is incorrect"
> 
> How is this write performed? via Explorer or Powershell?

Explorer.
> 
>> -on Mac and Windows, the contents of the volume in Finder/Explorer windows sometimes disappears,
>> sometimes reappearing later, sometimes not.
>> - on Mac (and similar on Windows), volume icon sometimes disappears - maybe the volume unmounts,
>> but it's unclear.
> 
> Is this a clustered Samba setup i.e, with CTDB for high availability?

Not clustered.  Just a plain vanilla Samba.
> 
>> All these issues vanish when I switch to sharing the FUSE-mounted volume, but, of course, I lose
>> the advantages of vfs_gluster.
> 
> Can you please attach the output of `testparm -s` so as to look through how Samba is setup?

From our test server (“nomodule-nofruit” is currently the only well-behaved share):

root at mfsuat-01 ~]#testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
Processing section "[share1]"
Processing section "[share2]"
Processing section "[nomodule]"
Processing section "[nomodule-nofruit]"
Processing section "[module]"
Processing section "[IPC$]"
WARNING: No path in service IPC$ - making it unavailable!
NOTE: Service IPC$ is flagged unavailable.
Loaded services file OK.
idmap range not specified for domain '*'
ERROR: Invalid idmap range for domain *!

WARNING: You have some share names that are longer than 12 characters.
These may not be accessible to some older clients.
(Eg. Windows9x, WindowsMe, and smbclient prior to Samba 3.0.)
WARNING: some services use vfs_fruit, others don't. Mounting them in conjunction on OS X clients results in undefined behaviour.

Server role: ROLE_DOMAIN_MEMBER

# Global parameters
[global]
	log file = /var/log/samba/log.%m
	map to guest = Bad User
	max log size = 50
	realm = XXXX.AD.UALBERTA.CA
	security = ADS
	workgroup = STS
	glusterfs:volume = mfs1
	idmap config * : backend = tdb
	access based share enum = Yes
	force create mode = 0777
	force directory mode = 0777
	include = /mfsmount/admin/etc/mfs/smb_shares.conf
	kernel share modes = No
	read only = No
	smb encrypt = desired
	vfs objects = glusterfs


[share1]
	path = /share1
	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca


[share2]
	path = /share2
	valid users = @mfs-test-group at xxxx.ad.ualberta.ca


[nomodule]
	kernel share modes = Yes
	path = /mfsmount/share1
	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
	vfs objects = fruit streams_xattr


[nomodule-nofruit]
	kernel share modes = Yes
	path = /mfsmount/share1
	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca
	vfs objects = 


[module]
	path = /share1
	valid users = @mfs-sa1 at xxxx.ad.ualberta.ca


[IPC$]
	available = No
	vfs objects = 


> 
>> My gluster version initially was 3.10.12.  I’ve since updated to gluster 3.12.13, but the symptoms
>> are the same.
>> 
>> Does this sound familiar to anyone?
> 
> All mentioned symptoms point towards a disconnection. We need to find out the origin of this
> disconnection. What do we have in logs under /var/log/samba/? Any errors?

Actually, yes.  Large numbers of:

[2018/09/12 09:37:17.873711,  0] ../source3/modules/vfs_glusterfs.c:996(vfs_gluster_stat)
  glfs_stat(.) failed: Invalid argument

There appears to be some sort of connection remaining, as I can continue to cause these errors in the server log by attempting I/O with the share.

This seems like the most promising lead to find the root cause.  Hopefully you (or someone) can interpret what it means, and what I might do about it (besides not using vfs_gluster anymore).

Regards,
Terry