[Gluster-users] [Nfs-ganesha-support] [SOLVED] volume start: gv01: failed: Quorum not met. Volume operation not allowed.

[Daniel Gryniewicz]

Thanks, Tom.  Good to know.

Daniel

On 05/22/2018 01:43 AM, TomK wrote:
> This list has been deprecated. Please subscribe to the new support list 
> at lists.nfs-ganesha.org.
> Hey All,
> 
> Appears I solved this one and NFS mounts now work on all my clients.  No 
> issues since fixing it a few hours back.
> 
> RESOLUTION
> 
> Auditd is to blame for the trouble.  Noticed this in the logs on 2 of 
> the 3 NFS servers (nfs01, nfs02, nfs03):
> 
> type=AVC msg=audit(1526965320.850:4094): avc:  denied  { write } for 
> pid=8714 comm="ganesha.nfsd" name="nfs_0" dev="dm-0" ino=201547689 
> scontext=system_u:system_r:ganesha_t:s0 
> tcontext=system_u:object_r:krb5_host_rcache_t:s0 tclass=file
> type=SYSCALL msg=audit(1526965320.850:4094): arch=c000003e syscall=2 
> success=no exit=-13 a0=7f23b0003150 a1=2 a2=180 a3=2 items=0 ppid=1 
> pid=8714 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 
> fsgid=0 tty=(none) ses=4294967295 comm="ganesha.nfsd" 
> exe="/usr/bin/ganesha.nfsd" subj=system_u:system_r:ganesha_t:s0 key=(null)
> type=PROCTITLE msg=audit(1526965320.850:4094): 
> proctitle=2F7573722F62696E2F67616E657368612E6E667364002D4C002F7661722F6C6F672F67616E657368612F67616E657368612E6C6F67002D66002F6574632F67616E657368612F67616E657368612E636F6E66002D4E004E49565F4556454E54 
> 
> type=AVC msg=audit(1526965320.850:4095): avc:  denied  { unlink } for 
> pid=8714 comm="ganesha.nfsd" name="nfs_0" dev="dm-0" ino=201547689 
> scontext=system_u:system_r:ganesha_t:s0 
> tcontext=system_u:object_r:krb5_host_rcache_t:s0 tclass=file
> type=SYSCALL msg=audit(1526965320.850:4095): arch=c000003e syscall=87 
> success=no exit=-13 a0=7f23b0004100 a1=7f23b0000050 a2=7f23b0004100 a3=5 
> items=0 ppid=1 pid=8714 auid=4294967295 uid=0 gid=0 euid=0 suid=0 
> fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 
> comm="ganesha.nfsd" exe="/usr/bin/ganesha.nfsd" 
> subj=system_u:system_r:ganesha_t:s0 key=(null)
> type=PROCTITLE msg=audit(1526965320.850:4095): 
> proctitle=2F7573722F62696E2F67616E657368612E6E667364002D4C002F7661722F6C6F672F67616E657368612F67616E657368612E6C6F67002D66002F6574632F67616E657368612F67616E657368612E636F6E66002D4E004E49565F4556454E54 
> 
> 
> Fix was to adjust the SELinux rules using audit2allow.
> 
> All the errors below including the one in the link below, were due to that.
> 
> Turns out that when ever it worked, it hit the only working server in 
> the system, nfs03.  Whenever it didn't work, it was hitting the non 
> working servers.  So sometimes it worked, and other times it didn't.  It 
> looked like it was to do with Haproxy / Keepalived as well since I 
> couldn't mount using the VIP but could using the host.  But that wasn't 
> the case either.
> 
> I've also added the third brick to the Gluster FS, nfs03, trying to see 
> if the backend FS was to blame since Gluster FS recommends 3 bricks 
> minimum for replication, but that had no effect.
> 
> In case anyone runs into this, I've added notes here as well:
> 
> http://microdevsys.com/wp/kernel-nfs-nfs4_discover_server_trunking-unhandled-error-512-exiting-with-error-eio-and-mount-hangs/ 
> 
> 
> http://microdevsys.com/wp/nfs-reply-xid-3844308326-reply-err-20-auth-rejected-credentials-client-should-begin-new-session/ 
> 
> 
> The errors thrown included:
> 
> NFS reply xid 3844308326 reply ERR 20: Auth Rejected Credentials (client 
> should begin new session)
> 
> kernel: NFS: nfs4_discover_server_trunking unhandled error -512. Exiting 
> with error EIO and mount hangs
> 
> + the kernel exception below.
>