[Gluster-users] Getting error in nfs.log file

Fri Apr 6 18:35:30 UTC 2012

Hi,

On April 1st, I started seeing the following errors in my nfs.log file.  I didn't notice until today, when most of my VM's under Citrix XenServer made their drives read-only.  The VM disk images are NFS mounted via gluster nfs.

I didn't notice any issues until the remount as read-only today.  Three of my VM's failed to boot after this, just sitting at initramfs.  All VM's that re-mounted read-only needs an fsck.

I'm running Gluster 3.2.5 under Ubuntu 10.04.4 (64 bit), and everything is happening via NFS.

Gerald

NFS.LOG:

Given volfile:
+------------------------------------------------------------------------------+
  1: volume gl-raid1-client-0
  2:     type protocol/client
  3:     option remote-host 192.168.20.1
  4:     option remote-subvolume /export/gluster/RAID-1
  5:     option transport-type tcp
  6: end-volume
  7:
  8: volume gl-raid1-replicate-0
  9:     type cluster/replicate
 10:     subvolumes gl-raid1-client-0
 11: end-volume
 12:
 13: volume gl-raid1-write-behind
 14:     type performance/write-behind
 15:     option cache-size 4mb
 16:     subvolumes gl-raid1-replicate-0
 17: end-volume
 18:
 19: volume gl-raid1-read-ahead
 20:     type performance/read-ahead
 21:     subvolumes gl-raid1-write-behind
 22: end-volume
 23:
 24: volume gl-raid1-io-cache
 25:     type performance/io-cache
 26:     option cache-size 268435456
 27:     subvolumes gl-raid1-read-ahead
 28: end-volume
 29:
 30: volume gl-raid1-quick-read
 31:     type performance/quick-read
 32:     option cache-size 268435456
 33:     subvolumes gl-raid1-io-cache
 34: end-volume
 35:
 36: volume gl-raid1
 37:     type debug/io-stats
 38:     option latency-measurement off
 39:     option count-fop-hits off
 40:     subvolumes gl-raid1-quick-read
 41: end-volume
 42:
 43: volume gl-raid6-client-0
 44:     type protocol/client
 45:     option remote-host 192.168.10.1
 46:     option remote-subvolume /export/gluster/RAID-6
 47:     option transport-type tcp
 48: end-volume
 49:
 50: volume gl-raid6-replicate-0
 51:     type cluster/replicate
 52:     subvolumes gl-raid6-client-0
 53: end-volume
 54:
 55: volume gl-raid6-write-behind
 56:     type performance/write-behind
 57:     option cache-size 4mb
 58:     subvolumes gl-raid6-replicate-0
 59: end-volume
 60:
 61: volume gl-raid6-read-ahead
 62:     type performance/read-ahead
 63:     subvolumes gl-raid6-write-behind
 64: end-volume
 65:
 66: volume gl-raid6-io-cache
 67:     type performance/io-cache
 68:     option cache-size 268435456
 69:     subvolumes gl-raid6-read-ahead
 70: end-volume
 71:
 72: volume gl-raid6-quick-read
 73:     type performance/quick-read
 74:     option cache-size 268435456
 75:     subvolumes gl-raid6-io-cache
 76: end-volume
 77:
 78: volume gl-raid6
 79:     type debug/io-stats
 80:     option latency-measurement off
 81:     option count-fop-hits off
 82:     subvolumes gl-raid6-quick-read
 83: end-volume
 84:
 85: volume nfs-server
 86:     type nfs/server
 87:     option nfs.dynamic-volumes on
 88:     option rpc-auth.addr.gl-raid6.allow *
 89:     option nfs3.gl-raid6.volume-id 149df6f8-ffbf-44cc-90f0-ee894d6626ba
 90:     option rpc-auth.addr.gl-raid1.allow *
 91:     option nfs3.gl-raid1.volume-id f8c1b8c4-8511-40dc-a4eb-f338522b9662
 92:     option nfs.port 2049
 93:     option nfs3.gl-raid6.trusted-sync on
 94:     option nfs3.gl-raid1.trusted-sync on
 95:     subvolumes gl-raid6 gl-raid1
 96: end-volume

+------------------------------------------------------------------------------+
[2012-03-12 22:28:53.706616] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-gl-raid6-client-0: changing port to 24009 (from 0)
[2012-03-12 22:28:53.706650] I [rpc-clnt.c:1536:rpc_clnt_reconfig] 0-gl-raid1-client-0: changing port to 24010 (from 0)
[2012-03-12 22:28:57.706828] I [client-handshake.c:1090:select_server_supported_programs] 0-gl-raid6-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2012-03-12 22:28:57.706993] I [client-handshake.c:1090:select_server_supported_programs] 0-gl-raid1-client-0: Using Program GlusterFS 3.2.5, Num (1298437), Version (310)
[2012-03-12 22:28:57.708254] I [client-handshake.c:913:client_setvolume_cbk] 0-gl-raid6-client-0: Connected to 192.168.10.1:24009, attached to remote volume '/export/gluster/R
[2012-03-12 22:28:57.708296] I [afr-common.c:3141:afr_notify] 0-gl-raid6-replicate-0: Subvolume 'gl-raid6-client-0' came back up; going online.
[2012-03-12 22:28:57.708447] I [client-handshake.c:913:client_setvolume_cbk] 0-gl-raid1-client-0: Connected to 192.168.20.1:24010, attached to remote volume '/export/gluster/R
[2012-03-12 22:28:57.708464] I [afr-common.c:3141:afr_notify] 0-gl-raid1-replicate-0: Subvolume 'gl-raid1-client-0' came back up; going online.
[2012-03-12 22:28:57.708568] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-gl-raid6-replicate-0: added root inode
[2012-03-12 22:28:57.708635] I [afr-common.c:1520:afr_set_root_inode_on_first_lookup] 0-gl-raid1-replicate-0: added root inode
[2012-04-01 13:34:05.870006] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.715548] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:07.715720] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.715742] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:07.715806] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.715825] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:07.715886] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.715904] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:07.715963] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.715981] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:07.716040] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.716057] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:07.716117] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:07.716135] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:19.146413] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:19.146463] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:19.146775] E [event.c:703:event_select_on_epoll] 0-epoll: index not found for fd=-1 (idx_hint=0)
[2012-04-01 13:34:42.297040] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:42.297081] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:42.297172] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:42.297188] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:42.297246] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:42.297261] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:42.297310] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:42.297325] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:42.297373] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:42.297387] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:42.297438] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message
[2012-04-01 13:34:42.297451] E [nfs3.c:522:nfs3svc_submit_reply] 0-nfs-nfsv3: Reply submission failed
[2012-04-01 13:34:42.297499] E [rpcsvc.c:1710:nfs_rpcsvc_submit_generic] 0-nfsrpc: Failed to submit message