[Gluster-users] NFS Client issues with Gluster Server 3.6.9

Mark Selby mselby at unseelie.name
Thu Mar 10 02:41:22 UTC 2016


The problem turns out to be with ZFS

I use ZFS on linux as the underlying filesystem and as per the docs you 
need to set zfs set acltype=posixacl on all the pools you are exporting 
as NFS. I did not do this and it causes chaos.

Once I did this the NFS issues go away and all is good.

Thanks everyone!


On 3/8/16 6:42 AM, Soumya Koduri wrote:
> The log file didn't have any errors logged. Please check the NFS 
> client logs in '/var/log/messages' or using dmesg and brick logs as well.
>
> Probably strace or packet trace could help too. You could use the 
> below command to capture the pkt trace while running the I/Os on the 
> node where gluster-nfs server is running :
>
> $ tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22
>
> Check the file later to see what operation had failed (using filters: 
> nfs, glusterfs).
>
> Thanks,
> Soumya
>
>
> On 03/07/2016 09:41 PM, Mark Selby wrote:
>> Here are the logs that you requested
>>
>> Please let me know if I can send you anything else.
>>
>> I really appreciate you taking a look at this - thanks!
>>
>>
>> root at dc1strg001x /root 547# gluster vol info backups
>>
>> Volume Name: backups
>> Type: Replicate
>> Volume ID: 71a26ea6-632d-4a1d-8610-e782ce2a5100
>> Status: Started
>> Number of Bricks: 1 x 2 = 2
>> Transport-type: tcp
>> Bricks:
>> Brick1: dc1strg001x:/zfspool/glusterfs/backups/data
>> Brick2: dc1strg002x:/zfspool/glusterfs/backups/data
>> Options Reconfigured:
>> nfs.disable: off
>>
>>
>> root at dc1strg001x /var/log/glusterfs 551# cat nfs.log
>> [2016-03-07 16:03:14.257919] I [MSGID: 100030] [glusterfsd.c:2318:main]
>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6
>> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
>> /var/run/gluster/ad38be3bd1baece29e1b672e6659ae60.socket)
>> [2016-03-07 16:03:14.267862] I [MSGID: 101190]
>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 1
>> [2016-03-07 16:03:14.273283] I
>> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
>> Configured rpc.outstanding-rpc-limit with value 16
>> [2016-03-07 16:03:14.284154] W [MSGID: 112153]
>> [mount3.c:3929:mnt3svc_init] 0-nfs-mount: Exports auth has been 
>> disabled!
>> [2016-03-07 16:03:14.306163] I [rpc-drc.c:694:rpcsvc_drc_init]
>> 0-rpc-service: DRC is turned OFF
>> [2016-03-07 16:03:14.306216] I [MSGID: 112110] [nfs.c:1494:init] 0-nfs:
>> NFS service started
>> [2016-03-07 16:03:14.312901] I [MSGID: 101190]
>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
>> with index 2
>> [2016-03-07 16:03:14.314078] W [graph.c:357:_log_if_unknown_option]
>> 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>> [2016-03-07 16:03:14.314137] W [graph.c:357:_log_if_unknown_option]
>> 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>> [2016-03-07 16:03:14.314185] W [graph.c:357:_log_if_unknown_option]
>> 0-nfs-server: option 'transport-type' is not recognized
>> [2016-03-07 16:03:14.314270] I [MSGID: 114020] [client.c:2118:notify]
>> 0-backups-client-0: parent translators are ready, attempting connect on
>> transport
>> [2016-03-07 16:03:14.315341] I [MSGID: 114020] [client.c:2118:notify]
>> 0-backups-client-1: parent translators are ready, attempting connect on
>> transport
>> [2016-03-07 16:03:14.315923] I [rpc-clnt.c:1847:rpc_clnt_reconfig]
>> 0-backups-client-0: changing port to 49152 (from 0)
>> Final graph:
>> +------------------------------------------------------------------------------+ 
>>
>>
>>    1: volume backups-client-0
>>    2:     type protocol/client
>>    3:     option ping-timeout 42
>>    4:     option remote-host dc1strg001x
>>    5:     option remote-subvolume /zfspool/glusterfs/backups/data
>>    6:     option transport-type socket
>>    7:     option username 42fa7a62-1420-4169-ad00-53c3481dbe5b
>>    8:     option password b71b3c88-51e0-464c-8b09-14b661fdb4d3
>>    9:     option send-gids true
>>   10: end-volume
>>   11:
>>   12: volume backups-client-1
>>   13:     type protocol/client
>>   14:     option ping-timeout 42
>>   15:     option remote-host dc1strg002x
>> [2016-03-07 16:03:14.317412] I [MSGID: 114057]
>> [client-handshake.c:1437:select_server_supported_programs]
>> 0-backups-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
>> (330)
>>   16:     option remote-subvolume /zfspool/glusterfs/backups/data
>>   17:     option transport-type socket
>>   18:     option username 42fa7a62-1420-4169-ad00-53c3481dbe5b
>>   19:     option password b71b3c88-51e0-464c-8b09-14b661fdb4d3
>>   20:     option send-gids true
>>   21: end-volume
>>   22:
>>   23: volume backups-replicate-0
>>   24:     type cluster/replicate
>>   25:     subvolumes backups-client-0 backups-client-1
>>   26: end-volume
>>   27:
>>   28: volume backups-dht
>>   29:     type cluster/distribute
>>   30:     subvolumes backups-replicate-0
>>   31: end-volume
>>   32:
>>   33: volume backups-write-behind
>>   34:     type performance/write-behind
>>   35:     subvolumes backups-dht
>>   36: end-volume
>>   37:
>>   38: volume backups
>>   39:     type debug/io-stats
>>   40:     option latency-measurement off
>>   41:     option count-fop-hits off
>>   42:     subvolumes backups-write-behind
>>   43: end-volume
>>   44:
>>   45: volume nfs-server
>>   46:     type nfs/server
>>   47:     option rpc-auth.auth-glusterfs on
>>   48:     option rpc-auth.auth-unix on
>>   49:     option rpc-auth.auth-null on
>>   50:     option rpc-auth.ports.insecure on
>>   51:     option rpc-auth-allow-insecure on
>>   52:     option transport-type socket
>>   53:     option transport.socket.listen-port 2049
>>   54:     option nfs.dynamic-volumes on
>>   55:     option nfs.nlm on
>>   56:     option nfs.drc off
>>   57:     option rpc-auth.addr.backups.allow *
>>   58:     option nfs3.backups.volume-id
>> 71a26ea6-632d-4a1d-8610-e782ce2a5100
>>   59:     option nfs.backups.disable off
>>   60:     option nfs.logs.disable off
>>   61:     option nfs.users.disable off
>>   62:     subvolumes backups
>>   63: end-volume
>>   64:
>> +------------------------------------------------------------------------------+ 
>>
>>
>> [2016-03-07 16:03:14.318157] I [MSGID: 114046]
>> [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-0:
>> Connected to backups-client-0, attached to remote volume
>> '/zfspool/glusterfs/backups/data'.
>> [2016-03-07 16:03:14.318276] I [MSGID: 114047]
>> [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-0:
>> Server and Client lk-version numbers are not same, reopening the fds
>> [2016-03-07 16:03:14.318400] I [MSGID: 108005]
>> [afr-common.c:3841:afr_notify] 0-backups-replicate-0: Subvolume
>> 'backups-client-0' came back up; going online.
>> [2016-03-07 16:03:14.318470] I [MSGID: 114035]
>> [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-0:
>> Server lk version = 1
>> [2016-03-07 16:03:14.496642] I [rpc-clnt.c:1847:rpc_clnt_reconfig]
>> 0-backups-client-1: changing port to 49152 (from 0)
>> [2016-03-07 16:03:14.498394] I [MSGID: 114057]
>> [client-handshake.c:1437:select_server_supported_programs]
>> 0-backups-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
>> (330)
>> [2016-03-07 16:03:14.505580] I [MSGID: 114046]
>> [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-1:
>> Connected to backups-client-1, attached to remote volume
>> '/zfspool/glusterfs/backups/data'.
>> [2016-03-07 16:03:14.505627] I [MSGID: 114047]
>> [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-1:
>> Server and Client lk-version numbers are not same, reopening the fds
>> [2016-03-07 16:03:14.506210] I [MSGID: 114035]
>> [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-1:
>> Server lk version = 1
>> [2016-03-07 16:03:14.507836] I [MSGID: 108031]
>> [afr-common.c:1782:afr_local_discovery_cbk] 0-backups-replicate-0:
>> selecting local read_child backups-client-0
>>
>>
>>
>>
>> On 3/6/16 9:13 PM, Jiffin Tony Thottan wrote:
>>>
>>>
>>> On 05/03/16 07:12, Mark Selby wrote:
>>>> I am trying to use GlusterFS as a general purpose NFS file server. I
>>>> have tried using the FUSE client but the performance fall off vs NFS
>>>> is quite large
>>>>
>>>> Both the client and the server are Ubuntu 14.04.
>>>>
>>>> I am using Gluster 3.6.9 because of the FUSE performance issues that
>>>> have been reported with 3.7.8 (see
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1309462)
>>>>
>>>> I am having serious issues with a generic NFS client as shown by the
>>>> issues below. Basically most FOPs are giving me a Remote I/O error.
>>>>
>>>> I would not think I was 1st person to see these issues - but my
>>>> Google Fu is not working.
>>>>
>>>> Any and all help would be much appreciated
>>>>
>>>> BTW - These operation against a plain Linux NFS server work fine.
>>>>
>>>>
>>>> root at dc1strg001x /var/log 448# gluster volume status
>>>> Status of volume: backups
>>>> Gluster process Port    Online  Pid
>>>> ------------------------------------------------------------------------------ 
>>>>
>>>>
>>>> Brick dc1strg001x:/zfspool/glusterfs/backups/data 49152   Y 6462
>>>> Brick dc1strg002x:/zfspool/glusterfs/backups/data 49152   Y 6382
>>>> NFS Server on localhost 2049    Y       6619
>>>> Self-heal Daemon on localhost N/A     Y       6626
>>>> NFS Server on dc1strg002x 2049    Y       6502
>>>> Self-heal Daemon on dc1strg002x N/A     Y       6509
>>>>
>>>>
>>>> root at vc1test001 /root 735# mount -o vers=3 -t nfs
>>>> dc1strg001x:/backups /mnt/backups_nfs
>>>>
>>>> root at vc1test001 /mnt/backups_nfs 737# dd if=/dev/zero of=testfile
>>>> bs=16k count=16384
>>>> 16384+0 records in
>>>> 16384+0 records out
>>>> 268435456 bytes (268 MB) copied, 2.46237 s, 109 MB/s
>>>>
>>>> root at vc1test001 /mnt/backups_nfs 738# rm testfile
>>>>
>>>> root at vc1test001 /mnt/backups_nfs 739# dd if=/dev/zero of=testfile
>>>> bs=16k count=16384
>>>> dd: failed to open ~testfile~: Remote I/O error
>>>>
>>>> root at vc1test001 /var/tmp 743# rsync -av testfile /mnt/backups_nfs/
>>>> sending incremental file list
>>>> testfile
>>>> rsync: mkstemp "/mnt/backups_nfs/.testfile.bzg47C" failed: Remote I/O
>>>> error (121)
>>>>
>>>> sent 1,074,004,056 bytes  received 121 bytes 165,231,411.85 bytes/sec
>>>> total size is 1,073,741,824  speedup is 1.00
>>>> rsync error: some files/attrs were not transferred (see previous
>>>> errors) (code 23) at main.c(1183) [sender=3.1.0]
>>>>
>>>
>>> Can you please provide the volume configuration(gluster vol info )
>>> and log file for nfs server which u mounted (/var/log/glusterfs)
>>>
>>> -- 
>>> Jiffin
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users



More information about the Gluster-users mailing list