[Gluster-users] NFS Client issues with Gluster Server 3.6.9
Gmail
b.s.mikhael at gmail.com
Thu Mar 10 03:35:12 UTC 2016
ZFS by default stores the extended attributes in a hidden directory instead of extending the file inode size like what XFS do!
There is a problem in ZFS on Linux implementation which the function responsible for deleting the files, it deletes only the files and forgets to delete the hidden directory which stores the extended attributes, and there seem no interest from ZFS on Linux community to fix this bug, it’s filed since 2013.
There is an option to change this to write the extended attributes to the inode instead of a hidden directory.
# zfs set xattr=sa <POOL_NAME>
— Bishoy
> On Mar 9, 2016, at 6:41 PM, Mark Selby <mselby at unseelie.name> wrote:
>
> The problem turns out to be with ZFS
>
> I use ZFS on linux as the underlying filesystem and as per the docs you need to set zfs set acltype=posixacl on all the pools you are exporting as NFS. I did not do this and it causes chaos.
>
> Once I did this the NFS issues go away and all is good.
>
> Thanks everyone!
>
>
> On 3/8/16 6:42 AM, Soumya Koduri wrote:
>> The log file didn't have any errors logged. Please check the NFS client logs in '/var/log/messages' or using dmesg and brick logs as well.
>>
>> Probably strace or packet trace could help too. You could use the below command to capture the pkt trace while running the I/Os on the node where gluster-nfs server is running :
>>
>> $ tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22
>>
>> Check the file later to see what operation had failed (using filters: nfs, glusterfs).
>>
>> Thanks,
>> Soumya
>>
>>
>> On 03/07/2016 09:41 PM, Mark Selby wrote:
>>> Here are the logs that you requested
>>>
>>> Please let me know if I can send you anything else.
>>>
>>> I really appreciate you taking a look at this - thanks!
>>>
>>>
>>> root at dc1strg001x /root 547# gluster vol info backups
>>>
>>> Volume Name: backups
>>> Type: Replicate
>>> Volume ID: 71a26ea6-632d-4a1d-8610-e782ce2a5100
>>> Status: Started
>>> Number of Bricks: 1 x 2 = 2
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: dc1strg001x:/zfspool/glusterfs/backups/data
>>> Brick2: dc1strg002x:/zfspool/glusterfs/backups/data
>>> Options Reconfigured:
>>> nfs.disable: off
>>>
>>>
>>> root at dc1strg001x /var/log/glusterfs 551# cat nfs.log
>>> [2016-03-07 16:03:14.257919] I [MSGID: 100030] [glusterfsd.c:2318:main]
>>> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.6
>>> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/nfs -p
>>> /var/lib/glusterd/nfs/run/nfs.pid -l /var/log/glusterfs/nfs.log -S
>>> /var/run/gluster/ad38be3bd1baece29e1b672e6659ae60.socket)
>>> [2016-03-07 16:03:14.267862] I [MSGID: 101190]
>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
>>> with index 1
>>> [2016-03-07 16:03:14.273283] I
>>> [rpcsvc.c:2215:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service:
>>> Configured rpc.outstanding-rpc-limit with value 16
>>> [2016-03-07 16:03:14.284154] W [MSGID: 112153]
>>> [mount3.c:3929:mnt3svc_init] 0-nfs-mount: Exports auth has been disabled!
>>> [2016-03-07 16:03:14.306163] I [rpc-drc.c:694:rpcsvc_drc_init]
>>> 0-rpc-service: DRC is turned OFF
>>> [2016-03-07 16:03:14.306216] I [MSGID: 112110] [nfs.c:1494:init] 0-nfs:
>>> NFS service started
>>> [2016-03-07 16:03:14.312901] I [MSGID: 101190]
>>> [event-epoll.c:632:event_dispatch_epoll_worker] 0-epoll: Started thread
>>> with index 2
>>> [2016-03-07 16:03:14.314078] W [graph.c:357:_log_if_unknown_option]
>>> 0-nfs-server: option 'rpc-auth.auth-glusterfs' is not recognized
>>> [2016-03-07 16:03:14.314137] W [graph.c:357:_log_if_unknown_option]
>>> 0-nfs-server: option 'rpc-auth-allow-insecure' is not recognized
>>> [2016-03-07 16:03:14.314185] W [graph.c:357:_log_if_unknown_option]
>>> 0-nfs-server: option 'transport-type' is not recognized
>>> [2016-03-07 16:03:14.314270] I [MSGID: 114020] [client.c:2118:notify]
>>> 0-backups-client-0: parent translators are ready, attempting connect on
>>> transport
>>> [2016-03-07 16:03:14.315341] I [MSGID: 114020] [client.c:2118:notify]
>>> 0-backups-client-1: parent translators are ready, attempting connect on
>>> transport
>>> [2016-03-07 16:03:14.315923] I [rpc-clnt.c:1847:rpc_clnt_reconfig]
>>> 0-backups-client-0: changing port to 49152 (from 0)
>>> Final graph:
>>> +------------------------------------------------------------------------------+
>>>
>>> 1: volume backups-client-0
>>> 2: type protocol/client
>>> 3: option ping-timeout 42
>>> 4: option remote-host dc1strg001x
>>> 5: option remote-subvolume /zfspool/glusterfs/backups/data
>>> 6: option transport-type socket
>>> 7: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b
>>> 8: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3
>>> 9: option send-gids true
>>> 10: end-volume
>>> 11:
>>> 12: volume backups-client-1
>>> 13: type protocol/client
>>> 14: option ping-timeout 42
>>> 15: option remote-host dc1strg002x
>>> [2016-03-07 16:03:14.317412] I [MSGID: 114057]
>>> [client-handshake.c:1437:select_server_supported_programs]
>>> 0-backups-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
>>> (330)
>>> 16: option remote-subvolume /zfspool/glusterfs/backups/data
>>> 17: option transport-type socket
>>> 18: option username 42fa7a62-1420-4169-ad00-53c3481dbe5b
>>> 19: option password b71b3c88-51e0-464c-8b09-14b661fdb4d3
>>> 20: option send-gids true
>>> 21: end-volume
>>> 22:
>>> 23: volume backups-replicate-0
>>> 24: type cluster/replicate
>>> 25: subvolumes backups-client-0 backups-client-1
>>> 26: end-volume
>>> 27:
>>> 28: volume backups-dht
>>> 29: type cluster/distribute
>>> 30: subvolumes backups-replicate-0
>>> 31: end-volume
>>> 32:
>>> 33: volume backups-write-behind
>>> 34: type performance/write-behind
>>> 35: subvolumes backups-dht
>>> 36: end-volume
>>> 37:
>>> 38: volume backups
>>> 39: type debug/io-stats
>>> 40: option latency-measurement off
>>> 41: option count-fop-hits off
>>> 42: subvolumes backups-write-behind
>>> 43: end-volume
>>> 44:
>>> 45: volume nfs-server
>>> 46: type nfs/server
>>> 47: option rpc-auth.auth-glusterfs on
>>> 48: option rpc-auth.auth-unix on
>>> 49: option rpc-auth.auth-null on
>>> 50: option rpc-auth.ports.insecure on
>>> 51: option rpc-auth-allow-insecure on
>>> 52: option transport-type socket
>>> 53: option transport.socket.listen-port 2049
>>> 54: option nfs.dynamic-volumes on
>>> 55: option nfs.nlm on
>>> 56: option nfs.drc off
>>> 57: option rpc-auth.addr.backups.allow *
>>> 58: option nfs3.backups.volume-id
>>> 71a26ea6-632d-4a1d-8610-e782ce2a5100
>>> 59: option nfs.backups.disable off
>>> 60: option nfs.logs.disable off
>>> 61: option nfs.users.disable off
>>> 62: subvolumes backups
>>> 63: end-volume
>>> 64:
>>> +------------------------------------------------------------------------------+
>>>
>>> [2016-03-07 16:03:14.318157] I [MSGID: 114046]
>>> [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-0:
>>> Connected to backups-client-0, attached to remote volume
>>> '/zfspool/glusterfs/backups/data'.
>>> [2016-03-07 16:03:14.318276] I [MSGID: 114047]
>>> [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-0:
>>> Server and Client lk-version numbers are not same, reopening the fds
>>> [2016-03-07 16:03:14.318400] I [MSGID: 108005]
>>> [afr-common.c:3841:afr_notify] 0-backups-replicate-0: Subvolume
>>> 'backups-client-0' came back up; going online.
>>> [2016-03-07 16:03:14.318470] I [MSGID: 114035]
>>> [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-0:
>>> Server lk version = 1
>>> [2016-03-07 16:03:14.496642] I [rpc-clnt.c:1847:rpc_clnt_reconfig]
>>> 0-backups-client-1: changing port to 49152 (from 0)
>>> [2016-03-07 16:03:14.498394] I [MSGID: 114057]
>>> [client-handshake.c:1437:select_server_supported_programs]
>>> 0-backups-client-1: Using Program GlusterFS 3.3, Num (1298437), Version
>>> (330)
>>> [2016-03-07 16:03:14.505580] I [MSGID: 114046]
>>> [client-handshake.c:1213:client_setvolume_cbk] 0-backups-client-1:
>>> Connected to backups-client-1, attached to remote volume
>>> '/zfspool/glusterfs/backups/data'.
>>> [2016-03-07 16:03:14.505627] I [MSGID: 114047]
>>> [client-handshake.c:1224:client_setvolume_cbk] 0-backups-client-1:
>>> Server and Client lk-version numbers are not same, reopening the fds
>>> [2016-03-07 16:03:14.506210] I [MSGID: 114035]
>>> [client-handshake.c:193:client_set_lk_version_cbk] 0-backups-client-1:
>>> Server lk version = 1
>>> [2016-03-07 16:03:14.507836] I [MSGID: 108031]
>>> [afr-common.c:1782:afr_local_discovery_cbk] 0-backups-replicate-0:
>>> selecting local read_child backups-client-0
>>>
>>>
>>>
>>>
>>> On 3/6/16 9:13 PM, Jiffin Tony Thottan wrote:
>>>>
>>>>
>>>> On 05/03/16 07:12, Mark Selby wrote:
>>>>> I am trying to use GlusterFS as a general purpose NFS file server. I
>>>>> have tried using the FUSE client but the performance fall off vs NFS
>>>>> is quite large
>>>>>
>>>>> Both the client and the server are Ubuntu 14.04.
>>>>>
>>>>> I am using Gluster 3.6.9 because of the FUSE performance issues that
>>>>> have been reported with 3.7.8 (see
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1309462)
>>>>>
>>>>> I am having serious issues with a generic NFS client as shown by the
>>>>> issues below. Basically most FOPs are giving me a Remote I/O error.
>>>>>
>>>>> I would not think I was 1st person to see these issues - but my
>>>>> Google Fu is not working.
>>>>>
>>>>> Any and all help would be much appreciated
>>>>>
>>>>> BTW - These operation against a plain Linux NFS server work fine.
>>>>>
>>>>>
>>>>> root at dc1strg001x /var/log 448# gluster volume status
>>>>> Status of volume: backups
>>>>> Gluster process Port Online Pid
>>>>> ------------------------------------------------------------------------------
>>>>>
>>>>> Brick dc1strg001x:/zfspool/glusterfs/backups/data 49152 Y 6462
>>>>> Brick dc1strg002x:/zfspool/glusterfs/backups/data 49152 Y 6382
>>>>> NFS Server on localhost 2049 Y 6619
>>>>> Self-heal Daemon on localhost N/A Y 6626
>>>>> NFS Server on dc1strg002x 2049 Y 6502
>>>>> Self-heal Daemon on dc1strg002x N/A Y 6509
>>>>>
>>>>>
>>>>> root at vc1test001 /root 735# mount -o vers=3 -t nfs
>>>>> dc1strg001x:/backups /mnt/backups_nfs
>>>>>
>>>>> root at vc1test001 /mnt/backups_nfs 737# dd if=/dev/zero of=testfile
>>>>> bs=16k count=16384
>>>>> 16384+0 records in
>>>>> 16384+0 records out
>>>>> 268435456 bytes (268 MB) copied, 2.46237 s, 109 MB/s
>>>>>
>>>>> root at vc1test001 /mnt/backups_nfs 738# rm testfile
>>>>>
>>>>> root at vc1test001 /mnt/backups_nfs 739# dd if=/dev/zero of=testfile
>>>>> bs=16k count=16384
>>>>> dd: failed to open ~testfile~: Remote I/O error
>>>>>
>>>>> root at vc1test001 /var/tmp 743# rsync -av testfile /mnt/backups_nfs/
>>>>> sending incremental file list
>>>>> testfile
>>>>> rsync: mkstemp "/mnt/backups_nfs/.testfile.bzg47C" failed: Remote I/O
>>>>> error (121)
>>>>>
>>>>> sent 1,074,004,056 bytes received 121 bytes 165,231,411.85 bytes/sec
>>>>> total size is 1,073,741,824 speedup is 1.00
>>>>> rsync error: some files/attrs were not transferred (see previous
>>>>> errors) (code 23) at main.c(1183) [sender=3.1.0]
>>>>>
>>>>
>>>> Can you please provide the volume configuration(gluster vol info )
>>>> and log file for nfs server which u mounted (/var/log/glusterfs)
>>>>
>>>> --
>>>> Jiffin
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160309/29c64371/attachment.html>
More information about the Gluster-users
mailing list