[Gluster-devel] NFS re-export works for me
Brent A Nelson
brent at phys.ufl.edu
Mon Mar 9 21:37:36 UTC 2009
Note that you can also use the UNFS3 user-mode NFS server rather than the
nfs-kernel-server. With it, you don't need a custom FUSE, and you don't
need to disable direct-IO. Probably the big downside, though, is that it
doesn't support NFS file locking.
Thanks,
Brent
On Mon, 9 Mar 2009, Raghavendra G wrote:
> Hi,
>
> starting glusterfs with direct-io disabled (glusterfs --disable-direct-io)
> ensures working of NFS re-export of glusterfs.
>
> regards,
>
> On Fri, Apr 27, 2007 at 12:47 AM, gpvcs at tiscali.fr <gpvcs at tiscali.fr> wrote:
>
>> Hi Avati,
>>
>> I first thought that I found the error when suddenly it was
>> working on my machine but I never managed to reproduce it.
>> Then I realized that in cluster mode, one of my machine was
>> running using the default fuse but it did it again on my
>> standalone machine.
>> One think for sure is that with default fuse (API 7.6)
>> NFS mount fail always. You have to use a latter fuse.
>>
>> After many try I started to trace NFS and Fuse in kernel mode:
>>
>> cd /proc/sys/sunrpc
>> echo 32767 > nfs_debug
>> echo 32767 > nfsd_debug
>>
>> and did my test again:
>>
>> exportfs:
>> localhost:/gfs/fs1 on /gfs/d610.nfsV3 type nfs
>> (rw,nfsvers=3,noac,hard,intr,tcp,lock,posix,addr=127.0.0.1)
>>
>> cd /gfs/nfsV3
>> ls > 11
>>
>>
>> The trace between a successfull writing other NFS and error
>> change here:
>> Apr 26 16:21:06 fturi kernel: NFS call write 6 @ 0
>> Apr 26 16:21:06 fturi kernel: nfsd_dispatch: vers 3 proc 7
>> Apr 26 16:21:06 fturi kernel: nfsd: WRITE(3) 24: 02010001
>> 00000000 00000003 00000000 00000001 00000000 6 bytes at 0 stable
>> Apr 26 16:21:06 fturi kernel: nfsd: fh_verify(24: 02010001
>> 00000000 00000003 00000000 00000001 00000000)
>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>> Apr 26 16:21:06 fturi kernel: # NO PERM 531 nfsd: write
>> complete err=-1
>> Apr 26 16:21:06 fturi kernel: NFS reply write: -1
>> <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
>>
>> The NO PERM message comes from this function in fuse
>> 2.6.3/kernel/file.c
>>
>> static int fuse_get_user_pages(struct fuse_req *req, const
>> char __user *buf,
>> unsigned nbytes, int write)
>> {
>> unsigned long user_addr = (unsigned long) buf;
>> unsigned offset = user_addr & ~PAGE_MASK;
>> int npages;
>>
>> /* This doesn't work with nfsd */
>> if (!current->mm)
>> {printk(KERN_CRIT "# NO PERM
>> %d",__LINE__);return -EPERM;};
>>
>> nbytes = min(nbytes, (unsigned)
>> FUSE_MAX_PAGES_PER_REQ << PAGE_SHIFT);
>> npages = (nbytes + offset + PAGE_SIZE - 1) >>
>> PAGE_SHIFT;
>> npages = max(npages, 1);
>> npages = min(npages, FUSE_MAX_PAGES_PER_REQ);
>> down_read(¤t->mm->mmap_sem);
>> npages = get_user_pages(current, current->mm,
>> user_addr, npages, write,
>> 0, req->pages, NULL);
>> up_read(¤t->mm->mmap_sem);
>> if (npages < 0)
>> return npages;
>>
>> req->num_pages = npages;
>> req->page_offset = offset;
>> return 0;
>> }
>>
>>
>> I Just added the dprintk to make sure it was there. There is
>> a comment about: /* doesnt work with nfsd */
>>
>> So, its look like its fuse refusing (error=-1 = NOPERM) the
>> writting other NFS.
>> I'm puzzled by the fact it works under another linux. maybe
>> the fuse_get_user_pages is not called in your case of the
>> structure "current->mm" is filled up in your case.
>>
>> This is the trace collected by the debug mode of server.vol
>> and client.vol ( debug on top of the brick):
>>
>> glusterfsd is empty
>> glusterfs contains:
>> [Apr 26 16:21:02] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:21:02] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/11)
>>
>> [Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_
>> mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
>> st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
>> 15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
>> 15:54:32]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/11)
>>
>> [Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=64356, st_
>> mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
>> st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
>> 15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
>> 15:54:32]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:967/trace_open()]
>> trace:trace_open (*this=0x8395650, path=/11, flags=32769,
>> mode=0)
>> [Apr 26 16:21:04] [DEBUG/trace.c:88/trace_open_cbk()]
>> trace:trace_open_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *ctx=0x8396f58), *buf=0x8396c60 {st_dev=64768,
>> st_ino=
>> 64356, st_mode=33206, st_nlink=1, st_uid=99, st_gid=99,
>> st_rdev=0, st_size=0, st_blksize=4096, st_blocks=8,
>> st_atime=[Apr 26 15:48:19], st_mtime=[Apr 26 15:54:32], st_cti
>> me=[Apr 26 15:54:32]})
>> [Apr 26 16:21:04] [DEBUG/trace.c:1089/trace_release()]
>> trace:trace_release (*this=0x8395650, *ctx=0x8396f58)
>> [Apr 26 16:21:04] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/11)
>>
>> [Apr 26 16:21:04] [DEBUG/trace.c:386/trace_release_cbk()]
>> trace:trace_release_cbk (*this=0x8395650, op_ret=0, op_errno=95)
>> [Apr 26 16:21:04] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8397098 {st_dev=64768, st_ino=64356, st_
>> mode=33206, st_nlink=1, st_uid=99, st_gid=99, st_rdev=0,
>> st_size=0, st_blksize=4096, st_blocks=8, st_atime=[Apr 26
>> 15:48:19], st_mtime=[Apr 26 15:54:32], st_ctime=[Apr 26
>> 15:54:32]})
>> [Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:28] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:28] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:31] [DEBUG/trace.c:688/trace_getattr()]
>> trace:trace_getattr (*this=0x8395650, path=/)
>>
>> [Apr 26 16:28:31] [DEBUG/trace.c:109/trace_getattr_cbk()]
>> trace:trace_getattr_cbk (*this=0x8395650, op_ret=0,
>> op_errno=95, *buf=0x8396c60 {st_dev=64768, st_ino=68494, st_
>> mode=16895, st_nlink=2, st_uid=0, st_gid=500, st_rdev=0,
>> st_size=4096, st_blksize=4096, st_blocks=16, st_atime=[Apr
>> 26 16:19:02], st_mtime=[Apr 26 09:42:58], st_ctime=[Ap
>> r 26 09:42:58]})
>> [Apr 26 16:28:31] [DEBUG/trace.c:1155/trace_getxattr()]
>> trace:trace_getxattr (*this=0x8395650, path=/,
>> name=system.posix_acl_access, size=0)
>> [Apr 26 16:28:31] [DEBUG/trace.c:513/trace_getxattr_cbk()]
>> trace:trace_getxattr_cbk (*this=0x8395650, op_ret=-1,
>> op_errno=95)
>>
>>
>> Cdt FT
>>
>>
>>
>>
>>
>> ---------- Initial Header -----------
>>
>> From : Anand Avati <avati at zresearch.com>
>> To : gpvcs at tiscali.fr
>> Cc : gluster-devel at nongnu.org
>> Date : Wed, 18 Apr 2007 08:55:26 -0700
>> Subject : Re: Re:[Gluster-devel] NFS re-export works for me
>>
>> I was able to successfully mount a glusterfs export via NFS
>> too. I am
>> using slackware 10.2 and did nothing special apart from using
>> fuse-2.6.0. Maybe your problem could be because of the AIX
>> client.
>> please try from a GNU/Linux client and see if it helps.
>> Attaching the
>> log file by loading debug/trace will help get a better picture.
>>
>> regards,
>> avati
>>
>> On Wed, Apr 18, 2007 at 11:25:49AM -0400, Brent A Nelson wrote:
>>> rw,fsid=10 are the export options.
>>>
>>> 2.6.15 is the kernel (it's a Ubuntu Dapper with its
>> kernel, probably
>>> closer to 2.6.16). Fuse 2.6.3 was added from an extra
>> repository.
>>>
>>> I used no options at all on the Ubuntu Breezy NFS client
>> mount. Looking
>>> at the mtab, no special options are listed there, either,
>> so the options
>>> are whatever is default for this platform.
>>>
>>> Thanks,
>>>
>>> Brent
>>>
>>> On Wed, 18 Apr 2007, gpvcs at tiscali.fr wrote:
>>>
>>>> Hi Brent,
>>>>
>>>> That's a very good news for me. Could you please tell me
>>>> which linux kernel version are you using (uname -a) and
>>>> /etc/exports line and mount command you used ?
>>>> I'm not lucky with NFS and will very appreciate thoses
>> details.
>>>>
>>>> Have a nice day, Francois
>>>> ---------- Initial Header -----------
>>>>
>>>> From : gluster-devel-bounces+gpvcs=tiscali.fr at nongnu.org
>>>> To : gluster-devel at nongnu.org
>>>> Cc :
>>>> Date : Wed, 18 Apr 2007 00:51:51 -0400 (EDT)
>>>> Subject : [Gluster-devel] NFS re-export works for me
>>>>
>>>> I tried a quick NFS re-export to another Linux box (with the
>>>> fsid= option
>>>> and fuse 2.6.3 on the NFS server, of course). I didn't test
>>>> it much, but
>>>> copies to and from the NFS filesystem worked just fine. I
>>>> think writes
>>>> were very slow, but they didn't give me any trouble.
>>>>
>>>> No special mount options (such as NFS version or transport)
>>>> were needed.
>>>>
>>>> Thanks,
>>>>
>>>> Brent
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at nongnu.org
>>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>>>
>>>
>>> ------------------------ ALICE C'EST ENCORE MIEUX AVEC
>> CANAL+ LE BOUQUET !
>>> ---------------
>>> Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE
>> BOUQUET, en cliquant
>>> ici http://alicebox.fr
>>> Soumis à conditions.
>>>>
>>>>
>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at nongnu.org
>>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>>
>> --
>> ultimate_answer_t
>> deep_thought (void)
>> {
>> sleep (years2secs (7500000));
>> return 42;
>> }
>>
>>
>> ------------------------ ALICE C'EST ENCORE MIEUX AVEC CANAL+ LE BOUQUET !
>> ---------------
>> Découvrez vite l'offre exclusive ALICEBOX et CANAL+ LE BOUQUET, en cliquant
>> ici http://alicebox.fr
>> Soumis à conditions.
>>
>>
>>
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>
> --
> Raghavendra G
>
More information about the Gluster-devel
mailing list