[Gluster-devel] Nodeid changed due to write-behind option chagned online will lead to un-expected umount by kernel

Raghavendra Gowdappa rgowdapp at redhat.com
Mon Mar 20 13:35:08 UTC 2017


+csaba

----- Original Message -----
> From: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
> To: "George Lian (Nokia - CN/Hangzhou)" <george.lian at nokia.com>
> Cc: "Bingxuan Zhang (Nokia - CN/Hangzhou)" <bingxuan.zhang at nokia.com>, Gluster-devel at gluster.org, "Tero Venetjoki
> (Nokia - FI/Espoo)" <tero.venetjoki at nokia.com>, "Cynthia Zhou (Nokia - CN/Hangzhou)" <cynthia.zhou at nokia.com>
> Sent: Monday, March 20, 2017 7:02:50 PM
> Subject: Re: [Gluster-devel] Nodeid changed due to write-behind option chagned online will lead to un-expected umount
> by kernel
> 
> 
> 
> ----- Original Message -----
> > From: "George Lian (Nokia - CN/Hangzhou)" <george.lian at nokia.com>
> > To: "Raghavendra Gowdappa" <rgowdapp at redhat.com>, Gluster-devel at gluster.org
> > Cc: "Bingxuan Zhang (Nokia - CN/Hangzhou)" <bingxuan.zhang at nokia.com>,
> > "Cynthia Zhou (Nokia - CN/Hangzhou)"
> > <cynthia.zhou at nokia.com>, "Tero Venetjoki (Nokia - FI/Espoo)"
> > <tero.venetjoki at nokia.com>
> > Sent: Monday, March 20, 2017 8:44:30 AM
> > Subject: Nodeid changed due to write-behind option chagned online will lead
> > to un-expected umount by kernel
> > 
> > 
> > Hi, GlusterFS expert,
> > 
> > In our latest test, We found an issue potentially related to glusterfs.
> > When
> > I execute “gluster volume set <volume> performance.write-behind on/off”,
> > some bind mounts will get lost, this issue is permanent!
> > Test steps:
> > a)	mkdir -p /mnt/log/node1/test1/test2; mkdir -p /mnt/log/node2/test1/test2
> > b)	mkdir -p /mnt/test1/test2
> > c)	mount –bind /mnt/log/node1/test1 /mnt/test1
> > d)	mount –bind /mnt/log/node2/test1/test2 /mnt/test1/test2
> > e)	mount | cut -d " " -f 3|xargs stat
> > f)	gluster volume set log performance.write-behind on/off    check
> > /mnt/test1/test2 , could found this bind mount get lost
> > 
> > We’ve consulted the linux kernel side guys, their explanations for this
> > “bind
> > mount lost” issue is when kernel side do lookup or stat(), it will find the
> > nodeid has changed and trigger fuse_dentry_revalidate, and
> > fuse_dentry_relivalidate() fails for some dentry and invalidates it which
> > leads to unmounting.
> > 
> >    88.912262 |   0)    stat-437    |               |
> >    fuse_dentry_revalidate [fuse]() {
> > <...>
> >    88.912264 |   0)    stat-437    |               |
> >    fuse_simple_request [fuse]() {
> > <...>
> >    88.921383 |   0)    stat-437    | # 9119.255 us |                  } /*
> >    fuse_simple_request [fuse] */
> >    88.921383 |   0)    stat-437    |   0.093 us    |
> >    dput();
> >    88.921384 |   0)    stat-437    |               |
> >    fuse_queue_forget [fuse]() {
> > <...>
> >    88.921427 |   0)    stat-437    | + 42.737 us   |                  }
> >    88.921427 |   0)    stat-437    | # 9164.967 us |                } /*
> >    fuse_dentry_revalidate [fuse] */
> >    88.921427 |   0)    stat-437    |               |
> >    d_invalidate() {
> >    88.921427 |   0)    stat-437    |   0.048 us    |
> >    _raw_spin_lock();
> >    88.921428 |   0)    stat-437    |               |
> >    d_walk() {
> >    88.921428 |   0)    stat-437    |   0.040 us    |
> >    _raw_spin_lock();
> >    88.921428 |   0)    stat-437    |   0.046 us    |
> >    detach_and_collect();
> >    88.921429 |   0)    stat-437    |   0.723 us    |                  }
> >    88.921429 |   0)    stat-437    |               |
> >    __detach_mounts() {
> > 
> > fuse_queue_forget() got called inside fuse_dentry_revalidate(), so the
> > failure is clearly caused by  (outarg.nodeid != get_node_id(inode)) check:
> > 
> > 7078187a795f8 (Miklos Szeredi     2014-12-12 09:49:05 +0100  230)
> > if (!ret) {
> > 6314efee3cfee (Miklos Szeredi     2013-10-01 16:41:22 +0200  231)
> > fi = get_fuse_inode(inode);
> > 9e6268db496a2 (Miklos Szeredi     2005-09-09 13:10:29 -0700  232)
> > if (outarg.nodeid != get_node_id(inode)) {
> > 07e77dca8a1f1 (Miklos Szeredi     2010-12-07 20:16:56 +0100  233)
> > fuse_queue_forget(fc, forget, outarg.nodeid,
> > 1);
> > e2a6b95236eba (Miklos Szeredi     2013-09-05 11:44:43 +0200  234)
> > goto invalid;
> > 9e6268db496a2 (Miklos Szeredi     2005-09-09 13:10:29 -0700  235)
> > }
> > 
> > The latest  kernel is doing the following in fs/fuse/dir.c:
> > 
> > 			if (outarg.nodeid != get_node_id(inode)) {
> > 				fuse_queue_forget(fc, forget, outarg.nodeid, 1);
> > 				goto invalid;
> > 			}
> > 
> > invalid:
> > 	ret = 0;
> > 	goto out;
> > }
> > 
> > When We check the glusterfs code, I find that when execute command “gluster
> > volume set <volume> performance.write-behind on/off”, the glusterfs client
> > side graph will be re-built. And all xlators will be re-initialized ,
> > So I am wondering if this will lead to node id change and cause kernel side
> > do the umount, because I find in function fuse_graph_setup, inode_table_new
> > is called, does it mean all inode will be changed?
> 
> Yes nodeids change after graph switch. Nice catch btw :).
> 
> > If so, we need consider the following solutions
> >  1) is there some way to notify kernel side in graceful way so that kernel
> >  side can do update the nodeid which stored before?
> 
> AFAIK, there is no such way.
> 
> >  2) is there some way let itable only initialized once, that's mean, we
> >  only
> >  initialized while fuse_graph_setup is first called, and reused the itable
> >  address, while new graph is created and fuse_graph_setup called due to
> >  options changed such like performance.write_behind.
> 
> We cannot use the inodes from old graph, because per xlator state from
> xlators of new graph is not present in them. So, we build new inodes and let
> interested xlators in the new graph store the state. This is done by sending
> a lookup call for each nodeid the kernel is using (on demand basis. We don't
> do this if kernel doesn't do an operation on the inodes it has cached).
> 
> The other possible approach is to build the state in old inodes instead of
> creating new inodes (that way we don't change nodeid). But how scalable is
> that approach is the question (what happens if there are too many graph
> switches and functions like inode_ctx_get/set slow down due to too many
> xlator states). Also, this is just a thought and I've to think more on this.
> Even if we do this, it will be a more disruptive change and needs more
> testing to be accepted as a solution.
> 
> >  3) is there some way let ino(which showed in Annex 1)  not changed due to
> >  graph rebuild, I mean not through address, such like the mapping of the
> >  gfid, then even if the itable rebuild, glusterfs have the same nodeid
> >  which
> >  stored in kernel if solution 1 and solution 2 can't be used?
> 
> Using plain gfids as nodeids would be the best solution. But kernel allows
> only 64bit nodeids and gfids are bigger. Mapping requires one more
> data-structure (say hash-table etc), that brings more overhead and I guess
> we wanted to keep things simple (by just using address of inode object as
> nodeid - Annex 1).
> 
> >  4) any other solutions?
> 
> I can't think of an easy simple solution now. Will update if I can find
> something.
> 
> > 
> > I would like to know your opinion about this issue and our solutions, and
> > your reply will be appreciated!
> > 
> > Annex 1:  inode mapping with the address of node stored in itable.
> > fuse_ino_to_inode (uint64_t ino, xlator_t *fuse)
> > {
> >         inode_t  *inode = NULL;
> >         xlator_t *active_subvol = NULL;
> > 
> >         if (ino == 1) {
> >                 active_subvol = fuse_active_subvol (fuse);
> >                 if (active_subvol)
> >                         inode = active_subvol->itable->root;
> >         } else {
> >                 inode = (inode_t *) (unsigned long) ino;
> >                 inode_ref (inode);
> >         }
> > 
> >         return inode;
> > }
> > 
> > Annex 2:  kernel code changed for this issue, which mean for some kernel
> > there is no un-expect umount issue, but has DOS-attack issue, and while the
> > latest kernel will have the un-expect umount issue due to fuse_graph_setup
> > rebuild.
> > 1) no un-expected umount issue but has DOS-attack issue.
> > kernel is doing the following in fs/fuse/dir.c:
> > 
> > 			if (outarg.nodeid != get_node_id(inode)) {
> > 				fuse_queue_forget(fc, forget, outarg.nodeid, 1);
> > 				goto invalid;
> > 			}
> > 
> > invalid:
> > 	ret = 0;
> > 	if (check_submounts_and_drop(entry) != 0)
> > 		ret = 1;
> > 	goto out;
> > }
> > 
> > 
> > The above check_submounts_and_drop() call was added in September 2013:
> > 
> > commit 46ea1562da792a94ee8391f7aed882eb7771e18c
> > Author: Anand Avati <avati at redhat.com>
> > Date:   Thu Sep 5 11:44:44 2013 +0200
> > 
> >     fuse: drop dentry on failed revalidate
> > 
> >     Drop a subtree when we find that it has moved or been delated.  This
> >     can
> >     be
> >     done as long as there are no submounts under this location.
> > 
> >     If the directory was moved and we come across the same directory in a
> >     future lookup it will be reconnected by d_materialise_unique().
> > 
> >     Signed-off-by: Anand Avati <avati at redhat.com>
> >     Signed-off-by: Miklos Szeredi <mszeredi at suse.cz>
> >     Signed-off-by: Al Viro <viro at zeniv.linux.org.uk>
> > 
> > diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c
> > index 25c6cfe98801..0e6961aae6c0 100644
> > --- a/fs/fuse/dir.c
> > +++ b/fs/fuse/dir.c
> > @@ -259,6 +259,8 @@ out:
> > 
> >  invalid:
> >         ret = 0;
> > +       if (check_submounts_and_drop(entry) != 0)
> > +               ret = 1;
> >         goto out;
> >  }
> > 
> > Ie. this version of fuse specifically avoided invalidating the dentry if
> > there were directories mounted under it.
> > (The check_submounts_and_drop() function is part of general vfs code, not
> > fuse specific.)
> > 
> > Then, in October 2013 there was a wider kernel change which caused invalid
> > mounts to be deleted more aggressively
> > for security reasons (elimination of certain kind of DOS attacks):
> > 
> > commit 8ed936b5671bfb33d89bc60bdcc7cf0470ba52fe
> > Author: Eric W. Biederman <ebiederman at twitter.com>
> > Date:   Tue Oct 1 18:33:48 2013 -0700
> > 
> >     vfs: Lazily remove mounts on unlinked files and directories.
> > 
> >     With the introduction of mount namespaces and bind mounts it became
> >     possible to access files and directories that on some paths are mount
> >     points but are not mount points on other paths.  It is very confusing
> >     when rm -rf somedir returns -EBUSY simply because somedir is mounted
> >     somewhere else.  With the addition of user namespaces allowing
> >     unprivileged mounts this condition has gone from annoying to allowing
> >     a DOS attack on other users in the system.
> > 
> >     The possibility for mischief is removed by updating the vfs to support
> >     rename, unlink and rmdir on a dentry that is a mountpoint and by
> >     lazily unmounting mountpoints on deleted dentries.
> > 
> > 
> > The above commit changed check_submounts_and_drop() function so that it
> > will
> > not return non-zero error status any more if the entry is a mount point (or
> > has mount points under it).
> > Instead it simply unmounts those entries unconditionally.
> > 
> > 
> > Finally, that call was removed from fuse code because it was not doing
> > anything anymore:
> > 
> > commit 9b053f3207e8887fed88162a339fdd4001abcdb2
> > Author: Eric W. Biederman <ebiederm at xmission.com>
> > Date:   Thu Feb 13 09:34:30 2014 -0800
> > 
> >     vfs: Remove unnecessary calls of check_submounts_and_drop
> > 
> >     Now that check_submounts_and_drop can not fail and is called from
> >     d_invalidate there is no longer a need to call check_submounts_and_drom
> >     from filesystem d_revalidate methods so remove it.
> > 
> >     Reviewed-by: Miklos Szeredi <miklos at szeredi.hu>
> >     Signed-off-by: "Eric W. Biederman" <ebiederm at xmission.com>
> >     Signed-off-by: Al Viro <viro at zeniv.linux.org.uk>
> > 
> > diff --git a/fs/fuse/dir.c b/fs/fuse/dir.c
> > index de1d84af9f7c..820efd74ca9f 100644
> > --- a/fs/fuse/dir.c
> > +++ b/fs/fuse/dir.c
> > @@ -274,9 +274,6 @@ out:
> > 
> > 
> > 2) has un-expected umount issue.
> >  invalid:
> >         ret = 0;
> > -
> > -       if (!(flags & LOOKUP_RCU) && check_submounts_and_drop(entry) != 0)
> > -               ret = 1;
> > 	goto out;
> >  }
> > 
> > Later, the check_submounts_and_drop() function was merged to d_invalidate()
> > and ceased to exist.
> > 
> > So, the unmounting does not happen in CentOS because it is using quite old
> > and more relaxed kernel where filesystem code lets the mount exists even if
> > glusterfs node id has changed.
> > But newer kernels released during the last 3 years are using stricter mount
> > handling policy which causes the unmounting (and most likely will not go
> > back).
> > 
> > 
> > Best regards,
> > George
> > 
> > 
> >
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-devel


More information about the Gluster-devel mailing list