[Gluster-devel] Duplicate entries and other weirdness in a 3*4 volume

[Niels de Vos]

On Sat, Jul 19, 2014 at 10:02:33AM -0400, Benjamin Turner wrote:
> On Fri, Jul 18, 2014 at 10:43 PM, Pranith Kumar Karampuri <
> pkarampu at redhat.com> wrote:
> 
> >
> > On 07/18/2014 07:57 PM, Anders Blomdell wrote:
> >
> >> During testing of a 3*4 gluster (from master as of yesterday), I
> >> encountered
> >> two major weirdnesses:
> >>
> >>    1. A 'rm -rf <some_dir>' needed several invocations to finish, each
> >> time
> >>       reporting a number of lines like these:
> >>                 rm: cannot remove ‘a/b/c/d/e/f’: Directory not empty
> >>
> >
> This is reproducible for me when running dbench on nfs mounts.  I think I
> may have seen it on glusterfs mounts as well but it seems more reproducible
> on nfs.  I should have caught it sooner but it doesn't error out client
> side when cleaning up, and the next test I run the deletes are successful.
>  When this happens in the nfs.log I see:
> 
> This spams the log, from what I can tell it happens when dbench is creating
> the files:
> [2014-07-19 13:37:03.271651] I [MSGID: 109036]
> [dht-common.c:5694:dht_log_new_layout_for_dir_selfheal] 0-testvol-dht:
> Setting layout of /clients/client3/~dmtmp/SEED with [Subvol_name:
> testvol-replicate-0, Err: -1 , Start: 2147483647 , Stop: 4294967295 ],
> [Subvol_name: testvol-replicate-1, Err: -1 , Start: 0 , Stop: 2147483646 ],

My guess is that DHT/AFR fail to create the whole directory structure on 
all bricks (remember that directories should get created on all bricks, 
even for a dht only volume). If creating a directory fails on 
a particular brick, self-heal should pick it up... But well, maybe 
self-heal is not run when deleting directories, causing some directories 
on some bricks to be non-empty, but empty on others. It may be that this 
conflict is not handled correctly.

You could maybe test with different volumes, and narrow down where the 
issue occurs:
- a volume of one brick
- a replicate volume with two bricks
- a distribute volume with two bricks

Potentially increase the number of bricks when a 2-brick afr-only, or 
dht-only volume does not trigger the issue reliably or quickly.

> Occasionally I see:
> [2014-07-19 13:50:46.091429] W [socket.c:529:__socket_rwv] 0-NLM-client:
> readv on 192.168.11.102:45823 failed (No data available)
> [2014-07-19 13:50:46.091570] E [rpc-transport.c:485:rpc_transport_unref]
> (-->/usr/lib64/glusterfs/3.5qa2/xlator/nfs/server.so(nlm_rpcclnt_notify+0x5a)
> [0x7f53775128ea]
> (-->/usr/lib64/glusterfs/3.5qa2/xlator/nfs/server.so(nlm_unset_rpc_clnt+0x75)
> [0x7f537750e3e5] (-->/usr/lib64/libgfrpc.so.0(rpc_clnt_unref+0x63)
> [0x7f5388914693]))) 0-rpc_transport: invalid argument: this

This looks like a bug in the NFS-server, I suggest to file is 
independently from the directory tree create/delete issue.

> I'm opening a BZ now, I'll leave systems up and put the repro steps +
> hostnames in the BZ in case anyone wants to poke around.

Thanks! The NFS problem does not need any checking on the running 
system.

Niels

> 
> -b
> 
> 
> 
> >
> >>    2. After having successfully deleted all files from the volume,
> >>       i have a single directory that is duplicated in gluster-fuse,
> >>       like this:
> >>         # ls -l /mnt/gluster
> >>          total 24
> >>          drwxr-xr-x 2 root root 12288 18 jul 16.17 work2/
> >>          drwxr-xr-x 2 root root 12288 18 jul 16.17 work2/
> >>
> >> any idea on how to debug this issue?
> >>
> > What are the steps to recreate? We need to first find what lead to this.
> > Then probably which xlator leads to this.
> >
> 
> I have not seen this but I am running on a 6x2 volume.  I wonder if this
> may only happen with replica > 2?
> 
> 
> >
> > Pranith
> >
> >>
> >> /Anders
> >>
> >>
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://supercolony.gluster.org/mailman/listinfo/gluster-devel
> >

> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-devel