[Bugs] [Bug 1225323] New: Glusterfs client crash during fd migration after graph switch

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1225323

            Bug ID: 1225323
           Summary: Glusterfs client crash during fd migration after graph
                    switch
           Product: GlusterFS
           Version: mainline
         Component: unclassified
          Assignee: bugs at gluster.org
          Reporter: rgowdapp at redhat.com
                CC: bugs at gluster.org, gluster-bugs at redhat.com



Description of problem:
(gdb) bt
#0  0x00007fe56cb27503 in dht_fsync (frame=0x7fe576d701c4, this=<optimized
out>, fd=0x7fe560002cec, datasync=0, xdata=0x0)
    at ../../../../../xlators/cluster/dht/src/dht-inode-read.c:818
#1  0x00007fe56c8db4b8 in wb_fsync (frame=0x7fe576d70118, this=0x7fe56800a2e0,
fd=0x7fe560002cec, datasync=0, xdata=0x0)
    at ../../../../../xlators/performance/write-behind/src/write-behind.c:1523
#2  0x00007fe56c6ccb3a in ra_fsync (frame=0x7fe576d7006c, this=0x7fe56800b6b0,
fd=0x7fe560002cec, datasync=0, xdata=0x0)
    at ../../../../../xlators/performance/read-ahead/src/read-ahead.c:628
#3  0x00007fe577eefb1d in default_fsync (frame=0x7fe576d7006c,
this=0x7fe56800c9e0, fd=0x7fe560002cec, flags=0, xdata=0x0)
    at ../../../libglusterfs/src/defaults.c:1817
#4  0x00007fe577eefb1d in default_fsync (frame=0x7fe576d7006c,
this=0x7fe56800dde0, fd=0x7fe560002cec, flags=0, xdata=0x0)
    at ../../../libglusterfs/src/defaults.c:1817
#5  0x00007fe577eefb1d in default_fsync (frame=0x7fe576d7006c,
this=0x7fe56800f0d0, fd=0x7fe560002cec, flags=0, xdata=0x0)
    at ../../../libglusterfs/src/defaults.c:1817
#6  0x00007fe577efab3b in default_fsync_resume (frame=0x7fe576d70a80,
this=0x7fe568010540, fd=0x7fe560002cec, flags=0, xdata=0x0)
    at ../../../libglusterfs/src/defaults.c:1376
#7  0x00007fe577f14fdd in call_resume (stub=0x7fe5767f7dd4) at
../../../libglusterfs/src/call-stub.c:2576
#8  0x00007fe567dfa578 in open_and_resume (this=this at entry=0x7fe568010540,
fd=fd at entry=0x7fe560002cec, stub=0x7fe5767f7dd4)
    at ../../../../../xlators/performance/open-behind/src/open-behind.c:241
#9  0x00007fe567dfa922 in ob_fsync (frame=0x7fe576d70a80, this=0x7fe568010540,
fd=0x7fe560002cec, flag=<optimized out>, xdata=<optimized out>)
    at ../../../../../xlators/performance/open-behind/src/open-behind.c:498
#10 0x00007fe567becbaf in mdc_fsync (frame=0x7fe576d709d4, this=0x7fe5680118d0,
fd=0x7fe560002cec, datasync=0, xdata=0x0)
    at ../../../../../xlators/performance/md-cache/src/md-cache.c:1669
#11 0x00007fe5679d0f76 in io_stats_fsync (frame=0x7fe576d70270,
this=0x7fe568012c90, fd=0x7fe560002cec, flags=0, xdata=0x0)
    at ../../../../../xlators/debug/io-stats/src/io-stats.c:2207
#12 0x00007fe577eefb1d in default_fsync (frame=0x7fe576d70270,
this=0x7fe568014160, fd=0x7fe560002cec, flags=0, xdata=0x0)
    at ../../../libglusterfs/src/defaults.c:1817
#13 0x00007fe577f3040a in syncop_fsync (subvol=subvol at entry=0x7fe568014160,
fd=fd at entry=0x7fe560002cec, dataonly=dataonly at entry=0, 
    xdata_in=xdata_in at entry=0x0, xdata_out=xdata_out at entry=0x0) at
../../../libglusterfs/src/syncop.c:2280
#14 0x00007fe56fe0257a in fuse_migrate_fd (this=this at entry=0x1661700,
basefd=basefd at entry=0x7fe560002cec, 
    old_subvol=old_subvol at entry=0x7fe568014160,
new_subvol=new_subvol at entry=0x7fe560030be0)
    at ../../../../../xlators/mount/fuse/src/fuse-bridge.c:4423
#15 0x00007fe56fe0271d in fuse_handle_opened_fds (this=0x1661700,
old_subvol=0x7fe568014160, new_subvol=0x7fe560030be0)
    at ../../../../../xlators/mount/fuse/src/fuse-bridge.c:4508
#16 0x00007fe56fe027c9 in fuse_graph_switch_task (data=<optimized out>) at
../../../../../xlators/mount/fuse/src/fuse-bridge.c:4559
---Type <return> to continue, or q <return> to quit---q
Quit
(gdb) f 0
#0  0x00007fe56cb27503 in dht_fsync (frame=0x7fe576d701c4, this=<optimized
out>, fd=0x7fe560002cec, datasync=0, xdata=0x0)
    at ../../../../../xlators/cluster/dht/src/dht-inode-read.c:818
818            STACK_WIND (frame, dht_fsync_cbk, subvol, subvol->fops->fsync,
(gdb) l
813            local->call_cnt = 1;
814            local->rebalance.flags = datasync;
815    
816            subvol = local->cached_subvol;
817    
818            STACK_WIND (frame, dht_fsync_cbk, subvol, subvol->fops->fsync,
819                        fd, datasync, xdata);
820    
821            return 0;
822    
(gdb) p subvol
$1 = (xlator_t *) 0x0

Version-Release number of selected component (if applicable):
master

How reproducible:
Fairly consistently in some setups

Steps to Reproduce:
1. run ./tests/performance/open-behind.t in a loop. Sometimes Gluster mount
crashes after we switch open-behind off. The crash is seen when when
"num_graphs" is executed.

2.
3.

Actual results:


Expected results:


Additional info:

RCA: Its a race. If a graph switch happens when we have an fd opened under
".meta" subtree, crash is seen when migrating that fd. meta-fsync is not
implemented and hence call is wound to real volume. Other translators have no
idea of handling this virtual inode. Fix is to implement fsync in meta. As a
complete fix, meta should implement all fops, so that it can check whether an
inode belongs to it or not and wind the call down only if it does not belong to
itself.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.