[Bugs] [Bug 1181048] lockless lookup cause disk to be kicked out

Thu Jan 15 03:49:06 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1181048

--- Comment #12 from lidi <lidi at perabytes.com> ---
When I use latest master for this test with one virtual machine, I found
another phenomenon.

After kill one brick, console 1 and 3 work well. console 2 seems like dead
lock, and report nothing. 

[root at node-1 ~]# getfattr -m. -de hex /sd{b,c,d}/bb
getfattr: Removing leading '/' from absolute path names
# file: sdb/bb
trusted.ec.config=0x0000080301000200
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x0000000000000c82
trusted.gfid=0x5db7082224804e00816bc03309449577

# file: sdc/bb
trusted.ec.config=0x0000080301000200
trusted.ec.size=0x0000000000000000
trusted.ec.version=0x0000000000000c82
trusted.gfid=0x5db7082224804e00816bc03309449577

# file: sdd/bb
trusted.ec.config=0x0000080301000200
trusted.ec.size=0x000000000c800000
trusted.ec.version=0x0000000000000c81
trusted.gfid=0x5db7082224804e00816bc03309449577

[2015-01-15 03:19:13.662595] W [socket.c:611:__socket_rwv] 0-test-client-2:
readv on 10.10.21.20:49154 failed (Connection reset by peer)
[2015-01-15 03:19:13.663052] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1ab)[0x7f75d023c4db] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1cb)[0x7f75d001237b] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f75d001248e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f75d001255b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f75d0012b1f] )))))
0-test-client-2: forced unwinding frame type(GlusterFS 3.3) op(FTRUNCATE(24))
called at 2015-01-15 03:18:48.958574 (xid=0x508d)
[2015-01-15 03:19:13.663096] W [client-rpc-fops.c:1421:client3_3_ftruncate_cbk]
0-test-client-2: remote operation failed: Transport endpoint is not connected
[2015-01-15 03:19:13.663120] W [ec-common.c:162:ec_check_status]
0-test-disperse-0: Operation failed on some subvolumes (up=7, mask=7,
remaining=0, good=3, bad=4)
[2015-01-15 03:19:13.663187] I [ec-heal.c:546:ec_heal_init] 0-ec: Healing
'/aa', gfid 269a9bc0-7a85-404f-a126-af878520847b
[2015-01-15 03:19:13.666965] I [socket.c:3301:socket_submit_request]
0-test-client-2: not connected (priv->connected = 0)
[2015-01-15 03:19:13.667001] W [rpc-clnt.c:1565:rpc_clnt_submit]
0-test-client-2: failed to submit rpc-request (XID: 0x5296 Program: GlusterFS
3.3, ProgVers: 330, Proc: 31) to rpc-transport (test-client-2)
[2015-01-15 03:19:13.667029] E [client-rpc-fops.c:1630:client3_3_entrylk_cbk]
0-test-client-2: remote operation failed: Transport endpoint is not connected
[2015-01-15 03:19:13.667131] W [ec-common.c:397:ec_child_select]
0-test-disperse-0: Executing operation with some subvolumes unavailable (4)
[2015-01-15 03:19:13.667451] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1ab)[0x7f75d023c4db] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1cb)[0x7f75d001237b] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f75d001248e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f75d001255b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f75d0012b1f] )))))
0-test-client-2: forced unwinding frame type(GlusterFS 3.3) op(FTRUNCATE(24))
called at 2015-01-15 03:18:49.325357 (xid=0x5294)
[2015-01-15 03:19:13.667508] W [client-rpc-fops.c:1421:client3_3_ftruncate_cbk]
0-test-client-2: remote operation failed: Transport endpoint is not connected
[2015-01-15 03:19:13.667544] W [ec-common.c:162:ec_check_status]
0-test-disperse-0: Operation failed on some subvolumes (up=7, mask=7,
remaining=0, good=3, bad=4)
[2015-01-15 03:19:13.667583] I [ec-heal.c:546:ec_heal_init] 0-ec: Healing
'/bb', gfid 5db70822-2480-4e00-816b-c03309449577
[2015-01-15 03:19:13.667679] W [rpc-clnt.c:1565:rpc_clnt_submit]
0-test-client-2: failed to submit rpc-request (XID: 0x5297 Program: GlusterFS
3.3, ProgVers: 330, Proc: 31) to rpc-transport (test-client-2)
[2015-01-15 03:19:13.667717] E [client-rpc-fops.c:1630:client3_3_entrylk_cbk]
0-test-client-2: remote operation failed: Transport endpoint is not connected
[2015-01-15 03:19:13.667979] W [rpc-clnt.c:1565:rpc_clnt_submit]
0-test-client-2: failed to submit rpc-request (XID: 0x5298 Program: GlusterFS
3.3, ProgVers: 330, Proc: 33) to rpc-transport (test-client-2)
[2015-01-15 03:19:13.668031] W [client-rpc-fops.c:1743:client3_3_xattrop_cbk]
0-test-client-2: remote operation failed: Success. Path: /bb
(5db70822-2480-4e00-816b-c03309449577)
[2015-01-15 03:19:13.668098] W [rpc-clnt.c:1565:rpc_clnt_submit]
0-test-client-2: failed to submit rpc-request (XID: 0x5299 Program: GlusterFS
3.3, ProgVers: 330, Proc: 29) to rpc-transport (test-client-2)
[2015-01-15 03:19:13.668135] E [client-rpc-fops.c:1526:client3_3_inodelk_cbk]
0-test-client-2: remote operation failed: Transport endpoint is not connected
[2015-01-15 03:19:13.668373] E [rpc-clnt.c:362:saved_frames_unwind] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn+0x1ab)[0x7f75d023c4db] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_unwind+0x1cb)[0x7f75d001237b] (-->
/usr/lib64/libgfrpc.so.0(saved_frames_destroy+0xe)[0x7f75d001248e] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0xab)[0x7f75d001255b] (-->
/usr/lib64/libgfrpc.so.0(rpc_clnt_notify+0x15f)[0x7f75d0012b1f] )))))
0-test-client-2: forced unwinding frame type(GF-DUMP) op(NULL(2)) called at
2015-01-15 03:18:59.744160 (xid=0x5295)
[2015-01-15 03:19:13.668421] W [rpc-clnt-ping.c:154:rpc_clnt_ping_cbk]
0-test-client-2: socket disconnected
[2015-01-15 03:19:13.668452] I [client.c:2215:client_rpc_notify]
0-test-client-2: disconnected from test-client-2. Client process will keep
trying to connect to glusterd until brick's port is available
[2015-01-15 03:19:13.668663] W [ec-common.c:397:ec_child_select]
0-test-disperse-0: Executing operation with some subvolumes unavailable (4)

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=F1Gl00Tulm&a=cc_unsubscribe