[Gluster-users] Can't mount particular brick even though the brick port is reachable, error message "Transport endpoint is not connected"
Olaf Buitelaar
olaf.buitelaar at gmail.com
Mon Mar 28 10:44:22 UTC 2022
Hi Peter,
I think Staril means, running the command; hosted-engine --set-maintenance
--mode=local, this is also possible from the ovirt ui, via the ribbon on
the hosts section;
[image: image.png]
>From the log's it seems gluster has difficulty find the shared's, e.g.;
.shard/e5f699e2-de11-41be-bd24-e29876928f0f.1279
(be318638-e8a0-4c6d-977d-7a937aa84806/e5f699e2-de11-41be-bd24-e29876928f0f.1279
Do these files exist within your brick's directories?
Did you try to repair the filesystem using xfs_repair?
Best regards,
Olaf
Op ma 28 mrt. 2022 om 10:01 schreef Peter Schmidt <
peterschmidt18351 at yandex.com>:
>
> Hi Olaf,
>
> I tried running "gluster volume start hdd force" but sadly it did not
> change anything.
>
> the raid rebuild has finished now and everything seems to be fine:
> md6 : active raid6 sdu1[2] sdx1[5] sds1[0] sdt1[1] sdz1[7] sdv1[3] sdw1[4]
> sdaa1[8] sdy1[6]
> 68364119040 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9]
> [UUUUUUUUU]
> bitmap: 0/73 pages [0KB], 65536KB chunk
>
> Best regards
> Peter
>
> 25.03.2022, 12:36, "Olaf Buitelaar" <olaf.buitelaar at gmail.com>:
>
> Hi Peter,
>
> I see your raid array is rebuilding, could it be your xfs needs a repair,
> using xfs_repair?
> did you try running gluster v hdd start force?
>
> Kind regards,
>
> Olaf
>
>
> Op do 24 mrt. 2022 om 15:54 schreef Peter Schmidt <
> peterschmidt18351 at yandex.com>:
>
> Hello everyone,
>
> I'm running an oVirt cluster on top of a distributed-replicate gluster
> volume and one of the bricks cannot be mounted anymore from my oVirt hosts.
> This morning I also noticed a stack trace and a spike in TCP connections on
> one of the three gluster nodes (storage2), which I have attached at the end
> of this mail. Only this particular brick on storage2 seems to be causing
> trouble:
> *Brick storage2:/data/glusterfs/hdd/brick3/brick*
> *Status: Transport endpoint is not connected*
>
> I don't know what's causing this or how to resolve this issue. I would
> appreciate it if someone could take a look at my logs and point me in the
> right direction. If any additional logs are required, please let me know.
> Thank you in advance!
>
> Operating system on all hosts: Centos 7.9.2009
> oVirt version: 4.3.10.4-1
> Gluster versions:
> - storage1: 6.10-1
> - storage2: 6.7-1
> - storage3: 6.7-1
>
> ####################################
> # brick is not connected/mounted on the oVirt hosts
>
> *[xlator.protocol.client.hdd-client-7.priv]*
> *fd.0.remote_fd = -1*
> *------ = ------*
> *granted-posix-lock[0] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type =
> F_RDLCK, fl_start = 100, fl_end = 100, user_flock: l_type = F_RDLCK,
> l_start = 100, l_len = 1*
> *granted-posix-lock[1] = owner = 9d673ffe323e25cd, cmd = F_SETLK fl_type =
> F_RDLCK, fl_start = 101, fl_end = 101, user_flock: l_type = F_RDLCK,
> l_start = 101, l_len = 1*
> *------ = ------*
> *connected = 0*
> *total_bytes_read = 11383136800*
> *ping_timeout = 10*
> *total_bytes_written = 16699851552*
> *ping_msgs_sent = 1*
> *msgs_sent = 2*
>
> ####################################
> # mount log from one of the oVirt hosts
> # the IP 172.22.102.142 corresponds to my gluster node "storage2"
> # the port 49154 corresponds to the brick
> storage2:/data/glusterfs/hdd/brick3/brick
>
> *[2022-03-24 10:59:28.138178] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk]
> 0-hdd-client-7: socket disconnected*
> *[2022-03-24 10:59:38.142698] I [rpc-clnt.c:2028:rpc_clnt_reconfig]
> 0-hdd-client-7: changing port to 49154 (from 0)*
> *The message "I [MSGID: 114018] [client.c:2331:client_rpc_notify]
> 0-hdd-client-7: disconnected from hdd-client-7. Client process will keep
> trying to connect to glusterd until brick's port is available" repeated 4
> times between [2022-03-24 10:58:04.114741] and [2022-03-24 10:59:28.137380]*
> *The message "W [MSGID: 114032]
> [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received
> RPC status error [Transport endpoint is not connected]" repeated 4 times
> between [2022-03-24 10:58:04.115169] and [2022-03-24 10:59:28.138052]*
> *[2022-03-24 10:59:49.143217] C
> [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server
> 172.22.102.142:49154 <http://172.22.102.142:49154/> has not responded in
> the last 10 seconds, disconnecting.*
> *[2022-03-24 10:59:49.143838] I [MSGID: 114018]
> [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from
> hdd-client-7. Client process will keep trying to connect to glusterd until
> brick's port is available*
> *[2022-03-24 10:59:49.144540] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f6724643adb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (-->
> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (-->
> /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ))))) 0-hdd-client-7: forced
> unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2022-03-24
> 10:59:38.145208 (xid=0x861)*
> *[2022-03-24 10:59:49.144557] W [MSGID: 114032]
> [client-handshake.c:1546:client_dump_version_cbk] 0-hdd-client-7: received
> RPC status error [Transport endpoint is not connected]*
> *[2022-03-24 10:59:49.144653] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7f6724643adb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7f67243ea7e4] (-->
> /lib64/libgfrpc.so.0(+0xd8fe)[0x7f67243ea8fe] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7f67243eb987] (-->
> /lib64/libgfrpc.so.0(+0xf518)[0x7f67243ec518] ))))) 0-hdd-client-7: forced
> unwinding frame type(GF-DUMP) op(NULL(2)) called at 2022-03-24
> 10:59:38.145218 (xid=0x862)*
> *[2022-03-24 10:59:49.144665] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk]
> 0-hdd-client-7: socket disconnected*
>
> ####################################
> # netcat/telnet to the brick's port of storage2 are working
>
> *[root at storage1 <root at storage1> ~]# netcat -z -v 172.22.102.142 49154*
> *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!*
>
> *[root at storage3 <root at storage3> ~]# netcat -z -v 172.22.102.142 49154*
> *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!*
>
> *[root at ovirthost1 <root at ovirthost1> /var/log/glusterfs]# netcat -z -v
> 172.22.102.142 49154*
> *Connection to 172.22.102.142 49154 port [tcp/*] succeeded!*
>
> ####################################
> # gluster peer status - all gluster peers are connected
> *[root at storage3 <root at storage3> ~]# gluster peer status*
> *Number of Peers: 2*
>
> *Hostname: storage1*
> *Uuid: 055e79c2-b1ff-4a82-9296-205d6877904e*
> *State: Peer in Cluster (Connected)*
>
> *Hostname: storage2*
> *Uuid: d7adcb92-2e71-41a9-80d4-13180ee673cf*
> *State: Peer in Cluster (Connected)*
>
> ####################################
> # Configuration of the volume
> *Volume Name: hdd*
> *Type: Distributed-Replicate*
> *Volume ID: 1b47c2f8-5024-4b85-aa7f-a3f767bb076c*
> *Status: Started*
> *Snapshot Count: 0*
> *Number of Bricks: 4 x 3 = 12*
> *Transport-type: tcp*
> *Bricks:*
> *Brick1: storage1:/data/glusterfs/hdd/brick1/brick*
> *Brick2: storage2:/data/glusterfs/hdd/brick1/brick*
> *Brick3: storage3:/data/glusterfs/hdd/brick1/brick*
> *Brick4: storage1:/data/glusterfs/hdd/brick2/brick*
> *Brick5: storage2:/data/glusterfs/hdd/brick2/brick*
> *Brick6: storage3:/data/glusterfs/hdd/brick2/brick*
> *Brick7: storage1:/data/glusterfs/hdd/brick3/brick*
> *Brick8: storage2:/data/glusterfs/hdd/brick3/brick*
> *Brick9: storage3:/data/glusterfs/hdd/brick3/brick*
> *Brick10: storage1:/data/glusterfs/hdd/brick4/brick*
> *Brick11: storage2:/data/glusterfs/hdd/brick4/brick*
> *Brick12: storage3:/data/glusterfs/hdd/brick4/brick*
> *Options Reconfigured:*
> *storage.owner-gid: 36*
> *storage.owner-uid: 36*
> *server.event-threads: 4*
> *client.event-threads: 4*
> *cluster.choose-local: off*
> *user.cifs: off*
> *features.shard: on*
> *cluster.shd-wait-qlength: 10000*
> *cluster.shd-max-threads: 8*
> *cluster.locking-scheme: granular*
> *cluster.data-self-heal-algorithm: full*
> *cluster.server-quorum-type: server*
> *cluster.eager-lock: enable*
> *network.remote-dio: enable*
> *performance.low-prio-threads: 32*
> *performance.io-cache: off*
> *performance.read-ahead: off*
> *performance.quick-read: off*
> *auth.allow: **
> *network.ping-timeout: 10*
> *cluster.quorum-type: auto*
> *transport.address-family: inet*
> *nfs.disable: on*
> *performance.client-io-threads: on*
>
> ####################################
> # gluster volume status. The brick running on port 49154 is supposedly
> online
>
> *Status of volume: hdd*
> *Gluster process TCP Port RDMA Port Online
> Pid*
>
> *------------------------------------------------------------------------------*
> *Brick storage1:/data/gluste*
> *rfs/hdd/brick1/brick 49158 0 Y
> 9142*
> *Brick storage2:/data/gluste*
> *rfs/hdd/brick1/brick 49152 0 Y
> 115896*
> *Brick storage3:/data/gluste*
> *rfs/hdd/brick1/brick 49158 0 Y
> 131775*
> *Brick storage1:/data/gluste*
> *rfs/hdd/brick2/brick 49159 0 Y
> 9151*
> *Brick storage2:/data/gluste*
> *rfs/hdd/brick2/brick 49153 0 Y
> 115904*
> *Brick storage3:/data/gluste*
> *rfs/hdd/brick2/brick 49159 0 Y
> 131783*
> *Brick storage1:/data/gluste*
> *rfs/hdd/brick3/brick 49160 0 Y
> 9163*
> *Brick storage2:/data/gluste*
> *rfs/hdd/brick3/brick 49154 0 Y
> 115913*
> *Brick storage3:/data/gluste*
> *rfs/hdd/brick3/brick 49160 0 Y
> 131792*
> *Brick storage1:/data/gluste*
> *rfs/hdd/brick4/brick 49161 0 Y
> 9170*
> *Brick storage2:/data/gluste*
> *rfs/hdd/brick4/brick 49155 0 Y
> 115923*
> *Brick storage3:/data/gluste*
> *rfs/hdd/brick4/brick 49161 0 Y
> 131800*
> *Self-heal Daemon on localhost N/A N/A Y
> 170468*
> *Self-heal Daemon on storage3 N/A N/A Y
> 132263*
> *Self-heal Daemon on storage1 N/A N/A Y
> 9512*
>
> *Task Status of Volume hdd*
>
> *------------------------------------------------------------------------------*
> *There are no active volume tasks*
>
> ####################################
> # gluster volume heal hdd info split-brain. All bricks are connected and
> showing no entries (0), except for brick3 on storage2
> *Brick storage2:/data/glusterfs/hdd/brick3/brick*
> *Status: Transport endpoint is not connected*
> *Number of entries in split-brain: -*
>
> ####################################
> # gluster volume heal hdd info. Only brick3 seems to be affected and it
> has lots of entries. brick3 on storage2 is not connected
>
> *Brick storage1:/data/glusterfs/hdd/brick3/brick*
>
> */538befbf-ffa7-4a8c-8827-cee679d589f4/images/615fa020-9737-4b83-a3c1-a61e32400d59/f4917758-deae-4a62-bf4d-5b9a95a7db5b*
> *<gfid:f3d0b19a-2544-48c5-90b7-addd561113bc>*
> */.shard/753a8a81-bd06-4c8c-9515-d54123f6fe4d.1*
> */.shard/c7f5f88f-dc85-4645-9178-c7df8e46a99d.83*
>
> */538befbf-ffa7-4a8c-8827-cee679d589f4/images/bc4362e6-cd43-4ab8-b8fa-0ea72405b7da/ea9c0e7c-d2c7-43c8-b19f-7a3076cc6743*
> */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.10872*
> */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.1901*
>
> */538befbf-ffa7-4a8c-8827-cee679d589f4/images/e48e80fb-d42f-47a4-9a56-07fd7ad868b3/31fd839f-85bf-4c42-ac0e-7055d903df40*
> */.shard/82700f9b-c7e0-4568-a565-64c9a770449f.223*
> */.shard/82700f9b-c7e0-4568-a565-64c9a770449f.243*
> */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.10696*
> */.shard/dc46e963-2b68-4802-9537-42f25ea97ae2.10902*
> *..*
> *Status: Connected*
> *Number of entries: 664*
>
> *Brick storage2:/data/glusterfs/hdd/brick3/brick*
> *Status: Transport endpoint is not connected*
> *Number of entries: -*
>
> *Brick storage3:/data/glusterfs/hdd/brick3/brick*
>
> */538befbf-ffa7-4a8c-8827-cee679d589f4/images/615fa020-9737-4b83-a3c1-a61e32400d59/f4917758-deae-4a62-bf4d-5b9a95a7db5b*
> *<gfid:f3d0b19a-2544-48c5-90b7-addd561113bc>*
> */.shard/753a8a81-bd06-4c8c-9515-d54123f6fe4d.1*
> *..*
> *Status: Connected*
> *Number of entries: 664*
>
> ####################################
> # /data/glusterfs/hdd/brick3 on storage2 is running inside of a software
> RAID
>
> *md6 : active raid6 sdac1[6] sdz1[3] sdx1[1] sdad1[7] sdaa1[4] sdy1[2]
> sdw1[0] sdab1[5] sdae1[8]*
> * 68364119040 blocks super 1.2 level 6, 512k chunk, algorithm 2 [9/9]
> [UUUUUUUUU]*
> * [============>........] check = 64.4% (6290736128/9766302720)
> finish=3220.5min speed=17985K/sec*
> * bitmap: 10/73 pages [40KB], 65536KB chunk*
>
> ####################################
> # glfsheal-hdd.log on storage2
>
> *[2022-03-24 10:15:33.238884] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-hdd-client-10: Connected
> to hdd-client-10, attached to remote volume
> '/data/glusterfs/hdd/brick4/brick'.*
> *[2022-03-24 10:15:33.238931] I [MSGID: 108002]
> [afr-common.c:5607:afr_notify] 0-hdd-replicate-3: Client-quorum is met*
> *[2022-03-24 10:15:33.241616] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-hdd-client-11: Connected
> to hdd-client-11, attached to remote volume
> '/data/glusterfs/hdd/brick4/brick'.*
> *[2022-03-24 10:15:44.078651] C
> [rpc-clnt-ping.c:155:rpc_clnt_ping_timer_expired] 0-hdd-client-7: server
> 172.22.102.142:49154 <http://172.22.102.142:49154/> has not responded in
> the last 10 seconds, disconnecting.*
> *[2022-03-24 10:15:44.078891] I [MSGID: 114018]
> [client.c:2331:client_rpc_notify] 0-hdd-client-7: disconnected from
> hdd-client-7. Client process will keep trying to connect to glusterd until
> brick's port is available*
> *[2022-03-24 10:15:44.079954] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc6c0cadadb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7fc6c019f7e4] (-->
> /lib64/libgfrpc.so.0(+0xd8fe)[0x7fc6c019f8fe] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7fc6c01a0987] (-->
> /lib64/libgfrpc.so.0(+0xf518)[0x7fc6c01a1518] ))))) 0-hdd-client-7: forced
> unwinding frame type(GF-DUMP) op(DUMP(1)) called at 2022-03-24
> 10:15:33.209640 (xid=0x5)*
> *[2022-03-24 10:15:44.080008] W [MSGID: 114032]
> [client-handshake.c:1547:client_dump_version_cbk] 0-hdd-client-7: received
> RPC status error [Transport endpoint is not connected]*
> *[2022-03-24 10:15:44.080526] E [rpc-clnt.c:346:saved_frames_unwind] (-->
> /lib64/libglusterfs.so.0(_gf_log_callingfn+0x13b)[0x7fc6c0cadadb] (-->
> /lib64/libgfrpc.so.0(+0xd7e4)[0x7fc6c019f7e4] (-->
> /lib64/libgfrpc.so.0(+0xd8fe)[0x7fc6c019f8fe] (-->
> /lib64/libgfrpc.so.0(rpc_clnt_connection_cleanup+0x97)[0x7fc6c01a0987] (-->
> /lib64/libgfrpc.so.0(+0xf518)[0x7fc6c01a1518] ))))) 0-hdd-client-7: forced
> unwinding frame type(GF-DUMP) op(NULL(2)) called at 2022-03-24
> 10:15:33.209655 (xid=0x6)*
> *[2022-03-24 10:15:44.080574] W [rpc-clnt-ping.c:210:rpc_clnt_ping_cbk]
> 0-hdd-client-7: socket disconnected*
>
> ####################################
> # stack trace on storage2 that happened this morning
>
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr000:115974 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr000 D ffff9b91b8951070 0
> 115974 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>]
> _xfs_log_force_lsn+0x2d1/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr001:121353 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr001 D ffff9b9b7d4dac80 0
> 121353 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>]
> _xfs_log_force_lsn+0x2d1/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr002:121354 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr002 D ffff9b9b7d75ac80 0
> 121354 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>]
> _xfs_log_force_lsn+0x2d1/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr003:121355 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr003 D ffff9b9b7d51ac80 0
> 121355 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>]
> schedule_timeout+0x221/0x2d0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ?
> ttwu_do_wakeup+0x19/0xe0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ?
> ttwu_do_activate+0x6f/0x80*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ?
> try_to_wake_up+0x190/0x390*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>]
> wait_for_completion+0xfd/0x140*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>]
> flush_work+0x10a/0x1b0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ?
> move_linked_works+0x90/0x90*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>]
> xlog_cil_force_lsn+0x8a/0x210 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>]
> _xfs_log_force_lsn+0x74/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa15bcb1f>] ?
> filemap_fdatawait_range+0x1f/0x30*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ?
> down_read+0x12/0x40*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr004:121356 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr004 D ffff9b9b7d75ac80 0
> 121356 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>]
> schedule_timeout+0x221/0x2d0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ?
> ttwu_do_wakeup+0x19/0xe0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ?
> ttwu_do_activate+0x6f/0x80*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ?
> try_to_wake_up+0x190/0x390*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>]
> wait_for_completion+0xfd/0x140*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>]
> flush_work+0x10a/0x1b0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ?
> move_linked_works+0x90/0x90*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>]
> xlog_cil_force_lsn+0x8a/0x210 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>]
> _xfs_log_force_lsn+0x74/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa15bcb1f>] ?
> filemap_fdatawait_range+0x1f/0x30*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ?
> down_read+0x12/0x40*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr005:153774 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr005 D ffff9b9b7d61ac80 0
> 153774 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>]
> schedule_timeout+0x221/0x2d0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ?
> ttwu_do_wakeup+0x19/0xe0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ?
> ttwu_do_activate+0x6f/0x80*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ?
> try_to_wake_up+0x190/0x390*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>]
> wait_for_completion+0xfd/0x140*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>]
> flush_work+0x10a/0x1b0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ?
> move_linked_works+0x90/0x90*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>]
> xlog_cil_force_lsn+0x8a/0x210 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ?
> getxattr+0x11b/0x180*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>]
> _xfs_log_force_lsn+0x74/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ?
> down_read+0x12/0x40*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr006:153775 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr006 D ffff9b9b7d49ac80 0
> 153775 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>]
> schedule_timeout+0x221/0x2d0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ?
> ttwu_do_wakeup+0x19/0xe0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d78df>] ?
> ttwu_do_activate+0x6f/0x80*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ?
> try_to_wake_up+0x190/0x390*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>]
> wait_for_completion+0xfd/0x140*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>]
> flush_work+0x10a/0x1b0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ?
> move_linked_works+0x90/0x90*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>]
> xlog_cil_force_lsn+0x8a/0x210 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ?
> getxattr+0x11b/0x180*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>]
> _xfs_log_force_lsn+0x74/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ?
> down_read+0x12/0x40*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr007:153776 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr007 D ffff9b9958c962a0 0
> 153776 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7e531>]
> schedule_timeout+0x221/0x2d0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d7782>] ?
> check_preempt_curr+0x92/0xa0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14d77a9>] ?
> ttwu_do_wakeup+0x19/0xe0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db210>] ?
> try_to_wake_up+0x190/0x390*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80ddd>]
> wait_for_completion+0xfd/0x140*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14be9aa>]
> flush_work+0x10a/0x1b0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14bb6c0>] ?
> move_linked_works+0x90/0x90*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05070ba>]
> xlog_cil_force_lsn+0x8a/0x210 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167335b>] ?
> getxattr+0x11b/0x180*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc0505484>]
> _xfs_log_force_lsn+0x74/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b7fd22>] ?
> down_read+0x12/0x40*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr008:153777 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr008 D ffff9b9b7d61ac80 0
> 153777 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>]
> _xfs_log_force_lsn+0x2d1/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> *Mar 24 06:24:06 storage2 kernel: INFO: task glfs_iotwr009:153778 blocked
> for more than 120 seconds.*
> *Mar 24 06:24:06 storage2 kernel: "echo 0 >
> /proc/sys/kernel/hung_task_timeout_secs" disables this message.*
> *Mar 24 06:24:06 storage2 kernel: glfs_iotwr009 D ffff9b9958c920e0 0
> 153778 1 0x00000080*
> *Mar 24 06:24:06 storage2 kernel: Call Trace:*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b80a29>] schedule+0x29/0x70*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc05056e1>]
> _xfs_log_force_lsn+0x2d1/0x310 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa14db4d0>] ?
> wake_up_state+0x20/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffc04e5a3d>]
> xfs_file_fsync+0xfd/0x1c0 [xfs]*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167fbf7>] do_fsync+0x67/0xb0*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa167ff03>]
> SyS_fdatasync+0x13/0x20*
> *Mar 24 06:24:06 storage2 kernel: [<ffffffffa1b8dede>]
> system_call_fastpath+0x25/0x2a*
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220328/673a4e68/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 22704 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20220328/673a4e68/attachment.png>
More information about the Gluster-users
mailing list