[Gluster-users] gluster stuck when trying to list a successful mount
Ilan Schwarts
ilan84 at gmail.com
Mon Aug 7 14:18:03 UTC 2017
Hi all,
My infrastructure is GlusterFS with 2 Nodes:
L137B-GlusterFS-Node1.L137B-root.com
L137B-GlusterFS-Node2.L137B-root.com
I have followed this guide:
http://www.itzgeek.com/how-tos/linux/centos-how-tos/install-and-configure-glusterfs-on-centos-7-rhel-7.html/2
I have created a device, formatted as ext4, mounted it.
The next step was to install gluster FS.
I installed glusterfs 3.10.3, created a volume and started it
[root at L137B-GlusterFS-Node2 someuser]# gluster volume info
Volume Name: gv0
Type: Replicate
Volume ID: a606f77b-c0df-427c-99ec-cee98b3ecd71
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: L137B-GlusterFS-Node1.L137B-root.com:/mnt/gluster/gv0
Brick2: L137B-GlusterFS-Node2.L137B-root.com:/mnt/gluster/gv0
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
Now when I try to mount to Node1 from Node2, with the command:
mount -t glusterfs L137B-GlusterFS-Node2.L137B-root.com:/gv0 /mnt
L137B-GlusterFS-Node2.L137B-root.com:/gv0 on /mnt type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,default_permissions,allow_other,max_read=131072)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
it returns ok, no errors, but when i do: ls -la /mnt the console get stucked.
on dmesg -T i get (from a duplicate session on another console):
[Mon Aug 7 17:05:58 2017] fuse init (API version 7.22)
[Mon Aug 7 17:09:32 2017] INFO: task glusterfsd:3273 blocked for more
than 120 seconds.
[Mon Aug 7 17:09:32 2017] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Aug 7 17:09:32 2017] glusterfsd D ffff880092e380b0 0
3273 1 0x00000080
[Mon Aug 7 17:09:32 2017] ffff8800a1e3bb70 0000000000000086
ffff880139b7af10 ffff8800a1e3bfd8
[Mon Aug 7 17:09:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8
ffff880139b7af10 ffff880092e380a8
[Mon Aug 7 17:09:32 2017] ffff880092e380ac ffff880139b7af10
00000000ffffffff ffff880092e380b0
[Mon Aug 7 17:09:32 2017] Call Trace:
[Mon Aug 7 17:09:32 2017] [<ffffffff8168d259>]
schedule_preempt_disabled+0x29/0x70
[Mon Aug 7 17:09:32 2017] [<ffffffff8168aeb5>]
__mutex_lock_slowpath+0xc5/0x1c0
[Mon Aug 7 17:09:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140
[Mon Aug 7 17:09:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f
[Mon Aug 7 17:09:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7
[Mon Aug 7 17:09:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0
[Mon Aug 7 17:09:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0
[Mon Aug 7 17:09:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0
[Mon Aug 7 17:09:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280
[Mon Aug 7 17:09:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0
[Mon Aug 7 17:09:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0
[Mon Aug 7 17:09:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0
[Mon Aug 7 17:09:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0
[Mon Aug 7 17:09:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160
[Mon Aug 7 17:09:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20
[Mon Aug 7 17:09:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0
[Mon Aug 7 17:09:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60
[Mon Aug 7 17:09:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180
[Mon Aug 7 17:09:32 2017] [<ffffffff8111f426>] ?
__audit_syscall_exit+0x1e6/0x280
[Mon Aug 7 17:09:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10
[Mon Aug 7 17:09:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
[Mon Aug 7 17:11:32 2017] INFO: task glusterfsd:3273 blocked for more
than 120 seconds.
[Mon Aug 7 17:11:32 2017] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Aug 7 17:11:32 2017] glusterfsd D ffff880092e380b0 0
3273 1 0x00000080
[Mon Aug 7 17:11:32 2017] ffff8800a1e3bb70 0000000000000086
ffff880139b7af10 ffff8800a1e3bfd8
[Mon Aug 7 17:11:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8
ffff880139b7af10 ffff880092e380a8
[Mon Aug 7 17:11:32 2017] ffff880092e380ac ffff880139b7af10
00000000ffffffff ffff880092e380b0
[Mon Aug 7 17:11:32 2017] Call Trace:
[Mon Aug 7 17:11:32 2017] [<ffffffff8168d259>]
schedule_preempt_disabled+0x29/0x70
[Mon Aug 7 17:11:32 2017] [<ffffffff8168aeb5>]
__mutex_lock_slowpath+0xc5/0x1c0
[Mon Aug 7 17:11:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140
[Mon Aug 7 17:11:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f
[Mon Aug 7 17:11:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7
[Mon Aug 7 17:11:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0
[Mon Aug 7 17:11:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0
[Mon Aug 7 17:11:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0
[Mon Aug 7 17:11:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280
[Mon Aug 7 17:11:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0
[Mon Aug 7 17:11:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0
[Mon Aug 7 17:11:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0
[Mon Aug 7 17:11:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0
[Mon Aug 7 17:11:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160
[Mon Aug 7 17:11:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20
[Mon Aug 7 17:11:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0
[Mon Aug 7 17:11:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60
[Mon Aug 7 17:11:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180
[Mon Aug 7 17:11:32 2017] [<ffffffff8111f426>] ?
__audit_syscall_exit+0x1e6/0x280
[Mon Aug 7 17:11:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10
[Mon Aug 7 17:11:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
[Mon Aug 7 17:13:32 2017] INFO: task glusterfsd:3273 blocked for more
than 120 seconds.
[Mon Aug 7 17:13:32 2017] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Aug 7 17:13:32 2017] glusterfsd D ffff880092e380b0 0
3273 1 0x00000080
[Mon Aug 7 17:13:32 2017] ffff8800a1e3bb70 0000000000000086
ffff880139b7af10 ffff8800a1e3bfd8
[Mon Aug 7 17:13:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8
ffff880139b7af10 ffff880092e380a8
[Mon Aug 7 17:13:32 2017] ffff880092e380ac ffff880139b7af10
00000000ffffffff ffff880092e380b0
[Mon Aug 7 17:13:32 2017] Call Trace:
[Mon Aug 7 17:13:32 2017] [<ffffffff8168d259>]
schedule_preempt_disabled+0x29/0x70
[Mon Aug 7 17:13:32 2017] [<ffffffff8168aeb5>]
__mutex_lock_slowpath+0xc5/0x1c0
[Mon Aug 7 17:13:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140
[Mon Aug 7 17:13:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f
[Mon Aug 7 17:13:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7
[Mon Aug 7 17:13:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0
[Mon Aug 7 17:13:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0
[Mon Aug 7 17:13:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0
[Mon Aug 7 17:13:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280
[Mon Aug 7 17:13:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0
[Mon Aug 7 17:13:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0
[Mon Aug 7 17:13:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0
[Mon Aug 7 17:13:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0
[Mon Aug 7 17:13:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160
[Mon Aug 7 17:13:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20
[Mon Aug 7 17:13:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0
[Mon Aug 7 17:13:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60
[Mon Aug 7 17:13:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180
[Mon Aug 7 17:13:32 2017] [<ffffffff8111f426>] ?
__audit_syscall_exit+0x1e6/0x280
[Mon Aug 7 17:13:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10
[Mon Aug 7 17:13:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
[Mon Aug 7 17:15:32 2017] INFO: task glusterfsd:3273 blocked for more
than 120 seconds.
[Mon Aug 7 17:15:32 2017] "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Mon Aug 7 17:15:32 2017] glusterfsd D ffff880092e380b0 0
3273 1 0x00000080
[Mon Aug 7 17:15:32 2017] ffff8800a1e3bb70 0000000000000086
ffff880139b7af10 ffff8800a1e3bfd8
[Mon Aug 7 17:15:32 2017] ffff8800a1e3bfd8 ffff8800a1e3bfd8
ffff880139b7af10 ffff880092e380a8
[Mon Aug 7 17:15:32 2017] ffff880092e380ac ffff880139b7af10
00000000ffffffff ffff880092e380b0
[Mon Aug 7 17:15:32 2017] Call Trace:
[Mon Aug 7 17:15:32 2017] [<ffffffff8168d259>]
schedule_preempt_disabled+0x29/0x70
[Mon Aug 7 17:15:32 2017] [<ffffffff8168aeb5>]
__mutex_lock_slowpath+0xc5/0x1c0
[Mon Aug 7 17:15:32 2017] [<ffffffff81208e17>] ? unlazy_walk+0x87/0x140
[Mon Aug 7 17:15:32 2017] [<ffffffff8168a31f>] mutex_lock+0x1f/0x2f
[Mon Aug 7 17:15:32 2017] [<ffffffff81683d93>] lookup_slow+0x33/0xa7
[Mon Aug 7 17:15:32 2017] [<ffffffff8120bfef>] link_path_walk+0x80f/0x8b0
[Mon Aug 7 17:15:32 2017] [<ffffffff810b4866>] ? __remove_hrtimer+0x46/0xe0
[Mon Aug 7 17:15:32 2017] [<ffffffff8120c1eb>] path_lookupat+0x6b/0x7a0
[Mon Aug 7 17:15:32 2017] [<ffffffff810f5fb3>] ? futex_wait+0x1a3/0x280
[Mon Aug 7 17:15:32 2017] [<ffffffff811de5e5>] ? kmem_cache_alloc+0x35/0x1e0
[Mon Aug 7 17:15:32 2017] [<ffffffff8120f23f>] ? getname_flags+0x4f/0x1a0
[Mon Aug 7 17:15:32 2017] [<ffffffff8120c94b>] filename_lookup+0x2b/0xc0
[Mon Aug 7 17:15:32 2017] [<ffffffff81210367>] user_path_at_empty+0x67/0xc0
[Mon Aug 7 17:15:32 2017] [<ffffffff810f5570>] ? futex_wake+0x80/0x160
[Mon Aug 7 17:15:32 2017] [<ffffffff812103d1>] user_path_at+0x11/0x20
[Mon Aug 7 17:15:32 2017] [<ffffffff81203843>] vfs_fstatat+0x63/0xc0
[Mon Aug 7 17:15:32 2017] [<ffffffff81203e11>] SYSC_newlstat+0x31/0x60
[Mon Aug 7 17:15:32 2017] [<ffffffff810f85c0>] ? SyS_futex+0x80/0x180
[Mon Aug 7 17:15:32 2017] [<ffffffff8111f426>] ?
__audit_syscall_exit+0x1e6/0x280
[Mon Aug 7 17:15:32 2017] [<ffffffff8120409e>] SyS_newlstat+0xe/0x10
[Mon Aug 7 17:15:32 2017] [<ffffffff81697189>] system_call_fastpath+0x16/0x1b
More information about the Gluster-users
mailing list