[Gluster-users] gluster performance issue

[paf1 at email.cz]

Hello,
can anybody help with gluster performace ?

I'm running gluster replica 3 arbiter 1 ( 2+1) with following version

OS Version:RHEL - 7 - 3.1611.el7.centos
OS Description:CentOS Linux 7 (Core)
Kernel Version:3.10.0 - 514.6.1.el7.x86_64
KVM Version:2.6.0 - 28.el7_3.3.1
LIBVIRT Version:libvirt-2.0.0-10.el7_3.4
VDSM Version:vdsm-4.18.21-1.el7.centos
SPICE Version:0.12.4 - 19.el7
GlusterFS Version:glusterfs-3.7.19-1.el7
CEPH Version:librbd1-0.94.5-1.el7

After huge gluster load ( runnig DB + 2x import over 10Gb Lan ) system 
generate messages to "messages"
and dramaticaly goes down, even node rebooted .

Jan 28 09:45:19 1kvm2 kernel: INFO: task worker:11384 blocked for more 
than 120 seconds.
Jan 28 09:45:19 1kvm2 kernel: "echo 0 > 
/proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 28 09:45:19 1kvm2 kernel: worker          D ffff880167566dd0     0 
11384      1 0x00000080
Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76fa60 0000000000000086 
ffff882fba75de20 ffff8807ab76ffd8
Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76ffd8 ffff8807ab76ffd8 
ffff882fba75de20 ffff882fbf396c40
Jan 28 09:45:19 1kvm2 kernel: 0000000000000000 7fffffffffffffff 
ffff882fa8ee2500 ffff8807ab76fd60
Jan 28 09:45:19 1kvm2 kernel: Call Trace:
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8168b899>] schedule+0x29/0x70
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff816892e9>] 
schedule_timeout+0x239/0x2d0
Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa0889ea2>] ? 
fuse_direct_io+0x492/0x820 [fuse]
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8168ae3e>] 
io_schedule_timeout+0xae/0x130
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8168aed8>] io_schedule+0x18/0x20
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8124d945>] 
wait_on_sync_kiocb+0x35/0x80
Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa088b091>] 
fuse_direct_IO+0x231/0x380 [fuse]
Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa088d1f8>] ? 
fuse_change_attributes_common+0x88/0x120 [fuse]
Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa088d32f>] ? 
fuse_change_attributes+0x9f/0x100 [fuse]
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff8118283c>] 
generic_file_aio_read+0x70c/0x790
Jan 28 09:45:19 1kvm2 kernel: [<ffffffffa0887d78>] 
fuse_file_aio_read+0x78/0xb0 [fuse]
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811fdad9>] 
do_sync_readv_writev+0x79/0xd0
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811ff17e>] 
do_readv_writev+0xce/0x260
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff810f7bae>] ? do_futex+0xfe/0x5b0
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811ff342>] vfs_readv+0x32/0x60
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff811ff6b2>] SyS_preadv+0xc2/0xf0
Jan 28 09:45:19 1kvm2 kernel: [<ffffffff816967c9>] 
system_call_fastpath+0x16/0x1b
Jan 28 09:50:01 1kvm2 systemd: Started Session 58 of user root.
Jan 28 09:50:01 1kvm2 systemd: Starting Session 58 of user root.


I tried to set kernel params via web workaround recommendations, but 
without any success
( current /etc/sysctl.conf settings )
vm.swappiness=1
vm.vfs_cache_pressure=100
vm.min_free_kbytes = 512000
vm.zone_reclaim_mode = 1

Any idea ??
regs.
Paf1
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170128/c26a5f28/attachment.html>