<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000066">
<font face="Ubuntu">Hello, <br>
can anybody help with gluster performace ?<br>
<br>
I'm running gluster replica 3 arbiter 1 ( 2+1) with following
version <br>
</font><br>
<div class="GPTSHQKCD0D">
<div>
<div class="GPTSHQKCHHD container-fluid GPTSHQKCHUD">
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">OS Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row0_value"
class="GPTSHQKCJHD">RHEL - 7 - 3.1611.el7.centos</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">OS Description:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row1_value"
class="GPTSHQKCJHD">CentOS Linux 7 (Core)</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">Kernel Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row2_value"
class="GPTSHQKCJHD">3.10.0 - 514.6.1.el7.x86_64</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">KVM Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row3_value"
class="GPTSHQKCJHD">2.6.0 - 28.el7_3.3.1</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">LIBVIRT Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row4_value"
class="GPTSHQKCJHD">libvirt-2.0.0-10.el7_3.4</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">VDSM Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row5_value"
class="GPTSHQKCJHD">vdsm-4.18.21-1.el7.centos</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">SPICE Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row6_value"
class="GPTSHQKCJHD">0.12.4 - 19.el7</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">GlusterFS Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row7_value"
class="GPTSHQKCJHD">glusterfs-3.7.19-1.el7</span></div>
</div>
</div>
</div>
</div>
<div class="row">
<div class="col-md-12">
<div class="row">
<div class="col-md-2">
<div class="col-md-10">CEPH Version:<span
id="SubTabHostGeneralSoftwareView_formPanel_col0_row8_value"
class="GPTSHQKCJHD">librbd1-0.94.5-1.el7<br>
<br>
After huge gluster load ( runnig DB + 2x import
over 10Gb Lan ) system generate messages to
"messages"<br>
and dramaticaly goes down, even node rebooted .<br>
<br>
Jan 28 09:45:19 1kvm2 kernel: INFO: task
worker:11384 blocked for more than 120 seconds.<br>
Jan 28 09:45:19 1kvm2 kernel: "echo 0 >
/proc/sys/kernel/hung_task_timeout_secs" disables
this message.<br>
Jan 28 09:45:19 1kvm2 kernel: worker D
ffff880167566dd0 0 11384 1 0x00000080<br>
Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76fa60
0000000000000086 ffff882fba75de20 ffff8807ab76ffd8<br>
Jan 28 09:45:19 1kvm2 kernel: ffff8807ab76ffd8
ffff8807ab76ffd8 ffff882fba75de20 ffff882fbf396c40<br>
Jan 28 09:45:19 1kvm2 kernel: 0000000000000000
7fffffffffffffff ffff882fa8ee2500 ffff8807ab76fd60<br>
Jan 28 09:45:19 1kvm2 kernel: Call Trace:<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff8168b899>] schedule+0x29/0x70<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff816892e9>]
schedule_timeout+0x239/0x2d0<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffffa0889ea2>] ?
fuse_direct_io+0x492/0x820 [fuse]<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff8168ae3e>]
io_schedule_timeout+0xae/0x130<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff8168aed8>] io_schedule+0x18/0x20<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff8124d945>]
wait_on_sync_kiocb+0x35/0x80<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffffa088b091>]
fuse_direct_IO+0x231/0x380 [fuse]<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffffa088d1f8>] ?
fuse_change_attributes_common+0x88/0x120 [fuse]<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffffa088d32f>] ?
fuse_change_attributes+0x9f/0x100 [fuse]<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff8118283c>]
generic_file_aio_read+0x70c/0x790<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffffa0887d78>]
fuse_file_aio_read+0x78/0xb0 [fuse]<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff811fdad9>]
do_sync_readv_writev+0x79/0xd0<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff811ff17e>]
do_readv_writev+0xce/0x260<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff810f7bae>] ? do_futex+0xfe/0x5b0<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff811ff342>] vfs_readv+0x32/0x60<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff811ff6b2>] SyS_preadv+0xc2/0xf0<br>
Jan 28 09:45:19 1kvm2 kernel:
[<ffffffff816967c9>]
system_call_fastpath+0x16/0x1b<br>
Jan 28 09:50:01 1kvm2 systemd: Started Session 58
of user root.<br>
Jan 28 09:50:01 1kvm2 systemd: Starting Session 58
of user root.<br>
<br>
<br>
I tried to set kernel params via web workaround
recommendations, but without any success<br>
( current /etc/sysctl.conf settings )<br>
vm.swappiness=1 <br>
vm.vfs_cache_pressure=100<br>
vm.min_free_kbytes = 512000<br>
vm.zone_reclaim_mode = 1<br>
<br>
Any idea ??<br>
regs.<br>
Paf1<br>
</span></div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>