<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=koi8-r">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Dmitry:<br>
<br>
Is there a way to check and see if the GlusterFS write requests is being routed through the network interface?<br>
<br>
I am asking this because of your bricks/host definition as you showed below.<br>
<br>
Thanks.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Sincerely,</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Ewen<br>
</div>
<div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> gluster-users-bounces@gluster.org <gluster-users-bounces@gluster.org> on behalf of Strahil Nikolov <hunter86_bg@yahoo.com><br>
<b>Sent:</b> November 25, 2020 12:42 PM<br>
<b>To:</b> Dmitry Antipov <dmantipov@yandex.ru><br>
<b>Cc:</b> gluster-users <gluster-users@gluster.org><br>
<b>Subject:</b> Re: [Gluster-users] Poor performance on a server-class system vs. desktop</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">Having the same performance on 2 very fast disks indicate that you are<br>
hitting a limit.<br>
You can start with this article: <br>
<a href="https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements">https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements</a><br>
<br>
Most probably increasing the performance.io-thread-count could help.<br>
<br>
<br>
Best Regards,<br>
Strahil Nikolov<br>
<br>
<br>
В 19:08 +0300 на 25.11.2020 (ср), Dmitry Antipov написа:<br>
> I'm trying to investigate the poor I/O performance results<br>
> observed on a server-class system vs. the desktop-class one.<br>
> <br>
> The second one is 8-core notebook with NVME disk. According to<br>
> <br>
> fio --name test --filename=XXX --bs=4k --rw=randwrite --<br>
> ioengine=libaio --direct=1 \<br>
> --iodepth=128 --numjobs=1 --runtime=60 --time_based=1<br>
> <br>
> this disk is able to perform 4K random writes at ~100K IOPS. When I<br>
> create the<br>
> glusterfs volume using the same disk as backing store:<br>
> <br>
> Volume Name: test1<br>
> Type: Replicate<br>
> Volume ID: 87bad2a9-7a4a-43fc-94d2-de72965b63d6<br>
> Status: Started<br>
> Snapshot Count: 0<br>
> Number of Bricks: 1 x 3 = 3<br>
> Transport-type: tcp<br>
> Bricks:<br>
> Brick1: 192.168.1.112:/glusterfs/test1-000<br>
> Brick2: 192.168.1.112:/glusterfs/test1-001<br>
> Brick3: 192.168.1.112:/glusterfs/test1-002<br>
> Options Reconfigured:<br>
> storage.fips-mode-rchecksum: on<br>
> transport.address-family: inet<br>
> nfs.disable: on<br>
> performance.client-io-threads: off<br>
> <br>
> and run:<br>
> <br>
> [global]<br>
> name=ref-write<br>
> filename=testfile<br>
> ioengine=gfapi_async<br>
> volume=test1<br>
> brick=localhost<br>
> create_on_open=1<br>
> rw=randwrite<br>
> direct=1<br>
> numjobs=1<br>
> time_based=1<br>
> runtime=60<br>
> <br>
> [test-4-kbytes]<br>
> bs=4k<br>
> size=1G<br>
> iodepth=128<br>
> <br>
> I'm seeing ~10K IOPS. So adding an extra layer (glusterfs :-) between<br>
> an I/O client<br>
> (fio in this case) and NVME disk introduces ~10x overhead. Maybe<br>
> worse than expected,<br>
> but the things goes even worse when I'm switching to the server.<br>
> <br>
> The server is 32-core machine with NVME disk capable to serve the<br>
> same I/O pattern<br>
> at ~200K IOPS. I've expected something similar to linear scalability,<br>
> i.e. ~20K<br>
> IOPS then running the same fio workload on a gluster volume. But I<br>
> surprisingly<br>
> got something very close to the same ~10K IOPS as seen on the<br>
> desktop-class machine.<br>
> So, here is ~20x overhead vs. ~10x one on the desktop.<br>
> <br>
> The OSes are different (Fedora Core 33 on a notebook and relatively<br>
> old Debian 9 on<br>
> server), but both systems runs the fairly recent 5.9.x kernels<br>
> (without massive tricky<br>
> tuning via sysctl or similar methods) and glusterfs 8.2, using XFS as<br>
> the filesystem<br>
> under the bricks.<br>
> <br>
> I would greatly appreciate any ideas on debugging this.<br>
> <br>
> Dmitry<br>
> ________<br>
> <br>
> <br>
> <br>
> Community Meeting Calendar:<br>
> <br>
> Schedule -<br>
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
> Bridge: <a href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><br>
> Gluster-users mailing list<br>
> Gluster-users@gluster.org<br>
> <a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
Gluster-users@gluster.org<br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</span></font></div>
</div>
</body>
</html>