<div dir="ltr"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px">Hi, </div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px"><br></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px">I have one more question about the Gluster linear scale-out performance regarding the "write-behind off" case specifically -- when "write-behind" is off, and still the stripe volumes and other settings as early thread posted, the <span style="font-size:small">storage I/O seems not to relate to the number of storage nodes. In my experiment, no matter I have 2 brick server nodes or 8 brick server nodes, the aggregated gluster I/O performance is ~100MB/sec. And fio benchmark measurement gives the same result. If "write behind" is on, then the storage performance is linear scale-out along with the # of brick server nodes increasing. </span></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px"><span style="font-size:small"><br></span></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif">No matter the write behind option is on/off, I thought the gluster I/O performance should be pulled and aggregated together as a whole. If that is the case, why do I get a consistent gluster performance (~100MB/sec) when "write behind" is off? Please advise me if I misunderstood something. </div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif"><br></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif">Thanks,</div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif">Qing </div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif"><br></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif"><br></div></div></div></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 21, 2020 at 7:29 PM Qing Wang <<a href="mailto:qw@g.clemson.edu">qw@g.clemson.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px">fio gives me the correct linear scale-out results, and you're right, the storage cache is the root cause that makes the dd measurement results not accurate at all. </div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px"><br></div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px">Thanks,</div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px">Qing </div></div></div></div></div></div></div></div></div></div></div></div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 21, 2020 at 2:53 PM Yaniv Kaul <<a href="mailto:ykaul@redhat.com" target="_blank">ykaul@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 21 Jul 2020, 21:43 Qing Wang <<a href="mailto:qw@g.clemson.edu" target="_blank">qw@g.clemson.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Yaniv,<div><br></div><div>Thanks for the quick response. I forget to mention I am testing the writing performance, not reading. In this case, would the client cache hit rate still be a big issue? </div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">It's not hitting the storage directly. Since it's also single threaded, it may also not saturate it. I highly recommend testing properly. </div><div dir="auto">Y. </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br></div><div>I'll use fio to run my test once again, thanks for the suggestion. </div><div><br></div><div>Thanks,</div><div>Qing </div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <<a href="mailto:ykaul@redhat.com" rel="noreferrer" target="_blank">ykaul@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 21 Jul 2020, 21:30 Qing Wang <<a href="mailto:qw@g.clemson.edu" rel="noreferrer" target="_blank">qw@g.clemson.edu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px">Hi, </div><div style="margin:0px;padding:0px;border:0px;font-family:Arial,Helvetica,sans-serif;font-size:13px"><br></div><div style="margin:0px;padding:0px;border:0px">I am trying to test Gluster linear scale-out performance by adding more storage server/bricks, and measure the storage I/O performance. To vary the storage server number, I create several "stripe" volumes that contain 2 brick servers, 3 brick servers, 4 brick servers, and so on. On gluster client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g bs=1M count=26000" to create 26G data (or larger size), and those data will be distributed to the corresponding gluster servers (each has gluster brick on it) and "dd" returns the final I/O throughput. The Internet is 40G infiniband, although I didn't do any specific configurations to use advanced features. <br></div></div></div></div></div></div></div></div></div></div></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">Your dd command is inaccurate, as it'll hit the client cache. It is also single threaded. I suggest switching to fio. </div><div dir="auto">Y. </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="margin:0px;padding:0px;border:0px"></div><div style="margin:0px;padding:0px;border:0px"><br></div><div style="margin:0px;padding:0px;border:0px">What confuses me is that the storage I/O seems not to relate to the number of storage nodes, but Gluster documents said it should be linear scaling. For example, when "write-behind" is on, and when Infiniband "jumbo frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no matter I have 2 brick servers or 8 brick servers -- for 2 server case, each server can have ~400 MB/sec; for 4 server case, each server can have ~200MB/sec. That said, each server I/O does aggregate to the final storage I/O (800 MB/sec), but this is not "linear scale-out". </div><div style="margin:0px;padding:0px;border:0px"><br></div><div style="margin:0px;padding:0px;border:0px">Can somebody help me to understand why this is the case? I certainly can have some misunderstanding/misconfiguration here. Please correct me if I do, thanks! </div><div style="margin:0px;padding:0px;border:0px"><br></div><div style="margin:0px;padding:0px;border:0px">Best,</div><div style="margin:0px;padding:0px;border:0px">Qing</div></div></div></div></div></div></div></div></div></div></div></div>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://bluejeans.com/441850968" rel="noreferrer noreferrer noreferrer" target="_blank">https://bluejeans.com/441850968</a><br>
<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" rel="noreferrer noreferrer" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div></div></div>
</blockquote></div>
</blockquote></div>