<div dir="ltr"><div><div><div>Hi Krist,<br><br></div>What are your volume options on that setup? Have you tried tuning it for the kind of workload and files size you have?</div><div><br></div><div>I would definitely do some tests with feature.shard=on/off first. If shard is on, try playing with features.shard-block-size.</div><div>Do you have jumbo frames (MTU=9000) enabled across the switch and nodes? if you have concurrent clients writing/reading, it could be beneficial to increase the number of client and server threads as well, try setting higher values for client.event-threads and server.event-threads.<br></div><div><br></div><div>Best regards,</div><div>Everton Brogliatto<br></div><div><br></div></div><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 24, 2017 at 7:48 PM, Krist van Besien <span dir="ltr"><<a href="mailto:krist@redhat.com" target="_blank">krist@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div><div><div><div><div><div><div><div>Hi all,<br><br></div>I usualy advise clients to use the native client if at all possible, as it is very robust. But I am running in to problems here.<br><br></div>In this case the gluster system is used to store video streams. Basicaly the setup is the following:<br></div>- A gluster cluster of 3 nodes, with ample storage. They export several volumes.</div><div>- The network is 10GB, switched.<br></div>- A "recording server" which subscribes to multi cast video streams, and records them to disk. The recorder writes the streams in 10s blocks, so when it is for example recording 50 streams it is creating 5 files a second, each about 5M. it uses a write-then-rename process.<br><br></div>I simulated that with a small script, that wrote 5M files and renamed them as fast as it could, and could easily create around 100 files/s (which abouts saturates the network). So I think the cluster is up to the task.<br><br></div>However if we try the actualy workload we run in to trouble. Running the recorder software we can gradually ramp up the number of streams it records (and thus the number of files it creates), and at arou d 50 streams the recorder eventually stops writing files. According to the programmers that wrote it, it appears that it can no longer get the needed locks¸ and as a result just stops writing. <br><br></div>We decided to test using the NFS client as well, and there the problem does not exist. But again, I (and the customer) would prefer not to use NFS, but use the native client in stead.<br><br></div>So if the problem is file locking, and the problem exists with the native client, and not using NFS, what could be the cause? <br><br></div>In what way do locking differ between the two different file systems, between NFS and Fuse, and how can the programmers work around any issues the fuse client might be causing?<br><br></div>This video stream software is a bespoke solution, developped in house and it is thus possible to change the way it handles files so it works with the native client, but the programmers are looking at me for guidance.<br><br></div>Any suggestions?<br><br></div>Krist<br><br clear="all"><div><div><div><div><div><div><div><div><div><div><div><div><div><div><br></div><div><br></div><div><br>-- <br><div class="m_-3123396753398865405gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span><font color="#888888"><span></span></font></span><font style="color:rgb(0,0,0)" face="Liberation Sans">Vriendelijke Groet | Best Regards | </font><span style="color:rgb(0,0,0);font-family:'Liberation Sans'">Freundliche Grüße | </span><font style="color:rgb(0,0,0)" face="Liberation Sans" color="#000000">Cordialement</font><span><font color="#888888"><span><br><div><hr style="color:rgb(0,0,0)" width="100%" size="2"><br><p style="font-weight:bold;margin:0;padding:0;font-size:14px;text-transform:uppercase;margin-bottom:0"><span>Krist</span> <span>van Besien</span></p>
<p style="font-weight:normal;font-size:10px;margin:0px 0px 4px;text-transform:uppercase"><span>senior architect</span><span style="color:rgb(204,204,204)">, <span style="font-weight:normal;color:#aaa;margin:0">RHCE, RHCSA Open Stack </span></span></p>
<p style="font-weight:normal;margin:0;font-size:10px;color:#999"><a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="https://www.redhat.com" target="_blank">Red Hat <span>Red Hat Switzerland S.A.</span></a></p>
<p style="font-weight:normal;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span style="margin:0px;padding:0px">
<a style="color:#0088ce;font-size:10px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" href="mailto:krist@redhat.com" target="_blank">krist@redhat.com</a> </span>
<span>M: <a href="tel:+41-79-5936260" style="color:#0088ce;font-size:11px;margin:0;text-decoration:none;font-family:'overpass',sans-serif" target="_blank">+41-79-5936260</a> </span>
</p>
<a href="https://red.ht/sig" target="_blank"> <img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a>
<div><a href="https://redhat.com/trusted" style="text-decoration:none;color:#c00;font-weight:bold" target="_blank">TRIED. TESTED. TRUSTED.</a></div></div></span></font></span></div></div></div></div></div></div></div></div></div></div>
</div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>