<div dir="ltr">First - thanks for taking the time and providing the reply. <br><br>Due to a bad flu I was unable to verify till now but now saw we use fstream in our apps, to write the files. Assumed the buffer was large, but never tested it. <br>
<br>We have increased the fstream buffer to 100MB and will check if this helps (this specific test can only be scheduled for later as the cluster is busy).<br><br>A related question:<br>I was wondering if there is any "magic number" as to number of files per directory in Gluster volumes.<br>
Some of my directories contain an order of 10^5 files and I'm begininng to suspect this too (when I worked in local FS I didn't see a noticable degradation, though).<br><br>Thanks again!<br>Ayelet <br><br><div class="gmail_quote">
On Thu, Jan 17, 2013 at 5:28 PM, harry mangalam <span dir="ltr"><<a href="mailto:harry.mangalam@uci.edu" target="_blank">harry.mangalam@uci.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Just a guess, but how are the writes being done? If they're being written in<br>
zillions of tiny writes, then what you may be seeing is described here:<br>
<<a href="http://moo.nac.uci.edu/%7Ehjm/bduc/BDUC_USER_HOWTO.html#writeperfongl" target="_blank">http://moo.nac.uci.edu/~hjm/bduc/BDUC_USER_HOWTO.html#writeperfongl</a>><br>
and the following stanza on named pipes.<br>
<br>
This is often the case with the large files being used in NGS/HTS where the<br>
fasta/fastq files are composed of millions of short (60-100 char) lines of<br>
characters and are typically written line-by-line.<br>
<br>
hjm<br>
<div><div class="h5"><br>
<br>
<br>
On Wednesday, January 16, 2013 02:47:37 PM Ayelet Shemesh wrote:<br>
> Hi to all Gluster experts,<br>
><br>
> I have a cluster of 10 machines exposing a volume into which 12 other<br>
> machines do many writes of large files (~100-300MB each).<br>
> In general I'm very happy with gluster. It's a great solution, and is quite<br>
> stable (thanks for the great work!).<br>
><br>
> However, I have a problem which I was unable to solve yet, nor find any<br>
> solution to in the documentation or on this list archive.<br>
><br>
> When the client machines write locally, and then just copy the files they<br>
> created to the gluster mount - everything works great.<br>
> When the client machines write directly to the gluster mounted volume I get<br>
> a huge performance hit.<br>
> In one specific test case the difference was 20 minutes for the copy and 8<br>
> hours for the direct write.<br>
><br>
> I tried to set the iocache attributes of write-behind-window and<br>
> flush-behind, but to no avail.<br>
><br>
> I will very much appreciate your help in solving this problem.<br>
><br>
> Thanks,<br>
> Ayelet<br><br></div></div></blockquote></div></div>