<div dir="ltr"><div><div>Okay good. At least this validates my doubts. Handling O_SYNC in gluster NFS and fuse is a bit different.<br></div>When application opens a file with O_SYNC on fuse mount then each write syscall has to be written to disk as part of the syscall where as in case of NFS, there is no concept of open. NFS performs write though a handle saying it needs to be a synchronous write, so write() syscall is performed first then it performs fsync(). so an write on an fd with O_SYNC becomes write+fsync. I am suspecting that when multiple threads do this write+fsync() operation on the same file, multiple writes are batched together to be written do disk so the throughput on the disk is increasing is my guess.<br><br></div>Does it answer your doubts?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 10, 2017 at 9:35 PM, Pat Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<br>
Without the oflag=sync and only a single test of each, the FUSE is
going faster than NFS:<br>
<br>
FUSE:<br>
<tt>mseas-data2(dri_nascar)% dd if=/dev/zero count=4096 bs=1048576
of=zeros.txt conv=sync</tt><tt><br>
</tt><tt>4096+0 records in</tt><tt><br>
</tt><tt>4096+0 records out</tt><tt><br>
</tt><tt>4294967296 bytes (4.3 GB) copied, 7.46961 s, 575 MB/s</tt><tt><br>
</tt><tt><br>
<br>
</tt>NFS<br>
<tt>mseas-data2(HYCOM)% dd if=/dev/zero count=4096 bs=1048576
of=zeros.txt conv=sync</tt><tt><br>
</tt><tt>4096+0 records in</tt><tt><br>
</tt><tt>4096+0 records out</tt><tt><br>
</tt><tt>4294967296 bytes (4.3 GB) copied, 11.4264 s, 376 MB/s</tt><div><div class="h5"><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt>
<div class="m_7039630128565981365moz-cite-prefix">On 05/10/2017 11:53 AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Could you let me know the speed without oflag=sync
on both the mounts? No need to collect profiles.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at 9:17 PM, Pat
Haley <span dir="ltr"><<a href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Here is what I see now:<br>
<br>
<tt>[root@mseas-data2 ~]# gluster volume info</tt><span><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Volume Name: data-volume</tt><tt><br>
</tt><tt>Type: Distribute</tt><tt><br>
</tt><tt>Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
</tt><tt>Status: Started</tt><tt><br>
</tt><tt>Number of Bricks: 2</tt><tt><br>
</tt><tt>Transport-type: tcp</tt><tt><br>
</tt><tt>Bricks:</tt><tt><br>
</tt><tt>Brick1: mseas-data2:/mnt/brick1</tt><tt><br>
</tt><tt>Brick2: mseas-data2:/mnt/brick2</tt><tt><br>
</tt><tt>Options Reconfigured:</tt><tt><br>
</tt></span><tt>diagnostics.count-fop-hits: on</tt><tt><br>
</tt><tt>diagnostics.latency-measuremen<wbr>t: on</tt><tt><br>
</tt><tt>nfs.exports-auth-enable: on</tt><tt><br>
</tt><tt>diagnostics.brick-sys-log-leve<wbr>l: WARNING</tt><span><tt><br>
</tt><tt>performance.readdir-ahead: on</tt><tt><br>
</tt><tt>nfs.disable: on</tt><tt><br>
</tt><tt>nfs.export-volumes: off</tt><tt><br>
</tt><br>
<br>
<br>
</span>
<div>
<div class="m_7039630128565981365h5">
<div class="m_7039630128565981365m_5079054141158038028moz-cite-prefix">On
05/10/2017 11:44 AM, Pranith Kumar Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Is this the volume info you have?<br>
<br>
<pre>><i> [<a href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">root at mseas-data2</a> ~]# gluster volume info
</i>><i>
</i>><i> Volume Name: data-volume
</i>><i> Type: Distribute
</i>><i> Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18
</i>><i> Status: Started
</i>><i> Number of Bricks: 2
</i>><i> Transport-type: tcp
</i>><i> Bricks:
</i>><i> Brick1: mseas-data2:/mnt/brick1
</i>><i> Brick2: mseas-data2:/mnt/brick2
</i>><i> Options Reconfigured:
</i>><i> performance.readdir-ahead: on
</i>><i> nfs.disable: on
</i>><i> nfs.export-volumes: off
</i></pre>
</div>
I copied this from old thread from 2016. This is distribute
volume. Did you change any of the options in between?
</div>
</blockquote>
</div></div><span><pre class="m_7039630128565981365m_5079054141158038028moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_7039630128565981365m_5079054141158038028moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_7039630128565981365m_5079054141158038028moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</span></div>
</blockquote></div>
--
<div class="m_7039630128565981365gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="m_7039630128565981365moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a class="m_7039630128565981365moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="m_7039630128565981365moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>