<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
Hi Pranith,<br>
<br>
Not entirely sure (this isn't my area of expertise). I'll run your
answer by some other people who are more familiar with this.<br>
<br>
I am also uncertain about how to interpret the results when we also
add the dd tests writing to the /home area (no gluster, still on the
same machine)<br>
<ul>
<li>dd test without oflag=sync (rough average of multiple tests)<br>
</li>
<ul>
<li>gluster w/ fuse mount : 570 Mb/s</li>
<li>gluster w/ nfs mount: 390 Mb/s</li>
<li>nfs (no gluster): 1.2 Gb/s</li>
</ul>
<li>dd test with oflag=sync (rough average of multiple tests)</li>
<ul>
<li>gluster w/ fuse mount: 5 Mb/s</li>
<li>gluster w/ nfs mount: 200 Mb/s</li>
<li>nfs (no gluster): 20 Mb/s<br>
</li>
</ul>
</ul>
Given that the non-gluster area is a RAID-6 of 4 disks while each
brick of the gluster area is a RAID-6 of 32 disks, I would naively
expect the writes to the gluster area to be roughly 8x faster than
to the non-gluster.<br>
<br>
I still think we have a speed issue, I can't tell if fuse vs nfs is
part of the problem. Was there anything useful in the profiles?<br>
<br>
Pat<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/10/2017 12:15 PM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnZf9pQww1j+nM=2didLtK2V79JPxte6xbdAZ_rw03sATg@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div dir="ltr">
<div>
<div>Okay good. At least this validates my doubts. Handling
O_SYNC in gluster NFS and fuse is a bit different.<br>
</div>
When application opens a file with O_SYNC on fuse mount then
each write syscall has to be written to disk as part of the
syscall where as in case of NFS, there is no concept of open.
NFS performs write though a handle saying it needs to be a
synchronous write, so write() syscall is performed first then
it performs fsync(). so an write on an fd with O_SYNC becomes
write+fsync. I am suspecting that when multiple threads do
this write+fsync() operation on the same file, multiple writes
are batched together to be written do disk so the throughput
on the disk is increasing is my guess.<br>
<br>
</div>
Does it answer your doubts?<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at 9:35 PM, Pat
Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Without the oflag=sync and only a single test of each, the
FUSE is going faster than NFS:<br>
<br>
FUSE:<br>
<tt>mseas-data2(dri_nascar)% dd if=/dev/zero count=4096
bs=1048576 of=zeros.txt conv=sync</tt><tt><br>
</tt><tt>4096+0 records in</tt><tt><br>
</tt><tt>4096+0 records out</tt><tt><br>
</tt><tt>4294967296 bytes (4.3 GB) copied, 7.46961 s, 575
MB/s</tt><tt><br>
</tt><tt><br>
<br>
</tt>NFS<br>
<tt>mseas-data2(HYCOM)% dd if=/dev/zero count=4096
bs=1048576 of=zeros.txt conv=sync</tt><tt><br>
</tt><tt>4096+0 records in</tt><tt><br>
</tt><tt>4096+0 records out</tt><tt><br>
</tt><tt>4294967296 bytes (4.3 GB) copied, 11.4264 s, 376
MB/s</tt>
<div>
<div class="h5"><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt>
<div class="m_7039630128565981365moz-cite-prefix">On
05/10/2017 11:53 AM, Pranith Kumar Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Could you let me know the speed
without oflag=sync on both the mounts? No need to
collect profiles.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at
9:17 PM, Pat Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0
0 0 .8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Here is what I see now:<br>
<br>
<tt>[root@mseas-data2 ~]# gluster volume
info</tt><span><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Volume Name: data-volume</tt><tt><br>
</tt><tt>Type: Distribute</tt><tt><br>
</tt><tt>Volume ID:
c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
</tt><tt>Status: Started</tt><tt><br>
</tt><tt>Number of Bricks: 2</tt><tt><br>
</tt><tt>Transport-type: tcp</tt><tt><br>
</tt><tt>Bricks:</tt><tt><br>
</tt><tt>Brick1: mseas-data2:/mnt/brick1</tt><tt><br>
</tt><tt>Brick2: mseas-data2:/mnt/brick2</tt><tt><br>
</tt><tt>Options Reconfigured:</tt><tt><br>
</tt></span><tt>diagnostics.count-fop-hits:
on</tt><tt><br>
</tt><tt>diagnostics.latency-measuremen<wbr>t:
on</tt><tt><br>
</tt><tt>nfs.exports-auth-enable: on</tt><tt><br>
</tt><tt>diagnostics.brick-sys-log-leve<wbr>l:
WARNING</tt><span><tt><br>
</tt><tt>performance.readdir-ahead: on</tt><tt><br>
</tt><tt>nfs.disable: on</tt><tt><br>
</tt><tt>nfs.export-volumes: off</tt><tt><br>
</tt><br>
<br>
<br>
</span>
<div>
<div class="m_7039630128565981365h5">
<div
class="m_7039630128565981365m_5079054141158038028moz-cite-prefix">On
05/10/2017 11:44 AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Is this the volume info you
have?<br>
<br>
<pre>><i> [<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">root at mseas-data2</a> ~]# gluster volume info
</i>><i>
</i>><i> Volume Name: data-volume
</i>><i> Type: Distribute
</i>><i> Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18
</i>><i> Status: Started
</i>><i> Number of Bricks: 2
</i>><i> Transport-type: tcp
</i>><i> Bricks:
</i>><i> Brick1: mseas-data2:/mnt/brick1
</i>><i> Brick2: mseas-data2:/mnt/brick2
</i>><i> Options Reconfigured:
</i>><i> performance.readdir-ahead: on
</i>><i> nfs.disable: on
</i>><i> nfs.export-volumes: off
</i></pre>
</div>
I copied this from old thread from 2016. This is distribute
volume. Did you change any of the options in between?
</div>
</blockquote>
</div></div><span><pre class="m_7039630128565981365m_5079054141158038028moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_7039630128565981365m_5079054141158038028moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_7039630128565981365m_5079054141158038028moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</span></div>
</blockquote></div>
--
<div class="m_7039630128565981365gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="m_7039630128565981365moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_7039630128565981365moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_7039630128565981365moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></body></html>