<html>
<head>
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
On 05/10/17 14:18, Pat Haley wrote:<br>
<blockquote cite="mid:6dd8f0b6-016a-59f5-3fdb-afc9f1bb2f0f@mit.edu"
type="cite">
<meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
<br>
Hi Pranith,<br>
<br>
Since we are mounting the partitions as the bricks, I tried the dd
test writing to
<brick-path>/.glusterfs/<file-to-be-removed-after-test>.
The results without oflag=sync were 1.6 Gb/s (faster than gluster
but not as fast as I was expecting given the 1.2 Gb/s to the
no-gluster area w/ fewer disks).<br>
<br>
Pat<br>
<br>
</blockquote>
<br>
Is that true for every disk? If you're choosing the same filename
every time for your dd test, you're likely only doing that test
against one disk. If that disk is slow, you would get the same
results every time despite other disks performing normally.<br>
<br>
<blockquote cite="mid:6dd8f0b6-016a-59f5-3fdb-afc9f1bb2f0f@mit.edu"
type="cite"> <br>
<div class="moz-cite-prefix">On 05/10/2017 01:27 PM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnZjAG579+QtiGeViBsgJp0hNXx-x1J-4+1_4DuLU_PQgg@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=UTF-8">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017 at 10:15 PM,
Pat Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Hi Pranith,<br>
<br>
Not entirely sure (this isn't my area of expertise).
I'll run your answer by some other people who are more
familiar with this.<br>
<br>
I am also uncertain about how to interpret the results
when we also add the dd tests writing to the /home
area (no gluster, still on the same machine)<br>
<ul>
<li>dd test without oflag=sync (rough average of
multiple tests)<br>
</li>
<ul>
<li>gluster w/ fuse mount : 570 Mb/s</li>
<li>gluster w/ nfs mount: 390 Mb/s</li>
<li>nfs (no gluster): 1.2 Gb/s</li>
</ul>
<li>dd test with oflag=sync (rough average of
multiple tests)</li>
<ul>
<li>gluster w/ fuse mount: 5 Mb/s</li>
<li>gluster w/ nfs mount: 200 Mb/s</li>
<li>nfs (no gluster): 20 Mb/s<br>
</li>
</ul>
</ul>
Given that the non-gluster area is a RAID-6 of 4 disks
while each brick of the gluster area is a RAID-6 of 32
disks, I would naively expect the writes to the
gluster area to be roughly 8x faster than to the
non-gluster.<br>
</div>
</blockquote>
<div><br>
</div>
<div>I think a better test is to try and write to a file
using nfs without any gluster to a location that is not
inside the brick but someother location that is on same
disk(s). If you are mounting the partition as the brick,
then we can write to a file inside .glusterfs directory,
something like
<brick-path>/.glusterfs/<file-to-be-removed-after-test>.
<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
I still think we have a speed issue, I can't tell if
fuse vs nfs is part of the problem.</div>
</blockquote>
<div><br>
</div>
I got interested in the post because I read that fuse
speed is lesser than nfs speed which is counter-intuitive
to my understanding. So wanted clarifications. Now that I
got my clarifications where fuse outperformed nfs without
sync, we can resume testing as described above and try to
find what it is. Based on your email-id I am guessing you
are from Boston and I am from Bangalore so if you are okay
with doing this debugging for multiple days because of
timezones, I will be happy to help. Please be a bit
patient with me, I am under a release crunch but I am very
curious with the problem you posted.<br>
<br>
</div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> Was there
anything useful in the profiles?<span class="HOEnZb"><font
color="#888888"><br>
</font></span></div>
</blockquote>
<div><br>
</div>
<div>Unfortunately profiles didn't help me much, I think
we are collecting the profiles from an active volume, so
it has a lot of information that is not pertaining to dd
so it is difficult to find the contributions of dd. So I
went through your post again and found something I
didn't pay much attention to earlier i.e. oflag=sync, so
did my own tests on my setup with FUSE so sent that
reply.<br>
<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"><span
class="HOEnZb"><font color="#888888"> <br>
Pat</font></span>
<div>
<div class="h5"><br>
<br>
<br>
<div class="m_-1330903101597946930moz-cite-prefix">On
05/10/2017 12:15 PM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>
<div>Okay good. At least this validates my
doubts. Handling O_SYNC in gluster NFS and
fuse is a bit different.<br>
</div>
When application opens a file with O_SYNC on
fuse mount then each write syscall has to be
written to disk as part of the syscall where
as in case of NFS, there is no concept of
open. NFS performs write though a handle
saying it needs to be a synchronous write,
so write() syscall is performed first then
it performs fsync(). so an write on an fd
with O_SYNC becomes write+fsync. I am
suspecting that when multiple threads do
this write+fsync() operation on the same
file, multiple writes are batched together
to be written do disk so the throughput on
the disk is increasing is my guess.<br>
<br>
</div>
Does it answer your doubts?<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, May 10, 2017
at 9:35 PM, Pat Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu"
target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Without the oflag=sync and only a single
test of each, the FUSE is going faster
than NFS:<br>
<br>
FUSE:<br>
<tt>mseas-data2(dri_nascar)% dd
if=/dev/zero count=4096 bs=1048576
of=zeros.txt conv=sync</tt><tt><br>
</tt><tt>4096+0 records in</tt><tt><br>
</tt><tt>4096+0 records out</tt><tt><br>
</tt><tt>4294967296 bytes (4.3 GB)
copied, 7.46961 s, 575 MB/s</tt><tt><br>
</tt><tt><br>
<br>
</tt>NFS<br>
<tt>mseas-data2(HYCOM)% dd if=/dev/zero
count=4096 bs=1048576 of=zeros.txt
conv=sync</tt><tt><br>
</tt><tt>4096+0 records in</tt><tt><br>
</tt><tt>4096+0 records out</tt><tt><br>
</tt><tt>4294967296 bytes (4.3 GB)
copied, 11.4264 s, 376 MB/s</tt>
<div>
<div class="m_-1330903101597946930h5"><tt><br>
</tt><tt><br>
</tt><tt><br>
</tt>
<div
class="m_-1330903101597946930m_7039630128565981365moz-cite-prefix">On
05/10/2017 11:53 AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Could you let me
know the speed without
oflag=sync on both the mounts?
No need to collect profiles.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed,
May 10, 2017 at 9:17 PM, Pat
Haley <span dir="ltr"><<a
moz-do-not-send="true"
href="mailto:phaley@mit.edu"
target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div bgcolor="#FFFFFF"
text="#000000"> <br>
Here is what I see now:<br>
<br>
<tt>[root@mseas-data2 ~]#
gluster volume info</tt><span><tt><br>
</tt><tt> </tt><tt><br>
</tt><tt>Volume Name:
data-volume</tt><tt><br>
</tt><tt>Type:
Distribute</tt><tt><br>
</tt><tt>Volume ID:
c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18</tt><tt><br>
</tt><tt>Status: Started</tt><tt><br>
</tt><tt>Number of
Bricks: 2</tt><tt><br>
</tt><tt>Transport-type:
tcp</tt><tt><br>
</tt><tt>Bricks:</tt><tt><br>
</tt><tt>Brick1:
mseas-data2:/mnt/brick1</tt><tt><br>
</tt><tt>Brick2:
mseas-data2:/mnt/brick2</tt><tt><br>
</tt><tt>Options
Reconfigured:</tt><tt><br>
</tt></span><tt>diagnostics.count-fop-hits:
on</tt><tt><br>
</tt><tt>diagnostics.latency-measuremen<wbr>t:
on</tt><tt><br>
</tt><tt>nfs.exports-auth-enable:
on</tt><tt><br>
</tt><tt>diagnostics.brick-sys-log-leve<wbr>l:
WARNING</tt><span><tt><br>
</tt><tt>performance.readdir-ahead:
on</tt><tt><br>
</tt><tt>nfs.disable: on</tt><tt><br>
</tt><tt>nfs.export-volumes:
off</tt><tt><br>
</tt><br>
<br>
<br>
</span>
<div>
<div
class="m_-1330903101597946930m_7039630128565981365h5">
<div
class="m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-cite-prefix">On
05/10/2017 11:44 AM,
Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote
type="cite">
<div dir="ltr">
<div>Is this the
volume info you
have?<br>
<br>
<pre>><i> [<a moz-do-not-send="true" href="http://www.gluster.org/mailman/listinfo/gluster-users" target="_blank">root at mseas-data2</a> ~]# gluster volume info
</i>><i>
</i>><i> Volume Name: data-volume
</i>><i> Type: Distribute
</i>><i> Volume ID: c162161e-2a2d-4dac-b015-f31fd8<wbr>9ceb18
</i>><i> Status: Started
</i>><i> Number of Bricks: 2
</i>><i> Transport-type: tcp
</i>><i> Bricks:
</i>><i> Brick1: mseas-data2:/mnt/brick1
</i>><i> Brick2: mseas-data2:/mnt/brick2
</i>><i> Options Reconfigured:
</i>><i> performance.readdir-ahead: on
</i>><i> nfs.disable: on
</i>><i> nfs.export-volumes: off
</i></pre>
</div>
I copied this from old thread from 2016. This is distribute
volume. Did you change any of the options in between?
</div>
</blockquote>
</div></div><span><pre class="m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-1330903101597946930m_7039630128565981365m_5079054141158038028moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</span></div>
</blockquote></div>
--
<div class="m_-1330903101597946930m_7039630128565981365gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="m_-1330903101597946930m_7039630128565981365moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-1330903101597946930m_7039630128565981365moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-1330903101597946930m_7039630128565981365moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div>
--
<div class="m_-1330903101597946930gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="m_-1330903101597946930moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-1330903101597946930moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-1330903101597946930moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div></div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users">http://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</body></html>