<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
Hi,<br>
<br>
We finally managed to do the dd tests for an NFS-mounted gluster
file system. The profile results during that test are in<br>
<br>
<a class="moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_nfs_test">http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_nfs_test</a><br>
<br>
The summary of the dd tests are<br>
<ul>
<li>writing to gluster disk mounted with fuse: 5 Mb/s</li>
<li>writing to gluster disk mounted with nfs: 200 Mb/s<br>
</li>
</ul>
Pat<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/05/2017 08:11 PM, Pat Haley
wrote:<br>
</div>
<blockquote cite="mid:5c64c01f-9998-6c3b-d869-5acbb6ce3077@mit.edu"
type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<br>
Hi,<br>
<br>
We redid the dd tests (this time using conv=sync oflag=sync to
avoid caching questions). The profile results are in<br>
<br>
<a moz-do-not-send="true" class="moz-txt-link-freetext"
href="http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test">http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test</a><br>
<br>
<br>
<div class="moz-cite-prefix">On 05/05/2017 12:47 PM, Ravishankar N
wrote:<br>
</div>
<blockquote
cite="mid:412870d9-1003-f8f1-aaf0-6a701b393b80@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div class="moz-cite-prefix">On 05/05/2017 08:42 PM, Pat Haley
wrote:<br>
</div>
<blockquote
cite="mid:b1f4209e-a0e5-5df4-f954-399df8270b21@mit.edu"
type="cite"> <br>
Hi Pranith,<br>
<br>
I presume you are asking for some version of the profile data
that just shows the dd test (or a repeat of the dd test). If
yes, how do I extract just that data?<br>
</blockquote>
Yes, that is what he is asking for. Just clear the existing
profile info using `gluster volume profile volname clear` and
run the dd test once. Then when you run profile info again, it
should just give you the stats for the dd test.<br>
<blockquote
cite="mid:b1f4209e-a0e5-5df4-f954-399df8270b21@mit.edu"
type="cite"> <br>
Thanks<br>
<br>
Pat<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/05/2017 10:58 AM, Pranith
Kumar Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnYKv+v=Dv3y0tC9WVJSfN=dH2U9QmrXqwOOvbTGZD=yWw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>hi Pat,<br>
</div>
Let us concentrate on the performance numbers
part for now. We will look at the permissions one
after this?<br>
<br>
</div>
As per the profile info, only 2.6% of the work-load is
writes. There are too many Lookups.<br>
<br>
</div>
Would it be possible to get the data for just the dd test
you were doing earlier?<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 5, 2017 at 8:14 PM,
Pat Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Hi Pranith & Ravi,<br>
<br>
A couple of quick questions<br>
<br>
We have profile turned on. Are there specific
queries we should make that would help debug our
configuration? (The default profile info was
previously sent in <a moz-do-not-send="true"
class="m_-8630651799581052987moz-txt-link-freetext"
href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html"
target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2017-<wbr>May/030840.html</a>
but I'm not sure if that is what you were looking
for.)<br>
<br>
We also started to do a test on serving gluster over
NFS. We rediscovered an issue we previously
reported ( <a moz-do-not-send="true"
class="m_-8630651799581052987moz-txt-link-freetext"
href="http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html"
target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2016-<wbr>September/028289.html</a>
) in that the NFS mounted version was ignoring the
group write permissions. What specific information
would be useful in debugging this?<br>
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="h5"><br>
<br>
<br>
<div
class="m_-8630651799581052987moz-cite-prefix">On
04/14/2017 03:01 AM, Ravishankar N wrote:<br>
</div>
<blockquote type="cite">
<div
class="m_-8630651799581052987moz-cite-prefix">On
04/14/2017 12:20 PM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Apr 8,
2017 at 10:28 AM, Ravishankar N <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="m_-8630651799581052987gmail-m_1278894059907384689moz-cite-prefix">Hi
Pat,<br>
<br>
I'm assuming you are using
gluster native (fuse mount). If
it helps, you could try mounting
it via gluster NFS (gnfs) and
then see if there is an
improvement in speed. Fuse
mounts are slower than gnfs
mounts but you get the benefit
of avoiding a single point of
failure. Unlike fuse mounts, if
the gluster node containing the
gnfs server goes down, all
mounts done using that node will
fail). For fuse mounts, you
could try tweaking the
write-behind xlator settings to
see if it helps. See the
performance.write-behind and
performance.write-behind-windo<wbr>w-size
options in `gluster volume set
help`. Of course, even for gnfs
mounts, you can achieve
fail-over by using CTDB.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Ravi,<br>
</div>
<div> Do you have any data that
suggests fuse mounts are slower than
gNFS servers? <br>
</div>
</div>
</div>
</div>
</blockquote>
I have heard anecdotal evidence time and again
on the ML and IRC, which is why I wanted to
compare it with NFS numbers on his setup. <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Pat,<br>
</div>
<div> I see that I am late to the
thread, but do you happen to have
"profile info" of the workload?<br>
<br>
</div>
<div>You can follow <a
moz-do-not-send="true"
href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/"
target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Administrator%<wbr>20Guide/Monitoring%20Workload/</a>
to get the information.<br>
</div>
</div>
</div>
</div>
</blockquote>
Yeah, Let's see if profile info shows up
anything interesting.<br>
-Ravi<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="m_-8630651799581052987gmail-m_1278894059907384689moz-cite-prefix">
<br>
Thanks,<br>
Ravi
<div>
<div
class="m_-8630651799581052987gmail-h5"><br>
<br>
On 04/08/2017 12:07 AM, Pat
Haley wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div
class="m_-8630651799581052987gmail-h5">
<br>
Hi,<br>
<br>
We noticed a dramatic
slowness when writing to a
gluster disk when compared
to writing to an NFS disk.
Specifically when using dd
(data duplicator) to write a
4.3 GB file of zeros:<br>
<ul>
<li>on NFS disk (/home):
9.5 Gb/s</li>
<li>on gluster disk
(/gdata): 508 Mb/s<br>
</li>
</ul>
The gluser disk is 2 bricks
joined together, no
replication or anything
else. The hardware is
(literally) the same:<br>
<ul>
<li>one server with 70
hard disks and a
hardware RAID card.</li>
<li>4 disks in a RAID-6
group (the NFS disk)</li>
<li>32 disks in a RAID-6
group (the max allowed
by the card,
/mnt/brick1)</li>
<li>32 disks in another
RAID-6 group
(/mnt/brick2)</li>
<li>2 hot spare<br>
</li>
</ul>
<p>Some additional
information and more tests
results (after changing
the log level):<br>
</p>
<p><span>glusterfs 3.7.11
built on Apr 27 2016
14:09:22</span><br>
<span>CentOS release 6.8
(Final)</span><br>
RAID bus controller: LSI
Logic / Symbios Logic
MegaRAID SAS-3 3108
[Invader] (rev 02)<br>
<br>
<br>
<br>
<b>Create the file to
/gdata (gluster)</b><br>
[root@mseas-data2 gdata]#
dd if=/dev/zero
of=/gdata/zero1 bs=1M
count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 1.91876 s, <b>546
MB/s</b><br>
<br>
<b>Create the file to
/home (ext4)</b><br>
[root@mseas-data2 gdata]#
dd if=/dev/zero
of=/home/zero1 bs=1M
count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 0.686021 s, <b>1.5
GB/s - </b>3 times as
fast<b><br>
<br>
<br>
Copy from /gdata to
/gdata (gluster to
gluster)<br>
</b>[root@mseas-data2
gdata]# dd if=/gdata/zero1
of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 101.052 s, <b>10.4
MB/s</b> - realllyyy
slooowww<br>
<br>
<br>
<b>Copy from /gdata to
/gdata</b> <b>2nd time
<b>(gluster to gluster)</b></b><br>
[root@mseas-data2 gdata]#
dd if=/gdata/zero1
of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 92.4904 s, <b>11.3
MB/s</b> <span>-
realllyyy slooowww</span>
again<br>
<br>
<br>
<br>
<b>Copy from /home to
/home (ext4 to ext4)</b><br>
[root@mseas-data2 gdata]#
dd if=/home/zero1
of=/home/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 3.53263 s, <b>297
MB/s </b>30 times as
fast<br>
<br>
<br>
<b>Copy from /home to
/home (ext4 to ext4)</b><br>
[root@mseas-data2 gdata]#
dd if=/home/zero1
of=/home/zero3<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 4.1737 s, <b>251
MB/s</b> <span>- 30
times as fast<br>
<br>
<br>
As a test, can we copy
data directly to the xfs
mountpoint (/mnt/brick1)
and bypass gluster?<br>
<br>
<br>
Any help you could give
us would be appreciated.<br>
<br>
</span>Thanks<br>
</p>
<pre class="m_-8630651799581052987gmail-m_1278894059907384689moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
<fieldset class="m_-8630651799581052987gmail-m_1278894059907384689mimeAttachmentHeader"></fieldset>
</div></div><pre>______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></pre>
</blockquote>
<p>
</p>
</div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a>
</blockquote></div>
--
<div class="m_-8630651799581052987gmail_signature"><div dir="ltr">Pranith
</div></div>
</div></div>
</blockquote><p>
</p>
</blockquote>
<pre class="m_-8630651799581052987moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-8630651799581052987moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-8630651799581052987moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</blockquote><p>
</p>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></body></html>