<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
Hi,<br>
<br>
We redid the dd tests (this time using conv=sync oflag=sync to avoid
caching questions). The profile results are in<br>
<br>
<a class="moz-txt-link-freetext" href="http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test">http://mseas.mit.edu/download/phaley/GlusterUsers/profile_gluster_fuse_test</a><br>
<br>
<br>
<div class="moz-cite-prefix">On 05/05/2017 12:47 PM, Ravishankar N
wrote:<br>
</div>
<blockquote
cite="mid:412870d9-1003-f8f1-aaf0-6a701b393b80@redhat.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<div class="moz-cite-prefix">On 05/05/2017 08:42 PM, Pat Haley
wrote:<br>
</div>
<blockquote
cite="mid:b1f4209e-a0e5-5df4-f954-399df8270b21@mit.edu"
type="cite"> <br>
Hi Pranith,<br>
<br>
I presume you are asking for some version of the profile data
that just shows the dd test (or a repeat of the dd test). If
yes, how do I extract just that data?<br>
</blockquote>
Yes, that is what he is asking for. Just clear the existing
profile info using `gluster volume profile volname clear` and run
the dd test once. Then when you run profile info again, it should
just give you the stats for the dd test.<br>
<blockquote
cite="mid:b1f4209e-a0e5-5df4-f954-399df8270b21@mit.edu"
type="cite"> <br>
Thanks<br>
<br>
Pat<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/05/2017 10:58 AM, Pranith
Kumar Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnYKv+v=Dv3y0tC9WVJSfN=dH2U9QmrXqwOOvbTGZD=yWw@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div>
<div>hi Pat,<br>
</div>
Let us concentrate on the performance numbers part
for now. We will look at the permissions one after this?<br>
<br>
</div>
As per the profile info, only 2.6% of the work-load is
writes. There are too many Lookups.<br>
<br>
</div>
Would it be possible to get the data for just the dd test
you were doing earlier?<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 5, 2017 at 8:14 PM, Pat
Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Hi Pranith & Ravi,<br>
<br>
A couple of quick questions<br>
<br>
We have profile turned on. Are there specific queries
we should make that would help debug our
configuration? (The default profile info was
previously sent in <a moz-do-not-send="true"
class="m_-8630651799581052987moz-txt-link-freetext"
href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html"
target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2017-<wbr>May/030840.html</a>
but I'm not sure if that is what you were looking
for.)<br>
<br>
We also started to do a test on serving gluster over
NFS. We rediscovered an issue we previously reported
( <a moz-do-not-send="true"
class="m_-8630651799581052987moz-txt-link-freetext"
href="http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html"
target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2016-<wbr>September/028289.html</a>
) in that the NFS mounted version was ignoring the
group write permissions. What specific information
would be useful in debugging this?<br>
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="h5"><br>
<br>
<br>
<div class="m_-8630651799581052987moz-cite-prefix">On
04/14/2017 03:01 AM, Ravishankar N wrote:<br>
</div>
<blockquote type="cite">
<div
class="m_-8630651799581052987moz-cite-prefix">On
04/14/2017 12:20 PM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Apr 8,
2017 at 10:28 AM, Ravishankar N <span
dir="ltr"><<a
moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="m_-8630651799581052987gmail-m_1278894059907384689moz-cite-prefix">Hi
Pat,<br>
<br>
I'm assuming you are using gluster
native (fuse mount). If it helps,
you could try mounting it via
gluster NFS (gnfs) and then see if
there is an improvement in speed.
Fuse mounts are slower than gnfs
mounts but you get the benefit of
avoiding a single point of
failure. Unlike fuse mounts, if
the gluster node containing the
gnfs server goes down, all mounts
done using that node will fail).
For fuse mounts, you could try
tweaking the write-behind xlator
settings to see if it helps. See
the performance.write-behind and
performance.write-behind-windo<wbr>w-size
options in `gluster volume set
help`. Of course, even for gnfs
mounts, you can achieve fail-over
by using CTDB.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Ravi,<br>
</div>
<div> Do you have any data that
suggests fuse mounts are slower than
gNFS servers? <br>
</div>
</div>
</div>
</div>
</blockquote>
I have heard anecdotal evidence time and again
on the ML and IRC, which is why I wanted to
compare it with NFS numbers on his setup. <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Pat,<br>
</div>
<div> I see that I am late to the
thread, but do you happen to have
"profile info" of the workload?<br>
<br>
</div>
<div>You can follow <a
moz-do-not-send="true"
href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/"
target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Administrator%<wbr>20Guide/Monitoring%20Workload/</a>
to get the information.<br>
</div>
</div>
</div>
</div>
</blockquote>
Yeah, Let's see if profile info shows up
anything interesting.<br>
-Ravi<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="m_-8630651799581052987gmail-m_1278894059907384689moz-cite-prefix">
<br>
Thanks,<br>
Ravi
<div>
<div
class="m_-8630651799581052987gmail-h5"><br>
<br>
On 04/08/2017 12:07 AM, Pat
Haley wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div
class="m_-8630651799581052987gmail-h5">
<br>
Hi,<br>
<br>
We noticed a dramatic slowness
when writing to a gluster disk
when compared to writing to an
NFS disk. Specifically when
using dd (data duplicator) to
write a 4.3 GB file of zeros:<br>
<ul>
<li>on NFS disk (/home): 9.5
Gb/s</li>
<li>on gluster disk
(/gdata): 508 Mb/s<br>
</li>
</ul>
The gluser disk is 2 bricks
joined together, no
replication or anything else.
The hardware is (literally)
the same:<br>
<ul>
<li>one server with 70 hard
disks and a hardware RAID
card.</li>
<li>4 disks in a RAID-6
group (the NFS disk)</li>
<li>32 disks in a RAID-6
group (the max allowed by
the card, /mnt/brick1)</li>
<li>32 disks in another
RAID-6 group (/mnt/brick2)</li>
<li>2 hot spare<br>
</li>
</ul>
<p>Some additional information
and more tests results
(after changing the log
level):<br>
</p>
<p><span>glusterfs 3.7.11
built on Apr 27 2016
14:09:22</span><br>
<span>CentOS release 6.8
(Final)</span><br>
RAID bus controller: LSI
Logic / Symbios Logic
MegaRAID SAS-3 3108
[Invader] (rev 02)<br>
<br>
<br>
<br>
<b>Create the file to /gdata
(gluster)</b><br>
[root@mseas-data2 gdata]# dd
if=/dev/zero of=/gdata/zero1
bs=1M count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 1.91876 s, <b>546
MB/s</b><br>
<br>
<b>Create the file to /home
(ext4)</b><br>
[root@mseas-data2 gdata]# dd
if=/dev/zero of=/home/zero1
bs=1M count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 0.686021 s, <b>1.5
GB/s - </b>3 times as
fast<b><br>
<br>
<br>
Copy from /gdata to /gdata
(gluster to gluster)<br>
</b>[root@mseas-data2
gdata]# dd if=/gdata/zero1
of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 101.052 s, <b>10.4
MB/s</b> - realllyyy
slooowww<br>
<br>
<br>
<b>Copy from /gdata to
/gdata</b> <b>2nd time <b>(gluster
to gluster)</b></b><br>
[root@mseas-data2 gdata]# dd
if=/gdata/zero1
of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 92.4904 s, <b>11.3
MB/s</b> <span>-
realllyyy slooowww</span>
again<br>
<br>
<br>
<br>
<b>Copy from /home to /home
(ext4 to ext4)</b><br>
[root@mseas-data2 gdata]# dd
if=/home/zero1
of=/home/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 3.53263 s, <b>297
MB/s </b>30 times as fast<br>
<br>
<br>
<b>Copy from /home to /home
(ext4 to ext4)</b><br>
[root@mseas-data2 gdata]# dd
if=/home/zero1
of=/home/zero3<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 4.1737 s, <b>251
MB/s</b> <span>- 30 times
as fast<br>
<br>
<br>
As a test, can we copy
data directly to the xfs
mountpoint (/mnt/brick1)
and bypass gluster?<br>
<br>
<br>
Any help you could give us
would be appreciated.<br>
<br>
</span>Thanks<br>
</p>
<pre class="m_-8630651799581052987gmail-m_1278894059907384689moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
<fieldset class="m_-8630651799581052987gmail-m_1278894059907384689mimeAttachmentHeader"></fieldset>
</div></div><pre>______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></pre>
</blockquote>
<p>
</p>
</div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a>
</blockquote></div>
--
<div class="m_-8630651799581052987gmail_signature"><div dir="ltr">Pranith
</div></div>
</div></div>
</blockquote><p>
</p>
</blockquote>
<pre class="m_-8630651799581052987moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-8630651799581052987moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-8630651799581052987moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</blockquote><p>
</p>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></body></html>