<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">On 05/05/2017 08:42 PM, Pat Haley
wrote:<br>
</div>
<blockquote cite="mid:b1f4209e-a0e5-5df4-f954-399df8270b21@mit.edu"
type="cite">
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
<br>
Hi Pranith,<br>
<br>
I presume you are asking for some version of the profile data that
just shows the dd test (or a repeat of the dd test). If yes, how
do I extract just that data?<br>
</blockquote>
Yes, that is what he is asking for. Just clear the existing profile
info using `gluster volume profile volname clear` and run the dd
test once. Then when you run profile info again, it should just give
you the stats for the dd test.<br>
<blockquote cite="mid:b1f4209e-a0e5-5df4-f954-399df8270b21@mit.edu"
type="cite"> <br>
Thanks<br>
<br>
Pat<br>
<br>
<br>
<br>
<div class="moz-cite-prefix">On 05/05/2017 10:58 AM, Pranith Kumar
Karampuri wrote:<br>
</div>
<blockquote
cite="mid:CAOgeEnYKv+v=Dv3y0tC9WVJSfN=dH2U9QmrXqwOOvbTGZD=yWw@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=utf-8">
<div dir="ltr">
<div>
<div>
<div>hi Pat,<br>
</div>
Let us concentrate on the performance numbers part
for now. We will look at the permissions one after this?<br>
<br>
</div>
As per the profile info, only 2.6% of the work-load is
writes. There are too many Lookups.<br>
<br>
</div>
Would it be possible to get the data for just the dd test you
were doing earlier?<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 5, 2017 at 8:14 PM, Pat
Haley <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000"> <br>
Hi Pranith & Ravi,<br>
<br>
A couple of quick questions<br>
<br>
We have profile turned on. Are there specific queries we
should make that would help debug our configuration?
(The default profile info was previously sent in <a
moz-do-not-send="true"
class="m_-8630651799581052987moz-txt-link-freetext"
href="http://lists.gluster.org/pipermail/gluster-users/2017-May/030840.html"
target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2017-<wbr>May/030840.html</a>
but I'm not sure if that is what you were looking for.)<br>
<br>
We also started to do a test on serving gluster over
NFS. We rediscovered an issue we previously reported (
<a moz-do-not-send="true"
class="m_-8630651799581052987moz-txt-link-freetext"
href="http://lists.gluster.org/pipermail/gluster-users/2016-September/028289.html"
target="_blank">http://lists.gluster.org/<wbr>pipermail/gluster-users/2016-<wbr>September/028289.html</a>
) in that the NFS mounted version was ignoring the group
write permissions. What specific information would be
useful in debugging this?<br>
<br>
Thanks<span class="HOEnZb"><font color="#888888"><br>
<br>
Pat</font></span>
<div>
<div class="h5"><br>
<br>
<br>
<div class="m_-8630651799581052987moz-cite-prefix">On
04/14/2017 03:01 AM, Ravishankar N wrote:<br>
</div>
<blockquote type="cite">
<div class="m_-8630651799581052987moz-cite-prefix">On
04/14/2017 12:20 PM, Pranith Kumar Karampuri
wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Apr 8, 2017
at 10:28 AM, Ravishankar N <span
dir="ltr"><<a moz-do-not-send="true"
href="mailto:ravishankar@redhat.com"
target="_blank">ravishankar@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="m_-8630651799581052987gmail-m_1278894059907384689moz-cite-prefix">Hi
Pat,<br>
<br>
I'm assuming you are using gluster
native (fuse mount). If it helps,
you could try mounting it via
gluster NFS (gnfs) and then see if
there is an improvement in speed.
Fuse mounts are slower than gnfs
mounts but you get the benefit of
avoiding a single point of failure.
Unlike fuse mounts, if the gluster
node containing the gnfs server goes
down, all mounts done using that
node will fail). For fuse mounts,
you could try tweaking the
write-behind xlator settings to see
if it helps. See the
performance.write-behind and
performance.write-behind-windo<wbr>w-size
options in `gluster volume set
help`. Of course, even for gnfs
mounts, you can achieve fail-over by
using CTDB.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div>Ravi,<br>
</div>
<div> Do you have any data that
suggests fuse mounts are slower than
gNFS servers? <br>
</div>
</div>
</div>
</div>
</blockquote>
I have heard anecdotal evidence time and again on
the ML and IRC, which is why I wanted to compare
it with NFS numbers on his setup. <br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<div>Pat,<br>
</div>
<div> I see that I am late to the
thread, but do you happen to have
"profile info" of the workload?<br>
<br>
</div>
<div>You can follow <a
moz-do-not-send="true"
href="https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Monitoring%20Workload/"
target="_blank">https://gluster.readthedocs.<wbr>io/en/latest/Administrator%<wbr>20Guide/Monitoring%20Workload/</a>
to get the information.<br>
</div>
</div>
</div>
</div>
</blockquote>
Yeah, Let's see if profile info shows up anything
interesting.<br>
-Ravi<br>
<blockquote type="cite">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote"
style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div bgcolor="#FFFFFF">
<div
class="m_-8630651799581052987gmail-m_1278894059907384689moz-cite-prefix">
<br>
Thanks,<br>
Ravi
<div>
<div
class="m_-8630651799581052987gmail-h5"><br>
<br>
On 04/08/2017 12:07 AM, Pat
Haley wrote:<br>
</div>
</div>
</div>
<blockquote type="cite">
<div>
<div
class="m_-8630651799581052987gmail-h5">
<br>
Hi,<br>
<br>
We noticed a dramatic slowness
when writing to a gluster disk
when compared to writing to an
NFS disk. Specifically when
using dd (data duplicator) to
write a 4.3 GB file of zeros:<br>
<ul>
<li>on NFS disk (/home): 9.5
Gb/s</li>
<li>on gluster disk (/gdata):
508 Mb/s<br>
</li>
</ul>
The gluser disk is 2 bricks
joined together, no replication
or anything else. The hardware
is (literally) the same:<br>
<ul>
<li>one server with 70 hard
disks and a hardware RAID
card.</li>
<li>4 disks in a RAID-6 group
(the NFS disk)</li>
<li>32 disks in a RAID-6 group
(the max allowed by the
card, /mnt/brick1)</li>
<li>32 disks in another RAID-6
group (/mnt/brick2)</li>
<li>2 hot spare<br>
</li>
</ul>
<p>Some additional information
and more tests results (after
changing the log level):<br>
</p>
<p><span>glusterfs 3.7.11 built
on Apr 27 2016 14:09:22</span><br>
<span>CentOS release 6.8
(Final)</span><br>
RAID bus controller: LSI Logic
/ Symbios Logic MegaRAID SAS-3
3108 [Invader] (rev 02)<br>
<br>
<br>
<br>
<b>Create the file to /gdata
(gluster)</b><br>
[root@mseas-data2 gdata]# dd
if=/dev/zero of=/gdata/zero1
bs=1M count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 1.91876 s, <b>546
MB/s</b><br>
<br>
<b>Create the file to /home
(ext4)</b><br>
[root@mseas-data2 gdata]# dd
if=/dev/zero of=/home/zero1
bs=1M count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 0.686021 s, <b>1.5
GB/s - </b>3 times as fast<b><br>
<br>
<br>
Copy from /gdata to /gdata
(gluster to gluster)<br>
</b>[root@mseas-data2 gdata]#
dd if=/gdata/zero1
of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 101.052 s, <b>10.4
MB/s</b> - realllyyy
slooowww<br>
<br>
<br>
<b>Copy from /gdata to /gdata</b>
<b>2nd time <b>(gluster to
gluster)</b></b><br>
[root@mseas-data2 gdata]# dd
if=/gdata/zero1
of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 92.4904 s, <b>11.3
MB/s</b> <span>- realllyyy
slooowww</span> again<br>
<br>
<br>
<br>
<b>Copy from /home to /home
(ext4 to ext4)</b><br>
[root@mseas-data2 gdata]# dd
if=/home/zero1 of=/home/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 3.53263 s, <b>297
MB/s </b>30 times as fast<br>
<br>
<br>
<b>Copy from /home to /home
(ext4 to ext4)</b><br>
[root@mseas-data2 gdata]# dd
if=/home/zero1 of=/home/zero3<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB)
copied, 4.1737 s, <b>251 MB/s</b>
<span>- 30 times as fast<br>
<br>
<br>
As a test, can we copy data
directly to the xfs
mountpoint (/mnt/brick1) and
bypass gluster?<br>
<br>
<br>
Any help you could give us
would be appreciated.<br>
<br>
</span>Thanks<br>
</p>
<pre class="m_-8630651799581052987gmail-m_1278894059907384689moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
<fieldset class="m_-8630651799581052987gmail-m_1278894059907384689mimeAttachmentHeader"></fieldset>
</div></div><pre>______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" class="m_-8630651799581052987gmail-m_1278894059907384689moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a></pre>
</blockquote>
<p>
</p>
</div>
______________________________<wbr>_________________
Gluster-users mailing list
<a moz-do-not-send="true" href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a>
<a moz-do-not-send="true" href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a>
</blockquote></div>
--
<div class="m_-8630651799581052987gmail_signature"><div dir="ltr">Pranith
</div></div>
</div></div>
</blockquote><p>
</p>
</blockquote>
<pre class="m_-8630651799581052987moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=<wbr>-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="m_-8630651799581052987moz-txt-link-abbreviated" href="mailto:phaley@mit.edu" target="_blank">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="m_-8630651799581052987moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/" target="_blank">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre></div></div></div></blockquote></div>
--
<div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Pranith
</div></div>
</div>
</blockquote>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</blockquote><p>
</p></body></html>