<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<br>
Hi,<br>
<br>
We noticed a dramatic slowness when writing to a gluster disk when
compared to writing to an NFS disk. Specifically when using dd (data
duplicator) to write a 4.3 GB file of zeros:<br>
<ul>
<li>on NFS disk (/home): 9.5 Gb/s</li>
<li>on gluster disk (/gdata): 508 Mb/s<br>
</li>
</ul>
The gluser disk is 2 bricks joined together, no replication or
anything else. The hardware is (literally) the same:<br>
<ul>
<li>one server with 70 hard disks and a hardware RAID card.</li>
<li>4 disks in a RAID-6 group (the NFS disk)</li>
<li>32 disks in a RAID-6 group (the max allowed by the card,
/mnt/brick1)</li>
<li>32 disks in another RAID-6 group (/mnt/brick2)</li>
<li>2 hot spare<br>
</li>
</ul>
<p>Some additional information and more tests results (after
changing the log level):<br>
</p>
<p><span>glusterfs 3.7.11 built on Apr 27 2016 14:09:22</span><br>
<span>CentOS release 6.8 (Final)</span><br>
RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108
[Invader] (rev 02)<br>
<br>
<br>
<br>
<b>Create the file to /gdata (gluster)</b><br>
[root@mseas-data2 gdata]# dd if=/dev/zero of=/gdata/zero1 bs=1M
count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB) copied, 1.91876 s, <b>546 MB/s</b><br>
<br>
<b>Create the file to /home (ext4)</b><br>
[root@mseas-data2 gdata]# dd if=/dev/zero of=/home/zero1 bs=1M
count=1000<br>
1000+0 records in<br>
1000+0 records out<br>
1048576000 bytes (1.0 GB) copied, 0.686021 s, <b>1.5 GB/s - </b>3
times as fast<b><br>
<br>
<br>
Copy from /gdata to /gdata (gluster to gluster)<br>
</b>[root@mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB) copied, 101.052 s, <b>10.4 MB/s</b> -
realllyyy slooowww<br>
<br>
<br>
<b>Copy from /gdata to /gdata</b> <b>2nd time <b>(gluster to
gluster)</b></b><br>
[root@mseas-data2 gdata]# dd if=/gdata/zero1 of=/gdata/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB) copied, 92.4904 s, <b>11.3 MB/s</b> <span>-
realllyyy slooowww</span> again<br>
<br>
<br>
<br>
<b>Copy from /home to /home (ext4 to ext4)</b><br>
[root@mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero2<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB) copied, 3.53263 s, <b>297 MB/s </b>30
times as fast<br>
<br>
<br>
<b>Copy from /home to /home (ext4 to ext4)</b><br>
[root@mseas-data2 gdata]# dd if=/home/zero1 of=/home/zero3<br>
2048000+0 records in<br>
2048000+0 records out<br>
1048576000 bytes (1.0 GB) copied, 4.1737 s, <b>251 MB/s</b> <span>-
30 times as fast<br>
<br>
<br>
As a test, can we copy data directly to the xfs mountpoint
(/mnt/brick1) and bypass gluster?<br>
<br>
<br>
Any help you could give us would be appreciated.<br>
<br>
</span>Thanks<br>
</p>
<pre class="moz-signature" cols="72">--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: <a class="moz-txt-link-abbreviated" href="mailto:phaley@mit.edu">phaley@mit.edu</a>
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 <a class="moz-txt-link-freetext" href="http://web.mit.edu/phaley/www/">http://web.mit.edu/phaley/www/</a>
77 Massachusetts Avenue
Cambridge, MA 02139-4301
</pre>
</body>
</html>