[Gluster-devel] Iusses with Random read/write

[Benjamin Turner]

I run alot of random IO tests with gluster and it has really come a long
way in the 3.7 release.  What version are you running on?  I have a couple
of suggestions:

-Run on 3.7 if you are not already.
-Run the IOZone test you are running on the back end without gluster to
verify that your HW meets your perf needs.  SSDs really scream with random
IO if your current HW won't meet your needs.
-When you are running tests watch top -H on both clients and servers, look
for any threads hitting 100%
-If you see hot threads bump up the server.event-threads and/or
client.event-threads from the default of 2

HTH!

-b


On Thu, Jul 23, 2015 at 3:04 AM, Susant Palai <spalai at redhat.com> wrote:

> ++CCing gluster-devel to have more eye on this problem.
>
> Susant
>
> ----- Original Message -----
> > From: "Subrata Ghosh" <subrata.ghosh at ericsson.com>
> > To: "Susant Palai <spalai at redhat.com> (spalai at redhat.com)" <
> spalai at redhat.com>, "Vijay Bellur <vbellur at redhat.com>
> > (vbellur at redhat.com)" <vbellur at redhat.com>
> > Cc: "Subrata Ghosh" <subrata.ghosh at ericsson.com>
> > Sent: Sunday, 19 July, 2015 7:57:28 PM
> > Subject: Iusses with Random read/write
> >
> > Hi Vijay/Prashant,
> >
> > How you are  you :).
> >
> > We need your  immediate  help / suggestion  to meet our random I/IO
> > performance metrics.
> > Currently we have performance issues with random read/write - our basic
> > requirement 20 MB/sec for random I/O.
> >
> > We tried with both "iozone" and "fio", received almost same ( random I/O)
> > performance which is not meeting our fundamental I/IO requirements.
> >
> > Our use case is as below.
> >
> > "Application running on different cards Writes/Reads (random) continuous
> > files to the volume comprising with storage belonging from different
> cards
> > in the distributed system, where replica presence  across cards and
> > applications are using non-local storages."
> > We have verified and identified  the bottleneck mostly on Gluster Client
> side
> > inside the application, however gluster server to server I/O speed looks
> > enough good. Performance tuning on gluster server side would not
> expected to
> > help.
> >
> > We also cross verified checked using NFS client we are getting far better
> > performance, but we cannot use NFS client /libgfapi because of use case
> > limitation ( brick failures cases etc..)
> >
> > Please throw some lights or thoughts to improve gluster client to
> achieve >
> > 20 MB/Secs
> >
> > Observations:
> >
> > Fio:
> >
> >
> > lease find the test results of Random write & read in 2 APPs scenarios.
> >
> > Scenario
> >
> > APP_1
> >
> > APP_2
> >
> > File size
> >
> > No of AMC's
> >
> > Random-Write
> >
> > 3.06  MB/s
> >
> > 3.02 MB/s
> >
> > 100 MB
> >
> > 4
> >
> > Random-Read
> >
> > 8.1 MB/s
> >
> > 8.4 MB/s
> >
> > 100 MB
> >
> > 4
> >
> >
> >
> > Iozone:
> >
> > ./iozone -R -l 1 -u 1 -r 4k -s 2G -F /home/cdr/f1 | tee -a
> > /tmp/iozone_results.txt &
> >
> >
> > APP 1
> >
> > APP2
> >
> > File Size : 2GB
> >
> > File size : 2GB
> >
> > Record size = 4 Kbytes
> >
> > Record size = 4 Kbytes
> >
> > Output is in Kbytes/sec
> >
> > Output is in Kbytes/sec
> >
> >
> >
> >
> > Initial write    41061.78
> >
> > Initial write    41167.36
> >
> >
> > Rewrite    40395.64
> >
> > Rewrite    40810.41
> >
> >
> > Read   262685.69
> >
> > Read   269644.62
> >
> >
> > Re-read  263751.66
> >
> > Re-read   270760.62
> >
> >
> > Reverse Read   27715.72
> >
> > Reverse Read    28604.22
> >
> >
> > Stride read   83776.44
> >
> > Stride read    84347.88
> >
> >
> > Random read    16239.74 (15.8 MB/s )
> >
> > Random read    15815.94  (15.4 MB/s )
> >
> >
> > Mixed workload    16260.95
> >
> > Mixed workload    15787.55
> >
> >
> > Random write     3356.57 (3.3 MB/s )
> >
> > Random write     3365.17 ( 3.3 MB/s)
> >
> >
> > Pwrite    40914.55
> >
> > Pwrite    40692.34
> >
> >
> > Pread   260613.83
> >
> > Pread   269850.59
> >
> >
> > Fwrite    40412.40
> >
> > Fwrite    40369.78
> >
> >
> > Fread   261506.61
> >
> > Fread   267142.41
> >
> >
> >
> > Some of the info on performance testing is at
> >
> http://www.gluster.org/community/documentation/index.php/Performance_Testing
> > Also pls check iozone limitations listed there.
> >
> > "WARNING: random I/O testing in iozone is very restricted by iozone
> > constraint that it must randomly read then randomly write the entire
> file!
> > This is not what we want - instead it should randomly read/write for some
> > fraction of file size or time duration, allowing us to spread out more on
> > the disk while not waiting too long for test to finish. This is why fio
> > (below) is the preferred test tool for random I/O workloads."
> >
> >
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20150723/1c2e92b1/attachment.html>