[Gluster-users] GFS performance under heavy traffic

[Jorick Astrego]

Hi David,

Did you try setting "direct-io-mode=disable" on the client mounts? As it
is mostly static content it would help to use the kernel caching and
read-ahead mechanisms.

I think the default is enabled.

Regards,

Jorick Astrego

On 12/19/19 1:28 AM, David Cunningham wrote:
> Hi Raghavendra and Strahil,
>
> We are using GFS version 5.6-1.el7 from the CentOS repository.
> Unfortunately we can't modify the application and it expects to read
> and write from a normal filesystem.
>
> There's around 25GB of data being written during a business day, so
> over 10 hours that's around 0.7 MBps, which has me mystified as to how
> it can generate 114MBps of network traffic. Granted we have read
> traffic as well, but still. The chart shows much more inbound traffic
> to the GFS server than outbound, suggesting the problem is with data
> writes.
>
> Is it possible with GFS to not check with the other nodes when
> reading? Our data is mostly static and we don't require 100% guarantee
> that the data is up-to-date when reading.
>
> Thanks for any assistance.
>
>
> On Wed, 18 Dec 2019 at 16:39, Raghavendra Gowdappa
> <rgowdapp at redhat.com <mailto:rgowdapp at redhat.com>> wrote:
>
>     What version of Glusterfs are you using? Though, not sure what's
>     the root cause of your problem, just wanted to point out a bug
>     with read-ahead which would cause read-amplification over network
>     [1][2], which should be fixed in recent versions.
>
>     [1] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
>     [2] https://bugzilla.redhat.com/show_bug.cgi?id=1393419
>
>     On Wed, Dec 18, 2019 at 2:50 AM David Cunningham
>     <dcunningham at voisonics.com <mailto:dcunningham at voisonics.com>> wrote:
>
>         Hello,
>
>         We switched a production system to using GFS instead of NFS at
>         the weekend, however it didn't go well on Monday when full
>         load hit. The application started crashing regularly and we
>         had to revert to NFS. It seems that the problem was high
>         network traffic used by GFS.
>
>         We've two GFS nodes plus one arbiter node, each about 1.3ms
>         latency from each other. Attached is a chart of network
>         traffic on one of the GFS nodes. We see that it saturated the
>         1Gbps link before we reverted to NFS at 15:10.
>
>         The question is, why does GFS use so much network traffic and
>         is there anything we can do about it? NFS traffic doesn't
>         exceed 4MBps, so 120MBps for GFS seems awfully high.
>
>         It would also be good to have faster read performance from
>         GFS, but that's another issue.
>
>         Thanks in advance for any assistance.
>
>         -- 
>         David Cunningham, Voisonics Limited
>         http://voisonics.com/
>         USA: +1 213 221 1092
>         New Zealand: +64 (0)28 2558 3782
>         ________
>
>         Community Meeting Calendar:
>
>         APAC Schedule -
>         Every 2nd and 4th Tuesday at 11:30 AM IST
>         Bridge: https://bluejeans.com/441850968
>
>         NA/EMEA Schedule -
>         Every 1st and 3rd Tuesday at 01:00 PM EDT
>         Bridge: https://bluejeans.com/441850968
>
>         Gluster-users mailing list
>         Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>         https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> -- 
> David Cunningham, Voisonics Limited
> http://voisonics.com/
> USA: +1 213 221 1092
> New Zealand: +64 (0)28 2558 3782
>
> ________
>
> Community Meeting Calendar:
>
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/441850968
>
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users




Met vriendelijke groet, With kind regards,

Jorick Astrego

Netbulae Virtualization Experts 

----------------

	Tel: 053 20 30 270 	info at netbulae.eu 	Staalsteden 4-3A 	KvK 08198180
 	Fax: 053 20 30 271 	www.netbulae.eu 	7547 TA Enschede 	BTW NL821234584B01

----------------

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191219/ae27cfb0/attachment.html>