[Gluster-users] NFS versus Fuse file locking problem (NFS works, fuse doesn't...)

[Everton Brogliatto]

Hi Krist,

What are your volume options on that setup? Have you tried tuning it for
the kind of workload and files size you have?

I would definitely do some tests with feature.shard=on/off first. If shard
is on, try playing with features.shard-block-size.
Do you have jumbo frames (MTU=9000) enabled across the switch and nodes? if
you have concurrent clients writing/reading, it could be beneficial to
increase the number of client and server threads as well, try setting
higher values for client.event-threads and server.event-threads.

Best regards,
Everton Brogliatto



On Thu, Aug 24, 2017 at 7:48 PM, Krist van Besien <krist at redhat.com> wrote:

> Hi all,
>
> I usualy advise clients to use the native client if at all possible, as it
> is very robust. But I am running in to problems here.
>
> In this case the gluster system is used to store video streams. Basicaly
> the setup is the following:
> - A gluster cluster of 3 nodes, with ample storage. They export several
> volumes.
> - The network is 10GB, switched.
> - A "recording server" which subscribes to multi cast video streams, and
> records them to disk. The recorder writes the streams in 10s blocks, so
> when it is for example recording 50 streams it is creating 5 files a
> second, each about 5M. it uses a write-then-rename process.
>
> I simulated that with a small script, that wrote 5M files and renamed them
> as fast as it could, and could easily create around 100 files/s (which
> abouts saturates the network). So I think the cluster is up to the task.
>
> However if we try the actualy workload we run in to trouble. Running the
> recorder software we can gradually ramp up the number of streams it records
> (and thus the number of files it creates), and at arou d 50 streams the
> recorder eventually stops writing files. According to the programmers that
> wrote it, it appears that it can no longer get the needed locks¸ and as a
> result just stops writing.
>
> We decided to test using the NFS client as well, and there the problem
> does not exist. But again, I (and the customer) would prefer not to use
> NFS, but use the native client in stead.
>
> So if the problem is file locking, and the problem exists with the native
> client, and not using NFS, what could be the cause?
>
> In what way do locking differ between the two different file systems,
> between NFS and Fuse, and how can the programmers work around any issues
> the fuse client might be causing?
>
> This video stream software is a bespoke solution, developped in house and
> it is thus possible to change the way it handles files so it works with the
> native client, but the programmers are looking at me for guidance.
>
> Any suggestions?
>
> Krist
>
>
>
>
> --
> Vriendelijke Groet |  Best Regards | Freundliche Grüße | Cordialement
> ------------------------------
>
> Krist van Besien
>
> senior architect, RHCE, RHCSA Open Stack
>
> Red Hat Red Hat Switzerland S.A. <https://www.redhat.com>
>
> krist at redhat.com    M: +41-79-5936260
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170824/c87b6242/attachment.html>