[Gluster-users] [Gluster-devel] high load when copy directory with many files

[Xavi Hernandez]

On Sun, Apr 11, 2021 at 10:29 AM Amar Tumballi <amar at kadalu.io> wrote:

> Hi Marco, this is really good test/info. Thanks.
>
> One more thing to observe is you are running such tests is 'gluster
> profile info', so the bottleneck fop is listed.
>
> Mohit, Xavi, in this parallel operations, the load may be high due to
> inodelk used in mds xattr update in dht? Or you guys suspect something else?
>

A profile info would be very useful to know which fop gets more requests. I
think inodelk by itself shouldn't be an issue (I guess we are setting mds
only once, right ?). In theory we shouldn't be sending any operation on an
inode without a previous successful lookup, and in this case lookups should
fail, so I don't clearly see what's the difference compared to an stat.

We should investigate this. I'll try to do some experiments (not sure if
this week, though).

Regards,

Xavi


> Regards
> Amar
>
> On Sat, 10 Apr, 2021, 11:45 pm Marco Lerda - FOREACH S.R.L., <
> marco.lerda at foreach.it> wrote:
>
>> hi,
>> we have isolated the problem (meanwhile some hardware upgrade and code
>> optimization helped to limit the problem).
>> it happens when many request (HTTP over apache) comes to a non existent
>> file.
>> With 30 concurrent request to the same non existing file cause the load
>> go high without limit.
>> Same requests on existing files works fine.
>> I have tried to simulate che apache access to file excluding apache with
>> repeated command on files with the same parallelism (30):
>> - with ls works fine, file exists or not
>> - with stat works fine, file exists or not
>> - with xattr load go up, file exists or not
>>
>> thank you
>>
>>
>> Il 05/10/2020 19.45, Marco Lerda - FOREACH S.R.L. ha scritto:
>> > hi,
>> > we use glusterfs on a php application that have many small php files
>> > images etc...
>> > We use glusterfs in replication mode.
>> > We have 2 nodes connected in fiber with 100MBps and less than 1 ms
>> > latency.
>> > We have also an arbiter on slower network (but the issue is there also
>> > without the arbiter).
>> > When we copy a directory (cp command) with many files, cpu usage and
>> > load explode raplidly,
>> > our application become inaccessible until the copy ends.
>> >
>> > I wonder if is that normal or we have done something wrong.
>> > I know that glusterfs is not indicated with many small files, and I
>> > know that it slow down,
>> > but I want to avoid that a simple copy of a directory will put down
>> > out application.
>> >
>> > Any suggestion?
>> >
>> > Thanks a lot
>> >
>> >
>> >
>> > ________
>> >
>> >
>> >
>> > Community Meeting Calendar:
>> >
>> > Schedule -
>> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> > Bridge: https://bluejeans.com/441850968
>> >
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> --
>>
>> ------------------------------------------------------
>> Marco Lerda
>> FOREACH S.R.L.
>> Via Laghi di Avigliana 115, 12022 - Busca (CN)
>> Telefono: 0171-1984102
>> Centralino/Fax: 0171-1984100
>> Email:  marco.lerda at foreach.it
>> Web: http://www.foreach.it
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://meet.google.com/cpu-eiue-hvk
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
> -------
>
> Community Meeting Calendar:
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://meet.google.com/cpu-eiue-hvk
>
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210412/9ba463ee/attachment.html>