[Gluster-users] Poor performance with small files

[Ron Trompert]

Hi Ben,

Thanks for the info.

Cheers,

Ron


On 29/04/15 21:03, Ben Turner wrote:
> ----- Original Message -----
>> From: "Ron Trompert" <ron.trompert at surfsara.nl>
>> To: gluster-users at gluster.org
>> Sent: Wednesday, April 29, 2015 1:25:59 PM
>> Subject: [Gluster-users] Poor performance with small files
>>
>> Hi,
>>
>> We run gluster as storage solution for our Owncloud-based sync and share
>> service. At the moment we have about 30 million files in the system
>> which addup to a little more than  30TB. Most of these files are as you
>> may expect very small, i.e. in the 100KB ball park. For about a year
>> everything ran perfectly fine. We run 3.6.2 by the way.
> 
> Upgrade to 3.6.3 and set client.event-threads and server.event-threads to at least 4:
> 
> "Previously, epoll thread did socket even-handling and the same thread was used for serving the client or processing the response received from the server. Due to this, other requests were in a queue untill the current epoll thread completed its operation. With multi-threaded epoll, events are distributed that improves the performance due the parallel processing of requests/responses received."
> 
> Here are the guidelines for tuning them:
> 
> https://access.redhat.com/documentation/en-US/Red_Hat_Storage/3/html/Administration_Guide/Small_File_Performance_Enhancements.html
> 
> In my testing with epoll threads at 4 I saw a between a 15% and 50% increase depending on the workload.
> 
> There are several smallfile perf enhancements in the works:
> 
> *http://www.gluster.org/community/documentation/index.php/Features/Feature_Smallfile_Perf
> 
> *Lookup unhashed is the next feature and should be ready with 3.7(correct me if I am wrong).  
> 
> *If you are using RAID 6 you may want to do some testing with RAID 10 or JBOD, but the benefits here only come into play with alot of concurrent access(30+ processes / threads working with different files).
> 
> *Tiering may help here if you want to add some SSDs, this is also a 3.7 feature.
> 
> HTH!
> 
> -b
> 
>>
>> Now we are trying to commission new hardware. We have done this by
>> adding the new nodes to our cluster and using the add-brick and
>> remove-brick procedure to get the data to the new nodes. In a week we
>> have migrated only 8.5TB this way. What are we doing wrong here? Is
>> there a way to improve the gluster performance on small files?
>>
>> I have another question. If you want to setup a gluster that will
>> contain lots of very small files. What would be a good practice to set
>> things up in terms configuration, sizes of bricks related tot memory and
>> number of cores, number of brick per node etc.?
>>
>>
>>
>> Best regards and thanks in advance,
>>
>> Ron
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users
>>