[Gluster-devel] copying small files to glusterfs is slow

[Daniel van Ham Colchete]

Trung,

I'm no Gluster developer but I might have some insights on your problem.

The big question is: how do you copy those files? Are you doing one at a
time (like a "cp" would)? If so, them that's the answer. From now on I'm
taking a 'yes' as the answer.

Copying one file means open, write, write, write, write and close. Every
operation suffers the network latency delay. That latency is around 0.2ms if
you are using a TCP/IP interconnect but it can go as down as 0.007ms if you
use the not-so-cheap infiniband interconnect.

Doing some cheap linear interpolation, it seems that, in your test system,
the time necessary to create a tiny file is 19 milliseconds. So, when you
say you needed 20 ms to create a 5kb file, 95% is file creation and 5% is
writing. But, when you are creating a 1MB in 100 ms, only 19% of it is file
creation. The writing speed I got here from the interpolation is 106mbps,
witch is close to the expected value.

I'm working in a project here that will have a few millions of small files.
But, in my case all the files are written (or read) in a different thread,
so, no matter how fast I need to be in files per second, I just need more
threads and more Gluster servers.

Best regards,
Daniel

On 5/28/07, Trung Le Thanh <trung_bmt at yahoo.com> wrote:
>
> Hi all,
>
> I am new to glusterfs. During the last week I did some
> testing and found that glusterfs took quite a long
> time to deal with small files.
>
> My configuration:
> 3 server, 1 clien linked by a 100Mbps switch
> Centos 5
> Glusterfs 1.2.3
> Fuse 2.6.5
> The server and client vol files are the same as in the
> wiki
>
> The speed of copying 5kb files is:    ~2000kbit/s
> The speed of copying 50kb files is:  ~10000kbit/s
> The speed of copying 300kb files is: ~40000kbit/s
> The speed of copying 600kb files is: ~70000kbit/s
> The speed of copying 1MB files is:   ~85000kbit/s
>
> The above values are average speed resulted from
> copying 4000 files of each size.
>
> I also tested with different sizes of the aggregate
> value of write-behind and I saw no improvement when
> dealing with small files. Can anyone help me on this?
>
> Thank you very much
> Best regards,
> Trung
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>