[Gluster-users] Horrible performance with small files (DHT/AFR)
Benjamin Krein
superbenk at superk.org
Fri Jun 5 13:02:15 UTC 2009
On Jun 5, 2009, at 2:23 AM, Pavan Vilas Sondur wrote:
> Hi Benjamin,
> Can you provide us with the following information to narrow down the
> issue.
> 1. The network configuration of the GlusterFS deployment and the
> bandwidth that is available
> for GlusterFS to use
All these servers (servers/clients) are on a full-duplex gigabit LAN
with gigabit NICs. As I've noted in previous emails, I can easily
utilize the bandwidth with scp & rsync. As you'll see below,
glusterfs seems to do very well with large files as well.
> 2. Does performance also get affected the same way with large files
> too? Or is it with just small files
> as you have mentioned? Let us know the performance of GlusterFS
> with large files.
Here are a bunch of tests I did with various configurations & copying
large files:
* Single server - cfs1 - large files (~480MB each)
root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
webform_cache.tar /mnt/large_file_\$i.tar; done;"
`webform_cache.tar' -> `/mnt/large_file_1.tar'
`webform_cache.tar' -> `/mnt/large_file_2.tar'
`webform_cache.tar' -> `/mnt/large_file_3.tar'
`webform_cache.tar' -> `/mnt/large_file_4.tar'
`webform_cache.tar' -> `/mnt/large_file_5.tar'
real 0m23.726s
user 0m0.128s
sys 0m4.972s
# Interface RX Rate RX # TX
Rate TX #
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
────────────────────────
cfs1 (source: local)
0 eth1 91.39MiB 63911
1.98MiB 27326
* Two servers w/ AFR only - large files (~480MB each)
root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
webform_cache.tar /mnt/large_file_\$i.tar; done;"
`webform_cache.tar' -> `/mnt/large_file_1.tar'
`webform_cache.tar' -> `/mnt/large_file_2.tar'
`webform_cache.tar' -> `/mnt/large_file_3.tar'
`webform_cache.tar' -> `/mnt/large_file_4.tar'
`webform_cache.tar' -> `/mnt/large_file_5.tar'
real 0m43.354s
user 0m0.100s
sys 0m3.044s
# Interface RX Rate RX # TX
Rate TX #
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
────────────────────────
cfs1 (source: local)
0 eth1 57.73MiB 39931
910.95KiB 10733
* Two servers w/DHT+AFR - large files (~480MB each)
root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
webform_cache.tar /mnt/large_file_\$i.tar; done;"
`webform_cache.tar' -> `/mnt/large_file_1.tar'
`webform_cache.tar' -> `/mnt/large_file_2.tar'
`webform_cache.tar' -> `/mnt/large_file_3.tar'
`webform_cache.tar' -> `/mnt/large_file_4.tar'
`webform_cache.tar' -> `/mnt/large_file_5.tar'
real 0m43.294s
user 0m0.100s
sys 0m3.356s
# Interface RX Rate RX # TX
Rate TX #
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
────────────────────────
cfs1 (source: local)
0 eth1 58.15MiB 40224
1.52MiB 20174
# Interface RX Rate RX # TX
Rate TX #
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
────────────────────────
cfs2 (source: local)
0 eth1 55.99MiB 38684
755.51KiB 9521
* Two servers DHT *only* - large files (~480MB each) - NOTE: only cfs1
was ever populated, isn't DHT supposed to distribute the files?:
root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
webform_cache.tar /mnt/\$i/large_file.tar; done;"
`webform_cache.tar' -> `/mnt/1/large_file.tar'
`webform_cache.tar' -> `/mnt/2/large_file.tar'
`webform_cache.tar' -> `/mnt/3/large_file.tar'
`webform_cache.tar' -> `/mnt/4/large_file.tar'
`webform_cache.tar' -> `/mnt/5/large_file.tar'
real 0m40.062s
user 0m0.204s
sys 0m3.500s
# Interface RX Rate RX # TX
Rate TX #
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
────────────────────────
cfs1 (source: local)
0 eth1 112.96MiB 78190
1.66MiB 21994
# Interface RX Rate RX # TX
Rate TX #
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
─
────────────────────────
cfs2 (source: local)
0 eth1 1019.00B 7
83.00B 0
root at cfs1|/home/clusterfs/webform/cache2|# find -ls
294926 8 drwxrwxr-x 7 www-data www-data 4096 Jun 5 08:52 .
294927 8 drwxr-xr-x 2 root root 4096 Jun 5 08:53 ./1
294933 489716 -rw-r--r-- 1 root root 500971520 Jun 5
08:54 ./1/large_file.tar
294931 8 drwxr-xr-x 2 root root 4096 Jun 5 08:53 ./5
294932 489716 -rw-r--r-- 1 root root 500971520 Jun 5
08:54 ./5/large_file.tar
294930 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./4
294936 489716 -rw-r--r-- 1 root root 500971520 Jun 5
08:54 ./4/large_file.tar
294928 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./2
294934 489716 -rw-r--r-- 1 root root 500971520 Jun 5
08:54 ./2/large_file.tar
294929 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./3
294935 489716 -rw-r--r-- 1 root root 500971520 Jun 5
08:54 ./3/large_file.tar
root at cfs2|/home/clusterfs/webform/cache2|# find -ls
3547150 8 drwxrwxr-x 7 www-data www-data 4096 Jun 5 08:52 .
3547153 8 drwxr-xr-x 2 root root 4096 Jun 5 08:52 ./3
3547155 8 drwxr-xr-x 2 root root 4096 Jun 5 08:52 ./5
3547154 8 drwxr-xr-x 2 root root 4096 Jun 5 08:52 ./4
3547152 8 drwxr-xr-x 2 root root 4096 Jun 5 08:52 ./2
3547151 8 drwxr-xr-x 2 root root 4096 Jun 5 08:52 ./1
>
> There haven't really been an issue with using large number of small
> files as such in the past. Nevertheless,
> we'll look into this once you give us the above details.
>
I don't feel that the tests I'm performing are all that abnormal or
out of the ordinary, so I'm surprised that I'm the only one having the
problems. As you can see from the large file results above, the
problem is clearly related to small files only.
Thanks for your continued interest in resolving this!
Ben
>
> On 04/06/09 16:21 -0400, Benjamin Krein wrote:
>> Here are some more details with different configs:
>>
>> * Only AFR between cfs1 & cfs2:
>> root at dev1# time cp -rp * /mnt/
>>
>> real 16m45.995s
>> user 0m1.104s
>> sys 0m5.528s
>>
>> * Single server - cfs1:
>> root at dev1# time cp -rp * /mnt/
>>
>> real 10m33.967s
>> user 0m0.764s
>> sys 0m5.516s
>>
>> * Stats via bmon on cfs1 during above copy:
>> # Interface RX Rate RX # TX Rate
>> TX #
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ─
>> ───────────────────────
>> ─
>> ───────────────────────
>> cfs1 (source: local)
>> 0 eth1 951.25KiB 1892 254.00KiB
>> 1633
>>
>> It gets progressively better, but that's still a *long* way from <2
>> min
>> times with scp & <1 min times with rsync! And, I have no
>> redundancy or
>> distributed hash whatsoever.
>>
>> * Client config for the last test:
>> -----
>> # Webform Flat-File Cache Volume client configuration
>>
>> volume srv1
>> type protocol/client
>> option transport-type tcp
>> option remote-host cfs1
>> option remote-subvolume webform_cache_brick
>> end-volume
>>
>> volume writebehind
>> type performance/write-behind
>> option cache-size 4mb
>> option flush-behind on
>> subvolumes srv1
>> end-volume
>>
>> volume cache
>> type performance/io-cache
>> option cache-size 512mb
>> subvolumes writebehind
>> end-volume
>> -----
>>
>> Ben
>>
>> On Jun 3, 2009, at 4:33 PM, Vahriç Muhtaryan wrote:
>>
>>> For better understanding issue did you try 4 servers DHT only or 2
>>> servers
>>> DHT only or two servers replication only for find out real problem
>>> maybe
>>> replication or dht could have a bug ?
>>>
>>> -----Original Message-----
>>> From: gluster-users-bounces at gluster.org
>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Benjamin
>>> Krein
>>> Sent: Wednesday, June 03, 2009 11:00 PM
>>> To: Jasper van Wanrooy - Chatventure
>>> Cc: gluster-users at gluster.org
>>> Subject: Re: [Gluster-users] Horrible performance with small files
>>> (DHT/AFR)
>>>
>>> The current boxes I'm using for testing are as follows:
>>>
>>> * 2x dual-core Opteron ~2GHz (x86_64)
>>> * 4GB RAM
>>> * 4x 7200 RPM 73GB SATA - RAID1+0 w/3ware hardware controllers
>>>
>>> The server storage directories live in /home/clusterfs where /home
>>> is
>>> an ext3 partition mounted with noatime.
>>>
>>> These servers are not virtualized. They are running Ubuntu 8.04 LTS
>>> Server x86_64.
>>>
>>> The files I'm copying are all <2k javascript files (plain text)
>>> stored
>>> in 100 hash directories in each of 3 parent directories:
>>>
>>> /home/clusterfs/
>>> + parentdir1/
>>> | + 00/
>>> | | ...
>>> | + 99/
>>> + parentdir1/
>>> | + 00/
>>> | | ...
>>> | + 99/
>>> + parentdir1/
>>> + 00/
>>> | ...
>>> + 99/
>>>
>>> There are ~10k of these <2k javascript files distributed throughout
>>> the above directory structure totaling approximately 570MB. My
>>> tests
>>> have been copying that entire directory structure from a client
>>> machine into the glusterfs mountpoint on the client.
>>>
>>> Observing IO on both the client box & all the server boxes via
>>> iostat
>>> shows that the disks are doing *very* little work. Observing the
>>> CPU/
>>> memory load with top or htop shows that none of the boxes are CPU or
>>> memory bound. Observing the bandwidth in/out of the network
>>> interface
>>> shows <1MB/s throughput (we have a fully gigabit LAN!) which usually
>>> drops down to <150KB/s during the copy.
>>>
>>> scp'ing the same directory structure from the same client to one of
>>> the same servers will work at ~40-50MB/s sustained as a comparison.
>>> Here is the results of copying the same directory structure using
>>> rsync to the same partition:
>>>
>>> # time rsync -ap * benk at cfs1:~/cache/
>>> benk at cfs1's password:
>>>
>>> real 0m23.566s
>>> user 0m8.433s
>>> sys 0m4.580s
>>>
>>> Ben
>>>
>>> On Jun 3, 2009, at 3:16 PM, Jasper van Wanrooy - Chatventure wrote:
>>>
>>>> Hi Benjamin,
>>>>
>>>> That's not good news. What kind of hardware do you use? Is it
>>>> virtualised? Or do you use real boxes?
>>>> What kind of files are you copying in your test? What performance
>>>> do
>>>> you have when copying it to a local dir?
>>>>
>>>> Best regards Jasper
>>>>
>>>> ----- Original Message -----
>>>> From: "Benjamin Krein" <superbenk at superk.org>
>>>> To: "Jasper van Wanrooy - Chatventure" <jvanwanrooy at chatventure.nl>
>>>> Cc: "Vijay Bellur" <vijay at gluster.com>, gluster-users at gluster.org
>>>> Sent: Wednesday, 3 June, 2009 19:23:51 GMT +01:00 Amsterdam /
>>>> Berlin / Bern / Rome / Stockholm / Vienna
>>>> Subject: Re: [Gluster-users] Horrible performance with small files
>>>> (DHT/AFR)
>>>>
>>>> I reduced my config to only 2 servers (had to donate 2 of the 4 to
>>>> another project). I now have a single server using DHT (for future
>>>> scaling) and AFR to a mirrored server. Copy times are much better,
>>>> but still pretty horrible:
>>>>
>>>> # time cp -rp * /mnt/
>>>>
>>>> real 21m11.505s
>>>> user 0m1.000s
>>>> sys 0m6.416s
>>>>
>>>> Ben
>>>>
>>>> On Jun 3, 2009, at 3:13 AM, Jasper van Wanrooy - Chatventure wrote:
>>>>
>>>>> Hi Benjamin,
>>>>>
>>>>> Did you also try with a lower thread-count. Actually I'm using 3
>>>>> threads.
>>>>>
>>>>> Best Regards Jasper
>>>>>
>>>>>
>>>>> On 2 jun 2009, at 18:25, Benjamin Krein wrote:
>>>>>
>>>>>> I do not see any difference with autoscaling removed. Current
>>>>>> server config:
>>>>>>
>>>>>> # webform flat-file cache
>>>>>>
>>>>>> volume webform_cache
>>>>>> type storage/posix
>>>>>> option directory /home/clusterfs/webform/cache
>>>>>> end-volume
>>>>>>
>>>>>> volume webform_cache_locks
>>>>>> type features/locks
>>>>>> subvolumes webform_cache
>>>>>> end-volume
>>>>>>
>>>>>> volume webform_cache_brick
>>>>>> type performance/io-threads
>>>>>> option thread-count 32
>>>>>> subvolumes webform_cache_locks
>>>>>> end-volume
>>>>>>
>>>>>> <<snip>>
>>>>>>
>>>>>> # GlusterFS Server
>>>>>> volume server
>>>>>> type protocol/server
>>>>>> option transport-type tcp
>>>>>> subvolumes dns_public_brick dns_private_brick webform_usage_brick
>>>>>> webform_cache_brick wordpress_uploads_brick subs_exports_brick
>>>>>> option auth.addr.dns_public_brick.allow 10.1.1.*
>>>>>> option auth.addr.dns_private_brick.allow 10.1.1.*
>>>>>> option auth.addr.webform_usage_brick.allow 10.1.1.*
>>>>>> option auth.addr.webform_cache_brick.allow 10.1.1.*
>>>>>> option auth.addr.wordpress_uploads_brick.allow 10.1.1.*
>>>>>> option auth.addr.subs_exports_brick.allow 10.1.1.*
>>>>>> end-volume
>>>>>>
>>>>>> # time cp -rp * /mnt/
>>>>>>
>>>>>> real 70m13.672s
>>>>>> user 0m1.168s
>>>>>> sys 0m8.377s
>>>>>>
>>>>>> NOTE: the above test was also done during peak hours when the
>>>>>> LAN/
>>>>>> dev server were in use which would cause some of the extra time.
>>>>>> This is still WAY too much, though.
>>>>>>
>>>>>> Ben
>>>>>>
>>>>>>
>>>>>> On Jun 1, 2009, at 1:40 PM, Vijay Bellur wrote:
>>>>>>
>>>>>>> Hi Benjamin,
>>>>>>>
>>>>>>> Could you please try by turning autoscaling off?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vijay
>>>>>>>
>>>>>>> Benjamin Krein wrote:
>>>>>>>> I'm seeing extremely poor performance writing small files to a
>>>>>>>> glusterfs DHT/AFR mount point. Here are the stats I'm seeing:
>>>>>>>>
>>>>>>>> * Number of files:
>>>>>>>> root at dev1|/home/aweber/cache|# find |wc -l
>>>>>>>> 102440
>>>>>>>>
>>>>>>>> * Average file size (bytes):
>>>>>>>> root at dev1|/home/aweber/cache|# ls -lR | awk '{sum += $5; n++;}
>>>>>>>> END {print sum/n;}'
>>>>>>>> 4776.47
>>>>>>>>
>>>>>>>> * Using scp:
>>>>>>>> root at dev1|/home/aweber/cache|# time scp -rp * benk at cfs1:~/
>>>>>>>> cache/
>>>>>>>>
>>>>>>>> real 1m38.726s
>>>>>>>> user 0m12.173s
>>>>>>>> sys 0m12.141s
>>>>>>>>
>>>>>>>> * Using cp to glusterfs mount point:
>>>>>>>> root at dev1|/home/aweber/cache|# time cp -rp * /mnt
>>>>>>>>
>>>>>>>> real 30m59.101s
>>>>>>>> user 0m1.296s
>>>>>>>> sys 0m5.820s
>>>>>>>>
>>>>>>>> Here is my configuration (currently, single client writing to 4
>>>>>>>> servers (2 DHT servers doing AFR):
>>>>>>>>
>>>>>>>> SERVER:
>>>>>>>>
>>>>>>>> # webform flat-file cache
>>>>>>>>
>>>>>>>> volume webform_cache
>>>>>>>> type storage/posix
>>>>>>>> option directory /home/clusterfs/webform/cache
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume webform_cache_locks
>>>>>>>> type features/locks
>>>>>>>> subvolumes webform_cache
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume webform_cache_brick
>>>>>>>> type performance/io-threads
>>>>>>>> option thread-count 32
>>>>>>>> option max-threads 128
>>>>>>>> option autoscaling on
>>>>>>>> subvolumes webform_cache_locks
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> <<snip>>
>>>>>>>>
>>>>>>>> # GlusterFS Server
>>>>>>>> volume server
>>>>>>>> type protocol/server
>>>>>>>> option transport-type tcp
>>>>>>>> subvolumes dns_public_brick dns_private_brick
>>>>>>>> webform_usage_brick
>>>>>>>> webform_cache_brick wordpress_uploads_brick subs_exports_brick
>>>>>>>> option auth.addr.dns_public_brick.allow 10.1.1.*
>>>>>>>> option auth.addr.dns_private_brick.allow 10.1.1.*
>>>>>>>> option auth.addr.webform_usage_brick.allow 10.1.1.*
>>>>>>>> option auth.addr.webform_cache_brick.allow 10.1.1.*
>>>>>>>> option auth.addr.wordpress_uploads_brick.allow 10.1.1.*
>>>>>>>> option auth.addr.subs_exports_brick.allow 10.1.1.*
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> CLIENT:
>>>>>>>>
>>>>>>>> # Webform Flat-File Cache Volume client configuration
>>>>>>>>
>>>>>>>> volume srv1
>>>>>>>> type protocol/client
>>>>>>>> option transport-type tcp
>>>>>>>> option remote-host cfs1
>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume srv2
>>>>>>>> type protocol/client
>>>>>>>> option transport-type tcp
>>>>>>>> option remote-host cfs2
>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume srv3
>>>>>>>> type protocol/client
>>>>>>>> option transport-type tcp
>>>>>>>> option remote-host cfs3
>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume srv4
>>>>>>>> type protocol/client
>>>>>>>> option transport-type tcp
>>>>>>>> option remote-host cfs4
>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume afr1
>>>>>>>> type cluster/afr
>>>>>>>> subvolumes srv1 srv3
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume afr2
>>>>>>>> type cluster/afr
>>>>>>>> subvolumes srv2 srv4
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume dist
>>>>>>>> type cluster/distribute
>>>>>>>> subvolumes afr1 afr2
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume writebehind
>>>>>>>> type performance/write-behind
>>>>>>>> option cache-size 4mb
>>>>>>>> option flush-behind on
>>>>>>>> subvolumes dist
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> volume cache
>>>>>>>> type performance/io-cache
>>>>>>>> option cache-size 512mb
>>>>>>>> subvolumes writebehind
>>>>>>>> end-volume
>>>>>>>>
>>>>>>>> Benjamin Krein
>>>>>>>> www.superk.org
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list