[Gluster-users] Horrible performance with small files (DHT/AFR)
Benjamin Krein
superbenk at superk.org
Tue Jun 9 11:15:14 UTC 2009
Are there any other things I can try here or have I dug up a bug that
is being looked at still? We'd love to roll out glusterfs into
production use, but the performance with small files is unacceptable.
Ben
On Jun 5, 2009, at 9:02 AM, Benjamin Krein wrote:
>
> On Jun 5, 2009, at 2:23 AM, Pavan Vilas Sondur wrote:
>
>> Hi Benjamin,
>> Can you provide us with the following information to narrow down
>> the issue.
>> 1. The network configuration of the GlusterFS deployment and the
>> bandwidth that is available
>> for GlusterFS to use
>
> All these servers (servers/clients) are on a full-duplex gigabit LAN
> with gigabit NICs. As I've noted in previous emails, I can easily
> utilize the bandwidth with scp & rsync. As you'll see below,
> glusterfs seems to do very well with large files as well.
>
>> 2. Does performance also get affected the same way with large files
>> too? Or is it with just small files
>> as you have mentioned? Let us know the performance of GlusterFS
>> with large files.
>
> Here are a bunch of tests I did with various configurations &
> copying large files:
>
> * Single server - cfs1 - large files (~480MB each)
> root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
> webform_cache.tar /mnt/large_file_\$i.tar; done;"
> `webform_cache.tar' -> `/mnt/large_file_1.tar'
> `webform_cache.tar' -> `/mnt/large_file_2.tar'
> `webform_cache.tar' -> `/mnt/large_file_3.tar'
> `webform_cache.tar' -> `/mnt/large_file_4.tar'
> `webform_cache.tar' -> `/mnt/large_file_5.tar'
>
> real 0m23.726s
> user 0m0.128s
> sys 0m4.972s
>
> # Interface RX Rate RX # TX
> Rate TX #
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ───────────────────────
> cfs1 (source: local)
> 0 eth1 91.39MiB 63911
> 1.98MiB 27326
>
> * Two servers w/ AFR only - large files (~480MB each)
> root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
> webform_cache.tar /mnt/large_file_\$i.tar; done;"
> `webform_cache.tar' -> `/mnt/large_file_1.tar'
> `webform_cache.tar' -> `/mnt/large_file_2.tar'
> `webform_cache.tar' -> `/mnt/large_file_3.tar'
> `webform_cache.tar' -> `/mnt/large_file_4.tar'
> `webform_cache.tar' -> `/mnt/large_file_5.tar'
>
> real 0m43.354s
> user 0m0.100s
> sys 0m3.044s
>
> # Interface RX Rate RX # TX
> Rate TX #
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ───────────────────────
> cfs1 (source: local)
> 0 eth1 57.73MiB 39931
> 910.95KiB 10733
>
> * Two servers w/DHT+AFR - large files (~480MB each)
> root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
> webform_cache.tar /mnt/large_file_\$i.tar; done;"
> `webform_cache.tar' -> `/mnt/large_file_1.tar'
> `webform_cache.tar' -> `/mnt/large_file_2.tar'
> `webform_cache.tar' -> `/mnt/large_file_3.tar'
> `webform_cache.tar' -> `/mnt/large_file_4.tar'
> `webform_cache.tar' -> `/mnt/large_file_5.tar'
>
> real 0m43.294s
> user 0m0.100s
> sys 0m3.356s
>
> # Interface RX Rate RX # TX
> Rate TX #
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ───────────────────────
> cfs1 (source: local)
> 0 eth1 58.15MiB 40224
> 1.52MiB 20174
> # Interface RX Rate RX # TX
> Rate TX #
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ───────────────────────
> cfs2 (source: local)
> 0 eth1 55.99MiB 38684
> 755.51KiB 9521
>
> * Two servers DHT *only* - large files (~480MB each) - NOTE: only
> cfs1 was ever populated, isn't DHT supposed to distribute the files?:
> root at dev1|~|# time sh -c "for i in 1 2 3 4 5; do cp -v
> webform_cache.tar /mnt/\$i/large_file.tar; done;"
> `webform_cache.tar' -> `/mnt/1/large_file.tar'
> `webform_cache.tar' -> `/mnt/2/large_file.tar'
> `webform_cache.tar' -> `/mnt/3/large_file.tar'
> `webform_cache.tar' -> `/mnt/4/large_file.tar'
> `webform_cache.tar' -> `/mnt/5/large_file.tar'
>
> real 0m40.062s
> user 0m0.204s
> sys 0m3.500s
>
> # Interface RX Rate RX # TX
> Rate TX #
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ───────────────────────
> cfs1 (source: local)
> 0 eth1 112.96MiB 78190
> 1.66MiB 21994
> # Interface RX Rate RX # TX
> Rate TX #
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ─
> ───────────────────────
> cfs2 (source: local)
> 0 eth1 1019.00B 7
> 83.00B 0
>
> root at cfs1|/home/clusterfs/webform/cache2|# find -ls
> 294926 8 drwxrwxr-x 7 www-data www-data 4096 Jun 5 08:52 .
> 294927 8 drwxr-xr-x 2 root root 4096 Jun 5 08:53 ./1
> 294933 489716 -rw-r--r-- 1 root root 500971520 Jun 5
> 08:54 ./1/large_file.tar
> 294931 8 drwxr-xr-x 2 root root 4096 Jun 5 08:53 ./5
> 294932 489716 -rw-r--r-- 1 root root 500971520 Jun 5
> 08:54 ./5/large_file.tar
> 294930 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./4
> 294936 489716 -rw-r--r-- 1 root root 500971520 Jun 5
> 08:54 ./4/large_file.tar
> 294928 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./2
> 294934 489716 -rw-r--r-- 1 root root 500971520 Jun 5
> 08:54 ./2/large_file.tar
> 294929 8 drwxr-xr-x 2 root root 4096 Jun 5 08:54 ./3
> 294935 489716 -rw-r--r-- 1 root root 500971520 Jun 5
> 08:54 ./3/large_file.tar
>
>
> root at cfs2|/home/clusterfs/webform/cache2|# find -ls
> 3547150 8 drwxrwxr-x 7 www-data www-data 4096 Jun 5 08:52 .
> 3547153 8 drwxr-xr-x 2 root root 4096 Jun 5
> 08:52 ./3
> 3547155 8 drwxr-xr-x 2 root root 4096 Jun 5
> 08:52 ./5
> 3547154 8 drwxr-xr-x 2 root root 4096 Jun 5
> 08:52 ./4
> 3547152 8 drwxr-xr-x 2 root root 4096 Jun 5
> 08:52 ./2
> 3547151 8 drwxr-xr-x 2 root root 4096 Jun 5
> 08:52 ./1
>
>>
>> There haven't really been an issue with using large number of small
>> files as such in the past. Nevertheless,
>> we'll look into this once you give us the above details.
>>
>
> I don't feel that the tests I'm performing are all that abnormal or
> out of the ordinary, so I'm surprised that I'm the only one having
> the problems. As you can see from the large file results above, the
> problem is clearly related to small files only.
>
> Thanks for your continued interest in resolving this!
>
> Ben
>
>>
>> On 04/06/09 16:21 -0400, Benjamin Krein wrote:
>>> Here are some more details with different configs:
>>>
>>> * Only AFR between cfs1 & cfs2:
>>> root at dev1# time cp -rp * /mnt/
>>>
>>> real 16m45.995s
>>> user 0m1.104s
>>> sys 0m5.528s
>>>
>>> * Single server - cfs1:
>>> root at dev1# time cp -rp * /mnt/
>>>
>>> real 10m33.967s
>>> user 0m0.764s
>>> sys 0m5.516s
>>>
>>> * Stats via bmon on cfs1 during above copy:
>>> # Interface RX Rate RX # TX Rate
>>> TX #
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ─
>>> ──────────────────────
>>> ─
>>> ─
>>> ──────────────────────
>>> cfs1 (source: local)
>>> 0 eth1 951.25KiB 1892 254.00KiB
>>> 1633
>>>
>>> It gets progressively better, but that's still a *long* way from
>>> <2 min
>>> times with scp & <1 min times with rsync! And, I have no
>>> redundancy or
>>> distributed hash whatsoever.
>>>
>>> * Client config for the last test:
>>> -----
>>> # Webform Flat-File Cache Volume client configuration
>>>
>>> volume srv1
>>> type protocol/client
>>> option transport-type tcp
>>> option remote-host cfs1
>>> option remote-subvolume webform_cache_brick
>>> end-volume
>>>
>>> volume writebehind
>>> type performance/write-behind
>>> option cache-size 4mb
>>> option flush-behind on
>>> subvolumes srv1
>>> end-volume
>>>
>>> volume cache
>>> type performance/io-cache
>>> option cache-size 512mb
>>> subvolumes writebehind
>>> end-volume
>>> -----
>>>
>>> Ben
>>>
>>> On Jun 3, 2009, at 4:33 PM, Vahriç Muhtaryan wrote:
>>>
>>>> For better understanding issue did you try 4 servers DHT only or 2
>>>> servers
>>>> DHT only or two servers replication only for find out real problem
>>>> maybe
>>>> replication or dht could have a bug ?
>>>>
>>>> -----Original Message-----
>>>> From: gluster-users-bounces at gluster.org
>>>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Benjamin
>>>> Krein
>>>> Sent: Wednesday, June 03, 2009 11:00 PM
>>>> To: Jasper van Wanrooy - Chatventure
>>>> Cc: gluster-users at gluster.org
>>>> Subject: Re: [Gluster-users] Horrible performance with small files
>>>> (DHT/AFR)
>>>>
>>>> The current boxes I'm using for testing are as follows:
>>>>
>>>> * 2x dual-core Opteron ~2GHz (x86_64)
>>>> * 4GB RAM
>>>> * 4x 7200 RPM 73GB SATA - RAID1+0 w/3ware hardware controllers
>>>>
>>>> The server storage directories live in /home/clusterfs where /
>>>> home is
>>>> an ext3 partition mounted with noatime.
>>>>
>>>> These servers are not virtualized. They are running Ubuntu 8.04
>>>> LTS
>>>> Server x86_64.
>>>>
>>>> The files I'm copying are all <2k javascript files (plain text)
>>>> stored
>>>> in 100 hash directories in each of 3 parent directories:
>>>>
>>>> /home/clusterfs/
>>>> + parentdir1/
>>>> | + 00/
>>>> | | ...
>>>> | + 99/
>>>> + parentdir1/
>>>> | + 00/
>>>> | | ...
>>>> | + 99/
>>>> + parentdir1/
>>>> + 00/
>>>> | ...
>>>> + 99/
>>>>
>>>> There are ~10k of these <2k javascript files distributed throughout
>>>> the above directory structure totaling approximately 570MB. My
>>>> tests
>>>> have been copying that entire directory structure from a client
>>>> machine into the glusterfs mountpoint on the client.
>>>>
>>>> Observing IO on both the client box & all the server boxes via
>>>> iostat
>>>> shows that the disks are doing *very* little work. Observing the
>>>> CPU/
>>>> memory load with top or htop shows that none of the boxes are CPU
>>>> or
>>>> memory bound. Observing the bandwidth in/out of the network
>>>> interface
>>>> shows <1MB/s throughput (we have a fully gigabit LAN!) which
>>>> usually
>>>> drops down to <150KB/s during the copy.
>>>>
>>>> scp'ing the same directory structure from the same client to one of
>>>> the same servers will work at ~40-50MB/s sustained as a comparison.
>>>> Here is the results of copying the same directory structure using
>>>> rsync to the same partition:
>>>>
>>>> # time rsync -ap * benk at cfs1:~/cache/
>>>> benk at cfs1's password:
>>>>
>>>> real 0m23.566s
>>>> user 0m8.433s
>>>> sys 0m4.580s
>>>>
>>>> Ben
>>>>
>>>> On Jun 3, 2009, at 3:16 PM, Jasper van Wanrooy - Chatventure wrote:
>>>>
>>>>> Hi Benjamin,
>>>>>
>>>>> That's not good news. What kind of hardware do you use? Is it
>>>>> virtualised? Or do you use real boxes?
>>>>> What kind of files are you copying in your test? What
>>>>> performance do
>>>>> you have when copying it to a local dir?
>>>>>
>>>>> Best regards Jasper
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Benjamin Krein" <superbenk at superk.org>
>>>>> To: "Jasper van Wanrooy - Chatventure"
>>>>> <jvanwanrooy at chatventure.nl>
>>>>> Cc: "Vijay Bellur" <vijay at gluster.com>, gluster-users at gluster.org
>>>>> Sent: Wednesday, 3 June, 2009 19:23:51 GMT +01:00 Amsterdam /
>>>>> Berlin / Bern / Rome / Stockholm / Vienna
>>>>> Subject: Re: [Gluster-users] Horrible performance with small files
>>>>> (DHT/AFR)
>>>>>
>>>>> I reduced my config to only 2 servers (had to donate 2 of the 4 to
>>>>> another project). I now have a single server using DHT (for
>>>>> future
>>>>> scaling) and AFR to a mirrored server. Copy times are much
>>>>> better,
>>>>> but still pretty horrible:
>>>>>
>>>>> # time cp -rp * /mnt/
>>>>>
>>>>> real 21m11.505s
>>>>> user 0m1.000s
>>>>> sys 0m6.416s
>>>>>
>>>>> Ben
>>>>>
>>>>> On Jun 3, 2009, at 3:13 AM, Jasper van Wanrooy - Chatventure
>>>>> wrote:
>>>>>
>>>>>> Hi Benjamin,
>>>>>>
>>>>>> Did you also try with a lower thread-count. Actually I'm using 3
>>>>>> threads.
>>>>>>
>>>>>> Best Regards Jasper
>>>>>>
>>>>>>
>>>>>> On 2 jun 2009, at 18:25, Benjamin Krein wrote:
>>>>>>
>>>>>>> I do not see any difference with autoscaling removed. Current
>>>>>>> server config:
>>>>>>>
>>>>>>> # webform flat-file cache
>>>>>>>
>>>>>>> volume webform_cache
>>>>>>> type storage/posix
>>>>>>> option directory /home/clusterfs/webform/cache
>>>>>>> end-volume
>>>>>>>
>>>>>>> volume webform_cache_locks
>>>>>>> type features/locks
>>>>>>> subvolumes webform_cache
>>>>>>> end-volume
>>>>>>>
>>>>>>> volume webform_cache_brick
>>>>>>> type performance/io-threads
>>>>>>> option thread-count 32
>>>>>>> subvolumes webform_cache_locks
>>>>>>> end-volume
>>>>>>>
>>>>>>> <<snip>>
>>>>>>>
>>>>>>> # GlusterFS Server
>>>>>>> volume server
>>>>>>> type protocol/server
>>>>>>> option transport-type tcp
>>>>>>> subvolumes dns_public_brick dns_private_brick
>>>>>>> webform_usage_brick
>>>>>>> webform_cache_brick wordpress_uploads_brick subs_exports_brick
>>>>>>> option auth.addr.dns_public_brick.allow 10.1.1.*
>>>>>>> option auth.addr.dns_private_brick.allow 10.1.1.*
>>>>>>> option auth.addr.webform_usage_brick.allow 10.1.1.*
>>>>>>> option auth.addr.webform_cache_brick.allow 10.1.1.*
>>>>>>> option auth.addr.wordpress_uploads_brick.allow 10.1.1.*
>>>>>>> option auth.addr.subs_exports_brick.allow 10.1.1.*
>>>>>>> end-volume
>>>>>>>
>>>>>>> # time cp -rp * /mnt/
>>>>>>>
>>>>>>> real 70m13.672s
>>>>>>> user 0m1.168s
>>>>>>> sys 0m8.377s
>>>>>>>
>>>>>>> NOTE: the above test was also done during peak hours when the
>>>>>>> LAN/
>>>>>>> dev server were in use which would cause some of the extra time.
>>>>>>> This is still WAY too much, though.
>>>>>>>
>>>>>>> Ben
>>>>>>>
>>>>>>>
>>>>>>> On Jun 1, 2009, at 1:40 PM, Vijay Bellur wrote:
>>>>>>>
>>>>>>>> Hi Benjamin,
>>>>>>>>
>>>>>>>> Could you please try by turning autoscaling off?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Vijay
>>>>>>>>
>>>>>>>> Benjamin Krein wrote:
>>>>>>>>> I'm seeing extremely poor performance writing small files to a
>>>>>>>>> glusterfs DHT/AFR mount point. Here are the stats I'm seeing:
>>>>>>>>>
>>>>>>>>> * Number of files:
>>>>>>>>> root at dev1|/home/aweber/cache|# find |wc -l
>>>>>>>>> 102440
>>>>>>>>>
>>>>>>>>> * Average file size (bytes):
>>>>>>>>> root at dev1|/home/aweber/cache|# ls -lR | awk '{sum += $5; n++;}
>>>>>>>>> END {print sum/n;}'
>>>>>>>>> 4776.47
>>>>>>>>>
>>>>>>>>> * Using scp:
>>>>>>>>> root at dev1|/home/aweber/cache|# time scp -rp * benk at cfs1:~/
>>>>>>>>> cache/
>>>>>>>>>
>>>>>>>>> real 1m38.726s
>>>>>>>>> user 0m12.173s
>>>>>>>>> sys 0m12.141s
>>>>>>>>>
>>>>>>>>> * Using cp to glusterfs mount point:
>>>>>>>>> root at dev1|/home/aweber/cache|# time cp -rp * /mnt
>>>>>>>>>
>>>>>>>>> real 30m59.101s
>>>>>>>>> user 0m1.296s
>>>>>>>>> sys 0m5.820s
>>>>>>>>>
>>>>>>>>> Here is my configuration (currently, single client writing
>>>>>>>>> to 4
>>>>>>>>> servers (2 DHT servers doing AFR):
>>>>>>>>>
>>>>>>>>> SERVER:
>>>>>>>>>
>>>>>>>>> # webform flat-file cache
>>>>>>>>>
>>>>>>>>> volume webform_cache
>>>>>>>>> type storage/posix
>>>>>>>>> option directory /home/clusterfs/webform/cache
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume webform_cache_locks
>>>>>>>>> type features/locks
>>>>>>>>> subvolumes webform_cache
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume webform_cache_brick
>>>>>>>>> type performance/io-threads
>>>>>>>>> option thread-count 32
>>>>>>>>> option max-threads 128
>>>>>>>>> option autoscaling on
>>>>>>>>> subvolumes webform_cache_locks
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> <<snip>>
>>>>>>>>>
>>>>>>>>> # GlusterFS Server
>>>>>>>>> volume server
>>>>>>>>> type protocol/server
>>>>>>>>> option transport-type tcp
>>>>>>>>> subvolumes dns_public_brick dns_private_brick
>>>>>>>>> webform_usage_brick
>>>>>>>>> webform_cache_brick wordpress_uploads_brick subs_exports_brick
>>>>>>>>> option auth.addr.dns_public_brick.allow 10.1.1.*
>>>>>>>>> option auth.addr.dns_private_brick.allow 10.1.1.*
>>>>>>>>> option auth.addr.webform_usage_brick.allow 10.1.1.*
>>>>>>>>> option auth.addr.webform_cache_brick.allow 10.1.1.*
>>>>>>>>> option auth.addr.wordpress_uploads_brick.allow 10.1.1.*
>>>>>>>>> option auth.addr.subs_exports_brick.allow 10.1.1.*
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> CLIENT:
>>>>>>>>>
>>>>>>>>> # Webform Flat-File Cache Volume client configuration
>>>>>>>>>
>>>>>>>>> volume srv1
>>>>>>>>> type protocol/client
>>>>>>>>> option transport-type tcp
>>>>>>>>> option remote-host cfs1
>>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume srv2
>>>>>>>>> type protocol/client
>>>>>>>>> option transport-type tcp
>>>>>>>>> option remote-host cfs2
>>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume srv3
>>>>>>>>> type protocol/client
>>>>>>>>> option transport-type tcp
>>>>>>>>> option remote-host cfs3
>>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume srv4
>>>>>>>>> type protocol/client
>>>>>>>>> option transport-type tcp
>>>>>>>>> option remote-host cfs4
>>>>>>>>> option remote-subvolume webform_cache_brick
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume afr1
>>>>>>>>> type cluster/afr
>>>>>>>>> subvolumes srv1 srv3
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume afr2
>>>>>>>>> type cluster/afr
>>>>>>>>> subvolumes srv2 srv4
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume dist
>>>>>>>>> type cluster/distribute
>>>>>>>>> subvolumes afr1 afr2
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume writebehind
>>>>>>>>> type performance/write-behind
>>>>>>>>> option cache-size 4mb
>>>>>>>>> option flush-behind on
>>>>>>>>> subvolumes dist
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> volume cache
>>>>>>>>> type performance/io-cache
>>>>>>>>> option cache-size 512mb
>>>>>>>>> subvolumes writebehind
>>>>>>>>> end-volume
>>>>>>>>>
>>>>>>>>> Benjamin Krein
>>>>>>>>> www.superk.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list