[Gluster-devel] accessing glusterfs mounted share - really slow
Matthias Albert
gluster at linux4experts.de
Fri Aug 31 09:50:18 UTC 2007
Hi,
Anand Avati schrieb:
> Matthias,
> have you loaded io-threads on the server bricks? io-threads is meant
> to classify file I/O and metadata operations into seperate threads,
> thus, readdir() operations would not fall-in-line with ongoing writes,
> instead gets into a different queue and processed by a seperate
> thread. this should make ls more 'interactive' while disk I/O is
> happening.
>
on server side I only do a export of my volumes
#server-side
---snip---
...
volume sdb1
type storage/posix
option directory /export/sdb1
end-volume
volume sdc1
type storage/posix
option directory /export/sdc1
end-volume
volume server
type protocol/server
option transport-type tcp/server
option listen-port 6997
subvolumes sdb1 sdc1
option auth.ip.sdb1.allow *
option auth.ip.sdc1.allow *
end-volume
...
---snap---
and on client side I added the afr, unify ,writebehind and readahead
translator.
#client-side
---snip---
...
volume gluster1-sdb1
type protocol/client
option transport-type tcp/client
option remote-host gluster1
option remote-port 6997
option remote-subvolume sdb1
end-volume
volume gluster1-sdc1
type protocol/client
option transport-type tcp/client
option remote-host gluster1
option remote-port 6997
option remote-subvolume sdc1
end-volume
...
volume afr1
type cluster/afr
subvolumes gluster3-hdb1 gluster4-hdc1
option replicate *:2
end-volume
...
volume cluster
type cluster/unify
subvolumes afr1 afr2 afr3 afr4 gluster2-hdb1
option scheduler alu
option alu.limits.min-free-disk 6GB
option alu.limits.max-open-files 10000
option namespace brick
option alu.order
disk-usage:read-usage:write-usage:open-files-usage:disk-speed-usage
option alu.disk-usage.entry-threshold 100GB
option alu.disk-usage.exit-threshold 60MB
option alu.open-files-usage.entry-threshold 1024
option alu.open-files-usage.exit-threshold 32
option alu.stat-refresh.interval 10sec
end-volume
volume writebehind
type performance/write-behind
option aggregate-size 131072 # unit in bytes
option flush-behind off
subvolumes cluster
end-volume
volume readahead
type performance/read-ahead
option page-size 65536
option page-count 16
subvolumes writebehind
end-volume
---snap---
So I guess, I should add a readahead and writebehind translator on
server side?
Regards,
Matthias
> avati
>
> 2007/8/31, Matthias Albert <gluster at linux4experts.de
> <mailto:gluster at linux4experts.de>>:
>
> Hi Krishna,
>
>
> Krishna Srinivas schrieb:
> > Hi Matthias,
> >
> > If I understand correctly, for you all the operations are fine, but
> > when a "cp" is being done and simultaneously you do "ls" from
> > another client, the "ls" is slow?
> >
> yepp, absolutly correct. Only If I do a "cp or dd for example" the
> ls or
> tab completion is really slow and only in the glusterfs mounted share.
>
> Matthias
>
> > Krishna
> >
> > On 8/31/07, Matthias Albert <gluster at linux4experts.de
> <mailto:gluster at linux4experts.de>> wrote:
> >
> >> Hi all,
> >>
> >> first of all, I've to say that gluterfs is really cool and
> absolutly
> >> great. I'm not a cluster filesystem specialist but I
> tested/configured
> >> openafs and lustre and both of them are so huge and complicated.
> >> As I saw glusterfs and played a little bit with it, I was really
> >> surprised how easy it is to setup a cluster filesystem without
> extra
> >> acl's, without formatting the new filesystem without a
> >> metadata/objectserver :-). Thanks a lot for this.
> >>
> >> Of course I've some questions :-).
> >>
> >> I've setup 4 glusterfsd server, each of them with a storage of
> about
> >> 400-500 Gig pre-tax.
> >> On client side I made different afr's over my remote volumes and
> >> finally a unify over the afr's. Readahead and writebehind is
> also enabled.
> >>
> >> Everything is working fine. I can copy "tons" of Gigabytes in my
> >> glusterfs without any problms and also my performance is
> absolutly great.
> >>
> >> But every time I start a "cp" or do a "dd test (to write some
> testfiles
> >> in the gluster storage) on some of my clients (I've 3 glusterfs
> clients
> >> one of them is a bacula server which uses the glusterfs as
> storage)
> >> all access from my glusterfs clients to the mounted share is really
> >> slow. It takes sometimes about 3-4 seconds till my ls is
> printing the
> >> output of the directory.
> >>
> >> e.g.
> >> ---snip---
> >> bash# df -h
> >> glusterfs 892G 84G 809G 10% /backup
> >>
> >> gsx:/backup/vmware-images # time ll
> >> ...
> >> ...
> >> real 0m2.863s
> >> user 0m0.004s
> >> sys 0m0.005s
> >> gsx:/backup/vmware-images #
> >> ---snap---
> >>
> >> Also the "tab completion" in the mounted glusterfs share is
> really slow.
> >> Access of not mounted glusterfs share is just normal (accessing
> /etc
> >> /usr/ /root etc. )
> >>
> >> Does anyone know these "phenomenon"?
> >>
> >> I'm using Debian as distro for all of my servers and Debian and
> SuSE on
> >> Client side.
> >>
> >> glusterfs version: glusterfs--mainline--2.5 patch-459
> >> fuse: fuse-2.7.0-glfs3
> >>
> >> If needed I can post my configs, strace outputs of ls -la and
> so on.
> >>
> >> Regards,
> >>
> >> Matthias
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Gluster-devel mailing list
> >> Gluster-devel at nongnu.org <mailto:Gluster-devel at nongnu.org>
> >> http://lists.nongnu.org/mailman/listinfo/gluster-devel
> <http://lists.nongnu.org/mailman/listinfo/gluster-devel>
> >>
> >>
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org <mailto:Gluster-devel at nongnu.org>
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
> It always takes longer than you expect, even when you take into
> account Hofstadter's Law.
>
> -- Hofstadter's Law
More information about the Gluster-devel
mailing list