[Gluster-users] Write performance tragically slow

[Barry Jaspan]

On Thu, Nov 18, 2010 at 9:41 AM, Anand Avati <anand.avati at gmail.com> wrote:

> FUSE based filesystems have to face the unfortunate overhead of context
> switches between application and the filesystem process.


Gluster 2.x had a library named "booster" that bypassed the FUSE layer by
providing implementations of all the filesystem syscalls that recognized
glusterfs file descriptors and went directly to the glusterfs code instead
of the kernel for those operations.

Why was booster removed from Gluster 3.x?

Barry


> GlusterFS does
> aggregate small writes but the aggregation code lies beyond the context
> switch hill. Nothing much can be done about it. We do however expose
> stbuf.st_blksize to be 128KB which applications like 'cp' honor and perform
> IO with that block size.
>
> Avati
>
> On Mon, Nov 15, 2010 at 8:19 PM, Pavel Snajdr <snajpa at snajpa.net> wrote:
>
> > I played with it further after I wrote that e-mail and I found out, that
> > this interesting thing:
> >
> > dd if=/dev/zero bs=X count=10000
> >
> > X       troughput
> > 1k      ~3MB/s
> > 2k      ~6MB/s
> > 4k      ~12MB/s
> > 8k      ~24MB/s
> > 16      ~48MB/s
> > ...
> > 128k    110MB/s
> >
> > => meaning I have tragically low IOPS (+- according to my 0.030ms network
> > latency).
> >
> > Gluster config was one generated by "gluster" command (version 3.1.0),
> but
> > I've tried either latest 3.0.x, resulting the same.
> >
> > I've tried probably all performance translators available, even AFR
> instead
> > of replicate.
> >
> > Is there any way to aggregate the writes and propagate them to the
> network
> > in bigger chunks, ie. 128k?
> >
> > What I don't understand is how for example NFS does this? With NFS there
> is
> > no such problem and even though IOPS are slig
> >
> >
> > On 15.11.2010 14:22, Luis E. Cerezo wrote:
> >
> >> have you tried larger files? I have seen a note somewhere that refers to
> >> tons of itty bitty files, it even cites the kernel source as an example.
> I
> >> can't seem to find it.
> >>
> >> Could you try larger than 1k files?
> >>
> >> -luis
> >>
> >>
> >>
> >>
> >> On Nov 10, 2010, at 8:23 PM, Pavel Snajdr wrote:
> >>
> >>  Hello,
> >>>
> >>> I imagine you get this kind of messages all the time, but just in case:
> >>>
> >>> I have setup with 2 storage servers with debian package of gluster -
> >>> version 3.1.0
> >>>
> >>> They are connected by dedicated 1 gigabit ethernet cards.
> >>>
> >>> I've set up simple replicated storage with 2 replicas and transport
> over
> >>> TCP (just followed the how to on the wiki with obvious changes).
> >>>
> >>> Here goes my problem:
> >>>
> >>> If I try to copy small files (i.e. extract kernel source) I get a
> >>> horrible results:
> >>>
> >>> praha-storage2:/mnt/test# time tar xf linux-2.6.26.8.tar.bz2
> >>>
> >>>
> >>> real 15m19.825s
> >>> user 0m13.989s
> >>> sys 0m5.152s
> >>>
> >>> likewise when rsyncing OpenVZ VPSes - I just can't get over 2MB/s in
> >>> syncing.
> >>>
> >>> I've monitored all resources - CPU load, network, disk I/O - they are
> all
> >>> used up to 0.00000nothing %.
> >>>
> >>> Network latency is about 0.11 ms all the time.
> >>>
> >>> Is this normal, or am I doing something wrong?
> >>>
> >>> Please help. I am frustrated :(
> >>>
> >>> --
> >>> S pozdravem / Best Regards
> >>>
> >>> Pavel Šnajdr
> >>> +420 720 107 791
> >>>
> >>> http://vpsfree.cz
> >>>
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >>>
> >>>
> >> Luis E. Cerezo
> >>
> >> blog:     http://www.luiscerezo.org
> >> fotofun: http://www.flickr.com/photos/luiscerezo/
> >> twitter: http://twitter.com/luiscerezo/
> >> Voice: +1 412 223 7396
> >>
> >>
> >>  _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> >
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
>


-- 
Barry Jaspan
Senior Architect | Acquia <http://acquia.com>
barry.jaspan at acquia.com | (c) 617.905.2208 | (w) 978.296.5231

"Get a free, hosted Drupal 7 site: http://www.drupalgardens.com"