[Gluster-devel] performance question

[Krishna Srinivas]

Gerry,

If you are using xfs try this command:
mkfs.xfs -i attr=2 /dev/<hda>

if you are using ext3:
mkfs.ext3 -i 256 /dev/<hda>

This will ensure that extended attributes are put into the inode
structure itself. If this is not done, an extra block is allocated
for EAs thus taking more time. It dramatically improved the
performance for me (4 to 5 folds) for creation of files.
reiserfs was fast by default, so it must be using in-inode EAs
by default.

Regards
Krishna

On 7/5/07, Krishna Srinivas <krishna at zresearch.com> wrote:
> Hi Gerry,
>
> Good observation. I was checking on the performance with
> self-heal turned off and the turning it on.
>
> On glusterfs mounted directory:
> With self-heal on:
> bash-3.1# time cp -r /etc/ .
>
> real    0m28.048s
> user    0m0.004s
> sys     0m0.060s
> bash-3.1# time rm -rf etc/
>
> real    0m28.666s
> user    0m0.000s
> sys     0m0.032s
> bash-3.1#
>
> With self-heal off:
> bash-3.1# time cp -r /etc/ .
>
> real    0m2.376s
> user    0m0.012s
> sys     0m0.060s
> bash-3.1# time rm -rf etc/
>
> real    0m3.639s
> user    0m0.000s
> sys     0m0.000s
> bash-3.1#
>
> So there is significant difference. The difference is that there is no
> external attributes management when the self-heal is off. AFR code
> does getxattr/setxattr when self-heal is on, so this introduces
> some overhead. Also there will be overhead in the backend filesystem
> code to manage xttrs.
>
> However, if i try to delete the etc directory directly in the backend
> (it was copied through glusterfs with self-heal on)
> bash-3.1# time rm -rf /export/dir1/etc/
>
> real    0m18.414s
> user    0m0.000s
> sys     0m0.132s
> bash-3.1#
>
> So there is significant overhead in the backend filesystem itself
> when xattrs are involved.
>
> Checking the overhead on open/close calls:
> (here a.out opens, writes a byte, closes)
>
> Selfheal is on:
> bash-3.1# time find . -type f -exec /root/a.out {}  \;
>
> real    0m1.529s
> user    0m0.120s
> sys     0m0.284s
> bash-3.1#
>
> Selfheal off:
> bash-3.1# time find . -type f -exec /root/a.out {}  \;
>
> real    0m0.577s
> user    0m0.124s
> sys     0m0.260s
> bash-3.1#
>
>
>
> There is  not much difference here. So setxattr/getxattr
> does not take much time if xattrs are already existing on
> he file. Hence there will be lot overhead only during create/unlink.
>
> We will see if we can optimize anyway in the AFR code.
> We have to note that backend system takes a lot
> of time during create/delete which we dont have control
> over. But still it is acceptable as there is not much overhead
> during open/close/write calls.
>
> Regards
> Krishna
>
> On 7/5/07, Anand Avati <avati at zresearch.com> wrote:
> >
> > Gerry,
> > please use the write-behind translator on the client side (above AFR)
> >
> > thanks,
> > avati
> >
> > 2007/7/4, Gerry Reno <greno at verizon.net>:
> > >
> > > In copying my /usr tree (4.9G) to a gluster client mount with a 4 brick
> > > AFR with no other translators I see it is taking about 1 hr. 45 min.  Is
> > > this normal performance?
> > > Now this is with the bricks all on the same machine and same ext3
> > > filesystem, but that seems like a long time even still.
> > >
> > > Gerry
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at nongnu.org
> > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> > >
> >
> >
> >
> > --
> > Anand V. Avati
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>