[Gluster-users] gluster local vs local = gluster x4 slower

Wed Mar 24 03:27:23 UTC 2010

It might also be useful overall to know what you want to achieve. Its better to do sizing, performance etc if there is clarity on what is to be achieved. Once that is clear, it would be more useful to say if something is possible or not with the config you are trying and why or why not and whether even the expectations are justified or not from what is essentially a distributed networked FS.

----- Original Message -----
From: "Jeremy Enos" <jenos at ncsa.uiuc.edu>
To: "Stephan von Krawczynski" <skraw at ithnet.com>
Cc: "Tejas N. Bhise" <tejas at gluster.com>, gluster-users at gluster.org
Sent: Wednesday, March 24, 2010 5:41:28 AM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
Subject: Re: [Gluster-users] gluster local vs local = gluster x4 slower

Stephan is correct- I primarily did this test to show a demonstrable 
overhead example that I'm trying to eliminate.  It's pronounced enough 
that it can be seen on a single disk / single node configuration, which 
is good in a way (so anyone can easily repro).

My distributed/clustered solution would be ideal if it were fast enough 
for small block i/o as well as large block- I was hoping that single 
node systems would achieve that, hence the single node test.  Because 
the single node test performed poorly, I eventually reduced down to 
single disk to see if it could still be seen, and it clearly can be.  
Perhaps it's something in my configuration?  I've pasted my config files 
below.
thx-

     Jeremy

######################glusterfsd.vol######################
volume posix
   type storage/posix
   option directory /export
end-volume

volume locks
   type features/locks
   subvolumes posix
end-volume

volume disk
   type performance/io-threads
   option thread-count 4
   subvolumes locks
end-volume

volume server-ib
   type protocol/server
   option transport-type ib-verbs/server
   option auth.addr.disk.allow *
   subvolumes disk
end-volume

volume server-tcp
   type protocol/server
   option transport-type tcp/server
   option auth.addr.disk.allow *
   subvolumes disk
end-volume

######################ghome.vol######################

#-----------IB remotes------------------
volume ghome
   type protocol/client
   option transport-type ib-verbs/client
#  option transport-type tcp/client
   option remote-host acfs
   option remote-subvolume raid
end-volume

#------------Performance Options-------------------

volume readahead
   type performance/read-ahead
   option page-count 4           # 2 is default option
   option force-atime-update off # default is off
   subvolumes ghome
end-volume

volume writebehind
   type performance/write-behind
   option cache-size 1MB
   subvolumes readahead
end-volume

volume cache
   type performance/io-cache
   option cache-size 1GB
   subvolumes writebehind
end-volume

######################END######################

On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote:
> On Tue, 23 Mar 2010 02:59:35 -0600 (CST)
> "Tejas N. Bhise"<tejas at gluster.com>  wrote:
>
>    
>> Out of curiosity, if you want to do stuff only on one machine,
>> why do you want to use a distributed, multi node, clustered,
>> file system ?
>>      
> Because what he does is a very good way to show the overhead produced only by
> glusterfs and nothing else (i.e. no network involved).
> A pretty relevant test scenario I would say.
>
> --
> Regards,
> Stephan
>
>
>    
>> Am I missing something here ?
>>
>> Regards,
>> Tejas.
>>
>> ----- Original Message -----
>> From: "Jeremy Enos"<jenos at ncsa.uiuc.edu>
>> To: gluster-users at gluster.org
>> Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
>> Subject: [Gluster-users] gluster local vs local = gluster x4 slower
>>
>> This test is pretty easy to replicate anywhere- only takes 1 disk, one
>> machine, one tarball.  Untarring to local disk directly vs thru gluster
>> is about 4.5x faster.  At first I thought this may be due to a slow host
>> (Opteron 2.4ghz).  But it's not- same configuration, on a much faster
>> machine (dual 3.33ghz Xeon) yields the performance below.
>>
>> ####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
>> [root at ac33 jenos]# time tar xzf
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>
>> real    0m41.290s
>> user    0m14.246s
>> sys     0m2.957s
>>
>> ####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
>> [root at ac33 jenos]# cd /export/jenos/
>> [root at ac33 jenos]# time tar xzf
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>
>> real    0m8.983s
>> user    0m6.857s
>> sys     0m1.844s
>>
>> ####THESE ARE TEST FILE DETAILS####
>> [root at ac33 jenos]# tar tzvf
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz  |wc -l
>> 109
>> [root at ac33 jenos]# ls -l
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>> -rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>> [root at ac33 jenos]#
>>
>> These are the relevant performance options I'm using in my .vol file:
>>
>> #------------Performance Options-------------------
>>
>> volume readahead
>>     type performance/read-ahead
>>     option page-count 4           # 2 is default option
>>     option force-atime-update off # default is off
>>     subvolumes ghome
>> end-volume
>>
>> volume writebehind
>>     type performance/write-behind
>>     option cache-size 1MB
>>     subvolumes readahead
>> end-volume
>>
>> volume cache
>>     type performance/io-cache
>>     option cache-size 1GB
>>     subvolumes writebehind
>> end-volume
>>
>> What can I do to improve gluster's performance?
>>
>>       Jeremy
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>      
>
>    