[Gluster-users] gluster local vs local = gluster x4 slower
Jeremy Enos
jenos at ncsa.uiuc.edu
Wed Mar 24 21:47:52 UTC 2010
Good suggestion- I hadn't tried that yet. It brings them much closer.
##############GLUSTER SINGLE DISK##############
[root at ac33 gjenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )
real 0m32.089s
user 0m6.516s
sys 0m3.177s
##############DIRECT SINGLE DISK##############
[root at ac33 gjenos]# cd /export/jenos/
[root at ac33 jenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz && sync )
real 0m25.089s
user 0m6.850s
sys 0m2.058s
##############DIRECT SINGLE DISK CACHED##############
[root at ac33 jenos]# time (tar xzf
/scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz )
real 0m8.955s
user 0m6.785s
sys 0m1.848s
Oddly, I'm seeing better performance on the gluster system than previous
tests too (used to be ~39 s). The direct disk time is obviously
benefiting from cache. There is still a difference, but it appears most
of the difference disappears w/ the cache advantage removed. That said-
the relative performance issue then still exists with Gluster. What can
be done to make it benefit from cache the same way direct disk does?
thx-
Jeremy
P.S.
I'll be posting results w/ performance options completely removed from
gluster as soon as I get a chance.
Jeremy
On 3/24/2010 4:23 PM, Bryan Whitehead wrote:
> I'd like to see results with this:
>
> time ( tar xzf /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz&& sync )
>
> I've found local filesystems seem to use cache very heavily. The
> untarred file could mostly be sitting in ram with local fs vs going
> though fuse (which might do many more sync'ed flushes to disk?).
>
> On Wed, Mar 24, 2010 at 2:25 AM, Jeremy Enos<jenos at ncsa.uiuc.edu> wrote:
>
>> I also neglected to mention that the underlying filesystem is ext3.
>>
>> On 3/24/2010 3:44 AM, Jeremy Enos wrote:
>>
>>> I haven't tried all performance options disabled yet- I can try that
>>> tomorrow when the resource frees up. I was actually asking first before
>>> blindly trying different configuration matrices in case there's a clear
>>> direction I should take with it. I'll let you know.
>>>
>>> Jeremy
>>>
>>> On 3/24/2010 2:54 AM, Stephan von Krawczynski wrote:
>>>
>>>> Hi Jeremy,
>>>>
>>>> have you tried to reproduce with all performance options disabled? They
>>>> are
>>>> possibly no good idea on a local system.
>>>> What local fs do you use?
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Stephan
>>>>
>>>>
>>>> On Tue, 23 Mar 2010 19:11:28 -0500
>>>> Jeremy Enos<jenos at ncsa.uiuc.edu> wrote:
>>>>
>>>>
>>>>> Stephan is correct- I primarily did this test to show a demonstrable
>>>>> overhead example that I'm trying to eliminate. It's pronounced enough
>>>>> that it can be seen on a single disk / single node configuration, which
>>>>> is good in a way (so anyone can easily repro).
>>>>>
>>>>> My distributed/clustered solution would be ideal if it were fast enough
>>>>> for small block i/o as well as large block- I was hoping that single
>>>>> node systems would achieve that, hence the single node test. Because
>>>>> the single node test performed poorly, I eventually reduced down to
>>>>> single disk to see if it could still be seen, and it clearly can be.
>>>>> Perhaps it's something in my configuration? I've pasted my config files
>>>>> below.
>>>>> thx-
>>>>>
>>>>> Jeremy
>>>>>
>>>>> ######################glusterfsd.vol######################
>>>>> volume posix
>>>>> type storage/posix
>>>>> option directory /export
>>>>> end-volume
>>>>>
>>>>> volume locks
>>>>> type features/locks
>>>>> subvolumes posix
>>>>> end-volume
>>>>>
>>>>> volume disk
>>>>> type performance/io-threads
>>>>> option thread-count 4
>>>>> subvolumes locks
>>>>> end-volume
>>>>>
>>>>> volume server-ib
>>>>> type protocol/server
>>>>> option transport-type ib-verbs/server
>>>>> option auth.addr.disk.allow *
>>>>> subvolumes disk
>>>>> end-volume
>>>>>
>>>>> volume server-tcp
>>>>> type protocol/server
>>>>> option transport-type tcp/server
>>>>> option auth.addr.disk.allow *
>>>>> subvolumes disk
>>>>> end-volume
>>>>>
>>>>> ######################ghome.vol######################
>>>>>
>>>>> #-----------IB remotes------------------
>>>>> volume ghome
>>>>> type protocol/client
>>>>> option transport-type ib-verbs/client
>>>>> # option transport-type tcp/client
>>>>> option remote-host acfs
>>>>> option remote-subvolume raid
>>>>> end-volume
>>>>>
>>>>> #------------Performance Options-------------------
>>>>>
>>>>> volume readahead
>>>>> type performance/read-ahead
>>>>> option page-count 4 # 2 is default option
>>>>> option force-atime-update off # default is off
>>>>> subvolumes ghome
>>>>> end-volume
>>>>>
>>>>> volume writebehind
>>>>> type performance/write-behind
>>>>> option cache-size 1MB
>>>>> subvolumes readahead
>>>>> end-volume
>>>>>
>>>>> volume cache
>>>>> type performance/io-cache
>>>>> option cache-size 1GB
>>>>> subvolumes writebehind
>>>>> end-volume
>>>>>
>>>>> ######################END######################
>>>>>
>>>>>
>>>>>
>>>>> On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote:
>>>>>
>>>>>> On Tue, 23 Mar 2010 02:59:35 -0600 (CST)
>>>>>> "Tejas N. Bhise"<tejas at gluster.com> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Out of curiosity, if you want to do stuff only on one machine,
>>>>>>> why do you want to use a distributed, multi node, clustered,
>>>>>>> file system ?
>>>>>>>
>>>>>>>
>>>>>> Because what he does is a very good way to show the overhead produced
>>>>>> only by
>>>>>> glusterfs and nothing else (i.e. no network involved).
>>>>>> A pretty relevant test scenario I would say.
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Stephan
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>> Am I missing something here ?
>>>>>>>
>>>>>>> Regards,
>>>>>>> Tejas.
>>>>>>>
>>>>>>> ----- Original Message -----
>>>>>>> From: "Jeremy Enos"<jenos at ncsa.uiuc.edu>
>>>>>>> To: gluster-users at gluster.org
>>>>>>> Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai, Kolkata,
>>>>>>> Mumbai, New Delhi
>>>>>>> Subject: [Gluster-users] gluster local vs local = gluster x4 slower
>>>>>>>
>>>>>>> This test is pretty easy to replicate anywhere- only takes 1 disk, one
>>>>>>> machine, one tarball. Untarring to local disk directly vs thru
>>>>>>> gluster
>>>>>>> is about 4.5x faster. At first I thought this may be due to a slow
>>>>>>> host
>>>>>>> (Opteron 2.4ghz). But it's not- same configuration, on a much faster
>>>>>>> machine (dual 3.33ghz Xeon) yields the performance below.
>>>>>>>
>>>>>>> ####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
>>>>>>> [root at ac33 jenos]# time tar xzf
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>>
>>>>>>> real 0m41.290s
>>>>>>> user 0m14.246s
>>>>>>> sys 0m2.957s
>>>>>>>
>>>>>>> ####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
>>>>>>> [root at ac33 jenos]# cd /export/jenos/
>>>>>>> [root at ac33 jenos]# time tar xzf
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>>
>>>>>>> real 0m8.983s
>>>>>>> user 0m6.857s
>>>>>>> sys 0m1.844s
>>>>>>>
>>>>>>> ####THESE ARE TEST FILE DETAILS####
>>>>>>> [root at ac33 jenos]# tar tzvf
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l
>>>>>>> 109
>>>>>>> [root at ac33 jenos]# ls -l
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>> -rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
>>>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>>> [root at ac33 jenos]#
>>>>>>>
>>>>>>> These are the relevant performance options I'm using in my .vol file:
>>>>>>>
>>>>>>> #------------Performance Options-------------------
>>>>>>>
>>>>>>> volume readahead
>>>>>>> type performance/read-ahead
>>>>>>> option page-count 4 # 2 is default option
>>>>>>> option force-atime-update off # default is off
>>>>>>> subvolumes ghome
>>>>>>> end-volume
>>>>>>>
>>>>>>> volume writebehind
>>>>>>> type performance/write-behind
>>>>>>> option cache-size 1MB
>>>>>>> subvolumes readahead
>>>>>>> end-volume
>>>>>>>
>>>>>>> volume cache
>>>>>>> type performance/io-cache
>>>>>>> option cache-size 1GB
>>>>>>> subvolumes writebehind
>>>>>>> end-volume
>>>>>>>
>>>>>>> What can I do to improve gluster's performance?
>>>>>>>
>>>>>>> Jeremy
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>
>>
>
More information about the Gluster-users
mailing list