[Gluster-users] gluster local vs local = gluster x4 slower
Jeremy Enos
jenos at ncsa.uiuc.edu
Wed Mar 24 09:25:45 UTC 2010
I also neglected to mention that the underlying filesystem is ext3.
On 3/24/2010 3:44 AM, Jeremy Enos wrote:
> I haven't tried all performance options disabled yet- I can try that
> tomorrow when the resource frees up. I was actually asking first
> before blindly trying different configuration matrices in case there's
> a clear direction I should take with it. I'll let you know.
>
> Jeremy
>
> On 3/24/2010 2:54 AM, Stephan von Krawczynski wrote:
>> Hi Jeremy,
>>
>> have you tried to reproduce with all performance options disabled?
>> They are
>> possibly no good idea on a local system.
>> What local fs do you use?
>>
>>
>> --
>> Regards,
>> Stephan
>>
>>
>> On Tue, 23 Mar 2010 19:11:28 -0500
>> Jeremy Enos<jenos at ncsa.uiuc.edu> wrote:
>>
>>> Stephan is correct- I primarily did this test to show a demonstrable
>>> overhead example that I'm trying to eliminate. It's pronounced enough
>>> that it can be seen on a single disk / single node configuration, which
>>> is good in a way (so anyone can easily repro).
>>>
>>> My distributed/clustered solution would be ideal if it were fast enough
>>> for small block i/o as well as large block- I was hoping that single
>>> node systems would achieve that, hence the single node test. Because
>>> the single node test performed poorly, I eventually reduced down to
>>> single disk to see if it could still be seen, and it clearly can be.
>>> Perhaps it's something in my configuration? I've pasted my config
>>> files
>>> below.
>>> thx-
>>>
>>> Jeremy
>>>
>>> ######################glusterfsd.vol######################
>>> volume posix
>>> type storage/posix
>>> option directory /export
>>> end-volume
>>>
>>> volume locks
>>> type features/locks
>>> subvolumes posix
>>> end-volume
>>>
>>> volume disk
>>> type performance/io-threads
>>> option thread-count 4
>>> subvolumes locks
>>> end-volume
>>>
>>> volume server-ib
>>> type protocol/server
>>> option transport-type ib-verbs/server
>>> option auth.addr.disk.allow *
>>> subvolumes disk
>>> end-volume
>>>
>>> volume server-tcp
>>> type protocol/server
>>> option transport-type tcp/server
>>> option auth.addr.disk.allow *
>>> subvolumes disk
>>> end-volume
>>>
>>> ######################ghome.vol######################
>>>
>>> #-----------IB remotes------------------
>>> volume ghome
>>> type protocol/client
>>> option transport-type ib-verbs/client
>>> # option transport-type tcp/client
>>> option remote-host acfs
>>> option remote-subvolume raid
>>> end-volume
>>>
>>> #------------Performance Options-------------------
>>>
>>> volume readahead
>>> type performance/read-ahead
>>> option page-count 4 # 2 is default option
>>> option force-atime-update off # default is off
>>> subvolumes ghome
>>> end-volume
>>>
>>> volume writebehind
>>> type performance/write-behind
>>> option cache-size 1MB
>>> subvolumes readahead
>>> end-volume
>>>
>>> volume cache
>>> type performance/io-cache
>>> option cache-size 1GB
>>> subvolumes writebehind
>>> end-volume
>>>
>>> ######################END######################
>>>
>>>
>>>
>>> On 3/23/2010 6:02 AM, Stephan von Krawczynski wrote:
>>>> On Tue, 23 Mar 2010 02:59:35 -0600 (CST)
>>>> "Tejas N. Bhise"<tejas at gluster.com> wrote:
>>>>
>>>>
>>>>> Out of curiosity, if you want to do stuff only on one machine,
>>>>> why do you want to use a distributed, multi node, clustered,
>>>>> file system ?
>>>>>
>>>> Because what he does is a very good way to show the overhead
>>>> produced only by
>>>> glusterfs and nothing else (i.e. no network involved).
>>>> A pretty relevant test scenario I would say.
>>>>
>>>> --
>>>> Regards,
>>>> Stephan
>>>>
>>>>
>>>>
>>>>> Am I missing something here ?
>>>>>
>>>>> Regards,
>>>>> Tejas.
>>>>>
>>>>> ----- Original Message -----
>>>>> From: "Jeremy Enos"<jenos at ncsa.uiuc.edu>
>>>>> To: gluster-users at gluster.org
>>>>> Sent: Tuesday, March 23, 2010 2:07:06 PM GMT +05:30 Chennai,
>>>>> Kolkata, Mumbai, New Delhi
>>>>> Subject: [Gluster-users] gluster local vs local = gluster x4 slower
>>>>>
>>>>> This test is pretty easy to replicate anywhere- only takes 1 disk,
>>>>> one
>>>>> machine, one tarball. Untarring to local disk directly vs thru
>>>>> gluster
>>>>> is about 4.5x faster. At first I thought this may be due to a
>>>>> slow host
>>>>> (Opteron 2.4ghz). But it's not- same configuration, on a much faster
>>>>> machine (dual 3.33ghz Xeon) yields the performance below.
>>>>>
>>>>> ####THIS TEST WAS TO A LOCAL DISK THRU GLUSTER####
>>>>> [root at ac33 jenos]# time tar xzf
>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>
>>>>> real 0m41.290s
>>>>> user 0m14.246s
>>>>> sys 0m2.957s
>>>>>
>>>>> ####THIS TEST WAS TO A LOCAL DISK (BYPASS GLUSTER)####
>>>>> [root at ac33 jenos]# cd /export/jenos/
>>>>> [root at ac33 jenos]# time tar xzf
>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>>
>>>>> real 0m8.983s
>>>>> user 0m6.857s
>>>>> sys 0m1.844s
>>>>>
>>>>> ####THESE ARE TEST FILE DETAILS####
>>>>> [root at ac33 jenos]# tar tzvf
>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz |wc -l
>>>>> 109
>>>>> [root at ac33 jenos]# ls -l
>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>> -rw-r--r-- 1 jenos ac 804385203 2010-02-07 06:32
>>>>> /scratch/jenos/intel/l_cproc_p_11.1.064_intel64.tgz
>>>>> [root at ac33 jenos]#
>>>>>
>>>>> These are the relevant performance options I'm using in my .vol file:
>>>>>
>>>>> #------------Performance Options-------------------
>>>>>
>>>>> volume readahead
>>>>> type performance/read-ahead
>>>>> option page-count 4 # 2 is default option
>>>>> option force-atime-update off # default is off
>>>>> subvolumes ghome
>>>>> end-volume
>>>>>
>>>>> volume writebehind
>>>>> type performance/write-behind
>>>>> option cache-size 1MB
>>>>> subvolumes readahead
>>>>> end-volume
>>>>>
>>>>> volume cache
>>>>> type performance/io-cache
>>>>> option cache-size 1GB
>>>>> subvolumes writebehind
>>>>> end-volume
>>>>>
>>>>> What can I do to improve gluster's performance?
>>>>>
>>>>> Jeremy
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>>>>>
>>>>>
>>>>
>>
More information about the Gluster-users
mailing list