[Gluster-users] Performance

[Mohit Anchlia]

Please find


[root at dsdb1 ~]# cat /proc/sys/vm/drop_caches
3
[root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=80k oflag=direct

81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 521.553 seconds, 20.6 MB/s
[root at dsdb1 ~]#
[root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=80k iflag=direct
dd: opening `/dev/zero': Invalid argument
[root at dsdb1 ~]# dd of=/dev/null if=/data/big.file bs=128k iflag=direct
81920+0 records in
81920+0 records out
10737418240 bytes (11 GB) copied, 37.854 seconds, 284 MB/s
[root at dsdb1 ~]#


On Wed, Apr 20, 2011 at 11:05 AM, Joe Landman
<landman at scalableinformatics.com> wrote:
> On 04/20/2011 02:01 PM, Mohit Anchlia wrote:
>>
>> Is 128K block size right size given that file sizes I have is from 70K -
>> 2MB?
>>
>> Please find
>>
>> [root at dsdb1 ~]# dd if=/dev/zero of=/data/big.file bs=128k count=80k
>> 81920+0 records in
>> 81920+0 records out
>> 10737418240 bytes (11 GB) copied, 14.751 seconds, 728 MB/s
>
>
>
>> [root at dsdb1 ~]# echo 3>  /proc/sys/vm/drop_caches
>> [root at dsdb1 ~]# dd of=/dev/null if=/data/big.file bs=128k
>> 81920+0 records in
>> 81920+0 records out
>> 10737418240 bytes (11 GB) copied, 3.10485 seconds, 3.5 GB/s
>
> Hmm ... this looks like it came from cache.  4 drive RAID0's aren't even
> remotely this fast.
>
> Add an oflag=direct to the first dd, and an iflag=direct to the second dd so
> we can avoid the OS memory cache for the moment (looks like the driver isn't
> respecting the drop caches command, or you had no space between the 3 and
> the > sign).
>
>
>
>>
>> On Wed, Apr 20, 2011 at 10:49 AM, Joe Landman
>> <landman at scalableinformatics.com>  wrote:
>>>
>>> On 04/20/2011 01:42 PM, Mohit Anchlia wrote:
>>>>
>>>>  mount
>>>> /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
>>>> proc on /proc type proc (rw)
>>>> sysfs on /sys type sysfs (rw)
>>>> devpts on /dev/pts type devpts (rw,gid=5,mode=620)
>>>> /dev/sdb1 on /boot type ext3 (rw)
>>>> tmpfs on /dev/shm type tmpfs (rw)
>>>> none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
>>>> sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
>>>> /dev/sda1 on /data type ext3 (rw)
>>>> glusterfs#dsdb1:/stress-volume on /data/mnt-stress type fuse
>>>> (rw,allow_other,default_permissions,max_read=131072)
>>>
>>> ok ...
>>>
>>> so gluster is running atop /data/mnt-stress, which is on /dev/sda1 and is
>>> ext3.
>>>
>>> Could you do this
>>>
>>>        dd if=/dev/zero of=/data/big.file bs=128k count=80k
>>>        echo 3>  /proc/sys/vm/drop_caches
>>>        dd of=/dev/null if=/data/big.file bs=128k
>>>
>>> so we can see the write and then read performance using 128k blocks?
>>>
>>> Also, since you are using the gluster native client, you don't get all
>>> the
>>> nice NFS caching bits.  Gluster native client is somewhat slower than the
>>> NFS client.
>>>
>>> So lets start with the write/read speed of the system before we deal with
>>> the gluster side of things.
>>>
>>>>
>>>>
>>>> On Wed, Apr 20, 2011 at 10:39 AM, Joe Landman
>>>> <landman at scalableinformatics.com>    wrote:
>>>>>
>>>>> On 04/20/2011 01:35 PM, Mohit Anchlia wrote:
>>>>>>
>>>>>> Should that command be there by default? I couldn't find lsscsi
>>>>>
>>>>> How about
>>>>>
>>>>>        mount
>>>>>
>>>>> output?
>>>>>
>>>>>
>>>>> --
>>>>> Joseph Landman, Ph.D
>>>>> Founder and CEO
>>>>> Scalable Informatics Inc.
>>>>> email: landman at scalableinformatics.com
>>>>> web  : http://scalableinformatics.com
>>>>>       http://scalableinformatics.com/sicluster
>>>>> phone: +1 734 786 8423 x121
>>>>> fax  : +1 866 888 3112
>>>>> cell : +1 734 612 4615
>>>>>
>>>
>>>
>>> --
>>> Joseph Landman, Ph.D
>>> Founder and CEO
>>> Scalable Informatics Inc.
>>> email: landman at scalableinformatics.com
>>> web  : http://scalableinformatics.com
>>>       http://scalableinformatics.com/sicluster
>>> phone: +1 734 786 8423 x121
>>> fax  : +1 866 888 3112
>>> cell : +1 734 612 4615
>>>
>
>
> --
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics Inc.
> email: landman at scalableinformatics.com
> web  : http://scalableinformatics.com
>       http://scalableinformatics.com/sicluster
> phone: +1 734 786 8423 x121
> fax  : +1 866 888 3112
> cell : +1 734 612 4615
>