[Gluster-users] [ovirt-users] ovirt glusterfs performance

Bill James bill.james at j2.com
Thu Feb 11 23:13:13 UTC 2016

xml attached.

On 02/11/2016 12:28 PM, Nir Soffer wrote:
> On Thu, Feb 11, 2016 at 8:27 PM, Bill James <bill.james at j2.com> wrote:
>> thank you for the reply.
>> We setup gluster using the names associated with  NIC 2 IP.
>>   Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>   Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>   Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>> That's NIC 2's IP.
>> Using 'iftop -i eno2 -L 5 -t' :
>> dd if=/dev/zero of=/root/testfile bs=1M count=1000 oflag=direct
>> 1048576000 bytes (1.0 GB) copied, 68.0714 s, 15.4 MB/s
> Can you share the xml of this vm? You can find it in vdsm log,
> at the time you start the vm.
> Or you can do (on the host):
> # virsh
> virsh # list
> (username: vdsm at ovirt password: shibboleth)
> virsh # dumpxml vm-id
>> Peak rate (sent/received/total):                      281Mb 5.36Mb
>> 282Mb
>> Cumulative (sent/received/total):                    1.96GB 14.6MB
>> 1.97GB
>> gluster volume info gv1:
>>   Options Reconfigured:
>>   performance.write-behind-window-size: 4MB
>>   performance.readdir-ahead: on
>>   performance.cache-size: 1GB
>>   performance.write-behind: off
>> performance.write-behind: off didn't help.
>> Neither did any other changes I've tried.
>> There is no VM traffic on this VM right now except my test.
>> On 02/10/2016 11:55 PM, Nir Soffer wrote:
>>> On Thu, Feb 11, 2016 at 2:42 AM, Ravishankar N <ravishankar at redhat.com>
>>> wrote:
>>>> +gluster-users
>>>> Does disabling 'performance.write-behind' give a better throughput?
>>>> On 02/10/2016 11:06 PM, Bill James wrote:
>>>>> I'm setting up a ovirt cluster using glusterfs and noticing not stellar
>>>>> performance.
>>>>> Maybe my setup could use some adjustments?
>>>>> 3 hardware nodes running centos7.2, glusterfs, ovirt
>>>>> Each node has 8 spindles configured in 1 array which is split using LVM
>>>>> with one logical volume for system and one for gluster.
>>>>> They each have 4 NICs,
>>>>>    NIC1 = ovirtmgmt
>>>>>    NIC2 = gluster  (1GbE)
>>> How do you ensure that gluster trafic is using this nic?
>>>>>    NIC3 = VM traffic
>>> How do you ensure that vm trafic is using this nic?
>>>>> I tried with default glusterfs settings
>>> And did you find any difference?
>>>>> and also with:
>>>>> performance.cache-size: 1GB
>>>>> performance.readdir-ahead: on
>>>>> performance.write-behind-window-size: 4MB
>>>>> [root at ovirt3 test scripts]# gluster volume info gv1
>>>>> Volume Name: gv1
>>>>> Type: Replicate
>>>>> Volume ID: 71afc35b-09d7-4384-ab22-57d032a0f1a2
>>>>> Status: Started
>>>>> Number of Bricks: 1 x 3 = 3
>>>>> Transport-type: tcp
>>>>> Bricks:
>>>>> Brick1: ovirt1-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Brick2: ovirt2-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Brick3: ovirt3-ks.test.j2noc.com:/gluster-store/brick1/gv1
>>>>> Options Reconfigured:
>>>>> performance.cache-size: 1GB
>>>>> performance.readdir-ahead: on
>>>>> performance.write-behind-window-size: 4MB
>>>>> Using simple dd test on VM in ovirt:
>>>>>     dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>> block size of 1G?!
>>> Try 1M (our default for storage operations)
>>>>>         1073741824 bytes (1.1 GB) copied, 65.9337 s, 16.3 MB/s
>>>>> Another VM not in ovirt using nfs:
>>>>>      dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct
>>>>>         1073741824 bytes (1.1 GB) copied, 27.0079 s, 39.8 MB/s
>>>>> Is that expected or is there a better way to set it up to get better
>>>>> performance?
>>> Adding Niels for advice.
>>>>> This email, its contents and ....
>>> Please avoid this, this is a public mailing list, everything you write
>>> here is public.
>>> Nir
>> I'll have to look into how to remove this sig for this mailing list....
>> Cloud Services for Business www.j2.com
>> j2 | eFax | eVoice | FuseMail | Campaigner | KeepItSafe | Onebox
>> This email, its contents and attachments contain information from j2 Global,
>> Inc. and/or its affiliates which may be privileged, confidential or
>> otherwise protected from disclosure. The information is intended to be for
>> the addressee(s) only. If you are not an addressee, any disclosure, copy,
>> distribution, or use of the contents of this message is prohibited. If you
>> have received this email in error please notify the sender by reply e-mail
>> and delete the original message and any copies. (c) 2015 j2 Global, Inc. All
>> rights reserved. eFax, eVoice, Campaigner, FuseMail, KeepItSafe, and Onebox
>> are registered trademarks of j2 Global, Inc. and its affiliates.

-------------- next part --------------
Please enter your authentication name: Please enter your password: <domain type='kvm' id='3'>
  <metadata xmlns:ovirt="http://ovirt.org/vm/tune/1.0">
  <maxMemory slots='16' unit='KiB'>4294967296</maxMemory>
  <memory unit='KiB'>2097152</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <vcpu placement='static' current='1'>16</vcpu>
    <memory mode='interleave' nodeset='0-1'/>
  <sysinfo type='smbios'>
      <entry name='manufacturer'>oVirt</entry>
      <entry name='product'>oVirt Node</entry>
      <entry name='version'>7-2.1511.el7.centos.2.10</entry>
      <entry name='serial'>30343536-3138-5355-4533-323134593738</entry>
      <entry name='uuid'>c6aa56b4-f387-4a5b-84b6-a7db6ef89686</entry>
    <type arch='x86_64' machine='pc-i440fx-rhel7.2.0'>hvm</type>
    <smbios mode='sysinfo'/>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <topology sockets='16' cores='1' threads='1'/>
      <cell id='0' cpus='0' memory='2097152' unit='KiB'/>
  <clock offset='variable' adjustment='0' basis='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source startupPolicy='optional'/>
      <target dev='hdc' bus='ide'/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    <disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='raw' cache='none' error_policy='stop' io='threads'/>
      <source file='/rhev/data-center/00000001-0001-0001-0001-00000000009d/f11b0914-f067-4ee9-85e2-c9009be9ede5/images/eb0ccbf9-1ad8-4af8-944f-bc0d06981ed0/2c20816c-2559-46ad-acc3-300638126d6e'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    <controller type='virtio-serial' index='0' ports='16'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    <controller type='usb' index='0'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    <interface type='bridge'>
      <mac address='00:1a:4a:16:01:51'/>
      <source bridge='QAVlan110'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <filterref filter='vdsm-no-mac-spoofing'/>
      <link state='up'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channels/c6aa56b4-f387-4a5b-84b6-a7db6ef89686.com.redhat.rhevm.vdsm'/>
      <target type='virtio' name='com.redhat.rhevm.vdsm' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channels/c6aa56b4-f387-4a5b-84b6-a7db6ef89686.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel1'/>
      <address type='virtio-serial' controller='0' bus='0' port='2'/>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='0' passwdValidTo='2016-02-11T22:59:07'>
      <listen type='address' address='0'/>
      <model type='cirrus' vram='32768' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    <memballoon model='none'>
      <alias name='balloon0'/>

More information about the Gluster-users mailing list