<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 23, 2017 at 4:56 PM, Jiří Sléžka <span dir="ltr"><<a href="mailto:jiri.slezka@slu.cz" target="_blank">jiri.slezka@slu.cz</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<span class=""><br>
On 11/22/2017 07:30 PM, Nir Soffer wrote:<br>
> On Mon, Nov 20, 2017 at 5:22 PM Jiří Sléžka <<a href="mailto:jiri.slezka@slu.cz">jiri.slezka@slu.cz</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:jiri.slezka@slu.cz">jiri.slezka@slu.cz</a>>> wrote:<br>
><br>
> Hi,<br>
><br>
> I am trying realize why is exporting of vm to export storage on<br>
> glusterfs such slow.<br>
><br>
> I am using oVirt and RHV, both instalations on version 4.1.7.<br>
><br>
> Hosts have dedicated nics for rhevm network - 1gbps, data storage itself<br>
> is on FC.<br>
><br>
> GlusterFS cluster lives separate on 4 dedicated hosts. It has slow disks<br>
> but I can achieve about 200-400mbit throughput in other applications (we<br>
> are using it for "cold" data, backups mostly).<br>
><br>
> I am using this glusterfs cluster as backend for export storage. When I<br>
> am exporting vm I can see only about 60-80mbit throughput.<br>
><br>
> What could be the bottleneck here?<br>
><br>
> Could it be qemu-img utility?<br>
><br>
> vdsm 97739 0.3 0.0 354212 29148 ? S<l 15:43 0:06<br>
> /usr/bin/qemu-img convert -p -t none -T none -f raw<br>
> /rhev/data-center/2ff6d0ee-<wbr>a10b-473d-b77c-be9149945f5f/<wbr>ff3cd56a-1005-4426-8137-<wbr>8f422c0b47c1/images/ba42cbcc-<wbr>c068-4df8-af3d-00f2077b1e27/<wbr>c57acd5f-d6cf-48cc-ad0c-<wbr>4a7d979c0c1e<br>
> -O raw<br>
> /rhev/data-center/mnt/<wbr>glusterSD/10.20.30.41:_rhv__<wbr>export/81094499-a392-4ea2-<wbr>b081-7c6288fbb636/images/<wbr>ba42cbcc-c068-4df8-af3d-<wbr>00f2077b1e27/c57acd5f-d6cf-<wbr>48cc-ad0c-4a7d979c0c1e<br>
><br>
> Any idea how to make it work faster or what throughput should I<br>
> expected?<br>
><br>
><br>
> gluster storage operations are using fuse mount - so every write:<br>
> - travel to the kernel<br>
> - travel back to the gluster fuse helper process<br>
> - travel to all 3 replicas - replication is done on client side<br>
> - return to kernel when all writes succeeded<br>
> - return to caller<br>
><br>
> So gluster will never set any speed record.<br>
><br>
> Additionally, you are copying from raw lv on FC - qemu-img cannot do<br>
> anything<br>
> smart and avoid copying unused clusters. Instead if copies gigabytes of<br>
> zeros<br>
> from FC.<br>
<br>
</div></div>ok, it does make sense<br>
<span class=""><br>
> However 7.5-10 MiB/s sounds too slow.<br>
><br>
> I would try to test with dd - how much time it takes to copy<br>
> the same image from FC to your gluster storage?<br>
><br>
> dd<br>
> if=/rhev/data-center/2ff6d0ee-<wbr>a10b-473d-b77c-be9149945f5f/<wbr>ff3cd56a-1005-4426-8137-<wbr>8f422c0b47c1/images/ba42cbcc-<wbr>c068-4df8-af3d-00f2077b1e27/<wbr>c57acd5f-d6cf-48cc-ad0c-<wbr>4a7d979c0c1e<br>
> of=/rhev/data-center/mnt/<wbr>glusterSD/10.20.30.41:_rhv__<wbr>export/81094499-a392-4ea2-<wbr>b081-7c6288fbb636/__test__<br>
> bs=8M oflag=direct status=progress<br>
<br>
</span>unfrotunately dd performs the same<br>
<br>
1778384896 bytes (1.8 GB) copied, 198.565265 s, 9.0 MB/s<br>
<span class=""><br>
<br>
> If dd can do this faster, please ask on qemu-discuss mailing list:<br>
> <a href="https://lists.nongnu.org/mailman/listinfo/qemu-discuss" rel="noreferrer" target="_blank">https://lists.nongnu.org/<wbr>mailman/listinfo/qemu-discuss</a><br>
><br>
> If both give similar results, I think asking in gluster mailing list<br>
> about this can help. Maybe your gluster setup can be optimized.<br>
<br>
</span>ok, this is definitly on the gluster side. Thanks for your guidance.<br>
<br>
I will investigate the gluster side and also will try Export on NFS share.<br></blockquote><div><br></div><div>[Adding gluster users ml]<br></div><div><br></div><div>Please provide "gluster volume info" output for the rhv_export gluster volume and also volume profile details (refer to earlier mail from Shani on how to run this) while performing the dd operation above.<br><br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Cheers,<br>
<br>
Jiri<br>
<span class=""><br>
<br>
><br>
> Nir<br>
> <br>
><br>
><br>
> Cheers,<br>
><br>
> Jiri<br>
><br>
><br>
> ______________________________<wbr>_________________<br>
> Users mailing list<br>
</span>> <a href="mailto:Users@ovirt.org">Users@ovirt.org</a> <mailto:<a href="mailto:Users@ovirt.org">Users@ovirt.org</a>><br>
> <a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
><br>
<br>
<br>
<br>______________________________<wbr>_________________<br>
Users mailing list<br>
<a href="mailto:Users@ovirt.org">Users@ovirt.org</a><br>
<a href="http://lists.ovirt.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.ovirt.org/<wbr>mailman/listinfo/users</a><br>
<br></blockquote></div><br></div></div>