<div dir="ltr">Dear Krutika,<div><br></div><div>1. </div><div>I've changed the volume settings, write performance seems to increased somewhat, however the profile doesn't really support that since latencies increased. However read performance has diminished, which does seem to be supported by the profile runs (attached).</div><div>Also the IO does seem to behave more consistent than before.</div><div>I don't really understand the idea behind them, maybe you can explain why these suggestions are good?</div><div>These settings seems to avoid as much local caching and access as possible and push everything to the gluster processes. While i would expect local access and local caches are a good thing, since it would lead to having less network access or disk access.</div><div>I tried to investigate these settings a bit more, and this is what i understood of them;</div><div>- network.remote-dio; when on it seems to ignore the O_DIRECT flag in the client, thus causing the files to be cached and buffered in the page cache on the client, i would expect this to be a good thing especially if the server process would access the same page cache? <br></div><div>At least that is what grasp from this commit; <a href="https://review.gluster.org/#/c/glusterfs/+/4206/2/xlators/protocol/client/src/client.c" target="_blank">https://review.gluster.org/#/c/glusterfs/+/4206/2/xlators/protocol/client/src/client.c</a> line 867</div><div>Also found this commit; <a href="https://github.com/gluster/glusterfs/commit/06c4ba589102bf92c58cd9fba5c60064bc7a504e#diff-938709e499b4383c3ed33c3979b9080c" target="_blank">https://github.com/gluster/glusterfs/commit/06c4ba589102bf92c58cd9fba5c60064bc7a504e#diff-938709e499b4383c3ed33c3979b9080c</a> suggesting remote-dio actually improves performance, not sure it's a write or read benchmark </div><div>When a file is opened with O_DIRECT it will also disable the write-behind functionality</div><div><br></div><div>- performance.strict-o-direct: when on, the AFR, will not ignore the O_DIRECT flag. and will invoke: fop_writev_stub with the wb_writev_helper, which seems to stack the operation, no idea why that is. But generally i suppose not ignoring the O_DIRECT flag in the AFR is a good thing, when a processes requests to have O_DIRECT. So this makes sense to me.</div><div><br></div><div>- cluster.choose-local: when off, it doesn't prefer the local node, but would always choose a brick. Since it's a 9 node cluster, with 3 subvolumes, only a 1/3 could end-up local, and the other 2/3 should be pushed to external nodes anyway. Or am I making the total wrong assumption here?</div><div><br></div><div>It seems to this config is moving to the gluster-block config side of things, which does make sense.</div><div>Since we're running quite some mysql instances, which opens the files with O_DIRECt i believe, it would mean the only layer of cache is within mysql it self. Which you could argue is a good thing. But i would expect a little of write-behind buffer, and maybe some of the data cached within gluster would alleviate things a bit on gluster's side. But i wouldn't know if that's the correct mind set, and so might be totally off here.</div><div>Also i would expect these gluster v set <VOL> command to be online operations, but somehow the bricks went down, after applying these changes. What appears to have happened is that after the update the brick process was restarted, but due to multiple brick process start issue, multiple processes were started, and the brick didn't came online again.</div><div>However i'll try to reproduce this, since i would like to test with cluster.choose-local: on, and see how performance compares. And hopefully when it occurs collect some useful info.</div><div>Question; are network.remote-dio and performance.strict-o-direct mutually exclusive settings, or can they both be on?</div><div><br></div><div>2. I've attached all brick logs, the only thing relevant i found was;</div><div><div>[2019-03-28 20:20:07.170452] I [MSGID: 113030] [posix-entry-ops.c:1146:posix_unlink] 0-ovirt-kube-posix: open-fd-key-status: 0 for /data/gfs/bricks/brick1/ovirt-kube/.shard/a38d64bc-a28b-4ee1-a0bb-f919e7a1022c.109886</div><div>[2019-03-28 20:20:07.170491] I [MSGID: 113031] [posix-entry-ops.c:1053:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /data/gfs/bricks/brick1/ovirt-kube/.shard/a38d64bc-a28b-4ee1-a0bb-f919e7a1022c.109886</div><div>[2019-03-28 20:20:07.248480] I [MSGID: 113030] [posix-entry-ops.c:1146:posix_unlink] 0-ovirt-kube-posix: open-fd-key-status: 0 for /data/gfs/bricks/brick1/ovirt-kube/.shard/a38d64bc-a28b-4ee1-a0bb-f919e7a1022c.109886</div><div>[2019-03-28 20:20:07.248491] I [MSGID: 113031] [posix-entry-ops.c:1053:posix_skip_non_linkto_unlink] 0-posix: linkto_xattr status: 0 for /data/gfs/bricks/brick1/ovirt-kube/.shard/a38d64bc-a28b-4ee1-a0bb-f919e7a1022c.109886</div></div><div><br></div><div>Thanks Olaf</div><div><br></div><div>ps. sorry needed to resend since it exceed the file limit</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Op ma 1 apr. 2019 om 07:56 schreef Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com">kdhananj@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div>Adding back gluster-users </div>Comments inline ...<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Mar 29, 2019 at 8:11 PM Olaf Buitelaar <<a href="mailto:olaf.buitelaar@gmail.com" target="_blank">olaf.buitelaar@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Dear Krutika,</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1. I’ve made 2 profile runs of around 10 minutes (see files
profile_data.txt and profile_data2.txt). Looking at it, most time seems be spent at the fop’s fsync and readdirp.</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Unfortunate I don’t have the profile info for the 3.12.15
version so it’s a bit hard to compare. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">One additional thing I do notice on 1 machine (10.32.9.5)
the iowait time increased a lot, from an average below the 1% it’s now around
the 12% after the upgrade. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">So first suspicion with be lighting strikes twice, and I’ve
also just now a bad disk, but that doesn’t appear to be the case, since all
smart status report ok. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Also dd shows performance I would more or less expect;</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">dd if=/dev/zero of=/data/test_file bs=100M count=1 oflag=dsync</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1+0 records in</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1+0 records out</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">104857600 bytes (105 MB) copied, 0.686088 s, 153 MB/s</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">dd if=/dev/zero of=/data/test_file bs=1G count=1
oflag=dsync</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1+0 records in</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1+0 records out</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1073741824 bytes (1.1 GB) copied, 7.61138 s, 141 MB/s</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">if=/dev/urandom of=/data/test_file bs=1024 count=1000000</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1000000+0 records in</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1000000+0 records out</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1024000000 bytes (1.0 GB) copied, 6.35051 s, 161 MB/s</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">dd if=/dev/zero of=/data/test_file bs=1024 count=1000000</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1000000+0 records in</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1000000+0 records out</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">1024000000 bytes (1.0 GB) copied, 1.6899 s, 606 MB/s</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">When I disable this brick (service glusterd stop; pkill
glusterfsd) performance in gluster is better, but not on par with what it was.
Also the cpu usages on the “neighbor” nodes which hosts the other bricks in the
same subvolume increases quite a lot in this case, which I wouldn’t expect
actually since they shouldn't handle much more work, except flagging shards to heal. Iowait also goes to idle once gluster is stopped, so it’s for sure gluster which waits for io.</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p></div></blockquote><div><br></div><div>So I see that FSYNC %-latency is on the higher side. And I also noticed you don't have direct-io options enabled on the volume.</div><div>Could you set the following options on the volume -</div><div># gluster volume set <VOLNAME> network.remote-dio off</div><div># gluster volume set <VOLNAME> performance.strict-o-direct on</div><div>and also disable choose-local <br></div><div># gluster volume set <VOLNAME> cluster.choose-local off<br></div><div> </div><div>let me know if this helps.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">2. I’ve attached the mnt log and volume info, but I couldn’t
find anything relevant in in those logs. I think this is because we run the VM’s
with libgfapi;</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">[root@ovirt-host-01 ~]# engine-config -g LibgfApiSupported</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">LibgfApiSupported: true version: 4.2</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">LibgfApiSupported: true version: 4.1</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">LibgfApiSupported: true version: 4.3</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">And I can confirm the qemu process is invoked with the
gluster:// address for the images.</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">The message is logged in the
/var/lib/libvert/qemu/<machine>
file, which I’ve also included. For a sample case see around; 2019-03-28
20:20:07</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Which has the error; E [MSGID: 133010]
[shard.c:2294:shard_common_lookup_shards_cbk] 0-ovirt-kube-shard: Lookup on
shard 109886 failed. Base file gfid = a38d64bc-a28b-4ee1-a0bb-f919e7a1022c
[Stale file handle]</p></div></blockquote><div><br></div><div>Could you also attach the brick logs for this volume?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">3. yes I see multiple instances for the same brick directory,
like; </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">/usr/sbin/glusterfsd -s 10.32.9.6 --volfile-id
ovirt-core.10.32.9.6.data-gfs-bricks-brick1-ovirt-core -p
/var/run/gluster/vols/ovirt-core/10.32.9.6-data-gfs-bricks-brick1-ovirt-core.pid
-S /var/run/gluster/452591c9165945d9.socket --brick-name
/data/gfs/bricks/brick1/ovirt-core -l
/var/log/glusterfs/bricks/data-gfs-bricks-brick1-ovirt-core.log --xlator-option
*-posix.glusterd-uuid=fb513da6-f3bd-4571-b8a2-db5efaf60cc1 --process-name brick
--brick-port 49154 --xlator-option ovirt-core-server.listen-port=49154</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">I’ve made an export of the output of ps from the time I observed
these multiple processes. </p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">In addition the brick_mux bug as noted by Atin. I might also
have another possible cause, as ovirt moves nodes from none-operational state
or maintenance state to active/activating, it also seems to restart gluster,
however I don’t have direct proof for this theory.</p>
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif"> </p></div></blockquote><div><br></div><div><a class="gmail_plusreply" id="gmail-m_-4660889944992995096plusReplyChip-0" href="mailto:amukherj@redhat.com" target="_blank">+Atin Mukherjee</a> ^^<br></div><div><a class="gmail_plusreply" id="gmail-m_-4660889944992995096plusReplyChip-1" href="mailto:moagrawa@redhat.com" target="_blank">+Mohit Agrawal</a> ^^</div><div><br></div><div>-Krutika</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">
<p class="MsoNormal" style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">Thanks Olaf</p></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Op vr 29 mrt. 2019 om 10:03 schreef Sandro Bonazzola <<a href="mailto:sbonazzo@redhat.com" target="_blank">sbonazzo@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Il giorno gio 28 mar 2019 alle ore 17:48 <<a href="mailto:olaf.buitelaar@gmail.com" target="_blank">olaf.buitelaar@gmail.com</a>> ha scritto:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Dear All,<br>
<br>
I wanted to share my experience upgrading from 4.2.8 to 4.3.1. While previous upgrades from 4.1 to 4.2 etc. went rather smooth, this one was a different experience. After first trying a test upgrade on a 3 node setup, which went fine. i headed to upgrade the 9 node production platform, unaware of the backward compatibility issues between gluster 3.12.15 -> 5.3. After upgrading 2 nodes, the HA engine stopped and wouldn't start. Vdsm wasn't able to mount the engine storage domain, since /dom_md/metadata was missing or couldn't be accessed. Restoring this file by getting a good copy of the underlying bricks, removing the file from the underlying bricks where the file was 0 bytes and mark with the stickybit, and the corresponding gfid's. Removing the file from the mount point, and copying back the file on the mount point. Manually mounting the engine domain, and manually creating the corresponding symbolic links in /rhev/data-center and /var/run/vdsm/storage and fixing the ownership back to vdsm.kvm (which was root.root), i was able to start the HA engine again. Since the engine was up again, and things seemed rather unstable i decided to continue the upgrade on the other nodes suspecting an incompatibility in gluster versions, i thought would be best to have them all on the same version rather soonish. However things went from bad to worse, the engine stopped again, and all vm’s stopped working as well. So on a machine outside the setup and restored a backup of the engine taken from version 4.2.8 just before the upgrade. With this engine I was at least able to start some vm’s again, and finalize the upgrade. Once the upgraded, things didn’t stabilize and also lose 2 vm’s during the process due to image corruption. After figuring out gluster 5.3 had quite some issues I was as lucky to see gluster 5.5 was about to be released, on the moment the RPM’s were available I’ve installed those. This helped a lot in terms of stability, for which I’m very grateful! However the performance is unfortunate terrible, it’s about 15% of what the performance was running gluster 3.12.15. It’s strange since a simple dd shows ok performance, but our actual workload doesn’t. While I would expect the performance to be better, due to all improvements made since gluster version 3.12. Does anybody share the same experience?<br>
I really hope gluster 6 will soon be tested with ovirt and released, and things start to perform and stabilize again..like the good old days. Of course when I can do anything, I’m happy to help.<br></blockquote><div><br></div><div>Opened <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1693998" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1693998</a> to track the rebase on Gluster 6.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I think the following short list of issues we have after the migration;<br>
Gluster 5.5;<br>
- Poor performance for our workload (mostly write dependent)<br>
- VM’s randomly pause on unknown storage errors, which are “stale file’s”. corresponding log; Lookup on shard 797 failed. Base file gfid = 8a27b91a-ff02-42dc-bd4c-caa019424de8 [Stale file handle]<br>
- Some files are listed twice in a directory (probably related the stale file issue?)<br>
Example;<br>
ls -la /rhev/data-center/59cd53a9-0003-02d7-00eb-0000000001e3/313f5d25-76af-4ecd-9a20-82a2fe815a3c/images/4add6751-3731-4bbd-ae94-aaeed12ea450/<br>
total 3081<br>
drwxr-x---. 2 vdsm kvm 4096 Mar 18 11:34 .<br>
drwxr-xr-x. 13 vdsm kvm 4096 Mar 19 09:42 ..<br>
-rw-rw----. 1 vdsm kvm 1048576 Mar 28 12:55 1a7cf259-6b29-421d-9688-b25dfaafb13c<br>
-rw-rw----. 1 vdsm kvm 1048576 Mar 28 12:55 1a7cf259-6b29-421d-9688-b25dfaafb13c<br>
-rw-rw----. 1 vdsm kvm 1048576 Jan 27 2018 1a7cf259-6b29-421d-9688-b25dfaafb13c.lease<br>
-rw-r--r--. 1 vdsm kvm 290 Jan 27 2018 1a7cf259-6b29-421d-9688-b25dfaafb13c.meta<br>
-rw-r--r--. 1 vdsm kvm 290 Jan 27 2018 1a7cf259-6b29-421d-9688-b25dfaafb13c.meta<br>
<br>
- brick processes sometimes starts multiple times. Sometimes I’ve 5 brick processes for a single volume. Killing all glusterfsd’s for the volume on the machine and running gluster v start <vol> force usually just starts one after the event, from then on things look all right. <br>
<br></blockquote><div><br></div><div>May I kindly ask to open bugs on Gluster for above issues at <a href="https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS" target="_blank">https://bugzilla.redhat.com/enter_bug.cgi?product=GlusterFS</a> ?</div><div>Sahina?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Ovirt 4.3.2.1-1.el7<br>
- All vms images ownership are changed to root.root after the vm is shutdown, probably related to; <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1666795" rel="noreferrer" target="_blank">https://bugzilla.redhat.com/show_bug.cgi?id=1666795</a> but not only scoped to the HA engine. I’m still in compatibility mode 4.2 for the cluster and for the vm’s, but upgraded to version ovirt 4.3.2<br></blockquote><div><br></div><div>Ryan?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- The network provider is set to ovn, which is fine..actually cool, only the “ovs-vswitchd” is a CPU hog, and utilizes 100%<br></blockquote><div><br></div><div>Miguel? Dominik?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- It seems on all nodes vdsm tries to get the the stats for the HA engine, which is filling the logs with (not sure if this is new);<br>
[api.virt] FINISH getStats return={'status': {'message': "Virtual machine does not exist: {'vmId': u'20d69acd-edfd-4aeb-a2ae-49e9c121b7e9'}", 'code': 1}} from=::1,59290, vmId=20d69acd-edfd-4aeb-a2ae-49e9c121b7e9 (api:54)<br></blockquote><div><br></div><div>Simone?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
- It seems the package os_brick [root] managedvolume not supported: Managed Volume Not Supported. Missing package os-brick.: ('Cannot import os_brick',) (caps:149) which fills the vdsm.log, but for this I also saw another message, so I suspect this will already be resolved shortly<br>
- The machine I used to run the backup HA engine, doesn’t want to get removed from the hosted-engine –vm-status, not even after running; hosted-engine --clean-metadata --host-id=10 --force-clean or hosted-engine --clean-metadata --force-clean from the machine itself.<br></blockquote><div><br></div><div>Simone?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Think that's about it.<br>
<br>
Don’t get me wrong, I don’t want to rant, I just wanted to share my experience and see where things can made better. <br></blockquote><div><br></div><div>If not already done, can you please open bugs for above issues at <a href="https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt" target="_blank">https://bugzilla.redhat.com/enter_bug.cgi?classification=oVirt</a> ?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
Best Olaf<br>
_______________________________________________<br>
Users mailing list -- <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
To unsubscribe send an email to <a href="mailto:users-leave@ovirt.org" target="_blank">users-leave@ovirt.org</a><br>
Privacy Statement: <a href="https://www.ovirt.org/site/privacy-policy/" rel="noreferrer" target="_blank">https://www.ovirt.org/site/privacy-policy/</a><br>
oVirt Code of Conduct: <a href="https://www.ovirt.org/community/about/community-guidelines/" rel="noreferrer" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br>
List Archives: <a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/3CO35Q7VZMWNHS4LPUJNO7S47MGLSKS5/" rel="noreferrer" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/3CO35Q7VZMWNHS4LPUJNO7S47MGLSKS5/</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-4660889944992995096gmail-m_-1229915118150265429gmail-m_-6615391421469341239gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><p style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-weight:bold;margin:0px;padding:0px;font-size:14px;text-transform:uppercase"><span>SANDRO</span> <span>BONAZZOLA</span></p><p style="margin:0px 0px 4px"><span style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:10px;text-transform:uppercase">MANAGER, SOFTWARE ENGINEERING, </span><font face="overpass, sans-serif" color="#000000"><span style="font-size:10px;text-transform:uppercase"><span style="margin:0px"></span>EMEA R&D RHV</span></font></p><p style="font-family:overpass,sans-serif;margin:0px;font-size:10px;color:rgb(153,153,153)"><a href="https://www.redhat.com/" style="color:rgb(0,136,206);margin:0px" target="_blank">Red Hat <span>EMEA</span></a></p><p style="font-family:overpass,sans-serif;margin:0px 0px 6px;font-size:10px;color:rgb(153,153,153)"><span style="margin:0px;padding:0px"><a href="mailto:sbonazzo@redhat.com" style="color:rgb(0,136,206);margin:0px" target="_blank">sbonazzo@redhat.com</a> </span></p><table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium" border="0"><tbody><tr><td width="100px"><a href="https://red.ht/sig" target="_blank"><img src="https://www.redhat.com/files/brand/email/sig-redhat.png" width="90" height="auto"></a></td></tr></tbody></table><table style="color:rgb(0,0,0);font-family:overpass,sans-serif;font-size:medium" border="0"><tbody><tr></tr></tbody></table></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</blockquote></div>
_______________________________________________<br>
Users mailing list -- <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
To unsubscribe send an email to <a href="mailto:users-leave@ovirt.org" target="_blank">users-leave@ovirt.org</a><br>
Privacy Statement: <a href="https://www.ovirt.org/site/privacy-policy/" rel="noreferrer" target="_blank">https://www.ovirt.org/site/privacy-policy/</a><br>
oVirt Code of Conduct: <a href="https://www.ovirt.org/community/about/community-guidelines/" rel="noreferrer" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br>
List Archives: <a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/HAGTA64LF7LLE6YMHQ6DLT26MD2GZ2PK/" rel="noreferrer" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/HAGTA64LF7LLE6YMHQ6DLT26MD2GZ2PK/</a><br>
</blockquote></div></div>
</blockquote></div>