<div dir="ltr"><div>Do you see errors reported in the mount logs for the volume? If so, could you attach the logs?<br></div>Any issues with your underlying disks. Can you also attach output of volume profiling?<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 30, 2018 at 12:13 AM, Jim Kusznir <span dir="ltr"><<a href="mailto:jim@palousetech.com" target="_blank">jim@palousetech.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Ok, things have gotten MUCH worse this morning. I'm getting random errors from VMs, right now, about a third of my VMs have been paused due to storage issues, and most of the remaining VMs are not performing well.<div><br></div><div>At this point, I am in full EMERGENCY mode, as my production services are now impacted, and I'm getting calls coming in with problems...</div><div><br></div><div>I'd greatly appreciate help...VMs are running VERY slowly (when they run), and they are steadily getting worse. I don't know why. I was seeing CPU peaks (to 100%) on several VMs, in perfect sync, for a few minutes at a time (while the VM became unresponsive and any VMs I was logged into that were linux were giving me the CPU stuck messages in my origional post). Is all this storage related?</div><div><br></div><div>I also have two different gluster volumes for VM storage, and only one had the issues, but now VMs in both are being affected at the same time and same way.</div><span class="HOEnZb"><font color="#888888"><div><br></div><div>--Jim</div></font></span></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 28, 2018 at 10:50 PM, Sahina Bose <span dir="ltr"><<a href="mailto:sabose@redhat.com" target="_blank">sabose@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">[Adding gluster-users to look at the heal issue]<br></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_2506865858631215125h5">On Tue, May 29, 2018 at 9:17 AM, Jim Kusznir <span dir="ltr"><<a href="mailto:jim@palousetech.com" target="_blank">jim@palousetech.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_2506865858631215125h5"><div dir="ltr">Hello:<div><br></div><div>I've been having some cluster and gluster performance issues lately. I also found that my cluster was out of date, and was trying to apply updates (hoping to fix some of these), and discovered the ovirt 4.1 repos were taken completely offline. So, I was forced to begin an upgrade to 4.2. According to docs I found/read, I needed only add the new repo, do a yum update, reboot, and be good on my hosts (did the yum update, the engine-setup on my hosted engine). Things seemed to work relatively well, except for a gluster sync issue that showed up.</div><div><br></div><div>My cluster is a 3 node hyperconverged cluster. I upgraded the hosted engine first, then engine 3. When engine 3 came back up, for some reason one of my gluster volumes would not sync. Here's sample output:</div><div><br></div><div><div>[root@ovirt3 ~]# gluster volume heal data-hdd info</div><div>Brick 172.172.1.11:/gluster/brick3/d<wbr>ata-hdd</div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/48d7ecb8-7ac5-4<wbr>725-bca5-b3519681cf2f/0d6080b0<wbr>-7018-4fa3-bb82-1dd9ef07d9b9 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/647be733-f153-4<wbr>cdc-85bd-ba72544c2631/b453a300<wbr>-0602-4be1-8310-8bd5abe00971 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/6da854d1-b6be-4<wbr>46b-9bf0-90a0dbbea830/3c93bd1f<wbr>-b7fa-4aa2-b445-6904e31839ba </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/7f647567-d18c-4<wbr>4f1-a58e-9b8865833acb/f9364470<wbr>-9770-4bb1-a6b9-a54861849625 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/f3c8e7aa-6ef2-4<wbr>2a7-93d4-e0a4df6dd2fa/2eb0b1ad<wbr>-2606-44ef-9cd3-ae59610a504b </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/b1ea3f62-0f05-4<wbr>ded-8c82-9c91c90e0b61/d5d6bf5a<wbr>-499f-431d-9013-5453db93ed32 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/8c8b5147-e9d6-4<wbr>810-b45b-185e3ed65727/16f08231<wbr>-93b0-489d-a2fd-687b6bf88eaa </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/12924435-b9c2-4<wbr>aab-ba19-1c1bc31310ef/07b3db69<wbr>-440e-491e-854c-bbfa18a7cff2 </div><div>Status: Connected</div><div>Number of entries: 8</div><div><br></div><div>Brick 172.172.1.12:/gluster/brick3/d<wbr>ata-hdd</div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/48d7ecb8-7ac5-4<wbr>725-bca5-b3519681cf2f/0d6080b0<wbr>-7018-4fa3-bb82-1dd9ef07d9b9 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/647be733-f153-4<wbr>cdc-85bd-ba72544c2631/b453a300<wbr>-0602-4be1-8310-8bd5abe00971 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/b1ea3f62-0f05-4<wbr>ded-8c82-9c91c90e0b61/d5d6bf5a<wbr>-499f-431d-9013-5453db93ed32 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/6da854d1-b6be-4<wbr>46b-9bf0-90a0dbbea830/3c93bd1f<wbr>-b7fa-4aa2-b445-6904e31839ba </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/7f647567-d18c-4<wbr>4f1-a58e-9b8865833acb/f9364470<wbr>-9770-4bb1-a6b9-a54861849625 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/8c8b5147-e9d6-4<wbr>810-b45b-185e3ed65727/16f08231<wbr>-93b0-489d-a2fd-687b6bf88eaa </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/12924435-b9c2-4<wbr>aab-ba19-1c1bc31310ef/07b3db69<wbr>-440e-491e-854c-bbfa18a7cff2 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/f3c8e7aa-6ef2-4<wbr>2a7-93d4-e0a4df6dd2fa/2eb0b1ad<wbr>-2606-44ef-9cd3-ae59610a504b </div><div>Status: Connected</div><div>Number of entries: 8</div><div><br></div><div>Brick 172.172.1.13:/gluster/brick3/d<wbr>ata-hdd</div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/b1ea3f62-0f05-4<wbr>ded-8c82-9c91c90e0b61/d5d6bf5a<wbr>-499f-431d-9013-5453db93ed32 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/8c8b5147-e9d6-4<wbr>810-b45b-185e3ed65727/16f08231<wbr>-93b0-489d-a2fd-687b6bf88eaa </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/12924435-b9c2-4<wbr>aab-ba19-1c1bc31310ef/07b3db69<wbr>-440e-491e-854c-bbfa18a7cff2 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/f3c8e7aa-6ef2-4<wbr>2a7-93d4-e0a4df6dd2fa/2eb0b1ad<wbr>-2606-44ef-9cd3-ae59610a504b </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/647be733-f153-4<wbr>cdc-85bd-ba72544c2631/b453a300<wbr>-0602-4be1-8310-8bd5abe00971 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/48d7ecb8-7ac5-4<wbr>725-bca5-b3519681cf2f/0d6080b0<wbr>-7018-4fa3-bb82-1dd9ef07d9b9 </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/6da854d1-b6be-4<wbr>46b-9bf0-90a0dbbea830/3c93bd1f<wbr>-b7fa-4aa2-b445-6904e31839ba </div><div>/cc65f671-3377-494a-a7d4-1d9f7<wbr>c3ae46c/images/7f647567-d18c-4<wbr>4f1-a58e-9b8865833acb/f9364470<wbr>-9770-4bb1-a6b9-a54861849625 </div><div>Status: Connected</div><div>Number of entries: 8</div></div><div><br></div><div>---------</div><div>Its been in this state for a couple days now, and bandwidth monitoring shows no appreciable data moving. I've tried repeatedly commanding a full heal from all three clusters in the node. Its always the same files that need healing.</div><div><br></div><div>When running gluster volume heal data-hdd statistics, I see sometimes different information, but always some number of "heal failed" entries. It shows 0 for split brain.</div><div><br></div><div>I'm not quite sure what to do. I suspect it may be due to nodes 1 and 2 still being on the older ovirt/gluster release, but I'm afraid to upgrade and reboot them until I have a good gluster sync (don't need to create a split brain issue). How do I proceed with this?</div><div><br></div><div>Second issue: I've been experiencing VERY POOR performance on most of my VMs. To the tune that logging into a windows 10 vm via remote desktop can take 5 minutes, launching quickbooks inside said vm can easily take 10 minutes. On some linux VMs, I get random messages like this:</div><div><div>Message from syslogd@unifi at May 28 20:39:23 ...</div><div> kernel:[6171996.308904] NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [mongod:14766]</div></div><div><br></div><div>(the process and PID are often different)</div><div><br></div><div>I'm not quite sure what to do about this either. My initial thought was upgrad everything to current and see if its still there, but I cannot move forward with that until my gluster is healed...</div><div><br></div><div>Thanks!</div><span class="m_2506865858631215125m_-3484925472286407273HOEnZb"><font color="#888888"><div>--Jim</div></font></span></div>
<br></div></div>______________________________<wbr>_________________<br>
Users mailing list -- <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
To unsubscribe send an email to <a href="mailto:users-leave@ovirt.org" target="_blank">users-leave@ovirt.org</a><br>
Privacy Statement: <a href="https://www.ovirt.org/site/privacy-policy/" rel="noreferrer" target="_blank">https://www.ovirt.org/site/pri<wbr>vacy-policy/</a><br>
oVirt Code of Conduct: <a href="https://www.ovirt.org/community/about/community-guidelines/" rel="noreferrer" target="_blank">https://www.ovirt.org/communit<wbr>y/about/community-guidelines/</a><br>
List Archives: <a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/3LEV6ZQ3JV2XLAL7NYBTXOYMYUOTIRQF/" rel="noreferrer" target="_blank">https://lists.ovirt.org/archiv<wbr>es/list/users@ovirt.org/messag<wbr>e/3LEV6ZQ3JV2XLAL7NYBTXOYMYUOT<wbr>IRQF/</a><br>
<br></blockquote></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div>