<div dir="ltr"><div><div><div><div><div><div>Hi,<br><br></div>I took a look at your logs.<br></div>It very much seems like an issue that is caused by a mismatch in glusterfs client and server packages.<br></div>So your client (mount) seems to be still running 3.7.20, as confirmed by the occurrence of the following log message:<br><br>[2017-05-26 08:58:23.647458] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol /rhev/data-center/mnt/glusterSD/s1:_testvol)<br>[2017-05-26 08:58:40.901204] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol /rhev/data-center/mnt/glusterSD/s1:_testvol)<br>[2017-05-26 08:58:48.923452] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol /rhev/data-center/mnt/glusterSD/s1:_testvol)<br><br></div>whereas the servers have rightly been upgraded to 3.10.2, as seen in rebalance log:<br><br>[2017-05-26 09:36:36.075940] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.2 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/testvol --xlator-option *dht.use-readdirp=yes --xlator-option *dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *dht.readdir-optimize=on --xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=7c0bf49e-1ede-47b1-b9a5-bfde6e60f07b --xlator-option *dht.commit-hash=3376396580 --socket-file /var/run/gluster/gluster-rebalance-801faefa-a583-46b4-8eef-e0ec160da9ea.sock --pid-file /var/lib/glusterd/vols/testvol/rebalance/7c0bf49e-1ede-47b1-b9a5-bfde6e60f07b.pid -l /var/log/glusterfs/testvol-rebalance.log)<br><br><br></div>Could you upgrade all packages to 3.10.2 and try again?<br><br></div>-Krutika<br><div><div><br></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 26, 2017 at 4:46 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-1745404107683404034divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Attached are the logs for both the rebalance and the mount.</p><span class="">
<p><br>
</p>
<div id="m_-1745404107683404034Signature"><br>
<div class="m_-1745404107683404034ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-1745404107683404034divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Friday, May 26, 2017 1:12:28 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> gluster-user; Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier<br>
<b>Subject:</b> Re: Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">
<div>Could you provide the rebalance and mount logs?<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 26, 2017 at 3:17 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-1745404107683404034m_4798580117692497506divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Good morning,</p>
<p><br>
</p>
<p>So i have tested the new Gluster 3.10.2, and after starting rebalance two VMs were paused due to storage error and third one was not responding.</p>
<p>After rebalance completed i started the VMs and it did not boot, and throw an XFS wrong inode error into the screen.</p>
<p><br>
</p>
<p>My setup:</p>
<p>4 nodes running CentOS7.3 with Gluster 3.10.2</p>
<p>4 bricks in distributed replica with group set to virt.</p>
<p>I added the volume to ovirt and created three VMs, i ran a loop to create 5GB file inside the VMs.</p>
<p>Added new 4 bricks to the existing nodes. </p>
<p>Started rebalane "with force to bypass the warning message"</p>
<p>VMs started to fail after rebalancing.</p>
<span>
<p><br>
</p>
<p><br>
</p>
<div id="m_-1745404107683404034m_4798580117692497506Signature"><br>
<div class="m_-1745404107683404034m_4798580117692497506ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-1745404107683404034m_4798580117692497506divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><span><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Wednesday, May 17, 2017 6:59:20 AM<br>
<b>To:</b> gluster-user<br>
<b>Cc:</b> Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier; Mahdi Adnan<br>
</span><b>Subject:</b> Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div>
<div>
<div class="m_-1745404107683404034h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
In the past couple of weeks, we've sent the following fixes concerning VM corruption upon doing rebalance -
<a href="https://review.gluster.org/#/q/status:merged+project:glusterfs+branch:master+topic:bug-1440051" target="_blank">
https://review.gluster.org/#/q<wbr>/status:merged+project:gluster<wbr>fs+branch:master+topic:bug-<wbr>1440051</a><br>
<br>
</div>
These fixes are very much part of the latest 3.10.2 release.<br>
<br>
</div>
Satheesaran within Red Hat also verified that they work and he's not seeing corruption issues anymore.<br>
<br>
</div>
I'd like to hear feedback from the users themselves on these fixes (on your test environments to begin with) before even changing the status of the bug to CLOSED.<br>
<br>
</div>
<div>Although 3.10.2 has a patch that prevents rebalance sub-commands from being executed on sharded volumes, you can override the check by using the 'force' option.<br>
<br>
</div>
<div>For example,<br>
<br>
</div>
<div># gluster volume rebalance myvol start force<br>
<br>
</div>
<div>Very much looking forward to hearing from you all.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Krutika<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</blockquote></div><br></div>