<div dir="ltr"><div><div>Thanks for that update. Very happy to hear it ran fine without any issues. :)<br><br></div>Yeah so you can ignore those 'No such file or directory' errors. They represent a transient state where DHT in the client process is yet to figure out the new location of the file.<br><br></div>-Krutika<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, May 29, 2017 at 6:51 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-6460752556636082476divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hello,</p>
<p><br>
</p>
<p>Yes, i forgot to upgrade the client as well. </p>
<p>I did the upgrade and created a new volume, same options as before, with one VM running and doing lots of IOs. i started the rebalance with force and after it completed the process i rebooted the VM, and it did start normally without issues.</p>
<p>I repeated the process and did another rebalance while the VM running and everything went fine.</p>
<p>But the logs in the client throwing lots of warning messages:</p>
<p><br>
</p>
<p></p>
<div>[2017-05-29 13:14:59.416382] W [MSGID: 114031] [client-rpc-fops.c:2928:<wbr>client3_3_lookup_cbk] 2-gfs_vol2-client-2: remote operation failed. Path: /50294ed6-db7a-418d-965f-<wbr>9b44c69a83fd/images/d59487fe-<wbr>f3a9-4bad-a607-3a181c871711/<wbr>aa01c3a0-5aa0-432d-82ad-<wbr>d1f515f1d87f
(93c403f5-c769-44b9-a087-<wbr>dc51fc21412e) [No such file or directory]</div>
<div>[2017-05-29 13:14:59.416427] W [MSGID: 114031] [client-rpc-fops.c:2928:<wbr>client3_3_lookup_cbk] 2-gfs_vol2-client-3: remote operation failed. Path: /50294ed6-db7a-418d-965f-<wbr>9b44c69a83fd/images/d59487fe-<wbr>f3a9-4bad-a607-3a181c871711/<wbr>aa01c3a0-5aa0-432d-82ad-<wbr>d1f515f1d87f
(93c403f5-c769-44b9-a087-<wbr>dc51fc21412e) [No such file or directory]</div>
<div>[2017-05-29 13:14:59.808251] W [MSGID: 114031] [client-rpc-fops.c:2928:<wbr>client3_3_lookup_cbk] 2-gfs_vol2-client-2: remote operation failed. Path: /50294ed6-db7a-418d-965f-<wbr>9b44c69a83fd/images/d59487fe-<wbr>f3a9-4bad-a607-3a181c871711/<wbr>aa01c3a0-5aa0-432d-82ad-<wbr>d1f515f1d87f
(93c403f5-c769-44b9-a087-<wbr>dc51fc21412e) [No such file or directory]</div>
<div>[2017-05-29 13:14:59.808287] W [MSGID: 114031] [client-rpc-fops.c:2928:<wbr>client3_3_lookup_cbk] 2-gfs_vol2-client-3: remote operation failed. Path: /50294ed6-db7a-418d-965f-<wbr>9b44c69a83fd/images/d59487fe-<wbr>f3a9-4bad-a607-3a181c871711/<wbr>aa01c3a0-5aa0-432d-82ad-<wbr>d1f515f1d87f
(93c403f5-c769-44b9-a087-<wbr>dc51fc21412e) [No such file or directory]</div>
<p></p>
<p><br>
</p>
<div id="m_-6460752556636082476Signature"><br>
</div>
<div id="m_-6460752556636082476Signature">Although the process went smooth, i will run another extensive test tomorrow just to be sure.</div><span class="">
<div id="m_-6460752556636082476Signature"><br>
<div class="m_-6460752556636082476ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6460752556636082476divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Monday, May 29, 2017 9:20:29 AM<div><div class="h5"><br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> gluster-user; Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier<br>
<b>Subject:</b> Re: Rebalance + VM corruption - current status and request for feedback</div></div></font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
I took a look at your logs.<br>
</div>
It very much seems like an issue that is caused by a mismatch in glusterfs client and server packages.<br>
</div>
So your client (mount) seems to be still running 3.7.20, as confirmed by the occurrence of the following log message:<br>
<br>
[2017-05-26 08:58:23.647458] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
/rhev/data-center/mnt/<wbr>glusterSD/s1:_testvol)<br>
[2017-05-26 08:58:40.901204] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
/rhev/data-center/mnt/<wbr>glusterSD/s1:_testvol)<br>
[2017-05-26 08:58:48.923452] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
/rhev/data-center/mnt/<wbr>glusterSD/s1:_testvol)<br>
<br>
</div>
whereas the servers have rightly been upgraded to 3.10.2, as seen in rebalance log:<br>
<br>
[2017-05-26 09:36:36.075940] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.2 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/testvol --xlator-option *dht.use-readdirp=yes --xlator-option
*dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-<wbr>heal=off --xlator-option *replicate*.entry-self-heal=<wbr>off --xlator-option *dht.readdir-optimize=on
--xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=7c0bf49e-1ede-<wbr>47b1-b9a5-bfde6e60f07b --xlator-option *dht.commit-hash=3376396580 --socket-file /var/run/gluster/gluster-<wbr>rebalance-801faefa-a583-46b4-<wbr>8eef-e0ec160da9ea.sock --pid-file /var/lib/glusterd/vols/<wbr>testvol/rebalance/7c0bf49e-<wbr>1ede-47b1-b9a5-bfde6e60f07b.<wbr>pid
-l /var/log/glusterfs/testvol-<wbr>rebalance.log)<br>
<br>
<br>
</div>
Could you upgrade all packages to 3.10.2 and try again?<br>
<br>
</div>
-Krutika<br>
<div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 26, 2017 at 4:46 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-6460752556636082476m_-1745404107683404034divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Attached are the logs for both the rebalance and the mount.</p>
<span>
<p><br>
</p>
<div id="m_-6460752556636082476m_-1745404107683404034Signature"><br>
<div class="m_-6460752556636082476m_-1745404107683404034ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6460752556636082476m_-1745404107683404034divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Friday, May 26, 2017 1:12:28 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> gluster-user; Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier<br>
<b>Subject:</b> Re: Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div>
<div>
<div class="m_-6460752556636082476h5">
<div>
<div dir="ltr">
<div>Could you provide the rebalance and mount logs?<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 26, 2017 at 3:17 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-6460752556636082476m_-1745404107683404034m_4798580117692497506divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Good morning,</p>
<p><br>
</p>
<p>So i have tested the new Gluster 3.10.2, and after starting rebalance two VMs were paused due to storage error and third one was not responding.</p>
<p>After rebalance completed i started the VMs and it did not boot, and throw an XFS wrong inode error into the screen.</p>
<p><br>
</p>
<p>My setup:</p>
<p>4 nodes running CentOS7.3 with Gluster 3.10.2</p>
<p>4 bricks in distributed replica with group set to virt.</p>
<p>I added the volume to ovirt and created three VMs, i ran a loop to create 5GB file inside the VMs.</p>
<p>Added new 4 bricks to the existing nodes. </p>
<p>Started rebalane "with force to bypass the warning message"</p>
<p>VMs started to fail after rebalancing.</p>
<span>
<p><br>
</p>
<p><br>
</p>
<div id="m_-6460752556636082476m_-1745404107683404034m_4798580117692497506Signature"><br>
<div class="m_-6460752556636082476m_-1745404107683404034m_4798580117692497506ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6460752556636082476m_-1745404107683404034m_4798580117692497506divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><span><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Wednesday, May 17, 2017 6:59:20 AM<br>
<b>To:</b> gluster-user<br>
<b>Cc:</b> Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier; Mahdi Adnan<br>
</span><b>Subject:</b> Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div>
<div>
<div class="m_-6460752556636082476m_-1745404107683404034h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
In the past couple of weeks, we've sent the following fixes concerning VM corruption upon doing rebalance -
<a href="https://review.gluster.org/#/q/status:merged+project:glusterfs+branch:master+topic:bug-1440051" target="_blank">
https://review.gluster.org/#/q<wbr>/status:merged+project:gluster<wbr>fs+branch:master+topic:bug-144<wbr>0051</a><br>
<br>
</div>
These fixes are very much part of the latest 3.10.2 release.<br>
<br>
</div>
Satheesaran within Red Hat also verified that they work and he's not seeing corruption issues anymore.<br>
<br>
</div>
I'd like to hear feedback from the users themselves on these fixes (on your test environments to begin with) before even changing the status of the bug to CLOSED.<br>
<br>
</div>
<div>Although 3.10.2 has a patch that prevents rebalance sub-commands from being executed on sharded volumes, you can override the check by using the 'force' option.<br>
<br>
</div>
<div>For example,<br>
<br>
</div>
<div># gluster volume rebalance myvol start force<br>
<br>
</div>
<div>Very much looking forward to hearing from you all.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Krutika<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</blockquote></div><br></div>