<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Hello,</p>
<p><br>
</p>
<p>Yes, i forgot to upgrade the client as well. </p>
<p>I did the upgrade and created a new volume, same options as before, with one VM running and doing lots of IOs. i started the rebalance with force and after it completed the process i rebooted the VM, and it did start normally without issues.</p>
<p>I repeated the process and did another rebalance while the VM running and everything went fine.</p>
<p>But the logs in the client throwing lots of warning messages:</p>
<p><br>
</p>
<p></p>
<div>[2017-05-29 13:14:59.416382] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]</div>
<div>[2017-05-29 13:14:59.416427] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-3: remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]</div>
<div>[2017-05-29 13:14:59.808251] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-2: remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]</div>
<div>[2017-05-29 13:14:59.808287] W [MSGID: 114031] [client-rpc-fops.c:2928:client3_3_lookup_cbk] 2-gfs_vol2-client-3: remote operation failed. Path: /50294ed6-db7a-418d-965f-9b44c69a83fd/images/d59487fe-f3a9-4bad-a607-3a181c871711/aa01c3a0-5aa0-432d-82ad-d1f515f1d87f
(93c403f5-c769-44b9-a087-dc51fc21412e) [No such file or directory]</div>
<p></p>
<p><br>
</p>
<div id="Signature"><br>
</div>
<div id="Signature">Although the process went smooth, i will run another extensive test tomorrow just to be sure.</div>
<div id="Signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Krutika Dhananjay <kdhananj@redhat.com><br>
<b>Sent:</b> Monday, May 29, 2017 9:20:29 AM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> gluster-user; Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier<br>
<b>Subject:</b> Re: Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
I took a look at your logs.<br>
</div>
It very much seems like an issue that is caused by a mismatch in glusterfs client and server packages.<br>
</div>
So your client (mount) seems to be still running 3.7.20, as confirmed by the occurrence of the following log message:<br>
<br>
[2017-05-26 08:58:23.647458] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
/rhev/data-center/mnt/glusterSD/s1:_testvol)<br>
[2017-05-26 08:58:40.901204] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
/rhev/data-center/mnt/glusterSD/s1:_testvol)<br>
[2017-05-26 08:58:48.923452] I [MSGID: 100030] [glusterfsd.c:2338:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.7.20 (args: /usr/sbin/glusterfs --volfile-server=s1 --volfile-server=s2 --volfile-server=s3 --volfile-server=s4 --volfile-id=/testvol
/rhev/data-center/mnt/glusterSD/s1:_testvol)<br>
<br>
</div>
whereas the servers have rightly been upgraded to 3.10.2, as seen in rebalance log:<br>
<br>
[2017-05-26 09:36:36.075940] I [MSGID: 100030] [glusterfsd.c:2475:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.10.2 (args: /usr/sbin/glusterfs -s localhost --volfile-id rebalance/testvol --xlator-option *dht.use-readdirp=yes --xlator-option
*dht.lookup-unhashed=yes --xlator-option *dht.assert-no-child-down=yes --xlator-option *replicate*.data-self-heal=off --xlator-option *replicate*.metadata-self-heal=off --xlator-option *replicate*.entry-self-heal=off --xlator-option *dht.readdir-optimize=on
--xlator-option *dht.rebalance-cmd=5 --xlator-option *dht.node-uuid=7c0bf49e-1ede-47b1-b9a5-bfde6e60f07b --xlator-option *dht.commit-hash=3376396580 --socket-file /var/run/gluster/gluster-rebalance-801faefa-a583-46b4-8eef-e0ec160da9ea.sock --pid-file /var/lib/glusterd/vols/testvol/rebalance/7c0bf49e-1ede-47b1-b9a5-bfde6e60f07b.pid
-l /var/log/glusterfs/testvol-rebalance.log)<br>
<br>
<br>
</div>
Could you upgrade all packages to 3.10.2 and try again?<br>
<br>
</div>
-Krutika<br>
<div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 26, 2017 at 4:46 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-1745404107683404034divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Attached are the logs for both the rebalance and the mount.</p>
<span class="">
<p><br>
</p>
<div id="m_-1745404107683404034Signature"><br>
<div class="m_-1745404107683404034ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-1745404107683404034divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Friday, May 26, 2017 1:12:28 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> gluster-user; Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier<br>
<b>Subject:</b> Re: Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div>
<div>
<div class="h5">
<div>
<div dir="ltr">
<div>Could you provide the rebalance and mount logs?<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, May 26, 2017 at 3:17 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_-1745404107683404034m_4798580117692497506divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Good morning,</p>
<p><br>
</p>
<p>So i have tested the new Gluster 3.10.2, and after starting rebalance two VMs were paused due to storage error and third one was not responding.</p>
<p>After rebalance completed i started the VMs and it did not boot, and throw an XFS wrong inode error into the screen.</p>
<p><br>
</p>
<p>My setup:</p>
<p>4 nodes running CentOS7.3 with Gluster 3.10.2</p>
<p>4 bricks in distributed replica with group set to virt.</p>
<p>I added the volume to ovirt and created three VMs, i ran a loop to create 5GB file inside the VMs.</p>
<p>Added new 4 bricks to the existing nodes. </p>
<p>Started rebalane "with force to bypass the warning message"</p>
<p>VMs started to fail after rebalancing.</p>
<span>
<p><br>
</p>
<p><br>
</p>
<div id="m_-1745404107683404034m_4798580117692497506Signature"><br>
<div class="m_-1745404107683404034m_4798580117692497506ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-1745404107683404034m_4798580117692497506divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><span><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Wednesday, May 17, 2017 6:59:20 AM<br>
<b>To:</b> gluster-user<br>
<b>Cc:</b> Gandalf Corvotempesta; Lindsay Mathieson; Kevin Lemonnier; Mahdi Adnan<br>
</span><b>Subject:</b> Rebalance + VM corruption - current status and request for feedback</font>
<div> </div>
</div>
<div>
<div class="m_-1745404107683404034h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>Hi,<br>
<br>
</div>
In the past couple of weeks, we've sent the following fixes concerning VM corruption upon doing rebalance -
<a href="https://review.gluster.org/#/q/status:merged+project:glusterfs+branch:master+topic:bug-1440051" target="_blank">
https://review.gluster.org/#/q<wbr>/status:merged+project:gluster<wbr>fs+branch:master+topic:bug-<wbr>1440051</a><br>
<br>
</div>
These fixes are very much part of the latest 3.10.2 release.<br>
<br>
</div>
Satheesaran within Red Hat also verified that they work and he's not seeing corruption issues anymore.<br>
<br>
</div>
I'd like to hear feedback from the users themselves on these fixes (on your test environments to begin with) before even changing the status of the bug to CLOSED.<br>
<br>
</div>
<div>Although 3.10.2 has a patch that prevents rebalance sub-commands from being executed on sharded volumes, you can override the check by using the 'force' option.<br>
<br>
</div>
<div>For example,<br>
<br>
</div>
<div># gluster volume rebalance myvol start force<br>
<br>
</div>
<div>Very much looking forward to hearing from you all.<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Krutika<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>