<div dir="ltr"><div><div><div><div><div>So Raghavendra has an RCA for this issue.<br><br></div>Copy-pasting his comment here:<br><br></div><RCA><br><pre class="gmail-bz_comment_text gmail-bz_wrap_comment_text" id="gmail-comment_text_12">Following is a rough algorithm of shard_writev:
1. Based on the offset, calculate the shards touched by current write.
2. Look for inodes corresponding to these shard files in itable.
3. If one or more inodes are missing from itable, issue mknod for corresponding shard files and ignore EEXIST in cbk.
4. resume writes on respective shards.
Now, imagine a write which falls to an existing "shard_file". For the sake of discussion lets consider a distribute of three subvols - s1, s2, s3
1. "shard_file" hashes to subvolume s2 and is present on s2
2. add a subvolume s4 and initiate a fix layout. The layout of ".shard" is fixed to include s4 and hash ranges are changed.
3. write that touches "shard_file" is issued.
4. The inode for "shard_file" is not present in itable after a graph switch and features/shard issues an mknod.
5. With new layout of .shard, lets say "shard_file" hashes to s3 and mknod (shard_file) on s3 succeeds. But, the shard_file is already present on s2.
So, we have two files on two different subvols of dht representing same shard and this will lead to corruption.
</pre></div></RCA><br><br></div>Raghavendra will be sending out a patch in DHT to fix this issue.<br><br></div>-Krutika<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 28, 2017 at 11:49 PM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="">On Mon, Mar 27, 2017 at 11:29 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<div id="m_-6836490455585458468gmail-m_1183689012231501332divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Do you guys have any update regarding this issue ?<br></p></div></div></blockquote></span><div>I do not actively work on this issue so I do not have an accurate update, but from what I heard from Krutika and Raghavendra(works on DHT) is: Krutika debugged initially and found that the issue seems more likely to be in DHT, Satheesaran who helped us recreate this issue in lab found that just fix-layout without rebalance also caused the corruption 1 out of 3 times. Raghavendra came up with a possible RCA for why this can happen. Raghavendra(CCed) would be the right person to provide accurate update. <br></div><div><div class="h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div id="m_-6836490455585458468gmail-m_1183689012231501332divtagdefaultwrapper" style="font-size:12pt;color:rgb(0,0,0);font-family:calibri,arial,helvetica,sans-serif" dir="ltr"><span class="m_-6836490455585458468gmail-">
<p><br>
</p>
<div id="m_-6836490455585458468gmail-m_1183689012231501332Signature"><br>
<div class="m_-6836490455585458468gmail-m_1183689012231501332ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_-6836490455585458468gmail-m_1183689012231501332divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" face="Calibri, sans-serif" color="#000000"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Tuesday, March 21, 2017 3:02:55 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai; <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List<div><div class="m_-6836490455585458468gmail-h5"><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</div></div></font>
<div> </div>
</div><div><div class="m_-6836490455585458468gmail-h5">
<div>
<div dir="ltr">
<div>
<div>Hi,<br>
<br>
</div>
So it looks like Satheesaran managed to recreate this issue. We will be seeking his help in debugging this. It will be easier that way.<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hello and thank you for your email. <br>
Actually no, i didn't check the gfid of the vms. <br>
If this will help, i can setup a new test cluster and get all the data you need.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Get <a href="https://aka.ms/ghei36" target="_blank">Outlook for Android</a></p>
<br>
<p></p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
From: Nithya Balachandran<br>
Sent: Monday, March 20, 20:57<span><br>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption<br>
</span>To: Krutika Dhananjay<br>
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai, <a href="mailto:gluster-users@gluster.org" target="_blank">
gluster-users@gluster.org</a> List<br>
</p>
<div>
<div class="m_-6836490455585458468gmail-m_1183689012231501332h5">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hi, <br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Do you know the GFIDs of the VM images which were corrupted?<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Regards,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Nithya<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On 20 March 2017 at 20:37, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>> wrote:</p>
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I looked at the logs.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
>From the time the new graph (since the add-brick command you shared where bricks 41 through 44 are added) is switched to (line 3011 onwards in nfs-gfapi.log), I see the following kinds of errors:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
1. Lookups to a bunch of files failed with ENOENT on both replicas which protocol/client converts to ESTALE. I am guessing these entries got migrated to</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
other subvolumes leading to 'No such file or directory' errors.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
DHT and thereafter shard get the same error code and log the following:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] [dht-helper.c:1198:dht_migrati<wbr>on_complete_check_task] 17-vmware2-dht: <gfid:a68ce411-e381-46a3-93cd-<wbr>d2af6a7c3532>: failed to lookup the file on vmware2-dht [Stale file handle] <wbr> <wbr> <wbr> <wbr>
<br>
1 [2017-03-17 14:04:26.353528] E [MSGID: 133014] [shard.c:1253:shard_common_sta<wbr>t_cbk] 17-vmware2-shard: stat failed: a68ce411-e381-46a3-93cd-d2af6a<wbr>7c3532 [Stale file handle]</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
which is fine.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
2. The other kind are from AFR logging of possible split-brain which I suppose are harmless too.<br>
[2017-03-17 14:23:36.968883] W [MSGID: 108008] [afr-read-txn.c:228:afr_read_t<wbr>xn] 17-vmware2-replicate-13: Unreadable subvolume -1 found with event generation 2 for gfid 74d49288-8452-40d4-893e-ff4672<wbr>557ff9. (Possible split-brain)</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Since you are saying the bug is hit only on VMs that are undergoing IO while rebalance is running (as opposed to those that remained powered off),</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
rebalance + IO could be causing some issues.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
CC'ing DHT devs</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Raghavendra/Nithya/Susant,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Could you take a look?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</blockquote>
</div>
</div>
<blockquote type="cite">
<blockquote type="cite">
<div>
<div class="m_-6836490455585458468gmail-m_1183689012231501332h5">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Thank you for your email mate.<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Yes, im aware of this but, to save costs i chose replica 2, this cluster is all flash.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
In version 3.7.x i had issues with ping timeout, if one hosts went down for few seconds the whole cluster hangs and become unavailable, to avoid this i adjusted the ping timeout to 5 seconds.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE or gfapi) im stuck with NFS for this volume.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
The other volume is mounted using gfapi in oVirt cluster.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</div>
</div>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Sunday, March 19, 2017 2:01:49 PM
</font><br>
</span></p>
<span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
While I'm still going through the logs, just wanted to point out a couple of things:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
1. It is recommended that you use 3-way replication (replica count 3) for VM store use case</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Will get back with anything else I might find or more questions if I have any.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Thanks mate,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Kindly, check the attachment.<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Sunday, March 19, 2017 10:00:22 AM
</font><br>
</span></p>
<span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
In that case could you share the ganesha-gfapi logs?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I have two volumes, one is mounted using libgfapi for ovirt mount, the other one is exported via NFS-Ganesha for VMWare which is the one im testing now.<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Sunday, March 19, 2017 8:02:19 AM
</font><br>
</span></p>
<span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
</blockquote>
<span>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Kindly, check the attached new log file, i dont know if it's helpful or not but, i couldn't find the log with the name you just described.<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</span>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Saturday, March 18, 2017 6:10:40 PM
</font><br>
</span></p>
<span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse mount logs? It should be right under /var/log/glusterfs/ directory</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
named after the mount point name, only hyphenated.<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hello Krutika,<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Kindly, check the attached logs.<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font></p>
<div>
<div class="m_-6836490455585458468gmail-m_1183689012231501332h5"><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Saturday, March 18, 2017 3:29:03 PM</font><br>
<font color="#000000"><b>To:</b></font><font color="#000000"> Mahdi Adnan</font><br>
<font color="#000000"><b>Cc:</b></font><font color="#000000"> </font><font color="#000000"><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a></font><br>
<font color="#000000"><b>Subject:</b></font><font color="#000000"> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</font>
</div>
</div>
<p></p>
<div>
<div class="m_-6836490455585458468gmail-m_1183689012231501332h5">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hi Mahdi,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Could you attach mount, brick and rebalance logs?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</div>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<div>
<div class="m_-6836490455585458468gmail-m_1183689012231501332h5">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hi,<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a volume contains few VMs.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
After the completion of rebalance, i have rebooted the VMs, some of ran just fine, and others just crashed.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Windows boot to recovery mode and Linux throw xfs errors and does not boot.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I ran the test again and it happened just as the first one, but i have noticed only VMs doing disk IOs are affected by this bug.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
The VMs in power off mode started fine and even md5 of the disk file did not change after the rebalance.<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
anyone else can confirm this ?<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Volume info:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Volume Name: vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Type: Distributed-Replicate</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Volume ID: 02328d46-a285-4533-aa3a-fb9bfe<wbr>b688bf</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Status: Started</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Snapshot Count: 0</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Number of Bricks: 22 x 2 = 44</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Transport-type: tcp</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Bricks:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick1: gluster01:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick2: gluster03:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick3: gluster02:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick4: gluster04:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick5: gluster01:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick6: gluster03:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick7: gluster02:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick8: gluster04:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick9: gluster01:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick10: gluster03:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick11: gluster02:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick12: gluster04:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick13: gluster01:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick14: gluster03:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick15: gluster02:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick16: gluster04:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick17: gluster01:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick18: gluster03:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick19: gluster02:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick20: gluster04:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick21: gluster01:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick22: gluster03:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick23: gluster02:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick24: gluster04:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick25: gluster01:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick26: gluster03:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick27: gluster02:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick28: gluster04:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick29: gluster01:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick30: gluster03:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick31: gluster02:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick32: gluster04:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick33: gluster01:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick34: gluster03:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick35: gluster02:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick36: gluster04:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick37: gluster01:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick38: gluster03:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick39: gluster02:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick40: gluster04:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick41: gluster01:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick42: gluster03:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick43: gluster02:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick44: gluster04:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Options Reconfigured:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.server-quorum-type: server</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
nfs.disable: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.readdir-ahead: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
transport.address-family: inet</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.quick-read: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.read-ahead: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.io-cache: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.stat-prefetch: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.eager-lock: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
network.remote-dio: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
features.shard: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.data-self-heal-algorit<wbr>hm: full</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
features.cache-invalidation: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
ganesha.enable: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
features.shard-block-size: 256MB</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
client.event-threads: 2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
server.event-threads: 2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.favorite-child-policy: size</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
storage.build-pgfid: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
network.ping-timeout: 5</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.enable-shared-storage: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
nfs-ganesha: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.server-quorum-ratio: 51%<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Adding bricks:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 gluster04:/mnt/disk11/vmware2<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
starting fix layout:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
gluster volume rebalance vmware2 fix-layout start<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Starting rebalance:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
gluster volume rebalance vmware2 start<br>
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font><br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
<br>
</p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-6836490455585458468gmail_signature"><div dir="ltr">Pranith<br></div></div>
</font></span></div></div>
</blockquote></div><br></div>