<div dir="ltr"><div><div><div><div><div><div><div>I looked at the logs.<br><br></div>From the time the new graph (since the add-brick command you shared where bricks 41 through 44 are added) is switched to (line 3011 onwards in nfs-gfapi.log), I see the following kinds of errors:<br><br></div>1. Lookups to a bunch of files failed with ENOENT on both replicas which protocol/client converts to ESTALE. I am guessing these entries got migrated to<br></div>other subvolumes leading to 'No such file or directory' errors.<br></div>DHT and thereafter shard get the same error code and log the following:<br><br> 0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] [dht-helper.c:1198:dht_migration_complete_check_task] 17-vmware2-dht: <gfid:a68ce411-e381-46a3-93cd-d2af6a7c3532>: failed to lookup the file on vmware2-dht [Stale file handle] <br> 1 [2017-03-17 14:04:26.353528] E [MSGID: 133014] [shard.c:1253:shard_common_stat_cbk] 17-vmware2-shard: stat failed: a68ce411-e381-46a3-93cd-d2af6a7c3532 [Stale file handle]<br><br></div>which is fine.<br><br></div>2. The other kind are from AFR logging of possible split-brain which I suppose are harmless too.<br>[2017-03-17 14:23:36.968883] W [MSGID: 108008] [afr-read-txn.c:228:afr_read_txn] 17-vmware2-replicate-13: Unreadable subvolume -1 found with event generation 2 for gfid 74d49288-8452-40d4-893e-ff4672557ff9. (Possible split-brain)<br><br></div><div>Since you are saying the bug is hit only on VMs that are undergoing IO while rebalance is running (as opposed to those that remained powered off),<br></div><div>rebalance + IO could be causing some issues.<br><br></div><div>CC'ing DHT devs<br><br></div><div>Raghavendra/Nithya/Susant,<br><br></div><div>Could you take a look?<br><br></div><div>-Krutika<br></div><div><div><div><br><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan <span dir="ltr"><<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_3610917189403001996divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Thank you for your email mate.</p>
<p><br>
</p>
<p>Yes, im aware of this but, to save costs i chose replica 2, this cluster is all flash.</p>
<p>In version 3.7.x i had issues with ping timeout, if one hosts went down for few seconds the whole cluster hangs and become unavailable, to avoid this i adjusted the ping timeout to 5 seconds.</p>
<p>As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE or gfapi) im stuck with NFS for this volume.</p>
<p>The other volume is mounted using gfapi in oVirt cluster.</p><span class="">
<p> </p>
<p><br>
</p>
<div id="m_3610917189403001996Signature"><br>
<div class="m_3610917189403001996ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_3610917189403001996divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Sunday, March 19, 2017 2:01:49 PM<div><div class="h5"><br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</div></div></font>
<div> </div>
</div><div><div class="h5">
<div>
<div dir="ltr">
<div>
<div>
<div>
<div>While I'm still going through the logs, just wanted to point out a couple of things:<br>
<br>
</div>
1. It is recommended that you use 3-way replication (replica count 3) for VM store use case<br>
</div>
2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.<br>
<br>
</div>
<div>Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?<br>
</div>
<div><br>
</div>
Will get back with anything else I might find or more questions if I have any.<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_3610917189403001996m_-1210732860838431600divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Thanks mate,</p>
<p>Kindly, check the attachment.</p>
<span>
<p><br>
</p>
<div id="m_3610917189403001996m_-1210732860838431600Signature"><br>
<div class="m_3610917189403001996m_-1210732860838431600ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_3610917189403001996m_-1210732860838431600divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Sunday, March 19, 2017 10:00:22 AM
<div>
<div class="m_3610917189403001996h5"><br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_3610917189403001996h5">
<div>
<div dir="ltr">
<div>In that case could you share the ganesha-gfapi logs?<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>I have two volumes, one is mounted using <span>libgfapi</span> for ovirt mount, the other one is exported via NFS-Ganesha for VMWare which is the one im testing now.</p>
<span>
<p><br>
</p>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625Signature"><br>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625divRplyFwdMsg" dir="ltr"><font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Sunday, March 19, 2017 8:02:19 AM
<div>
<div class="m_3610917189403001996m_-1210732860838431600h5"><br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_3610917189403001996m_-1210732860838431600h5">
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Kindly, check the attached new log file, i dont know if it's helpful or not but, i couldn't find the log with the name you just described.</p>
</div>
</div>
</blockquote>
<div>No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?<br>
</div>
<div> <br>
</div>
<div>-Krutika<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<span>
<p><br>
</p>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687Signature">
<br>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687ecxmoz-signature">
-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Saturday, March 18, 2017 6:10:40 PM
<div>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625h5"><br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</div>
</div>
</font>
<div> </div>
</div>
<div>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625h5">
<div>
<div dir="ltr">
<div>mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse mount logs? It should be right under /var/log/glusterfs/ directory<br>
</div>
<div>named after the mount point name, only hyphenated.<br>
</div>
<div><br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p>Hello <span>Krutika,</span></p>
<p><span><br>
</span></p>
<p>Kindly, check the attached logs.</p>
<span>
<p><br>
</p>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024Signature">
<br>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024ecxmoz-signature">
-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</span></div>
<hr style="display:inline-block;width:98%">
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024divRplyFwdMsg" dir="ltr">
<font style="font-size:11pt" color="#000000" face="Calibri, sans-serif"><b>From:</b> Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>><br>
<b>Sent:</b> Saturday, March 18, 2017 3:29:03 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</font>
<div> </div>
</div>
<div>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687h5">
<div>
<div dir="ltr">
<div>
<div>Hi Mahdi,<br>
<br>
</div>
Could you attach mount, brick and rebalance logs?<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024m_-1991684123538814306divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a volume contains few VMs.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
After the completion of rebalance, i have rebooted the VMs, some of ran just fine, and others just crashed.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Windows boot to recovery mode and Linux throw xfs errors and does not boot.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
I ran the test again and it happened just as the first one, but i have noticed only VMs doing disk IOs are affected by this bug.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
The VMs in power off mode started fine and even md5 of the disk file did not change after the rebalance.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
anyone else can confirm this ?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Volume info:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Volume Name: vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Type: Distributed-Replicate</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Volume ID: 02328d46-a285-4533-aa3a-fb9bfe<wbr>b688bf</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Status: Started</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Snapshot Count: 0</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Number of Bricks: 22 x 2 = 44</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Transport-type: tcp</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Bricks:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick1: gluster01:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick2: gluster03:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick3: gluster02:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick4: gluster04:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick5: gluster01:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick6: gluster03:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick7: gluster02:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick8: gluster04:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick9: gluster01:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick10: gluster03:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick11: gluster02:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick12: gluster04:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick13: gluster01:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick14: gluster03:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick15: gluster02:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick16: gluster04:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick17: gluster01:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick18: gluster03:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick19: gluster02:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick20: gluster04:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick21: gluster01:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick22: gluster03:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick23: gluster02:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick24: gluster04:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick25: gluster01:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick26: gluster03:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick27: gluster02:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick28: gluster04:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick29: gluster01:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick30: gluster03:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick31: gluster02:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick32: gluster04:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick33: gluster01:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick34: gluster03:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick35: gluster02:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick36: gluster04:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick37: gluster01:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick38: gluster03:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick39: gluster02:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick40: gluster04:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick41: gluster01:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick42: gluster03:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick43: gluster02:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Brick44: gluster04:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Options Reconfigured:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">cluster.server-quorum-type: server</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">nfs.disable: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">performance.readdir-ahead: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">transport.address-family: inet</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">performance.quick-read: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">performance.read-ahead: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">performance.io-cache: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">performance.stat-prefetch: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">cluster.eager-lock: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">network.remote-dio: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">features.shard: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">cluster.data-self-heal-algorit<wbr>hm: full</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">features.cache-invalidation: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">ganesha.enable: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">features.shard-block-size: 256MB</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">client.event-threads: 2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">server.event-threads: 2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">cluster.favorite-child-policy: size</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">storage.build-pgfid: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">network.ping-timeout: 5</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">cluster.enable-shared-storage: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">nfs-ganesha: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">cluster.server-quorum-ratio: 51%</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Adding bricks:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 gluster04:/mnt/disk11/vmware2<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">starting fix layout:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">gluster volume rebalance vmware2 fix-layout start<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px"><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">Starting rebalance:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">gluster volume rebalance vmware2 start</div>
<span class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024HOEnZb"><font color="#888888"><br>
<p></p>
<p><br>
</p>
<div id="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024m_-1991684123538814306Signature">
<br>
<div class="m_3610917189403001996m_-1210732860838431600m_9180764191956123625m_1062672590910331687m_-3705862106183925024m_-1991684123538814306ecxmoz-signature">
-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</font></span></div>
</div>
<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</blockquote></div><br></div>