<div dir="ltr"><div><div>Hi Mahdi,<br><br></div>Could you attach mount, brick and rebalance logs?<br><br></div>-Krutika<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan <span dir="ltr">&lt;<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>&gt;</span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">




<div dir="ltr">
<div id="m_-1991684123538814306divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif" dir="ltr">
<p></p>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Hi,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a volume contains few VMs.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
After the completion of rebalance, i have rebooted the VMs, some of ran just fine, and others just crashed.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Windows boot to recovery mode and Linux throw xfs errors and does not boot.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
I ran the test again and it happened just as the first one, but i have noticed only VMs doing disk IOs are affected by this bug.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
The VMs in power off mode started fine and even md5 of the disk file did not change after the rebalance.</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
anyone else can confirm this ?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
Volume info:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px;margin-top:0px;margin-bottom:0px">
 </div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Volume Name: vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Type: Distributed-Replicate</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Volume ID: 02328d46-a285-4533-aa3a-<wbr>fb9bfeb688bf</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Status: Started</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Snapshot Count: 0</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Number of Bricks: 22 x 2 = 44</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Transport-type: tcp</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Bricks:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick1: gluster01:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick2: gluster03:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick3: gluster02:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick4: gluster04:/mnt/disk1/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick5: gluster01:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick6: gluster03:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick7: gluster02:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick8: gluster04:/mnt/disk2/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick9: gluster01:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick10: gluster03:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick11: gluster02:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick12: gluster04:/mnt/disk3/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick13: gluster01:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick14: gluster03:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick15: gluster02:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick16: gluster04:/mnt/disk4/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick17: gluster01:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick18: gluster03:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick19: gluster02:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick20: gluster04:/mnt/disk5/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick21: gluster01:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick22: gluster03:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick23: gluster02:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick24: gluster04:/mnt/disk6/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick25: gluster01:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick26: gluster03:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick27: gluster02:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick28: gluster04:/mnt/disk7/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick29: gluster01:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick30: gluster03:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick31: gluster02:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick32: gluster04:/mnt/disk8/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick33: gluster01:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick34: gluster03:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick35: gluster02:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick36: gluster04:/mnt/disk9/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick37: gluster01:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick38: gluster03:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick39: gluster02:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick40: gluster04:/mnt/disk10/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick41: gluster01:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick42: gluster03:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick43: gluster02:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Brick44: gluster04:/mnt/disk11/vmware2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Options Reconfigured:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
cluster.server-quorum-type: server</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
nfs.disable: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
performance.readdir-ahead: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
transport.address-family: inet</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
performance.quick-read: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
performance.read-ahead: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
performance.io-cache: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
performance.stat-prefetch: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
cluster.eager-lock: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
network.remote-dio: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
features.shard: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
cluster.data-self-heal-<wbr>algorithm: full</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
features.cache-invalidation: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
ganesha.enable: on</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
features.shard-block-size: 256MB</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
client.event-threads: 2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
server.event-threads: 2</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
cluster.favorite-child-policy: size</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
storage.build-pgfid: off</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
network.ping-timeout: 5</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
cluster.enable-shared-storage: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
nfs-ganesha: enable</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
cluster.server-quorum-ratio: 51%</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Adding bricks:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 gluster04:/mnt/disk11/vmware2<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
starting fix layout:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
gluster volume rebalance vmware2 fix-layout start<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
Starting rebalance:</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif;font-size:16px">
gluster volume rebalance vmware2  start</div><span class="HOEnZb"><font color="#888888">
<br>
<p></p>
<p><br>
</p>
<div id="m_-1991684123538814306Signature"><br>
<div class="m_-1991684123538814306ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</font></span></div>
</div>

<br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br></blockquote></div><br></div>