<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;" dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Do you guys have any update regarding this issue ?</p>
<p><br>
</p>
<div id="Signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font color="#3366ff"><br>
<br>
</font><font color="#3366ff"></font></div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Krutika Dhananjay <kdhananj@redhat.com><br>
<b>Sent:</b> Tuesday, March 21, 2017 3:02:55 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> Nithya Balachandran; Gowdappa, Raghavendra; Susant Palai; gluster-users@gluster.org List<br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>Hi,<br>
<br>
</div>
So it looks like Satheesaran managed to recreate this issue. We will be seeking his help in debugging this. It will be easier that way.<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Mar 21, 2017 at 1:35 PM, Mahdi Adnan <span dir="ltr">
<<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hello and thank you for your email. <br>
Actually no, i didn't check the gfid of the vms. <br>
If this will help, i can setup a new test cluster and get all the data you need.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Get <a href="https://aka.ms/ghei36" target="_blank">Outlook for Android</a></p>
<br>
<p></p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
From: Nithya Balachandran<br>
Sent: Monday, March 20, 20:57<span class=""><br>
Subject: Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption<br>
</span>To: Krutika Dhananjay<br>
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai, <a href="mailto:gluster-users@gluster.org" target="_blank">
gluster-users@gluster.org</a> List<br>
</p>
<div>
<div class="h5">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hi, <br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Do you know the GFIDs of the VM images which were corrupted?<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Regards,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Nithya<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On 20 March 2017 at 20:37, Krutika Dhananjay <<a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a>> wrote:</p>
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I looked at the logs.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
>From the time the new graph (since the add-brick command you shared where bricks 41 through 44 are added) is switched to (line 3011 onwards in nfs-gfapi.log), I see the following kinds of errors:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
1. Lookups to a bunch of files failed with ENOENT on both replicas which protocol/client converts to ESTALE. I am guessing these entries got migrated to</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
other subvolumes leading to 'No such file or directory' errors.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
DHT and thereafter shard get the same error code and log the following:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
0 [2017-03-17 14:04:26.353444] E [MSGID: 109040] [dht-helper.c:1198:dht_<wbr>migration_complete_check_task] 17-vmware2-dht: <gfid:a68ce411-e381-46a3-93cd-<wbr>d2af6a7c3532>: failed to lookup the file on vmware2-dht [Stale file handle] <wbr> <wbr> <wbr> <wbr>
<br>
1 [2017-03-17 14:04:26.353528] E [MSGID: 133014] [shard.c:1253:shard_common_<wbr>stat_cbk] 17-vmware2-shard: stat failed: a68ce411-e381-46a3-93cd-<wbr>d2af6a7c3532 [Stale file handle]</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
which is fine.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
2. The other kind are from AFR logging of possible split-brain which I suppose are harmless too.<br>
[2017-03-17 14:23:36.968883] W [MSGID: 108008] [afr-read-txn.c:228:afr_read_<wbr>txn] 17-vmware2-replicate-13: Unreadable subvolume -1 found with event generation 2 for gfid 74d49288-8452-40d4-893e-<wbr>ff4672557ff9. (Possible split-brain)</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Since you are saying the bug is hit only on VMs that are undergoing IO while rebalance is running (as opposed to those that remained powered off),</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
rebalance + IO could be causing some issues.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
CC'ing DHT devs</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Raghavendra/Nithya/Susant,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Could you take a look?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</blockquote>
</div>
</div>
<blockquote type="cite">
<blockquote type="cite">
<div>
<div class="h5">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Thank you for your email mate.<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Yes, im aware of this but, to save costs i chose replica 2, this cluster is all flash.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
In version 3.7.x i had issues with ping timeout, if one hosts went down for few seconds the whole cluster hangs and become unavailable, to avoid this i adjusted the ping timeout to 5 seconds.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
As for choosing Ganesha over gfapi, VMWare does not support Gluster (FUSE or gfapi) im stuck with NFS for this volume.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
The other volume is mounted using gfapi in oVirt cluster.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</div>
</div>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Sunday, March 19, 2017 2:01:49 PM
</font><br>
</span></p>
<span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
While I'm still going through the logs, just wanted to point out a couple of things:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
1. It is recommended that you use 3-way replication (replica count 3) for VM store use case</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
2. network.ping-timeout at 5 seconds is way too low. Please change it to 30.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Is there any specific reason for using NFS-Ganesha over gfapi/FUSE?<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Will get back with anything else I might find or more questions if I have any.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Thanks mate,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Kindly, check the attachment.<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Sunday, March 19, 2017 10:00:22 AM
</font><br>
</span></p>
<span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
In that case could you share the ganesha-gfapi logs?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I have two volumes, one is mounted using libgfapi for ovirt mount, the other one is exported via NFS-Ganesha for VMWare which is the one im testing now.<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Sunday, March 19, 2017 8:02:19 AM
</font><br>
</span></p>
<span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
</blockquote>
<span class="">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Kindly, check the attached new log file, i dont know if it's helpful or not but, i couldn't find the log with the name you just described.<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</span>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Saturday, March 18, 2017 6:10:40 PM
</font><br>
</span></p>
<span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse mount logs? It should be right under /var/log/glusterfs/ directory</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
named after the mount point name, only hyphenated.<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hello Krutika,<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Kindly, check the attached logs.<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<b><font color="#000000">From:</font></b><font color="#000000"> Krutika Dhananjay <</font><font color="#000000"><a href="mailto:kdhananj@redhat.com" target="_blank">kdhananj@redhat.com</a></font><font color="#000000">></font></p>
<div>
<div class="h5"><br>
<font color="#000000"><b>Sent:</b></font><font color="#000000"> Saturday, March 18, 2017 3:29:03 PM</font><br>
<font color="#000000"><b>To:</b></font><font color="#000000"> Mahdi Adnan</font><br>
<font color="#000000"><b>Cc:</b></font><font color="#000000"> </font><font color="#000000"><a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a></font><br>
<font color="#000000"><b>Subject:</b></font><font color="#000000"> Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption</font>
</div>
</div>
<p></p>
<div>
<div class="h5">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hi Mahdi,</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Could you attach mount, brick and rebalance logs?</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-Krutika<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan <<a href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>> wrote:</p>
</div>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<div>
<div class="h5">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Hi,<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure in a volume contains few VMs.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
After the completion of rebalance, i have rebooted the VMs, some of ran just fine, and others just crashed.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Windows boot to recovery mode and Linux throw xfs errors and does not boot.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
I ran the test again and it happened just as the first one, but i have noticed only VMs doing disk IOs are affected by this bug.</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
The VMs in power off mode started fine and even md5 of the disk file did not change after the rebalance.<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
anyone else can confirm this ?<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Volume info:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Volume Name: vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Type: Distributed-Replicate</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Volume ID: 02328d46-a285-4533-aa3a-<wbr>fb9bfeb688bf</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Status: Started</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Snapshot Count: 0</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Number of Bricks: 22 x 2 = 44</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Transport-type: tcp</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Bricks:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick1: gluster01:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick2: gluster03:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick3: gluster02:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick4: gluster04:/mnt/disk1/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick5: gluster01:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick6: gluster03:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick7: gluster02:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick8: gluster04:/mnt/disk2/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick9: gluster01:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick10: gluster03:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick11: gluster02:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick12: gluster04:/mnt/disk3/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick13: gluster01:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick14: gluster03:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick15: gluster02:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick16: gluster04:/mnt/disk4/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick17: gluster01:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick18: gluster03:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick19: gluster02:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick20: gluster04:/mnt/disk5/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick21: gluster01:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick22: gluster03:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick23: gluster02:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick24: gluster04:/mnt/disk6/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick25: gluster01:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick26: gluster03:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick27: gluster02:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick28: gluster04:/mnt/disk7/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick29: gluster01:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick30: gluster03:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick31: gluster02:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick32: gluster04:/mnt/disk8/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick33: gluster01:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick34: gluster03:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick35: gluster02:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick36: gluster04:/mnt/disk9/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick37: gluster01:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick38: gluster03:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick39: gluster02:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick40: gluster04:/mnt/disk10/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick41: gluster01:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick42: gluster03:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick43: gluster02:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Brick44: gluster04:/mnt/disk11/vmware2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Options Reconfigured:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.server-quorum-type: server</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
nfs.disable: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.readdir-ahead: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
transport.address-family: inet</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.quick-read: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.read-ahead: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.io-cache: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
performance.stat-prefetch: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.eager-lock: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
network.remote-dio: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
features.shard: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.data-self-heal-<wbr>algorithm: full</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
features.cache-invalidation: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
ganesha.enable: on</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
features.shard-block-size: 256MB</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
client.event-threads: 2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
server.event-threads: 2</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.favorite-child-policy: size</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
storage.build-pgfid: off</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
network.ping-timeout: 5</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.enable-shared-storage: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
nfs-ganesha: enable</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
cluster.server-quorum-ratio: 51%<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Adding bricks:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2 gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2 gluster04:/mnt/disk11/vmware2<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
starting fix layout:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
gluster volume rebalance vmware2 fix-layout start<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
Starting rebalance:</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
gluster volume rebalance vmware2 start<br>
<br>
<br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
-- </p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<font color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi A. Mahdi</b></font></font><br>
</p>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
</blockquote>
<blockquote type="cite">
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
</p>
</blockquote>
<p dir="auto" style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">
<br>
<br>
</p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</body>
</html>