<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<p>Based on what I know of the workflow, there is no update. There
is no bug report in bugzilla so there are no patches in review for
it.<br>
</p>
<br>
<div class="moz-cite-prefix">On 03/27/2017 10:59 AM, Mahdi Adnan
wrote:<br>
</div>
<blockquote
cite="mid:BY2PR01MB34529799C5006591B242FEDFF330@BY2PR01MB345.prod.exchangelabs.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper"
style="font-size:12pt;color:#000000;font-family:Calibri,Arial,Helvetica,sans-serif;"
dir="ltr">
<p>Hi,</p>
<p><br>
</p>
<p>Do you guys have any update regarding this issue ?</p>
<p><br>
</p>
<div id="Signature"><br>
<div class="ecxmoz-signature">-- <br>
<br>
<font color="#3366ff"><font color="#000000">Respectfully<b><br>
</b><b>Mahdi A. Mahdi</b></font></font><font
color="#3366ff"><br>
<br>
</font></div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
Krutika Dhananjay <a class="moz-txt-link-rfc2396E" href="mailto:kdhananj@redhat.com"><kdhananj@redhat.com></a><br>
<b>Sent:</b> Tuesday, March 21, 2017 3:02:55 PM<br>
<b>To:</b> Mahdi Adnan<br>
<b>Cc:</b> Nithya Balachandran; Gowdappa, Raghavendra; Susant
Palai; <a class="moz-txt-link-abbreviated" href="mailto:gluster-users@gluster.org">gluster-users@gluster.org</a> List<br>
<b>Subject:</b> Re: [Gluster-users] Gluster 3.8.10 rebalance
VMs corruption</font>
<div> </div>
</div>
<div>
<div dir="ltr">
<div>
<div>Hi,<br>
<br>
</div>
So it looks like Satheesaran managed to recreate this issue.
We will be seeking his help in debugging this. It will be
easier that way.<br>
<br>
</div>
-Krutika<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Mar 21, 2017 at 1:35 PM,
Mahdi Adnan <span dir="ltr">
<<a moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com" target="_blank">mahdi.adnan@outlook.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Hello
and thank you for your email. <br>
Actually no, i didn't check the gfid of the vms. <br>
If this will help, i can setup a new test cluster and
get all the data you need.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"></p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Get
<a moz-do-not-send="true" href="https://aka.ms/ghei36"
target="_blank">Outlook for Android</a></p>
<br>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">From:
Nithya Balachandran<br>
Sent: Monday, March 20, 20:57<span class=""><br>
Subject: Re: [Gluster-users] Gluster 3.8.10
rebalance VMs corruption<br>
</span>To: Krutika Dhananjay<br>
Cc: Mahdi Adnan, Gowdappa, Raghavendra, Susant Palai,
<a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">
gluster-users@gluster.org</a> List<br>
</p>
<div>
<div class="h5">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Hi,
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Do
you know the GFIDs of the VM images which were
corrupted?<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Regards,</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Nithya<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
20 March 2017 at 20:37, Krutika Dhananjay <<a
moz-do-not-send="true"
href="mailto:kdhananj@redhat.com"
target="_blank">kdhananj@redhat.com</a>>
wrote:</p>
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">I
looked at the logs.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">From
the time the new graph (since the add-brick
command you shared where bricks 41 through 44
are added) is switched to (line 3011 onwards in
nfs-gfapi.log), I see the following kinds of
errors:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">1.
Lookups to a bunch of files failed with ENOENT
on both replicas which protocol/client converts
to ESTALE. I am guessing these entries got
migrated to</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">other
subvolumes leading to 'No such file or
directory' errors.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">DHT
and thereafter shard get the same error code and
log the following:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> 0
[2017-03-17 14:04:26.353444] E [MSGID: 109040]
[dht-helper.c:1198:dht_<wbr>migration_complete_check_task]
17-vmware2-dht:
<gfid:a68ce411-e381-46a3-93cd-<wbr>d2af6a7c3532>:
failed to lookup the file on vmware2-dht
[Stale file handle] <wbr> <wbr> <wbr> <wbr>
<br>
1 [2017-03-17 14:04:26.353528] E [MSGID:
133014] [shard.c:1253:shard_common_<wbr>stat_cbk]
17-vmware2-shard: stat failed:
a68ce411-e381-46a3-93cd-<wbr>d2af6a7c3532 [Stale
file handle]</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">which
is fine.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">2.
The other kind are from AFR logging of possible
split-brain which I suppose are harmless too.<br>
[2017-03-17 14:23:36.968883] W [MSGID: 108008]
[afr-read-txn.c:228:afr_read_<wbr>txn]
17-vmware2-replicate-13: Unreadable subvolume -1
found with event generation 2 for gfid
74d49288-8452-40d4-893e-<wbr>ff4672557ff9.
(Possible split-brain)</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Since
you are saying the bug is hit only on VMs that
are undergoing IO while rebalance is running (as
opposed to those that remained powered off),</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">rebalance
+ IO could be causing some issues.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">CC'ing
DHT devs</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Raghavendra/Nithya/Susant,</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Could
you take a look?</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">-Krutika<br>
<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
Sun, Mar 19, 2017 at 4:55 PM, Mahdi Adnan <<a
moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com"
target="_blank">mahdi.adnan@outlook.com</a>>
wrote:</p>
</blockquote>
</div>
</div>
<blockquote type="cite">
<blockquote type="cite">
<div>
<div class="h5">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Thank
you for your email mate.<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Yes,
im aware of this but, to save costs i chose
replica 2, this cluster is all flash.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">In
version 3.7.x i had issues with ping timeout,
if one hosts went down for few seconds the
whole cluster hangs and become unavailable, to
avoid this i adjusted the ping timeout to 5
seconds.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">As
for choosing Ganesha over gfapi, VMWare does
not support Gluster (FUSE or gfapi) im stuck
with NFS for this volume.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">The
other volume is mounted using gfapi in oVirt
cluster.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> <br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">--
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><font
color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi
A. Mahdi</b></font></font></p>
</div>
</div>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b><font
color="#000000">From:</font></b><font
color="#000000"> Krutika Dhananjay <</font><font
color="#000000"><a moz-do-not-send="true"
href="mailto:kdhananj@redhat.com"
target="_blank">kdhananj@redhat.com</a></font><font
color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font
color="#000000"> Sunday, March 19, 2017
2:01:49 PM
</font><br>
</span></p>
<span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b>To:</b>
Mahdi Adnan<br>
<b>Cc:</b> <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster
3.8.10 rebalance VMs corruption</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> </p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">While
I'm still going through the logs, just wanted to
point out a couple of things:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">1.
It is recommended that you use 3-way replication
(replica count 3) for VM store use case</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">2.
network.ping-timeout at 5 seconds is way too
low. Please change it to 30.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Is
there any specific reason for using NFS-Ganesha
over gfapi/FUSE?<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Will
get back with anything else I might find or more
questions if I have any.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">-Krutika<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
Sun, Mar 19, 2017 at 2:36 PM, Mahdi Adnan <<a
moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com"
target="_blank">mahdi.adnan@outlook.com</a>>
wrote:</p>
</span></blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Thanks
mate,</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Kindly,
check the attachment.<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">--
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><font
color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi
A. Mahdi</b></font></font></p>
</span>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b><font
color="#000000">From:</font></b><font
color="#000000"> Krutika Dhananjay <</font><font
color="#000000"><a moz-do-not-send="true"
href="mailto:kdhananj@redhat.com"
target="_blank">kdhananj@redhat.com</a></font><font
color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font
color="#000000"> Sunday, March 19, 2017
10:00:22 AM
</font><br>
</span></p>
<span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b>To:</b>
Mahdi Adnan<br>
<b>Cc:</b> <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster
3.8.10 rebalance VMs corruption</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> </p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">In
that case could you share the ganesha-gfapi
logs?</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">-Krutika<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
Sun, Mar 19, 2017 at 12:13 PM, Mahdi Adnan
<<a moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com"
target="_blank">mahdi.adnan@outlook.com</a>>
wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">I
have two volumes, one is mounted using
libgfapi for ovirt mount, the other one is
exported via NFS-Ganesha for VMWare which is
the one im testing now.<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">--
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><font
color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font color="#3366ff"><b>Mahdi
A. Mahdi</b></font></font></p>
</span>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b><font
color="#000000">From:</font></b><font
color="#000000"> Krutika Dhananjay <</font><font
color="#000000"><a moz-do-not-send="true"
href="mailto:kdhananj@redhat.com"
target="_blank">kdhananj@redhat.com</a></font><font
color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font
color="#000000"> Sunday, March 19, 2017
8:02:19 AM
</font><br>
</span></p>
<span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b>To:</b>
Mahdi Adnan<br>
<b>Cc:</b> <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users] Gluster
3.8.10 rebalance VMs corruption</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> <br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan
<<a moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com"
target="_blank">mahdi.adnan@outlook.com</a>>
wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
</blockquote>
<span class="">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Kindly,
check the attached new log file, i dont
know if it's helpful or not but, i
couldn't find the log with the name you
just described.<br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
No. Are you using FUSE or libgfapi for
accessing the volume? Or is it NFS?</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> </p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">-Krutika</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</span>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"></p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">--
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><font
color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font
color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b><font
color="#000000">From:</font></b><font
color="#000000"> Krutika Dhananjay <</font><font
color="#000000"><a moz-do-not-send="true"
href="mailto:kdhananj@redhat.com"
target="_blank">kdhananj@redhat.com</a></font><font
color="#000000">></font><span class=""><br>
<font color="#000000"><b>Sent:</b></font><font
color="#000000"> Saturday, March 18,
2017 6:10:40 PM
</font><br>
</span></p>
<span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b>To:</b>
Mahdi Adnan<br>
<b>Cc:</b> <a moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">gluster-users@gluster.org</a><br>
<b>Subject:</b> Re: [Gluster-users]
Gluster 3.8.10 rebalance VMs corruption</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> </p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">mnt-disk11-vmware2.log
seems like a brick log. Could you attach
the fuse mount logs? It should be right
under /var/log/glusterfs/ directory</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">named
after the mount point name, only
hyphenated.<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">-Krutika<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan
<<a moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com"
target="_blank">mahdi.adnan@outlook.com</a>>
wrote:</p>
</span></blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite"><span class="">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Hello Krutika,<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Kindly,
check the attached logs.<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">--
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><font
color="#000000"><font color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font
color="#3366ff"><b>Mahdi A. Mahdi</b></font></font></p>
</span>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><b><font
color="#000000">From:</font></b><font
color="#000000"> Krutika Dhananjay <</font><font
color="#000000"><a
moz-do-not-send="true"
href="mailto:kdhananj@redhat.com"
target="_blank">kdhananj@redhat.com</a></font><font
color="#000000">></font></p>
<div>
<div class="h5"><br>
<font color="#000000"><b>Sent:</b></font><font
color="#000000"> Saturday, March 18,
2017 3:29:03 PM</font><br>
<font color="#000000"><b>To:</b></font><font
color="#000000"> Mahdi Adnan</font><br>
<font color="#000000"><b>Cc:</b></font><font
color="#000000"> </font><font
color="#000000"><a
moz-do-not-send="true"
href="mailto:gluster-users@gluster.org"
target="_blank">gluster-users@gluster.org</a></font><br>
<font color="#000000"><b>Subject:</b></font><font
color="#000000"> Re: [Gluster-users]
Gluster 3.8.10 rebalance VMs
corruption</font>
</div>
</div>
<div>
<div class="h5">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> </p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Hi
Mahdi,</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Could
you attach mount, brick and rebalance
logs?</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">-Krutika<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">On
Sat, Mar 18, 2017 at 12:14 AM, Mahdi
Adnan <<a moz-do-not-send="true"
href="mailto:mahdi.adnan@outlook.com"
target="_blank">mahdi.adnan@outlook.com</a>>
wrote:</p>
</div>
</div>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<div>
<div class="h5">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Hi,<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">I
have upgraded to Gluster 3.8.10
today and ran the add-brick
procedure in a volume contains few
VMs.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">After
the completion of rebalance, i have
rebooted the VMs, some of ran just
fine, and others just crashed.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Windows
boot to recovery mode and Linux
throw xfs errors and does not boot.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">I
ran the test again and it happened
just as the first one, but i have
noticed only VMs doing disk IOs
are affected by this bug.</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">The
VMs in power off mode started fine
and even md5 of the disk file did
not change after the rebalance.<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">anyone
else can confirm this ?<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Volume
info:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"> </p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Volume
Name: vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Type:
Distributed-Replicate</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Volume
ID: 02328d46-a285-4533-aa3a-<wbr>fb9bfeb688bf</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Status:
Started</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Snapshot
Count: 0</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Number
of Bricks: 22 x 2 = 44</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Transport-type:
tcp</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Bricks:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick1:
gluster01:/mnt/disk1/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick2:
gluster03:/mnt/disk1/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick3:
gluster02:/mnt/disk1/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick4:
gluster04:/mnt/disk1/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick5:
gluster01:/mnt/disk2/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick6:
gluster03:/mnt/disk2/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick7:
gluster02:/mnt/disk2/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick8:
gluster04:/mnt/disk2/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick9:
gluster01:/mnt/disk3/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick10:
gluster03:/mnt/disk3/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick11:
gluster02:/mnt/disk3/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick12:
gluster04:/mnt/disk3/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick13:
gluster01:/mnt/disk4/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick14:
gluster03:/mnt/disk4/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick15:
gluster02:/mnt/disk4/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick16:
gluster04:/mnt/disk4/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick17:
gluster01:/mnt/disk5/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick18:
gluster03:/mnt/disk5/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick19:
gluster02:/mnt/disk5/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick20:
gluster04:/mnt/disk5/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick21:
gluster01:/mnt/disk6/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick22:
gluster03:/mnt/disk6/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick23:
gluster02:/mnt/disk6/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick24:
gluster04:/mnt/disk6/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick25:
gluster01:/mnt/disk7/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick26:
gluster03:/mnt/disk7/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick27:
gluster02:/mnt/disk7/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick28:
gluster04:/mnt/disk7/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick29:
gluster01:/mnt/disk8/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick30:
gluster03:/mnt/disk8/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick31:
gluster02:/mnt/disk8/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick32:
gluster04:/mnt/disk8/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick33:
gluster01:/mnt/disk9/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick34:
gluster03:/mnt/disk9/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick35:
gluster02:/mnt/disk9/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick36:
gluster04:/mnt/disk9/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick37:
gluster01:/mnt/disk10/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick38:
gluster03:/mnt/disk10/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick39:
gluster02:/mnt/disk10/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick40:
gluster04:/mnt/disk10/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick41:
gluster01:/mnt/disk11/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick42:
gluster03:/mnt/disk11/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick43:
gluster02:/mnt/disk11/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Brick44:
gluster04:/mnt/disk11/vmware2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Options
Reconfigured:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">cluster.server-quorum-type:
server</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">nfs.disable:
on</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">performance.readdir-ahead:
on</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">transport.address-family:
inet</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">performance.quick-read:
off</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">performance.read-ahead:
off</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">performance.io-cache:
off</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">performance.stat-prefetch:
off</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">cluster.eager-lock:
enable</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">network.remote-dio:
enable</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">features.shard:
on</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">cluster.data-self-heal-<wbr>algorithm:
full</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">features.cache-invalidation:
on</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">ganesha.enable:
on</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">features.shard-block-size:
256MB</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">client.event-threads:
2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">server.event-threads:
2</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">cluster.favorite-child-policy:
size</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">storage.build-pgfid:
off</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">network.ping-timeout:
5</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">cluster.enable-shared-storage:
enable</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">nfs-ganesha:
enable</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">cluster.server-quorum-ratio:
51%<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Adding
bricks:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">gluster
volume add-brick vmware2 replica 2
gluster01:/mnt/disk11/vmware2
gluster03:/mnt/disk11/vmware2
gluster02:/mnt/disk11/vmware2
gluster04:/mnt/disk11/vmware2<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">starting
fix layout:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">gluster
volume rebalance vmware2 fix-layout
start<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">Starting
rebalance:</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">gluster
volume rebalance vmware2 start<br>
<br>
<br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">--
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><font
color="#000000"><font
color="#3366ff">Respectfully</font></font><br>
<font color="#000000"><font
color="#3366ff"><b>Mahdi A.
Mahdi</b></font></font><br>
</p>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white">______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a moz-do-not-send="true"
href="mailto:Gluster-users@gluster.org"
target="_blank">Gluster-users@gluster.org</a><br>
<a moz-do-not-send="true"
href="http://lists.gluster.org/mailman/listinfo/gluster-users"
target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
</p>
</blockquote>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
</p>
</blockquote>
</blockquote>
</blockquote>
<blockquote type="cite">
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
</p>
</blockquote>
</blockquote>
<blockquote type="cite">
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
</p>
</blockquote>
<p dir="auto"
style="text-align:left;margin-top:25px;margin-bottom:25px;font-family:sans-serif;font-size:11pt;color:black;background-color:white"><br>
<br>
</p>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://lists.gluster.org/mailman/listinfo/gluster-users">http://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
<br>
</body>
</html>