<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
        {font-family:"Cambria Math";
        panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
        {font-family:Calibri;
        panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
        {margin:0in;
        margin-bottom:.0001pt;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
        {mso-style-priority:99;
        color:blue;
        text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
        {mso-style-priority:99;
        color:purple;
        text-decoration:underline;}
p
        {mso-style-priority:99;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;}
p.msonormal0, li.msonormal0, div.msonormal0
        {mso-style-name:msonormal;
        mso-margin-top-alt:auto;
        margin-right:0in;
        mso-margin-bottom-alt:auto;
        margin-left:0in;
        font-size:12.0pt;
        font-family:"Times New Roman",serif;}
span.EmailStyle19
        {mso-style-type:personal;
        font-family:"Calibri",sans-serif;
        color:#1F497D;}
span.EmailStyle20
        {mso-style-type:personal-compose;
        font-family:"Calibri",sans-serif;
        color:windowtext;}
.MsoChpDefault
        {mso-style-type:export-only;
        font-family:"Calibri",sans-serif;}
@page WordSection1
        {size:8.5in 11.0in;
        margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
        {page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1F497D">Yep, first unmount/remounted, then rebooted clients. &nbsp;Stopped/started the volumes, and rebooted all nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1F497D"><o:p>&nbsp;</o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Davide Obbi &lt;davide.obbi@booking.com&gt;
<br>
<b>Sent:</b> Monday, January 7, 2019 12:47 PM<br>
<b>To:</b> Matt Waymack &lt;mwaymack@nsgdv.com&gt;<br>
<b>Cc:</b> Raghavendra Gowdappa &lt;rgowdapp@redhat.com&gt;; gluster-users@gluster.org List &lt;gluster-users@gluster.org&gt;<br>
<b>Subject:</b> Re: [External] Re: [Gluster-users] Input/output error on FUSE log<o:p></o:p></span></p>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<p class="MsoNormal">i guess you tried already unmounting, stop/star and mounting?<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p>&nbsp;</o:p></p>
<div>
<div>
<p class="MsoNormal">On Mon, Jan 7, 2019 at 7:44 PM Matt Waymack &lt;<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1F497D">Yes, all volumes use sharding.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1F497D">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Davide Obbi &lt;<a href="mailto:davide.obbi@booking.com" target="_blank">davide.obbi@booking.com</a>&gt;
<br>
<b>Sent:</b> Monday, January 7, 2019 12:43 PM<br>
<b>To:</b> Matt Waymack &lt;<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>&gt;<br>
<b>Cc:</b> Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;;
<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List &lt;<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>&gt;<br>
<b>Subject:</b> Re: [External] Re: [Gluster-users] Input/output error on FUSE log</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">are all the volumes being configured with sharding?<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Mon, Jan 7, 2019 at 5:35 PM Matt Waymack &lt;<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1F497D">I think that I can rule out network as I have multiple volumes on the same nodes and not all volumes
 are affected.&nbsp; Additionally, access via SMB using samba-vfs-glusterfs is not affected, even on the same volumes.&nbsp;&nbsp; This is seemingly only affecting the FUSE clients.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif;color:#1F497D">&nbsp;</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:&quot;Calibri&quot;,sans-serif"> Davide Obbi &lt;<a href="mailto:davide.obbi@booking.com" target="_blank">davide.obbi@booking.com</a>&gt;
<br>
<b>Sent:</b> Sunday, January 6, 2019 12:26 PM<br>
<b>To:</b> Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt;<br>
<b>Cc:</b> Matt Waymack &lt;<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>&gt;;
<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List &lt;<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>&gt;<br>
<b>Subject:</b> Re: [External] Re: [Gluster-users] Input/output error on FUSE log</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">i would start doing some checks like: &quot;(Input/output error)&quot; seems returned by the operating system, this happens for instance trying to access a file system which is on a device
 not available so i would check the network connectivity between the client to servers&nbsp; and server to server during the reported time.
<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Regards<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Davide<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Sun, Jan 6, 2019 at 3:32 AM Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa &lt;<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack &lt;<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>&gt; wrote:<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div id="gmail-m_-1517320296147174849gmail-m_3080909756899822203gmail-m_-8138506196405553470gmail-m_6692858748533838528gmail-m_-506362477572935915divtagdefaultwrapper">
<p style="background:white"><span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Hi all,</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">&nbsp;</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">I'm having a problem writing to our volume.&nbsp; When writing files larger than about 2GB, I get an intermittent issue where the write will fail and return Input/Output error.&nbsp; This is also shown in the
 FUSE log of the client (this is affecting all clients).&nbsp; A snip of a client log is below:</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51040978: WRITE =&gt; -1 gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51040979: FLUSH() ERR =&gt; -1 (Input/output error)</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51041266: WRITE =&gt; -1 gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51041267: FLUSH() ERR =&gt; -1 (Input/output error)</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51041548: WRITE =&gt; -1 gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51041549: FLUSH() ERR =&gt; -1 (Input/output error)</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">The message &quot;W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1311504267&quot; repeated 1721 times between [2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">The message &quot;E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-gv1-dht: dict is null&quot; repeated 1714 times between [2019-01-05 22:39:33.925981] and [2019-01-05 22:39:50.451862]</span><o:p></o:p></p>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">The message &quot;W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1137142622&quot; repeated 1707 times between [2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]</span><o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This looks to be a DHT issue. Some questions:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">* Are all subvolumes of DHT up and client is connected to them? Particularly the subvolume which contains the file in question.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">* Can you get all extended attributes of parent directory of the file from all bricks?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">* set diagnostics.client-log-level to TRACE, capture these errors again and attach the client log file.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt">I spoke a bit early. dht_writev doesn't search hashed subvolume as its already been looked up in lookup. So, these msgs looks to be of a different issue - not&nbsp; writev failure.<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">&nbsp;<o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div id="gmail-m_-1517320296147174849gmail-m_3080909756899822203gmail-m_-8138506196405553470gmail-m_6692858748533838528gmail-m_-506362477572935915divtagdefaultwrapper">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">&nbsp;</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">This is intermittent for most files, but eventually if a file is large enough it will not write.&nbsp; The workflow is SFTP tot he client which then writes to the volume over FUSE.&nbsp; When files get to a certain
 point,w e can no longer write to them.&nbsp; The file sizes are different as well, so it's not like they all get to the same size and just stop either.&nbsp; I've ruled out a free space issue, our files at their largest are only a few hundred GB and we have tens of
 terrabytes free on each brick.&nbsp; We are also sharding at 1GB.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">&nbsp;</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">I'm not sure where to go from here as the error seems vague and I can only see it on the client log.&nbsp; I'm not seeing these errors on the nodes themselves.&nbsp; This is also seen if I mount the volume via
 FUSE on any of the nodes as well and it is only reflected in the FUSE log.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">&nbsp;</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Here is the volume info:</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Volume Name: gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Type: Distributed-Replicate</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Volume ID: 1472cc78-e2a0-4c3f-9571-dab840239b3c</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Status: Started</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Snapshot Count: 0</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Number of Bricks: 8 x (2 &#43; 1) = 24</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Transport-type: tcp</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Bricks:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick1: tpc-glus4:/exp/b1/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick2: tpc-glus2:/exp/b1/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick3: tpc-arbiter1:/exp/b1/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick4: tpc-glus2:/exp/b2/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick5: tpc-glus4:/exp/b2/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick6: tpc-arbiter1:/exp/b2/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick7: tpc-glus4:/exp/b3/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick8: tpc-glus2:/exp/b3/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick9: tpc-arbiter1:/exp/b3/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick10: tpc-glus4:/exp/b4/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick11: tpc-glus2:/exp/b4/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick12: tpc-arbiter1:/exp/b4/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick13: tpc-glus1:/exp/b5/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick14: tpc-glus3:/exp/b5/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick15: tpc-arbiter2:/exp/b5/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick16: tpc-glus1:/exp/b6/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick17: tpc-glus3:/exp/b6/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick18: tpc-arbiter2:/exp/b6/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick19: tpc-glus1:/exp/b7/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick20: tpc-glus3:/exp/b7/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick21: tpc-arbiter2:/exp/b7/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick22: tpc-glus1:/exp/b8/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick23: tpc-glus3:/exp/b8/gv1</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Brick24: tpc-arbiter2:/exp/b8/gv1 (arbiter)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">Options Reconfigured:</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">performance.cache-samba-metadata: on</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">performance.cache-invalidation: off</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">features.shard-block-size: 1000MB</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">features.shard: on</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">transport.address-family: inet</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">nfs.disable: on</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">cluster.lookup-optimize: on</span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">&nbsp;</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">I'm a bit stumped on this, any help is appreciated.&nbsp; Thank you!</span><o:p></o:p></p>
</div>
<p style="background:white;background-attachment:scroll;background-position-x:0%;background-position-y:0%">
<span style="font-family:&quot;Calibri&quot;,sans-serif;color:black">&nbsp;</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#003580">Davide Obbi</span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00AEEF">Senior System Administrator</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">Booking.com B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00AEEF">Direct&nbsp;</span><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">&#43;31207031558</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-family:&quot;Arial&quot;,sans-serif;color:#003580"><a href="https://www.booking.com/" target="_blank"><span style="color:#007FFF;text-decoration:none"><img border="0" id="gmail-m_-1517320296147174849gmail-m_3080909756899822203_x005f_x0000_i1025" src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com"></span></a></span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">Empowering people to experience the world since 1996</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">43 languages, 214&#43; offices worldwide, 141,000&#43; global destinations, 29 million reported listings&nbsp;<br>
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#003580">Davide Obbi</span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00AEEF">Senior System Administrator</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">Booking.com B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00AEEF">Direct&nbsp;</span><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">&#43;31207031558</span><o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-family:&quot;Arial&quot;,sans-serif;color:#003580"><a href="https://www.booking.com/" target="_blank"><span style="color:#007FFF;text-decoration:none"><img border="0" id="gmail-m_-1517320296147174849_x005f_x0000_i1026" src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com"></span></a></span></b><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">Empowering people to experience the world since 1996</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">43 languages, 214&#43; offices worldwide, 141,000&#43; global destinations, 29 million reported listings&nbsp;<br>
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#003580">Davide Obbi<o:p></o:p></span></b></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00AEEF">Senior System Administrator<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">Booking.com B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#00AEEF">Direct&nbsp;</span><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">&#43;31207031558<o:p></o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-family:&quot;Arial&quot;,sans-serif;color:#003580"><a href="https://www.booking.com/" target="_blank"><span style="color:#007FFF;text-decoration:none"><img border="0" id="_x0000_i1027" src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com"></span></a><o:p></o:p></span></b></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">Empowering people to experience the world since 1996</span><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666">43 languages, 214&#43; offices worldwide, 141,000&#43; global destinations, 29 million reported listings&nbsp;<br>
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><span style="font-size:10.0pt;font-family:&quot;Arial&quot;,sans-serif;color:#666666"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>