<div dir="ltr">i guess you tried already unmounting, stop/star and mounting?<br></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Jan 7, 2019 at 7:44 PM Matt Waymack <<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-1517320296147174849WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">Yes, all volumes use sharding.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Davide Obbi <<a href="mailto:davide.obbi@booking.com" target="_blank">davide.obbi@booking.com</a>>
<br>
<b>Sent:</b> Monday, January 7, 2019 12:43 PM<br>
<b>To:</b> Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>><br>
<b>Cc:</b> Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>; <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [External] Re: [Gluster-users] Input/output error on FUSE log<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">are all the volumes being configured with sharding?<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Mon, Jan 7, 2019 at 5:35 PM Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)">I think that I can rule out network as I have multiple volumes on the same nodes and not all volumes
are affected. Additionally, access via SMB using samba-vfs-glusterfs is not affected, even on the same volumes. This is seemingly only affecting the FUSE clients.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:"Calibri",sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11pt;font-family:"Calibri",sans-serif"> Davide Obbi <<a href="mailto:davide.obbi@booking.com" target="_blank">davide.obbi@booking.com</a>>
<br>
<b>Sent:</b> Sunday, January 6, 2019 12:26 PM<br>
<b>To:</b> Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>><br>
<b>Cc:</b> Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>>;
<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [External] Re: [Gluster-users] Input/output error on FUSE log</span><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Hi,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">i would start doing some checks like: "(Input/output error)" seems returned by the operating system, this happens for instance trying to access a file system which is on a device
not available so i would check the network connectivity between the client to servers and server to server during the reported time.
<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Regards<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Davide<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jan 6, 2019 at 3:32 AM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div id="gmail-m_-1517320296147174849gmail-m_3080909756899822203gmail-m_-8138506196405553470gmail-m_6692858748533838528gmail-m_-506362477572935915divtagdefaultwrapper">
<p style="background:white none repeat scroll 0% 0%"><span style="font-family:"Calibri",sans-serif;color:black">Hi all,</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black"> </span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">I'm having a problem writing to our volume. When writing files larger than about 2GB, I get an intermittent issue where the write will fail and return Input/Output error. This is also shown in the
FUSE log of the client (this is affecting all clients). A snip of a client log is below:</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51040978: WRITE => -1 gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output error)</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51041266: WRITE => -1 gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output error)</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51041548: WRITE => -1 gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output error)</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1311504267" repeated 1721 times between [2019-01-05 22:39:33.906241] and [2019-01-05 22:39:44.598371]</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 22:39:33.925981] and [2019-01-05 22:39:50.451862]</span><u></u><u></u></p>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1137142622" repeated 1707 times between [2019-01-05 22:39:39.636552] and [2019-01-05 22:39:50.451895]</span><u></u><u></u></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This looks to be a DHT issue. Some questions:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* Are all subvolumes of DHT up and client is connected to them? Particularly the subvolume which contains the file in question.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* Can you get all extended attributes of parent directory of the file from all bricks?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* set diagnostics.client-log-level to TRACE, capture these errors again and attach the client log file.<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">I spoke a bit early. dht_writev doesn't search hashed subvolume as its already been looked up in lookup. So, these msgs looks to be of a different issue - not writev failure.<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-width:medium medium medium 1pt;padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt;border-color:currentcolor currentcolor currentcolor rgb(204,204,204)">
<div>
<div id="gmail-m_-1517320296147174849gmail-m_3080909756899822203gmail-m_-8138506196405553470gmail-m_6692858748533838528gmail-m_-506362477572935915divtagdefaultwrapper">
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">This is intermittent for most files, but eventually if a file is large enough it will not write. The workflow is SFTP tot he client which then writes to the volume over FUSE. When files get to a certain
point,w e can no longer write to them. The file sizes are different as well, so it's not like they all get to the same size and just stop either. I've ruled out a free space issue, our files at their largest are only a few hundred GB and we have tens of
terrabytes free on each brick. We are also sharding at 1GB.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">I'm not sure where to go from here as the error seems vague and I can only see it on the client log. I'm not seeing these errors on the nodes themselves. This is also seen if I mount the volume via
FUSE on any of the nodes as well and it is only reflected in the FUSE log.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Here is the volume info:</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Volume Name: gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Type: Distributed-Replicate</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Volume ID: 1472cc78-e2a0-4c3f-9571-dab840239b3c</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Status: Started</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Snapshot Count: 0</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Number of Bricks: 8 x (2 + 1) = 24</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Transport-type: tcp</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Bricks:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick1: tpc-glus4:/exp/b1/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick2: tpc-glus2:/exp/b1/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick3: tpc-arbiter1:/exp/b1/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick4: tpc-glus2:/exp/b2/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick5: tpc-glus4:/exp/b2/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick6: tpc-arbiter1:/exp/b2/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick7: tpc-glus4:/exp/b3/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick8: tpc-glus2:/exp/b3/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick9: tpc-arbiter1:/exp/b3/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick10: tpc-glus4:/exp/b4/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick11: tpc-glus2:/exp/b4/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick12: tpc-arbiter1:/exp/b4/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick13: tpc-glus1:/exp/b5/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick14: tpc-glus3:/exp/b5/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick15: tpc-arbiter2:/exp/b5/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick16: tpc-glus1:/exp/b6/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick17: tpc-glus3:/exp/b6/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick18: tpc-arbiter2:/exp/b6/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick19: tpc-glus1:/exp/b7/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick20: tpc-glus3:/exp/b7/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick21: tpc-arbiter2:/exp/b7/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick22: tpc-glus1:/exp/b8/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick23: tpc-glus3:/exp/b8/gv1</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Brick24: tpc-arbiter2:/exp/b8/gv1 (arbiter)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">Options Reconfigured:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">performance.cache-samba-metadata: on</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">performance.cache-invalidation: off</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">features.shard-block-size: 1000MB</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">features.shard: on</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">transport.address-family: inet</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">nfs.disable: on</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">cluster.lookup-optimize: on</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black">I'm a bit stumped on this, any help is appreciated. Thank you!</span><u></u><u></u></p>
</div>
<p style="background:white none repeat scroll 0% 0%">
<span style="font-family:"Calibri",sans-serif;color:black"> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:rgb(0,53,128)">Davide Obbi</span></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(0,174,239)">Senior System Administrator</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">Booking.com B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(0,174,239)">Direct </span><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">+31207031558</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:rgb(0,53,128)"><a href="https://www.booking.com/" target="_blank"><span style="color:rgb(0,127,255);text-decoration:none"><img id="gmail-m_-1517320296147174849gmail-m_3080909756899822203_x005f_x0000_i1025" src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" border="0"></span></a></span></b><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">Empowering people to experience the world since 1996</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million reported listings <br>
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-size:11.5pt;font-family:"Arial",sans-serif;color:rgb(0,53,128)">Davide Obbi<u></u><u></u></span></b></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(0,174,239)">Senior System Administrator<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">Booking.com B.V.<br>
Vijzelstraat 66-80 Amsterdam 1017HL Netherlands<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(0,174,239)">Direct </span><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">+31207031558<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Arial",sans-serif;color:rgb(0,53,128)"><a href="https://www.booking.com/" target="_blank"><span style="color:rgb(0,127,255);text-decoration:none"><img id="gmail-m_-1517320296147174849_x0000_i1026" src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" border="0"></span></a><u></u><u></u></span></b></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">Empowering people to experience the world since 1996</span><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:8.5pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)">43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million reported listings <br>
Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><span style="font-size:10pt;font-family:"Arial",sans-serif;color:rgb(102,102,102)"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><span style="display:block;font-size:11pt;font-family:Century Gothic;color:rgb(0,53,128)"><div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px"><div>Davide Obbi</div><div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">Senior System Administrator<br><br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com B.V.<br>Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="https://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" target="_blank"><img src="https://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div><div><span style="font-size:11px">Empowering people to experience the world since 1996</span></div><div><span style="font-size:11px">43 languages, 214+ offices worldwide, 141,000+ global destinations, 29 million reported listings <br></span><span style="font-size:11px">Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span></div></div></div></span></div>