<div dir="ltr"><div dir="ltr">I don't see write failures in the log but I do see fallocate failing with EIO.<div><br></div><div><div>[2019-01-07 19:16:44.846187] W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1285124113</div><div>[2019-01-07 19:16:44.846194] D [MSGID: 0] [dht-helper.c:969:dht_subvol_get_hashed] 0-gv1-dht: No hashed subvolume for path=/.shard/aa3ef10e-95e0-40d3-9464-133d72fa8a95.185</div><div>[2019-01-07 19:16:44.846200] D [MSGID: 0] [dht-common.c:7631:dht_mknod] 0-gv1-dht: no subvolume in layout for path=/.shard/aa3ef10e-95e0-40d3-9464-133d72fa8a95.185 <--- *** <b>DHT failed to find a hashed subvol</b> ***</div><div><br></div><div>[2019-01-07 19:16:44.846207] D [MSGID: 0] [dht-common.c:7712:dht_mknod] 0-stack-trace: stack-address: 0x7f6748006778, gv1-dht returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846215] D [MSGID: 0] [shard.c:3645:shard_common_mknod_cbk] 0-gv1-shard: mknod of shard 185 failed: Input/output error</div><div>[2019-01-07 19:16:44.846223] D [MSGID: 0] [shard.c:720:shard_common_inode_write_failure_unwind] 0-stack-trace: stack-address: 0x7f6748006778, gv1-shard returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846234] D [MSGID: 0] [defaults.c:1352:default_fallocate_cbk] 0-stack-trace: stack-address: 0x7f6748006778, gv1-quick-read returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846244] D [MSGID: 0] [defaults.c:1352:default_fallocate_cbk] 0-stack-trace: stack-address: 0x7f6748006778, gv1-open-behind returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846254] D [MSGID: 0] [md-cache.c:2715:mdc_fallocate_cbk] 0-stack-trace: stack-address: 0x7f6748006778, gv1-md-cache returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846264] D [MSGID: 0] [defaults.c:1352:default_fallocate_cbk] 0-stack-trace: stack-address: 0x7f6748006778, gv1-io-threads returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846274] D [MSGID: 0] [io-stats.c:2528:io_stats_fallocate_cbk] 0-stack-trace: stack-address: 0x7f6748006778, gv1 returned -1 error: Input/output error [Input/output error]</div><div>[2019-01-07 19:16:44.846284] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 1373: FALLOCATE() ERR => -1 (Input/output error)</div><div>[2019-01-07 19:16:44.846298] T [fuse-bridge.c:278:send_fuse_iov] 0-glusterfs-fuse: writev() result 16/16 </div></div><div><br></div><div>Please get the xattrs on the .shard directory on each brick of the volume so we can check if the layout is complete:</div><div><br></div><div>getfattr -e hex -m . -d <brick_root>/.shard</div><div><br></div><div>Thanks,</div><div>Nithya</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, 10 Jan 2019 at 02:25, Matt Waymack <<a href="mailto:mwaymack@nsgdv.com">mwaymack@nsgdv.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-5543314113905380092WordSection1">
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Has anyone any other ideas where to look? This is only affecting FUSE clients. SMB clients are unaffected by this problem.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thanks!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> <a href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster.org</a> <<a href="mailto:gluster-users-bounces@gluster.org" target="_blank">gluster-users-bounces@gluster.org</a>>
<b>On Behalf Of </b>Matt Waymack<br>
<b>Sent:</b> Monday, January 7, 2019 1:19 PM<br>
<b>To:</b> Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] Input/output error on FUSE log<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Attached are the logs from when a failure occurred with diagnostics set to trace.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)">Thank you!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:Calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:Calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:Calibri,sans-serif"> Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>>
<br>
<b>Sent:</b> Saturday, January 5, 2019 8:32 PM<br>
<b>To:</b> Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>><br>
<b>Cc:</b> <a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a> List <<a href="mailto:gluster-users@gluster.org" target="_blank">gluster-users@gluster.org</a>><br>
<b>Subject:</b> Re: [Gluster-users] Input/output error on FUSE log<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jan 6, 2019 at 7:58 AM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jan 6, 2019 at 4:19 AM Matt Waymack <<a href="mailto:mwaymack@nsgdv.com" target="_blank">mwaymack@nsgdv.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div id="gmail-m_-5543314113905380092gmail-m_6692858748533838528gmail-m_-506362477572935915divtagdefaultwrapper">
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Hi all,<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><u></u> <u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">I'm having a problem writing to our volume. When writing files larger than about 2GB, I get an intermittent issue where the write will fail and return Input/Output error.
This is also shown in the FUSE log of the client (this is affecting all clients). A snip of a client log is below:<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">[2019-01-05 22:39:44.581371] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51040978: WRITE => -1 gfid=82a0b5c4-7ef3-43c2-ad86-41e16673d7c2 fd=0x7f949839a368 (Input/output
error)<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">[2019-01-05 22:39:44.598392] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51040979: FLUSH() ERR => -1 (Input/output error)<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">[2019-01-05 22:39:47.420920] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51041266: WRITE => -1 gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949809b7f8 (Input/output
error)<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">[2019-01-05 22:39:47.433377] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51041267: FLUSH() ERR => -1 (Input/output error)<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">[2019-01-05 22:39:50.441531] W [fuse-bridge.c:2474:fuse_writev_cbk] 0-glusterfs-fuse: 51041548: WRITE => -1 gfid=0e8e1e13-97a5-478a-bc58-e81ddf3698a3 fd=0x7f949839a368 (Input/output
error)<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">[2019-01-05 22:39:50.451914] W [fuse-bridge.c:1441:fuse_err_cbk] 0-glusterfs-fuse: 51041549: FLUSH() ERR => -1 (Input/output error)<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1311504267" repeated 1721 times between [2019-01-05 22:39:33.906241]
and [2019-01-05 22:39:44.598371]<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">The message "E [MSGID: 101046] [dht-common.c:1502:dht_lookup_dir_cbk] 0-gv1-dht: dict is null" repeated 1714 times between [2019-01-05 22:39:33.925981] and [2019-01-05 22:39:50.451862]<u></u><u></u></span></p>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black">The message "W [MSGID: 109011] [dht-layout.c:163:dht_layout_search] 0-gv1-dht: no subvolume for hash (value) = 1137142622" repeated 1707 times between [2019-01-05 22:39:39.636552]
and [2019-01-05 22:39:50.451895]<u></u><u></u></span></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This looks to be a DHT issue. Some questions:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* Are all subvolumes of DHT up and client is connected to them? Particularly the subvolume which contains the file in question.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* Can you get all extended attributes of parent directory of the file from all bricks?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">* set diagnostics.client-log-level to TRACE, capture these errors again and attach the client log file.<u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">I spoke a bit early. dht_writev doesn't search hashed subvolume as its already been looked up in lookup. So, these msgs looks to be of a different issue - not writev failure.<u></u><u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<blockquote style="border-top:none;border-right:none;border-bottom:none;border-left:1pt solid rgb(204,204,204);padding:0in 0in 0in 6pt;margin:5pt 0in 5pt 4.8pt">
<div>
<div id="gmail-m_-5543314113905380092gmail-m_6692858748533838528gmail-m_-506362477572935915divtagdefaultwrapper">
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">This is intermittent for most files, but eventually if a file is large enough it will not write. The workflow is SFTP tot he client which then writes to
the volume over FUSE. When files get to a certain point,w e can no longer write to them. The file sizes are different as well, so it's not like they all get to the same size and just stop either. I've ruled out a free space issue, our files at their largest
are only a few hundred GB and we have tens of terrabytes free on each brick. We are also sharding at 1GB.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">I'm not sure where to go from here as the error seems vague and I can only see it on the client log. I'm not seeing these errors on the nodes themselves.
This is also seen if I mount the volume via FUSE on any of the nodes as well and it is only reflected in the FUSE log.<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Here is the volume info:<u></u><u></u></span></p>
</div>
<div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Volume Name: gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Type: Distributed-Replicate<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Volume ID: 1472cc78-e2a0-4c3f-9571-dab840239b3c<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Status: Started<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Snapshot Count: 0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Number of Bricks: 8 x (2 + 1) = 24<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Transport-type: tcp<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Bricks:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick1: tpc-glus4:/exp/b1/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick2: tpc-glus2:/exp/b1/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick3: tpc-arbiter1:/exp/b1/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick4: tpc-glus2:/exp/b2/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick5: tpc-glus4:/exp/b2/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick6: tpc-arbiter1:/exp/b2/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick7: tpc-glus4:/exp/b3/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick8: tpc-glus2:/exp/b3/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick9: tpc-arbiter1:/exp/b3/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick10: tpc-glus4:/exp/b4/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick11: tpc-glus2:/exp/b4/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick12: tpc-arbiter1:/exp/b4/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick13: tpc-glus1:/exp/b5/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick14: tpc-glus3:/exp/b5/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick15: tpc-arbiter2:/exp/b5/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick16: tpc-glus1:/exp/b6/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick17: tpc-glus3:/exp/b6/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick18: tpc-arbiter2:/exp/b6/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick19: tpc-glus1:/exp/b7/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick20: tpc-glus3:/exp/b7/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick21: tpc-arbiter2:/exp/b7/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick22: tpc-glus1:/exp/b8/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick23: tpc-glus3:/exp/b8/gv1<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Brick24: tpc-arbiter2:/exp/b8/gv1 (arbiter)<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">Options Reconfigured:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">performance.cache-samba-metadata: on<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">performance.cache-invalidation: off<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">features.shard-block-size: 1000MB<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">features.shard: on<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">transport.address-family: inet<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">nfs.disable: on<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">cluster.lookup-optimize: on<u></u><u></u></span></p>
</div>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="background:white"><span style="font-family:Calibri,sans-serif;color:black">I'm a bit stumped on this, any help is appreciated. Thank you!<u></u><u></u></span></p>
</div>
<p style="background:white"><span style="font-family:Calibri,sans-serif;color:black"><u></u> <u></u></span></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>