<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='color:black'>Hi all,<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>We are having a sporadic issue with our Gluster mounts that is affecting several of our Kubernetes environments. We are having trouble understanding what is causing it, and we could use some guidance from the pros!<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><u><span style='color:black'>Scenario</span></u><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>We have an environment running a single-node Kubernetes with Heketi and several pods using Gluster mounts. The environment runs fine and the mounts appear to be healthy for up to several days. Suddenly, one or more (sometimes all) Gluster mounts have a problem and shut down the brick. The affected containers enter a crash loop that continues indefinitely, until someone intervenes. To work-around the crash loop, a user needs to trigger the bricks to be started again--either through manually starting them, restarting the Gluster pod or restarting the entire node.<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><u><span style='color:black'>Diagnostics</span></u><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>The tell-tale error message is seeing the following when describing a pod that is in a crash loop:<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>Message: error while creating mount source path '/var/lib/kubelet/pods/4a2574bb-6fa4-11e9-a315-005056b83c80/volumes/kubernetes.io~glusterfs/db': mkdir /var/lib/kubelet/pods/4a2574bb-6fa4-11e9-a315-005056b83c80/volumes/kubernetes.io~glusterfs/db: file exists</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='margin-left:.5in;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>We always see that "file exists" message when this error occurs.<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Looking at the </span><span style='font-size:10.5pt;font-family:Helvetica;color:#24292E;background:white'>glusterd.log</span><span style='color:black'> file, there had been nothing in the log for over a day and then suddenly, at the time the crash loop started, this:<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:04.733147] I [MSGID: 106143] [glusterd-pmap.c:397:pmap_registry_remove] 0-pmap: removing brick /var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_a3cef78a5914a2808da0b5736e3daec7/brick on port 49168</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:04.733374] I [MSGID: 106143] [glusterd-pmap.c:397:pmap_registry_remove] 0-pmap: removing brick /var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_7614e5014a0e402630a0e1fd776acf0a/brick on port 49167</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.003848] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/fe4ac75011a4de0e.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.065420] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/85e9fb223aa121f2.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.066479] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/e2a66e8cd8f5f606.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.067444] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/a0625e5b78d69bb8.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.068471] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/770bc294526d0360.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.074278] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/adbd37fe3e1eed36.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.075497] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/17712138f3370e53.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.076545] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/a6cf1aca8b23f394.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.077511] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/d0f83b191213e877.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.078447] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/d5dd08945d4f7f6d.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:05.079424] W [socket.c:599:__socket_rwv] 0-management: readv on /var/run/gluster/c8d7b10108758e2f.socket failed (No data available)</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>[2019-05-08 13:49:14.778619] I [MSGID: 106143] [glusterd-pmap.c:397:pmap_registry_remove] 0-pmap: removing brick /var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_0ed4f7f941de388cda678fe273e9ceb4/brick on port 49166</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>... (and more of the same)<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Nothing further has been printed to the gluster log since. The bricks do not come back on their own.<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>The version of gluster we are using (running in a container, using the gluster/gluster-centos image from dockerhub):<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'># rpm -qa | grep gluster</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-rdma-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>-block-0.3-2.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>python2-</span><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>centos-release-</span><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>41-1.0-3.el7.centos.noarch</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-api-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-cli-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-geo-replication-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-libs-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-client-xlators-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-fuse-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-server-4.1.7-1.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>The version of glusterfs running on our Kubernetes node (a CentOS system):<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>]$ rpm -qa | grep gluster</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-libs-3.12.2-18.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-3.12.2-18.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-fuse-3.12.2-18.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#B42419'>gluster</span><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>fs-client-xlators-3.12.2-18.el7.x86_64</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>The Kubernetes version:<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>$ kubectl version</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:black;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:7.5pt;font-family:Monaco;color:#F2F2F2'>Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Our gluster settings/volume options:<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>apiVersion: storage.k8s.io/v1</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>kind: StorageClass</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>metadata:</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> name: gluster-heketi</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> selfLink: /apis/storage.k8s.io/v1/storageclasses/gluster-heketi</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>parameters:</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> gidMax: "50000"</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> gidMin: "2000"</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> resturl:<span class=apple-converted-space> </span><a href="http://10.233.35.158:8080"><span style='color:#954F72'>http://10.233.35.158:8080</span></a></span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> restuser: "null"</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> restuserkey: "null"</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> volumetype: "none"</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> volumeoptions: cluster.post-op-delay-secs 0, performance.client-io-threads off, performance.open-behind off, performance.readdir-ahead off, performance.read-ahead off, performance.stat-prefetch off, performance.write-behind off, performance.io-cache off, cluster.consistent-metadata on, performance.quick-read off, performance.strict-o-direct on</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>provisioner: kubernetes.io/glusterfs</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>reclaimPolicy: Delete</span><span style='color:black'><o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Volume info for the heketi volume:<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>gluster> volume info heketidbstorage</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'> </span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Volume Name: heketidbstorage</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Type: Distribute</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Volume ID: 34b897d0-0953-4f8f-9c5c-54e043e55d92</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Status: Started</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Snapshot Count: 0</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Number of Bricks: 1</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Transport-type: tcp</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Bricks:</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Brick1: 10.10.168.25:/var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_a16f9f0374fe5db948a60a017a3f5e60/brick</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>Options Reconfigured:</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>user.heketi.id: 1d2400626dac780fce12e45a07494853</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>transport.address-family: inet</span></code><span style='color:black'><o:p></o:p></span></pre><pre style='background:#F6F8FA;caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;background-position:initial initial;background-repeat:initial initial;word-spacing:0px'><code><span style='font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in'>nfs.disable: on</span></code><span style='color:black'><o:p></o:p></span></pre><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Full Gluster logs available if needed, just let me know how best to provide them.<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Thanks in advance for any help or suggestions on this!<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Best,<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Jeff Bischoff<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'>Turbonomic<o:p></o:p></span></p><p class=MsoNormal style='caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px'><span style='color:black'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt'><o:p> </o:p></span></p></div></body></html>