<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:Monaco;
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
p.p1, li.p1, div.p1
{mso-style-name:p1;
margin:0in;
margin-bottom:.0001pt;
background:black;
font-size:7.5pt;
font-family:Monaco;
color:#F2F2F2;}
span.s2
{mso-style-name:s2;
color:#B42419;}
span.s1
{mso-style-name:s1;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We are having a sporadic issue with our Gluster mounts that is affecting several of our Kubernetes environments. We are having trouble understanding what is causing it, and we could use some guidance from
the pros!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt">Scenario<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">We have an environment running a single-node Kubernetes with Heketi and several pods using Gluster mounts. The environment runs fine and the mounts appear to be healthy for up to several days. Suddenly, one
or more (sometimes all) Gluster mounts report a stale mount and shut down the brick. The affected containers enter a crash loop that continues indefinitely, until someone intervenes. To work-around the crash loop, a user needs to trigger the bricks to be started
again--either through manually starting them, restarting the Gluster pod or restarting the entire node.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div style="mso-element:para-border-div;border:none;border-bottom:solid windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span style="font-size:11.0pt">Diagnostics<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt">Looking at the </span><span style="font-size:10.5pt;font-family:Helvetica;color:#24292E;background:white">glusterd.log</span><span style="font-size:11.0pt"> file, the error message at the time the problem
starts looks something like this:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E">got disconnect from stale rpc on /var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_d0456279568a623a16a5508daa89b4d5/brick</span><span style="font-size:11.0pt"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">This message occurs once for each brick that stops responding. The brick does not recover on its own. Here is that same message again, with surrounding context included.<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in"><o:p> </o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:38.663362] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0x3a7a5)
[0x7f795f0d77a5] -->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0xe2765) [0x7f795f17f765] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f79643180f5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=vol_d0a0dcf9903e236f68a3933c3060ec5a
--last=no<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:38.905338] E [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0x3a7a5)
[0x7f795f0d77a5] -->/usr/lib64/glusterfs/4.1.7/xlator/mgmt/glusterd.so(+0xe26c3) [0x7f795f17f6c3] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f79643180f5] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=vol_d0a0dcf9903e236f68a3933c3060ec5a
--last=no<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:38.982785] I [MSGID: 106542] [glusterd-utils.c:8253:glusterd_brick_signal] 0-glusterd:
sending signal 15 to brick with pid 8951<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:39.983244] I [MSGID: 106143] [glusterd-pmap.c:397:pmap_registry_remove] 0-pmap: removing
brick /var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_d0456279568a623a16a5508daa89b4d5/brick on port 49169<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:39.984656] W [glusterd-handler.c:6124:__glusterd_brick_rpc_notify] 0-management: got
disconnect from stale rpc on /var/lib/heketi/mounts/vg_c197878af606e71a874ad28e3bd7e4e1/brick_d0456279568a623a16a5508daa89b4d5/brick<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.316466] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
nfs already stopped<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.316601] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
nfs service is stopped<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.316644] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management:
nfs/server.so xlator is not installed<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.319650] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
bitd already stopped<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.319708] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
bitd service is stopped<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.321091] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
scrub already stopped<o:p></o:p></span></p>
<p class="MsoNormal" style="background:#F6F8FA"><span style="font-size:9.0pt;font-family:Consolas;color:#24292E;border:none windowtext 1.0pt;padding:0in">[2019-05-07 11:53:40.321132] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
scrub service is stopped</span><span style="font-size:9.0pt;font-family:Consolas;color:#24292E"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The version of gluster we are using (running in a container, using the gluster/gluster-centos image from dockerhub):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span class="s1"># rpm -qa | grep gluster</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-rdma-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">-block-0.3-2.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s1">python2-</span><span class="s2">gluster</span><span class="s1">-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s1">centos-release-</span><span class="s2">gluster</span><span class="s1">41-1.0-3.el7.centos.noarch</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-api-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-cli-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-geo-replication-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-libs-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-client-xlators-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-fuse-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-server-4.1.7-1.el7.x86_64</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The version of gluster running on our Kubernetes node (a CentOS system):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">]$ rpm -qa | grep gluster</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-libs-3.12.2-18.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-3.12.2-18.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-fuse-3.12.2-18.el7.x86_64</span><o:p></o:p></p>
<p class="p1"><span class="s2">gluster</span><span class="s1">fs-client-xlators-3.12.2-18.el7.x86_64</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">The Kubernetes version:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="p1"><span class="s1">$</span><span class="apple-converted-space"> </span>
<span class="s1">kubectl version</span><o:p></o:p></p>
<p class="p1"><span class="s1">Client Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}</span><o:p></o:p></p>
<p class="p1"><span class="s1">Server Version: version.Info{Major:"1", Minor:"13", GitVersion:"v1.13.5", GitCommit:"2166946f41b36dea2c4626f90a77706f426cdea2", GitTreeState:"clean", BuildDate:"2019-03-25T15:19:22Z", GoVersion:"go1.11.5", Compiler:"gc", Platform:"linux/amd64"}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Full Gluster logs available if needed, just let me know how to provide them.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thanks in advance for any help or suggestions on this!<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Best,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Jeff Bischoff<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Turbonomic<o:p></o:p></span></p>
</div>
This message and its attachments are intended only for the designated recipient(s). It may contain confidential or proprietary information and may be subject to legal or other confidentiality protections. If you are not a designated recipient, you may not review,
copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
</body>
</html>