<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hello Folks,<div><br></div><div>I am new to heketi and was following the documentation word to work.</div><div><br></div><div>I successfully created a 3 node topolgy and the cluster was successfully created.</div><div><br></div><div><div>./heketi-cli --server <a href="http://localhost:8085">http://localhost:8085</a> topology load --json=topology.json</div><div>Creating cluster ... ID: 1233163d3b76a6538a34d9b27a855ead</div><div><span style="white-space:pre">        </span>Allowing file volumes on cluster.</div><div><span style="white-space:pre">        </span>Allowing block volumes on cluster.</div><div><span style="white-space:pre">        </span>Creating node glust-0 ... ID: 82af3c3335b7d24d0e5fc7559d44f8d7</div><div><span style="white-space:pre">                </span>Adding device /dev/sdb ... OK</div><div><span style="white-space:pre">                </span>Adding device /dev/sdc ... OK</div><div><span style="white-space:pre">        </span>Creating node glust-1 ... ID: ec106227ea3580571a252b28ad223109</div><div><span style="white-space:pre">                </span>Adding device /dev/sdb ... OK</div><div><span style="white-space:pre">                </span>Adding device /dev/sdc ... OK</div><div><span style="white-space:pre">        </span>Creating node glust-2 ... ID: fc6535a147b39d65e7037015dcc0cc20</div><div><span style="white-space:pre">                </span>Adding device /dev/sdb ... OK</div><div><span style="white-space:pre">                </span>Adding device /dev/sdc ... OK</div></div><div><br></div><div><br></div><div>And on all 3 nodes, I can see that I have successfully peered.</div><div><br></div><div><div>glust-1:~# gluster volume info</div><div>No volumes present</div><div>glust-1:~# gluster peer status</div><div>Number of Peers: 2</div><div><br></div><div>Hostname: glust-0</div><div>Uuid: 22891efe-28ce-4c12-b97a-42ee78d50995</div><div>State: Peer in Cluster (Connected)</div><div><br></div><div>Hostname: 10.27.239.240</div><div>Uuid: fc5834ec-2b7e-402b-a412-6783bae0db66</div><div>State: Peer in Cluster (Connected)</div></div><div><br></div><div><br></div><div>But when I create a volume, </div><div><br></div><div>I get </div><div><br></div></div></div></div></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">./heketi-cli --server <a href="http://localhost:8085">http://localhost:8085</a> volume create --size=10<br>Error: Get <a href="http://localhost:8085/queue/729862db18ef8d968eb7d0017c09b04d">http://localhost:8085/queue/729862db18ef8d968eb7d0017c09b04d</a>: dial tcp [::1]:8085: getsockopt: connection refused</blockquote></div></blockquote><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br></div><div>And heketi crashes after spitting out HUGE amounts of logs. My client binaries are the latest Darwin ones.</div><div><br></div><div>My heketi.json looks like:</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br></blockquote></div></div></div></div></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">╰─ cat heketi.json</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">{</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;_port_comment&quot;: &quot;Heketi Server Port Number&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;port&quot;: &quot;8085&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="white-space:pre">        </span>&quot;_enable_tls_comment&quot;: &quot;Enable TLS in Heketi Server&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="white-space:pre">        </span>&quot;enable_tls&quot;: false,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="white-space:pre">        </span>&quot;_cert_file_comment&quot;: &quot;Path to a valid certificate file&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="white-space:pre">        </span>&quot;cert_file&quot;: &quot;&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="white-space:pre">        </span>&quot;_key_file_comment&quot;: &quot;Path to a valid private key file&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span style="white-space:pre">        </span>&quot;key_file&quot;: &quot;&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br></blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;_use_auth&quot;: &quot;Enable JWT authorization. Please enable for deployment&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;use_auth&quot;: false,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;_jwt&quot;: &quot;Private keys for access&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;jwt&quot;: {</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_admin&quot;: &quot;Admin has access to all APIs&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;admin&quot;: {</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;key&quot;: &quot;My Secret&quot;</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    },</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_user&quot;: &quot;User only has access to /volumes endpoint&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;user&quot;: {</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;key&quot;: &quot;My Secret&quot;</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    }</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  },</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;_backup_db_to_kube_secret&quot;: &quot;Backup the heketi database to a Kubernetes secret when running in Kubernetes. Default is off.&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;backup_db_to_kube_secret&quot;: false,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;_glusterfs_comment&quot;: &quot;GlusterFS Configuration&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  &quot;glusterfs&quot;: {</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_executor_comment&quot;: [</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;Execute plugin. Possible choices: mock, ssh&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;mock: This setting is used for testing and development.&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;      It will not send commands to any node.&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;ssh:  This setting will notify Heketi to ssh to the nodes.&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;      It will need the values in sshexec to be configured.&quot;,</blockquote></div></div></div></div></div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;kubernetes: Communicate with GlusterFS containers over&quot;,</blockquote></div></div></div></div></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;            Kubernetes exec api.&quot;<br>    ],<br>    &quot;executor&quot;: &quot;ssh&quot;,<br>    &quot;_sshexec_comment&quot;: &quot;SSH username and private key file information&quot;,<br>    &quot;sshexec&quot;: {<br>      &quot;keyfile&quot;: &quot;id_rsa&quot;,<br>      &quot;user&quot;: &quot;root&quot;,<br>      &quot;port&quot;: &quot;22&quot;,<br>      &quot;fstab&quot;: &quot;/etc/fstab&quot;,<br>      &quot;backup_lvm_metadata&quot;: false</blockquote></div></blockquote><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    },</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br></blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_db_comment&quot;: &quot;Database file name&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;db&quot;: &quot;heketi.db&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">     &quot;_refresh_time_monitor_gluster_nodes&quot;: &quot;Refresh time in seconds to monitor Gluster nodes&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;refresh_time_monitor_gluster_nodes&quot;: 120,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_start_time_monitor_gluster_nodes&quot;: &quot;Start time in seconds to monitor Gluster nodes when the heketi comes up&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;start_time_monitor_gluster_nodes&quot;: 10,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_loglevel_comment&quot;: [</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;Set log level. Choices are:&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;  none, critical, error, warning, info, debug&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">      &quot;Default is warning&quot;</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    ],</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;loglevel&quot; : &quot;debug&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_auto_create_block_hosting_volume&quot;: &quot;Creates Block Hosting volumes automatically if not found or exsisting volume exhausted&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;auto_create_block_hosting_volume&quot;: true,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_block_hosting_volume_size&quot;: &quot;New block hosting volume will be created in size mentioned, This is considered only if auto-create is enabled.&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;block_hosting_volume_size&quot;: 500,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;_block_hosting_volume_options&quot;: &quot;New block hosting volume will be created with the following set of options. Removing the group gluster-block option is NOT recommended. Additional options can be added next to it separated by a comma.&quot;,</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">    &quot;block_hosting_volume_options&quot;: &quot;group gluster-block&quot;</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">  }</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">}</blockquote></div></div></div></div></div></div><div><div><div><div><div><div><br></div></div></div></div></div></div><div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[cmdexec] ERROR 2018/11/09 09:44:33 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/volume.go:145">github.com/heketi/heketi/executors/cmdexec/volume.go:145</a>: Unable to stop volume vol_41062d80d73fc9065d6f38344a5033b8: volume stop: vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume vol_41062d80d73fc9065d6f38344a5033b8 does not exist</blockquote></div></blockquote></div></blockquote><div dir="ltr"><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 55.106µs<br>[cmdexec] ERROR 2018/11/09 09:44:35 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;gluster --mode=script volume delete vol_41062d80d73fc9065d6f38344a5033b8&#39;] on glust-0:22: Err[Process exited with status 1]: Stdout []: Stderr [volume delete: vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume vol_41062d80d73fc9065d6f38344a5033b8 does not exist<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:35 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/volume.go:154">github.com/heketi/heketi/executors/cmdexec/volume.go:154</a>: Unable to delete volume vol_41062d80d73fc9065d6f38344a5033b8: volume delete: vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume vol_41062d80d73fc9065d6f38344a5033b8 does not exist<br>[heketi] INFO 2018/11/09 09:44:35 Deleting brick 140fa407e1dd1bbb5c342eaad1081a3e<br>[heketi] INFO 2018/11/09 09:44:35 Deleting brick b498b48679d9cbd3f380f865b95faa76<br>[heketi] INFO 2018/11/09 09:44:35 Deleting brick 4652460cce68f2d8ff6c4c34cb57e047<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 43.818µs<br>[cmdexec] ERROR 2018/11/09 09:44:36 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;umount /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e&#39;] on glust-2:22: Err[Process exited with status 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e: mountpoint not found<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:36 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/brick.go:198">github.com/heketi/heketi/executors/cmdexec/brick.go:198</a>: umount: /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e: mountpoint not found<br>[cmdexec] ERROR 2018/11/09 09:44:36 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;umount /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76&#39;] on glust-1:22: Err[Process exited with status 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76: mountpoint not found<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:36 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/brick.go:198">github.com/heketi/heketi/executors/cmdexec/brick.go:198</a>: umount: /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76: mountpoint not found<br>[cmdexec] ERROR 2018/11/09 09:44:36 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;umount /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047&#39;] on glust-0:22: Err[Process exited with status 32]: Stdout []: Stderr [umount: /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047: mountpoint not found<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:36 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/brick.go:198">github.com/heketi/heketi/executors/cmdexec/brick.go:198</a>: umount: /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047: mountpoint not found<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 50.819µs<br>[cmdexec] DEBUG 2018/11/09 09:44:37 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173</a>: Host: glust-2:22 Command: /bin/bash -c &#39;mount&#39;<br>Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)<br>proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)<br>udev on /dev type devtmpfs (rw,nosuid,relatime,size=8206656k,nr_inodes=2051664,mode=755)<br>devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)<br>tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755)<br>/dev/mapper/system-root on / type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)<br>securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)<br>tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)<br>tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)<br>tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)<br>cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)<br>pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)<br>cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)<br>cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)<br>cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)<br>cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)<br>cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)<br>cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)<br>cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)<br>cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)<br>cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)<br>systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11743)<br>debugfs on /sys/kernel/debug type debugfs (rw,relatime)<br>mqueue on /dev/mqueue type mqueue (rw,relatime)<br>hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)<br>/dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered)<br>fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)<br>[cmdexec] WARNING 2018/11/09 09:44:37 brick path [/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e] not mounted, assuming deleted<br>[cmdexec] DEBUG 2018/11/09 09:44:37 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173</a>: Host: glust-1:22 Command: /bin/bash -c &#39;mount&#39;<br>Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)<br>proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)<br>udev on /dev type devtmpfs (rw,nosuid,relatime,size=8206664k,nr_inodes=2051666,mode=755)<br>devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)<br>tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755)<br>/dev/mapper/system-root on / type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)<br>securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)<br>tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)<br>tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)<br>tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)<br>cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)<br>pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)<br>cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)<br>cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)<br>cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)<br>cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)<br>cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)<br>cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)<br>cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)<br>cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)<br>cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)<br>systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=861)<br>debugfs on /sys/kernel/debug type debugfs (rw,relatime)<br>mqueue on /dev/mqueue type mqueue (rw,relatime)<br>hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)<br>/dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered)<br>fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)<br>[cmdexec] WARNING 2018/11/09 09:44:37 brick path [/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76] not mounted, assuming deleted<br>[cmdexec] DEBUG 2018/11/09 09:44:37 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173</a>: Host: glust-0:22 Command: /bin/bash -c &#39;mount&#39;<br>Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)<br>proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)<br>udev on /dev type devtmpfs (rw,nosuid,relatime,size=8206664k,nr_inodes=2051666,mode=755)<br>devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)<br>tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755)<br>/dev/mapper/system-root on / type ext4 (rw,relatime,discard,errors=remount-ro,data=ordered)<br>securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)<br>tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)<br>tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)<br>tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)<br>cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)<br>pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)<br>cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)<br>cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)<br>cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)<br>cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)<br>cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)<br>cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)<br>cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)<br>cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)<br>cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)<br>systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1553)<br>mqueue on /dev/mqueue type mqueue (rw,relatime)<br>hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)<br>debugfs on /sys/kernel/debug type debugfs (rw,relatime)<br>/dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered)<br>fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[cmdexec] WARNING 2018/11/09 09:44:37 brick path [/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047] not mounted, assuming deleted<br>[negroni] Completed 200 OK in 16.726183ms<br>[cmdexec] DEBUG 2018/11/09 09:44:38 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173</a>: Host: glust-2:22 Command: /bin/bash -c &#39;sed -i.save &quot;/brick_140fa407e1dd1bbb5c342eaad1081a3e/d&quot; /etc/fstab&#39;<br>Result:<br>[cmdexec] DEBUG 2018/11/09 09:44:38 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173</a>: Host: glust-1:22 Command: /bin/bash -c &#39;sed -i.save &quot;/brick_b498b48679d9cbd3f380f865b95faa76/d&quot; /etc/fstab&#39;<br>Result:<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 50.054µs<br>[cmdexec] DEBUG 2018/11/09 09:44:38 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173</a>: Host: glust-0:22 Command: /bin/bash -c &#39;sed -i.save &quot;/brick_4652460cce68f2d8ff6c4c34cb57e047/d&quot; /etc/fstab&#39;<br>Result:<br>[cmdexec] ERROR 2018/11/09 09:44:39 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvremove --autobackup=n -f vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76&#39;] on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv: vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76<br>[cmdexec] ERROR 2018/11/09 09:44:39 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvremove --autobackup=n -f vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e&#39;] on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv: vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 47.799µs<br>[cmdexec] ERROR 2018/11/09 09:44:39 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvremove --autobackup=n -f vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047&#39;] on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv: vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047<br>[cmdexec] ERROR 2018/11/09 09:44:40 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvs --noheadings --options=thin_count vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76&#39;] on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin pool: vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76<br>[cmdexec] ERROR 2018/11/09 09:44:40 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvs --noheadings --options=thin_count vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e&#39;] on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin pool: vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 57.345µs<br>[cmdexec] ERROR 2018/11/09 09:44:40 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvs --noheadings --options=thin_count vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73&#39;] on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin pool: vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73<br>[cmdexec] ERROR 2018/11/09 09:44:41 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvremove --autobackup=n -f vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76&#39;] on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool: vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76<br>[cmdexec] ERROR 2018/11/09 09:44:41 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvremove --autobackup=n -f vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e&#39;] on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool: vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 67.474µs<br>[cmdexec] ERROR 2018/11/09 09:44:41 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;lvremove --autobackup=n -f vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73&#39;] on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [  Failed to find logical volume &quot;vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73&quot;<br>]<br>[cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool: vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73<br>[cmdexec] ERROR 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;rmdir /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76&#39;] on glust-1:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir: failed to remove &#39;/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76&#39;: No such file or directory<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/brick.go:279">github.com/heketi/heketi/executors/cmdexec/brick.go:279</a>: rmdir: failed to remove &#39;/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76&#39;: No such file or directory<br>[cmdexec] ERROR 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;rmdir /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e&#39;] on glust-2:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir: failed to remove &#39;/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e&#39;: No such file or directory<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/brick.go:279">github.com/heketi/heketi/executors/cmdexec/brick.go:279</a>: rmdir: failed to remove &#39;/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e&#39;: No such file or directory<br>[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d<br>[negroni] Completed 200 OK in 48.399µs<br>[cmdexec] ERROR 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169">github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169</a>: Failed to run command [/bin/bash -c &#39;rmdir /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047&#39;] on glust-0:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir: failed to remove &#39;/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047&#39;: No such file or directory<br>]<br>[cmdexec] ERROR 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/executors/cmdexec/brick.go:279">github.com/heketi/heketi/executors/cmdexec/brick.go:279</a>: rmdir: failed to remove &#39;/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047&#39;: No such file or directory<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/device_entry.go:411">github.com/heketi/heketi/apps/glusterfs/device_entry.go:411</a>: expected space needed for amount=10485760 snapFactor=1 : 10539008<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/device_entry.go:411">github.com/heketi/heketi/apps/glusterfs/device_entry.go:411</a>: expected space needed for amount=10485760 snapFactor=1 : 10539008<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/device_entry.go:411">github.com/heketi/heketi/apps/glusterfs/device_entry.go:411</a>: expected space needed for amount=10485760 snapFactor=1 : 10539008<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry.go:580">github.com/heketi/heketi/apps/glusterfs/volume_entry.go:580</a>: Using the following clusters: [1233163d3b76a6538a34d9b27a855ead]<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43">github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43</a>: brick_size = 10485760<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44">github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44</a>: sets = 1<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45">github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45</a>: num_bricks = 3<br>[heketi] INFO 2018/11/09 09:44:42 Allocating brick set #0<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251</a>: 0 / 3<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/device_entry.go:411">github.com/heketi/heketi/apps/glusterfs/device_entry.go:411</a>: expected space needed for amount=10485760 snapFactor=1 : 10539008<br>[heketi] DEBUG 2018/11/09 09:44:42 /src/<a href="http://github.com/heketi/heketi/apps/glusterfs/device_entry.go:368">github.com/heketi/heketi/apps/glusterfs/device_entry.go:368</a>: device d4b4ff32a56f249dcff9b4a88eef85f2[1073606656] &gt; required size [10539008] ?<br>panic: CHECK:<br><span style="white-space:pre">        </span>func (<a href="http://github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID">github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID</a>) 0x19e0c2b<br><span style="white-space:pre">        </span>File /var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/pkg/utils/uuid.go:34">github.com/heketi/heketi/pkg/utils/uuid.go:34</a><br><span style="white-space:pre">        </span>Info: [13 16]<br>goroutine 863 [running]:<br><a href="http://github.com/heketi/heketi/vendor/github.com/lpabon/godbc.dbc_panic(0x24aeb24">github.com/heketi/heketi/vendor/github.com/lpabon/godbc.dbc_panic(0x24aeb24</a>, 0x5, 0x0, 0xc4207fd7a0, 0x2, 0x2)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:85">github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:85</a> +0x40c<br><a href="http://github.com/heketi/heketi/vendor/github.com/lpabon/godbc.Check(0x224d900">github.com/heketi/heketi/vendor/github.com/lpabon/godbc.Check(0x224d900</a>, 0xc4207fd7a0, 0x2, 0x2)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:123">github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:123</a> +0x5d<br><a href="http://github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID(0x2f3d1e0">github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID(0x2f3d1e0</a>, 0xc4200741b0, 0x1, 0xc4207e3760)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/pkg/utils/uuid.go:34">github.com/heketi/heketi/pkg/utils/uuid.go:34</a> +0x19c<br><a href="http://github.com/heketi/heketi/pkg/utils.GenUUID(0x23fab80">github.com/heketi/heketi/pkg/utils.GenUUID(0x23fab80</a>, 0xc4207e3760)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/pkg/utils/uuid.go:42">github.com/heketi/heketi/pkg/utils/uuid.go:42</a> +0x39<br><a href="http://github.com/heketi/heketi/apps/glusterfs.NewBrickEntry(0xa00000">github.com/heketi/heketi/apps/glusterfs.NewBrickEntry(0xa00000</a>, 0xa00000, 0xd000, 0xc4207bdd00, 0x20, 0xc4207bdd20, 0x20, 0x0, 0xc42028c760, 0x20, ...)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_entry.go:64">github.com/heketi/heketi/apps/glusterfs/brick_entry.go:64</a> +0x119<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*DeviceEntry).NewBrickEntry(0xc4203ffd00">github.com/heketi/heketi/apps/glusterfs.(*DeviceEntry).NewBrickEntry(0xc4203ffd00</a>, 0xa00000, 0x3ff0000000000000, 0x0, 0xc42028c760, 0x20, 0xc4207a9100)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/device_entry.go:379">github.com/heketi/heketi/apps/glusterfs/device_entry.go:379</a> +0x239<br><a href="http://github.com/heketi/heketi/apps/glusterfs.tryAllocateBrickOnDevice(0x2f59a60">github.com/heketi/heketi/apps/glusterfs.tryAllocateBrickOnDevice(0x2f59a60</a>, 0xc42079ef40, 0x0, 0xc4203ffd00, 0xc4207fd760, 0x7)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:201">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:201</a> +0x1d2<br><a href="http://github.com/heketi/heketi/apps/glusterfs.findDeviceAndBrickForSet(0x2f59a60">github.com/heketi/heketi/apps/glusterfs.findDeviceAndBrickForSet(0x2f59a60</a>, 0xc42079ef40, 0xc4207a9350, 0x0, 0xc4205b25a0, 0xc4207fd760, 0xc42001c000, 0x6, 0x6, 0xc4205b2600)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:228">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:228</a> +0xd4<br><a href="http://github.com/heketi/heketi/apps/glusterfs.populateBrickSet(0x2f59a60">github.com/heketi/heketi/apps/glusterfs.populateBrickSet(0x2f59a60</a>, 0xc42079ef40, 0xc4207a9350, 0x0, 0xc4205b25a0, 0xc4207bd180, 0x20, 0x0, 0x0, 0x0, ...)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:253">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:253</a> +0x2ce<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*StandardBrickPlacer).PlaceAll(0x2ff8400">github.com/heketi/heketi/apps/glusterfs.(*StandardBrickPlacer).PlaceAll(0x2ff8400</a>, 0x2f4bfe0, 0xc420821650, 0x2f59a60, 0xc42079ef40, 0x0, 0x0, 0x0, 0x0)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:463">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:463</a> +0x336<br><a href="http://github.com/heketi/heketi/apps/glusterfs.allocateBricks.func1(0xc4208447e0">github.com/heketi/heketi/apps/glusterfs.allocateBricks.func1(0xc4208447e0</a>, 0x1, 0xc420821620)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:286">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:286</a> +0x1a5<br><a href="http://github.com/heketi/heketi/pkg/db.(*TxWrap).View(0xc420304e80">github.com/heketi/heketi/pkg/db.(*TxWrap).View(0xc420304e80</a>, 0xc420821620, 0xc4207a94f8, 0x11f6a4a)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/pkg/db/wrap.go:78">github.com/heketi/heketi/pkg/db/wrap.go:78</a> +0x33<br><a href="http://github.com/heketi/heketi/apps/glusterfs.allocateBricks(0x2f3d7e0">github.com/heketi/heketi/apps/glusterfs.allocateBricks(0x2f3d7e0</a>, 0xc420304e80, 0xc4207bcdc0, 0x20, 0xc4202d0580, 0x1, 0xa00000, 0x2fd4b40, 0x58, 0xa)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:282">github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:282</a> +0x108<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks.func2(0xc4208447e0">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks.func2(0xc4208447e0</a>, 0x23c56e0, 0xc4207a9501)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:468">github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:468</a> +0xf0<br><a href="http://github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304e10">github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304e10</a>, 0xc4206a4fc0, 0xc4207a96a0, 0xc4202d0580)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/pkg/db/wrap.go:87">github.com/heketi/heketi/pkg/db/wrap.go:87</a> +0x3f<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks(0xc4202d0580">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks(0xc4202d0580</a>, 0x2f48420, 0xc420304e10, 0xc4207bcdc0, 0x20, 0x1, 0xa00000, 0x3440000, 0x0, 0x2337d20, ...)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:466">github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:466</a> +0x14e<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricksInCluster(0xc4202d0580">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricksInCluster(0xc4202d0580</a>, 0x2f48420, 0xc420304e10, 0xc4207bcdc0, 0x20, 0xa, 0x140, 0xa6, 0xc4207a97e8, 0x1029a36, ...)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:54">github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:54</a> +0x378<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).tryAllocateBricks(0xc4202d0580">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).tryAllocateBricks(0xc4202d0580</a>, 0x2f48420, 0xc420304e10, 0xc420304d90, 0x1, 0x1, 0x28, 0x38, 0xc4206a4f80, 0x36a8288, ...)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry.go:436">github.com/heketi/heketi/apps/glusterfs/volume_entry.go:436</a> +0x183<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume.func1(0xc4208447e0">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume.func1(0xc4208447e0</a>, 0x23b00c0, 0x2f20401)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry.go:605">github.com/heketi/heketi/apps/glusterfs/volume_entry.go:605</a> +0xd1<br><a href="http://github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304af0">github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304af0</a>, 0xc4206a4f80, 0xc4207a99e0, 0x19dfe05)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/pkg/db/wrap.go:87">github.com/heketi/heketi/pkg/db/wrap.go:87</a> +0x3f<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume(0xc4202d0580">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume(0xc4202d0580</a>, 0x2f48420, 0xc420304af0, 0xc420304d90, 0x1, 0x1, 0x1, 0x1, 0xc420304d90, 0x1, ...)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry.go:602">github.com/heketi/heketi/apps/glusterfs/volume_entry.go:602</a> +0xfc<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).createVolumeComponents(0xc4202d0580">github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).createVolumeComponents(0xc4202d0580</a>, 0x2f48420, 0xc420304af0, 0xfffffffffffffffe, 0xc420302ae0, 0xc4207a9b40, 0x1029099, 0xc400000008)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/volume_entry.go:582">github.com/heketi/heketi/apps/glusterfs/volume_entry.go:582</a> +0x301<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build.func1(0xc4208447e0">github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build.func1(0xc4208447e0</a>, 0x2528490, 0xc4208447e0)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/operations.go:146">github.com/heketi/heketi/apps/glusterfs/operations.go:146</a> +0x7f<br><a href="http://github.com/heketi/heketi/vendor/github.com/boltdb/bolt.(*DB).Update(0xc420302960">github.com/heketi/heketi/vendor/github.com/boltdb/bolt.(*DB).Update(0xc420302960</a>, 0xc420304ae0, 0x0, 0x0)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/vendor/github.com/boltdb/bolt/db.go:595">github.com/heketi/heketi/vendor/github.com/boltdb/bolt/db.go:595</a> +0x9a<br><a href="http://github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build(0xc4208067b0">github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build(0xc4208067b0</a>, 0x2f5e620, 0xc420257800)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/operations.go:144">github.com/heketi/heketi/apps/glusterfs/operations.go:144</a> +0x6f<br><a href="http://github.com/heketi/heketi/apps/glusterfs.retryOperation(0x2f5a780">github.com/heketi/heketi/apps/glusterfs.retryOperation(0x2f5a780</a>, 0xc4208067b0, 0x2f5e620, 0xc420257800, 0x1, 0x1)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/operations_manage.go:175">github.com/heketi/heketi/apps/glusterfs/operations_manage.go:175</a> +0x217<br><a href="http://github.com/heketi/heketi/apps/glusterfs.AsyncHttpOperation.func1(0x0">github.com/heketi/heketi/apps/glusterfs.AsyncHttpOperation.func1(0x0</a>, 0x0, 0x0, 0x0)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/apps/glusterfs/operations_manage.go:104">github.com/heketi/heketi/apps/glusterfs/operations_manage.go:104</a> +0x552<br><a href="http://github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle.func1(0xc420818580">github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle.func1(0xc420818580</a>, 0xc42084ac30)<br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:291">github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:291</a> +0xf6<br>created by <a href="http://github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle">github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle</a><br><span style="white-space:pre">        </span>/var/tmp/gtg-wrapper/src/<a href="http://github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:287">github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:287</a> +0x49</blockquote></div></blockquote></div></div>