[heketi-devel] Failed to mount Volumes
Junaid Subhani
ijunaidsubhani at gmail.com
Fri Nov 9 15:00:28 UTC 2018
Hello Folks,
I am new to heketi and was following the documentation word to work.
I successfully created a 3 node topolgy and the cluster was successfully
created.
./heketi-cli --server http://localhost:8085 topology load
--json=topology.json
Creating cluster ... ID: 1233163d3b76a6538a34d9b27a855ead
Allowing file volumes on cluster.
Allowing block volumes on cluster.
Creating node glust-0 ... ID: 82af3c3335b7d24d0e5fc7559d44f8d7
Adding device /dev/sdb ... OK
Adding device /dev/sdc ... OK
Creating node glust-1 ... ID: ec106227ea3580571a252b28ad223109
Adding device /dev/sdb ... OK
Adding device /dev/sdc ... OK
Creating node glust-2 ... ID: fc6535a147b39d65e7037015dcc0cc20
Adding device /dev/sdb ... OK
Adding device /dev/sdc ... OK
And on all 3 nodes, I can see that I have successfully peered.
glust-1:~# gluster volume info
No volumes present
glust-1:~# gluster peer status
Number of Peers: 2
Hostname: glust-0
Uuid: 22891efe-28ce-4c12-b97a-42ee78d50995
State: Peer in Cluster (Connected)
Hostname: 10.27.239.240
Uuid: fc5834ec-2b7e-402b-a412-6783bae0db66
State: Peer in Cluster (Connected)
But when I create a volume,
I get
./heketi-cli --server http://localhost:8085 volume create --size=10
> Error: Get http://localhost:8085/queue/729862db18ef8d968eb7d0017c09b04d:
> dial tcp [::1]:8085: getsockopt: connection refused
And heketi crashes after spitting out HUGE amounts of logs. My client
binaries are the latest Darwin ones.
My heketi.json looks like:
>
> ╰─ cat heketi.json
{
"_port_comment": "Heketi Server Port Number",
"port": "8085",
"_enable_tls_comment": "Enable TLS in Heketi Server",
"enable_tls": false,
"_cert_file_comment": "Path to a valid certificate file",
"cert_file": "",
"_key_file_comment": "Path to a valid private key file",
"key_file": "",
> "_use_auth": "Enable JWT authorization. Please enable for deployment",
"use_auth": false,
"_jwt": "Private keys for access",
"jwt": {
"_admin": "Admin has access to all APIs",
"admin": {
"key": "My Secret"
},
"_user": "User only has access to /volumes endpoint",
"user": {
"key": "My Secret"
}
},
"_backup_db_to_kube_secret": "Backup the heketi database to a Kubernetes
> secret when running in Kubernetes. Default is off.",
"backup_db_to_kube_secret": false,
"_glusterfs_comment": "GlusterFS Configuration",
"glusterfs": {
"_executor_comment": [
"Execute plugin. Possible choices: mock, ssh",
"mock: This setting is used for testing and development.",
" It will not send commands to any node.",
"ssh: This setting will notify Heketi to ssh to the nodes.",
" It will need the values in sshexec to be configured.",
"kubernetes: Communicate with GlusterFS containers over",
" Kubernetes exec api."
> ],
> "executor": "ssh",
> "_sshexec_comment": "SSH username and private key file information",
> "sshexec": {
> "keyfile": "id_rsa",
> "user": "root",
> "port": "22",
> "fstab": "/etc/fstab",
> "backup_lvm_metadata": false
},
> "_db_comment": "Database file name",
"db": "heketi.db",
"_refresh_time_monitor_gluster_nodes": "Refresh time in seconds to
> monitor Gluster nodes",
"refresh_time_monitor_gluster_nodes": 120,
"_start_time_monitor_gluster_nodes": "Start time in seconds to monitor
> Gluster nodes when the heketi comes up",
"start_time_monitor_gluster_nodes": 10,
"_loglevel_comment": [
"Set log level. Choices are:",
" none, critical, error, warning, info, debug",
"Default is warning"
],
"loglevel" : "debug",
"_auto_create_block_hosting_volume": "Creates Block Hosting volumes
> automatically if not found or exsisting volume exhausted",
"auto_create_block_hosting_volume": true,
"_block_hosting_volume_size": "New block hosting volume will be created
> in size mentioned, This is considered only if auto-create is enabled.",
"block_hosting_volume_size": 500,
"_block_hosting_volume_options": "New block hosting volume will be
> created with the following set of options. Removing the group gluster-block
> option is NOT recommended. Additional options can be added next to it
> separated by a comma.",
"block_hosting_volume_options": "group gluster-block"
}
}
[cmdexec] ERROR 2018/11/09 09:44:33 /src/
> github.com/heketi/heketi/executors/cmdexec/volume.go:145: Unable to stop
> volume vol_41062d80d73fc9065d6f38344a5033b8: volume stop:
> vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume
> vol_41062d80d73fc9065d6f38344a5033b8 does not exist
[negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 55.106µs
> [cmdexec] ERROR 2018/11/09 09:44:35 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'gluster --mode=script volume delete
> vol_41062d80d73fc9065d6f38344a5033b8'] on glust-0:22: Err[Process exited
> with status 1]: Stdout []: Stderr [volume delete:
> vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume
> vol_41062d80d73fc9065d6f38344a5033b8 does not exist
> ]
> [cmdexec] ERROR 2018/11/09 09:44:35 /src/
> github.com/heketi/heketi/executors/cmdexec/volume.go:154: Unable to
> delete volume vol_41062d80d73fc9065d6f38344a5033b8: volume delete:
> vol_41062d80d73fc9065d6f38344a5033b8: failed: Volume
> vol_41062d80d73fc9065d6f38344a5033b8 does not exist
> [heketi] INFO 2018/11/09 09:44:35 Deleting brick
> 140fa407e1dd1bbb5c342eaad1081a3e
> [heketi] INFO 2018/11/09 09:44:35 Deleting brick
> b498b48679d9cbd3f380f865b95faa76
> [heketi] INFO 2018/11/09 09:44:35 Deleting brick
> 4652460cce68f2d8ff6c4c34cb57e047
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 43.818µs
> [cmdexec] ERROR 2018/11/09 09:44:36 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'umount
> /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e']
> on glust-2:22: Err[Process exited with status 32]: Stdout []: Stderr
> [umount:
> /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e:
> mountpoint not found
> ]
> [cmdexec] ERROR 2018/11/09 09:44:36 /src/
> github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount:
> /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e:
> mountpoint not found
> [cmdexec] ERROR 2018/11/09 09:44:36 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'umount
> /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76']
> on glust-1:22: Err[Process exited with status 32]: Stdout []: Stderr
> [umount:
> /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76:
> mountpoint not found
> ]
> [cmdexec] ERROR 2018/11/09 09:44:36 /src/
> github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount:
> /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76:
> mountpoint not found
> [cmdexec] ERROR 2018/11/09 09:44:36 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'umount
> /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047']
> on glust-0:22: Err[Process exited with status 32]: Stdout []: Stderr
> [umount:
> /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047:
> mountpoint not found
> ]
> [cmdexec] ERROR 2018/11/09 09:44:36 /src/
> github.com/heketi/heketi/executors/cmdexec/brick.go:198: umount:
> /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047:
> mountpoint not found
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 50.819µs
> [cmdexec] DEBUG 2018/11/09 09:44:37 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-2:22
> Command: /bin/bash -c 'mount'
> Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> udev on /dev type devtmpfs
> (rw,nosuid,relatime,size=8206656k,nr_inodes=2051664,mode=755)
> devpts on /dev/pts type devpts
> (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755)
> /dev/mapper/system-root on / type ext4
> (rw,relatime,discard,errors=remount-ro,data=ordered)
> securityfs on /sys/kernel/security type securityfs
> (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup
> (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> cgroup on /sys/fs/cgroup/cpuset type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpuset)
> cgroup on /sys/fs/cgroup/perf_event type cgroup
> (rw,nosuid,nodev,noexec,relatime,perf_event)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
> (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> cgroup on /sys/fs/cgroup/freezer type cgroup
> (rw,nosuid,nodev,noexec,relatime,freezer)
> cgroup on /sys/fs/cgroup/blkio type cgroup
> (rw,nosuid,nodev,noexec,relatime,blkio)
> cgroup on /sys/fs/cgroup/memory type cgroup
> (rw,nosuid,nodev,noexec,relatime,memory)
> cgroup on /sys/fs/cgroup/pids type cgroup
> (rw,nosuid,nodev,noexec,relatime,pids)
> cgroup on /sys/fs/cgroup/devices type cgroup
> (rw,nosuid,nodev,noexec,relatime,devices)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs
> (rw,relatime,fd=34,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=11743)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> mqueue on /dev/mqueue type mqueue (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> /dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered)
> fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
> [cmdexec] WARNING 2018/11/09 09:44:37 brick path
> [/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e]
> not mounted, assuming deleted
> [cmdexec] DEBUG 2018/11/09 09:44:37 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-1:22
> Command: /bin/bash -c 'mount'
> Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> udev on /dev type devtmpfs
> (rw,nosuid,relatime,size=8206664k,nr_inodes=2051666,mode=755)
> devpts on /dev/pts type devpts
> (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755)
> /dev/mapper/system-root on / type ext4
> (rw,relatime,discard,errors=remount-ro,data=ordered)
> securityfs on /sys/kernel/security type securityfs
> (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup
> (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
> cgroup on /sys/fs/cgroup/cpuset type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpuset)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
> (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> cgroup on /sys/fs/cgroup/memory type cgroup
> (rw,nosuid,nodev,noexec,relatime,memory)
> cgroup on /sys/fs/cgroup/perf_event type cgroup
> (rw,nosuid,nodev,noexec,relatime,perf_event)
> cgroup on /sys/fs/cgroup/freezer type cgroup
> (rw,nosuid,nodev,noexec,relatime,freezer)
> cgroup on /sys/fs/cgroup/blkio type cgroup
> (rw,nosuid,nodev,noexec,relatime,blkio)
> cgroup on /sys/fs/cgroup/pids type cgroup
> (rw,nosuid,nodev,noexec,relatime,pids)
> cgroup on /sys/fs/cgroup/devices type cgroup
> (rw,nosuid,nodev,noexec,relatime,devices)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs
> (rw,relatime,fd=38,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=861)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> mqueue on /dev/mqueue type mqueue (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> /dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered)
> fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
> [cmdexec] WARNING 2018/11/09 09:44:37 brick path
> [/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76]
> not mounted, assuming deleted
> [cmdexec] DEBUG 2018/11/09 09:44:37 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-0:22
> Command: /bin/bash -c 'mount'
> Result: sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> udev on /dev type devtmpfs
> (rw,nosuid,relatime,size=8206664k,nr_inodes=2051666,mode=755)
> devpts on /dev/pts type devpts
> (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=1643676k,mode=755)
> /dev/mapper/system-root on / type ext4
> (rw,relatime,discard,errors=remount-ro,data=ordered)
> securityfs on /sys/kernel/security type securityfs
> (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
> tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup
> (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/lib/systemd/systemd-cgroups-agent,name=systemd)
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
> (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
> cgroup on /sys/fs/cgroup/cpuset type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpuset)
> cgroup on /sys/fs/cgroup/perf_event type cgroup
> (rw,nosuid,nodev,noexec,relatime,perf_event)
> cgroup on /sys/fs/cgroup/memory type cgroup
> (rw,nosuid,nodev,noexec,relatime,memory)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
> (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
> cgroup on /sys/fs/cgroup/pids type cgroup
> (rw,nosuid,nodev,noexec,relatime,pids)
> cgroup on /sys/fs/cgroup/blkio type cgroup
> (rw,nosuid,nodev,noexec,relatime,blkio)
> cgroup on /sys/fs/cgroup/devices type cgroup
> (rw,nosuid,nodev,noexec,relatime,devices)
> cgroup on /sys/fs/cgroup/freezer type cgroup
> (rw,nosuid,nodev,noexec,relatime,freezer)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs
> (rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1553)
> mqueue on /dev/mqueue type mqueue (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> /dev/mapper/system-var on /var type ext4 (rw,relatime,discard,data=ordered)
> fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [cmdexec] WARNING 2018/11/09 09:44:37 brick path
> [/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047]
> not mounted, assuming deleted
> [negroni] Completed 200 OK in 16.726183ms
> [cmdexec] DEBUG 2018/11/09 09:44:38 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-2:22
> Command: /bin/bash -c 'sed -i.save
> "/brick_140fa407e1dd1bbb5c342eaad1081a3e/d" /etc/fstab'
> Result:
> [cmdexec] DEBUG 2018/11/09 09:44:38 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-1:22
> Command: /bin/bash -c 'sed -i.save
> "/brick_b498b48679d9cbd3f380f865b95faa76/d" /etc/fstab'
> Result:
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 50.054µs
> [cmdexec] DEBUG 2018/11/09 09:44:38 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:173: Host: glust-0:22
> Command: /bin/bash -c 'sed -i.save
> "/brick_4652460cce68f2d8ff6c4c34cb57e047/d" /etc/fstab'
> Result:
> [cmdexec] ERROR 2018/11/09 09:44:39 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvremove --autobackup=n -f
> vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76']
> on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv:
> vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76
> [cmdexec] ERROR 2018/11/09 09:44:39 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvremove --autobackup=n -f
> vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e']
> on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv:
> vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 47.799µs
> [cmdexec] ERROR 2018/11/09 09:44:39 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvremove --autobackup=n -f
> vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047']
> on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:39 did not delete missing lv:
> vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047
> [cmdexec] ERROR 2018/11/09 09:44:40 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvs --noheadings --options=thin_count
> vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76']
> on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin
> pool:
> vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76
> [cmdexec] ERROR 2018/11/09 09:44:40 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvs --noheadings --options=thin_count
> vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e']
> on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin
> pool:
> vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 57.345µs
> [cmdexec] ERROR 2018/11/09 09:44:40 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvs --noheadings --options=thin_count
> vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73']
> on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:40 unable to count lvs in missing thin
> pool:
> vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73
> [cmdexec] ERROR 2018/11/09 09:44:41 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvremove --autobackup=n -f
> vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76']
> on glust-1:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool:
> vg_b1cb327b466d7b9f0bcaab81cf1dd721/tp_b498b48679d9cbd3f380f865b95faa76
> [cmdexec] ERROR 2018/11/09 09:44:41 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvremove --autobackup=n -f
> vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e']
> on glust-2:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool:
> vg_8026f612718d4590a18b83a09b5ba019/tp_140fa407e1dd1bbb5c342eaad1081a3e
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 67.474µs
> [cmdexec] ERROR 2018/11/09 09:44:41 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'lvremove --autobackup=n -f
> vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73']
> on glust-0:22: Err[Process exited with status 5]: Stdout []: Stderr [
> Failed to find logical volume
> "vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73"
> ]
> [cmdexec] WARNING 2018/11/09 09:44:41 did not delete missing thin pool:
> vg_fcb58b98b306b0ce7f357e40c9f7a780/tp_f2bfbabcf9fed35efd0b8a1d6270ad73
> [cmdexec] ERROR 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'rmdir
> /var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76']
> on glust-1:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir:
> failed to remove
> '/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76':
> No such file or directory
> ]
> [cmdexec] ERROR 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/executors/cmdexec/brick.go:279: rmdir: failed to
> remove
> '/var/lib/heketi/mounts/vg_b1cb327b466d7b9f0bcaab81cf1dd721/brick_b498b48679d9cbd3f380f865b95faa76':
> No such file or directory
> [cmdexec] ERROR 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'rmdir
> /var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e']
> on glust-2:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir:
> failed to remove
> '/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e':
> No such file or directory
> ]
> [cmdexec] ERROR 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/executors/cmdexec/brick.go:279: rmdir: failed to
> remove
> '/var/lib/heketi/mounts/vg_8026f612718d4590a18b83a09b5ba019/brick_140fa407e1dd1bbb5c342eaad1081a3e':
> No such file or directory
> [negroni] Started GET /queue/729862db18ef8d968eb7d0017c09b04d
> [negroni] Completed 200 OK in 48.399µs
> [cmdexec] ERROR 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/pkg/utils/ssh/ssh.go:169: Failed to run command
> [/bin/bash -c 'rmdir
> /var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047']
> on glust-0:22: Err[Process exited with status 1]: Stdout []: Stderr [rmdir:
> failed to remove
> '/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047':
> No such file or directory
> ]
> [cmdexec] ERROR 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/executors/cmdexec/brick.go:279: rmdir: failed to
> remove
> '/var/lib/heketi/mounts/vg_fcb58b98b306b0ce7f357e40c9f7a780/brick_4652460cce68f2d8ff6c4c34cb57e047':
> No such file or directory
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected
> space needed for amount=10485760 snapFactor=1 : 10539008
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected
> space needed for amount=10485760 snapFactor=1 : 10539008
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected
> space needed for amount=10485760 snapFactor=1 : 10539008
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry.go:580: Using the
> following clusters: [1233163d3b76a6538a34d9b27a855ead]
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43:
> brick_size = 10485760
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets
> = 1
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45:
> num_bricks = 3
> [heketi] INFO 2018/11/09 09:44:42 Allocating brick set #0
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected
> space needed for amount=10485760 snapFactor=1 : 10539008
> [heketi] DEBUG 2018/11/09 09:44:42 /src/
> github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device
> d4b4ff32a56f249dcff9b4a88eef85f2[1073606656] > required size [10539008] ?
> panic: CHECK:
> func (github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID) 0x19e0c2b
> File /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/pkg/utils/uuid.go:34
> Info: [13 16]
> goroutine 863 [running]:
> github.com/heketi/heketi/vendor/github.com/lpabon/godbc.dbc_panic(0x24aeb24,
> 0x5, 0x0, 0xc4207fd7a0, 0x2, 0x2)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:85 +0x40c
> github.com/heketi/heketi/vendor/github.com/lpabon/godbc.Check(0x224d900,
> 0xc4207fd7a0, 0x2, 0x2)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/vendor/github.com/lpabon/godbc/godbc.go:123 +0x5d
> github.com/heketi/heketi/pkg/utils.IdSource.ReadUUID(0x2f3d1e0,
> 0xc4200741b0, 0x1, 0xc4207e3760)
> /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/utils/uuid.go:34
> +0x19c
> github.com/heketi/heketi/pkg/utils.GenUUID(0x23fab80, 0xc4207e3760)
> /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/utils/uuid.go:42
> +0x39
> github.com/heketi/heketi/apps/glusterfs.NewBrickEntry(0xa00000, 0xa00000,
> 0xd000, 0xc4207bdd00, 0x20, 0xc4207bdd20, 0x20, 0x0, 0xc42028c760, 0x20,
> ...)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_entry.go:64 +0x119
>
> github.com/heketi/heketi/apps/glusterfs.(*DeviceEntry).NewBrickEntry(0xc4203ffd00,
> 0xa00000, 0x3ff0000000000000, 0x0, 0xc42028c760, 0x20, 0xc4207a9100)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/device_entry.go:379 +0x239
> github.com/heketi/heketi/apps/glusterfs.tryAllocateBrickOnDevice(0x2f59a60,
> 0xc42079ef40, 0x0, 0xc4203ffd00, 0xc4207fd760, 0x7)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:201 +0x1d2
> github.com/heketi/heketi/apps/glusterfs.findDeviceAndBrickForSet(0x2f59a60,
> 0xc42079ef40, 0xc4207a9350, 0x0, 0xc4205b25a0, 0xc4207fd760, 0xc42001c000,
> 0x6, 0x6, 0xc4205b2600)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:228 +0xd4
> github.com/heketi/heketi/apps/glusterfs.populateBrickSet(0x2f59a60,
> 0xc42079ef40, 0xc4207a9350, 0x0, 0xc4205b25a0, 0xc4207bd180, 0x20, 0x0,
> 0x0, 0x0, ...)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:253 +0x2ce
>
> github.com/heketi/heketi/apps/glusterfs.(*StandardBrickPlacer).PlaceAll(0x2ff8400,
> 0x2f4bfe0, 0xc420821650, 0x2f59a60, 0xc42079ef40, 0x0, 0x0, 0x0, 0x0)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:463 +0x336
> github.com/heketi/heketi/apps/glusterfs.allocateBricks.func1(0xc4208447e0,
> 0x1, 0xc420821620)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:286 +0x1a5
> github.com/heketi/heketi/pkg/db.(*TxWrap).View(0xc420304e80,
> 0xc420821620, 0xc4207a94f8, 0x11f6a4a)
> /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/db/wrap.go:78 +0x33
> github.com/heketi/heketi/apps/glusterfs.allocateBricks(0x2f3d7e0,
> 0xc420304e80, 0xc4207bcdc0, 0x20, 0xc4202d0580, 0x1, 0xa00000, 0x2fd4b40,
> 0x58, 0xa)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:282 +0x108
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks.func2(0xc4208447e0,
> 0x23c56e0, 0xc4207a9501)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:468 +0xf0
> github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304e10,
> 0xc4206a4fc0, 0xc4207a96a0, 0xc4202d0580)
> /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/db/wrap.go:87 +0x3f
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricks(0xc4202d0580,
> 0x2f48420, 0xc420304e10, 0xc4207bcdc0, 0x20, 0x1, 0xa00000, 0x3440000, 0x0,
> 0x2337d20, ...)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:466
> +0x14e
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).allocBricksInCluster(0xc4202d0580,
> 0x2f48420, 0xc420304e10, 0xc4207bcdc0, 0x20, 0xa, 0x140, 0xa6,
> 0xc4207a97e8, 0x1029a36, ...)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:54 +0x378
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).tryAllocateBricks(0xc4202d0580,
> 0x2f48420, 0xc420304e10, 0xc420304d90, 0x1, 0x1, 0x28, 0x38, 0xc4206a4f80,
> 0x36a8288, ...)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry.go:436 +0x183
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume.func1(0xc4208447e0,
> 0x23b00c0, 0x2f20401)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry.go:605 +0xd1
> github.com/heketi/heketi/pkg/db.(*TxWrap).Update(0xc420304af0,
> 0xc4206a4f80, 0xc4207a99e0, 0x19dfe05)
> /var/tmp/gtg-wrapper/src/github.com/heketi/heketi/pkg/db/wrap.go:87 +0x3f
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).saveCreateVolume(0xc4202d0580,
> 0x2f48420, 0xc420304af0, 0xc420304d90, 0x1, 0x1, 0x1, 0x1, 0xc420304d90,
> 0x1, ...)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry.go:602 +0xfc
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeEntry).createVolumeComponents(0xc4202d0580,
> 0x2f48420, 0xc420304af0, 0xfffffffffffffffe, 0xc420302ae0, 0xc4207a9b40,
> 0x1029099, 0xc400000008)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/volume_entry.go:582 +0x301
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build.func1(0xc4208447e0,
> 0x2528490, 0xc4208447e0)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/operations.go:146 +0x7f
>
> github.com/heketi/heketi/vendor/github.com/boltdb/bolt.(*DB).Update(0xc420302960,
> 0xc420304ae0, 0x0, 0x0)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/vendor/github.com/boltdb/bolt/db.go:595 +0x9a
>
> github.com/heketi/heketi/apps/glusterfs.(*VolumeCreateOperation).Build(0xc4208067b0,
> 0x2f5e620, 0xc420257800)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/operations.go:144 +0x6f
> github.com/heketi/heketi/apps/glusterfs.retryOperation(0x2f5a780,
> 0xc4208067b0, 0x2f5e620, 0xc420257800, 0x1, 0x1)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/operations_manage.go:175 +0x217
> github.com/heketi/heketi/apps/glusterfs.AsyncHttpOperation.func1(0x0,
> 0x0, 0x0, 0x0)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/apps/glusterfs/operations_manage.go:104 +0x552
>
> github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle.func1(0xc420818580,
> 0xc42084ac30)
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:291
> +0xf6
> created by
> github.com/heketi/heketi/vendor/github.com/heketi/rest.(*AsyncHttpHandler).handle
> /var/tmp/gtg-wrapper/src/
> github.com/heketi/heketi/vendor/github.com/heketi/rest/asynchttp.go:287
> +0x49
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/heketi-devel/attachments/20181109/9d0b18ae/attachment-0001.html>
More information about the heketi-devel
mailing list