<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-CA" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal">Hi everybody,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am testing heketi autoprovisioning of gluster volumes and I am hitting a problem where heketi reports "No space" in the log, even there should be a space.
<o:p></o:p></p>
<p class="MsoNormal">Would somebody be able to point me what I need to look into or where my problem is? Here are the details of my problem:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I am using a curl to simulate my failing request from kubernetes. As a side note, our kubernetes is old, 1.4 and therefore I cannot
<o:p></o:p></p>
<p class="MsoNormal">robert@workstation:~/src/ifds/helm/tapremium$ curl -v -X POST -H "Authorization: bearer ****" -H "Content-Type: application/json" POST http://localhost:8080/volumes
<a href="mailto:-d@/tmp/volume.json">-d@/tmp/volume.json</a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">=================================================================================================================<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is a content of /tmp/volume.json file<o:p></o:p></p>
<p class="MsoNormal">{<o:p></o:p></p>
<p class="MsoNormal"> "size": 10,<o:p></o:p></p>
<p class="MsoNormal"> "name": "",<o:p></o:p></p>
<p class="MsoNormal"> "durability": {<o:p></o:p></p>
<p class="MsoNormal"> "type": "replicate",<o:p></o:p></p>
<p class="MsoNormal"> "replicate": {<o:p></o:p></p>
<p class="MsoNormal"> "replica": 3<o:p></o:p></p>
<p class="MsoNormal"> },<o:p></o:p></p>
<p class="MsoNormal"> "disperse": {}<o:p></o:p></p>
<p class="MsoNormal"> },<o:p></o:p></p>
<p class="MsoNormal"> "gid": 2000,<o:p></o:p></p>
<p class="MsoNormal"> "snapshot": {<o:p></o:p></p>
<p class="MsoNormal"> "enable": false,<o:p></o:p></p>
<p class="MsoNormal"> "factor": 0<o:p></o:p></p>
<p class="MsoNormal"> }<o:p></o:p></p>
<p class="MsoNormal">}<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">This is my topology<o:p></o:p></p>
<p class="MsoNormal">=================================================================================================================<o:p></o:p></p>
<p class="MsoNormal">Cluster Id: 7396c8b03593f9bfa1288cdebd9412ac<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Volumes:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Name: vol_4708719dddaacda301e3b71763cb73fa<o:p></o:p></p>
<p class="MsoNormal"> Size: 10<o:p></o:p></p>
<p class="MsoNormal"> Id: 4708719dddaacda301e3b71763cb73fa<o:p></o:p></p>
<p class="MsoNormal"> Cluster Id: 7396c8b03593f9bfa1288cdebd9412ac<o:p></o:p></p>
<p class="MsoNormal"> Mount: 192.168.48.9:vol_4708719dddaacda301e3b71763cb73fa<o:p></o:p></p>
<p class="MsoNormal"> Mount Options: backup-volfile-servers=<o:p></o:p></p>
<p class="MsoNormal"> Durability Type: none<o:p></o:p></p>
<p class="MsoNormal"> Snapshot: Disabled<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Bricks:<o:p></o:p></p>
<p class="MsoNormal"> Id: 99b33f7f23347c1691e4a93c7220af04<o:p></o:p></p>
<p class="MsoNormal"> Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_99b33f7f23347c1691e4a93c7220af04/brick<o:p></o:p></p>
<p class="MsoNormal"> Size (GiB): 5<o:p></o:p></p>
<p class="MsoNormal"> Node: 31e1ac3014029ff48398a989c10952b6<o:p></o:p></p>
<p class="MsoNormal"> Device: 05df77bacca68b941f8d3d20d64a1aa2<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Id: acebc8961ffb304fa2b97c201c7af8da<o:p></o:p></p>
<p class="MsoNormal"> Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_acebc8961ffb304fa2b97c201c7af8da/brick<o:p></o:p></p>
<p class="MsoNormal"> Size (GiB): 5<o:p></o:p></p>
<p class="MsoNormal"> Node: 31e1ac3014029ff48398a989c10952b6<o:p></o:p></p>
<p class="MsoNormal"> Device: 05df77bacca68b941f8d3d20d64a1aa2<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Nodes:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"> Node Id: 31e1ac3014029ff48398a989c10952b6<o:p></o:p></p>
<p class="MsoNormal"> State: online<o:p></o:p></p>
<p class="MsoNormal"> Cluster Id: 7396c8b03593f9bfa1288cdebd9412ac<o:p></o:p></p>
<p class="MsoNormal"> Zone: 1<o:p></o:p></p>
<p class="MsoNormal"> Management Hostname: 192.168.48.9<o:p></o:p></p>
<p class="MsoNormal"> Storage Hostname: 192.168.48.9<o:p></o:p></p>
<p class="MsoNormal"> Devices:<o:p></o:p></p>
<p class="MsoNormal"> Id:05df77bacca68b941f8d3d20d64a1aa2 Name:/dev/gluster_vg/heketi_volumeState:online Size (GiB):99 Used (GiB):10 Free (GiB):89
<o:p></o:p></p>
<p class="MsoNormal"> Bricks:<o:p></o:p></p>
<p class="MsoNormal"> Id:99b33f7f23347c1691e4a93c7220af04 Size (GiB):5 Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_99b33f7f23347c1691e4a93c7220af04/brick<o:p></o:p></p>
<p class="MsoNormal"> Id:acebc8961ffb304fa2b97c201c7af8da Size (GiB):5 Path: /var/lib/heketi/mounts/vg_05df77bacca68b941f8d3d20d64a1aa2/brick_acebc8961ffb304fa2b97c201c7af8da/brick<o:p></o:p></p>
<p class="MsoNormal"> Id:6c0325d31e1b84a4f3aa088ba352e2ec Name:/dev/gluster_vg/heketi_volume2State:online Size (GiB):24 Used (GiB):0 Free (GiB):24
<o:p></o:p></p>
<p class="MsoNormal"> Bricks:<o:p></o:p></p>
<p class="MsoNormal"> Id:bac78b5e4d558d387218d12fdfb7bfa0 Name:/dev/gluster_vg/heketi_volume1State:online Size (GiB):24 Used (GiB):0 Free (GiB):24
<o:p></o:p></p>
<p class="MsoNormal"> Bricks:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">=================================================================================================================<o:p></o:p></p>
<p class="MsoNormal">and this is the log output<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[negroni] Started POST /volumes<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:101: [e8cb84a8b977a00d77e137a05a5b2530] Replica 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:580: Using the following clusters: [7396c8b03593f9bfa1288cdebd9412ac]<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 10485760<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 1<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=10485760 snapFactor=1 : 10539008<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device 6c0325d31e1b84a4f3aa088ba352e2ec[26079232] > required size [10539008] ?<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0)
<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 5242880<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 2<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 6<o:p></o:p></p>
<p class="MsoNormal">[heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=5242880 snapFactor=1 : 5271552<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device bac78b5e4d558d387218d12fdfb7bfa0[26079232] > required size [5271552] ?<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0)
<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 2621440<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 4<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 12<o:p></o:p></p>
<p class="MsoNormal">[heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=2621440 snapFactor=1 : 2637824<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device 05df77bacca68b941f8d3d20d64a1aa2[94179328] > required size [2637824] ?<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0)
<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:43: brick_size = 1310720<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:44: sets = 8<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:45: num_bricks = 24<o:p></o:p></p>
<p class="MsoNormal">[heketi] INFO 2018/10/25 14:08:28 Allocating brick set #0<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 0 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:411: expected space needed for amount=1310720 snapFactor=1 : 1318912<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/device_entry.go:368: device 05df77bacca68b941f8d3d20d64a1aa2[94179328] > required size [1318912] ?<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/brick_allocate.go:251: 1 / 3<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:455: Error detected. Cleaning up volume e8cb84a8b977a00d77e137a05a5b2530: Len(0)
<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:56: No space, re-trying with smaller brick size<o:p></o:p></p>
<p class="MsoNormal">[heketi] ERROR 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry_allocate.go:37: Minimum brick size limit reached. Out of space.<o:p></o:p></p>
<p class="MsoNormal">[heketi] DEBUG 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/volume_entry.go:445: Cluster 7396c8b03593f9bfa1288cdebd9412ac can not accommodate volume (Minimum brick size limit reached. Out of space.), trying next cluster<o:p></o:p></p>
<p class="MsoNormal">[heketi] ERROR 2018/10/25 14:08:28 /src/github.com/heketi/heketi/apps/glusterfs/operations_manage.go:89: Create Volume Build Failed: No space<o:p></o:p></p>
<p class="MsoNormal">[negroni] Completed 500 Internal Server Error in 3.397009ms<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial",sans-serif;color:#5B6770">This message is marked
<b>Public</b></span><o:p></o:p></p>
</div>
<DIV><P><HR>
Please consider the environment before printing this email and any attachments.<BR>
<BR>
This e-mail and any attachments are intended only for the individual or company to which it is addressed and may contain information which is privileged, confidential and prohibited from disclosure or unauthorized use under applicable law. If you are not the intended recipient of this e-mail, you are hereby notified that any use, dissemination, or copying of this e-mail or the information contained in this e-mail is strictly prohibited by the sender. If you have received this transmission in error, please return the material received to the sender and delete all copies from your system.
</P></DIV>
</body>
</html>