From acucciarre at cloudian.com Wed May 19 10:17:47 2021 From: acucciarre at cloudian.com (Andrea Cucciarre') Date: Wed, 19 May 2021 12:17:47 +0200 Subject: [heketi-devel] Create Volume Build Failed: No space Message-ID: Hello, I'm trying to create a large volume but it fails shortly showing "No space", though there is plenty of space. The debug logs doesn't say much: [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #122 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #123 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #124 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #125 [heketi] INFO 2021/05/19 08:11:17 Allocating brick set #126 [heketi] ERROR 2021/05/19 08:11:17 heketi/apps/glusterfs/operations_manage.go:220:glusterfs.AsyncHttpOperation: Create Volume Build Failed: No space I'm using the following heketi-cli command: heketi-cli --user admin --secret xZbphMQHQ volume create --name=cache --size=95545 --gluster-volume-options="performance.write-behind off","performance.open-behind off" --replica=3 If I change the volume size to 94800, it works. It's a 3 cluster nodes, each node have 9 disks 10Tb each and I have set max_bricks_per_volume to 600 and brick_max_size_gb 10240. My topology info is the end of the mail. volume size / (max_bricks_per_volume/ 3 nodes) = 95545/200 = 477.725 brick size device size = 11175, so 11175/477.725 = 23 bricks per device 9 devices * 23 bricks = 207 So the free space on the devices allow the creation of 207 bricks (477.725 size each) on every node,? I need 200 bricks per node to create the volume, then why it fails for "No space" ? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 ??? File:? true ??? Block: true ??? Volumes: ??????? Name: heketidbstorage ??????? Size: 2 ??????? Id: cc07e3205650842e31324cab359bca43 ??????? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 ??????? Mount: 10.50.50.51:heketidbstorage ??????? Mount Options: backup-volfile-servers=10.50.50.52,10.50.50.53 ??????? Durability Type: replicate ??????? Replica: 3 ??????? Snapshot: Disabled ??????????????? Bricks: ??????????????????????? Id: 7a4004f395cce5fe89f4a0fa5af61549 ??????????????????????? Path: /var/lib/heketi/mounts/vg_a81b43597c607f9a3ba54e2552b821e8/brick_7a4004f395cce5fe89f4a0fa5af61549/brick ??????????????????????? Size (GiB): 2 ??????????????????????? Node: e18b193a75bc688ea86659bd1aa0a96c ??????????????????????? Device: a81b43597c607f9a3ba54e2552b821e8 ??????????????????????? Id: 84879c1d761d54788285a8728fe7100f ??????????????????????? Path: /var/lib/heketi/mounts/vg_d25b7a27273da7d25d1e1d8ca60ec922/brick_84879c1d761d54788285a8728fe7100f/brick ??????????????????????? Size (GiB): 2 ??????????????????????? Node: fc13ff5b8a48893e31b1c3da2815ac16 ??????????????????????? Device: d25b7a27273da7d25d1e1d8ca60ec922 ??????????????????????? Id: 9d7dd2cf5ae65b87111f4481f32fb7d7 ??????????????????????? Path: /var/lib/heketi/mounts/vg_bbd7550f3fdeccf291397337375b0cd2/brick_9d7dd2cf5ae65b87111f4481f32fb7d7/brick ??????????????????????? Size (GiB): 2 ??????????????????????? Node: 7ad303638bf94f974ebae8aaf11ac127 ??????????????????????? Device: bbd7550f3fdeccf291397337375b0cd2 ??? Nodes: ??????? Node Id: 7ad303638bf94f974ebae8aaf11ac127 ??????? State: online ??????? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 ??????? Zone: 1 ??????? Management Hostnames: appliance1 ??????? Storage Hostnames: 10.50.50.51 ??????? Devices: ??????????????? Id:00599faa322c0ef3cd5de1f29c6fb38e State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdg ??????????????????????? Bricks: ??????????????? Id:35c678931dd899477289034ca9c4be87 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdc ??????????????????????? Bricks: ??????????????? Id:4b21b903a642a65557fee7f38745a393 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdh ??????????????????????? Bricks: ??????????????? Id:718ef6478d1209909e38567a81ee3dc5 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdj ??????????????????????? Bricks: ??????????????? Id:7569be6559a050ae87c18096a94f7ee1 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdk ??????????????????????? Bricks: ??????????????? Id:88e42232a848d4dce05fbc9e6ab5fe24 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdf ??????????????????????? Bricks: ??????????????? Id:a7129a57851de3c8f74f059486ef801d State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sde ??????????????????????? Bricks: ??????????????? Id:bbd7550f3fdeccf291397337375b0cd2 State:online Size (GiB):11175?? Used (GiB):2?????? Free (GiB):11173 ??????????????????????? Known Paths: /dev/sdd ??????????????????????? Bricks: Id:9d7dd2cf5ae65b87111f4481f32fb7d7?? Size (GiB):2?????? Path: /var/lib/heketi/mounts/vg_bbd7550f3fdeccf291397337375b0cd2/brick_9d7dd2cf5ae65b87111f4481f32fb7d7/brick ??????????????? Id:c5d7fe344f184120594983b305596b6a State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdi ??????????????????????? Bricks: ??????? Node Id: e18b193a75bc688ea86659bd1aa0a96c ??????? State: online ??????? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 ??????? Zone: 1 ??????? Management Hostnames: appliance2 ??????? Storage Hostnames: 10.50.50.52 ??????? Devices: ??????????????? Id:038fb10f2f56c79155b415c2065cc4be State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdf ??????????????????????? Bricks: ??????????????? Id:17987efce61b40397d7809074aacb669 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdk ??????????????????????? Bricks: ??????????????? Id:1ead1ac8442017e28bfd7a2f62bf944c State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdh ??????????????????????? Bricks: ??????????????? Id:271ff7788b1204a15075cb1ce750876a State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sde ??????????????????????? Bricks: ??????????????? Id:63bdcd6738e2f9a0ff9631d6e4608ab4 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdj ??????????????????????? Bricks: ??????????????? Id:7168580b23549f88031328de2d4d82bd State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdc ??????????????????????? Bricks: ??????????????? Id:9a12e60bb5c6793f125eafd81a339a6e State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdi ??????????????????????? Bricks: ??????????????? Id:a81b43597c607f9a3ba54e2552b821e8 State:online Size (GiB):11175?? Used (GiB):2?????? Free (GiB):11173 ??????????????????????? Known Paths: /dev/sdd ??????????????????????? Bricks: Id:7a4004f395cce5fe89f4a0fa5af61549?? Size (GiB):2?????? Path: /var/lib/heketi/mounts/vg_a81b43597c607f9a3ba54e2552b821e8/brick_7a4004f395cce5fe89f4a0fa5af61549/brick ??????????????? Id:b025f8d107533230de67ed2ce9f75aab State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdg ??????????????????????? Bricks: ??????? Node Id: fc13ff5b8a48893e31b1c3da2815ac16 ??????? State: online ??????? Cluster Id: 1def3eb774703edd5dad5687d86ec1f5 ??????? Zone: 1 ??????? Management Hostnames: appliance3 ??????? Storage Hostnames: 10.50.50.53 ??????? Devices: ??????????????? Id:29ccf7627df71dd8e8781841c76c3e38 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sde ??????????????????????? Bricks: ??????????????? Id:3c8bd93e84138f4075e4697c5be2a6f1 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdj ??????????????????????? Bricks: ??????????????? Id:42dd1c3486974f3905f41b12890285ae State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdk ??????????????????????? Bricks: ??????????????? Id:8dd5e8558a5669a85e24342661218bb7 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdg ??????????????????????? Bricks: ??????????????? Id:972a57af87a24a3bad819d3b78a7fa35 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdc ??????????????????????? Bricks: ??????????????? Id:bc679d5c2db68f320904bc486da64bac State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdd ??????????????????????? Bricks: ??????????????? Id:c8f540c5837982973624821ed0dd1d61 State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdh ??????????????????????? Bricks: ??????????????? Id:d25b7a27273da7d25d1e1d8ca60ec922 State:online Size (GiB):11175?? Used (GiB):2?????? Free (GiB):11173 ??????????????????????? Known Paths: /dev/sdi ??????????????????????? Bricks: Id:84879c1d761d54788285a8728fe7100f?? Size (GiB):2?????? Path: /var/lib/heketi/mounts/vg_d25b7a27273da7d25d1e1d8ca60ec922/brick_84879c1d761d54788285a8728fe7100f/brick ??????????????? Id:dd67cbd491766a0ce81d7e751e19c79d State:online Size (GiB):11175?? Used (GiB):0?????? Free (GiB):11175 ??????????????????????? Known Paths: /dev/sdf ??????????????????????? Bricks: -- Regards Andrea Cucciarre'