<div dir="ltr"> I would leave it on 64M in volumes with spindle disks, but with SSD volumes, I would increase it to 128M or even 256M, but it varies from one workload to another.<br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 27, 2021 at 10:02 PM Erik Jacobson <<a href="mailto:erik.jacobson@hpe.com" target="_blank">erik.jacobson@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Also, I would like to point that I have VMs with large disks 1TB and 2TB, and<br>
> have no issues. definitely would upgrade Gluster version like let's say at<br>
> least 7.9.<br>
<br>
Great! Thank you! We can update but it's very sensitive due to the<br>
workload. I can't officially update our gluster until we have a cluster<br>
with a couple thousand nodes to test with. However, for this problem,<br>
this is on my list on the test machine. I'm hoping I can reproduce it. So far<br>
no luck making it happen again. Once I hit it, I will try to collect more data<br>
and at the end update gluster.<br>
<br>
What do you think about the suggestion to increase the shard size? Are<br>
you using the default size on your 1TB and 2TB images?<br>
<br>
> Amar also asked a question regarding enabling Sharding in the volume after<br>
> creating the VMs disks, which would certainly mess up the volume if that what<br>
> happened.<br>
<br>
Oh I missed this question. I basically scripted it quick since I was<br>
doing it so often.. I have a similar script that takes it away to start<br>
over.<br>
<br>
set -x<br>
pdsh -g gluster mkdir /data/brick_adminvm/<br>
gluster volume create adminvm replica 3 transport tcp 172.23.255.151:/data/brick_adminvm 172.23.255.152:/data/brick_adminvm 172.23.255.153:/data/brick_adminvm<br>
gluster volume set adminvm group virt<br>
gluster volume set adminvm granular-entry-heal enable<br>
gluster volume set adminvm storage.owner-uid 439<br>
gluster volume set adminvm storage.owner-gid 443<br>
gluster volume start adminvm<br>
<br>
pdsh -g gluster mount /adminvm<br>
<br>
echo -n "press enter to continue for restore tarball"<br>
<br>
pushd /adminvm<br>
tar xvf /root/backup.tar<br>
popd<br>
<br>
echo -n "press enter to continue for qemu-img"<br>
<br>
pushd /adminvm<br>
qemu-img create -f raw -o preallocation=falloc /adminvm/images/adminvm.img 5T<br>
popd<br>
<br>
<br>
Thanks again for the kind responses,<br>
<br>
Erik<br>
<br>
> <br>
> On Wed, Jan 27, 2021 at 5:28 PM Erik Jacobson <<a href="mailto:erik.jacobson@hpe.com" target="_blank">erik.jacobson@hpe.com</a>> wrote:<br>
> <br>
> > > Shortly after the sharded volume is made, there are some fuse mount<br>
> > > messages. I'm not 100% sure if this was just before or during the<br>
> > > big qemu-img command to make the 5T image<br>
> > > (qemu-img create -f raw -o preallocation=falloc<br>
> > > /adminvm/images/adminvm.img 5T)<br>
> > Any reason to have a single disk with this size ?<br>
> <br>
> > Usually in any<br>
> > virtualization I have used , it is always recommended to keep it lower.<br>
> > Have you thought about multiple disks with smaller size ?<br>
> <br>
> Yes, because the actual virtual machine is an admin node/head node cluster<br>
> manager for a supercomputer that hosts big OS images and drives<br>
> multi-thousand-node-clusters (boot, monitoring, image creation,<br>
> distribution, sometimes NFS roots, etc) . So this VM is a biggie.<br>
> <br>
> We could make multiple smaller images but it would be very painful since<br>
> it differs from the normal non-VM setup.<br>
> <br>
> So unlike many solutions where you have lots of small VMs with their<br>
> images small images, this solution is one giant VM with one giant image.<br>
> We're essentially using gluster in this use case (as opposed to others I<br>
> have posted about in the past) for head node failover (combined with<br>
> pacemaker).<br>
> <br>
> > Also worth<br>
> > noting is that RHII is supported only when the shard size is 512MB, so<br>
> > it's worth trying bigger shard size .<br>
> <br>
> I have put larger shard size and newer gluster version on the list to<br>
> try. Thank you! Hoping to get it failing again to try these things!<br>
> <br>
> <br>
> <br>
> --<br>
> Respectfully<br>
> Mahdi<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Respectfully<div>Mahdi</div></div></div></div>