<div dir="ltr">Hello Erik,<div><br></div><div> Anything in the logs of the fuse mount? can you stat the file from the mount? also, the report of an image is only 64M makes me think about Sharding as the default value of Shard size is 64M.</div><div>Do you have any clues on when this issue start to happen? was there any operation done to the Gluster cluster?</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 26, 2021 at 2:40 AM Erik Jacobson <<a href="mailto:erik.jacobson@hpe.com">erik.jacobson@hpe.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hello all. Thanks again for gluster. We're having a strange problem<br>
getting virtual machines started that are hosted on a gluster volume.<br>
<br>
One of the ways we use gluster now is to make a HA-ish cluster head<br>
node. A virtual machine runs in the shared storage and is backed up by 3<br>
physical servers that contribute to the gluster storage share.<br>
<br>
We're using sharding in this volume. The VM image file is around 5T and<br>
we use qemu-img with falloc to get all the blocks allocated in advance.<br>
<br>
We are not using gfapi largely because it would mean we have to build<br>
our own libvirt and qemu and we'd prefer not to do that. So we're using<br>
a glusterfs fuse mount to host the image. The virtual machine is using<br>
virtio disks but we had similar trouble using scsi emulation.<br>
<br>
The issue: - all seems well, the VM head node installs, boots, etc.<br>
<br>
However, at some point, it stops being able to boot! grub2 acts like it<br>
cannot find /boot. At the grub2 prompt, it can see the partitions, but<br>
reports no filesystem found where there are indeed filesystems.<br>
<br>
If we switch qemu to use "direct kernel load" (bypass grub2), this often<br>
works around the problem but in one case Linux gave us a clue. Linux<br>
reported /dev/vda as only being 64 megabytes, which would explain a lot.<br>
This means the virtual machine Linux though the disk supplied by the<br>
disk image was tiny! 64M instead of 5T<br>
<br>
We are using sles15sp2 and hit the problem more often with updates<br>
applied than without. I'm in the process of trying to isolate if there<br>
is a sles15sp2 update causing this, or if we're within "random chance".<br>
<br>
On one of the physical nodes, if it is in the failure mode, if I use<br>
'kpartx' to create the partitions from the image file, then mount the<br>
giant root filesystem (ie mount /dev/mapper/loop0p31 /mnt) and then<br>
umount /mnt, then that physical node starts the VM fine, grub2 loads,<br>
the virtual machine is fully happy! Until I try to shut it down and<br>
start it up again, at which point it sticks at grub2 again! What about<br>
mounting the image file makes it so qemu sees the whole disk?<br>
<br>
The problem doesn't always happen but once it starts, the same VM image has<br>
trouble starting on any of the 3 physical nodes sharing the storage.<br>
But using the trick to force-mount the root within the image with<br>
kpartx, then the machine can come up. My only guess is this changes the<br>
file just a tiny bit in the middle of the image.<br>
<br>
Once the problem starts, it keeps happening except temporarily working<br>
when I do the loop mount trick on the physical admin.<br>
<br>
<br>
Here is some info about what I have in place:<br>
<br>
<br>
nano-1:/adminvm/images # gluster volume info<br>
<br>
Volume Name: adminvm<br>
Type: Replicate<br>
Volume ID: 67de902c-8c00-4dc9-8b69-60b93b5f6104<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: 172.23.255.151:/data/brick_adminvm<br>
Brick2: 172.23.255.152:/data/brick_adminvm<br>
Brick3: 172.23.255.153:/data/brick_adminvm<br>
Options Reconfigured:<br>
performance.client-io-threads: on<br>
nfs.disable: on<br>
storage.fips-mode-rchecksum: on<br>
transport.address-family: inet<br>
performance.quick-read: off<br>
performance.read-ahead: off<br>
performance.io-cache: off<br>
performance.low-prio-threads: 32<br>
network.remote-dio: enable<br>
cluster.eager-lock: enable<br>
cluster.quorum-type: auto<br>
cluster.server-quorum-type: server<br>
cluster.data-self-heal-algorithm: full<br>
cluster.locking-scheme: granular<br>
cluster.shd-max-threads: 8<br>
cluster.shd-wait-qlength: 10000<br>
features.shard: on<br>
user.cifs: off<br>
cluster.choose-local: off<br>
client.event-threads: 4<br>
server.event-threads: 4<br>
cluster.granular-entry-heal: enable<br>
storage.owner-uid: 439<br>
storage.owner-gid: 443<br>
<br>
<br>
<br>
<br>
libglusterfs0-7.2-4723.1520.210122T1700.a.sles15sp2hpe.x86_64<br>
glusterfs-7.2-4723.1520.210122T1700.a.sles15sp2hpe.x86_64<br>
python3-gluster-7.2-4723.1520.210122T1700.a.sles15sp2hpe.noarch<br>
<br>
<br>
<br>
nano-1:/adminvm/images # uname -a<br>
Linux nano-1 5.3.18-24.46-default #1 SMP Tue Jan 5 16:11:50 UTC 2021 (4ff469b) x86_64 x86_64 x86_64 GNU/Linux<br>
nano-1:/adminvm/images # rpm -qa | grep qemu-4<br>
qemu-4.2.0-9.4.x86_64<br>
<br>
<br>
<br>
Would love any advice!!!!<br>
<br>
<br>
Erik<br>
________<br>
<br>
<br>
<br>
Community Meeting Calendar:<br>
<br>
Schedule -<br>
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC<br>
Bridge: <a href="https://meet.google.com/cpu-eiue-hvk" rel="noreferrer" target="_blank">https://meet.google.com/cpu-eiue-hvk</a><br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr">Respectfully<div>Mahdi</div></div></div>