<div dir="ltr"><div dir="ltr"><div class="gmail_default" style="font-size:small;color:#000000"><span style="color:rgb(34,34,34)">On Sun, Aug 18, 2019 at 12:21 AM Nir Soffer <<a href="mailto:nirsof@gmail.com">nirsof@gmail.com</a>> wrote:</span><br></div></div><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Implement alignment probing similar to file-posix, by reading from the<br>
first 4k of the image.<br>
<br>
Before this change, provisioning a VM on storage with sector size of<br>
4096 bytes would fail when the installer try to create filesystems. Here<br>
is an example command that reproduces this issue:<br>
<br>
$ qemu-system-x86_64 -accel kvm -m 2048 -smp 2 \<br>
-drive file=gluster://gluster1/gv0/fedora29.raw,format=raw,cache=none \<br>
-cdrom Fedora-Server-dvd-x86_64-29-1.2.iso<br>
<br>
The installer fails in few seconds when trying to create filesystem on<br>
/dev/mapper/fedora-root. In error report we can see that it failed with<br>
EINVAL (I could not extract the error from guest).<br>
<br>
Copying disk fails with EINVAL:<br>
<br>
$ qemu-img convert -p -f raw -O raw -t none -T none \<br>
gluster://gluster1/gv0/fedora29.raw \<br>
gluster://gluster1/gv0/fedora29-clone.raw<br>
qemu-img: error while writing sector 4190208: Invalid argument<br>
<br>
This is a fix to same issue fixed in commit a6b257a08e3d (file-posix:<br>
Handle undetectable alignment) for gluster:// images.<br>
<br>
This fix has the same limit, that the first block of the image should be<br>
allocated, otherwise we cannot detect the alignment and fallback to a<br>
safe value (4096) even when using storage with sector size of 512 bytes.<br>
<br>
Signed-off-by: Nir Soffer <<a href="mailto:nsoffer@redhat.com" target="_blank">nsoffer@redhat.com</a>><br>
---<br>
block/gluster.c | 47 +++++++++++++++++++++++++++++++++++++++++++++++<br>
1 file changed, 47 insertions(+)<br>
<br>
diff --git a/block/gluster.c b/block/gluster.c<br>
index f64dc5b01e..d936240b72 100644<br>
--- a/block/gluster.c<br>
+++ b/block/gluster.c<br>
@@ -52,6 +52,9 @@<br>
<br>
#define GERR_INDEX_HINT "hint: check in 'server' array index '%d'\n"<br>
<br>
+/* The value is known only on the server side. */<br>
+#define MAX_ALIGN 4096<br>
+<br>
typedef struct GlusterAIOCB {<br>
int64_t size;<br>
int ret;<br>
@@ -902,8 +905,52 @@ out:<br>
return ret;<br>
}<br>
<br>
+/*<br>
+ * Check if read is allowed with given memory buffer and length.<br>
+ *<br>
+ * This function is used to check O_DIRECT request alignment.<br>
+ */<br>
+static bool gluster_is_io_aligned(struct glfs_fd *fd, void *buf, size_t len)<br>
+{<br>
+ ssize_t ret = glfs_pread(fd, buf, len, 0, 0, NULL);<br>
+ return ret >= 0 || errno != EINVAL;<br>
+}<br>
+<br>
+static void gluster_probe_alignment(BlockDriverState *bs, struct glfs_fd *fd,<br>
+ Error **errp)<br>
+{<br>
+ char *buf;<br>
+ size_t alignments[] = {1, 512, 1024, 2048, 4096};<br>
+ size_t align;<br>
+ int i;<br>
+<br>
+ buf = qemu_memalign(MAX_ALIGN, MAX_ALIGN);<br>
+<br>
+ for (i = 0; i < ARRAY_SIZE(alignments); i++) {<br>
+ align = alignments[i];<br>
+ if (gluster_is_io_aligned(fd, buf, align)) {<br>
+ /* Fallback to safe value. */<br>
+ bs->bl.request_alignment = (align != 1) ? align : MAX_ALIGN;<br>
+ break;<br>
+ }<br>
+ }<br>
+<br>
+ qemu_vfree(buf);<br>
+<br>
+ if (!bs->bl.request_alignment) {<br>
+ error_setg(errp, "Could not find working O_DIRECT alignment");<br>
+ error_append_hint(errp, "Try cache.direct=off\n");<br>
+ }<br>
+}<br>
+<br>
static void qemu_gluster_refresh_limits(BlockDriverState *bs, Error **errp)<br>
{<br>
+ BDRVGlusterState *s = bs->opaque;<br>
+<br>
+ gluster_probe_alignment(bs, s->fd, errp);<br>
+<br>
+ bs->bl.min_mem_alignment = bs->bl.request_alignment;<br>
+ bs->bl.opt_mem_alignment = MAX(bs->bl.request_alignment, MAX_ALIGN);<br>
bs->bl.max_transfer = GLUSTER_MAX_TRANSFER;<br>
}<br>
<br>
-- <br>
2.20.1<br>
<br></blockquote><div><br></div><div><div class="gmail_default" style="font-size:small;color:rgb(0,0,0)">To debug this I added this temporary patch:</div><div class="gmail_default" style="font-size:small;color:rgb(0,0,0)"><br></div>diff --git a/block/gluster.c b/block/gluster.c<br>index d2d187490b..790ef4251b 100644<br>--- a/block/gluster.c<br>+++ b/block/gluster.c<br>@@ -912,6 +912,7 @@ out:<br> static bool gluster_is_io_aligned(struct glfs_fd *fd, void *buf, size_t len)<br> {<br> ssize_t ret = glfs_pread(fd, buf, len, 0, 0, NULL);<br>+ printf("gluster_is_io_aligned len=%ld ret=%ld errno=%d\n", len, ret, errno);<br> return ret >= 0 || errno != EINVAL;<br> }<br> <br>@@ -940,6 +941,9 @@ static void gluster_probe_alignment(BlockDriverState *bs, struct glfs_fd *fd,<br> error_setg(errp, "Could not find working O_DIRECT alignment");<br> error_append_hint(errp, "Try cache.direct=off\n");<br> }<br>+<br>+ printf("Probed aligment for %s request_alignment=%d\n",<br>+ bs->filename, bs->bl.request_alignment);<br> }<br> <br> static void qemu_gluster_refresh_limits(BlockDriverState *bs, Error **errp)<br><div class="gmail_default" style="font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-size:small;color:rgb(0,0,0)">Here is example run with volume with sector size of 512 bytes:<br><br>$ sudo mount -t glusterfs gluster1:/gv1 /tmp/gv1<br>$ dd if=/dev/zero bs=1M count=100 | tr "\0" "x" > /tmp/gv1/src.raw<br>$ truncate -s 100m /tmp/gv1/dst.raw<br>$ dd if=/dev/zero bs=1 count=1 of=/tmp/gv1/dst.raw conv=notrunc<br><br>$ ./qemu-img convert -n -f raw -O raw -t none -T none gluster://gluster1/gv1/src.raw gluster://gluster1/gv1/dst.raw<br>gluster_is_io_aligned len=1 ret=-1 errno=22<br>gluster_is_io_aligned len=512 ret=512 errno=0<br>Probed aligment for gluster://gluster1/gv1/src.raw request_alignment=512<br>gluster_is_io_aligned len=1 ret=-1 errno=22<br>gluster_is_io_aligned len=512 ret=512 errno=0<br>Probed aligment for gluster://gluster1/gv1/dst.raw request_alignment=512<br><br>And with volume with sector size of 4096 bytes:<br><br>$ sudo mount -t glusterfs gluster1:/gv0 /tmp/gv0<br>$ dd if=/dev/zero bs=1M count=100 | tr "\0" "x" > /tmp/gv0/src.raw<br>$ truncate -s 100m /tmp/gv0/dst.raw<br>$ dd if=/dev/zero bs=1 count=1 of=/tmp/gv0/dst.raw conv=notrunc<br><br>$ ./qemu-img convert -n -f raw -O raw -t none -T none gluster://gluster1/gv0/src.raw gluster://gluster1/gv0/dst.raw<br>gluster_is_io_aligned len=1 ret=-1 errno=22<br>gluster_is_io_aligned len=512 ret=-1 errno=22<br>gluster_is_io_aligned len=1024 ret=-1 errno=22<br>gluster_is_io_aligned len=2048 ret=-1 errno=22<br>gluster_is_io_aligned len=4096 ret=4096 errno=0<br>Probed aligment for gluster://gluster1/gv0/src.raw request_alignment=4096<br>gluster_is_io_aligned len=1 ret=-1 errno=22<br>gluster_is_io_aligned len=512 ret=-1 errno=22<br>gluster_is_io_aligned len=1024 ret=-1 errno=22<br>gluster_is_io_aligned len=2048 ret=-1 errno=22<br>gluster_is_io_aligned len=4096 ret=4096 errno=0<br>Probed aligment for gluster://gluster1/gv0/dst.raw request_alignment=4096<br></div></div><div> </div></div></div>