[Gluster-users] [Gluster-devel] 3.7.13 & proxmox/qemu

[Samuli Heinonen]

Here is a quick way how to test this:
GlusterFS 3.7.13 volume with default settings with brick on ZFS dataset. gluster-test1 is server and gluster-test2 is client mounting with FUSE.

Writing file with oflag=direct is not ok:
[root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000
dd: failed to open ‘file’: Invalid argument

Enable network.remote-dio on Gluster Volume:
[root at gluster-test1 gluster]# gluster volume set gluster network.remote-dio enable
volume set: success

Writing small file with oflag=direct is ok:
[root at gluster-test2 gluster]# dd if=/dev/zero of=file oflag=direct count=1 bs=1024000
1+0 records in
1+0 records out
1024000 bytes (1.0 MB) copied, 0.0103793 s, 98.7 MB/s

Writing bigger file with oflag=direct is ok:
[root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.10583 s, 94.8 MB/s

Enable Sharding on Gluster Volume:
[root at gluster-test1 gluster]# gluster volume set gluster features.shard enable
volume set: success

Writing small file  with oflag=direct is ok:
[root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=1 bs=1M
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.0115247 s, 91.0 MB/s

Writing bigger file with oflag=direct is not ok:
[root at gluster-test2 gluster]# dd if=/dev/zero of=file3 oflag=direct count=100 bs=1M
dd: error writing ‘file3’: Operation not permitted
dd: closing output file ‘file3’: Operation not permitted

-samuli


> On 22 Jul 2016, at 16:12, Vijay Bellur <vbellur at redhat.com> wrote:
> 
> 2016-07-22 1:54 GMT-04:00 Frank Rothenstein <f.rothenstein at bodden-kliniken.de>:
>> The point is that even if all other backend storage filesystems do correctly
>> untill 3.7.11 there was no error on ZFS. Something happened nobody ever
>> could explain in the release of 3.7.12 that makes FUSE-mount _in ovirt_ (it
>> partly uses dd with iflag=direct  , using iflag=direct yourself gives also
>> errors on the FUSE-mounts ) unusable.
>> 
>> So 3.7.11 is the last usable version when using ZFS on bricks, afaik.
>> 
> 
> Can you please share the exact dd command that causes this problem?
> 
> Thanks,
> Vijay
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users