[Gluster-users] fuse access seems ok but native qemu access is too slow
algol at sapo.pt
algol at sapo.pt
Fri Mar 19 14:34:55 UTC 2021
Hi,
I have just configure a distributed-replicate 2x3 gluster volume.
It seems to be working fine when I mount the volume with fuse but
native gluster access with qemu-img is much slower and outputs errors.
Log files are filled with warning and errors.
Starting vm with the created images is possible but it takes much more
time when using gluster native access. Nevertheless, after starting
up, there seems to be no penalty in using the gluster native access
image being some times faster.
Can anyone, please, help me in figuring out if there is and where is
the problem?
Thanks
rm
It seems to be working fine when I mount it with fuse:
root at srv-31:~# mount -t glusterfs
srv-32.lan.example.com/ssd-volume/[1] /mnt/ssd-volume/
root at srv-31:~# echo 'This is a test' > /mnt/ssd-volume/test/text.txt
root at srv-31:~# ll /data/glusterfs/ssd/brick[12]/brick/test/*
-rw-r--r-- 2 root root 15 mar 17 14:08
/data/glusterfs/ssd/brick1/brick/test/text.txt
root at srv-31:~# time qemu-img create -f qcow2
/mnt/ssd-volume/libvirt/images/test_5.qcow2.img 64G
Formatting '/mnt/ssd-volume/libvirt/images/test_5.qcow2.img',
fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off
refcount_bits=16
real 0m0,204s
user 0m0,009s
sys 0m0,009s
Creating a image file onteh mouted gluster volume is quick but I get
some warnings on log files:
root at srv-31:~# tail -f /var/log/glusterfs/glusterd.log
/var/log/glusterfs/glustershd.log
/var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:37:11.301388 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.303694 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.305808 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.307868 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.309863 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.312139 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.317780 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.319007 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.321096 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.323111 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.325284 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.327319 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.329488 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.331792 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.333597 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.335864 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.339551 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.341110 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.343079 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.345174 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.349569 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.351920 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.353282 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.355272 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.357467 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.359584 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.361771 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.364013 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.366094 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.368094 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.370261 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.372389 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.375550 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
(...)
[2021-03-19 11:37:11.477472 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.479897 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.482318 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
[2021-03-19 11:37:11.484489 +0000] W [dict.c:1532:dict_get_with_ref]
(-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e)
[0x7fb1dbdccc3e]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37)
[0x7fb1e1a1f3f7]
-->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d)
[0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL
[Invalid argument]
Native gluster access with qemu-img is much slower and outputs errors:
root at srv-31:~# time qemu-img create -f qcow2
gluster://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img
64G
Formatting
'gluster://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img[2]', fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off
refcount_bits=16
[2021-03-19 11:47:42.490844 +0000] I
[io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume:
Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2021-03-19 11:47:42.615942 +0000] E [MSGID: 108006]
[afr-common.c:6146:__afr_handle_child_down_event]
0-ssd-volume-replicate-0: All subvolumes are down. Going offline until
at least one of them comes back up.
[2021-03-19 11:47:42.617033 +0000] E [MSGID: 108006]
[afr-common.c:6146:__afr_handle_child_down_event]
0-ssd-volume-replicate-1: All subvolumes are down. Going offline until
at least one of them comes back up.
[2021-03-19 11:47:52.502174 +0000] I [io-stats.c:4038:fini]
0-ssd-volume: io-stats translator unloaded
[2021-03-19 11:47:53.514364 +0000] I
[io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume:
Configure ios_sample_buf size is 1024 because ios_sample_interval is 0
[2021-03-19 11:47:53.654336 +0000] E [MSGID: 108006]
[afr-common.c:6146:__afr_handle_child_down_event]
0-ssd-volume-replicate-0: All subvolumes are down. Going offline until
at least one of them comes back up.
[2021-03-19 11:47:53.655251 +0000] E [MSGID: 108006]
[afr-common.c:6146:__afr_handle_child_down_event]
0-ssd-volume-replicate-1: All subvolumes are down. Going offline until
at least one of them comes back up.
[2021-03-19 11:48:03.525947 +0000] I [io-stats.c:4038:fini]
0-ssd-volume: io-stats translator unloaded
real 0m22,068s
user 0m0,055s
sys 0m0,043s
In both cases images are written correctly (and usable):
root at srv-31:~# l /data/glusterfs/ssd/brick[12]/brick/libvirt/images/test_[56]*
-rw-r--r-- 2 root root 193K mar 19 11:37
/data/glusterfs/ssd/brick2/brick/libvirt/images/test_5.qcow2.img
-rw------- 2 root root 193K mar 19 11:47
/data/glusterfs/ssd/brick2/brick/libvirt/images/test_6.qcow2.img
But I get lots of warnings on log files:
root at srv-31:~# tail -f /var/log/glusterfs/glusterd.log
/var/log/glusterfs/glustershd.log
/var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/glusterd.log <==
[2021-03-19 11:47:42.483736 +0000] I [MSGID: 106496]
[glusterd-handshake.c:969:__server_getspec] 0-management: Received
mount request for volume ssd-volume
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.498908 +0000] I
[addr.c:54:compare_addr_and_update]
0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr =
"127.0.0.1"
[2021-03-19 11:47:42.498967 +0000] I [login.c:110:gf_auth]
0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:42.498989 +0000] I [MSGID: 115029]
[server-handshake.c:561:server_setvolume] 0-ssd-volume-server:
accepted client from
CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol
/data/glusterfs/ssd/brick1/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.502029 +0000] I
[addr.c:54:compare_addr_and_update]
0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr =
"127.0.0.1"
[2021-03-19 11:47:42.502087 +0000] I [login.c:110:gf_auth]
0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:42.502109 +0000] I [MSGID: 115029]
[server-handshake.c:561:server_setvolume] 0-ssd-volume-server:
accepted client from
CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol
/data/glusterfs/ssd/brick2/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.615182 +0000] W [socket.c:767:__socket_rwv]
0-tcp.ssd-volume-server: readv on 127.0.0.1:49151 failed (No data
available)
[2021-03-19 11:47:42.615258 +0000] I [MSGID: 115036]
[server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting
connection
[{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.615215 +0000] W [socket.c:767:__socket_rwv]
0-tcp.ssd-volume-server: readv on 127.0.0.1:49150 failed (No data
available)
[2021-03-19 11:47:42.615316 +0000] I [MSGID: 115036]
[server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting
connection
[{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:42.615459 +0000] I [MSGID: 101055]
[client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down
connection
CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:42.615524 +0000] I [MSGID: 101055]
[client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down
connection
CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.522422 +0000] I
[addr.c:54:compare_addr_and_update]
0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr =
"127.0.0.1"
[2021-03-19 11:47:53.522480 +0000] I [login.c:110:gf_auth]
0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:53.522502 +0000] I [MSGID: 115029]
[server-handshake.c:561:server_setvolume] 0-ssd-volume-server:
accepted client from
CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol
/data/glusterfs/ssd/brick1/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.525344 +0000] I
[addr.c:54:compare_addr_and_update]
0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr =
"127.0.0.1"
[2021-03-19 11:47:53.525396 +0000] I [login.c:110:gf_auth]
0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681
[2021-03-19 11:47:53.525419 +0000] I [MSGID: 115029]
[server-handshake.c:561:server_setvolume] 0-ssd-volume-server:
accepted client from
CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol
/data/glusterfs/ssd/brick2/brick
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.653652 +0000] W [socket.c:767:__socket_rwv]
0-tcp.ssd-volume-server: readv on 127.0.0.1:49146 failed (No data
available)
[2021-03-19 11:47:53.653731 +0000] I [MSGID: 115036]
[server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting
connection
[{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.653752 +0000] W [socket.c:767:__socket_rwv]
0-tcp.ssd-volume-server: readv on 127.0.0.1:49144 failed (No data
available)
[2021-03-19 11:47:53.653840 +0000] I [MSGID: 115036]
[server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting
connection
[{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:47:53.653928 +0000] I [MSGID: 101055]
[client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down
connection
CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:47:53.654040 +0000] I [MSGID: 101055]
[client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down
connection
CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0
==> /var/log/glusterfs/glusterd.log <==
[2021-03-19 11:47:53.507207 +0000] I [MSGID: 106496]
[glusterd-handshake.c:969:__server_getspec] 0-management: Received
mount request for volume ssd-volume
Starting a vm with a virtual disk on the mounted gluster volume
(-drive file=/mnt/ssd-volume/libvirt/images/test_6.qcow2.img)is quick:
root at srv-31:~# time virsh start vm-11
Domain vm-11 started
real 0m0,277s
user 0m0,020s
sys 0m0,004s
Statring a vm with a virtual disk on gluster volume with native access
(-drive
file=gluster://srv-31.lan.example.com:24007/ssd-volume/libvirt/images/test_7.qcow2.img)is much
slower:
root at srv-31:~# time virsh start vm-11
Domain vm-11 started
real 0m22,339s
user 0m0,021s
sys 0m0,007s
From time to time, even without user operations, log files report errors:
root at srv-31:~# tail -f /var/log/glusterfs/glusterd.log
/var/log/glusterfs/glustershd.log
/var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:09.525079 +0000] E [MSGID: 113002]
[posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null
for (null) [Invalid argument]
[2021-03-19 11:45:09.525212 +0000] E [MSGID: 115056]
[server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server:
MKDIR info [{frame=11817}, {MKDIR_path=},
{uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=},
{client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid
argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:13.525581 +0000] E [MSGID: 113002]
[posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null
for (null) [Invalid argument]
[2021-03-19 11:45:13.525711 +0000] E [MSGID: 115056]
[server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server:
MKDIR info [{frame=10055}, {MKDIR_path=},
{uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=},
{client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid
argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:38.829803 +0000] E [MSGID: 115056]
[server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server:
MKDIR info [{frame=11820}, {MKDIR_path=},
{uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=},
{client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid
argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:38.829970 +0000] E [MSGID: 115056]
[server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server:
MKDIR info [{frame=10058}, {MKDIR_path=},
{uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=},
{client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid
argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:38.829721 +0000] E [MSGID: 113002]
[posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null
for (null) [Invalid argument]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==
[2021-03-19 11:45:38.829883 +0000] E [MSGID: 113002]
[posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null
for (null) [Invalid argument]
[2021-03-19 11:45:50.005995 +0000] E [MSGID: 113002]
[posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null
for (null) [Invalid argument]
[2021-03-19 11:45:50.006115 +0000] E [MSGID: 115056]
[server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server:
MKDIR info [{frame=10082}, {MKDIR_path=},
{uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=},
{client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid
argument}]
==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==
[2021-03-19 11:45:50.006096 +0000] E [MSGID: 113002]
[posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null
for (null) [Invalid argument]
[2021-03-19 11:45:50.006212 +0000] E [MSGID: 115056]
[server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server:
MKDIR info [{frame=11844}, {MKDIR_path=},
{uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=},
{client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid
argument}]
==> /var/log/glusterfs/glustershd.log <==
[2021-03-19 11:45:50.006255 +0000] E [MSGID: 114031]
[client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-3:
remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid
argument}]
[2021-03-19 11:45:50.006352 +0000] E [MSGID: 114031]
[client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-0:
remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid
argument}]
[2021-03-19 11:45:50.006408 +0000] E [MSGID: 114031]
[client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-4:
remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid
argument}]
[2021-03-19 11:45:50.006505 +0000] E [MSGID: 114031]
[client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-5:
remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid
argument}]
[2021-03-19 11:45:50.006562 +0000] E [MSGID: 114031]
[client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-1:
remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid
argument}]
[2021-03-19 11:45:50.006645 +0000] E [MSGID: 114031]
[client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-2:
remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid
argument}]
Everything seems to be ok
root at srv-31:~# gluster volume status ssd-volume
Status of volume: ssd-volume
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick
srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick
49156 0 Y 9242
Brick
srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick
49156 0 Y 7968
Brick
srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick
49156 0 Y 8211
Brick
srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick
49157 0 Y 9258
Brick
srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick
49157 0 Y 7984
Brick
srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick
49157 0 Y 8227
Self-heal Daemon on localhost N/A N/A Y 2486
Self-heal Daemon on
srv-32.lan.example.com N/A
N/A Y 2291
Self-heal Daemon on
srv-33.lan.example.com N/A
N/A Y 2062
Task Status of Volume ssd-volume
------------------------------------------------------------------------------
There are no active volume tasks
root at srv-31:~# gluster volume info ssd-volume
Volume Name: ssd-volume
Type: Distributed-Replicate
Volume ID: a6f3426f-5b33-404b-ab61-581e24e0c36d
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x 3 = 6
Transport-type: tcp
Bricks:
Brick1: srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick2: srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick3: srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick
Brick4: srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick
Brick5: srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick
Brick6: srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick
Options Reconfigured:
server.allow-insecure: on
storage.owner-gid: 64055
storage.owner-uid: 64055
ssl.cipher-list: HIGH:!SSLv2
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
storage.fips-mode-rchecksum: on
client.ssl: off
server.ssl: off
auth.ssl-allow: *
root at srv-31:~# gluster peer status
Number of Peers: 2
Hostname: srv-32.lan.example.com
Uuid: 0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a
State: Peer in Cluster (Connected)
Hostname: srv-33.lan.example.com
Uuid: 111d93d2-64aa-463f-8f33-7b53a3f05cab
State: Peer in Cluster (Connected)
root at srv-31:~# gluster pool list
UUID
Hostname State
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a
srv-32.lan.example.com Connected
111d93d2-64aa-463f-8f33-7b53a3f05cab
srv-33.lan.example.com Connected
67e9da67-90c5-486b-acb1-eb91abebcefb
localhost Connected
root at srv-32:~# gluster pool list
UUID
Hostname State
111d93d2-64aa-463f-8f33-7b53a3f05cab
srv-33.lan.example.com Connected
67e9da67-90c5-486b-acb1-eb91abebcefb
10.0.0.31 Connected
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a
localhost Connected
root at srv-33:~# gluster pool list
UUID
Hostname State
0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a
srv-32.lan.example.com Connected
67e9da67-90c5-486b-acb1-eb91abebcefb
10.0.0.31 Connected
111d93d2-64aa-463f-8f33-7b53a3f05cab
localhost Connected
root at srv-31:~# cat /etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option transport.socket.listen-port 24007
option ping-timeout 0
option event-threads 1
# option lock-timer 180
# option transport.address-family inet6
# option base-port 49152
option max-port 60999
option rpc-auth-allow-insecure on
end-volume
root at srv-31:~# ss -lntp
State Recv-Q Send-Q Local Address:Port Peer
Address:Port
LISTEN 0 128 0.0.0.0:49153
0.0.0.0:*
users:(("glusterfsd",pid=2455,fd=11))
LISTEN 0 128 0.0.0.0:49156
0.0.0.0:*
users:(("glusterfsd",pid=9242,fd=11))
LISTEN 0 128 0.0.0.0:49157
0.0.0.0:*
users:(("glusterfsd",pid=9258,fd=11))
LISTEN 0 128 0.0.0.0:24007
0.0.0.0:*
users:(("glusterd",pid=1954,fd=10))
LISTEN 0 128 0.0.0.0:111
0.0.0.0:*
users:(("rpcbind",pid=1716,fd=4),("systemd",pid=1,fd=36))
LISTEN 0 128 0.0.0.0:2224
0.0.0.0:*
users:(("pcsd",pid=2126,fd=4))
LISTEN 0 128 0.0.0.0:22
0.0.0.0:*
users:(("sshd",pid=1963,fd=3))
LISTEN 0 20 127.0.0.1:25
0.0.0.0:*
users:(("exim4",pid=2377,fd=3))
LISTEN 0 128 0.0.0.0:49152
0.0.0.0:*
users:(("glusterfsd",pid=2446,fd=11))
LISTEN 0 128 [::]:111
[::]:*
users:(("rpcbind",pid=1716,fd=6),("systemd",pid=1,fd=38))
LISTEN 0 128 [::]:2224
[::]:* users:(("pcsd",pid=2126,fd=5))
root at srv-31:~# ss -tnp | grep gluster
ESTAB 0 0 127.0.0.1:49149
127.0.1.1:24007
users:(("glusterfsd",pid=2446,fd=9))
ESTAB 0 0 10.0.0.31:49144
10.0.0.33:49153
users:(("glusterfs",pid=2486,fd=17))
ESTAB 0 0 127.0.0.1:24007
127.0.0.1:49145
users:(("glusterd",pid=1954,fd=26))
ESTAB 0 0 127.0.1.1:49157
127.0.0.1:49140
users:(("glusterfsd",pid=9258,fd=10))
ESTAB 0 0 127.0.0.1:49137
127.0.1.1:49156
users:(("glusterfs",pid=2917,fd=10))
ESTAB 0 0 10.0.0.31:49146
10.0.0.32:49152
users:(("glusterfs",pid=2486,fd=25))
ESTAB 0 0 10.0.0.31:49153
10.0.0.33:49147
users:(("glusterfsd",pid=2455,fd=270))
ESTAB 0 0 127.0.1.1:24007
127.0.0.1:49148
users:(("glusterd",pid=1954,fd=12))
ESTAB 0 0 10.0.0.31:49157
10.0.0.33:49141
users:(("glusterfsd",pid=9258,fd=271))
ESTAB 0 0 127.0.1.1:24007
127.0.0.1:49131
users:(("glusterd",pid=1954,fd=31))
ESTAB 0 0 10.0.0.31:49152
10.0.0.32:49146
users:(("glusterfsd",pid=2446,fd=271))
ESTAB 0 0 10.0.0.31:49134
10.0.0.33:24007
users:(("glusterd",pid=1954,fd=33))
ESTAB 0 0 10.0.0.31:49127
10.0.0.33:49157
users:(("glusterfs",pid=2917,fd=11))
ESTAB 0 0 127.0.0.1:49142
127.0.1.1:49156
users:(("glusterfs",pid=2486,fd=15))
ESTAB 0 0 127.0.0.1:49135
127.0.1.1:49157
users:(("glusterfs",pid=2917,fd=14))
ESTAB 0 0 127.0.0.1:49145
127.0.0.1:24007
users:(("glusterfs",pid=2486,fd=9))
ESTAB 0 0 127.0.1.1:49157
127.0.0.1:49135
users:(("glusterfsd",pid=9258,fd=272))
ESTAB 0 0 127.0.1.1:49152
127.0.0.1:49141
users:(("glusterfsd",pid=2446,fd=10))
ESTAB 0 0 10.0.0.31:49157
10.0.0.32:49144
users:(("glusterfsd",pid=9258,fd=270))
ESTAB 0 0 127.0.1.1:24007
127.0.0.1:49132
users:(("glusterd",pid=1954,fd=25))
ESTAB 0 0 127.0.0.1:49148
127.0.1.1:24007
users:(("glusterfsd",pid=2455,fd=9))
ESTAB 0 0 10.0.0.31:49118
10.0.0.32:49156
users:(("glusterfs",pid=2486,fd=13))
ESTAB 0 0 127.0.1.1:49156
127.0.0.1:49137
users:(("glusterfsd",pid=9242,fd=272))
ESTAB 0 0 10.0.0.31:49121
10.0.0.33:49156
users:(("glusterfs",pid=2917,fd=13))
ESTAB 0 0 127.0.0.1:49132
127.0.1.1:24007
users:(("glusterfsd",pid=9242,fd=9))
ESTAB 0 0 127.0.0.1:49141
127.0.1.1:49152
users:(("glusterfs",pid=2486,fd=22))
ESTAB 0 0 10.0.0.31:49156
10.0.0.32:49148
users:(("glusterfsd",pid=9242,fd=270))
ESTAB 0 0 10.0.0.31:49152
10.0.0.33:49149
users:(("glusterfsd",pid=2446,fd=270))
ESTAB 0 0 127.0.0.1:49140
127.0.1.1:49157
users:(("glusterfs",pid=2486,fd=11))
ESTAB 0 0 10.0.0.31:49123
10.0.0.32:49157
users:(("glusterfs",pid=2917,fd=12))
ESTAB 0 0 10.0.0.31:49141
10.0.0.33:49152
users:(("glusterfs",pid=2486,fd=23))
ESTAB 0 0 127.0.0.1:49131
127.0.1.1:24007
users:(("glusterfsd",pid=9258,fd=9))
ESTAB 0 0 10.0.0.31:49153
10.0.0.32:49145
users:(("glusterfsd",pid=2455,fd=271))
ESTAB 0 0 10.0.0.31:49117
10.0.0.33:49156
users:(("glusterfs",pid=2486,fd=5))
ESTAB 0 0 10.0.0.31:49114
10.0.0.32:49157
users:(("glusterfs",pid=2486,fd=16))
ESTAB 0 0 127.0.1.1:49153
127.0.0.1:49138
users:(("glusterfsd",pid=2455,fd=10))
ESTAB 0 0 10.0.0.31:49143
10.0.0.32:24007
users:(("glusterfs",pid=2917,fd=9))
ESTAB 0 0 10.0.0.31:49113
10.0.0.33:49157
users:(("glusterfs",pid=2486,fd=18))
ESTAB 0 0 10.0.0.31:49156
10.0.0.33:49143
users:(("glusterfsd",pid=9242,fd=271))
ESTAB 0 0 127.0.0.1:49138
127.0.1.1:49153
users:(("glusterfs",pid=2486,fd=19))
ESTAB 0 0 10.0.0.31:24007
10.0.0.33:49150
users:(("glusterd",pid=1954,fd=32))
ESTAB 0 0 10.0.0.31:49131
10.0.0.32:49156
users:(("glusterfs",pid=2917,fd=7))
ESTAB 0 0 10.0.0.31:49145
10.0.0.32:49153
users:(("glusterfs",pid=2486,fd=8))
ESTAB 0 0 127.0.1.1:24007
127.0.0.1:49149
users:(("glusterd",pid=1954,fd=13))
ESTAB 0 0 10.0.0.31:24007
10.0.0.32:49150
users:(("glusterd",pid=1954,fd=7))
ESTAB 0 0 10.0.0.31:49151
10.0.0.32:24007
users:(("glusterd",pid=1954,fd=8))
ESTAB 0 0 127.0.1.1:49156
127.0.0.1:49142 users:(("glusterfsd",pid=9242,fd=10))
hostnames resolve correctly:
root at srv-31:~# for i in {31..33}; do host srv-$i.lan.example.com; done
srv-31.lan.example.com has address 10.0.0.31
srv-32.lan.example.com has address 10.0.0.32
srv-33.lan.example.com has address 10.0.0.33
And network conectivity is ok:
root at srv-31:~# for i in {31..33}; do ping -c 1 -q
srv-$i.lan.example.com | echo "srv-$i ok"; done
srv-31 ok
srv-32 ok
srv-33 ok
root at srv-31:~# for i in {31..33}; do nc -v srv-$i.lan.example.com 24007; done
Connection to srv-31.lan.example.com 24007 port [tcp/*] succeeded!
Connection to srv-32.lan.example.com 24007 port [tcp/*] succeeded!
Connection to srv-33.lan.example.com 24007 port [tcp/*] succeeded!
root at srv-31:~# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Ligações:
---------
[1] http://srv-31.lan.example.com/ssd-volume/libvirt/images/
[2] http://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.im
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20210319/08159680/attachment-0001.html>
More information about the Gluster-users
mailing list