<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN"
"http://www.w3.org/TR/REC-html40/loose.dtd">
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
<title></title>
</head>
<body style="font-family:Arial">
<div>
<div>Hi,<br></div>
<div>
<div><br></div>
</div>
<div>I have just configure a distributed-replicate 2x3 gluster volume.</div>
<div>It seems to be working fine when I mount the volume with fuse but native gluster access with qemu-img is much slower and outputs errors. Log files are filled with warning and errors.<br></div>
<div>Starting vm with the created images is possible but it takes much more time when using gluster native access. Nevertheless, after starting up, there seems to be no penalty in using the gluster native access image being some times faster.<br></div>
<div><br></div>
<div>Can anyone, please, help me in figuring out if there is and where is the problem?<br></div>
<div><br></div>
<div>Thanks<br></div>
<div>rm<br></div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>It seems to be working fine when I mount it with fuse:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# mount -t glusterfs <a href="http://srv-31.lan.example.com/ssd-volume/libvirt/images/">srv-32.lan.example.com/ssd-volume/</a> /mnt/ssd-volume/</span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# echo 'This is a test' > /mnt/ssd-volume/test/text.txt<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# ll /data/glusterfs/ssd/brick[12]/brick/test/*<br></span></div>
</div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">-rw-r--r-- 2 root root 15 mar 17 14:08 /data/glusterfs/ssd/brick1/brick/test/text.txt<br>
<br>
root@srv-31:~# time qemu-img create -f qcow2 /mnt/ssd-volume/libvirt/images/test_5.qcow2.img 64G<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Formatting '/mnt/ssd-volume/libvirt/images/test_5.qcow2.img', fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off refcount_bits=16<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">real 0m0,204s<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">user 0m0,009s<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">sys 0m0,009s</span><br></div>
<div><br></div>
<div><br></div>
<div>
<div>Creating a image file onteh mouted gluster volume is quick but I get some warnings on log files:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log /var/log/glusterfs/glustershd.log /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
</div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.301388 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.303694 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.305808 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.307868 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.309863 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.312139 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.317780 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.319007 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.321096 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.323111 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.325284 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.327319 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.329488 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.331792 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.333597 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.335864 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.339551 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.341110 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.343079 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.345174 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.349569 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.351920 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.353282 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.355272 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.357467 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.359584 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.361771 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.364013 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.366094 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.368094 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.370261 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.372389 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.375550 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">(...)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.477472 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.479897 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.482318 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:37:11.484489 +0000] W [dict.c:1532:dict_get_with_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/9.0/xlator/features/locks.so(+0x26c3e) [0x7fb1dbdccc3e] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_uint32+0x37) [0x7fb1e1a1f3f7] -->/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_get_with_ref+0x7d) [0x7fb1e1a1e86d] ) 0-dict: dict OR key (glusterfs.lk.lkmode) is NULL [Invalid argument]</span><br></div>
<div><br></div>
<div><br></div>
<div>Native gluster access with qemu-img is much slower and outputs errors:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# time qemu-img create -f qcow2 gluster://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img</span> <span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">64G<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Formatting 'gluster://<a href="http://srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.im">srv-31.lan.example.com/ssd-volume/libvirt/images/test_6.qcow2.img</a>', fmt=qcow2 size=68719476736 cluster_size=65536 lazy_refcounts=off refcount_bits=16</span><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.490844 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615942 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.617033 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:52.502174 +0000] I [io-stats.c:4038:fini] 0-ssd-volume: io-stats translator unloaded<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.514364 +0000] I [io-stats.c:3708:ios_sample_buf_size_configure] 0-ssd-volume: Configure ios_sample_buf size is 1024 because ios_sample_interval is 0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.654336 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-0: All subvolumes are down. Going offline until at least one of them comes back up.<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.655251 +0000] E [MSGID: 108006] [afr-common.c:6146:__afr_handle_child_down_event] 0-ssd-volume-replicate-1: All subvolumes are down. Going offline until at least one of them comes back up.<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:48:03.525947 +0000] I [io-stats.c:4038:fini] 0-ssd-volume: io-stats translator unloaded<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">real 0m22,068s<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">user 0m0,055s<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">sys 0m0,043s</span><br></div>
<div><br></div>
<div><br></div>
<div>In both cases images are written correctly (and usable):<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# l /data/glusterfs/ssd/brick[12]/brick/libvirt/images/test_[56]*<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">-rw-r--r-- 2 root root 193K mar 19 11:37 /data/glusterfs/ssd/brick2/brick/libvirt/images/test_5.qcow2.img<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">-rw------- 2 root root 193K mar 19 11:47 /data/glusterfs/ssd/brick2/brick/libvirt/images/test_6.qcow2.img</span><br></div>
<div><br></div>
<div>But I get lots of warnings on log files:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log /var/log/glusterfs/glustershd.log /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/glusterd.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.483736 +0000] I [MSGID: 106496] [glusterd-handshake.c:969:__server_getspec] 0-management: Received mount request for volume ssd-volume<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.498908 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr = "127.0.0.1"<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.498967 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.498989 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick1/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.502029 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr = "127.0.0.1"<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.502087 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.502109 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick2/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615182 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49151 failed (No data available)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615258 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615215 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49150 failed (No data available)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615316 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615459 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:42.615524 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:d0cc883b-859c-4143-ab22-83e72673904a-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.522422 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick1/brick: allowed = "*", received addr = "127.0.0.1"<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.522480 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.522502 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick1/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.525344 +0000] I [addr.c:54:compare_addr_and_update] 0-/data/glusterfs/ssd/brick2/brick: allowed = "*", received addr = "127.0.0.1"<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.525396 +0000] I [login.c:110:gf_auth] 0-auth/login: allowed user names: 80c9a2bf-6fad-4023-8458-a175cef7f681<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.525419 +0000] I [MSGID: 115029] [server-handshake.c:561:server_setvolume] 0-ssd-volume-server: accepted client from CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0 (version: 9.0) with subvol /data/glusterfs/ssd/brick2/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.653652 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49146 failed (No data available)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.653731 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.653752 +0000] W [socket.c:767:__socket_rwv] 0-tcp.ssd-volume-server: readv on 127.0.0.1:49144 failed (No data available)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.653840 +0000] I [MSGID: 115036] [server.c:500:server_rpc_notify] 0-ssd-volume-server: disconnecting connection [{client-uid=CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.653928 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.654040 +0000] I [MSGID: 101055] [client_t.c:397:gf_client_unref] 0-ssd-volume-server: Shutting down connection CTX_ID:53e36496-2f5a-4599-9d6b-aeda7822a7a0-GRAPH_ID:0-PID:53235-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/glusterd.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:47:53.507207 +0000] I [MSGID: 106496] [glusterd-handshake.c:969:__server_getspec] 0-management: Received mount request for volume ssd-volume</span><br></div>
<div><br></div>
<div><br></div>
<div>Starting a vm with a virtual disk on the mounted gluster volume <span style="font-family: Courier New,Courier,monospace, sans-serif;" class="font">(-drive file=/mnt/ssd-volume/libvirt/images/test_6.qcow2.img)</span>is quick:</div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# time virsh start vm-11</span><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Domain vm-11 started</span><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">real 0m0,277s</span><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">user 0m0,020s</span><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">sys 0m0,004s</span><br></div>
<div><br></div>
<div>Statring a vm with a virtual disk on gluster volume with native access <span style="font-family: Courier New,Courier,monospace, sans-serif;" class="font">(-drive file=gluster://srv-31.lan.example.com:24007/ssd-volume/libvirt/images/test_7.qcow2.img)</span>is much slower:</div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# time virsh start vm-11<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Domain vm-11 started<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">real 0m22,339s<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">user 0m0,021s<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">sys 0m0,007s</span><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>From time to time, even without user operations, log files report errors:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# tail -f /var/log/glusterfs/glusterd.log /var/log/glusterfs/glustershd.log /var/log/glusterfs/bricks/data-glusterfs-ssd-brick[12]-brick.log<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:09.525079 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:09.525212 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11817}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:13.525581 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:13.525711 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10055}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4b47408f-323c-4c6a-9a20-2ae2a3a2cdb8-GRAPH_ID:3-PID:2291-HOST:srv-32-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:38.829803 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11820}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:38.829970 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10058}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:974f4637-64ef-42e6-afad-1dc9c67c4a43-GRAPH_ID:3-PID:2062-HOST:srv-33-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:38.829721 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick2-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:38.829883 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.005995 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006115 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=10082}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-3-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/bricks/data-glusterfs-ssd-brick1-brick.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006096 +0000] E [MSGID: 113002] [posix-entry-ops.c:682:posix_mkdir] 0-ssd-volume-posix: gfid is null for (null) [Invalid argument]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006212 +0000] E [MSGID: 115056] [server-rpc-fops_v2.c:497:server4_mkdir_cbk] 0-ssd-volume-server: MKDIR info [{frame=11844}, {MKDIR_path=}, {uuid_utoa=00000000-0000-0000-0000-000000000001}, {bname=}, {client=CTX_ID:4945546f-f368-4fa7-8bfc-3dd7abda5d1b-GRAPH_ID:3-PID:2486-HOST:srv-31-PC_NAME:ssd-volume-client-0-RECON_NO:-0}, {error-xlator=ssd-volume-posix}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">==> /var/log/glusterfs/glustershd.log <==<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006255 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-3: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006352 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-0: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006408 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-4: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006505 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-5: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006562 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-1: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">[2021-03-19 11:45:50.006645 +0000] E [MSGID: 114031] [client-rpc-fops_v2.c:214:client4_0_mkdir_cbk] 3-ssd-volume-client-2: remote operation failed. [{path=(null)}, {errno=22}, {error=Invalid argument}]</span><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
<div>Everything seems to be ok<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# gluster volume status ssd-volume<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Status of volume: ssd-volume<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Gluster process TCP Port RDMA Port Online Pid<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">------------------------------------------------------------------------------<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick 49156 0 Y 9242<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick 49156 0 Y 7968<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick 49156 0 Y 8211<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick 49157 0 Y 9258<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick 49157 0 Y 7984<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick 49157 0 Y 8227<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Self-heal Daemon on localhost N/A N/A Y 2486<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Self-heal Daemon on srv-32.lan.example.com N/A N/A Y 2291<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Self-heal Daemon on srv-33.lan.example.com N/A N/A Y 2062<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Task Status of Volume ssd-volume<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">------------------------------------------------------------------------------<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">There are no active volume tasks</span><br></div>
<div><br></div>
<div><br></div>
<div>
<div>
<div><br></div>
</div>
<div>
<div><br></div>
</div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# gluster volume info ssd-volume<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Volume Name: ssd-volume<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Type: Distributed-Replicate<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Volume ID: a6f3426f-5b33-404b-ab61-581e24e0c36d<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Status: Started<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Snapshot Count: 0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Number of Bricks: 2 x 3 = 6<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Transport-type: tcp<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Bricks:<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick1: srv-31.lan.example.com:/data/glusterfs/ssd/brick1/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick2: srv-32.lan.example.com:/data/glusterfs/ssd/brick1/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick3: srv-33.lan.example.com:/data/glusterfs/ssd/brick1/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick4: srv-31.lan.example.com:/data/glusterfs/ssd/brick2/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick5: srv-32.lan.example.com:/data/glusterfs/ssd/brick2/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Brick6: srv-33.lan.example.com:/data/glusterfs/ssd/brick2/brick<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Options Reconfigured:<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">server.allow-insecure: on<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">storage.owner-gid: 64055<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">storage.owner-uid: 64055<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ssl.cipher-list: HIGH:!SSLv2<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">performance.client-io-threads: off<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">nfs.disable: on<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">transport.address-family: inet<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">storage.fips-mode-rchecksum: on<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">client.ssl: off<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">server.ssl: off<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">auth.ssl-allow: *</span><br></div>
<div><br></div>
</div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# gluster peer status<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Number of Peers: 2<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Hostname: srv-32.lan.example.com<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Uuid: 0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">State: Peer in Cluster (Connected)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Hostname: srv-33.lan.example.com<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Uuid: 111d93d2-64aa-463f-8f33-7b53a3f05cab<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">State: Peer in Cluster (Connected)</span><br></div>
<div><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# gluster pool list<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">UUID Hostname State<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a srv-32.lan.example.com Connected<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">111d93d2-64aa-463f-8f33-7b53a3f05cab srv-33.lan.example.com Connected<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">67e9da67-90c5-486b-acb1-eb91abebcefb localhost Connected</span><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-32:~# gluster pool list<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">UUID Hostname State<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">111d93d2-64aa-463f-8f33-7b53a3f05cab srv-33.lan.example.com Connected<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">67e9da67-90c5-486b-acb1-eb91abebcefb 10.0.0.31 Connected<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a localhost Connected</span><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-33:~# gluster pool list<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">UUID Hostname State<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">0a11bfb9-5821-42f0-8ef3-18b36bfcfa8a srv-32.lan.example.com Connected<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">67e9da67-90c5-486b-acb1-eb91abebcefb 10.0.0.31 Connected<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">111d93d2-64aa-463f-8f33-7b53a3f05cab localhost Connected</span><br></div>
<div><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# cat /etc/glusterfs/glusterd.vol<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">volume management<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> type mgmt/glusterd<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option working-directory /var/lib/glusterd<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option transport-type socket<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option transport.socket.keepalive-time 10<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option transport.socket.keepalive-interval 2<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option transport.socket.read-fail-log off<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option transport.socket.listen-port 24007<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option ping-timeout 0<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option event-threads 1<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"># option lock-timer 180<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"># option transport.address-family inet6<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"># option base-port 49152<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option max-port 60999<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"> option rpc-auth-allow-insecure on<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">end-volume</span><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# ss -lntp<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">State Recv-Q Send-Q Local Address:Port Peer Address:Port <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:49153 0.0.0.0:* users:(("glusterfsd",pid=2455,fd=11)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:49156 0.0.0.0:* users:(("glusterfsd",pid=9242,fd=11)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:49157 0.0.0.0:* users:(("glusterfsd",pid=9258,fd=11)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:24007 0.0.0.0:* users:(("glusterd",pid=1954,fd=10)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:111 0.0.0.0:* users:(("rpcbind",pid=1716,fd=4),("systemd",pid=1,fd=36)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:2224 0.0.0.0:* users:(("pcsd",pid=2126,fd=4)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=1963,fd=3)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 20 127.0.0.1:25 0.0.0.0:* users:(("exim4",pid=2377,fd=3)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 0.0.0.0:49152 0.0.0.0:* users:(("glusterfsd",pid=2446,fd=11)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 [::]:111 [::]:* users:(("rpcbind",pid=1716,fd=6),("systemd",pid=1,fd=38)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">LISTEN 0 128 [::]:2224 [::]:* users:(("pcsd",pid=2126,fd=5))</span><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# ss -tnp | grep gluster<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49149 127.0.1.1:24007 users:(("glusterfsd",pid=2446,fd=9)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49144 10.0.0.33:49153 users:(("glusterfs",pid=2486,fd=17)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:24007 127.0.0.1:49145 users:(("glusterd",pid=1954,fd=26)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:49157 127.0.0.1:49140 users:(("glusterfsd",pid=9258,fd=10)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49137 127.0.1.1:49156 users:(("glusterfs",pid=2917,fd=10)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49146 10.0.0.32:49152 users:(("glusterfs",pid=2486,fd=25)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49153 10.0.0.33:49147 users:(("glusterfsd",pid=2455,fd=270)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49148 users:(("glusterd",pid=1954,fd=12)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49157 10.0.0.33:49141 users:(("glusterfsd",pid=9258,fd=271)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49131 users:(("glusterd",pid=1954,fd=31)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49152 10.0.0.32:49146 users:(("glusterfsd",pid=2446,fd=271)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49134 10.0.0.33:24007 users:(("glusterd",pid=1954,fd=33)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49127 10.0.0.33:49157 users:(("glusterfs",pid=2917,fd=11)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49142 127.0.1.1:49156 users:(("glusterfs",pid=2486,fd=15)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49135 127.0.1.1:49157 users:(("glusterfs",pid=2917,fd=14)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49145 127.0.0.1:24007 users:(("glusterfs",pid=2486,fd=9)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:49157 127.0.0.1:49135 users:(("glusterfsd",pid=9258,fd=272)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:49152 127.0.0.1:49141 users:(("glusterfsd",pid=2446,fd=10)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49157 10.0.0.32:49144 users:(("glusterfsd",pid=9258,fd=270)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49132 users:(("glusterd",pid=1954,fd=25)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49148 127.0.1.1:24007 users:(("glusterfsd",pid=2455,fd=9)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49118 10.0.0.32:49156 users:(("glusterfs",pid=2486,fd=13)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:49156 127.0.0.1:49137 users:(("glusterfsd",pid=9242,fd=272)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49121 10.0.0.33:49156 users:(("glusterfs",pid=2917,fd=13)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49132 127.0.1.1:24007 users:(("glusterfsd",pid=9242,fd=9)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49141 127.0.1.1:49152 users:(("glusterfs",pid=2486,fd=22)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49156 10.0.0.32:49148 users:(("glusterfsd",pid=9242,fd=270)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49152 10.0.0.33:49149 users:(("glusterfsd",pid=2446,fd=270)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49140 127.0.1.1:49157 users:(("glusterfs",pid=2486,fd=11)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49123 10.0.0.32:49157 users:(("glusterfs",pid=2917,fd=12)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49141 10.0.0.33:49152 users:(("glusterfs",pid=2486,fd=23)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49131 127.0.1.1:24007 users:(("glusterfsd",pid=9258,fd=9)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49153 10.0.0.32:49145 users:(("glusterfsd",pid=2455,fd=271)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49117 10.0.0.33:49156 users:(("glusterfs",pid=2486,fd=5)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49114 10.0.0.32:49157 users:(("glusterfs",pid=2486,fd=16)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:49153 127.0.0.1:49138 users:(("glusterfsd",pid=2455,fd=10)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49143 10.0.0.32:24007 users:(("glusterfs",pid=2917,fd=9)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49113 10.0.0.33:49157 users:(("glusterfs",pid=2486,fd=18)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49156 10.0.0.33:49143 users:(("glusterfsd",pid=9242,fd=271)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.0.1:49138 127.0.1.1:49153 users:(("glusterfs",pid=2486,fd=19)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:24007 10.0.0.33:49150 users:(("glusterd",pid=1954,fd=32)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49131 10.0.0.32:49156 users:(("glusterfs",pid=2917,fd=7)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49145 10.0.0.32:49153 users:(("glusterfs",pid=2486,fd=8)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:24007 127.0.0.1:49149 users:(("glusterd",pid=1954,fd=13)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:24007 10.0.0.32:49150 users:(("glusterd",pid=1954,fd=7)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 10.0.0.31:49151 10.0.0.32:24007 users:(("glusterd",pid=1954,fd=8)) <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">ESTAB 0 0 127.0.1.1:49156 127.0.0.1:49142 users:(("glusterfsd",pid=9242,fd=10))</span><br></div>
<div><br></div>
<div>hostnames resolve correctly:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# for i in {31..33}; do host srv-$i.lan.example.com; done<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">srv-31.lan.example.com has address 10.0.0.31<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">srv-32.lan.example.com has address 10.0.0.32<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">srv-33.lan.example.com has address 10.0.0.33</span><br></div>
<div><br></div>
<div><br></div>
<div>And network conectivity is ok:<br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# for i in {31..33}; do ping -c 1 -q srv-$i.lan.example.com | echo "srv-$i ok"; done<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">srv-31 ok<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">srv-32 ok<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">srv-33 ok</span><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# for i in {31..33}; do nc -v srv-$i.lan.example.com 24007; done<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Connection to srv-31.lan.example.com 24007 port [tcp/*] succeeded!<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Connection to srv-32.lan.example.com 24007 port [tcp/*] succeeded!<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Connection to srv-33.lan.example.com 24007 port [tcp/*] succeeded!</span><br></div>
<div><br></div>
<div><br></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">root@srv-31:~# iptables -L<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Chain INPUT (policy ACCEPT)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">target prot opt source destination <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Chain FORWARD (policy ACCEPT)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">target prot opt source destination <br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;"><br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">Chain OUTPUT (policy ACCEPT)<br></span></div>
<div><span class="font" style="font-family: Courier New,Courier,monospace, sans-serif;">target prot opt source destination</span><br></div>
<div><br></div>
<div><br></div>
<div><br></div>
</body>
</html>