<html><head></head><body><div class="ydpca800cd5yahoo-style-wrap" style="font-family:courier new, courier, monaco, monospace, sans-serif;font-size:16px;"><div id="ydpca800cd5yiv3543031743"><div><div class="ydpca800cd5yiv3543031743ydp39efa2dyahoo-style-wrap" style="font-family:courier new, courier, monaco, monospace, sans-serif;font-size:16px;"><div></div>
<div>Hello All,</div><div><br clear="none"></div><div>I have tried to enable debug and see the reason for the issue. Here is the relevant glusterd.log:</div><div><br clear="none"></div><div><span>[2019-04-12 07:56:54.526508] E [MSGID: 106077] [glusterd-snapshot.c:1882:glusterd_is_thinp_brick] 0-management: Failed to get pool name for device systemd-1<br clear="none">[2019-04-12 07:56:54.527509] E [MSGID: 106121] [glusterd-snapshot.c:2523:glusterd_snapshot_create_prevalidate] 0-management: Failed to pre validate<br clear="none">[2019-04-12 07:56:54.527525] E [MSGID: 106024] [glusterd-snapshot.c:2547:glusterd_snapshot_create_prevalidate] 0-management: Snapshot is supported only for thin provisioned LV. Ensure that all bricks of isos are thinly provisioned LV.<br clear="none">[2019-04-12 07:56:54.527539] W [MSGID: 106029] [glusterd-snapshot.c:8613:glusterd_snapshot_prevalidate] 0-management: Snapshot create pre-validation failed<br clear="none">[2019-04-12 07:56:54.527552] W [MSGID: 106121] [glusterd-mgmt.c:147:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed<br clear="none">[2019-04-12 07:56:54.527568] E [MSGID: 106121] [glusterd-mgmt.c:1015:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node<br clear="none">[2019-04-12 07:56:54.527583] E [MSGID: 106121] [glusterd-mgmt.c:2377:glusterd_mgmt_v3_initiate_snap_phases] 0-management: Pre Validation Failed<br clear="none"><br clear="none"></span><div>here is the output of lvscan & lvs:</div><div><br clear="none"></div><div><span>[root@ovirt1 ~]# lvscan<br clear="none"> ACTIVE '/dev/gluster_vg_md0/my_vdo_thinpool' [9.86 TiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_md0/gluster_lv_data' [500.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_md0/gluster_lv_isos' [50.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_ssd/my_ssd_thinpool' [168.59 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_ssd/gluster_lv_engine' [40.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt1/swap' [6.70 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt1/home' [1.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt1/root' [60.00 GiB] inherit<br clear="none">[root@ovirt1 ~]# lvs --noheadings -o pool_lv<br clear="none"><br clear="none"><br clear="none"><br clear="none"> my_vdo_thinpool<br clear="none"> my_vdo_thinpool<br clear="none"><br clear="none"> my_ssd_thinpool<br clear="none"><br clear="none">[root@ovirt1 ~]# ssh ovirt2 "lvscan;lvs --noheadings -o pool_lv"<br clear="none"> ACTIVE '/dev/gluster_vg_md0/my_vdo_thinpool' [<9.77 TiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_md0/gluster_lv_data' [500.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_md0/gluster_lv_isos' [50.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_ssd/my_ssd_thinpool' [<161.40 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_ssd/gluster_lv_engine' [40.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt2/root' [15.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt2/home' [1.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt2/swap' [16.00 GiB] inherit<br clear="none"><br clear="none"><br clear="none"><br clear="none"> my_vdo_thinpool<br clear="none"> my_vdo_thinpool<br clear="none"><br clear="none"> my_ssd_thinpool<br clear="none"><br clear="none">[root@ovirt1 ~]# ssh ovirt3 "lvscan;lvs --noheadings -o pool_lv"<br clear="none"> ACTIVE '/dev/gluster_vg_sda3/gluster_thinpool_sda3' [41.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_sda3/gluster_lv_data' [15.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_sda3/gluster_lv_isos' [15.00 GiB] inherit<br clear="none"> ACTIVE '/dev/gluster_vg_sda3/gluster_lv_engine' [15.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt3/root' [20.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt3/home' [1.00 GiB] inherit<br clear="none"> ACTIVE '/dev/centos_ovirt3/swap' [8.00 GiB] inherit<br clear="none"><br clear="none"><br clear="none"><br clear="none"> gluster_thinpool_sda3<br clear="none"> gluster_thinpool_sda3<br clear="none"> gluster_thinpool_sda3<br clear="none"><br clear="none"><br clear="none"></span><div>I am mounting my bricks via systemd , as I have issues with bricks being started before VDO.</div><div><br clear="none"></div><div><span>[root@ovirt1 ~]# findmnt /gluster_bricks/isos<br>TARGET SOURCE FSTYPE OPTIONS<br>/gluster_bricks/isos systemd-1 autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21843<br>/gluster_bricks/isos /dev/mapper/gluster_vg_md0-gluster_lv_isos xfs rw,noatime,nodiratime,seclabel,attr2,inode64,noquota<br>[root@ovirt1 ~]# ssh ovirt2 "findmnt /gluster_bricks/isos "<br>TARGET SOURCE FSTYPE OPTIONS<br>/gluster_bricks/isos systemd-1 autofs rw,relatime,fd=26,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14279<br>/gluster_bricks/isos /dev/mapper/gluster_vg_md0-gluster_lv_isos xfs rw,noatime,nodiratime,seclabel,attr2,inode64,noquota<br>[root@ovirt1 ~]# ssh ovirt3 "findmnt /gluster_bricks/isos "<br>TARGET SOURCE FSTYPE OPTIONS<br>/gluster_bricks/isos systemd-1 autofs rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17770<br>/gluster_bricks/isos /dev/mapper/gluster_vg_sda3-gluster_lv_isos xfs rw,noatime,nodiratime,seclabel,attr2,inode64,logbsize=256k,sunit=512,swidth=1024,noquota<br></span><span></span><div><br></div><div><br></div><div><span>[root@ovirt1 ~]# grep "gluster_bricks" /proc/mounts<br>systemd-1 /gluster_bricks/data autofs rw,relatime,fd=22,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21513 0 0<br>systemd-1 /gluster_bricks/engine autofs rw,relatime,fd=25,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21735 0 0<br>systemd-1 /gluster_bricks/isos autofs rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=21843 0 0<br>/dev/mapper/gluster_vg_ssd-gluster_lv_engine /gluster_bricks/engine xfs rw,seclabel,noatime,nodiratime,attr2,inode64,sunit=256,swidth=256,noquota 0 0<br>/dev/mapper/gluster_vg_md0-gluster_lv_isos /gluster_bricks/isos xfs rw,seclabel,noatime,nodiratime,attr2,inode64,noquota 0 0<br>/dev/mapper/gluster_vg_md0-gluster_lv_data /gluster_bricks/data xfs rw,seclabel,noatime,nodiratime,attr2,inode64,noquota 0 0<br><br></span><div><br></div><div><br></div></div><div><br></div><div>Obviously , gluster is catching "systemd-1" as a device and tries to check if it's a thin LV.</div><div>Where should I open a bug for that ?</div><div><br></div><div>P.S.: Adding oVirt User list.<br></div></div><div><br clear="none"></div></div><div>Best Regards,</div><div>Strahil Nikolov<br clear="none"></div><div><br clear="none"></div></div><div><br clear="none"></div>
</div><div class="ydpca800cd5yiv3543031743ydpd10f78b9yahoo_quoted" id="ydpca800cd5yiv3543031743ydpd10f78b9yahoo_quoted_5432582668">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
В четвъртък, 11 април 2019 г., 4:00:31 ч. Гринуич-4, Strahil Nikolov <hunter86_bg@yahoo.com> написа:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
</div>
</div></div></div></div><div class="ydpd87bd613yiv3543031743yqt7882736846" id="ydpd87bd613yiv3543031743yqt17582"><div><div id="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118"><div><div class="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydpf10cc4c6yahoo-style-wrap" style="font-family:courier new, courier, monaco, monospace, sans-serif;font-size:16px;"><div></div>
<div>Hi Rafi,</div><div><br clear="none"></div><div>thanks for your update.</div><div><br clear="none"></div><div>I have tested again with another gluster volume.</div><div><span>[root@ovirt1 glusterfs]# gluster volume info isos<br clear="none"><br clear="none">Volume Name: isos<br clear="none">Type: Replicate<br clear="none">Volume ID: 9b92b5bd-79f5-427b-bd8d-af28b038ed2a<br clear="none">Status: Started<br clear="none">Snapshot Count: 0<br clear="none">Number of Bricks: 1 x (2 + 1) = 3<br clear="none">Transport-type: tcp<br clear="none">Bricks:<br clear="none">Brick1: ovirt1:/gluster_bricks/isos/isos<br clear="none">Brick2: ovirt2:/gluster_bricks/isos/isos<br clear="none">Brick3: ovirt3.localdomain:/gluster_bricks/isos/isos (arbiter)<br clear="none">Options Reconfigured:<br clear="none">cluster.granular-entry-heal: enable<br clear="none">performance.strict-o-direct: on<br clear="none">network.ping-timeout: 30<br clear="none">storage.owner-gid: 36<br clear="none">storage.owner-uid: 36<br clear="none">user.cifs: off<br clear="none">features.shard: on<br clear="none">cluster.shd-wait-qlength: 10000<br clear="none">cluster.shd-max-threads: 8<br clear="none">cluster.locking-scheme: granular<br clear="none">cluster.data-self-heal-algorithm: full<br clear="none">cluster.server-quorum-type: server<br clear="none">cluster.quorum-type: auto<br clear="none">cluster.eager-lock: enable<br clear="none">network.remote-dio: off<br clear="none">performance.low-prio-threads: 32<br clear="none">performance.io-cache: off<br clear="none">performance.read-ahead: off<br clear="none">performance.quick-read: off<br clear="none">transport.address-family: inet<br clear="none">nfs.disable: on<br clear="none">performance.client-io-threads: off<br clear="none">cluster.enable-shared-storage: enable</span><br clear="none"></div><div><br clear="none"></div><div>Command run:<br clear="none"></div><div><span>logrotate -f glusterfs ; logrotate -f glusterfs-georep; gluster snapshot create isos-snap-2019-04-11 isos description TEST<br clear="none"></span><span></span><div><br clear="none"></div><div>Logs:</div><div><span>[root@ovirt1 glusterfs]# cat cli.log<br clear="none">[2019-04-11 07:51:02.367453] I [cli.c:769:main] 0-cli: Started running gluster with version 5.5<br clear="none">[2019-04-11 07:51:02.486863] I [MSGID: 101190] [event-epoll.c:621:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<br clear="none">[2019-04-11 07:51:02.556813] E [cli-rpc-ops.c:11293:gf_cli_snapshot] 0-cli: cli_to_glusterd for snapshot failed<br clear="none">[2019-04-11 07:51:02.556880] I [input.c:31:cli_batch] 0-: Exiting with: -1<br clear="none">[root@ovirt1 glusterfs]# cat glusterd.log<br clear="none">[2019-04-11 07:51:02.553357] E [MSGID: 106024] [glusterd-snapshot.c:2547:glusterd_snapshot_create_prevalidate] 0-management: Snapshot is supported only for thin provisioned LV. Ensure that all bricks of isos are thinly provisioned LV.<br clear="none">[2019-04-11 07:51:02.553365] W [MSGID: 106029] [glusterd-snapshot.c:8613:glusterd_snapshot_prevalidate] 0-management: Snapshot create pre-validation failed<br clear="none">[2019-04-11 07:51:02.553703] W [MSGID: 106121] [glusterd-mgmt.c:147:gd_mgmt_v3_pre_validate_fn] 0-management: Snapshot Prevalidate Failed<br clear="none"></span><div><span>[2019-04-11 07:51:02.553719] E [MSGID: 106121] [glusterd-mgmt.c:1015:glusterd_mgmt_v3_pre_validate] 0-management: Pre Validation failed for operation Snapshot on local node<br clear="none"></span></div><div><br clear="none"></div><div><div>My LVs hosting the bricks are:</div><div><span>[root@ovirt1 ~]# lvs gluster_vg_md0<br clear="none"> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert<br clear="none"> gluster_lv_data gluster_vg_md0 Vwi-aot--- 500.00g my_vdo_thinpool 35.97<br clear="none"> gluster_lv_isos gluster_vg_md0 Vwi-aot--- 50.00g my_vdo_thinpool 52.11<br clear="none"> my_vdo_thinpool gluster_vg_md0 twi-aot--- 9.86t 2.04 11.45<br clear="none"><br clear="none"></span><span>[root@ovirt1 ~]# ssh ovirt2 "lvs gluster_vg_md0"<br clear="none"> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert<br clear="none"> gluster_lv_data gluster_vg_md0 Vwi-aot--- 500.00g my_vdo_thinpool 35.98<br clear="none"> gluster_lv_isos gluster_vg_md0 Vwi-aot--- 50.00g my_vdo_thinpool 25.94<br clear="none"> my_vdo_thinpool gluster_vg_md0 twi-aot--- <9.77t 1.93 11.39<br clear="none">[root@ovirt1 ~]# ssh ovirt3 "lvs gluster_vg_sda3"<br clear="none"> LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert<br clear="none"> gluster_lv_data gluster_vg_sda3 Vwi-aotz-- 15.00g gluster_thinpool_sda3 0.17<br clear="none"> gluster_lv_engine gluster_vg_sda3 Vwi-aotz-- 15.00g gluster_thinpool_sda3 0.16<br clear="none"> gluster_lv_isos gluster_vg_sda3 Vwi-aotz-- 15.00g gluster_thinpool_sda3 0.12<br clear="none"> gluster_thinpool_sda3 gluster_vg_sda3 twi-aotz-- 41.00g 0.16 1.58<br clear="none"><br clear="none"></span><div>As you can see - all bricks are thin LV and space is not the issue.</div><div><br clear="none"></div><div>Can someone hint me how to enable debug , so gluster logs can show the reason for that pre-check failure ?</div><div><br clear="none"></div><div>Best Regards,</div><div>Strahil Nikolov<br clear="none"></div></div><span></span></div></div><div><br clear="none"></div></div><div><br clear="none"></div>
</div><div class="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydp848d3c78yahoo_quoted" id="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydp848d3c78yahoo_quoted_5091806849">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
В сряда, 10 април 2019 г., 9:05:15 ч. Гринуич-4, Rafi Kavungal Chundattu Parambil <rkavunga@redhat.com> написа:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div class="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118yqt2100580676" id="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118yqt11158"><div><div dir="ltr">Hi Strahil,<br clear="none"><br clear="none">The name of device is not at all a problem here. Can you please check the log of glusterd, and see if there is any useful information about the failure. Also please provide the output of `lvscan` and `lvs --noheadings -o pool_lv` from all nodes<br clear="none"><br clear="none">Regards<br clear="none">Rafi KC<br clear="none"><div class="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydp848d3c78yqt0187791214" id="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydp848d3c78yqtfd10892"><br clear="none">----- Original Message -----<br clear="none">From: "Strahil Nikolov" <<a shape="rect" href="mailto:hunter86_bg@yahoo.com" rel="nofollow" target="_blank">hunter86_bg@yahoo.com</a>><br clear="none">To: <a shape="rect" href="mailto:gluster-users@gluster.org" rel="nofollow" target="_blank">gluster-users@gluster.org</a><br clear="none">Sent: Wednesday, April 10, 2019 2:36:39 AM<br clear="none">Subject: [Gluster-users] Gluster snapshot fails<br clear="none"><br clear="none">Hello Community, <br clear="none"><br clear="none">I have a problem running a snapshot of a replica 3 arbiter 1 volume. <br clear="none"><br clear="none">Error: <br clear="none">[<a shape="rect" href="mailto:root@ovirt2" rel="nofollow" target="_blank">root@ovirt2</a> ~]# gluster snapshot create before-423 engine description "Before upgrade of engine from 4.2.2 to 4.2.3" <br clear="none">snapshot create: failed: Snapshot is supported only for thin provisioned LV. Ensure that all bricks of engine are thinly provisioned LV. <br clear="none">Snapshot command failed <br clear="none"><br clear="none">Volume info: <br clear="none"><br clear="none">Volume Name: engine <br clear="none">Type: Replicate <br clear="none">Volume ID: 30ca1cc2-f2f7-4749-9e2e-cee9d7099ded <br clear="none">Status: Started <br clear="none">Snapshot Count: 0 <br clear="none">Number of Bricks: 1 x (2 + 1) = 3 <br clear="none">Transport-type: tcp <br clear="none">Bricks: <br clear="none">Brick1: ovirt1:/gluster_bricks/engine/engine <br clear="none">Brick2: ovirt2:/gluster_bricks/engine/engine <br clear="none">Brick3: ovirt3:/gluster_bricks/engine/engine (arbiter) <br clear="none">Options Reconfigured: <br clear="none">cluster.granular-entry-heal: enable <br clear="none">performance.strict-o-direct: on <br clear="none">network.ping-timeout: 30 <br clear="none">storage.owner-gid: 36 <br clear="none">storage.owner-uid: 36 <br clear="none">user.cifs: off <br clear="none">features.shard: on <br clear="none">cluster.shd-wait-qlength: 10000 <br clear="none">cluster.shd-max-threads: 8 <br clear="none">cluster.locking-scheme: granular <br clear="none">cluster.data-self-heal-algorithm: full <br clear="none">cluster.server-quorum-type: server <br clear="none">cluster.quorum-type: auto <br clear="none">cluster.eager-lock: enable <br clear="none">network.remote-dio: off <br clear="none">performance.low-prio-threads: 32 <br clear="none">performance.io-cache: off <br clear="none">performance.read-ahead: off <br clear="none">performance.quick-read: off <br clear="none">transport.address-family: inet <br clear="none">nfs.disable: on <br clear="none">performance.client-io-threads: off <br clear="none">cluster.enable-shared-storage: enable <br clear="none"><br clear="none"><br clear="none">All bricks are on thin lvm with plenty of space, the only thing that could be causing it is that ovirt1 & ovirt2 are on /dev/gluster_vg_ssd/gluster_lv_engine , while arbiter is on /dev/gluster_vg_sda3/gluster_lv_engine. <br clear="none"><br clear="none">Is that the issue ? Should I rename my brick's VG ? <br clear="none">If so, why there is no mentioning in the documentation ? <br clear="none"><br clear="none"><br clear="none">Best Regards, <br clear="none">Strahil Nikolov </div><br clear="none"><br clear="none"><br clear="none">_______________________________________________<br clear="none">Gluster-users mailing list<br clear="none"><a shape="rect" href="mailto:Gluster-users@gluster.org" rel="nofollow" target="_blank">Gluster-users@gluster.org</a><br clear="none"><a shape="rect" href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="nofollow" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a><div class="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydp848d3c78yqt0187791214" id="ydpd87bd613yiv3543031743ydpd10f78b9yiv7519249118ydp848d3c78yqtfd14983"><br clear="none"></div></div></div></div>
</div>
</div></div></div></div></div></body></html>