<div dir="ltr"><div>Could you enable strict-o-direct and disable remote-dio on the src volume as well, restart the vms on "old" and retry migration?</div><div><br></div><div># gluster volume set <VOLNAME> performance.strict-o-direct on</div><div># gluster volume set <VOLNAME> network.remote-dio off</div><div><br></div><div>-Krutika<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Mar 26, 2019 at 10:32 PM Sander Hoentjen <<a href="mailto:sander@hoentjen.eu">sander@hoentjen.eu</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 26-03-19 14:23, Sahina Bose wrote:<br>
> +Krutika Dhananjay and gluster ml<br>
><br>
> On Tue, Mar 26, 2019 at 6:16 PM Sander Hoentjen <<a href="mailto:sander@hoentjen.eu" target="_blank">sander@hoentjen.eu</a>> wrote:<br>
>> Hello,<br>
>><br>
>> tl;dr We have disk corruption when doing live storage migration on oVirt<br>
>> 4.2 with gluster 3.12.15. Any idea why?<br>
>><br>
>> We have a 3-node oVirt cluster that is both compute and gluster-storage.<br>
>> The manager runs on separate hardware. We are running out of space on<br>
>> this volume, so we added another Gluster volume that is bigger, put a<br>
>> storage domain on it and then we migrated VM's to it with LSM. After<br>
>> some time, we noticed that (some of) the migrated VM's had corrupted<br>
>> filesystems. After moving everything back with export-import to the old<br>
>> domain where possible, and recovering from backups where needed we set<br>
>> off to investigate this issue.<br>
>><br>
>> We are now at the point where we can reproduce this issue within a day.<br>
>> What we have found so far:<br>
>> 1) The corruption occurs at the very end of the replication step, most<br>
>> probably between START and FINISH of diskReplicateFinish, before the<br>
>> START merge step<br>
>> 2) In the corrupted VM, at some place where data should be, this data is<br>
>> replaced by zero's. This can be file-contents or a directory-structure<br>
>> or whatever.<br>
>> 3) The source gluster volume has different settings then the destination<br>
>> (Mostly because the defaults were different at creation time):<br>
>><br>
>> Setting old(src) new(dst)<br>
>> cluster.op-version 30800 30800 (the same)<br>
>> cluster.max-op-version 31202 31202 (the same)<br>
>> cluster.metadata-self-heal off on<br>
>> cluster.data-self-heal off on<br>
>> cluster.entry-self-heal off on<br>
>> performance.low-prio-threads 16 32<br>
>> performance.strict-o-direct off on<br>
>> network.ping-timeout 42 30<br>
>> network.remote-dio enable off<br>
>> transport.address-family - inet<br>
>> performance.stat-prefetch off on<br>
>> features.shard-block-size 512MB 64MB<br>
>> cluster.shd-max-threads 1 8<br>
>> cluster.shd-wait-qlength 1024 10000<br>
>> cluster.locking-scheme full granular<br>
>> cluster.granular-entry-heal no enable<br>
>><br>
>> 4) To test, we migrate some VM's back and forth. The corruption does not<br>
>> occur every time. To this point it only occurs from old to new, but we<br>
>> don't have enough data-points to be sure about that.<br>
>><br>
>> Anybody an idea what is causing the corruption? Is this the best list to<br>
>> ask, or should I ask on a Gluster list? I am not sure if this is oVirt<br>
>> specific or Gluster specific though.<br>
> Do you have logs from old and new gluster volumes? Any errors in the<br>
> new volume's fuse mount logs?<br>
<br>
Around the time of corruption I see the message:<br>
The message "I [MSGID: 133017] [shard.c:4941:shard_seek] 0-ZoneA_Gluster1-shard: seek called on 7fabc273-3d8a-4a49-8906-b8ccbea4a49f. [Operation not supported]" repeated 231 times between [2019-03-26 13:14:22.297333] and [2019-03-26 13:15:42.912170]<br>
<br>
I also see this message at other times, when I don't see the corruption occur, though.<br>
<br>
-- <br>
Sander<br>
_______________________________________________<br>
Users mailing list -- <a href="mailto:users@ovirt.org" target="_blank">users@ovirt.org</a><br>
To unsubscribe send an email to <a href="mailto:users-leave@ovirt.org" target="_blank">users-leave@ovirt.org</a><br>
Privacy Statement: <a href="https://www.ovirt.org/site/privacy-policy/" rel="noreferrer" target="_blank">https://www.ovirt.org/site/privacy-policy/</a><br>
oVirt Code of Conduct: <a href="https://www.ovirt.org/community/about/community-guidelines/" rel="noreferrer" target="_blank">https://www.ovirt.org/community/about/community-guidelines/</a><br>
List Archives: <a href="https://lists.ovirt.org/archives/list/users@ovirt.org/message/M3T2VGGGV6DE643ZKKJUAF274VSWTJFH/" rel="noreferrer" target="_blank">https://lists.ovirt.org/archives/list/users@ovirt.org/message/M3T2VGGGV6DE643ZKKJUAF274VSWTJFH/</a><br>
</blockquote></div></div>