[Gluster-users] [ovirt-users] Re: VM disk corruption with LSM on Gluster

[Sander Hoentjen]

On 26-03-19 14:23, Sahina Bose wrote:
> +Krutika Dhananjay and gluster ml
>
> On Tue, Mar 26, 2019 at 6:16 PM Sander Hoentjen <sander at hoentjen.eu> wrote:
>> Hello,
>>
>> tl;dr We have disk corruption when doing live storage migration on oVirt
>> 4.2 with gluster 3.12.15. Any idea why?
>>
>> We have a 3-node oVirt cluster that is both compute and gluster-storage.
>> The manager runs on separate hardware. We are running out of space on
>> this volume, so we added another Gluster volume that is bigger, put a
>> storage domain on it and then we migrated VM's to it with LSM. After
>> some time, we noticed that (some of) the migrated VM's had corrupted
>> filesystems. After moving everything back with export-import to the old
>> domain where possible, and recovering from backups where needed we set
>> off to investigate this issue.
>>
>> We are now at the point where we can reproduce this issue within a day.
>> What we have found so far:
>> 1) The corruption occurs at the very end of the replication step, most
>> probably between START and FINISH of diskReplicateFinish, before the
>> START merge step
>> 2) In the corrupted VM, at some place where data should be, this data is
>> replaced by zero's. This can be file-contents or a directory-structure
>> or whatever.
>> 3) The source gluster volume has different settings then the destination
>> (Mostly because the defaults were different at creation time):
>>
>> Setting                                 old(src)  new(dst)
>> cluster.op-version                      30800     30800 (the same)
>> cluster.max-op-version                  31202     31202 (the same)
>> cluster.metadata-self-heal              off       on
>> cluster.data-self-heal                  off       on
>> cluster.entry-self-heal                 off       on
>> performance.low-prio-threads            16        32
>> performance.strict-o-direct             off       on
>> network.ping-timeout                    42        30
>> network.remote-dio                      enable    off
>> transport.address-family                -         inet
>> performance.stat-prefetch               off       on
>> features.shard-block-size               512MB     64MB
>> cluster.shd-max-threads                 1         8
>> cluster.shd-wait-qlength                1024      10000
>> cluster.locking-scheme                  full      granular
>> cluster.granular-entry-heal             no        enable
>>
>> 4) To test, we migrate some VM's back and forth. The corruption does not
>> occur every time. To this point it only occurs from old to new, but we
>> don't have enough data-points to be sure about that.
>>
>> Anybody an idea what is causing the corruption? Is this the best list to
>> ask, or should I ask on a Gluster list? I am not sure if this is oVirt
>> specific or Gluster specific though.
> Do you have logs from old and new gluster volumes? Any errors in the
> new volume's fuse mount logs?

Around the time of corruption I see the message:
The message "I [MSGID: 133017] [shard.c:4941:shard_seek] 0-ZoneA_Gluster1-shard: seek called on 7fabc273-3d8a-4a49-8906-b8ccbea4a49f. [Operation not supported]" repeated 231 times between [2019-03-26 13:14:22.297333] and [2019-03-26 13:15:42.912170]

I also see this message at other times, when I don't see the corruption occur, though.

-- 
Sander