[Gluster-users] Gluster 3.8.10 rebalance VMs corruption

[Krutika Dhananjay]

mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse
mount logs? It should be right under /var/log/glusterfs/ directory
named after the mount point name, only hyphenated.

-Krutika

On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan <mahdi.adnan at outlook.com>
wrote:

> Hello Krutika,
>
>
> Kindly, check the attached logs.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ------------------------------
> *From:* Krutika Dhananjay <kdhananj at redhat.com>
> *Sent:* Saturday, March 18, 2017 3:29:03 PM
> *To:* Mahdi Adnan
> *Cc:* gluster-users at gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> Hi Mahdi,
>
> Could you attach mount, brick and rebalance logs?
>
> -Krutika
>
> On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan <mahdi.adnan at outlook.com>
> wrote:
>
>> Hi,
>>
>> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure
>> in a volume contains few VMs.
>> After the completion of rebalance, i have rebooted the VMs, some of ran
>> just fine, and others just crashed.
>> Windows boot to recovery mode and Linux throw xfs errors and does not
>> boot.
>> I ran the test again and it happened just as the first one, but i have
>> noticed only VMs doing disk IOs are affected by this bug.
>> The VMs in power off mode started fine and even md5 of the disk file did
>> not change after the rebalance.
>>
>> anyone else can confirm this ?
>>
>>
>> Volume info:
>>
>> Volume Name: vmware2
>> Type: Distributed-Replicate
>> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 22 x 2 = 44
>> Transport-type: tcp
>> Bricks:
>> Brick1: gluster01:/mnt/disk1/vmware2
>> Brick2: gluster03:/mnt/disk1/vmware2
>> Brick3: gluster02:/mnt/disk1/vmware2
>> Brick4: gluster04:/mnt/disk1/vmware2
>> Brick5: gluster01:/mnt/disk2/vmware2
>> Brick6: gluster03:/mnt/disk2/vmware2
>> Brick7: gluster02:/mnt/disk2/vmware2
>> Brick8: gluster04:/mnt/disk2/vmware2
>> Brick9: gluster01:/mnt/disk3/vmware2
>> Brick10: gluster03:/mnt/disk3/vmware2
>> Brick11: gluster02:/mnt/disk3/vmware2
>> Brick12: gluster04:/mnt/disk3/vmware2
>> Brick13: gluster01:/mnt/disk4/vmware2
>> Brick14: gluster03:/mnt/disk4/vmware2
>> Brick15: gluster02:/mnt/disk4/vmware2
>> Brick16: gluster04:/mnt/disk4/vmware2
>> Brick17: gluster01:/mnt/disk5/vmware2
>> Brick18: gluster03:/mnt/disk5/vmware2
>> Brick19: gluster02:/mnt/disk5/vmware2
>> Brick20: gluster04:/mnt/disk5/vmware2
>> Brick21: gluster01:/mnt/disk6/vmware2
>> Brick22: gluster03:/mnt/disk6/vmware2
>> Brick23: gluster02:/mnt/disk6/vmware2
>> Brick24: gluster04:/mnt/disk6/vmware2
>> Brick25: gluster01:/mnt/disk7/vmware2
>> Brick26: gluster03:/mnt/disk7/vmware2
>> Brick27: gluster02:/mnt/disk7/vmware2
>> Brick28: gluster04:/mnt/disk7/vmware2
>> Brick29: gluster01:/mnt/disk8/vmware2
>> Brick30: gluster03:/mnt/disk8/vmware2
>> Brick31: gluster02:/mnt/disk8/vmware2
>> Brick32: gluster04:/mnt/disk8/vmware2
>> Brick33: gluster01:/mnt/disk9/vmware2
>> Brick34: gluster03:/mnt/disk9/vmware2
>> Brick35: gluster02:/mnt/disk9/vmware2
>> Brick36: gluster04:/mnt/disk9/vmware2
>> Brick37: gluster01:/mnt/disk10/vmware2
>> Brick38: gluster03:/mnt/disk10/vmware2
>> Brick39: gluster02:/mnt/disk10/vmware2
>> Brick40: gluster04:/mnt/disk10/vmware2
>> Brick41: gluster01:/mnt/disk11/vmware2
>> Brick42: gluster03:/mnt/disk11/vmware2
>> Brick43: gluster02:/mnt/disk11/vmware2
>> Brick44: gluster04:/mnt/disk11/vmware2
>> Options Reconfigured:
>> cluster.server-quorum-type: server
>> nfs.disable: on
>> performance.readdir-ahead: on
>> transport.address-family: inet
>> performance.quick-read: off
>> performance.read-ahead: off
>> performance.io-cache: off
>> performance.stat-prefetch: off
>> cluster.eager-lock: enable
>> network.remote-dio: enable
>> features.shard: on
>> cluster.data-self-heal-algorithm: full
>> features.cache-invalidation: on
>> ganesha.enable: on
>> features.shard-block-size: 256MB
>> client.event-threads: 2
>> server.event-threads: 2
>> cluster.favorite-child-policy: size
>> storage.build-pgfid: off
>> network.ping-timeout: 5
>> cluster.enable-shared-storage: enable
>> nfs-ganesha: enable
>> cluster.server-quorum-ratio: 51%
>>
>>
>> Adding bricks:
>> gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2
>> gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2
>> gluster04:/mnt/disk11/vmware2
>>
>>
>> starting fix layout:
>> gluster volume rebalance vmware2 fix-layout start
>>
>> Starting rebalance:
>> gluster volume rebalance vmware2  start
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170318/c3cb32ed/attachment.html>