[Gluster-users] Gluster 3.8.10 rebalance VMs corruption

Sun Mar 19 05:02:19 UTC 2017

On Sat, Mar 18, 2017 at 10:36 PM, Mahdi Adnan <mahdi.adnan at outlook.com>
wrote:

> Kindly, check the attached new log file, i dont know if it's helpful or
> not but, i couldn't find the log with the name you just described.
>
No. Are you using FUSE or libgfapi for accessing the volume? Or is it NFS?

-Krutika

>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ------------------------------
> *From:* Krutika Dhananjay <kdhananj at redhat.com>
> *Sent:* Saturday, March 18, 2017 6:10:40 PM
>
> *To:* Mahdi Adnan
> *Cc:* gluster-users at gluster.org
> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>
> mnt-disk11-vmware2.log seems like a brick log. Could you attach the fuse
> mount logs? It should be right under /var/log/glusterfs/ directory
> named after the mount point name, only hyphenated.
>
> -Krutika
>
> On Sat, Mar 18, 2017 at 7:27 PM, Mahdi Adnan <mahdi.adnan at outlook.com>
> wrote:
>
>> Hello Krutika,
>>
>>
>> Kindly, check the attached logs.
>>
>>
>>
>> --
>>
>> Respectfully
>> *Mahdi A. Mahdi*
>>
>> ------------------------------
>> *From:* Krutika Dhananjay <kdhananj at redhat.com>
>> *Sent:* Saturday, March 18, 2017 3:29:03 PM
>> *To:* Mahdi Adnan
>> *Cc:* gluster-users at gluster.org
>> *Subject:* Re: [Gluster-users] Gluster 3.8.10 rebalance VMs corruption
>>
>> Hi Mahdi,
>>
>> Could you attach mount, brick and rebalance logs?
>>
>> -Krutika
>>
>> On Sat, Mar 18, 2017 at 12:14 AM, Mahdi Adnan <mahdi.adnan at outlook.com>
>> wrote:
>>
>>> Hi,
>>>
>>> I have upgraded to Gluster 3.8.10 today and ran the add-brick procedure
>>> in a volume contains few VMs.
>>> After the completion of rebalance, i have rebooted the VMs, some of ran
>>> just fine, and others just crashed.
>>> Windows boot to recovery mode and Linux throw xfs errors and does not
>>> boot.
>>> I ran the test again and it happened just as the first one, but i have
>>> noticed only VMs doing disk IOs are affected by this bug.
>>> The VMs in power off mode started fine and even md5 of the disk file did
>>> not change after the rebalance.
>>>
>>> anyone else can confirm this ?
>>>
>>>
>>> Volume info:
>>>
>>> Volume Name: vmware2
>>> Type: Distributed-Replicate
>>> Volume ID: 02328d46-a285-4533-aa3a-fb9bfeb688bf
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 22 x 2 = 44
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: gluster01:/mnt/disk1/vmware2
>>> Brick2: gluster03:/mnt/disk1/vmware2
>>> Brick3: gluster02:/mnt/disk1/vmware2
>>> Brick4: gluster04:/mnt/disk1/vmware2
>>> Brick5: gluster01:/mnt/disk2/vmware2
>>> Brick6: gluster03:/mnt/disk2/vmware2
>>> Brick7: gluster02:/mnt/disk2/vmware2
>>> Brick8: gluster04:/mnt/disk2/vmware2
>>> Brick9: gluster01:/mnt/disk3/vmware2
>>> Brick10: gluster03:/mnt/disk3/vmware2
>>> Brick11: gluster02:/mnt/disk3/vmware2
>>> Brick12: gluster04:/mnt/disk3/vmware2
>>> Brick13: gluster01:/mnt/disk4/vmware2
>>> Brick14: gluster03:/mnt/disk4/vmware2
>>> Brick15: gluster02:/mnt/disk4/vmware2
>>> Brick16: gluster04:/mnt/disk4/vmware2
>>> Brick17: gluster01:/mnt/disk5/vmware2
>>> Brick18: gluster03:/mnt/disk5/vmware2
>>> Brick19: gluster02:/mnt/disk5/vmware2
>>> Brick20: gluster04:/mnt/disk5/vmware2
>>> Brick21: gluster01:/mnt/disk6/vmware2
>>> Brick22: gluster03:/mnt/disk6/vmware2
>>> Brick23: gluster02:/mnt/disk6/vmware2
>>> Brick24: gluster04:/mnt/disk6/vmware2
>>> Brick25: gluster01:/mnt/disk7/vmware2
>>> Brick26: gluster03:/mnt/disk7/vmware2
>>> Brick27: gluster02:/mnt/disk7/vmware2
>>> Brick28: gluster04:/mnt/disk7/vmware2
>>> Brick29: gluster01:/mnt/disk8/vmware2
>>> Brick30: gluster03:/mnt/disk8/vmware2
>>> Brick31: gluster02:/mnt/disk8/vmware2
>>> Brick32: gluster04:/mnt/disk8/vmware2
>>> Brick33: gluster01:/mnt/disk9/vmware2
>>> Brick34: gluster03:/mnt/disk9/vmware2
>>> Brick35: gluster02:/mnt/disk9/vmware2
>>> Brick36: gluster04:/mnt/disk9/vmware2
>>> Brick37: gluster01:/mnt/disk10/vmware2
>>> Brick38: gluster03:/mnt/disk10/vmware2
>>> Brick39: gluster02:/mnt/disk10/vmware2
>>> Brick40: gluster04:/mnt/disk10/vmware2
>>> Brick41: gluster01:/mnt/disk11/vmware2
>>> Brick42: gluster03:/mnt/disk11/vmware2
>>> Brick43: gluster02:/mnt/disk11/vmware2
>>> Brick44: gluster04:/mnt/disk11/vmware2
>>> Options Reconfigured:
>>> cluster.server-quorum-type: server
>>> nfs.disable: on
>>> performance.readdir-ahead: on
>>> transport.address-family: inet
>>> performance.quick-read: off
>>> performance.read-ahead: off
>>> performance.io-cache: off
>>> performance.stat-prefetch: off
>>> cluster.eager-lock: enable
>>> network.remote-dio: enable
>>> features.shard: on
>>> cluster.data-self-heal-algorithm: full
>>> features.cache-invalidation: on
>>> ganesha.enable: on
>>> features.shard-block-size: 256MB
>>> client.event-threads: 2
>>> server.event-threads: 2
>>> cluster.favorite-child-policy: size
>>> storage.build-pgfid: off
>>> network.ping-timeout: 5
>>> cluster.enable-shared-storage: enable
>>> nfs-ganesha: enable
>>> cluster.server-quorum-ratio: 51%
>>>
>>>
>>> Adding bricks:
>>> gluster volume add-brick vmware2 replica 2 gluster01:/mnt/disk11/vmware2
>>> gluster03:/mnt/disk11/vmware2 gluster02:/mnt/disk11/vmware2
>>> gluster04:/mnt/disk11/vmware2
>>>
>>>
>>> starting fix layout:
>>> gluster volume rebalance vmware2 fix-layout start
>>>
>>> Starting rebalance:
>>> gluster volume rebalance vmware2  start
>>>
>>>
>>>
>>> --
>>>
>>> Respectfully
>>> *Mahdi A. Mahdi*
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170319/75135d77/attachment.html>