[Gluster-users] Disperse mkdir fails

[Ankireddypalle Reddy]

Xavi,
            Thanks for checking this.
            1) mkdir returns errnum 5. EIO.  
            2)  The specified directory is the parent directory under which all the data in the gluster volume will be stored. Current around 160TB of 262 TB is  consumed.
            3)  It is extremely difficult to list the exact sequence of FOPS that would have been issued to the directory. The storage is heavily used and lot of sub directories are present inside this directory.

           Are you looking for the extended attributes for this directory from all the bricks inside the volume.  There are about 60 bricks.

Thanks and Regards,
Ram                                    

-----Original Message-----
From: Xavier Hernandez [mailto:xhernandez at datalab.es] 
Sent: Thursday, March 09, 2017 11:15 AM
To: Ankireddypalle Reddy; Gluster Devel (gluster-devel at gluster.org); gluster-users at gluster.org
Subject: Re: [Gluster-users] Disperse mkdir fails

Hi Ram,

On 09/03/17 16:52, Ankireddypalle Reddy wrote:
> Attachment (1):
>
> 1
>
> 	
>
> info.txt
> <https://imap.commvault.com/webconsole/embedded.do?url=https://imap.co
> mmvault.com/webconsole/api/drive/publicshare/346714/file/3037641a3f9b4
> 133920b1b251ed32d5d/action/preview&downloadUrl=https://imap.commvault.
> com/webconsole/api/contentstore/publicshare/346714/file/3037641a3f9b41
> 33920b1b251ed32d5d/action/download>
> [Download]
> <https://imap.commvault.com/webconsole/api/contentstore/publicshare/34
> 6714/file/3037641a3f9b4133920b1b251ed32d5d/action/download>(3.35
> KB)
>
> Hi,
>
>         I have a disperse gluster volume  with 6 servers. 262TB of 
> usable capacity.  Gluster version is 3.7.19.
>
>         glusterfs1, glusterf2 and glusterfs3 nodes were initially used 
> for creating the volume. Nodes glusterf4, glusterfs5 and glusterfs6 
> were later added to the volume.
>
>
>
>         Directory creation failed on a directory called 
> /ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC.
>
>         # file: ws/glus/Folder_07.11.2016_23.02/CV_MAGNETIC
>
>         glusterfs.gfid.string="e8e51015-616f-4f04-b9d2-92f46eb5cfc7"
>
>
>
>         gluster mount log contains lot of following errors:
>
>         [2017-03-09 15:32:36.773937] W [MSGID: 122056] 
> [ec-combine.c:875:ec_combine_check] 0-StoragePool-disperse-7:
> Mismatching xdata in answers of 'LOOKUP' for
> e8e51015-616f-4f04-b9d2-92f46eb5cfc7
>
>
>
>         The directory seems to be out of sync between nodes 
> glusterfs1,
> glusterfs2 and glusterfs3. Each has different version.
>
>
>
>          trusted.ec.version=0x00000000000839f00000000000083a4d
>
>          trusted.ec.version=0x0000000000082ea40000000000083a4b
>
>          trusted.ec.version=0x0000000000083a760000000000083a7b
>
>
>
>          Self-heal does not seem to be healing this directory.
>

This is very similar to what happened the other time. Once more than 1 brick is damaged, self-heal cannot do anything to heal it on a 2+1 configuration.

What error does return the mkdir request ?

Does the directory you are trying to create already exist on some brick ?

Can you show all the remaining extended attributes of the directory ?

It would also be useful to have the directory contents on each brick (an 'ls -l'). In this case, include the name of the directory you are trying to create.

Can you explain a detailed sequence of operations done on that directory since the last time you successfully created a new subdirectory ? 
including any metadata change.

Xavi

>
>
> Thanks and Regards,
>
> Ram
>
> ***************************Legal Disclaimer***************************
> "This communication may contain confidential and privileged material 
> for the sole use of the intended recipient. Any unauthorized review, 
> use or distribution by others is strictly prohibited. If you have 
> received the message by mistake, please advise the sender by reply 
> email and delete the message. Thank you."
> **********************************************************************
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>

***************************Legal Disclaimer***************************
"This communication may contain confidential and privileged material for the
sole use of the intended recipient. Any unauthorized review, use or distribution
by others is strictly prohibited. If you have received the message by mistake,
please advise the sender by reply email and delete the message. Thank you."
**********************************************************************