[Gluster-users] [Gluster-devel] missing files
Pranith Kumar Karampuri
pkarampu at redhat.com
Thu Feb 5 10:18:29 UTC 2015
I believe David already fixed this. I hope this is the same issue he
told about permissions issue.
Pranith
On 02/05/2015 03:44 PM, Xavier Hernandez wrote:
> Is the failure repeatable ? with the same directories ?
>
> It's very weird that the directories appear on the volume when you do
> an 'ls' on the bricks. Could it be that you only made a single 'ls' on
> fuse mount which not showed the directory ? Is it possible that this
> 'ls' triggered a self-heal that repaired the problem, whatever it was,
> and when you did another 'ls' on the fuse mount after the 'ls' on the
> bricks, the directories were there ?
>
> The first 'ls' could have healed the files, causing that the following
> 'ls' on the bricks showed the files as if nothing were damaged. If
> that's the case, it's possible that there were some disconnections
> during the copy.
>
> Added Pranith because he knows better replication and self-heal details.
>
> Xavi
>
> On 02/04/2015 07:23 PM, David F. Robinson wrote:
>> Distributed/replicated
>>
>> Volume Name: homegfs
>> Type: Distributed-Replicate
>> Volume ID: 1e32672a-f1b7-4b58-ba94-58c085e59071
>> Status: Started
>> Number of Bricks: 4 x 2 = 8
>> Transport-type: tcp
>> Bricks:
>> Brick1: gfsib01a.corvidtec.com:/data/brick01a/homegfs
>> Brick2: gfsib01b.corvidtec.com:/data/brick01b/homegfs
>> Brick3: gfsib01a.corvidtec.com:/data/brick02a/homegfs
>> Brick4: gfsib01b.corvidtec.com:/data/brick02b/homegfs
>> Brick5: gfsib02a.corvidtec.com:/data/brick01a/homegfs
>> Brick6: gfsib02b.corvidtec.com:/data/brick01b/homegfs
>> Brick7: gfsib02a.corvidtec.com:/data/brick02a/homegfs
>> Brick8: gfsib02b.corvidtec.com:/data/brick02b/homegfs
>> Options Reconfigured:
>> performance.io-thread-count: 32
>> performance.cache-size: 128MB
>> performance.write-behind-window-size: 128MB
>> server.allow-insecure: on
>> network.ping-timeout: 10
>> storage.owner-gid: 100
>> geo-replication.indexing: off
>> geo-replication.ignore-pid-check: on
>> changelog.changelog: on
>> changelog.fsync-interval: 3
>> changelog.rollover-time: 15
>> server.manage-gids: on
>>
>>
>> ------ Original Message ------
>> From: "Xavier Hernandez" <xhernandez at datalab.es>
>> To: "David F. Robinson" <david.robinson at corvidtec.com>; "Benjamin
>> Turner" <bennyturns at gmail.com>
>> Cc: "gluster-users at gluster.org" <gluster-users at gluster.org>; "Gluster
>> Devel" <gluster-devel at gluster.org>
>> Sent: 2/4/2015 6:03:45 AM
>> Subject: Re: [Gluster-devel] missing files
>>
>>> On 02/04/2015 01:30 AM, David F. Robinson wrote:
>>>> Sorry. Thought about this a little more. I should have been clearer.
>>>> The files were on both bricks of the replica, not just one side. So,
>>>> both bricks had to have been up... The files/directories just don't
>>>> show
>>>> up on the mount.
>>>> I was reading and saw a related bug
>>>> (https://bugzilla.redhat.com/show_bug.cgi?id=1159484). I saw it
>>>> suggested to run:
>>>> find <mount> -d -exec getfattr -h -n trusted.ec.heal {} \;
>>>
>>> This command is specific for a dispersed volume. It won't do anything
>>> (aside from the error you are seeing) on a replicated volume.
>>>
>>> I think you are using a replicated volume, right ?
>>>
>>> In this case I'm not sure what can be happening. Is your volume a pure
>>> replicated one or a distributed-replicated ? on a pure replicated it
>>> doesn't make sense that some entries do not show in an 'ls' when the
>>> file is in both replicas (at least without any error message in the
>>> logs). On a distributed-replicated it could be caused by some problem
>>> while combining contents of each replica set.
>>>
>>> What's the configuration of your volume ?
>>>
>>> Xavi
>>>
>>>>
>>>> I get a bunch of errors for operation not supported:
>>>> [root at gfs02a homegfs]# find wks_backup -d -exec getfattr -h -n
>>>> trusted.ec.heal {} \;
>>>> find: warning: the -d option is deprecated; please use -depth instead,
>>>> because the latter is a POSIX-compliant feature.
>>>> wks_backup/homer_backup/backup: trusted.ec.heal: Operation not
>>>> supported
>>>> wks_backup/homer_backup/logs/2014_05_20.log: trusted.ec.heal:
>>>> Operation
>>>> not supported
>>>> wks_backup/homer_backup/logs/2014_05_21.log: trusted.ec.heal:
>>>> Operation
>>>> not supported
>>>> wks_backup/homer_backup/logs/2014_05_18.log: trusted.ec.heal:
>>>> Operation
>>>> not supported
>>>> wks_backup/homer_backup/logs/2014_05_19.log: trusted.ec.heal:
>>>> Operation
>>>> not supported
>>>> wks_backup/homer_backup/logs/2014_05_22.log: trusted.ec.heal:
>>>> Operation
>>>> not supported
>>>> wks_backup/homer_backup/logs: trusted.ec.heal: Operation not supported
>>>> wks_backup/homer_backup: trusted.ec.heal: Operation not supported
>>>> ------ Original Message ------
>>>> From: "Benjamin Turner" <bennyturns at gmail.com
>>>> <mailto:bennyturns at gmail.com>>
>>>> To: "David F. Robinson" <david.robinson at corvidtec.com
>>>> <mailto:david.robinson at corvidtec.com>>
>>>> Cc: "Gluster Devel" <gluster-devel at gluster.org
>>>> <mailto:gluster-devel at gluster.org>>; "gluster-users at gluster.org"
>>>> <gluster-users at gluster.org <mailto:gluster-users at gluster.org>>
>>>> Sent: 2/3/2015 7:12:34 PM
>>>> Subject: Re: [Gluster-devel] missing files
>>>>> It sounds to me like the files were only copied to one replica,
>>>>> werent
>>>>> there for the initial for the initial ls which triggered a self heal,
>>>>> and were there for the last ls because they were healed. Is there any
>>>>> chance that one of the replicas was down during the rsync? It could
>>>>> be that you lost a brick during copy or something like that. To
>>>>> confirm I would look for disconnects in the brick logs as well as
>>>>> checking glusterfshd.log to verify the missing files were actually
>>>>> healed.
>>>>>
>>>>> -b
>>>>>
>>>>> On Tue, Feb 3, 2015 at 5:37 PM, David F. Robinson
>>>>> <david.robinson at corvidtec.com <mailto:david.robinson at corvidtec.com>>
>>>>> wrote:
>>>>>
>>>>> I rsync'd 20-TB over to my gluster system and noticed that I had
>>>>> some directories missing even though the rsync completed
>>>>> normally.
>>>>> The rsync logs showed that the missing files were transferred.
>>>>> I went to the bricks and did an 'ls -al
>>>>> /data/brick*/homegfs/dir/*' the files were on the bricks. After I
>>>>> did this 'ls', the files then showed up on the FUSE mounts.
>>>>> 1) Why are the files hidden on the fuse mount?
>>>>> 2) Why does the ls make them show up on the FUSE mount?
>>>>> 3) How can I prevent this from happening again?
>>>>> Note, I also mounted the gluster volume using NFS and saw the
>>>>> same
>>>>> behavior. The files/directories were not shown until I did the
>>>>> "ls" on the bricks.
>>>>> David
>>>>> ===============================
>>>>> David F. Robinson, Ph.D.
>>>>> President - Corvid Technologies
>>>>> 704.799.6944 x101 <tel:704.799.6944%20x101> [office]
>>>>> 704.252.1310 <tel:704.252.1310> [cell]
>>>>> 704.799.7974 <tel:704.799.7974> [fax]
>>>>> David.Robinson at corvidtec.com
>>>>> <mailto:David.Robinson at corvidtec.com>
>>>>> http://www.corvidtechnologies.com
>>>>> <http://www.corvidtechnologies.com/>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-devel mailing list
>>>>> Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>
More information about the Gluster-users
mailing list