[Gluster-users] missing files

[Jeremy Enos]

I have another clue to report:
So I have my export directory as:
/export
Mounted as:
/scratch

If I do "ls -lR /scratch", it's supposed to synchronize all files and
metadata, right?  Well, it doesn't seem to be doing that.

I have approx 100 files in one problematic folder.  Only 50 show up to
ls.  That is, until I list it specifically.  They also don't show up in
the export directory until ls'd by name in /scratch.

ls /scratch/file* # results in files1-49 being listed
ls /export/file*  # same result as above
ls /export/file50.dat  #  no such file or directory
ls /scratch/file50.dat  # lists file as if nothing was ever wrong
ls /export/file50.dat  # shows up now after specific ls call in /scratch
ls /scratch/file*  # results in files 1-50 being listed now  (magic?)
ls /export/file*  # also results in files 1-50 being listed now

I'm considering doing a:
for n in `seq 51-100` ; do ls /scratch/file$n.dat ; done
just to recover the files.  However, I'm delaying that so I can keep
some in the problematic state should someone give me some additional
debugging steps here.  Don't get me wrong- I appreciate any help I can
get w/ a free product like this.  But I'm actually surprised that a
report like this just seems to be hitting a dead end on this list in
terms of responses.  Isn't this alarming behavior?  Somehow the
filesystem got into a state where files still were recorded, but weren't
represented until specifically listed.  That should tell us something,
but I'm no expert here.
thx-

    Jeremy

Jeremy Enos wrote:
> Can anyone tell me if there's hope of recovering data here?  Steps to
> take?  Anything?  Is something wrong with my configuration?  (raid1 over
> raid0)  If I don't have a clue what went wrong or why, or how to
> recover, then even formatting and starting fresh doesn't lend much hope
> in future reliability.
> thx-
>
>     Jeremy
>
> Jeremy Enos wrote:
>   
>> plain text send...
>>
>> Jeremy Enos wrote:
>>     
>>> What kind of tweaking and tampering was necessary to recover the lost
>>> data?
>>>
>>>     Jeremy
>>>
>>> My configuration:
>>> Oh yes- of course- don't know why I left this out.  Version and
>>> config files follow.
>>>
>>> [jenos at ac glusterfs]$ rpm -qa |grep gluster
>>> glusterfs-common-2.0.7-1.fc10.x86_64
>>> glusterfs-client-2.0.7-1.fc10.x86_64
>>>
>>>
>>> [jenos at ac glusterfs]$ cat glusterfs.vol
>>> #-----------IB remotes------------------
>>> volume remote1
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac11
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote2
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac12
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote3
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac13
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote4
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac14
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote5
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac15
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote6
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac16
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote7
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac17
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote8
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac18
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote9
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac19
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> volume remote10
>>>   type protocol/client
>>>   option transport-type ib-verbs/client
>>>   option remote-host ac20
>>>   option remote-subvolume ibstripe
>>> end-volume
>>>
>>> #----------Stripe and Replicate------------------
>>>
>>> volume stripe1
>>>   type cluster/stripe
>>>   option block-size 1MB
>>>   subvolumes remote1 remote2 remote3 remote4 remote5
>>> end-volume
>>>
>>> volume stripe2
>>>   type cluster/stripe
>>>   option block-size 1MB
>>>   subvolumes remote6 remote7 remote8 remote9 remote10
>>> end-volume
>>>
>>> volume replicate
>>>   type cluster/replicate
>>>   option metadata-self-heal on
>>>   subvolumes stripe1 stripe2
>>> end-volume
>>>
>>> #------------Performance Options-------------------
>>>
>>> volume readahead
>>>   type performance/read-ahead
>>>   option page-count 4           # 2 is default option
>>>   option force-atime-update off # default is off
>>>   subvolumes replicate
>>> end-volume
>>>
>>> volume writebehind
>>>   type performance/write-behind
>>>   option cache-size 1MB
>>>   subvolumes readahead
>>> end-volume
>>>
>>> volume cache
>>>   type performance/io-cache
>>>   option cache-size 1GB
>>>   subvolumes writebehind
>>> end-volume
>>>
>>> [jenos at ac glusterfs]$ cat glusterfsd.vol
>>> volume posix
>>>   type storage/posix
>>>   option directory /export
>>> end-volume
>>>
>>> volume locks
>>>   type features/locks
>>>   subvolumes posix
>>> end-volume
>>>
>>> volume ibstripe
>>>   type performance/io-threads
>>>   option thread-count 4
>>>   subvolumes locks
>>> end-volume
>>>
>>> volume server-ib
>>>   type protocol/server
>>>   option transport-type ib-verbs/server
>>>   option auth.addr.ibstripe.allow *
>>>   subvolumes ibstripe
>>> end-volume
>>>
>>> volume server-tcp
>>>   type protocol/server
>>>   option transport-type tcp/server
>>>   option auth.addr.ibstripe.allow *
>>>   subvolumes ibstripe
>>> end-volume
>>>
>>> [jenos at ac glusterfs]$
>>>
>>>
>>>
>>> Krzysztof Strasburger wrote:
>>>       
>>>> On Wed, Nov 04, 2009 at 01:31:30AM -0600, Jeremy Enos wrote:
>>>>  
>>>>         
>>>>> Hi-
>>>>> I've got a problem where certain batches of files written out to
>>>>> gluster have disappeared.  Also, newly created files sometimes
>>>>> don't show up to ls unless they are explicitly specified to ls and
>>>>> other tools.
>>>>>
>>>>> In my export folder, everything appears fine.
>>>>> I have found that when I touch the missing file in gluster, it
>>>>> comes back, shows a file size, but appears empty.  I've tried
>>>>> umounting, restarting all glusterfsds, remounting, and it stayed
>>>>> the same.  Also, this problem did not show up immediately after
>>>>> setting up the filesystem, at least during basic tests.  Any ideas?
>>>>>     
>>>>>           
>>>> What is your configuration? I experienced similar problems with unify
>>>> after a disk crash. The namespace (replicated) was not rebuilt
>>>> correctly
>>>> after replacing the failing unit and I had to add some files manually
>>>> (OK, using a script, but an intervention was needed). No data loss,
>>>> only a bit of tweaking and tampering ;).
>>>> Krzysztof
>>>>