[Gluster-users] Replication deleting file contents

[Adrian Revill]

Hi Vikas

This is a puzzle, I cant seem to be able to reproduce the problem, since 
I had the problem I had deleted the backing store directories. All I can 
think is when I originally started up the servers, I had not installed 
the fuse rpm's, maybe that created a funny state.
I will continue testing for the failure scenarios, as I want to start 
using glusterfs in a live environment in the new year, It looks like an 
excellent solution to distribute data files for our Beowulf clusters.

I now see the reported file sizes go to zero when the empty server 
joins, but the original data is still available in the first servers 
backing store.

Attached are the logs
I started the server and client on factory2, mounting the directory with 
mount -t glusterfs localhost /mnt/export/
copied in 3 files
ran ls -al /mnt/export and got correct file sizes.
Started factory2 and ran ls -al /mnt/ again and the file sizes are 0.

I do have one more question. When should a joining server get updated? 
 From the forum and docs it seems it should be as soon as the directory 
is read.
But I cant get the empty server to update its contents till I read the 
file contents eg md5sum the file. Is there any way to force a server to 
update?

regards
Adrian

Vikas Gorur wrote:
> Adrian Revill wrote:
>> Hi
>> I have read through the docs and google and I think i am trying to do 
>> this right, but just wanted to be sure i have it correctly configured.
>>
>> I have 2 servers factory1 and factory2, both clean installs of 
>> RHEL5.4 on basic hardware.
>> I followed the instructions from 
>> http://gluster.com/community/documentation/index.php/GlusterFS_User_Guide, 
>> and have installed 3.0.0-1
>>
>> On factory1 I ran the command
>> /usr/bin/glusterfs-volgen --name dataexp --raid 1 
>> factory1:/data/export factory2:/data/export
>>
>> I copied factory1-dataexp-export.vol to /etc/glusterfsd.vol on factory1
>> factory2-dataexp-export.vol to /etc/glusterfsd.vol on factory2
>>
>> and coppied dataexp-tcp.vol to /etc/glusterfs.vol on both factory1 
>> and factory2
>>
>> Then on both servers I started the server with the init script and ran
>> mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/export
>>
>> Every thing works fine, i can copy files into either servers 
>> /mnt/export directory and they appear on the other server.
>>
>> I then tried a failure scenario. With both servers and clients up i 
>> deleted all the files from /mnt/export, so far so good.
>>
>> I then shut down the server on factory1, leaving the client up and 
>> copied three large files (715MB each) into /mnt/export on factory2
>> Both factory1 and factroy2 both show the files with the correct size 
>> and md5sum.
>>
>> I then started the server on factory1.
>> This is where it goes wrong, as soon as factory1 starts, both clients 
>> show the file sizes of the files as 0, checking the backing store 
>> /data/export i see the files are also 0 size there too. The md5Sum is 
>> now wrong.
>>
>> It seems that the replication has created the files on factory2, but 
>> before copying the contents, it has decided that the empty files on 
>> factory1 are newer than factory2 and decided to copy the zero size 
>> files back to factory1.
>>
>> Any one got any ideas?
>>
> Adrian, If this is reproducible easily, can you run the clients with 
> log level TRACE and send us the logs (or perhaps log a bug and attach 
> it there if they are big)?
>
> Vikas
>
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email 
> ______________________________________________________________________


______________________________________________________________________
This email has been scanned by the MessageLabs Email Security System.
For more information please visit http://www.messagelabs.com/email 
______________________________________________________________________
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: mnt-export-.log
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20091222/84639d46/attachment.ksh>