[Gluster-devel] Question about afr/self-heal

[Brian Hirt]

Hello,

I'm running some tests with with GlusterFS and so far I like what I'm  
seeing.   I've got a test 4 node system set up with AFR-Unify.   node1  
and node3 are replicated and node2 and node4 are replicated.   These  
are then unified together into a single brick.   afr and unify are  
done on the client side.  All of the servers are running ubuntu +  
glusterfs 1.3.12 with an underlaying ext3 filesystem.

Durning one of my tests i took down a server during an update/addition  
of a few thousand files.  After the update was complete, i brought up  
the downed node.  I was able to see all the new files after i did a  
directory listing on the client, but they all had a size of 0 and the  
updated files still had the old contents.   When I opened these files  
on the client, the correct contents were returned and the once down  
node was then corrected for that file.

 From searching through the email archives, this seems like the  
intended way it supposed to work.   However, in the state that the  
filesystem is in now, my redundancy is lost for those changes until i  
open every file and directory on the client.   In my configuration I  
intend to have many million files.  Am I supposed to open every single  
one of them after a node goes down to get the replication back in  
sync?  There will often be times where servers are brought down for  
routine maintenance for 10-15 minutes at a time and during that time  
only a few hundred files might change.     What is the proper  
procedure for resynchronizing?   How are other people handling this?    
I've seen a few comments about fsck in the mail archive referencing a  
path that doesn't exist in my GlusterFS distribution (possibly it's  
the 1.4 branch)

Also the log file is very verbose about the downed server.   There are  
lots of messages like:

2008-12-09 11:18:08 E [tcp-client.c:190:tcp_connect] brick2: non- 
blocking connect() returned: 111 (Connection refused)
2008-12-09 11:18:08 W [client-protocol.c:332:client_protocol_xfer]  
brick2: not connected at the moment to submit frame type(1) op(34)
2008-12-09 11:18:08 E [client-protocol.c:4430:client_lookup_cbk]  
brick2: no proper reply from server, returning ENOTCONN
2008-12-09 11:18:08 E [tcp-client.c:190:tcp_connect] brick2: non- 
blocking connect() returned: 111 (Connection refused)
2008-12-09 11:18:08 W [client-protocol.c:332:client_protocol_xfer]  
brick2: not connected at the moment to submit frame type(1) op(9)
2008-12-09 11:18:08 E [client-protocol.c:2787:client_chmod_cbk]  
brick2: no proper reply from server, returning ENOTCONN

In some of my tests I'm seeing several hundred a second logged.   Is  
there some way to make this a bit less verbose?

I'm sorry if these are FAQ, but I've so far been unable to find  
anything on the wiki or mailing lists.

Thanks in advance for you help and this great project.

Brian Hirt