[Gluster-users] 2.0_rc1 and replicate

[Łukasz Mierzwa]

Hi,

I've tested replicate a little today and I've got this problems:

1. client-side replicate with 2 nodes, both acting as server and client
2. I start copying big file to gluster mount point
3. during file copy I stop gluster server on second node
4. I got a pice of file on second node and whole file on first one
5. during or after the copy I start halted server on second node
6. I run ls -lh on second node in gluster mount point
7. logs are telling me to delete one of files and my second node keeps
incomplited copy

I've checked if I got xattrs enabled with attr command:

# attr -l <gluster_data_dir>/iso
Attribute "glusterfs.afr.entry-pending" has a 8 byte value for iso
# attr -g glusterfs.afr.entry-pending <gluster_data_dir>/iso
attr_get: No data available
Could not get "glusterfs.afr.entry-pending" for iso

With client side it seems that if im inside gluster mount point and stop
server on second node, after starting it again I need to reenter this dir
for self-heal to work, it does not seem to be needed with server-side.
With server side afr (both node afr'ing each other), when I stop second node
during big file copy and after while start it again if I run 'ls -lh' on
gluster mount point I'm getting some self-heal race condition, looking at
dstat output it seems that running 'ls -lh' on one node makes that node to
self-heal other with it current file content, I forgot about the log files
so I don't know what they say about it, I will retest it tomorrow.
Using replicate slows down writing a lot, with client-side to 1-2MB/s (!) on
1Gb link, with server-side to 10-12MB/s.

==============================================
client-side replicate, server.vol:

volume brick-data
    type storage/posix
    option directory /var/glusterfs/data/
end-volume

volume locks
    type features/locks
    subvolumes brick-data
end-volume

volume client
    type performance/io-threads
    option thread-count 4
    subvolumes locks
end-volume

volume server
    type protocol/server
    option transport-type tcp
    subvolumes client
    option auth.login.client.allow gluster
    option auth.login.gluster.password pass
end-volume

========================================================================
client-side replicate, client vol:
volume gluster_134
    type protocol/client
    option transport-type tcp
    option remote-host 172.16.110.134
    option remote-subvolume client
    option username gluster
    option password pass
end-volume

volume gluster_135
    type protocol/client
    option transport-type tcp
    option remote-host 172.16.110.135
    option remote-subvolume client
    option username gluster
    option password pass
end-volume

volume afr
    type cluster/replicate
    subvolumes gluster_134 gluster_135
end-volume

volume readahead
    type performance/read-ahead
    option page-size 1MB
    option page-count 2
    subvolumes afr
end-volume

volume iocache
    type performance/io-cache
    option cache-size 64MB
    option page-size 256KB
    option page-count 2
    subvolumes readahead
end-volume

volume writeback
    type performance/write-behind
    option aggregate-size 1MB
    option window-size 2MB
    option flush-behind off
    subvolumes iocache
end-volume

volume iothreads
    type performance/io-threads
    option thread-count 4
    subvolumes writeback
end-volume

=============================================================
server-side replicate, server.vol:

volume brick
    type storage/posix
    option directory /var/glusterfs/data/
end-volume

volume client_afr
    type features/locks
    subvolumes brick
end-volume

volume brick_134
    type protocol/client
    option transport-type tcp/client
    option remote-host 172.16.110.134
    option remote-subvolume client_afr
    option username gluster
    option password Tj72pAz44
end-volume

volume afr
    type cluster/replicate
    subvolumes client_afr brick_134
end-volume

volume client
    type performance/io-threads
    option thread-count 4
    subvolumes afr
end-volume

volume server
    type protocol/server
    option transport-type tcp
    subvolumes client_afr client
    option auth.login.client.allow gluster
    option auth.login.client_afr.allow gluster
    option auth.login.gluster.password pass
end-volume

================================================================
server-side replicate, client.vol:
volume gluster_134
    type protocol/client
    option transport-type tcp
    option remote-host 172.16.110.134
    option remote-subvolume client
    option username gluster
    option password Tj72pAz44
end-volume

volume gluster_135
    type protocol/client
    option transport-type tcp
    option remote-host 172.16.110.135
    option remote-subvolume client
    option username gluster
    option password Tj72pAz44
end-volume

volume ha
    type cluster/ha
    subvolumes gluster_135 gluster_134
end-volume

volume readahead
    type performance/read-ahead
    option page-size 1MB
    option page-count 2
    subvolumes ha
end-volume

volume iocache
    type performance/io-cache
    option cache-size 64MB
    option page-size 256KB
    option page-count 2
    subvolumes readahead
end-volume

volume writeback
    type performance/write-behind
    option aggregate-size 1MB
    option window-size 2MB
    option flush-behind off
    subvolumes iocache
end-volume

volume iothreads
    type performance/io-threads
    option thread-count 4
    subvolumes writeback
end-volume
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20090201/c8a8cdc4/attachment.html>