[Gluster-devel] Question about unify over afr

[Łukasz Mierzwa]

1. I got 6 servers in glusterfs cluster 192.168.1.40-45, each one is running 
glusterfs-server with config file that is in debug.tar.
Data and namespace dirs were placed on different raid arrays.
2. On 192.168.1.40 I started glusterfs-client
3. Then I created ~200k dirs (no files), all servers were online when I was 
creating dirs, no network problems.
4. I started gluster client through strace with debugging enabled.
5. On 192.168.1.45 I attached strace to '[glusterfs]' process (server)
6. After that I run 'ls -lh' (first 3-4 minutes)
7. and 'find > /dev/null' on client mount point (192.168.1.40). (rest of 
graphs)
8. After ~15 minutes I terminated find.

On 192.168.1.40 I checked /proc/diskstats:
a) before any reads from gluster:
   9    0 md0 89202 0 713592 0 10590 0 84720 0 0 0 0
   9    1 md1 90853 0 726812 0 14580 0 116640 0 0 0 0
b) after all reads:
   9    0 md0 100430 0 803416 0 19256 0 154048 0 0 0 0
   9    1 md1 102083 0 816652 0 26604 0 212832 0 0 0 0

(154048 - 84720) * 512b = ~34MB written to namespace array (vs ~44MB read but 
some reads where cached from previous tests, I forgot to clear it) 
(212832 - 116640) * 512b = ~47MB written to data array (vs ~44MB read but some 
reads where cached from previous tests, I forgot to clear it)
all while doing simple ls and find, no other data (logs or whatever, strace and 
gluster logs were written to tmpfs) were written to those arrays, they were 
only used by glusterfs. There were only dirs so 34MB is a not so small amount 
of data.
All arrays were formatted with ext3 and mounted with 
"rw,noatime,nodiratime,user_xattr,acl,commit=60".
I've checked if running 'find' directly on my ext3 mount point will make any 
writes to fs, I stopped gluster-server and client
before:
   9    1 md1 18714 0 149700 0 23931 0 191448 0 0 0 0
after find (5 minutes later):
   9    1 md1 294078 0 2352612 0 23931 0 191448 0 0 0 0

~1GB of reads and no writes at all, gluster is the source of writes to those 
arrays not fs itself.

debug.tar with all logs, configs and rrd graphs:
https://doc.grono.org/debug.tar

-- 
Łukasz Mierzwa

Grono.net S.A.
 ul. Szturmowa 2a, 02-678 Warszawa
 Sąd Rejonowy dla m.st. Warszawy, XIII Wydział Gospodarczy;
 Nr KRS 0000292169 , NIP: 929-173-90-15  Regon: 141197097 Kapitał
 zakładowy: 550.000,00 zł
 http://grono.net/
 
 Treść tej wiadomości jest poufna i prawnie chroniona. Odbiorca może być
 jedynie jej adresat z wyłączeniem dostępu osób trzecich. Jeżeli nie
 jesteś adresatem niniejszej wiadomości, jej rozpowszechnianie,
 kopiowanie, rozprowadzanie lub inne działanie o podobnym charakterze
 jest prawnie zabronione i może by karalne. Jeżeli wiadomość ta trafiła
 do Ciebie omyłkowo, uprzejmie prosimy o odesłanie jej na adres nadawcy i
 usunięcie.