[Gluster-devel] Re: AFR/Replicate read-subvolume option not working right

[Gordan Bobic]

Alex Attarian wrote:
> I'll give that a try Gordan! Thanks for the info! 
> 
> My other question is does gluster perform a read for the extended 
> attributes on all servers regardless read-subvolume?
> This is what my tcpdump shows:
> ,3..h...:O....5....*.....L......... 
> ......+....c...c..............D........z/production/public_images/15932/images/JetLease_Aircraft_Wanted_For_Lease_Oct_09.gif.JetLease_Aircraft_Wanted_For_Lease_Oct_09.gif.............glusterfs.open-fd-count.0.........trusted.afr.gfs3.12.........trusted.afr.gfs2.12.........trusted.afr.gfs1.12.
> 09:17:23.498739 IP 192.168.27.26.6996 > 192.168.13.14.1022: P 
> 10821:11057(236) ack 13336 win 1525 <nop,nop,timestamp 1758080706 741598866>
> ......]...........
> h...,3..:O.........*.....L......... 
> ...................... at ....................................c...c....KFGX....J.......J..........p............glusterfs.open-fd-count.0.........trusted.afr.gfs2.....................trusted.afr.gfs1.............
> 09:17:23.511931 IP 192.168.13.14.1022 > 192.168.27.26.6996: P 
> 13336:13625(289) ack 11057 win 1525 <nop,nop,timestamp 741598867 1758080706>
> ..J..................
> ,3..h...:O.........*.....L......... 
> ......+....c...c..............D........z/production/public_images/15932/images/Jet_1Hawker_Multi_July_09.gif.Jet_1Hawker_Multi_July_09.gif.............glusterfs.open-fd-count.0.........trusted.afr.gfs3.12.........trusted.afr.gfs2.12.........trusted.afr.gfs1.12.
> 09:17:23.512218 IP 192.168.27.26.6996 > 192.168.13.14.1022: P 
> 11057:11293(236) ack 13625 win 1525 <nop,nop,timestamp 1758080707 741598867>
> ..................
> h...,3..:O.........*.....L......... 
> ......................A[.......g...........................c...c....KBp.....Jw......Jw.........p............glusterfs.open-fd-count.0.........trusted.afr.gfs2.....................trusted.afr.gfs1.............
> 09:17:23.525473 IP 192.168.13.14.1022 > 192.168.27.26.6996: P 
> 13625:13944(319) ack 11293 win 1525 <nop,nop,timestamp 741598869 1758080707>
> ..J........-..
> 
> 
> Does it have to read the trusted.afr.* attributes from every server 
> every time it reads a file?

In short - yes, AFAIK. Healing is done when files are accessed, thus the 
metadata checks have to be done across all the servers when a file is 
accessed to verify that they are all in sync.

The end result is that this ends up being the dominant performance 
factor for small I/Os and negligible for large I/O. So if your files are 
small and many (e.g. maildir), performance will be poor. But then again 
all cluster file systems suffer from that.

Gordan