[Gluster-devel] Ganesha caches bad directory entries when other glusterfs client(ganesha) creating files

[Kinglong Mee]

I use ganesha on glusterfs as,
1. Glusterfs contains two bricks (brick1,brick2),
2. Two ganeshas bases on glusterfs, and two nfs clients through the
two ganeshas,
3. One nfs client(ganeshaA, clientA) copy small files in;
   the other nfs client(ganeshaB, clientB) do "find /mnt/nfs/ | wc -l" by cycle.

After some times later, ctrl+c the copying and the finding,
do "find /mnt/nfs/ | wc -l" on clientA and clientB.

When the system time deviation between brick1 and brick2 is larger
than 0.1 seconds
(eg, the system time of brick1 is 0.1s lesser than brick2),
the files count showed by clientB is less than clientA.

Ganesha caches the directory entries at memory depends on directory's mtime,
those cached entries are valid until the directory's mtime change.
But if ganesha caches bad entries, the client will never shows the right entries
until the directory's mtime change. It's the problem I have meet.

All following analysis are based on the system time deviation between
brick1 is 0.1s
less than brick2;
1. DHT readdirs from brick1 to brick2 in proper order;
2  A directory's mtime at brick2 is newer than brick1;
3. GaneshaB's dht readdirs from brick1, gets old entries from the directory;
4. GaneshaA's dht creates one file under the directory, and update the
mtime of directory on brick1;
   ** for the 0.1s lesser than brick2, the new mtime at brick1 maybe
less than brick2.
5. GaneshaB's dht readdirs from brick2, gets latest entries from the
same name directory;
6. GaneshaB gets the brick2's mtime as the directory's mtime;
7. At bad case, ganeshaB caches those entries (without the file created at #4).

Is it a known issue?
Maybe we should update the directory's mtime on all nodes at #4 (on
brick1 and brick2)?

thanks,
Kinglong Mee