[Gluster-devel] Performance problem with XFS

[Xavier Hernandez]

Hi,

I've reproduced a problem I've seen with directory listing of 
directories not accessed for a long time (some hours). Gluster version 
is 3.3.1.

I've made the tests with different hardware and the behavior is quite 
similar.

The problem can be clearly seen doing this:

1. Format bricks with XFS, inode size 512, and mount them
2. Create a gluster volume (I've tried several combinations, see later)
3. Start and mount it
4. Create a directory <vol>/dirs and fill it with 300 subdirectories
5. Unmount the volume, stop it and flush kernel caches of all servers 
(sync ; echo 3 > /proc/sys/vm/drop_caches)
6. Start the volume, mount it, and execute "time ls -l <vol>/dirs | wc -l"
7. Create 80.000 directories at <vol>/ (notice that these directories 
are not created inside <vol>/dirs)
8. Unmount the volume, stop it and flush kernel caches of all servers 
(sync ; echo 3 > /proc/sys/vm/drop_caches)
9. Start the volume, mount it, and execute "time ls -l <vol>/dirs | wc -l"
10. Delete directory <vol>/dirs and recreate it with 300 subdirectories also
11. Unmount the volume, stop it and flush kernel caches of all servers 
(sync ; echo 3 > /proc/sys/vm/drop_caches)
12. Start the volume, mount it, and execute "time ls -l <vol>/dirs | wc -l"

With this test, I get the following times:

first ls: 1 second
second ls: 3.5 seconds
third ls: 10 seconds

I don't understand the second ls because the <vol>/dirs directory still 
have the same 300 subdirectories. But the third one is worst.

I've tried with different kinds of volumes (distributed-replicated, 
distributed, and even a single brick), and the behavior is the same 
(though the times are smaller when less bricks are involved).

After reaching this situation, I've tried to get the previous ls times 
by deleting directories, however the times do not seem to improve. Only 
after doing some "dirty" tests and removing empty gfid directories from 
<vol>/.glusterfs on all bricks I get better times, though not as good as 
the first ls (3 - 4 seconds better than the third ls).

This is always reproducible if the volume is stopped and the caches are 
emptied before each ls. With more files and/or directories, it can take 
up to 20 or more seconds to list a directory with 100-200 subdirectories.

Without stopping anything, a second ls responds in about 0.2 seconds.

I've also tested this with ext4 and BTRFS (I know it is not supported, 
but tested anyway). These are the results:

ext4 first ls: 0.5 seconds
ext4 second ls: 0.8 seconds
ext4 third ls: 7 seconds

btrfs first ls: 0.5 seconds
btrfs second ls: 0.5 seconds
btrfs third ls: 0.5 seconds

It seems clear that it depends on the file system, but if I access 
directly the bricks, all ls take at most 0.1 seconds to complete.

Repairing and defragmenting the bricks does not help.

strace'ing the glusterfs process of the bricks, I see that for each 
directory a lot of entries from <vol>/.glusterfs are lstat'ed and a lot 
of lgetxattr are called. For 300 directories I've counted more than 4500 
lstat's and more than 5300 lgetxattr, many of them repeated. I've also 
noticed that some lstat's take from 10 to 60 ms to complete (with XFS).

Is there any way to minimize these effects ? I'm doing something wrong ?

Thanks in advance for your help,

Xavi