[Gluster-devel] Df, ls, reads slow during I/O

[Jordan Mendler]

I have optimized my glusterfs system pretty well for writes, but I am having
an issue where doing an ls is very slow, and during I/O operations df takes
forever and often hangs on glusterfs. Below are some examples of the slow ls
(at the time of the tests, there are no filesystem operations occuring other
than the ls).

I am finding that during heavy reads problems are much worse, though they
also can be bad during heavy writes.

Does anyone have any ideas of how I can track this problem down? It is the
only thing preventing our system from being production-ready. The current
hardware setup is 2 storage servers using ALU on a 16xSATA RAID6 (8cores
each w/ 8gb RAM), and an AFR'ed namespace on the same hardware RAIDs. My
configuration is available at http://genome.ucla.edu/~jordan/gluster/

Thanks so much,
Jordan
-----------------

Examples of slow ls during no FS operations:

[root at solexa solexa]# time ls solexa_datasets/
070403_SLXA-EAS20_0001_FC5063  071121_HWI-EAS172_200C0             docs
070405_SLXA-EAS20_0005_FC5065  071203_HWI-EAS172_200BA             genomes
070418_SLXA-EAS20_5256         071207_HWI-EAS172_2015A
Instruments
070427_SLXA-EAS20_5152         071211_HWI-EAS172_200gh             L001.tar
070515_SLXA-EAS20_5612         071217_HWI-EAS172_14759             L007.tar
070523_SLXA-EAS20_5153         071221_HWI-EAS172_200M1
ls_d_images.output
070529_SLXA-EAS20_5594         080108_HWI-EAS172_200G8
ls_d_images.output.20071029
070604_SLXA-EAS20_5447         080116_HWI-EAS172_14758
ls_d_images.output.20071029.truncated
070608_SLXA-EAS20_FC5459       080121_HWI-EAS172_2040F
ls_d_images.output.2007103016.truncated
070612_SLXA-EAS20_5646         080125_HWI-EAS172_13471R
ls_d_images.output.2007103017.truncated
070621_SLXA-EAS20_5590         080128_HWI-EAS172_201UD
ls_d_images.output.20071030.truncated
070625_SLXA-EAS20_5701         080204_HWI-EAS172_201ET
ls_d_images.output.200710311329.truncated
070629_SLXA-EAS20_5861         080206_SLXA-EAS20_203NA
ls_d_images.output.200710311349.truncated
070703_SLXA-EAS20_5731         080215_HWI-EAS172_2009B
ls_d_images.output.200710311350.nottruncatedbutmissingdoneitems
070709_SLXA-EAS20_5611         080226_HWI-EAS172_205A0
ls_d_images.output.200710311749.truncated
070713_SLXA-EAS20_5863         080304_HWI-EAS172_204AY
ls_d_images.output.200711010821.truncated
070717_SLXA-EAS20_5699         080307_HWI-EAS172_20ANM
ls_d_images.output.200711010845.truncated
070723_SLXA-EAS20_5540         080311_HWI-EAS172_204KJ
ls_d_images.output.200711051802.truncated
070727_SLXA-EAS20_4606         080314_HWI-EAS172_204MG
ls_d_images.output.clustertemp1.200711011622
070731_SLXA-EAS20_4611         080320_HWI-EAS172_204KB
redo.200711061339
070810_SLXA-EAS20_5866         080326_HWI-EAS172_204KP             Reports
070814_SLXA-EAS20_5697         080401_HWI-EAS172_20CLK
rsync_to_storage.pl
070817_SLXA-EAS20_5603         080407_HWI-EAS172_204MN
Runs_Comments
070821_SLXA-EAS20_11290        080411_HWI-EAS172_204K1
Shawn_basecalls_U87
070827_SLXA-EAS20_11279        080418_HWI-EAS172_204T7
solexatmp0.tarfile2dirname
070831_SLXA-EAS20_5406         080429_HWI-EAS172_204KD
solexatmp0.tarfile2dirname.new
070925_SLXA-EAS20_11296        080505_HWI-EAS172_20DYC
solexatmp1.tarfile2dirname
071002_SLXA-EAS20_4977         080513_HWI-EAS172_20DYG             testdir
071022_HWI-EAS172_11989        080515_HWI-EAS172_20DYG_RE
testdir.tar
071026_HWI-EAS172_11989reseq   080516_HWI-EAS335_305T4             test.pl
071030_HWI-EAS172_14517        1
U87.fasta.bz2
071105_HWI-EAS172_14515        4276.fasta
verify_JMM.log
071109_HWI-EAS172_14034        -al
verify_JMM.sh
071115_HWI-EAS172_14055        clustertemp1.tarfiles.200711051731

real    0m2.298s
user    0m0.001s
sys    0m0.001s

[root at solexa solexa]# time ls solexa_datasets/070817_SLXA-EAS20_5603/Data
C2-36_Firecrest1.8.28_04-10-2007_solexa  default_offsets.txt

real    0m6.246s
user    0m0.001s
sys    0m0.002s