[Gluster-devel] Df, ls, reads slow during I/O

Mon May 26 21:11:24 UTC 2008

And now, after a while of running, the cache doesn't seem to get used  
at all anymore. My jobs are running fine, but glusterfs memory usage  
is super low and all jobs are pulling data directly from the network.

How does io-cache determine what files to cache and for how long?

Dan Parsons

On May 26, 2008, at 1:17 PM, Jordan Mendler wrote:

> I have optimized my glusterfs system pretty well for writes, but I  
> am having
> an issue where doing an ls is very slow, and during I/O operations  
> df takes
> forever and often hangs on glusterfs. Below are some examples of the  
> slow ls
> (at the time of the tests, there are no filesystem operations  
> occuring other
> than the ls).
>
> I am finding that during heavy reads problems are much worse, though  
> they
> also can be bad during heavy writes.
>
> Does anyone have any ideas of how I can track this problem down? It  
> is the
> only thing preventing our system from being production-ready. The  
> current
> hardware setup is 2 storage servers using ALU on a 16xSATA RAID6  
> (8cores
> each w/ 8gb RAM), and an AFR'ed namespace on the same hardware  
> RAIDs. My
> configuration is available at http://genome.ucla.edu/~jordan/gluster/
>
> Thanks so much,
> Jordan
> -----------------
>
> Examples of slow ls during no FS operations:
>
> [root at solexa solexa]# time ls solexa_datasets/
> 070403_SLXA-EAS20_0001_FC5063  071121_HWI-EAS172_200C0              
> docs
> 070405_SLXA-EAS20_0005_FC5065  071203_HWI-EAS172_200BA              
> genomes
> 070418_SLXA-EAS20_5256         071207_HWI-EAS172_2015A
> Instruments
> 070427_SLXA-EAS20_5152         071211_HWI-EAS172_200gh              
> L001.tar
> 070515_SLXA-EAS20_5612         071217_HWI-EAS172_14759              
> L007.tar
> 070523_SLXA-EAS20_5153         071221_HWI-EAS172_200M1
> ls_d_images.output
> 070529_SLXA-EAS20_5594         080108_HWI-EAS172_200G8
> ls_d_images.output.20071029
> 070604_SLXA-EAS20_5447         080116_HWI-EAS172_14758
> ls_d_images.output.20071029.truncated
> 070608_SLXA-EAS20_FC5459       080121_HWI-EAS172_2040F
> ls_d_images.output.2007103016.truncated
> 070612_SLXA-EAS20_5646         080125_HWI-EAS172_13471R
> ls_d_images.output.2007103017.truncated
> 070621_SLXA-EAS20_5590         080128_HWI-EAS172_201UD
> ls_d_images.output.20071030.truncated
> 070625_SLXA-EAS20_5701         080204_HWI-EAS172_201ET
> ls_d_images.output.200710311329.truncated
> 070629_SLXA-EAS20_5861         080206_SLXA-EAS20_203NA
> ls_d_images.output.200710311349.truncated
> 070703_SLXA-EAS20_5731         080215_HWI-EAS172_2009B
> ls_d_images.output.200710311350.nottruncatedbutmissingdoneitems
> 070709_SLXA-EAS20_5611         080226_HWI-EAS172_205A0
> ls_d_images.output.200710311749.truncated
> 070713_SLXA-EAS20_5863         080304_HWI-EAS172_204AY
> ls_d_images.output.200711010821.truncated
> 070717_SLXA-EAS20_5699         080307_HWI-EAS172_20ANM
> ls_d_images.output.200711010845.truncated
> 070723_SLXA-EAS20_5540         080311_HWI-EAS172_204KJ
> ls_d_images.output.200711051802.truncated
> 070727_SLXA-EAS20_4606         080314_HWI-EAS172_204MG
> ls_d_images.output.clustertemp1.200711011622
> 070731_SLXA-EAS20_4611         080320_HWI-EAS172_204KB
> redo.200711061339
> 070810_SLXA-EAS20_5866         080326_HWI-EAS172_204KP              
> Reports
> 070814_SLXA-EAS20_5697         080401_HWI-EAS172_20CLK
> rsync_to_storage.pl
> 070817_SLXA-EAS20_5603         080407_HWI-EAS172_204MN
> Runs_Comments
> 070821_SLXA-EAS20_11290        080411_HWI-EAS172_204K1
> Shawn_basecalls_U87
> 070827_SLXA-EAS20_11279        080418_HWI-EAS172_204T7
> solexatmp0.tarfile2dirname
> 070831_SLXA-EAS20_5406         080429_HWI-EAS172_204KD
> solexatmp0.tarfile2dirname.new
> 070925_SLXA-EAS20_11296        080505_HWI-EAS172_20DYC
> solexatmp1.tarfile2dirname
> 071002_SLXA-EAS20_4977         080513_HWI-EAS172_20DYG              
> testdir
> 071022_HWI-EAS172_11989        080515_HWI-EAS172_20DYG_RE
> testdir.tar
> 071026_HWI-EAS172_11989reseq   080516_HWI-EAS335_305T4              
> test.pl
> 071030_HWI-EAS172_14517        1
> U87.fasta.bz2
> 071105_HWI-EAS172_14515        4276.fasta
> verify_JMM.log
> 071109_HWI-EAS172_14034        -al
> verify_JMM.sh
> 071115_HWI-EAS172_14055        clustertemp1.tarfiles.200711051731
>
> real    0m2.298s
> user    0m0.001s
> sys    0m0.001s
>
> [root at solexa solexa]# time ls solexa_datasets/070817_SLXA-EAS20_5603/ 
> Data
> C2-36_Firecrest1.8.28_04-10-2007_solexa  default_offsets.txt
>
> real    0m6.246s
> user    0m0.001s
> sys    0m0.002s
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>