[Gluster-devel] Df, ls, reads slow during I/O
Dan Parsons
dparsons at nyip.net
Mon May 26 21:11:24 UTC 2008
And now, after a while of running, the cache doesn't seem to get used
at all anymore. My jobs are running fine, but glusterfs memory usage
is super low and all jobs are pulling data directly from the network.
How does io-cache determine what files to cache and for how long?
Dan Parsons
On May 26, 2008, at 1:17 PM, Jordan Mendler wrote:
> I have optimized my glusterfs system pretty well for writes, but I
> am having
> an issue where doing an ls is very slow, and during I/O operations
> df takes
> forever and often hangs on glusterfs. Below are some examples of the
> slow ls
> (at the time of the tests, there are no filesystem operations
> occuring other
> than the ls).
>
> I am finding that during heavy reads problems are much worse, though
> they
> also can be bad during heavy writes.
>
> Does anyone have any ideas of how I can track this problem down? It
> is the
> only thing preventing our system from being production-ready. The
> current
> hardware setup is 2 storage servers using ALU on a 16xSATA RAID6
> (8cores
> each w/ 8gb RAM), and an AFR'ed namespace on the same hardware
> RAIDs. My
> configuration is available at http://genome.ucla.edu/~jordan/gluster/
>
> Thanks so much,
> Jordan
> -----------------
>
> Examples of slow ls during no FS operations:
>
> [root at solexa solexa]# time ls solexa_datasets/
> 070403_SLXA-EAS20_0001_FC5063 071121_HWI-EAS172_200C0
> docs
> 070405_SLXA-EAS20_0005_FC5065 071203_HWI-EAS172_200BA
> genomes
> 070418_SLXA-EAS20_5256 071207_HWI-EAS172_2015A
> Instruments
> 070427_SLXA-EAS20_5152 071211_HWI-EAS172_200gh
> L001.tar
> 070515_SLXA-EAS20_5612 071217_HWI-EAS172_14759
> L007.tar
> 070523_SLXA-EAS20_5153 071221_HWI-EAS172_200M1
> ls_d_images.output
> 070529_SLXA-EAS20_5594 080108_HWI-EAS172_200G8
> ls_d_images.output.20071029
> 070604_SLXA-EAS20_5447 080116_HWI-EAS172_14758
> ls_d_images.output.20071029.truncated
> 070608_SLXA-EAS20_FC5459 080121_HWI-EAS172_2040F
> ls_d_images.output.2007103016.truncated
> 070612_SLXA-EAS20_5646 080125_HWI-EAS172_13471R
> ls_d_images.output.2007103017.truncated
> 070621_SLXA-EAS20_5590 080128_HWI-EAS172_201UD
> ls_d_images.output.20071030.truncated
> 070625_SLXA-EAS20_5701 080204_HWI-EAS172_201ET
> ls_d_images.output.200710311329.truncated
> 070629_SLXA-EAS20_5861 080206_SLXA-EAS20_203NA
> ls_d_images.output.200710311349.truncated
> 070703_SLXA-EAS20_5731 080215_HWI-EAS172_2009B
> ls_d_images.output.200710311350.nottruncatedbutmissingdoneitems
> 070709_SLXA-EAS20_5611 080226_HWI-EAS172_205A0
> ls_d_images.output.200710311749.truncated
> 070713_SLXA-EAS20_5863 080304_HWI-EAS172_204AY
> ls_d_images.output.200711010821.truncated
> 070717_SLXA-EAS20_5699 080307_HWI-EAS172_20ANM
> ls_d_images.output.200711010845.truncated
> 070723_SLXA-EAS20_5540 080311_HWI-EAS172_204KJ
> ls_d_images.output.200711051802.truncated
> 070727_SLXA-EAS20_4606 080314_HWI-EAS172_204MG
> ls_d_images.output.clustertemp1.200711011622
> 070731_SLXA-EAS20_4611 080320_HWI-EAS172_204KB
> redo.200711061339
> 070810_SLXA-EAS20_5866 080326_HWI-EAS172_204KP
> Reports
> 070814_SLXA-EAS20_5697 080401_HWI-EAS172_20CLK
> rsync_to_storage.pl
> 070817_SLXA-EAS20_5603 080407_HWI-EAS172_204MN
> Runs_Comments
> 070821_SLXA-EAS20_11290 080411_HWI-EAS172_204K1
> Shawn_basecalls_U87
> 070827_SLXA-EAS20_11279 080418_HWI-EAS172_204T7
> solexatmp0.tarfile2dirname
> 070831_SLXA-EAS20_5406 080429_HWI-EAS172_204KD
> solexatmp0.tarfile2dirname.new
> 070925_SLXA-EAS20_11296 080505_HWI-EAS172_20DYC
> solexatmp1.tarfile2dirname
> 071002_SLXA-EAS20_4977 080513_HWI-EAS172_20DYG
> testdir
> 071022_HWI-EAS172_11989 080515_HWI-EAS172_20DYG_RE
> testdir.tar
> 071026_HWI-EAS172_11989reseq 080516_HWI-EAS335_305T4
> test.pl
> 071030_HWI-EAS172_14517 1
> U87.fasta.bz2
> 071105_HWI-EAS172_14515 4276.fasta
> verify_JMM.log
> 071109_HWI-EAS172_14034 -al
> verify_JMM.sh
> 071115_HWI-EAS172_14055 clustertemp1.tarfiles.200711051731
>
> real 0m2.298s
> user 0m0.001s
> sys 0m0.001s
>
> [root at solexa solexa]# time ls solexa_datasets/070817_SLXA-EAS20_5603/
> Data
> C2-36_Firecrest1.8.28_04-10-2007_solexa default_offsets.txt
>
> real 0m6.246s
> user 0m0.001s
> sys 0m0.002s
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
More information about the Gluster-devel
mailing list