[Gluster-devel] Df, ls, reads slow during I/O

Mon May 26 21:13:10 UTC 2008

Sorry, I replied to the wrong thread, please disregard.

Dan Parsons

On May 26, 2008, at 2:11 PM, Dan Parsons wrote:

> And now, after a while of running, the cache doesn't seem to get  
> used at all anymore. My jobs are running fine, but glusterfs memory  
> usage is super low and all jobs are pulling data directly from the  
> network.
>
> How does io-cache determine what files to cache and for how long?
>
>
> Dan Parsons
>
>
> On May 26, 2008, at 1:17 PM, Jordan Mendler wrote:
>
>> I have optimized my glusterfs system pretty well for writes, but I  
>> am having
>> an issue where doing an ls is very slow, and during I/O operations  
>> df takes
>> forever and often hangs on glusterfs. Below are some examples of  
>> the slow ls
>> (at the time of the tests, there are no filesystem operations  
>> occuring other
>> than the ls).
>>
>> I am finding that during heavy reads problems are much worse,  
>> though they
>> also can be bad during heavy writes.
>>
>> Does anyone have any ideas of how I can track this problem down? It  
>> is the
>> only thing preventing our system from being production-ready. The  
>> current
>> hardware setup is 2 storage servers using ALU on a 16xSATA RAID6  
>> (8cores
>> each w/ 8gb RAM), and an AFR'ed namespace on the same hardware  
>> RAIDs. My
>> configuration is available at http://genome.ucla.edu/~jordan/gluster/
>>
>> Thanks so much,
>> Jordan
>> -----------------
>>
>> Examples of slow ls during no FS operations:
>>
>> [root at solexa solexa]# time ls solexa_datasets/
>> 070403_SLXA-EAS20_0001_FC5063  071121_HWI-EAS172_200C0              
>> docs
>> 070405_SLXA-EAS20_0005_FC5065  071203_HWI-EAS172_200BA              
>> genomes
>> 070418_SLXA-EAS20_5256         071207_HWI-EAS172_2015A
>> Instruments
>> 070427_SLXA-EAS20_5152         071211_HWI-EAS172_200gh              
>> L001.tar
>> 070515_SLXA-EAS20_5612         071217_HWI-EAS172_14759              
>> L007.tar
>> 070523_SLXA-EAS20_5153         071221_HWI-EAS172_200M1
>> ls_d_images.output
>> 070529_SLXA-EAS20_5594         080108_HWI-EAS172_200G8
>> ls_d_images.output.20071029
>> 070604_SLXA-EAS20_5447         080116_HWI-EAS172_14758
>> ls_d_images.output.20071029.truncated
>> 070608_SLXA-EAS20_FC5459       080121_HWI-EAS172_2040F
>> ls_d_images.output.2007103016.truncated
>> 070612_SLXA-EAS20_5646         080125_HWI-EAS172_13471R
>> ls_d_images.output.2007103017.truncated
>> 070621_SLXA-EAS20_5590         080128_HWI-EAS172_201UD
>> ls_d_images.output.20071030.truncated
>> 070625_SLXA-EAS20_5701         080204_HWI-EAS172_201ET
>> ls_d_images.output.200710311329.truncated
>> 070629_SLXA-EAS20_5861         080206_SLXA-EAS20_203NA
>> ls_d_images.output.200710311349.truncated
>> 070703_SLXA-EAS20_5731         080215_HWI-EAS172_2009B
>> ls_d_images.output.200710311350.nottruncatedbutmissingdoneitems
>> 070709_SLXA-EAS20_5611         080226_HWI-EAS172_205A0
>> ls_d_images.output.200710311749.truncated
>> 070713_SLXA-EAS20_5863         080304_HWI-EAS172_204AY
>> ls_d_images.output.200711010821.truncated
>> 070717_SLXA-EAS20_5699         080307_HWI-EAS172_20ANM
>> ls_d_images.output.200711010845.truncated
>> 070723_SLXA-EAS20_5540         080311_HWI-EAS172_204KJ
>> ls_d_images.output.200711051802.truncated
>> 070727_SLXA-EAS20_4606         080314_HWI-EAS172_204MG
>> ls_d_images.output.clustertemp1.200711011622
>> 070731_SLXA-EAS20_4611         080320_HWI-EAS172_204KB
>> redo.200711061339
>> 070810_SLXA-EAS20_5866         080326_HWI-EAS172_204KP              
>> Reports
>> 070814_SLXA-EAS20_5697         080401_HWI-EAS172_20CLK
>> rsync_to_storage.pl
>> 070817_SLXA-EAS20_5603         080407_HWI-EAS172_204MN
>> Runs_Comments
>> 070821_SLXA-EAS20_11290        080411_HWI-EAS172_204K1
>> Shawn_basecalls_U87
>> 070827_SLXA-EAS20_11279        080418_HWI-EAS172_204T7
>> solexatmp0.tarfile2dirname
>> 070831_SLXA-EAS20_5406         080429_HWI-EAS172_204KD
>> solexatmp0.tarfile2dirname.new
>> 070925_SLXA-EAS20_11296        080505_HWI-EAS172_20DYC
>> solexatmp1.tarfile2dirname
>> 071002_SLXA-EAS20_4977         080513_HWI-EAS172_20DYG              
>> testdir
>> 071022_HWI-EAS172_11989        080515_HWI-EAS172_20DYG_RE
>> testdir.tar
>> 071026_HWI-EAS172_11989reseq   080516_HWI-EAS335_305T4              
>> test.pl
>> 071030_HWI-EAS172_14517        1
>> U87.fasta.bz2
>> 071105_HWI-EAS172_14515        4276.fasta
>> verify_JMM.log
>> 071109_HWI-EAS172_14034        -al
>> verify_JMM.sh
>> 071115_HWI-EAS172_14055        clustertemp1.tarfiles.200711051731
>>
>> real    0m2.298s
>> user    0m0.001s
>> sys    0m0.001s
>>
>> [root at solexa solexa]# time ls solexa_datasets/070817_SLXA- 
>> EAS20_5603/Data
>> C2-36_Firecrest1.8.28_04-10-2007_solexa  default_offsets.txt
>>
>> real    0m6.246s
>> user    0m0.001s
>> sys    0m0.002s
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at nongnu.org
>> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>>
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>