[Gluster-users] Fwd:Re: client is terrible with large amount of small files

gjprabu gjprabu at zohocorp.com
Thu May 7 06:53:02 UTC 2015


Hi Vijay,

    Do we have any other options to increase the performance.

Regards
Prabu





---- On Wed, 06 May 2015 15:51:20 +0530 gjprabu <gjprabu at zohocorp.com> wrote ---- 

Hi Vijay,

       We tired on physical machines but its doesn't improve speed.

# gluster volume info
 
Volume Name: integvoltest
Type: Replicate
Volume ID: 6c66afb9-d466-428e-b944-e15d7a1be5f2
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: integ-gluster3:/srv/sdb1/brick7
Brick2: integ-gluster4:/srv/sdb1/brick7
Options Reconfigured:
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
cluster.ensure-durability: off
cluster.readdir-optimize: on
performance.readdir-ahead: on
server.event-threads: 30
client.event-threads: 30

 
    
     
gluster volume profile integvoltest info
Brick: integ-gluster3:/srv/sdb1/brick7
--------------------------------------
Cumulative Stats:
   Block Size:                  4b+                   8b+                  16b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                    2                     4                     8 
 
   Block Size:                 32b+                  64b+                 128b+ 
 No. of Reads:                    2                     2                     2 
No. of Writes:                    8                     6                     6 
 
   Block Size:                256b+                 512b+                1024b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                     2                     6 
 
   Block Size:               2048b+                4096b+ 
 No. of Reads:                    0                     0 
No. of Writes:                    2                  6507 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             37      FORGET
      0.00       0.00 us       0.00 us       0.00 us             61     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         202442  RELEASEDIR
      0.00     142.00 us     142.00 us     142.00 us              1 REMOVEXATTR
      0.00     101.00 us      79.00 us     142.00 us              3        STAT
      0.00     313.00 us     313.00 us     313.00 us              1     XATTROP
      0.00     120.00 us      96.00 us     145.00 us              3        READ
      0.00     101.75 us      69.00 us     158.00 us              4      STATFS
      0.00     131.25 us     112.00 us     147.00 us              4    GETXATTR
      0.00     256.00 us     216.00 us     309.00 us              3      UNLINK
      0.00     820.00 us     820.00 us     820.00 us              1     SYMLINK
      0.00     109.80 us      72.00 us     197.00 us             10     READDIR
      0.00     125.58 us     100.00 us     161.00 us             12     SETATTR
      0.00     138.36 us     102.00 us     196.00 us             11        OPEN
      0.00      55.38 us      24.00 us     240.00 us             29       FLUSH
      0.00     445.00 us     125.00 us     937.00 us              4    READDIRP
      0.01     306.43 us     165.00 us     394.00 us              7      RENAME
      0.01     199.55 us     153.00 us     294.00 us             11    SETXATTR
      0.01      72.64 us      28.00 us     227.00 us             47    FINODELK
      0.02      67.69 us      30.00 us     241.00 us             96     ENTRYLK
      0.03    1038.18 us     943.00 us    1252.00 us             11       MKDIR
      0.03     251.49 us     147.00 us     865.00 us             53    FXATTROP
      0.06    1115.60 us     808.00 us    1860.00 us             20      CREATE
      0.07     323.83 us      31.00 us   22132.00 us             88     INODELK
      1.41     170.57 us      79.00 us    2022.00 us           3262       WRITE
     26.35     103.15 us       4.00 us     260.00 us         100471     OPENDIR
     71.98     139.07 us      47.00 us     471.00 us         203591      LOOKUP
 
    Duration: 1349 seconds
   Data Read: 624 bytes
Data Written: 26675732 bytes
 
Interval 25 Stats:
   Block Size:               4096b+ 
 No. of Reads:                    0 
No. of Writes:                  108 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           4887  RELEASEDIR
      1.06     155.19 us     117.00 us     248.00 us            108       WRITE
     25.69      83.24 us      35.00 us     260.00 us           4888     OPENDIR
     73.25     117.83 us      51.00 us     406.00 us           9844      LOOKUP
 
    Duration: 17 seconds
   Data Read: 0 bytes
Data Written: 442368 bytes
 
Brick: integ-gluster4:/srv/sdb1/brick7
--------------------------------------
Cumulative Stats:
   Block Size:                  4b+                   8b+                  16b+ 
 No. of Reads:                    0                     0                     1 
No. of Writes:                    2                     4                     8 
 
   Block Size:                 32b+                  64b+                 128b+ 
 No. of Reads:                    2                     2                     2 
No. of Writes:                    8                     6                     6 
 
   Block Size:                256b+                 512b+                1024b+ 
 No. of Reads:                    0                     0                     0 
No. of Writes:                    4                     2                     6 
 
   Block Size:               2048b+                4096b+ 
 No. of Reads:                    0                     0 
No. of Writes:                    2                  6507 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us             37      FORGET
      0.00       0.00 us       0.00 us       0.00 us             61     RELEASE
      0.00       0.00 us       0.00 us       0.00 us         202444  RELEASEDIR
      0.00     124.00 us     124.00 us     124.00 us              1 REMOVEXATTR
      0.00     186.00 us     186.00 us     186.00 us              1     XATTROP
      0.00      90.00 us      49.00 us     131.00 us              3    GETXATTR
      0.00     276.00 us     276.00 us     276.00 us              1     SYMLINK
      0.00      92.75 us      49.00 us     119.00 us              4      STATFS
      0.00     125.00 us      58.00 us     161.00 us              3      UNLINK
      0.00     135.75 us      90.00 us     173.00 us              4        READ
      0.00      64.73 us      54.00 us     131.00 us             11    SETXATTR
      0.00      93.80 us      40.00 us     129.00 us             10     READDIR
      0.00      95.00 us      75.00 us     159.00 us             12     SETATTR
      0.00     121.27 us      93.00 us     156.00 us             11        OPEN
      0.00     257.14 us     106.00 us     404.00 us              7      RENAME
      0.00      62.76 us      26.00 us     245.00 us             29       FLUSH
      0.00     313.73 us     256.00 us     397.00 us             11       MKDIR
      0.00      74.08 us      23.00 us     243.00 us             49    FINODELK
      0.01      62.12 us      20.00 us     145.00 us             88     INODELK
      0.01      58.81 us      18.00 us     120.00 us             96     ENTRYLK
      0.01     370.35 us     285.00 us     479.00 us             20      CREATE
      0.01     149.85 us      70.00 us     230.00 us             53    FXATTROP
      0.03      69.19 us      32.00 us     167.00 us            350        STAT
      0.46     130.36 us      59.00 us    1042.00 us           3262       WRITE
     10.07      91.73 us       5.00 us     759.00 us         100472     OPENDIR
     24.90     111.87 us      40.00 us     954.00 us         203595      LOOKUP
     64.49     293.59 us      28.00 us    2779.00 us         200938    READDIRP
 
    Duration: 1349 seconds
   Data Read: 624 bytes
Data Written: 26675732 bytes
 
Interval 25 Stats:
   Block Size:               4096b+ 
 No. of Reads:                    0 
No. of Writes:                  108 
 %-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
 ---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us           4888  RELEASEDIR
      0.03      92.18 us      55.00 us     161.00 us             17        STAT
      0.31     144.44 us     108.00 us     382.00 us            108       WRITE
     11.20     113.73 us      53.00 us     203.00 us           4887     OPENDIR
     26.70     134.65 us      79.00 us     265.00 us           9844      LOOKUP
     61.76     313.65 us      56.00 us     652.00 us           9774    READDIRP
 
    Duration: 17 seconds
   Data Read: 0 bytes
Data Written: 442368 bytes
 
strace -Tcf git clone
% time     seconds  usecs/call     calls    errors syscall
------ ----------- ----------- --------- --------- ----------------
100.00    0.000026           2        13           mmap
  0.00    0.000000           0         3           read
  0.00    0.000000           0        77           write
  0.00    0.000000           0         4           open
  0.00    0.000000           0         4           close
  0.00    0.000000           0         4           fstat
  0.00    0.000000           0         7           mprotect
  0.00    0.000000           0         1           munmap
  0.00    0.000000           0         3           brk
  0.00    0.000000           0         2           rt_sigaction
  0.00    0.000000           0         1           rt_sigprocmask
  0.00    0.000000           0         1         1 access
  0.00    0.000000           0         1           execve
  0.00    0.000000           0         1           getrlimit
  0.00    0.000000           0         1           arch_prctl
  0.00    0.000000           0         2         1 futex
  0.00    0.000000           0         1           set_tid_address
  0.00    0.000000           0         1           set_robust_list
------ ----------- ----------- --------- --------- ----------------
100.00    0.000026                   127         2 total
 

Regards
Prabu





---- On Tue, 05 May 2015 15:51:58 +0530 Vijay Bellur<vbellur at redhat.com> wrote ---- 

On 05/05/2015 03:43 PM, Kamal wrote: 
> Hi Vijay, 
> 
> We Tried the same. But its doesn't improve speed. 
> 
> For testing glusterfs, we are running storage in virtual machines. Is 
> that will make any difference ? 
 
Performance testing on physical machines is a better bet as it takes 
some variables away from the equation. 
 
Since you happen to use a replicated volume, can you try by enabling 
this option? 
 
gluster volume set <volname> cluster.ensure-durability off 
 
Additionally you might want to try "strace -Tcf git clone .." and 
"gluster volume profile ..." to figure out where the latency is stemming 
from. 
 
 
> But, copying the same folder between two storage machines is really 
> fast. fyi. 
> 
 
Does this copying involve gluster or not? 
 
Regards, 
Vijay 
 
> Regards, 
> Kamal 
> 
> 
> 
> ---- On Tue, 05 May 2015 15:08:27 +0530 *Vijay 
> Bellur<vbellur at redhat.com>* wrote ---- 
> 
> On 05/05/2015 12:59 PM, Kamal wrote: 
> > Hi Amukher, 
> > 
> > Even after upgrade to 3.7 small files transfer rate is slow. 
> > 
> > Below is the volume info. 
> > 
> > Volume Name: integvol1 
> > Type: Replicate 
> > Volume ID: 31793ba4-eeca-462a-a0cd-9adfb281225b 
> > Status: Started 
> > Number of Bricks: 1 x 2 = 2 
> > Transport-type: tcp 
> > Bricks: 
> > Brick1: integ-gluster1:/srv/sdb2/brick4 
> > Brick2: integ-gluster2:/srv/sdb2/brick4 
> > Options Reconfigured: 
> > server.event-threads: 30 
> > client.event-threads: 30 
> > ---- 
> > 
> > I understand that for replication it would take some more time, but 
> > here its taking more time. 
> > 
> > 
> > Time taken for git clone in non gluster directory = 25 sec 
> > 
> > Time taken for git clone in gluster directory = 14 minutes 
> > 
> > Its a huge difference. Plz let me know any other tuning 
> parameters need 
> > to be done. 
> > 
> > 
> 
> I have seen this before and it primarily seems to be related to the 
> readdir calls done by git clone. 
> 
> Turning on these options might help to some extent: 
> 
> gluster volume set <volname> performance.readdir-ahead on 
> 
> gluster volume set <volname> cluster.readdir-optimize on 
> 
> Please do let us know what you observe with these options enabled. 
> 
> Regards, 
> Vijay 
> 
> 
> 
 
_______________________________________________ 
Gluster-users mailing list 
Gluster-users at gluster.org 
http://www.gluster.org/mailman/listinfo/gluster-users 










-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20150507/f37a12fc/attachment.html>


More information about the Gluster-users mailing list