[Gluster-users] FW: Performance with Gluster+Fuse is 60x slower then Gluster+NFS ?

Van Renterghem Stijn stijn.vanrenterghem at cloudar.be
Wed Feb 17 15:23:35 UTC 2016


Hi,

I have setup a server with a new installation of Gluster.
The volume type is 'Replicate'.

1)
I mounted the volume with Fuse
IP1:/app       /srv/data               glusterfs       defaults,_netdev,backupvolfile-server=IP2,fetch-attempts=2      0 0

When I start my application, it takes 2h until the application is started
Below you can see the stats after the application is started. I can see a very high LOOKUP value.
Can you explain this high value ? The volume type is replicate, so I should think, I shouldn't have LOOKUPs ?

Interval2
Block Size:                  1b+                  16b+                  32b+
No. of Reads:                    0                     0                     0
No. of Writes:                  342                    25                   575

   Block Size:                 64b+                 128b+                 256b+
No. of Reads:                    0                     0                     0
No. of Writes:                  143                   898                   118

   Block Size:                512b+                1024b+                2048b+
No. of Reads:                    1                     4                    11
No. of Writes:                   82                     0                     0

   Block Size:               4096b+                8192b+               16384b+
No. of Reads:                   11                    31                    39
No. of Writes:                    0                     0                     0

   Block Size:              32768b+               65536b+              131072b+
No. of Reads:                   59                   148                   555
No. of Writes:                    0                     0                     0

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              1      FORGET
      0.00       0.00 us       0.00 us       0.00 us            201     RELEASE
      0.00       0.00 us       0.00 us       0.00 us          54549  RELEASEDIR
      0.00      47.00 us      47.00 us      47.00 us              1 REMOVEXATTR
      0.00      94.00 us      74.00 us     114.00 us              2     XATTROP
      0.00     191.00 us     191.00 us     191.00 us              1    TRUNCATE
      0.00      53.50 us      35.00 us      74.00 us              4      STATFS
      0.00      79.67 us      70.00 us      91.00 us              3      RENAME
      0.00      37.33 us      27.00 us      68.00 us             15     INODELK
      0.00     190.67 us     116.00 us     252.00 us              3      UNLINK
      0.00      28.83 us       8.00 us      99.00 us             30     ENTRYLK
      0.00     146.33 us     117.00 us     188.00 us              6      CREATE
      0.00      37.63 us      12.00 us      73.00 us             84     READDIR
      0.00      23.75 us       8.00 us      75.00 us            198       FLUSH
      0.00      65.33 us      42.00 us     141.00 us            204        OPEN
      0.01      45.78 us      11.00 us     191.00 us            944    FINODELK
      0.01      80.34 us      31.00 us     211.00 us            859        READ
      0.02      96.74 us      50.00 us     188.00 us            944    FXATTROP
      0.02      55.84 us      24.00 us     140.00 us           1707       FSTAT
      0.02      52.89 us      21.00 us     175.00 us           2183       WRITE
      0.02      59.69 us      11.00 us     235.00 us           2312    GETXATTR
      0.03      51.18 us       8.00 us     142.00 us           3091        STAT
      0.46      48.66 us       1.00 us     179.00 us          54549     OPENDIR
      1.13     135.93 us      18.00 us   16362.00 us          48124    READDIRP
     98.29      70.46 us      16.00 us    2903.00 us        8104385      LOOKUP

    Duration: 7560 seconds
   Data Read: 91208567 bytes = 91MB
Data Written: 292007 bytes = 0,292MB



2)
I have tried some tuning options, but that didn't changed anything:

#gluster volume info app

Volume Name: app
Type: Replicate
Volume ID: f1b59aec-adf8-41f8-ad95-839ace247041
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: IP1:/exports/app/app
Brick2: IP2:/exports/app/app
Options Reconfigured:
cluster.readdir-optimize: on
server.event-threads: 8
client.event-threads: 8
cluster.lookup-optimize: on
diagnostics.count-fop-hits: on
diagnostics.latency-measurement: on
auth.allow: client1,client2
nfs.rpc-auth-allow: client1,client2
nfs.export-volumes: on
nfs.addr-namelookup: off
nfs.disable: off
performance.readdir-ahead: on
performance.io-thread-count: 64




3)
I then have enabled NFS support.
I stopped the application and unmounted the volume. I then mounted it again with nfs:
IP1:/app        /srv/data               nfs             rsize=4096,wsize=4096,hard,intr  0 0

I started the application again and it was running within 3minutes.
The stats with NFS where very different then with Fuse. It seems that they are almost not logged.

Interval 11 Stats:
   Block Size:                128b+                 256b+                 512b+
No. of Reads:                    0                     0                     0
No. of Writes:                    9                     1                     1

   Block Size:               1024b+                2048b+                4096b+
No. of Reads:                    0                     0                     0
No. of Writes:                    1                     5                     8

%-latency   Avg-latency   Min-Latency   Max-Latency   No. of calls         Fop
---------   -----------   -----------   -----------   ------------        ----
      0.00       0.00 us       0.00 us       0.00 us              2     RELEASE
      0.00       0.00 us       0.00 us       0.00 us              1  RELEASEDIR
      0.02       2.00 us       2.00 us       2.00 us              1     OPENDIR
      0.57      34.00 us      19.00 us      49.00 us              2     READDIR
      0.81      96.00 us      96.00 us      96.00 us              1     SETATTR
      1.06      62.50 us      61.00 us      64.00 us              2        OPEN
      1.39     164.00 us     164.00 us     164.00 us              1    TRUNCATE
      1.39      41.25 us      30.00 us      52.00 us              4    GETXATTR
      1.54      91.00 us      86.00 us      96.00 us              2     XATTROP
      2.72      80.50 us      29.00 us     122.00 us              4      LOOKUP
      2.81      33.30 us      17.00 us      56.00 us             10     INODELK
     10.36      76.69 us      26.00 us     133.00 us             16       FLUSH
     15.83      75.00 us      61.00 us     105.00 us             25       WRITE
     17.22      48.55 us      13.00 us      78.00 us             42    FINODELK
     44.28     124.83 us      62.00 us     161.00 us             42    FXATTROP

    Duration: 580 seconds
   Data Read: 0 bytes
Data Written: 60839 bytes



What is wrong with the Fuse client ?
Why does my application start in 120min with Gluster+Fuse and in 3min with Gluster+NFS ?





Regards,
Stijn
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160217/9066b422/attachment.html>


More information about the Gluster-users mailing list