[Gluster-users] Gluster slow reads

[adi at telecompartners.ro]

  Hi,

  I have a very small distributed dispersed volume with 4+2 and 2 servers 
with 3 bricks on each of them. The volume is mounted via fuse client on 
another Linux server.

   The volume worked well for a few months in this setup. However in the 
last few days I have a very slow read speed (gluster uploads to the 
gluster client via the mount point).  By slow I mean 3-4 MB/sec on a 
gigabit link. I don't have small files stored on gluster, the smallest 
file is around 30-40 MB. The networking  between the client and the 
bricks is fine (all of them are connected in the same switch, no errors, 
some iptraf tests directly between client and gluster servers are 
looking good). On the same client I have mounted another 2 gluster 
volumes from other servers, both of them are ok.

  My 'gluster volume info' details are the following :

Volume Name: gluster4-vol
Type: Disperse
Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (4 + 2) = 6
Transport-type: tcp
Bricks:
Brick1: gluster4:/export/sdb1/brick
Brick2: gluster5:/export/sdb1/brick
Brick3: gluster4:/export/sdc1/brick
Brick4: gluster5:/export/sdc1/brick
Brick5: gluster4:/export/sdd1/brick
Brick6: gluster5:/export/sdd1/brick
Options Reconfigured:
network.ping-timeout: 60
performance.client-io-threads: on
performance.io-thread-count: 32
cluster.readdir-optimize: on
performance.cache-size: 1GB
client.event-threads: 10
server.event-threads: 10
cluster.lookup-optimize: on
server.allow-insecure: on
storage.reserve: 0
transport.address-family: inet
nfs.disable: on


  In the gluster client debug logs I noticed the following lines :

  [2019-10-10 13:49:58.571879] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN 
scheduled as fast fop
[2019-10-10 13:49:58.571964] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT 
scheduled as fast fop
[2019-10-10 13:49:58.572058] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH 
scheduled as normal fop
[2019-10-10 13:49:58.572728] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN 
scheduled as fast fop
[2019-10-10 13:49:58.576275] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT 
scheduled as fast fop
[2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk
The message "D [MSGID: 0] [io-threads.c:356:iot_schedule] 
0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285 
times between [2019-10-10 13:49:58.357922] and [2019-10-10 
13:50:07.837047]
[2019-10-10 13:50:07.837068] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH 
scheduled as normal fop
[2019-10-10 13:50:07.837165] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ 
scheduled as slow fop


  What does it mean exactly "READ scheduled as slow fop" ? Can I schedule 
READs as normal or fast fop like the other operations ?

  I'm using this gluster volume only for reading so I don't care about 
writes right now.


  Thanks.