[Gluster-users] Gluster slow reads

[Ashish Pandey]

Hi, 

I am keeping Raghvendra in loop and hope he can comment on the "Read being scheduled as slow fop". 

Other than that, I would request you to provide following information to debug this issue. 

1 - Profile information of the volume. You can find the steps here - 
https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/ 

2 - Gluster volume <volname> status 

3 - Gluster volume heal <volname> info 

4 - Check the size of each bricks to see if it has been filled. 

5 - gluster version of client and server. 

6 - Host OS version of client and server. 

--- 
Ashish 

----- Original Message -----

From: adi at telecompartners.ro 
To: gluster-users at gluster.org 
Sent: Thursday, October 10, 2019 8:08:24 PM 
Subject: [Gluster-users] Gluster slow reads 


Hi, 

I have a very small distributed dispersed volume with 4+2 and 2 servers 
with 3 bricks on each of them. The volume is mounted via fuse client on 
another Linux server. 

The volume worked well for a few months in this setup. However in the 
last few days I have a very slow read speed (gluster uploads to the 
gluster client via the mount point). By slow I mean 3-4 MB/sec on a 
gigabit link. I don't have small files stored on gluster, the smallest 
file is around 30-40 MB. The networking between the client and the 
bricks is fine (all of them are connected in the same switch, no errors, 
some iptraf tests directly between client and gluster servers are 
looking good). On the same client I have mounted another 2 gluster 
volumes from other servers, both of them are ok. 

My 'gluster volume info' details are the following : 

Volume Name: gluster4-vol 
Type: Disperse 
Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59 
Status: Started 
Snapshot Count: 0 
Number of Bricks: 1 x (4 + 2) = 6 
Transport-type: tcp 
Bricks: 
Brick1: gluster4:/export/sdb1/brick 
Brick2: gluster5:/export/sdb1/brick 
Brick3: gluster4:/export/sdc1/brick 
Brick4: gluster5:/export/sdc1/brick 
Brick5: gluster4:/export/sdd1/brick 
Brick6: gluster5:/export/sdd1/brick 
Options Reconfigured: 
network.ping-timeout: 60 
performance.client-io-threads: on 
performance.io-thread-count: 32 
cluster.readdir-optimize: on 
performance.cache-size: 1GB 
client.event-threads: 10 
server.event-threads: 10 
cluster.lookup-optimize: on 
server.allow-insecure: on 
storage.reserve: 0 
transport.address-family: inet 
nfs.disable: on 


In the gluster client debug logs I noticed the following lines : 

[2019-10-10 13:49:58.571879] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN 
scheduled as fast fop 
[2019-10-10 13:49:58.571964] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT 
scheduled as fast fop 
[2019-10-10 13:49:58.572058] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH 
scheduled as normal fop 
[2019-10-10 13:49:58.572728] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN 
scheduled as fast fop 
[2019-10-10 13:49:58.576275] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT 
scheduled as fast fop 
[2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal] 
0-logging-infra: Buffer overflow of a buffer whose size limit is 5. 
About to flush least recently used log message to disk 
The message "D [MSGID: 0] [io-threads.c:356:iot_schedule] 
0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285 
times between [2019-10-10 13:49:58.357922] and [2019-10-10 
13:50:07.837047] 
[2019-10-10 13:50:07.837068] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH 
scheduled as normal fop 
[2019-10-10 13:50:07.837165] D [MSGID: 0] 
[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ 
scheduled as slow fop 


What does it mean exactly "READ scheduled as slow fop" ? Can I schedule 
READs as normal or fast fop like the other operations ? 

I'm using this gluster volume only for reading so I don't care about 
writes right now. 


Thanks. 

________ 

Community Meeting Calendar: 

APAC Schedule - 
Every 2nd and 4th Tuesday at 11:30 AM IST 
Bridge: https://bluejeans.com/118564314 

NA/EMEA Schedule - 
Every 1st and 3rd Tuesday at 01:00 PM EDT 
Bridge: https://bluejeans.com/118564314 

Gluster-users mailing list 
Gluster-users at gluster.org 
https://lists.gluster.org/mailman/listinfo/gluster-users 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20191016/0004a4f4/attachment.html>