<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Hi,<br></div><div><br></div><div>I am keeping Raghvendra in loop and hope he can comment on the "Read being scheduled as slow fop".<br></div><div><br></div><div>Other than that, I would request you to provide following information to debug this issue.<br></div><div><br></div><div>1 - Profile information of the volume. You can find the steps here - <br></div><div><a href="https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/">https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/</a></div><div><br></div><div>2 - Gluster volume <volname> status<br></div><div><br></div><div>3 - Gluster volume heal <volname> info<br></div><div><br></div><div>4 - Check the size of each bricks to see if it has been filled.<br></div><div><br></div><div>5 - gluster version of client and server.<br></div><div><br></div><div>6 - Host OS version of client and server.<br></div><div><br></div><div>---<br></div><div>Ashish<br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;" data-mce-style="color: #000; font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica,Arial,sans-serif; font-size: 12pt;"><b>From: </b>adi@telecompartners.ro<br><b>To: </b>gluster-users@gluster.org<br><b>Sent: </b>Thursday, October 10, 2019 8:08:24 PM<br><b>Subject: </b>[Gluster-users] Gluster slow reads<br><div><br></div><br> Hi,<br><div><br></div> I have a very small distributed dispersed volume with 4+2 and 2 servers <br>with 3 bricks on each of them. The volume is mounted via fuse client on <br>another Linux server.<br><div><br></div> The volume worked well for a few months in this setup. However in the <br>last few days I have a very slow read speed (gluster uploads to the <br>gluster client via the mount point). By slow I mean 3-4 MB/sec on a <br>gigabit link. I don't have small files stored on gluster, the smallest <br>file is around 30-40 MB. The networking between the client and the <br>bricks is fine (all of them are connected in the same switch, no errors, <br>some iptraf tests directly between client and gluster servers are <br>looking good). On the same client I have mounted another 2 gluster <br>volumes from other servers, both of them are ok.<br><div><br></div> My 'gluster volume info' details are the following :<br><div><br></div>Volume Name: gluster4-vol<br>Type: Disperse<br>Volume ID: fa464bb9-b034-4fce-a56e-7ac157432d59<br>Status: Started<br>Snapshot Count: 0<br>Number of Bricks: 1 x (4 + 2) = 6<br>Transport-type: tcp<br>Bricks:<br>Brick1: gluster4:/export/sdb1/brick<br>Brick2: gluster5:/export/sdb1/brick<br>Brick3: gluster4:/export/sdc1/brick<br>Brick4: gluster5:/export/sdc1/brick<br>Brick5: gluster4:/export/sdd1/brick<br>Brick6: gluster5:/export/sdd1/brick<br>Options Reconfigured:<br>network.ping-timeout: 60<br>performance.client-io-threads: on<br>performance.io-thread-count: 32<br>cluster.readdir-optimize: on<br>performance.cache-size: 1GB<br>client.event-threads: 10<br>server.event-threads: 10<br>cluster.lookup-optimize: on<br>server.allow-insecure: on<br>storage.reserve: 0<br>transport.address-family: inet<br>nfs.disable: on<br><div><br></div><br> In the gluster client debug logs I noticed the following lines :<br><div><br></div> [2019-10-10 13:49:58.571879] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN <br>scheduled as fast fop<br>[2019-10-10 13:49:58.571964] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT <br>scheduled as fast fop<br>[2019-10-10 13:49:58.572058] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH <br>scheduled as normal fop<br>[2019-10-10 13:49:58.572728] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: OPEN <br>scheduled as fast fop<br>[2019-10-10 13:49:58.576275] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FSTAT <br>scheduled as fast fop<br>[2019-10-10 13:50:07.837069] D [logging.c:1952:_gf_msg_internal] <br>0-logging-infra: Buffer overflow of a buffer whose size limit is 5. <br>About to flush least recently used log message to disk<br>The message "D [MSGID: 0] [io-threads.c:356:iot_schedule] <br>0-gluster4-vol-io-threads: READ scheduled as slow fop" repeated 285 <br>times between [2019-10-10 13:49:58.357922] and [2019-10-10 <br>13:50:07.837047]<br>[2019-10-10 13:50:07.837068] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: FLUSH <br>scheduled as normal fop<br>[2019-10-10 13:50:07.837165] D [MSGID: 0] <br>[io-threads.c:356:iot_schedule] 0-gluster4-vol-io-threads: READ <br>scheduled as slow fop<br><div><br></div><br> What does it mean exactly "READ scheduled as slow fop" ? Can I schedule <br>READs as normal or fast fop like the other operations ?<br><div><br></div> I'm using this gluster volume only for reading so I don't care about <br>writes right now.<br><div><br></div><br> Thanks.<br><div><br></div>________<br><div><br></div>Community Meeting Calendar:<br><div><br></div>APAC Schedule -<br>Every 2nd and 4th Tuesday at 11:30 AM IST<br>Bridge: https://bluejeans.com/118564314<br><div><br></div>NA/EMEA Schedule -<br>Every 1st and 3rd Tuesday at 01:00 PM EDT<br>Bridge: https://bluejeans.com/118564314<br><div><br></div>Gluster-users mailing list<br>Gluster-users@gluster.org<br>https://lists.gluster.org/mailman/listinfo/gluster-users<br></div><div><br></div></div></body></html>