[Gluster-users] socket time out in fio benchmark

[shuau li]

Hi everyone,


    I am trying to run multipal jobs using fio benchamark in replica volume

with 3 bricks, but some hours later, warning message “W [socket.c:195:__socket_rwv]

0-tcp.ida-server: readv failed (Connection timed out)” appear in bricks logs, I think this

waring may due to high work loads, glusterfs with high work loads can not respond

socket timely. So I add codes in rpc/rpc-transport/socket/src/socket.c to expand timeout

threshold of socket, now the SO_RCVTIMEO is 180s, KEEP_ALIVE is 300s, then run

work load again, but it does not work.




My test enviroment is as follow:
   
    Three nodes work as gluster cluster, each nodes with 16GB memory, 8 core

3.3GHz cpu， two 10000baseT/full and one 1000baseT/full network cards, each nodes

use 16 * 2T raid5 disks working as brick. The glusterfs version is 3.3.1.

    I create a 1*3 replica volume use this three nodes, every node use fuse to mount

volume through a 10000baseT/full network card. At the sametime, every node use cifs to

mount fuse_mount_point through another 10000baseT/full card.

    Each node run two fio scripts, read and write jobs. Both scripts do operation in


cifs_mount_point. scripts is as follows:

write_jobs:
while true
do
mkdir -p ${DIR}_write_${i}
/usr/local/bin/fio --ioengine=libaio --iodepth=256 --numjobs=100 --rw=write --bs=1k --size=1000m --directory=${DIR}_write_${i} --name=job01_1k_write >> ${DIR}_write_${i}/job01_1k_write.log
i=`expr $i + 1`
done




read jobs:
mkdir -p ${DIR}_read_${i}
/usr/local/bin/fio --ioengine=libaio --iodepth=256 --numjobs=100 --rw=read --bs=1k --size=1000m --directory=${DIR}_read_${i} --name=job01_1k_read >> ${DIR}_read_${i}/job01_1k_read.log
i=`expr $i + 1`
done
 
    I change iodepth from 256 to 16, numjobs from 100 to 25, but it still does not

work. Is there anybody pay attention to this problem?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141111/871f3b8d/attachment.html>