[Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Ben Turner]

<re added gluster users, it looks like it was dropped from your email>

----- Original Message -----
> From: "Julio Guevara" <julioguevara150 at gmail.com>
> To: "Ben Turner" <bturner at redhat.com>
> Sent: Thursday, June 15, 2017 5:52:26 PM
> Subject: Re: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume
> 
> I stumble upon the problem.
> 
> We are using deep security agent (da_agent) as our main antivirus. When the
> antivirus gets activated it installs kernel modules:
>   redirfs
>   gsch
> 
> Apparently when this modules are present and loaded to the kernel, I see
> all the issues that i have described here.
> Once I uninstall the agent and reboot the system (To make sure modules are
> unloaded) glusterfs works without any issue.
> This is the sofware version that i'm using if it is useful for anybody:
> 
>   CentOS 6.8
>   kernel                2.6.32-696.3.1.el6
>   ds_agent           9.6.2-7723.el6 tried with ds_agent 9.6.2-7888.el6
>  same issue.
>   glusterfs-server  3.8.12-1.el6
> 
> @Ben the tail I sent before includes both server and client logs, even
> bricks.

Hmm, maybe the security SW is killing / interfering some how with the gluster stack?  Do you know the expected behavior of the antivirus when is sees binaries and / or behavior it doesn't recognize?  Maybe FUSE being in user space is tripping it up?  Is there any way to configure the anitvirus to white list / not interfere with the components of the gluster stack?

-b


> 
> Thanks
> Julio Guevara
> 
> On Wed, Jun 14, 2017 at 11:11 PM, Ben Turner <bturner at redhat.com> wrote:
> 
> > ----- Original Message -----
> > > From: "Julio Guevara" <julioguevara150 at gmail.com>
> > > To: gluster-users at gluster.org
> > > Sent: Tuesday, June 13, 2017 4:43:06 PM
> > > Subject: [Gluster-users] Transport Endpoint Not connected while running
> >      sysbench on Gluster Volume
> > >
> > > I'm having a hard time trying to get a gluster volume up and running. I
> > have
> > > setup other gluster volumes on other systems without much problems but
> > this
> > > one is killing me.
> > >
> > > The gluster vol was created with the command:
> > > gluster volume create mariadb_gluster_volume
> > > laeft-dccdb01p:/export/mariadb/brick
> > >
> > > I had to lower frame-timeout since the system would become unresponsive
> > until
> > > the frame failed by timeout:
> > > gluster volume set mariadb_gluster_volume networking.frame-timeout 5
> > >
> > > running gluster version: glusterfs 3.8.12
> > >
> > > The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> > > --file-num=64 prepare
> > >
> > > sysbench version: sysbench 0.4.12-5.el6
> > >
> > > kernel version: 2.6.32-696.1.1.el6
> > >
> > > centos: 6.8
> > >
> > > Issue: Whenever I run the sysbench over the mount /var/lib/mysql_backups
> > I
> > > get the error that is shown on the log output.
> > >
> > > It is a constant issue, I can reproduce it when I start increasing the
> > > --file-num for sysbench above 3.
> >
> > It looks like you may be seeing a crash.  If you look at /var/log/messages
> > on all of the clients / servers do you see any crashes / seg faults / ABRT
> > messages in the log?  If so can you open a BZ with the core / other info
> > here?  Here is an example of a crash on one of the bricks:
> >
> > http://lists.gluster.org/pipermail/gluster-users/2016-February/025460.html
> >
> > My guess is something is happening client sidesince we don't see anything
> > in the server logs, check the client mount
> > log(/var/log/glusterfs/<mount-point>.log
> > and the messages file on your client.  Also check messages on the servers.
> > If you see anything shoot us out the info and lets get a BZ open, if not
> > maybe someone else on the list has some other ideas.
> >
> > -b
> >
> > >
> > >
> > >
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://lists.gluster.org/mailman/listinfo/gluster-users
> >
>