[Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume

[Ben Turner]

----- Original Message -----
> From: "Ben Turner" <bturner at redhat.com>
> To: "Julio Guevara" <julioguevara150 at gmail.com>
> Cc: gluster-users at gluster.org
> Sent: Thursday, June 15, 2017 6:10:58 PM
> Subject: Re: [Gluster-users] Transport Endpoint Not connected while running sysbench on Gluster Volume
> 
> <re added gluster users, it looks like it was dropped from your email>
> 
> ----- Original Message -----
> > From: "Julio Guevara" <julioguevara150 at gmail.com>
> > To: "Ben Turner" <bturner at redhat.com>
> > Sent: Thursday, June 15, 2017 5:52:26 PM
> > Subject: Re: [Gluster-users] Transport Endpoint Not connected while running
> > sysbench on Gluster Volume
> > 
> > I stumble upon the problem.
> > 
> > We are using deep security agent (da_agent) as our main antivirus. When the
> > antivirus gets activated it installs kernel modules:
> >   redirfs
> >   gsch
> > 
> > Apparently when this modules are present and loaded to the kernel, I see
> > all the issues that i have described here.
> > Once I uninstall the agent and reboot the system (To make sure modules are
> > unloaded) glusterfs works without any issue.
> > This is the sofware version that i'm using if it is useful for anybody:
> > 
> >   CentOS 6.8
> >   kernel                2.6.32-696.3.1.el6
> >   ds_agent           9.6.2-7723.el6 tried with ds_agent 9.6.2-7888.el6
> >  same issue.
> >   glusterfs-server  3.8.12-1.el6
> > 
> > @Ben the tail I sent before includes both server and client logs, even
> > bricks.
> 
> Hmm, maybe the security SW is killing / interfering some how with the gluster
> stack?  Do you know the expected behavior of the antivirus when is sees
> binaries and / or behavior it doesn't recognize?  Maybe FUSE being in user
> space is tripping it up?  Is there any way to configure the anitvirus to
> white list / not interfere with the components of the gluster stack?

I just did a quick google and saw:

http://docs.trendmicro.com/all/ent/ds/v9.5_sp1/en-us/DS_Agent-Linux_9.5_SP1_readme.txt

   - Anti-Malware is unable to scan fuse-based file-system if the 
     mount owner is not root, and the mount does not allow other users to 
     access. [26265]

So it would appear that there have been some issues with FUSE based file systems.  It may be worth reaching out to the vendor if you have support and see if there are any known issues with FUSE based systems.  In the meantime you may want to try NFS if you NEED the antivirus else you could leave it disabled until you get the issue sorted.

-b


> 
> -b
> 
> 
> > 
> > Thanks
> > Julio Guevara
> > 
> > On Wed, Jun 14, 2017 at 11:11 PM, Ben Turner <bturner at redhat.com> wrote:
> > 
> > > ----- Original Message -----
> > > > From: "Julio Guevara" <julioguevara150 at gmail.com>
> > > > To: gluster-users at gluster.org
> > > > Sent: Tuesday, June 13, 2017 4:43:06 PM
> > > > Subject: [Gluster-users] Transport Endpoint Not connected while running
> > >      sysbench on Gluster Volume
> > > >
> > > > I'm having a hard time trying to get a gluster volume up and running. I
> > > have
> > > > setup other gluster volumes on other systems without much problems but
> > > this
> > > > one is killing me.
> > > >
> > > > The gluster vol was created with the command:
> > > > gluster volume create mariadb_gluster_volume
> > > > laeft-dccdb01p:/export/mariadb/brick
> > > >
> > > > I had to lower frame-timeout since the system would become unresponsive
> > > until
> > > > the frame failed by timeout:
> > > > gluster volume set mariadb_gluster_volume networking.frame-timeout 5
> > > >
> > > > running gluster version: glusterfs 3.8.12
> > > >
> > > > The workload i'm using is: sysbench --test=fileio --file-total-size=4G
> > > > --file-num=64 prepare
> > > >
> > > > sysbench version: sysbench 0.4.12-5.el6
> > > >
> > > > kernel version: 2.6.32-696.1.1.el6
> > > >
> > > > centos: 6.8
> > > >
> > > > Issue: Whenever I run the sysbench over the mount
> > > > /var/lib/mysql_backups
> > > I
> > > > get the error that is shown on the log output.
> > > >
> > > > It is a constant issue, I can reproduce it when I start increasing the
> > > > --file-num for sysbench above 3.
> > >
> > > It looks like you may be seeing a crash.  If you look at
> > > /var/log/messages
> > > on all of the clients / servers do you see any crashes / seg faults /
> > > ABRT
> > > messages in the log?  If so can you open a BZ with the core / other info
> > > here?  Here is an example of a crash on one of the bricks:
> > >
> > > http://lists.gluster.org/pipermail/gluster-users/2016-February/025460.html
> > >
> > > My guess is something is happening client sidesince we don't see anything
> > > in the server logs, check the client mount
> > > log(/var/log/glusterfs/<mount-point>.log
> > > and the messages file on your client.  Also check messages on the
> > > servers.
> > > If you see anything shoot us out the info and lets get a BZ open, if not
> > > maybe someone else on the list has some other ideas.
> > >
> > > -b
> > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://lists.gluster.org/mailman/listinfo/gluster-users
> > >
> > 
>