[Gluster-users] Write operations failing on clients

[Alex]

Ben Turner <bturner at ...> writes:

> 
> ----- Original Message -----
> > From: "Alex" <alex.m at ...>
> > To: gluster-users at ...
> > Sent: Thursday, April 30, 2015 6:52:58 AM
> > Subject: Re: [Gluster-users] Write operations failing on clients
> > 
> > Okay, I did some digging. On the client there was many errors such as:
> > 
> > [2015-04-29 15:47:08.700174] W [client-rpc-fops.c:2774:client3_3_lookup_cbk]
> > 0-img-client-0: remote operation failed: Transport endpoint is not
> > connected. Path: /www/img/gallery/9722926_4130.jpg
> > (00000000-0000-0000-0000-000000000000)
> > [2015-04-29 15:47:08.700268] I
> > [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk]
> > 0-img-replicate-0: looking up /www/img/gallery/9722926_4130.jpg under
> > img-client-0 failed (Transport endpoint is not connected)
> > 
> > And at the same time on the cluster:
> > [2015-04-29 15:47:59.989897] W [client-rpc-fops.c:2774:client3_3_lookup_cbk]
> > 0-img-client-0: remote operation failed: Transport endpoint is not
> > connected. Path: /www/pdf/23096091-1722.pdf
> > (00000000-0000-0000-0000-000000000000)
> > [2015-04-29 15:47:59.989923] I
> > [afr-self-heal-entry.c:607:afr_sh_entry_expunge_entry_cbk]
> > 0-img-replicate-0: looking up /www/pdf/23096091-1722.pdf under img-client-0
> > failed (Transport endpoint is not connected)
> > 
> > 
> > What could it mean? Is there some kind of network error? BTW there was
> > nothing that indicated any network connectivity problems between nodes and
> > clients.
> 
> Hi Alex.  You are correct when you see the "Transport endpoint is not
connected." it usually means that the
> client is unable to access the server.  Check gluster v status and make
sure all your bricks are online.  Try
> to unmount / remount the client if you see everything is up.  If you still
cant access and everythin is online
> I would do some basic NW troubleshooting, make sure selinux is off on the
servers, and check
> /var/log/glusterfs/bricks for errors on each of the servers.  If you see
any error messages of the
> severity { M | A | C | E } pastebin them to me and I'll have a look.
> 
> -b
> 
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at ...
> > http://www.gluster.org/mailman/listinfo/gluster-users
> > 
> 

Hi Ben.

I suspect "endpoint not connected" was occurring when client tried to write
anything. In same time reads from that volume from other clients processed
normally.

Keep in mind that problem began to occur at ~14:20 and continues now. So I
parsed logs for errors on yesterday and today. Also there was a couple of
cluster daemon restarting while troubleshooting.

I collected logs from one brick. If you need logs from all bricks, just say.
client log: http://termbin.com/1012

server logs:
cli.log-20150430 http://termbin.com/ui7r
etc-glusterfs-glusterd.vol.log-20150430 http://termbin.com/tmof
glustershd.log-20150430 http://termbin.com/jz22
img-rebalance.log-20150430 http://termbin.com/y5zi
nfs.log http://termbin.com/3qsm
nfs.log-20150430 http://termbin.com/u8e7
var-gl-images.log-20150430 http://termbin.com/wny4

Thank you in advance!
Alex