[Gluster-users] GlusterFS 3.7.11 crash issue

[Anoop C S]

On Mon, 2016-06-27 at 15:05 +0200, Yann LEMARIE wrote:
>  @Anoop,
> 
> Where can I find the coredump file ?
> 

You will get hints about the crash from entries inside
/var/log/messages(for example pid of the process, location of coredump
etc). 

> The crash occurs 2 times last 7 days, each time a sunday morning with
> no reason, no increase of traffic or something like this, the volume
> was mounted since 15 days.
> 
> The bricks are used as a CDN like, distributting small images and css
> files with a nginx https service (with a load balancer and 2 EC2), on
> a sunday morning there is not a lot of activity ...
> 

>From the very minimal back trace that we have from brick logs I would
assume that a truncate operation was being handled by trash translator
and it crashed.

> Volume infos: 
> > root at nfs05 /var/log/glusterfs # gluster volume info cdn
> >  
> > Volume Name: cdn
> > Type: Replicate
> > Volume ID: c53b9bae-5e12-4f13-8217-53d8c96c302c
> > Status: Started
> > Number of Bricks: 1 x 2 = 2
> > Transport-type: tcp
> > Bricks:
> > Brick1: nfs05:/srv/cdn
> > Brick2: nfs06:/srv/cdn
> > Options Reconfigured:
> > performance.readdir-ahead: on
> > features.trash: on
> > features.trash-max-filesize: 20MB
>  
> I don't know if there is a link with this crash problem, but I have
> another problem with my 2 servers that make GluserFS's clients
> disconnected (from another volume) :
> > Jun 24 02:28:04 nfs05 kernel: [2039468.818617] xen_netfront:
> > xennet: skb rides the rocket: 19 slots
> > Jun 24 02:28:11 nfs05 kernel: [2039475.744086] net_ratelimit: 66
> > callbacks suppressed
>  It seem to be a network interface problem :
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811
> 
> Yann
> 
> Le 27/06/2016 12:59, Anoop C S a écrit :
> > On Mon, 2016-06-27 at 09:47 +0200, Yann LEMARIE wrote:
> > > Hi,
> > > 
> > > I'm using GlusterFS since many years and never see this problem,
> > > but
> > > this is the second time in one week ...
> > > 
> > > I have 3 volumes with 2 bricks and 1 volume crash with no reason,
> > Did you observe the crash while mounting the volume? Or can you be
> > more
> > specific on what were you doing just before you saw the crash? Can
> > you
> > please share the output of `gluster volume info <VOLNAME>`?
> > 
> > >  I just have to stop/start the volume to make it up again.
> > > The only logs I can find are in syslog :
> > > 
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames:
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10)
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset:
> > > > git://git.gluster.com/glusterfs.git
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash:
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details:
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: glusterfs
> > > > 3.7.11
> > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: ---------
> > > > 
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames:
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10)
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset:
> > > > git://git.gluster.com/glusterfs.git
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash:
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details:
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: glusterfs
> > > > 3.7.11
> > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: ---------
> > > > 
> > >  
> > > Thanks for your help
> > > 
> > > 
> > > Regards
> > > -- 
> > > Yann Lemarié
> > > iRaiser - Support Technique
> > >  
> > > ylemarie at iraiser.eu
> > > _______________________________________________
> > > Gluster-users mailing list
> > > Gluster-users at gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-users
>  
> -- 
> Yann Lemarié
> iRaiser - Support Technique
>  
> ylemarie at iraiser.eu
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users