[Gluster-users] GlusterFS 3.7.11 crash issue

[Yann LEMARIE]

  @Anoop,

Where can I find the coredump file ?

The crash occurs 2 times last 7 days, each time a sunday morning with no 
reason, no increase of traffic or something like this, the volume was 
mounted since 15 days.

The bricks are used as a CDN like, distributting small images and css 
files with a nginx https service (with a load balancer and 2 EC2), on a 
sunday morning there is not a lot of activity ...

Volume infos:
> root at nfs05 /var/log/glusterfs # gluster volume info cdn
>
> Volume Name: cdn
> Type: Replicate
> Volume ID: c53b9bae-5e12-4f13-8217-53d8c96c302c
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: nfs05:/srv/cdn
> Brick2: nfs06:/srv/cdn
> Options Reconfigured:
> performance.readdir-ahead: on
> features.trash: on
> features.trash-max-filesize: 20MB

I don't know if there is a link with this crash problem, but I have 
another problem with my 2 servers that make GluserFS's clients 
disconnected (from another volume) :
> Jun 24 02:28:04 nfs05 kernel: [2039468.818617] xen_netfront: xennet: 
> skb rides the rocket: 19 slots
> Jun 24 02:28:11 nfs05 kernel: [2039475.744086] net_ratelimit: 66 
> callbacks suppressed
It seem to be a network interface problem :
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811

Yann

Le 27/06/2016 12:59, Anoop C S a écrit :
> On Mon, 2016-06-27 at 09:47 +0200, Yann LEMARIE wrote:
>> Hi,
>>
>> I'm using GlusterFS since many years and never see this problem, but
>> this is the second time in one week ...
>>
>> I have 3 volumes with 2 bricks and 1 volume crash with no reason,
> Did you observe the crash while mounting the volume? Or can you be more
> specific on what were you doing just before you saw the crash? Can you
> please share the output of `gluster volume info <VOLNAME>`?
>
>>   I just have to stop/start the volume to make it up again.
>> The only logs I can find are in syslog :
>>
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames:
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10)
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset:
>>> git://git.gluster.com/glusterfs.git
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash:
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details:
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: glusterfs
>>> 3.7.11
>>> Jun 26 11:27:44 nfs05 srv-cdn[7879]: ---------
>>>
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames:
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10)
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset:
>>> git://git.gluster.com/glusterfs.git
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash:
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details:
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: glusterfs
>>> 3.7.11
>>> Jun 26 11:27:44 nfs06 srv-cdn[1787]: ---------
>>>
>>   
>> Thanks for your help
>>
>>
>> Regards
>> -- 
>> Yann Lemarié
>> iRaiser - Support Technique
>>   
>> ylemarie at iraiser.eu
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Yann Lemarié
iRaiser - Support Technique
iRaiser Logotype <http://www.iraiser.eu>
ylemarie at iraiser.eu


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160627/c700446c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 1842 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160627/c700446c/attachment.png>