<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi,<br>
<br>
I found the coredump file, but it's a 15Mo file (zipped), I can't
post it on this mailling list.<br>
<br>
Here is some parts of the repport :<br>
<br>
<blockquote type="cite">ProblemType: Crash<br>
Architecture: amd64<br>
Date: Sun Jun 26 11:27:44 2016<br>
DistroRelease: Ubuntu 14.04<br>
ExecutablePath: /usr/sbin/glusterfsd<br>
ExecutableTimestamp: 1460982898<br>
ProcCmdline: /usr/sbin/glusterfsd -s nfs05 --volfile-id
cdn.nfs05.srv-cdn -p
/var/lib/glusterd/vols/cdn/run/nfs05-srv-cdn.pid -S
/var/run/gluster/d52ac3e6c0a3fa316a9e8360976f3af5.socket
--brick-name /srv/cdn -l /var/log/glusterfs/bricks/srv-cdn.log
--xlator-option
*-posix.glusterd-uuid=6af63b78-a3da-459d-a909-c010e6c9072c
--brick-port 49155 --xlator-option cdn-server.listen-port=49155<br>
ProcCwd: /<br>
ProcEnviron:<br>
PATH=(custom, no user)<br>
TERM=linux<br>
ProcMaps:<br>
7f25f18d9000-7f25f18da000 ---p 00000000 00:00 0<br>
7f25f18da000-7f25f19da000 rw-p 00000000 00:00
0 [stack:849]<br>
7f25f19da000-7f25f19db000 ---p 00000000 00:00 0<br>
</blockquote>
...<br>
<blockquote type="cite">ProcStatus:<br>
Name: glusterfsd<br>
State: D (disk sleep)<br>
Tgid: 7879<br>
Ngid: 0<br>
Pid: 7879<br>
PPid: 1<br>
TracerPid: 0<br>
Uid: 0 0 0 0<br>
Gid: 0 0 0 0<br>
FDSize: 64<br>
Groups: 0<br>
VmPeak: 878404 kB<br>
VmSize: 878404 kB<br>
VmLck: 0 kB<br>
VmPin: 0 kB<br>
VmHWM: 96104 kB<br>
VmRSS: 90652 kB<br>
VmData: 792012 kB<br>
VmStk: 276 kB<br>
VmExe: 84 kB<br>
VmLib: 7716 kB<br>
VmPTE: 700 kB<br>
VmSwap: 20688 kB<br>
Threads: 22<br>
SigQ: 0/30034<br>
SigPnd: 0000000000000000<br>
ShdPnd: 0000000000000000<br>
SigBlk: 0000000000004a01<br>
SigIgn: 0000000000001000<br>
SigCgt: 00000001800000fa<br>
CapInh: 0000000000000000<br>
CapPrm: 0000001fffffffff<br>
CapEff: 0000001fffffffff<br>
CapBnd: 0000001fffffffff<br>
Seccomp: 0<br>
Cpus_allowed: 7fff<br>
Cpus_allowed_list: 0-14<br>
Mems_allowed: 00000000,00000001<br>
Mems_allowed_list: 0<br>
voluntary_ctxt_switches: 3<br>
nonvoluntary_ctxt_switches: 1<br>
Signal: 11<br>
Uname: Linux 3.13.0-44-generic x86_64<br>
UserGroups:<br>
CoreDump: base64<br>
</blockquote>
...<br>
<br>
<br>
Yann<br>
<br>
<div class="moz-cite-prefix">Le 28/06/2016 09:31, Anoop C S a
écrit :<br>
</div>
<blockquote cite="mid:1467099100.6046.20.camel@redhat.com"
type="cite">
<pre wrap="">On Mon, 2016-06-27 at 15:05 +0200, Yann LEMARIE wrote:
</pre>
<blockquote type="cite">
<pre wrap=""> @Anoop,
Where can I find the coredump file ?
</pre>
</blockquote>
<pre wrap="">
You will get hints about the crash from entries inside
/var/log/messages(for example pid of the process, location of coredump
etc).
</pre>
<blockquote type="cite">
<pre wrap="">The crash occurs 2 times last 7 days, each time a sunday morning with
no reason, no increase of traffic or something like this, the volume
was mounted since 15 days.
The bricks are used as a CDN like, distributting small images and css
files with a nginx https service (with a load balancer and 2 EC2), on
a sunday morning there is not a lot of activity ...
</pre>
</blockquote>
<pre wrap="">
>From the very minimal back trace that we have from brick logs I would
assume that a truncate operation was being handled by trash translator
and it crashed.
</pre>
<blockquote type="cite">
<pre wrap="">Volume infos:
</pre>
<blockquote type="cite">
<pre wrap="">root@nfs05 /var/log/glusterfs # gluster volume info cdn
Volume Name: cdn
Type: Replicate
Volume ID: c53b9bae-5e12-4f13-8217-53d8c96c302c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: nfs05:/srv/cdn
Brick2: nfs06:/srv/cdn
Options Reconfigured:
performance.readdir-ahead: on
features.trash: on
features.trash-max-filesize: 20MB
</pre>
</blockquote>
<pre wrap="">
I don't know if there is a link with this crash problem, but I have
another problem with my 2 servers that make GluserFS's clients
disconnected (from another volume) :
</pre>
<blockquote type="cite">
<pre wrap="">Jun 24 02:28:04 nfs05 kernel: [2039468.818617] xen_netfront:
xennet: skb rides the rocket: 19 slots
Jun 24 02:28:11 nfs05 kernel: [2039475.744086] net_ratelimit: 66
callbacks suppressed
</pre>
</blockquote>
<pre wrap=""> It seem to be a network interface problem :
<a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811">https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1317811</a>
Yann
Le 27/06/2016 12:59, Anoop C S a écrit :
</pre>
<blockquote type="cite">
<pre wrap="">On Mon, 2016-06-27 at 09:47 +0200, Yann LEMARIE wrote:
</pre>
<blockquote type="cite">
<pre wrap="">Hi,
I'm using GlusterFS since many years and never see this problem,
but
this is the second time in one week ...
I have 3 volumes with 2 bricks and 1 volume crash with no reason,
</pre>
</blockquote>
<pre wrap="">Did you observe the crash while mounting the volume? Or can you be
more
specific on what were you doing just before you saw the crash? Can
you
please share the output of `gluster volume info <VOLNAME>`?
</pre>
<blockquote type="cite">
<pre wrap=""> I just have to stop/start the volume to make it up again.
The only logs I can find are in syslog :
</pre>
<blockquote type="cite">
<pre wrap="">Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames:
Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10)
Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset:
git://git.gluster.com/glusterfs.git
Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11
Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash:
Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44
Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details:
Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1
Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string: glusterfs
3.7.11
Jun 26 11:27:44 nfs05 srv-cdn[7879]: ---------
Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames:
Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10)
Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset:
git://git.gluster.com/glusterfs.git
Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11
Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash:
Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44
Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details:
Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1
Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string: glusterfs
3.7.11
Jun 26 11:27:44 nfs06 srv-cdn[1787]: ---------
</pre>
</blockquote>
<pre wrap="">
Thanks for your help
Regards
--
Yann Lemarié
iRaiser - Support Technique
<a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a>
_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
<pre wrap="">
--
Yann Lemarié
iRaiser - Support Technique
<a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a>
_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="http://www.gluster.org/mailman/listinfo/gluster-users">http://www.gluster.org/mailman/listinfo/gluster-users</a>
</pre>
</blockquote>
</blockquote>
<br>
<div class="moz-signature">-- <br>
<div dir="ltr">
<div
style="margin-bottom:0px;margin-left:0px;padding-bottom:5px">
<div style="overflow:hidden">
<div dir="ltr">
<div dir="ltr" style="font-size:13px">Yann Lemarié</div>
<div dir="ltr" style="font-size:13px"><span
style="font-size:x-small">iRaiser - Support Technique</span></div>
<div dir="ltr" style="background-color:white
paddind-left:5px;margin-top:5px;width:145px;margin-bottom:5px;">
<a href="http://www.iraiser.eu" target="_blank"> <img
title="" alt="iRaiser Logotype"
src="cid:part1.08020404.07020709@iraiser.eu"
height="22" width="102"> </a> </div>
<div dir="ltr" style="font-size:13px"><span
style="font-size:x-small"><a
href="mailto:ylemarie@iraiser.eu" target="_blank"><a class="moz-txt-link-abbreviated" href="mailto:ylemarie@iraiser.eu">ylemarie@iraiser.eu</a></a></span></div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>