[Gluster-users] help, all latency comes from FINODELK

Wed Feb 5 11:34:45 UTC 2014

sorry, I can't give the backtrace.
I got a coredump file for segmentfault, when I used gdb to get backtrace, I
just got lots of question marks (no debuginfo package installed)

gdb -c core.xxxx

[New Thread 11858]
[New Thread 11859]
[New Thread 11882]
[New Thread 11861]
[New Thread 11896]
[New Thread 11881]
[New Thread 14312]
[New Thread 11894]
[New Thread 14315]
[New Thread 14314]
[New Thread 11860]
[New Thread 11883]
[New Thread 14318]
[New Thread 11893]
[New Thread 12069]
[New Thread 14313]
[New Thread 14321]
[New Thread 14319]
[New Thread 14320]
[New Thread 11857]
[New Thread 12070]
[New Thread 14317]
[New Thread 14316]
Core was generated by `/usr/sbin/glusterfsd -s localhost --volfile-id
sh-ugc1-mams.shdx.miaoyan.cluste'.
Program terminated with signal 11, Segmentation fault.
#0  0x00007f4e824d10b4 in ?? ()
(gdb) bt
#0  0x00007f4e824d10b4 in ?? ()
#1  0x0000003000000018 in ?? ()
#2  0x00000000024a8510 in ?? ()
#3  0x0000000000000000 in ?? ()

(gdb) thread apply all bt full

Thread 23 (Thread 14316):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 22 (Thread 14317):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 21 (Thread 12070):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 20 (Thread 11857):
#0  0x00000035ef2e7c73 in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 19 (Thread 14320):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 18 (Thread 14319):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 17 (Thread 14321):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 16 (Thread 14313):
#0  0x00000035ef60b7bb in ?? ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 15 (Thread 12069):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 14 (Thread 11893):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 13 (Thread 14318):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 12 (Thread 11883):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 11 (Thread 11860):
#0  0x00000035ef60b43c in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 10 (Thread 14314):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 9 (Thread 14315):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.
---Type <return> to continue, or q <return> to quit---

Thread 8 (Thread 11894):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 7 (Thread 14312):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 6 (Thread 11881):
#0  0x00000035ef60b43c in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 5 (Thread 11896):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 4 (Thread 11861):
#0  0x00000035ef60ed2d in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 3 (Thread 11882):
#0  0x00000035ef60b7bb in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 2 (Thread 11859):
#0  0x00000035ef60b43c in ?? ()
No symbol table info available.
#1  0x0000000000000000 in ?? ()
No symbol table info available.

Thread 1 (Thread 11858):
#0  0x00007f4e824d10b4 in ?? ()
---Type <return> to continue, or q <return> to quit---
No symbol table info available.
#1  0x0000003000000018 in ?? ()
No symbol table info available.
#2  0x00000000024a8510 in ?? ()
No symbol table info available.
#3  0x0000000000000000 in ?? ()
No symbol table info available.

On Thu, Jan 30, 2014 at 3:14 PM, Pranith Kumar Karampuri <
pkarampu at redhat.com> wrote:

> Could you give us the back trace
>
> Pranith
> ----- Original Message -----
> > From: "Mingfan Lu" <mingfan.lu at gmail.com>
> > To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > Cc: "haiwei.xie-soulinfo" <haiwei.xie at soulinfo.com>, "
> Gluster-users at gluster.org List" <gluster-users at gluster.org>
> > Sent: Thursday, January 30, 2014 12:43:44 PM
> > Subject: Re: [Gluster-users] help, all latency comes from FINODELK
> >
> > When I triedto execute the statedump, the brick server of the BAD node
> > crashed.
> >
> >
> >
> > On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri <
> > pkarampu at redhat.com> wrote:
> >
> > > Could you take statedump of bricks and get that information please.
> > > You can use
> > >
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html
> > > for taking statedumps.
> > >
> > > Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276that
> > > you raised for? If yes could you attach the files to that bug so that
> I get
> > > the notifications immediately.
> > >
> > > Pranith
> > > ----- Original Message -----
> > > > From: "Mingfan Lu" <mingfan.lu at gmail.com>
> > > > To: "haiwei.xie-soulinfo" <haiwei.xie at soulinfo.com>
> > > > Cc: "Gluster-users at gluster.org List" <gluster-users at gluster.org>
> > > > Sent: Tuesday, January 28, 2014 7:44:14 AM
> > > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK
> > > >
> > > > About 200+ clients
> > > >
> > > > How to print print lock info in bricks, print lock requst info in
> afr &
> > > dht?
> > > > thanks.
> > > >
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo <
> > > > haiwei.xie at soulinfo.com > wrote:
> > > >
> > > >
> > > >
> > > > hi,
> > > > looks FINODELK deak lock. how many clients, and nfs or fuse?
> > > > Maybe the best way is to print lock info in bricks, print lock requst
> > > info in
> > > > afr & dht.
> > > >
> > > > > all my clients hang when they creating dir
> > > > >
> > > > >
> > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu <
> mingfan.lu at gmail.com >
> > > > > wrote:
> > > > >
> > > > > > I ran glusterfs volume profile my_volume info, I got some thing
> > > likes:
> > > > > >
> > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET
> > > > > > 0.00 0.00 us 0.00 us 0.00 us 185
> > > > > > RELEASE
> > > > > > 0.00 0.00 us 0.00 us 0.00 us 11
> > > > > > RELEASEDIR
> > > > > > 0.00 66.50 us 54.00 us 79.00 us 2
> > > > > > SETATTR
> > > > > > 0.00 44.83 us 25.00 us 94.00 us 6
> > > > > > READDIR
> > > > > > 0.00 57.55 us 28.00 us 95.00 us 11
> > > > > > OPENDIR
> > > > > > 0.00 325.00 us 279.00 us 371.00 us 2
> > > > > > RENAME
> > > > > > 0.00 147.80 us 84.00 us 206.00 us 5
> > > > > > LINK
> > > > > > 0.00 389.50 us 55.00 us 724.00 us 2
> > > > > > READDIRP
> > > > > > 0.00 164.25 us 69.00 us 287.00 us 8
> > > > > > UNLINK
> > > > > > 0.00 37.46 us 18.00 us 87.00 us 50
> > > > > > FSTAT
> > > > > > 0.00 70.32 us 29.00 us 210.00 us 37
> > > > > > GETXATTR
> > > > > > 0.00 77.75 us 42.00 us 216.00 us 55
> > > > > > SETXATTR
> > > > > > 0.00 36.39 us 11.00 us 147.00 us 119
> > > > > > FLUSH
> > > > > > 0.00 51.22 us 24.00 us 139.00 us 275
> > > > > > OPEN
> > > > > > 0.00 180.14 us 84.00 us 457.00 us 96
> > > > > > XATTROP
> > > > > > 0.00 3847.20 us 231.00 us 18218.00 us 5
> > > > > > MKNOD
> > > > > > 0.00 70.08 us 15.00 us 6539.00 us 342
> > > > > > ENTRYLK
> > > > > > 0.00 10338.86 us 184.00 us 34813.00 us 7
> > > > > > CREATE
> > > > > > 0.00 896.65 us 12.00 us 83103.00 us 235
> > > > > > INODELK
> > > > > > 0.00 187.86 us 50.00 us 668.00 us 1526
> > > > > > WRITE
> > > > > > 0.00 40.66 us 13.00 us 400.00 us 10400
> > > > > > STATFS
> > > > > > 0.00 313.13 us 66.00 us 2142.00 us 2049
> > > > > > FXATTROP
> > > > > > 0.00 2794.97 us 26.00 us 78048.00 us 295
> > > > > > READ
> > > > > > 0.00 24469.82 us 206.00 us 176157.00 us 34
> > > > > > MKDIR
> > > > > > 0.00 40.49 us 13.00 us 507.00 us 21420
> > > > > > STAT
> > > > > > 0.00 190.90 us 40.00 us 330032.00 us 45820
> > > > > > LOOKUP
> > > > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994
> > > > > > FINODELK
> > > > > >
> > > > > > what happend?
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140205/01d2db76/attachment.html>