[Gluster-users] help, all latency comes from FINODELK

Fri Feb 7 06:32:38 UTC 2014

Today， when I ran statedump, node22 still crashed and I got the pending
frames

[2014-02-07 13:32:52.356386] E
[marker-quota-helper.c:230:mq_dict_set_contribution]
(-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/debug/io-stats.so(io_stats_lookup+0x13e)
[0x7f81ca048a3e]
(-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/features/marker.so(marker_lookup+0x300)
[0x7f81ca25e170]
(-->/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/features/marker.so(mq_req_xattr+0x3c)
[0x7f81ca2676ec]))) 0-marker: invalid argument: loc->parent
pending frames:

patchset: git://git.gluster.com/glusterfs.git
signal received: 11
time of crash: 2014-02-07 13:32:52
configuration details:
argp 1
backtrace 1
dlfcn 1
fdatasync 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.3.0.5rhs_iqiyi_7
/lib64/libc.so.6[0x35ef232920]
/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/protocol/server.so(ltable_dump+0x54)[0x7f81c9e130b4]
/usr/lib64/glusterfs/3.3.0.5rhs_iqiyi_7/xlator/protocol/server.so(server_inode+0xc7)[0x7f81c9e13597]
/usr/lib64/libglusterfs.so.0(gf_proc_dump_xlator_info+0x112)[0x30630443f2]
/usr/lib64/libglusterfs.so.0(gf_proc_dump_info+0x4b2)[0x3063044bc2]
/usr/sbin/glusterfsd(glusterfs_sigwaiter+0xd2)[0x405cc2]
/lib64/libpthread.so.0[0x35ef607851]
/lib64/libc.so.6(clone+0x6d)[0x35ef2e767d]

On Thu, Jan 30, 2014 at 3:14 PM, Pranith Kumar Karampuri <
pkarampu at redhat.com> wrote:

> Could you give us the back trace
>
> Pranith
> ----- Original Message -----
> > From: "Mingfan Lu" <mingfan.lu at gmail.com>
> > To: "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> > Cc: "haiwei.xie-soulinfo" <haiwei.xie at soulinfo.com>, "
> Gluster-users at gluster.org List" <gluster-users at gluster.org>
> > Sent: Thursday, January 30, 2014 12:43:44 PM
> > Subject: Re: [Gluster-users] help, all latency comes from FINODELK
> >
> > When I triedto execute the statedump, the brick server of the BAD node
> > crashed.
> >
> >
> >
> > On Wed, Jan 29, 2014 at 8:13 PM, Pranith Kumar Karampuri <
> > pkarampu at redhat.com> wrote:
> >
> > > Could you take statedump of bricks and get that information please.
> > > You can use
> > >
> https://access.redhat.com/site/documentation/en-US/Red_Hat_Storage/2.0/html/Administration_Guide/sect-User_Guide-Monitor_Workload-Performing_Statedump.html
> > > for taking statedumps.
> > >
> > > Is this same as https://bugzilla.redhat.com/show_bug.cgi?id=1056276that
> > > you raised for? If yes could you attach the files to that bug so that
> I get
> > > the notifications immediately.
> > >
> > > Pranith
> > > ----- Original Message -----
> > > > From: "Mingfan Lu" <mingfan.lu at gmail.com>
> > > > To: "haiwei.xie-soulinfo" <haiwei.xie at soulinfo.com>
> > > > Cc: "Gluster-users at gluster.org List" <gluster-users at gluster.org>
> > > > Sent: Tuesday, January 28, 2014 7:44:14 AM
> > > > Subject: Re: [Gluster-users] help, all latency comes from FINODELK
> > > >
> > > > About 200+ clients
> > > >
> > > > How to print print lock info in bricks, print lock requst info in
> afr &
> > > dht?
> > > > thanks.
> > > >
> > > >
> > > >
> > > > On Tue, Jan 28, 2014 at 9:51 AM, haiwei.xie-soulinfo <
> > > > haiwei.xie at soulinfo.com > wrote:
> > > >
> > > >
> > > >
> > > > hi,
> > > > looks FINODELK deak lock. how many clients, and nfs or fuse?
> > > > Maybe the best way is to print lock info in bricks, print lock requst
> > > info in
> > > > afr & dht.
> > > >
> > > > > all my clients hang when they creating dir
> > > > >
> > > > >
> > > > > On Mon, Jan 27, 2014 at 11:33 PM, Mingfan Lu <
> mingfan.lu at gmail.com >
> > > > > wrote:
> > > > >
> > > > > > I ran glusterfs volume profile my_volume info, I got some thing
> > > likes:
> > > > > >
> > > > > > 0.00 0.00 us 0.00 us 0.00 us 30 FORGET
> > > > > > 0.00 0.00 us 0.00 us 0.00 us 185
> > > > > > RELEASE
> > > > > > 0.00 0.00 us 0.00 us 0.00 us 11
> > > > > > RELEASEDIR
> > > > > > 0.00 66.50 us 54.00 us 79.00 us 2
> > > > > > SETATTR
> > > > > > 0.00 44.83 us 25.00 us 94.00 us 6
> > > > > > READDIR
> > > > > > 0.00 57.55 us 28.00 us 95.00 us 11
> > > > > > OPENDIR
> > > > > > 0.00 325.00 us 279.00 us 371.00 us 2
> > > > > > RENAME
> > > > > > 0.00 147.80 us 84.00 us 206.00 us 5
> > > > > > LINK
> > > > > > 0.00 389.50 us 55.00 us 724.00 us 2
> > > > > > READDIRP
> > > > > > 0.00 164.25 us 69.00 us 287.00 us 8
> > > > > > UNLINK
> > > > > > 0.00 37.46 us 18.00 us 87.00 us 50
> > > > > > FSTAT
> > > > > > 0.00 70.32 us 29.00 us 210.00 us 37
> > > > > > GETXATTR
> > > > > > 0.00 77.75 us 42.00 us 216.00 us 55
> > > > > > SETXATTR
> > > > > > 0.00 36.39 us 11.00 us 147.00 us 119
> > > > > > FLUSH
> > > > > > 0.00 51.22 us 24.00 us 139.00 us 275
> > > > > > OPEN
> > > > > > 0.00 180.14 us 84.00 us 457.00 us 96
> > > > > > XATTROP
> > > > > > 0.00 3847.20 us 231.00 us 18218.00 us 5
> > > > > > MKNOD
> > > > > > 0.00 70.08 us 15.00 us 6539.00 us 342
> > > > > > ENTRYLK
> > > > > > 0.00 10338.86 us 184.00 us 34813.00 us 7
> > > > > > CREATE
> > > > > > 0.00 896.65 us 12.00 us 83103.00 us 235
> > > > > > INODELK
> > > > > > 0.00 187.86 us 50.00 us 668.00 us 1526
> > > > > > WRITE
> > > > > > 0.00 40.66 us 13.00 us 400.00 us 10400
> > > > > > STATFS
> > > > > > 0.00 313.13 us 66.00 us 2142.00 us 2049
> > > > > > FXATTROP
> > > > > > 0.00 2794.97 us 26.00 us 78048.00 us 295
> > > > > > READ
> > > > > > 0.00 24469.82 us 206.00 us 176157.00 us 34
> > > > > > MKDIR
> > > > > > 0.00 40.49 us 13.00 us 507.00 us 21420
> > > > > > STAT
> > > > > > 0.00 190.90 us 40.00 us 330032.00 us 45820
> > > > > > LOOKUP
> > > > > > 100.00 72004815.62 us 8.00 us 5783044563.00 us 3994
> > > > > > FINODELK
> > > > > >
> > > > > > what happend?
> > > > > >
> > > >
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140207/5bf08f32/attachment.html>