[Gluster-users] GlusterFS 3.7.11 crash issue

Anoop C S anoopcs at redhat.com
Tue Jun 28 09:17:57 UTC 2016 

On Mon, 2016-06-27 at 13:54 +0530, Atin Mukherjee wrote:
> +Anoop, Jiffin
> 
> On Mon, Jun 27, 2016 at 1:50 PM, Yann LEMARIE <ylemarie at iraiser.eu>
> wrote:
> > Thanks Atin,
> > 
> > Here is the log with "backtrace"
> > 
> > > time of crash:
> > > 2016-06-26 09:27:44
> > > configuration details:
> > > argp 1
> > > backtrace 1
> > > dlfcn 1
> > > libpthread 1
> > > llistxattr 1
> > > setfsid 1
> > > spinlock 1
> > > epoll.h 1
> > > xattr.h 1
> > > st_atim.tv_nsec 1
> > > package-string: glusterfs 3.7.11
> > > /usr/lib/x86_64-linux-
> > > gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f2626e7a5e
> > > 2]
> > > /usr/lib/x86_64-linux-
> > > gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f2626e966cd]
> > > /lib/x86_64-linux-gnu/libc.so.6(+0x36cb0)[0x7f2626278cb0]
> > > /lib/x86_64-linux-gnu/libc.so.6(strlen+0x2a)[0x7f26262ca9da]
> > > /usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_stat
> > > _cbk+0x279)[0x7f2620c7a909]
> >  
> > I have lot of message about [File exist] or [No such file or
> > directory] within the .trash directory, is ther any link with the
> > crash ?
> > 
> > > [2016-06-26 09:27:37.965461] E [MSGID: 113027]
> > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of /srv/cdn/ failed
> > > [File exists]
> > > [2016-06-26 09:27:37.965479] E [MSGID: 113027]
> > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of
> > > /srv/cdn/.trashcan failed [File exists]
> > > [2016-06-26 09:27:37.965495] E [MSGID: 113027]
> > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of
> > > /srv/cdn/.trashcan/ failed [File exists]
> > > [2016-06-26 09:27:37.965512] E [MSGID: 113027]
> > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of
> > > /srv/cdn/.trashcan//medias failed [File exists]
> > > [2016-06-26 09:27:37.965570] E [MSGID: 113001]
> > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on
> > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr
> > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b failed [No such file or directory]
> > > [2016-06-26 09:27:37.965615] E [MSGID: 113001]
> > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on
> > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr
> > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive failed [No such file or
> > > directory]
> > > [2016-06-26 09:27:37.965659] E [MSGID: 113001]
> > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on
> > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr
> > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin failed [No such
> > > file or directory]
> > > [2016-06-26 09:27:37.965683] E [MSGID: 113027]
> > > [posix.c:1427:posix_mkdir] 0-cdn-posix: mkdir of
> > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr
> > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin       failed [File
> > > exists]
> > > [2016-06-26 09:27:37.965720] E [MSGID: 113001]
> > > [posix.c:1451:posix_mkdir] 0-cdn-posix: setting xattrs on
> > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr
> > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp failed [No such
> > > file or directory]
> > > [2016-06-26 09:27:37.965767] E [MSGID: 113001]
> > > [posix.c:2644:posix_create] 0-cdn-posix: setting xattrs on
> > > /srv/cdn/.trashcan//medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yr
> > > l8ZiGVIvdiyBD5u1pJUXBzYJ5b/Dante_Olive/origin/tmp/IMG3920.JPG_201
> > > 6-06-26_092737 failed  [No such file or directory]
> > > [2016-06-26 09:27:37.965955] E [posix.c:2996:_fill_writev_xdata]
> > > (-->/usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_read
> > > v_cbk+0x16c) [0x7f2620c6fc4c] -->/usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc)
> > > [0x7f26214ae37c] -->/usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0
> > > x1ff) [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode:
> > > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid
> > > argument]
> > > [2016-06-26 09:27:37.966096] E [posix.c:2996:_fill_writev_xdata]
> > > (-->/usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/features/trash.so(trash_truncate_read
> > > v_cbk+0x16c) [0x7f2620c6fc4c] -->/usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(posix_writev+0x1dc)
> > > [0x7f26214ae37c] -->/usr/lib/x86_64-linux-
> > > gnu/glusterfs/3.7.11/xlator/storage/posix.so(_fill_writev_xdata+0
> > > x1ff) [0x7f26214ae16f] ) 0-cdn-posix: fd: 0x7f26140057fc inode:
> > > 0x7f2600932484gfid:00000000-0000-0000-0000-000000000000 [Invalid
> > > argument]
> > 

Can you confirm whether truncated files are present in .trashcan
directory or not? From the above brick logs check
whether medias/fjDtE6cAz80xweuAdoIcWCDYxo9FdPqIVL88Yrl8ZiGVIvdiyBD5u1pJ
UXBzYJ5b/Dante_Olive/origin/tmp/IMG3920.JPG_2016-06-26_092737 exists
under .trashcan (from mount or directly on bricks)?

> > Yann
> > 
> > 
> > Le 27/06/2016 10:02, Atin Mukherjee a écrit :
> > > Grep for "backtrace" in /var/log/glusterfs, that should give you
> > > a particular log file name by which you can determine which
> > > process crashed. You should also be able to see a coredump. Why
> > > don't you attach the same for further analysis?
> > > 
> > > ~Atin
> > > 
> > > On Mon, Jun 27, 2016 at 1:17 PM, Yann LEMARIE <ylemarie at iraiser.e
> > > u> wrote:
> > > > Hi,
> > > > 
> > > > I'm using GlusterFS since many years and never see this
> > > > problem, but this is the second time in one week ...
> > > > 
> > > > I have 3 volumes with 2 bricks and 1 volume crash with no
> > > > reason, I just have to stop/start the volume to make it up
> > > > again.
> > > > The only logs I can find are in syslog :
> > > > 
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: pending frames:
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: frame : type(0) op(10)
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: patchset:
> > > > > git://git.gluster.com/glusterfs.git
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: signal received: 11
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: time of crash:
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: 2016-06-26 09:27:44
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: configuration details:
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: argp 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: backtrace 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: dlfcn 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: libpthread 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: llistxattr 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: setfsid 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: spinlock 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: epoll.h 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: xattr.h 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: st_atim.tv_nsec 1
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: package-string:
> > > > > glusterfs 3.7.11
> > > > > Jun 26 11:27:44 nfs05 srv-cdn[7879]: ---------
> > > > > 
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: pending frames:
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: frame : type(0) op(10)
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: patchset:
> > > > > git://git.gluster.com/glusterfs.git
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: signal received: 11
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: time of crash:
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: 2016-06-26 09:27:44
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: configuration details:
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: argp 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: backtrace 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: dlfcn 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: libpthread 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: llistxattr 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: setfsid 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: spinlock 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: epoll.h 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: xattr.h 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: st_atim.tv_nsec 1
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: package-string:
> > > > > glusterfs 3.7.11
> > > > > Jun 26 11:27:44 nfs06 srv-cdn[1787]: ---------
> > > > > 
> > > >  
> > > > Thanks for your help
> > > > 
> > > > 
> > > > Regards
> > > > -- 
> > > > Yann Lemarié
> > > > iRaiser - Support Technique
> > > >  
> > > > ylemarie at iraiser.eu
> > > > 
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://www.gluster.org/mailman/listinfo/gluster-users
> > > > 
> >  
> > -- 
> > Yann Lemarié
> > iRaiser - Support Technique
> >  
> > ylemarie at iraiser.eu
> > 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list