[Bugs] [Bug 1258149] New: Data Tiering:sqlite db database lock and crash while querying on tier volume bricks while IO was going on

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1258149

            Bug ID: 1258149
           Summary: Data Tiering:sqlite db database lock and crash while
                    querying on tier volume bricks while IO was going on
           Product: GlusterFS
           Version: 3.7.3
         Component: tiering
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: nchilaka at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org



Description of problem:
======================
On a hot replica pair over EC volume, While I was doing IOs on fuse mount(quite
a lot of mp3 files), I issued a sql query.
This crashed the database as below on one node
==>>>>>>>>>>>>>>>>>>>>>>>>>> COLD BRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==
efebba2e-e22b-494e-b4ed-a031878219fa|0|0|0|0|0|0|0|0|1|1
71ded0ac-64ed-47da-a327-136fab01c4c1|0|0|0|0|0|0|0|0|1|1
6774b9d1-8486-4a33-ba66-7da9af41a8e7|0|0|0|0|0|0|0|0|1|1
7d6fbf0a-5043-418c-99af-2668ff6781d0|0|0|0|0|0|0|0|0|1|1
3885ff39-2616-481a-ad8b-81d2f40af4d1|0|0|0|0|0|0|0|0|1|1
efebba2e-e22b-494e-b4ed-a031878219fa|00000000-0000-0000-0000-000000000001|f1|/f1|0|0
71ded0ac-64ed-47da-a327-136fab01c4c1|00000000-0000-0000-0000-000000000001|f2|/f2|0|0
6774b9d1-8486-4a33-ba66-7da9af41a8e7|00000000-0000-0000-0000-000000000001|f3|/f3|0|0
7d6fbf0a-5043-418c-99af-2668ff6781d0|00000000-0000-0000-0000-000000000001|f4|/f4|0|0
3885ff39-2616-481a-ad8b-81d2f40af4d1|00000000-0000-0000-0000-000000000001|f5|/f5|0|0
==>>>>>>>>>>>>>>>>>>>>>>>>>> HOTBRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==
Error: near line 1: database is locked

Crash was seen:
[2015-08-29 21:38:00.172183] E [MSGID: 109037]
[tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing Ooh

[2015-08-29 21:38:00.172196] E [MSGID: 109037]
[tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing La

[2015-08-29 21:38:00.172209] E [MSGID: 109037]
[tier.c:194:tier_migrate_using_query_file] 0-ecvol-tier-dht: failed parsing Laa

pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.com/glusterfs.git
signal received: 6
time of crash: 
2015-08-29 21:38:00
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 3.7.3
/usr/lib64/libglusterfs.so.0(_gf_msg_backtrace_nomem+0xb6)[0x3560c25936]
/usr/lib64/libglusterfs.so.0(gf_print_trace+0x32f)[0x3560c4549f]
/lib64/libc.so.6[0x340e8326a0]
/lib64/libc.so.6(gsignal+0x35)[0x340e832625]
/lib64/libc.so.6(abort+0x175)[0x340e833e05]
/lib64/libc.so.6[0x340e870537]
/lib64/libc.so.6(__fortify_fail+0x37)[0x340e9025d7]
/lib64/libc.so.6[0x340e9004c0]
/usr/lib64/glusterfs/3.7.3/xlator/cluster/tier.so(+0x55110)[0x7f92bfb87110]
/usr/lib64/glusterfs/3.7.3/xlator/cluster/tier.so(+0x55ebf)[0x7f92bfb87ebf]
/lib64/libpthread.so.0[0x340ec07a51]
/lib64/libc.so.6(clone+0x6d)[0x340e8e89ad]
---------
(END) 


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.mounted a replica over ec volume(already mounted on nfs) using fuse
2.did some IOs
3.issued a sql query on both nodes

4. crash was seen after database lock was seen on one node

[root at nag-manual-node1 ~]#  echo
"===========Date=====================";date;echo
"================================ ColdBrick#1==============================" ; 
echo "select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3
/rhs/brick1/ecvol/.glusterfs/ecvol.db;echo "===================== ColdBrick#2
======================" ;  echo "select * from gf_file_tb; select * from
gf_flink_tb;" | sqlite3 /rhs/brick2/ecvol/.glusterfs/ecvol.db;echo
"==>>>>>>>>>>>>>>>>>>>>>>>>>> COLD BRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==" ;  echo
"select * from gf_file_tb; select * from gf_flink_tb;" | sqlite3
/rhs/brick3/ecvol/.glusterfs/ecvol.db ;echo "==>>>>>>>>>>>>>>>>>>>>>>>>>>
HOTBRICK <<<<<<<<<<<<<<<<<<<<<<<<<<==" ;  echo "select * from gf_file_tb;
select * from gf_flink_tb;" | sqlite3
/rhs/brick4/ecvolhot/.glusterfs/ecvolhot.db 
===========Date=====================
Sun Aug 30 03:06:23 IST 2015

sosreports will be updated

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.