[Bugs] [Bug 1665803] New: [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image file from ovirt
bugzilla at redhat.com
bugzilla at redhat.com
Mon Jan 14 06:07:44 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1665803
Bug ID: 1665803
Summary: [ovirt-gluster] Fuse mount crashed while deleting a 1
TB image file from ovirt
Product: GlusterFS
Version: 5
Hardware: x86_64
OS: Linux
Status: NEW
Component: sharding
Keywords: Triaged
Severity: high
Assignee: bugs at gluster.org
Reporter: kdhananj at redhat.com
QA Contact: bugs at gluster.org
CC: bugs at gluster.org, rhs-bugs at redhat.com,
sankarshan at redhat.com, sasundar at redhat.com,
storage-qa-internal at redhat.com
Depends On: 1662368
Blocks: 1662059
Target Milestone: ---
Classification: Community
+++ This bug was initially created as a clone of Bug #1662368 +++
+++ This bug was initially created as a clone of Bug #1662059 +++
Description of problem:
-----------------------
Attempts were made to try out the customer scenario, where the disks of bigger
sizes residing on gluster volumes are deleted from ovirt. During one such
attempts, its found that the fuse mount process has crashed.
Version-Release number of selected component (if applicable):
-------------------------------------------------------------
ovirt 4.0.5
gluster-master
How reproducible:
-----------------
1/1
Hit it once.
Steps to Reproduce:
-------------------
1. ovirt storage domain is configured to use gluster arbitrated replicate
volume,
with sharding enabled
2. Create disk of size 1TB from ovirt Manager UI
3. Delete the disk from ovirt Manager UI
Actual results:
---------------
Gluster fuse mount process crashed on one of the hypervisor
Expected results:
-----------------
No gluster process should crash
--- Additional comment from SATHEESARAN on 2018-12-25 18:03:56 UTC ---
1. Gluster cluster info
------------------------
There are 3 nodes in the gluster cluster
2. Gluster volume info
----------------------
[root at rhsqa-grafton7-nic2 ~]# gluster volume info imstore
Volume Name: imstore
Type: Replicate
Volume ID: 878eb828-0735-4ce8-a2b3-c52a757ee1b2
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: server1:/gluster_bricks/vol1/im1
Brick2: server2:/gluster_bricks/vol1/im1
Brick3: server3:/gluster_bricks/vol1/im1 (arbiter)
Options Reconfigured:
performance.strict-o-direct: on
storage.owner-gid: 36
storage.owner-uid: 36
network.ping-timeout: 30
cluster.granular-entry-heal: on
server.event-threads: 4
client.event-threads: 4
cluster.choose-local: off
user.cifs: off
features.shard: on
cluster.shd-wait-qlength: 10000
cluster.shd-max-threads: 8
cluster.locking-scheme: granular
cluster.data-self-heal-algorithm: full
cluster.server-quorum-type: server
cluster.quorum-type: auto
cluster.eager-lock: enable
network.remote-dio: off
performance.low-prio-threads: 32
performance.io-cache: off
performance.read-ahead: off
performance.quick-read: off
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
4. Test info
-------------
File that was deleted:
/rhev/data-center/mnt/glusterSD/server1\:_imstore/3d4f163a-c6e0-476e-a726-bd780e0d1b83/images/075c6ffd-318c-4108-8405-ccf8078c1e16/b62a4640-f02a-4aa1-b249-cfc4cb2f7f59
GFID of this file is: 3d231d2b-4fff-4c03-b593-70befaf77296
Before deleting the file:
[root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/ |grep 3d231 | wc -l
16383
While the deleting is in progress:
[root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/ |grep 3d231 | wc -l
3983
After the fuse mount crash, there were some ghost shards, but after 15 mins,
there are no ghost shards
[root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/.remove_me/
[root at server1 ~]# ls /gluster_bricks/vol1/im1/.shard/ |grep 3d231 | wc -l
0
--- Additional comment from SATHEESARAN on 2018-12-26 06:27:00 UTC ---
(In reply to Krutika Dhananjay from comment #5)
> So there is no core dump and I can't tell much from just the logs.
>
> From
> [root at dhcp37-127 ~]# cat /proc/sys/kernel/core_pattern
> |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i
>
> Seems like this should be set to a valid path for us to get the core dump.
>
> Would be great if you can change this value to a meaningful path and
> recreate the issue.
>
> -Krutika
I could reproduce the issue consistently outside of ovirt-gluster setup.
With 3 gluster servers and 1 client.
1. Create 5 VM image files on the fuse mounted gluster volume using qemu-img
command
# qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm1.img 10G
# qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm2.img 7G
# qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm3.img 5G
# qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm4.img 4G
2. Delete the files from the mount
# rm -rf /mnt/testdata/*
The above step hits the crash,close to consistent
I will reinstall the required debug packages and will provide the setup for
debugging
--- Additional comment from SATHEESARAN on 2018-12-26 06:33:43 UTC ---
Backtrace from the core file
Core was generated by `/usr/sbin/glusterfs --volfile-server=10.70.37.152
--volfile-id=/volume1 /mnt/te'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007fdb3d8eb53e in shard_unlink_block_inode
(local=local at entry=0x7fdb2400a400, shard_block_num=<optimized out>) at
shard.c:2945
2945 base_ictx->fsync_count--;
--- Additional comment from Krutika Dhananjay on 2018-12-26 14:23:28 UTC ---
(In reply to SATHEESARAN from comment #6)
> (In reply to Krutika Dhananjay from comment #5)
> > So there is no core dump and I can't tell much from just the logs.
> >
> > From
> > [root at dhcp37-127 ~]# cat /proc/sys/kernel/core_pattern
> > |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i
> >
> > Seems like this should be set to a valid path for us to get the core dump.
> >
> > Would be great if you can change this value to a meaningful path and
> > recreate the issue.
> >
> > -Krutika
>
> I could reproduce the issue consistently outside of RHV-RHGS setup.
> With 3 RHGS servers and 1 client.
>
> 1. Create 5 VM image files on the fuse mounted gluster volume using qemu-img
> command
> # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm1.img 10G
> # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm2.img 7G
> # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm3.img 5G
> # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm4.img 4G
>
> 2. Delete the files from the mount
> # rm -rf /mnt/testdata/*
>
> The above step hits the crash,close to consistent
>
> I will reinstall the required debug packages and will provide the setup for
> debugging
Is the mountpoint in step 1 different from the one used in 2? In step 1, files
are created under /mnt/test/. But the rm -rf is done from /mnt/testdata/
-Krutika
--- Additional comment from SATHEESARAN on 2018-12-26 15:38:21 UTC ---
(In reply to Krutika Dhananjay from comment #8)
> (In reply to SATHEESARAN from comment #6)
> > (In reply to Krutika Dhananjay from comment #5)
> > > So there is no core dump and I can't tell much from just the logs.
> > >
> > > From
> > > [root at dhcp37-127 ~]# cat /proc/sys/kernel/core_pattern
> > > |/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t %e %i
> > >
> > > Seems like this should be set to a valid path for us to get the core dump.
> > >
> > > Would be great if you can change this value to a meaningful path and
> > > recreate the issue.
> > >
> > > -Krutika
> >
> > I could reproduce the issue consistently outside of RHV-RHGS setup.
> > With 3 RHGS servers and 1 client.
> >
> > 1. Create 5 VM image files on the fuse mounted gluster volume using qemu-img
> > command
> > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm1.img 10G
> > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm2.img 7G
> > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm3.img 5G
> > # qemu-img create -f qcow2 -o preallocation=full /mnt/test/vm4.img 4G
> >
> > 2. Delete the files from the mount
> > # rm -rf /mnt/testdata/*
> >
> > The above step hits the crash,close to consistent
> >
> > I will reinstall the required debug packages and will provide the setup for
> > debugging
>
>
> Is the mountpoint in step 1 different from the one used in 2? In step 1,
> files are created under /mnt/test/. But the rm -rf is done from
> /mnt/testdata/
>
> -Krutika
I did it from same mount. No different mounts
--- Additional comment from Worker Ant on 2018-12-28 02:10:20 UTC ---
REVIEW: https://review.gluster.org/21946 (features/shard: Assign fop id during
background deletion to prevent excessive logging) posted (#1) for review on
master by Krutika Dhananjay
--- Additional comment from Worker Ant on 2018-12-28 15:43:45 UTC ---
REVIEW: https://review.gluster.org/21957 (features/shard: Fix launch of
multiple synctasks for background deletion) posted (#1) for review on master by
Krutika Dhananjay
--- Additional comment from Worker Ant on 2019-01-08 12:09:39 UTC ---
REVIEW: https://review.gluster.org/21946 (features/shard: Assign fop id during
background deletion to prevent excessive logging) posted (#7) for review on
master by Xavi Hernandez
--- Additional comment from Worker Ant on 2019-01-11 08:36:32 UTC ---
REVIEW: https://review.gluster.org/21957 (features/shard: Fix launch of
multiple synctasks for background deletion) posted (#7) for review on master by
Xavi Hernandez
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1662059
[Bug 1662059] [RHV-RHGS] Fuse mount crashed while deleting a 1 TB image file
from RHV
https://bugzilla.redhat.com/show_bug.cgi?id=1662368
[Bug 1662368] [ovirt-gluster] Fuse mount crashed while deleting a 1 TB image
file from ovirt
--
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list