[Bugs] [Bug 1337837] New: Files present in the .shard folder even after deleting all the vms from the UI
bugzilla at redhat.com
bugzilla at redhat.com
Fri May 20 08:11:36 UTC 2016
https://bugzilla.redhat.com/show_bug.cgi?id=1337837
Bug ID: 1337837
Summary: Files present in the .shard folder even after deleting
all the vms from the UI
Product: GlusterFS
Version: 3.7.11
Component: sharding
Keywords: Triaged
Severity: medium
Assignee: bugs at gluster.org
Reporter: kdhananj at redhat.com
QA Contact: bugs at gluster.org
CC: bugs at gluster.org, knarra at redhat.com,
pkarampu at redhat.com, sabose at redhat.com
Depends On: 1333643, 1335858
+++ This bug was initially created as a clone of Bug #1335858 +++
+++ This bug was initially created as a clone of Bug #1333643 +++
Description of problem:
I have a HC setup with templates created for both windows and linux vms. I have
created a pool out of those templates and launched vms . Now i went a head and
removed pool and the vms associated to it except the HostedEngine. I see that
there are some files still present under .shard folder plus in the mount point
too.
Files present under data storage domain:
===============================================
[root at sulphur images]# ll
total 12
drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:45 4ec5c4d3-89b5-4b75-93e7-3cb54773850c
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:03 a0685750-13af-4723-a257-e483138675ce
drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:44 f1ac00a8-836c-488f-8131-78e72911495e
.shard files present inside the data brick of the volume:
=========================================================
[root at sulphur .shard]# ll
total 7529900
-rw-rw----. 2 root root 0 Apr 26 19:34
71ce5baa-60c6-4015-846b-9d721b899ab2.1
-rw-rw----. 2 root root 0 Apr 26 19:38
71ce5baa-60c6-4015-846b-9d721b899ab2.10
-rw-rw----. 2 root root 0 Apr 26 19:39
71ce5baa-60c6-4015-846b-9d721b899ab2.11
-rw-rw----. 2 root root 0 Apr 26 19:39
71ce5baa-60c6-4015-846b-9d721b899ab2.12
-rw-rw----. 2 root root 0 Apr 26 19:40
71ce5baa-60c6-4015-846b-9d721b899ab2.13
-rw-rw----. 2 root root 0 Apr 26 19:40
71ce5baa-60c6-4015-846b-9d721b899ab2.14
-rw-rw----. 2 root root 0 Apr 26 19:41
71ce5baa-60c6-4015-846b-9d721b899ab2.15
-rw-rw----. 2 root root 0 Apr 26 19:41
71ce5baa-60c6-4015-846b-9d721b899ab2.16
-rw-rw----. 2 root root 0 Apr 26 19:42
71ce5baa-60c6-4015-846b-9d721b899ab2.17
-rw-rw----. 2 root root 0 Apr 26 19:42
71ce5baa-60c6-4015-846b-9d721b899ab2.18
-rw-rw----. 2 root root 0 Apr 26 19:42
71ce5baa-60c6-4015-846b-9d721b899ab2.19
-rw-rw----. 2 root root 0 Apr 26 19:34
71ce5baa-60c6-4015-846b-9d721b899ab2.2
-rw-rw----. 2 root root 0 Apr 26 19:43
71ce5baa-60c6-4015-846b-9d721b899ab2.20
-rw-rw----. 2 root root 0 Apr 26 19:43
71ce5baa-60c6-4015-846b-9d721b899ab2.21
-rw-rw----. 2 root root 0 Apr 26 19:44
71ce5baa-60c6-4015-846b-9d721b899ab2.22
-rw-rw----. 2 root root 0 Apr 26 19:44
71ce5baa-60c6-4015-846b-9d721b899ab2.23
-rw-rw----. 2 root root 0 Apr 26 19:45
71ce5baa-60c6-4015-846b-9d721b899ab2.24
-rw-rw----. 2 root root 0 Apr 26 19:21
71ce5baa-60c6-4015-846b-9d721b899ab2.25
-rw-rw----. 2 root root 0 Apr 26 19:46
71ce5baa-60c6-4015-846b-9d721b899ab2.26
-rw-rw----. 2 root root 0 Apr 26 19:46
71ce5baa-60c6-4015-846b-9d721b899ab2.27
-rw-rw----. 2 root root 0 Apr 26 19:47
71ce5baa-60c6-4015-846b-9d721b899ab2.28
-rw-rw----. 2 root root 0 Apr 26 19:47
71ce5baa-60c6-4015-846b-9d721b899ab2.29
-rw-rw----. 2 root root 0 Apr 26 19:35
71ce5baa-60c6-4015-846b-9d721b899ab2.3
-rw-rw----. 2 root root 0 Apr 26 19:48
71ce5baa-60c6-4015-846b-9d721b899ab2.30
-rw-rw----. 2 root root 0 Apr 26 19:48
71ce5baa-60c6-4015-846b-9d721b899ab2.31
-rw-rw----. 2 root root 0 Apr 26 19:49
71ce5baa-60c6-4015-846b-9d721b899ab2.32
-rw-rw----. 2 root root 0 Apr 26 19:49
71ce5baa-60c6-4015-846b-9d721b899ab2.33
-rw-rw----. 2 root root 0 Apr 26 19:50
71ce5baa-60c6-4015-846b-9d721b899ab2.34
-rw-rw----. 2 root root 0 Apr 26 19:50
71ce5baa-60c6-4015-846b-9d721b899ab2.35
-rw-rw----. 2 root root 0 Apr 26 19:50
71ce5baa-60c6-4015-846b-9d721b899ab2.36
-rw-rw----. 2 root root 0 Apr 26 19:51
71ce5baa-60c6-4015-846b-9d721b899ab2.37
-rw-rw----. 2 root root 0 Apr 26 19:51
71ce5baa-60c6-4015-846b-9d721b899ab2.38
-rw-rw----. 2 root root 0 Apr 26 19:52
71ce5baa-60c6-4015-846b-9d721b899ab2.39
-rw-rw----. 2 root root 0 Apr 26 19:35
71ce5baa-60c6-4015-846b-9d721b899ab2.4
-rw-rw----. 2 root root 0 Apr 26 19:52
71ce5baa-60c6-4015-846b-9d721b899ab2.40
-rw-rw----. 2 root root 0 Apr 26 19:53
71ce5baa-60c6-4015-846b-9d721b899ab2.41
-rw-rw----. 2 root root 0 Apr 26 19:53
71ce5baa-60c6-4015-846b-9d721b899ab2.42
-rw-rw----. 2 root root 0 Apr 26 19:54
71ce5baa-60c6-4015-846b-9d721b899ab2.43
-rw-rw----. 2 root root 0 Apr 26 19:54
71ce5baa-60c6-4015-846b-9d721b899ab2.44
-rw-rw----. 2 root root 0 Apr 26 19:55
71ce5baa-60c6-4015-846b-9d721b899ab2.45
-rw-rw----. 2 root root 0 Apr 26 19:55
71ce5baa-60c6-4015-846b-9d721b899ab2.46
-rw-rw----. 2 root root 0 Apr 26 19:56
71ce5baa-60c6-4015-846b-9d721b899ab2.47
-rw-rw----. 2 root root 0 Apr 26 19:56
71ce5baa-60c6-4015-846b-9d721b899ab2.48
-rw-rw----. 2 root root 0 Apr 26 19:57
71ce5baa-60c6-4015-846b-9d721b899ab2.49
-rw-rw----. 2 root root 0 Apr 26 19:36
71ce5baa-60c6-4015-846b-9d721b899ab2.5
-rw-rw----. 2 root root 0 Apr 26 19:21
71ce5baa-60c6-4015-846b-9d721b899ab2.50
-rw-rw----. 2 root root 0 Apr 26 19:57
71ce5baa-60c6-4015-846b-9d721b899ab2.51
-rw-rw----. 2 root root 0 Apr 26 19:58
71ce5baa-60c6-4015-846b-9d721b899ab2.52
-rw-rw----. 2 root root 0 Apr 26 19:58
71ce5baa-60c6-4015-846b-9d721b899ab2.53
-rw-rw----. 2 root root 0 Apr 26 19:59
71ce5baa-60c6-4015-846b-9d721b899ab2.54
-rw-rw----. 2 root root 0 Apr 26 19:59
71ce5baa-60c6-4015-846b-9d721b899ab2.55
-rw-rw----. 2 root root 0 Apr 26 20:00
71ce5baa-60c6-4015-846b-9d721b899ab2.56
-rw-rw----. 2 root root 0 Apr 26 20:00
71ce5baa-60c6-4015-846b-9d721b899ab2.57
-rw-rw----. 2 root root 0 Apr 26 20:01
71ce5baa-60c6-4015-846b-9d721b899ab2.58
-rw-rw----. 2 root root 0 Apr 26 20:01
71ce5baa-60c6-4015-846b-9d721b899ab2.59
-rw-rw----. 2 root root 0 Apr 26 19:36
71ce5baa-60c6-4015-846b-9d721b899ab2.6
-rw-rw----. 2 root root 0 Apr 26 20:02
71ce5baa-60c6-4015-846b-9d721b899ab2.60
-rw-rw----. 2 root root 0 Apr 26 20:02
71ce5baa-60c6-4015-846b-9d721b899ab2.61
-rw-rw----. 2 root root 0 Apr 26 20:03
71ce5baa-60c6-4015-846b-9d721b899ab2.62
-rw-rw----. 2 root root 0 Apr 26 20:03
71ce5baa-60c6-4015-846b-9d721b899ab2.63
-rw-rw----. 2 root root 0 Apr 26 20:04
71ce5baa-60c6-4015-846b-9d721b899ab2.64
-rw-rw----. 2 root root 0 Apr 26 20:04
71ce5baa-60c6-4015-846b-9d721b899ab2.65
-rw-rw----. 2 root root 0 Apr 26 20:04
71ce5baa-60c6-4015-846b-9d721b899ab2.66
-rw-rw----. 2 root root 0 Apr 26 20:05
71ce5baa-60c6-4015-846b-9d721b899ab2.67
-rw-rw----. 2 root root 0 Apr 26 20:05
71ce5baa-60c6-4015-846b-9d721b899ab2.68
-rw-rw----. 2 root root 0 Apr 26 20:06
71ce5baa-60c6-4015-846b-9d721b899ab2.69
-rw-rw----. 2 root root 0 Apr 26 19:37
71ce5baa-60c6-4015-846b-9d721b899ab2.7
-rw-rw----. 2 root root 0 Apr 26 20:06
71ce5baa-60c6-4015-846b-9d721b899ab2.70
-rw-rw----. 2 root root 0 Apr 26 20:07
71ce5baa-60c6-4015-846b-9d721b899ab2.71
-rw-rw----. 2 root root 0 Apr 26 20:07
71ce5baa-60c6-4015-846b-9d721b899ab2.72
-rw-rw----. 2 root root 0 Apr 26 20:08
71ce5baa-60c6-4015-846b-9d721b899ab2.73
-rw-rw----. 2 root root 0 Apr 26 20:08
71ce5baa-60c6-4015-846b-9d721b899ab2.74
-rw-rw----. 2 root root 0 Apr 26 19:21
71ce5baa-60c6-4015-846b-9d721b899ab2.75
-rw-rw----. 2 root root 0 Apr 26 20:09
71ce5baa-60c6-4015-846b-9d721b899ab2.76
-rw-rw----. 2 root root 0 Apr 26 20:10
71ce5baa-60c6-4015-846b-9d721b899ab2.77
-rw-rw----. 2 root root 0 Apr 26 20:10
71ce5baa-60c6-4015-846b-9d721b899ab2.78
-rw-rw----. 2 root root 0 Apr 26 20:11
71ce5baa-60c6-4015-846b-9d721b899ab2.79
-rw-rw----. 2 root root 0 Apr 26 19:37
71ce5baa-60c6-4015-846b-9d721b899ab2.8
-rw-rw----. 2 root root 0 Apr 26 20:11
71ce5baa-60c6-4015-846b-9d721b899ab2.80
-rw-rw----. 2 root root 0 Apr 26 20:11
71ce5baa-60c6-4015-846b-9d721b899ab2.81
-rw-rw----. 2 root root 0 Apr 26 20:12
71ce5baa-60c6-4015-846b-9d721b899ab2.82
-rw-rw----. 2 root root 0 Apr 26 20:12
71ce5baa-60c6-4015-846b-9d721b899ab2.83
-rw-rw----. 2 root root 0 Apr 26 20:13
71ce5baa-60c6-4015-846b-9d721b899ab2.84
-rw-rw----. 2 root root 0 Apr 26 20:13
71ce5baa-60c6-4015-846b-9d721b899ab2.85
-rw-rw----. 2 root root 0 Apr 26 20:14
71ce5baa-60c6-4015-846b-9d721b899ab2.86
-rw-rw----. 2 root root 0 Apr 26 20:14
71ce5baa-60c6-4015-846b-9d721b899ab2.87
-rw-rw----. 2 root root 0 Apr 26 20:15
71ce5baa-60c6-4015-846b-9d721b899ab2.88
-rw-rw----. 2 root root 0 Apr 26 20:15
71ce5baa-60c6-4015-846b-9d721b899ab2.89
-rw-rw----. 2 root root 0 Apr 26 19:38
71ce5baa-60c6-4015-846b-9d721b899ab2.9
-rw-rw----. 2 root root 0 Apr 26 20:16
71ce5baa-60c6-4015-846b-9d721b899ab2.90
-rw-rw----. 2 root root 0 Apr 26 20:16
71ce5baa-60c6-4015-846b-9d721b899ab2.91
-rw-rw----. 2 root root 0 Apr 26 20:17
71ce5baa-60c6-4015-846b-9d721b899ab2.92
-rw-rw----. 2 root root 0 Apr 26 20:17
71ce5baa-60c6-4015-846b-9d721b899ab2.93
-rw-rw----. 2 root root 0 Apr 26 20:18
71ce5baa-60c6-4015-846b-9d721b899ab2.94
-rw-rw----. 2 root root 0 Apr 26 20:18
71ce5baa-60c6-4015-846b-9d721b899ab2.95
-rw-rw----. 2 root root 0 Apr 26 20:19
71ce5baa-60c6-4015-846b-9d721b899ab2.96
-rw-rw----. 2 root root 0 Apr 26 20:19
71ce5baa-60c6-4015-846b-9d721b899ab2.97
-rw-rw----. 2 root root 0 Apr 26 20:19
71ce5baa-60c6-4015-846b-9d721b899ab2.98
-rw-rw----. 2 root root 0 Apr 13 11:15
71ce5baa-60c6-4015-846b-9d721b899ab2.99
-rw-rw----. 2 root root 536870912 May 4 17:58
f9333cb0-eab1-4899-be93-86d03d75ee0e.1
-rw-rw----. 2 root root 536870912 May 4 17:27
f9333cb0-eab1-4899-be93-86d03d75ee0e.10
-rw-rw----. 2 root root 536870912 May 4 17:26
f9333cb0-eab1-4899-be93-86d03d75ee0e.11
-rw-rw----. 2 root root 536870912 May 4 17:26
f9333cb0-eab1-4899-be93-86d03d75ee0e.12
-rw-rw----. 2 root root 47284224 May 4 17:27
f9333cb0-eab1-4899-be93-86d03d75ee0e.13
-rw-rw----. 2 root root 0 May 4 17:11
f9333cb0-eab1-4899-be93-86d03d75ee0e.14
-rw-rw----. 2 root root 0 May 4 17:11
f9333cb0-eab1-4899-be93-86d03d75ee0e.15
-rw-rw----. 2 root root 536870912 May 4 17:53
f9333cb0-eab1-4899-be93-86d03d75ee0e.16
-rw-rw----. 2 root root 536870912 May 4 17:54
f9333cb0-eab1-4899-be93-86d03d75ee0e.17
-rw-rw----. 2 root root 0 May 4 17:11
f9333cb0-eab1-4899-be93-86d03d75ee0e.18
-rw-rw----. 2 root root 143769600 May 4 17:55
f9333cb0-eab1-4899-be93-86d03d75ee0e.19
-rw-rw----. 2 root root 536870912 May 5 10:15
f9333cb0-eab1-4899-be93-86d03d75ee0e.2
-rw-rw----. 2 root root 802816 May 4 16:46
f9333cb0-eab1-4899-be93-86d03d75ee0e.25
-rw-rw----. 2 root root 536870912 May 4 18:00
f9333cb0-eab1-4899-be93-86d03d75ee0e.3
-rw-rw----. 2 root root 536870912 May 4 18:01
f9333cb0-eab1-4899-be93-86d03d75ee0e.4
-rw-rw----. 2 root root 536870912 May 4 16:57
f9333cb0-eab1-4899-be93-86d03d75ee0e.5
-rw-rw----. 2 root root 26750976 May 4 18:33
f9333cb0-eab1-4899-be93-86d03d75ee0e.50
-rw-rw----. 2 root root 536870912 May 4 17:20
f9333cb0-eab1-4899-be93-86d03d75ee0e.6
-rw-rw----. 2 root root 536870912 May 5 10:15
f9333cb0-eab1-4899-be93-86d03d75ee0e.7
-rw-rw----. 2 root root 278528 May 4 16:46
f9333cb0-eab1-4899-be93-86d03d75ee0e.75
-rw-rw----. 2 root root 536870912 May 5 10:15
f9333cb0-eab1-4899-be93-86d03d75ee0e.8
-rw-rw----. 2 root root 536870912 May 4 17:27
f9333cb0-eab1-4899-be93-86d03d75ee0e.9
-rw-rw----. 2 root root 536870912 May 4 16:46
f9333cb0-eab1-4899-be93-86d03d75ee0e.99
Images under vmstore storage domain:
======================================
[root at sulphur images]# ll
total 16
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:02 210ca44a-d0e3-4750-b3d3-4b1358df280b
drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:45 5dc92a56-ee20-4d2e-a6bd-9f3753263029
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 12:14 8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506
drwxr-xr-x. 2 vdsm kvm 4096 May 5 11:45 bf3ed8c9-fac0-4f42-931b-9c92c9af2fe5
.shard files under vmstore brick:
======================================
[root at sulphur .shard]# ll
total 9055212
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.1
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.10
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.11
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.12
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.13
-rw-rw----. 2 root root 536870912 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.14
-rw-rw----. 2 root root 375697408 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.15
-rw-rw----. 2 root root 0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.16
-rw-rw----. 2 root root 0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.17
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.18
-rw-rw----. 2 root root 0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.19
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.2
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.20
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.21
-rw-rw----. 2 root root 0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.22
-rw-rw----. 2 root root 536870912 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.23
-rw-rw----. 2 root root 536870912 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.24
-rw-rw----. 2 root root 77815808 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.25
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.26
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.27
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.28
-rw-rw----. 2 root root 0 Apr 15 21:50
026de0b6-9bea-4b14-8b6f-b7261eb6c030.29
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.3
-rw-rw----. 2 root root 0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.30
-rw-rw----. 2 root root 0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.31
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.32
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.33
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.34
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.35
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.36
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.37
-rw-rw----. 2 root root 0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.38
-rw-rw----. 2 root root 535822336 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.39
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.4
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.5
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.6
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.7
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.8
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.9
-rw-rw----. 2 root root 0 May 2 20:19
14215e87-1c82-45ec-9a34-4a8bb028d86f.1
-rw-rw----. 2 root root 0 May 2 20:23
14215e87-1c82-45ec-9a34-4a8bb028d86f.10
-rw-rw----. 2 root root 0 May 2 20:23
14215e87-1c82-45ec-9a34-4a8bb028d86f.11
-rw-rw----. 2 root root 0 May 2 20:24
14215e87-1c82-45ec-9a34-4a8bb028d86f.12
-rw-rw----. 2 root root 536870912 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.13
-rw-rw----. 2 root root 94326784 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.14
-rw-rw----. 2 root root 0 Apr 29 16:54
14215e87-1c82-45ec-9a34-4a8bb028d86f.15
-rw-rw----. 2 root root 0 Apr 29 16:55
14215e87-1c82-45ec-9a34-4a8bb028d86f.16
-rw-rw----. 2 root root 0 Apr 29 16:55
14215e87-1c82-45ec-9a34-4a8bb028d86f.17
-rw-rw----. 2 root root 0 May 2 20:26
14215e87-1c82-45ec-9a34-4a8bb028d86f.18
-rw-rw----. 2 root root 0 May 2 20:26
14215e87-1c82-45ec-9a34-4a8bb028d86f.19
-rw-rw----. 2 root root 0 May 2 20:20
14215e87-1c82-45ec-9a34-4a8bb028d86f.2
-rw-rw----. 2 root root 0 May 2 20:26
14215e87-1c82-45ec-9a34-4a8bb028d86f.20
-rw-rw----. 2 root root 0 May 2 20:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.21
-rw-rw----. 2 root root 499495424 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.22
-rw-rw----. 2 root root 0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.23
-rw-rw----. 2 root root 0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.24
-rw-rw----. 2 root root 0 Apr 26 19:28
14215e87-1c82-45ec-9a34-4a8bb028d86f.25
-rw-rw----. 2 root root 0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.26
-rw-rw----. 2 root root 0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.27
-rw-rw----. 2 root root 0 Apr 26 19:28
14215e87-1c82-45ec-9a34-4a8bb028d86f.28
-rw-rw----. 2 root root 0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.29
-rw-rw----. 2 root root 0 May 2 20:20
14215e87-1c82-45ec-9a34-4a8bb028d86f.3
-rw-rw----. 2 root root 0 Apr 26 19:28
14215e87-1c82-45ec-9a34-4a8bb028d86f.30
-rw-rw----. 2 root root 372343296 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.31
-rw-rw----. 2 root root 0 May 2 20:31
14215e87-1c82-45ec-9a34-4a8bb028d86f.32
-rw-rw----. 2 root root 0 May 2 20:31
14215e87-1c82-45ec-9a34-4a8bb028d86f.33
-rw-rw----. 2 root root 0 May 2 20:32
14215e87-1c82-45ec-9a34-4a8bb028d86f.34
-rw-rw----. 2 root root 0 May 2 20:32
14215e87-1c82-45ec-9a34-4a8bb028d86f.35
-rw-rw----. 2 root root 0 May 2 20:32
14215e87-1c82-45ec-9a34-4a8bb028d86f.36
-rw-rw----. 2 root root 0 May 2 20:33
14215e87-1c82-45ec-9a34-4a8bb028d86f.37
-rw-rw----. 2 root root 0 May 2 20:33
14215e87-1c82-45ec-9a34-4a8bb028d86f.38
-rw-rw----. 2 root root 0 Apr 13 11:15
14215e87-1c82-45ec-9a34-4a8bb028d86f.39
-rw-rw----. 2 root root 536869888 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.4
-rw-rw----. 2 root root 214117376 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.5
-rw-rw----. 2 root root 0 Apr 29 19:24
14215e87-1c82-45ec-9a34-4a8bb028d86f.6
-rw-rw----. 2 root root 0 Apr 29 19:24
14215e87-1c82-45ec-9a34-4a8bb028d86f.7
-rw-rw----. 2 root root 0 May 2 20:22
14215e87-1c82-45ec-9a34-4a8bb028d86f.8
-rw-rw----. 2 root root 0 May 2 20:22
14215e87-1c82-45ec-9a34-4a8bb028d86f.9
I ran the command virsh -r list command to make sure that there are no other
vms running other than HostedEngine from all the nodes:
[root at sulphur ~]# virsh -r list
Id Name State
----------------------------------------------------
616 HostedEngine running
[root at tettnang ~]# virsh -r list
Id Name State
----------------------------------------------------
[root at zod ~]# virsh -r list
Id Name State
----------------------------------------------------
Version-Release number of selected component (if applicable):
glusterfs-3.7.9-2.el7rhgs.x86_64
How reproducible:
Tried it once
Steps to Reproduce:
1. Create vms in HC setup using RHEV templates and Pools.
2. Now delete the pool so that all the vms are removed except HostedEngine
3.
Actual results:
I still see that there are some image files present inside the storage domain
plus entries under .shard folder for both data and vmstore volumes.
Expected results:
When all the virtual machines are removed ideally there should not be any
entries present in the .shard folder and in the storage domain.
Additional info:
--- Additional comment from Pranith Kumar K on 2016-05-12 06:02:21 EDT ---
RCA:
Handling of hardlinks in sharding xlator:
Sharding translator performs hardlinks only on the base-file. When the last
link is deleted, it deletes all the shard files associated with that base-file.
So when an unlink fop comes to shard, it does a lookup (Equivalent of stat for
user applications) to find out the number of links. If the number of links is >
1 then it just deletes that link but doesn't shards. If the number of links is
1, then it goes ahead and deletes all the shards as well.
What happens in the rhev-setup on VM deletion:
Kasturi (QE) selected all the VMs at once and performed delete. All these vm
data will be moved to remove_me_xyz directory and host which is the owner of
that VM will start doing unlinks in the background.
It seems like there are hardlinks on the setup where VMs are stored, where
links are shared between different hosts.
[root at sulphur vmstore-brick1]# find . -type f | xargs getfattr -d -m. -e hex |
grep trusted.gfid | sort | uniq -c | grep -v " 2 "
17 trusted.gfid=0x026de0b69bea4b148b6fb7261eb6c030
1 trusted.gfid=0x122c529ce3ab440f87edc2cf9ef79051
1 trusted.gfid=0x1383cf29af214231bbaafd3120fd4c27
16 trusted.gfid=0x13fd032531be41d4a5e8cff2aff02aca
16 trusted.gfid=0x14215e871c8245ec9a344a8bb028d86f
1 trusted.gfid=0x14807a3c2c0b4f6b92cea4a042856971
16 trusted.gfid=0x2b294ed3423c4e43afee404315a316ab
1 trusted.gfid=0x445a3b7cfb444fb290352b9087dc105b
1 trusted.gfid=0x4a21a35d946d4fe88283f3a2a29992cd
1 trusted.gfid=0x4ae3729419894b3bbb08467e32a088e7
1 trusted.gfid=0x655f5fdf9a59499bb3d329a7ab23ff5e
1 trusted.gfid=0x710d1bbe94e54a60ac64d916b1df672c
1 trusted.gfid=0x97abfcae67ff420797ad3014ce084e2e
1 trusted.gfid=0xc5816599efb547cca48a4a58a9751844
17 trusted.gfid=0xd939b5a7aa29453e814e32c0cf6e2608
1 trusted.gfid=0xd94592fb54ef46dbbf3da881ac94e465
17 trusted.gfid=0xdec2404ec7574494a78598ea4bd31381
1 trusted.gfid=0xfff5f116ef9140078c6536f23197695f
[root at sulphur vmstore-brick1]# ls -lih
.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
469762591 -rw-rw----. 17 vdsm kvm 512M Apr 13 12:10
.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
[root at sulphur vmstore-brick1]# find . -inum 469762591
./.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5308d74e-28e2-4158-b2f9-af898e14e6d0/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/7ca57bac-c614-402e-9bb4-0d172d69c476/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5ee9dc38-f4a5-41ec-bd95-878165c022b1/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/99f4c43a-dc2f-46f1-b4b4-2cd289f43757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/bedfd824-6560-454a-9ed1-6baf3e592428/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/9965825b-3e79-434a-8eec-b85994901ce5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/548b241a-7958-40f4-b84a-161628f4e0e5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/394f07f9-baab-4299-ba76-a13b9d2a949f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/6e97619b-bbfb-4856-b63c-05727f84556f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ff21f8ba-5038-42f6-9b29-d56d87d87b21/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/3f61feef-0eb3-4cee-875b-d2e22dc8e7e3/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/a96768b4-548c-4c9d-80ac-fb4552e97a94/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ae9715b8-c6a7-4125-948f-877fba1a7757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d
When all the VMs are deleted together, it can lead to different hosts trying to
delete hardlinks in parallel in their respective remove_me_xxx directory. i.e.
host1 can delete
"./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d"
where as host2 can delete
"./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d".
which are hardlinks to each other. So at the time when shard queries how many
links the file has by doing lookup both of them will get a number greater than
1, so they both go ahead and delete the file. But one of those unlinks will be
the last one but shard has no clue about this because at the time when it
queried number of links is more than 1. This leads to shard not deleting
shards. This is the reason for the bug.
We have seen files with hardlink count '1' which suggests there is a bug even
in posix-xlator which is not deleting gfid-link thinking it is not the last
link.
Pranith
--- Additional comment from Vijay Bellur on 2016-05-13 08:07:18 EDT ---
REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#1) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)
--- Additional comment from Vijay Bellur on 2016-05-16 06:47:42 EDT ---
REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#2) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)
--- Additional comment from Vijay Bellur on 2016-05-18 02:18:20 EDT ---
REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#3) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)
--- Additional comment from Vijay Bellur on 2016-05-20 02:56:54 EDT ---
REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#4) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)
--- Additional comment from Vijay Bellur on 2016-05-20 03:38:32 EDT ---
REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#5) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)
Referenced Bugs:
https://bugzilla.redhat.com/show_bug.cgi?id=1333643
[Bug 1333643] Files present in the .shard folder even after deleting all
the vms from the UI
https://bugzilla.redhat.com/show_bug.cgi?id=1335858
[Bug 1335858] Files present in the .shard folder even after deleting all
the vms from the UI
--
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list