[Bugs] [Bug 1337837] New: Files present in the .shard folder even after deleting all the vms from the UI

bugzilla at redhat.com bugzilla at redhat.com
Fri May 20 08:11:36 UTC 2016


https://bugzilla.redhat.com/show_bug.cgi?id=1337837

            Bug ID: 1337837
           Summary: Files present in the .shard folder even after deleting
                    all the vms from the UI
           Product: GlusterFS
           Version: 3.7.11
         Component: sharding
          Keywords: Triaged
          Severity: medium
          Assignee: bugs at gluster.org
          Reporter: kdhananj at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org, knarra at redhat.com,
                    pkarampu at redhat.com, sabose at redhat.com
        Depends On: 1333643, 1335858



+++ This bug was initially created as a clone of Bug #1335858 +++

+++ This bug was initially created as a clone of Bug #1333643 +++

Description of problem:
I have a HC setup with templates created for both windows and linux vms. I have
created a pool out of those templates and launched vms . Now i went a head and
removed pool and the vms associated to it except the HostedEngine. I see that
there are some files still present under .shard folder plus in the mount point
too.

Files present under data storage domain:
===============================================
[root at sulphur images]# ll
total 12
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 4ec5c4d3-89b5-4b75-93e7-3cb54773850c
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:03 a0685750-13af-4723-a257-e483138675ce
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:44 f1ac00a8-836c-488f-8131-78e72911495e

.shard files present inside the data brick of the volume:
=========================================================
[root at sulphur .shard]# ll
total 7529900
-rw-rw----. 2 root root         0 Apr 26 19:34
71ce5baa-60c6-4015-846b-9d721b899ab2.1
-rw-rw----. 2 root root         0 Apr 26 19:38
71ce5baa-60c6-4015-846b-9d721b899ab2.10
-rw-rw----. 2 root root         0 Apr 26 19:39
71ce5baa-60c6-4015-846b-9d721b899ab2.11
-rw-rw----. 2 root root         0 Apr 26 19:39
71ce5baa-60c6-4015-846b-9d721b899ab2.12
-rw-rw----. 2 root root         0 Apr 26 19:40
71ce5baa-60c6-4015-846b-9d721b899ab2.13
-rw-rw----. 2 root root         0 Apr 26 19:40
71ce5baa-60c6-4015-846b-9d721b899ab2.14
-rw-rw----. 2 root root         0 Apr 26 19:41
71ce5baa-60c6-4015-846b-9d721b899ab2.15
-rw-rw----. 2 root root         0 Apr 26 19:41
71ce5baa-60c6-4015-846b-9d721b899ab2.16
-rw-rw----. 2 root root         0 Apr 26 19:42
71ce5baa-60c6-4015-846b-9d721b899ab2.17
-rw-rw----. 2 root root         0 Apr 26 19:42
71ce5baa-60c6-4015-846b-9d721b899ab2.18
-rw-rw----. 2 root root         0 Apr 26 19:42
71ce5baa-60c6-4015-846b-9d721b899ab2.19
-rw-rw----. 2 root root         0 Apr 26 19:34
71ce5baa-60c6-4015-846b-9d721b899ab2.2
-rw-rw----. 2 root root         0 Apr 26 19:43
71ce5baa-60c6-4015-846b-9d721b899ab2.20
-rw-rw----. 2 root root         0 Apr 26 19:43
71ce5baa-60c6-4015-846b-9d721b899ab2.21
-rw-rw----. 2 root root         0 Apr 26 19:44
71ce5baa-60c6-4015-846b-9d721b899ab2.22
-rw-rw----. 2 root root         0 Apr 26 19:44
71ce5baa-60c6-4015-846b-9d721b899ab2.23
-rw-rw----. 2 root root         0 Apr 26 19:45
71ce5baa-60c6-4015-846b-9d721b899ab2.24
-rw-rw----. 2 root root         0 Apr 26 19:21
71ce5baa-60c6-4015-846b-9d721b899ab2.25
-rw-rw----. 2 root root         0 Apr 26 19:46
71ce5baa-60c6-4015-846b-9d721b899ab2.26
-rw-rw----. 2 root root         0 Apr 26 19:46
71ce5baa-60c6-4015-846b-9d721b899ab2.27
-rw-rw----. 2 root root         0 Apr 26 19:47
71ce5baa-60c6-4015-846b-9d721b899ab2.28
-rw-rw----. 2 root root         0 Apr 26 19:47
71ce5baa-60c6-4015-846b-9d721b899ab2.29
-rw-rw----. 2 root root         0 Apr 26 19:35
71ce5baa-60c6-4015-846b-9d721b899ab2.3
-rw-rw----. 2 root root         0 Apr 26 19:48
71ce5baa-60c6-4015-846b-9d721b899ab2.30
-rw-rw----. 2 root root         0 Apr 26 19:48
71ce5baa-60c6-4015-846b-9d721b899ab2.31
-rw-rw----. 2 root root         0 Apr 26 19:49
71ce5baa-60c6-4015-846b-9d721b899ab2.32
-rw-rw----. 2 root root         0 Apr 26 19:49
71ce5baa-60c6-4015-846b-9d721b899ab2.33
-rw-rw----. 2 root root         0 Apr 26 19:50
71ce5baa-60c6-4015-846b-9d721b899ab2.34
-rw-rw----. 2 root root         0 Apr 26 19:50
71ce5baa-60c6-4015-846b-9d721b899ab2.35
-rw-rw----. 2 root root         0 Apr 26 19:50
71ce5baa-60c6-4015-846b-9d721b899ab2.36
-rw-rw----. 2 root root         0 Apr 26 19:51
71ce5baa-60c6-4015-846b-9d721b899ab2.37
-rw-rw----. 2 root root         0 Apr 26 19:51
71ce5baa-60c6-4015-846b-9d721b899ab2.38
-rw-rw----. 2 root root         0 Apr 26 19:52
71ce5baa-60c6-4015-846b-9d721b899ab2.39
-rw-rw----. 2 root root         0 Apr 26 19:35
71ce5baa-60c6-4015-846b-9d721b899ab2.4
-rw-rw----. 2 root root         0 Apr 26 19:52
71ce5baa-60c6-4015-846b-9d721b899ab2.40
-rw-rw----. 2 root root         0 Apr 26 19:53
71ce5baa-60c6-4015-846b-9d721b899ab2.41
-rw-rw----. 2 root root         0 Apr 26 19:53
71ce5baa-60c6-4015-846b-9d721b899ab2.42
-rw-rw----. 2 root root         0 Apr 26 19:54
71ce5baa-60c6-4015-846b-9d721b899ab2.43
-rw-rw----. 2 root root         0 Apr 26 19:54
71ce5baa-60c6-4015-846b-9d721b899ab2.44
-rw-rw----. 2 root root         0 Apr 26 19:55
71ce5baa-60c6-4015-846b-9d721b899ab2.45
-rw-rw----. 2 root root         0 Apr 26 19:55
71ce5baa-60c6-4015-846b-9d721b899ab2.46
-rw-rw----. 2 root root         0 Apr 26 19:56
71ce5baa-60c6-4015-846b-9d721b899ab2.47
-rw-rw----. 2 root root         0 Apr 26 19:56
71ce5baa-60c6-4015-846b-9d721b899ab2.48
-rw-rw----. 2 root root         0 Apr 26 19:57
71ce5baa-60c6-4015-846b-9d721b899ab2.49
-rw-rw----. 2 root root         0 Apr 26 19:36
71ce5baa-60c6-4015-846b-9d721b899ab2.5
-rw-rw----. 2 root root         0 Apr 26 19:21
71ce5baa-60c6-4015-846b-9d721b899ab2.50
-rw-rw----. 2 root root         0 Apr 26 19:57
71ce5baa-60c6-4015-846b-9d721b899ab2.51
-rw-rw----. 2 root root         0 Apr 26 19:58
71ce5baa-60c6-4015-846b-9d721b899ab2.52
-rw-rw----. 2 root root         0 Apr 26 19:58
71ce5baa-60c6-4015-846b-9d721b899ab2.53
-rw-rw----. 2 root root         0 Apr 26 19:59
71ce5baa-60c6-4015-846b-9d721b899ab2.54
-rw-rw----. 2 root root         0 Apr 26 19:59
71ce5baa-60c6-4015-846b-9d721b899ab2.55
-rw-rw----. 2 root root         0 Apr 26 20:00
71ce5baa-60c6-4015-846b-9d721b899ab2.56
-rw-rw----. 2 root root         0 Apr 26 20:00
71ce5baa-60c6-4015-846b-9d721b899ab2.57
-rw-rw----. 2 root root         0 Apr 26 20:01
71ce5baa-60c6-4015-846b-9d721b899ab2.58
-rw-rw----. 2 root root         0 Apr 26 20:01
71ce5baa-60c6-4015-846b-9d721b899ab2.59
-rw-rw----. 2 root root         0 Apr 26 19:36
71ce5baa-60c6-4015-846b-9d721b899ab2.6
-rw-rw----. 2 root root         0 Apr 26 20:02
71ce5baa-60c6-4015-846b-9d721b899ab2.60
-rw-rw----. 2 root root         0 Apr 26 20:02
71ce5baa-60c6-4015-846b-9d721b899ab2.61
-rw-rw----. 2 root root         0 Apr 26 20:03
71ce5baa-60c6-4015-846b-9d721b899ab2.62
-rw-rw----. 2 root root         0 Apr 26 20:03
71ce5baa-60c6-4015-846b-9d721b899ab2.63
-rw-rw----. 2 root root         0 Apr 26 20:04
71ce5baa-60c6-4015-846b-9d721b899ab2.64
-rw-rw----. 2 root root         0 Apr 26 20:04
71ce5baa-60c6-4015-846b-9d721b899ab2.65
-rw-rw----. 2 root root         0 Apr 26 20:04
71ce5baa-60c6-4015-846b-9d721b899ab2.66
-rw-rw----. 2 root root         0 Apr 26 20:05
71ce5baa-60c6-4015-846b-9d721b899ab2.67
-rw-rw----. 2 root root         0 Apr 26 20:05
71ce5baa-60c6-4015-846b-9d721b899ab2.68
-rw-rw----. 2 root root         0 Apr 26 20:06
71ce5baa-60c6-4015-846b-9d721b899ab2.69
-rw-rw----. 2 root root         0 Apr 26 19:37
71ce5baa-60c6-4015-846b-9d721b899ab2.7
-rw-rw----. 2 root root         0 Apr 26 20:06
71ce5baa-60c6-4015-846b-9d721b899ab2.70
-rw-rw----. 2 root root         0 Apr 26 20:07
71ce5baa-60c6-4015-846b-9d721b899ab2.71
-rw-rw----. 2 root root         0 Apr 26 20:07
71ce5baa-60c6-4015-846b-9d721b899ab2.72
-rw-rw----. 2 root root         0 Apr 26 20:08
71ce5baa-60c6-4015-846b-9d721b899ab2.73
-rw-rw----. 2 root root         0 Apr 26 20:08
71ce5baa-60c6-4015-846b-9d721b899ab2.74
-rw-rw----. 2 root root         0 Apr 26 19:21
71ce5baa-60c6-4015-846b-9d721b899ab2.75
-rw-rw----. 2 root root         0 Apr 26 20:09
71ce5baa-60c6-4015-846b-9d721b899ab2.76
-rw-rw----. 2 root root         0 Apr 26 20:10
71ce5baa-60c6-4015-846b-9d721b899ab2.77
-rw-rw----. 2 root root         0 Apr 26 20:10
71ce5baa-60c6-4015-846b-9d721b899ab2.78
-rw-rw----. 2 root root         0 Apr 26 20:11
71ce5baa-60c6-4015-846b-9d721b899ab2.79
-rw-rw----. 2 root root         0 Apr 26 19:37
71ce5baa-60c6-4015-846b-9d721b899ab2.8
-rw-rw----. 2 root root         0 Apr 26 20:11
71ce5baa-60c6-4015-846b-9d721b899ab2.80
-rw-rw----. 2 root root         0 Apr 26 20:11
71ce5baa-60c6-4015-846b-9d721b899ab2.81
-rw-rw----. 2 root root         0 Apr 26 20:12
71ce5baa-60c6-4015-846b-9d721b899ab2.82
-rw-rw----. 2 root root         0 Apr 26 20:12
71ce5baa-60c6-4015-846b-9d721b899ab2.83
-rw-rw----. 2 root root         0 Apr 26 20:13
71ce5baa-60c6-4015-846b-9d721b899ab2.84
-rw-rw----. 2 root root         0 Apr 26 20:13
71ce5baa-60c6-4015-846b-9d721b899ab2.85
-rw-rw----. 2 root root         0 Apr 26 20:14
71ce5baa-60c6-4015-846b-9d721b899ab2.86
-rw-rw----. 2 root root         0 Apr 26 20:14
71ce5baa-60c6-4015-846b-9d721b899ab2.87
-rw-rw----. 2 root root         0 Apr 26 20:15
71ce5baa-60c6-4015-846b-9d721b899ab2.88
-rw-rw----. 2 root root         0 Apr 26 20:15
71ce5baa-60c6-4015-846b-9d721b899ab2.89
-rw-rw----. 2 root root         0 Apr 26 19:38
71ce5baa-60c6-4015-846b-9d721b899ab2.9
-rw-rw----. 2 root root         0 Apr 26 20:16
71ce5baa-60c6-4015-846b-9d721b899ab2.90
-rw-rw----. 2 root root         0 Apr 26 20:16
71ce5baa-60c6-4015-846b-9d721b899ab2.91
-rw-rw----. 2 root root         0 Apr 26 20:17
71ce5baa-60c6-4015-846b-9d721b899ab2.92
-rw-rw----. 2 root root         0 Apr 26 20:17
71ce5baa-60c6-4015-846b-9d721b899ab2.93
-rw-rw----. 2 root root         0 Apr 26 20:18
71ce5baa-60c6-4015-846b-9d721b899ab2.94
-rw-rw----. 2 root root         0 Apr 26 20:18
71ce5baa-60c6-4015-846b-9d721b899ab2.95
-rw-rw----. 2 root root         0 Apr 26 20:19
71ce5baa-60c6-4015-846b-9d721b899ab2.96
-rw-rw----. 2 root root         0 Apr 26 20:19
71ce5baa-60c6-4015-846b-9d721b899ab2.97
-rw-rw----. 2 root root         0 Apr 26 20:19
71ce5baa-60c6-4015-846b-9d721b899ab2.98
-rw-rw----. 2 root root         0 Apr 13 11:15
71ce5baa-60c6-4015-846b-9d721b899ab2.99
-rw-rw----. 2 root root 536870912 May  4 17:58
f9333cb0-eab1-4899-be93-86d03d75ee0e.1
-rw-rw----. 2 root root 536870912 May  4 17:27
f9333cb0-eab1-4899-be93-86d03d75ee0e.10
-rw-rw----. 2 root root 536870912 May  4 17:26
f9333cb0-eab1-4899-be93-86d03d75ee0e.11
-rw-rw----. 2 root root 536870912 May  4 17:26
f9333cb0-eab1-4899-be93-86d03d75ee0e.12
-rw-rw----. 2 root root  47284224 May  4 17:27
f9333cb0-eab1-4899-be93-86d03d75ee0e.13
-rw-rw----. 2 root root         0 May  4 17:11
f9333cb0-eab1-4899-be93-86d03d75ee0e.14
-rw-rw----. 2 root root         0 May  4 17:11
f9333cb0-eab1-4899-be93-86d03d75ee0e.15
-rw-rw----. 2 root root 536870912 May  4 17:53
f9333cb0-eab1-4899-be93-86d03d75ee0e.16
-rw-rw----. 2 root root 536870912 May  4 17:54
f9333cb0-eab1-4899-be93-86d03d75ee0e.17
-rw-rw----. 2 root root         0 May  4 17:11
f9333cb0-eab1-4899-be93-86d03d75ee0e.18
-rw-rw----. 2 root root 143769600 May  4 17:55
f9333cb0-eab1-4899-be93-86d03d75ee0e.19
-rw-rw----. 2 root root 536870912 May  5 10:15
f9333cb0-eab1-4899-be93-86d03d75ee0e.2
-rw-rw----. 2 root root    802816 May  4 16:46
f9333cb0-eab1-4899-be93-86d03d75ee0e.25
-rw-rw----. 2 root root 536870912 May  4 18:00
f9333cb0-eab1-4899-be93-86d03d75ee0e.3
-rw-rw----. 2 root root 536870912 May  4 18:01
f9333cb0-eab1-4899-be93-86d03d75ee0e.4
-rw-rw----. 2 root root 536870912 May  4 16:57
f9333cb0-eab1-4899-be93-86d03d75ee0e.5
-rw-rw----. 2 root root  26750976 May  4 18:33
f9333cb0-eab1-4899-be93-86d03d75ee0e.50
-rw-rw----. 2 root root 536870912 May  4 17:20
f9333cb0-eab1-4899-be93-86d03d75ee0e.6
-rw-rw----. 2 root root 536870912 May  5 10:15
f9333cb0-eab1-4899-be93-86d03d75ee0e.7
-rw-rw----. 2 root root    278528 May  4 16:46
f9333cb0-eab1-4899-be93-86d03d75ee0e.75
-rw-rw----. 2 root root 536870912 May  5 10:15
f9333cb0-eab1-4899-be93-86d03d75ee0e.8
-rw-rw----. 2 root root 536870912 May  4 17:27
f9333cb0-eab1-4899-be93-86d03d75ee0e.9
-rw-rw----. 2 root root 536870912 May  4 16:46
f9333cb0-eab1-4899-be93-86d03d75ee0e.99


Images under vmstore storage domain:
======================================
[root at sulphur images]# ll
total 16
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 11:02 210ca44a-d0e3-4750-b3d3-4b1358df280b
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 5dc92a56-ee20-4d2e-a6bd-9f3753263029
drwxr-xr-x. 2 vdsm kvm 4096 Apr 13 12:14 8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506
drwxr-xr-x. 2 vdsm kvm 4096 May  5 11:45 bf3ed8c9-fac0-4f42-931b-9c92c9af2fe5


.shard files under vmstore brick:
======================================
[root at sulphur .shard]# ll
total 9055212
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.1
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.10
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.11
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.12
-rw-rw----. 2 root root 536870912 Apr 13 12:13
026de0b6-9bea-4b14-8b6f-b7261eb6c030.13
-rw-rw----. 2 root root 536870912 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.14
-rw-rw----. 2 root root 375697408 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.15
-rw-rw----. 2 root root         0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.16
-rw-rw----. 2 root root         0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.17
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.18
-rw-rw----. 2 root root         0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.19
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.2
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.20
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.21
-rw-rw----. 2 root root         0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.22
-rw-rw----. 2 root root 536870912 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.23
-rw-rw----. 2 root root 536870912 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.24
-rw-rw----. 2 root root  77815808 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.25
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.26
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.27
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.28
-rw-rw----. 2 root root         0 Apr 15 21:50
026de0b6-9bea-4b14-8b6f-b7261eb6c030.29
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.3
-rw-rw----. 2 root root         0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.30
-rw-rw----. 2 root root         0 Apr 13 12:25
026de0b6-9bea-4b14-8b6f-b7261eb6c030.31
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.32
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.33
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.34
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.35
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.36
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.37
-rw-rw----. 2 root root         0 Apr 13 12:30
026de0b6-9bea-4b14-8b6f-b7261eb6c030.38
-rw-rw----. 2 root root 535822336 Apr 13 12:14
026de0b6-9bea-4b14-8b6f-b7261eb6c030.39
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.4
-rw-rw----. 2 root root 536870912 Apr 13 12:11
026de0b6-9bea-4b14-8b6f-b7261eb6c030.5
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.6
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.7
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.8
-rw-rw----. 2 root root 536870912 Apr 13 12:12
026de0b6-9bea-4b14-8b6f-b7261eb6c030.9
-rw-rw----. 2 root root         0 May  2 20:19
14215e87-1c82-45ec-9a34-4a8bb028d86f.1
-rw-rw----. 2 root root         0 May  2 20:23
14215e87-1c82-45ec-9a34-4a8bb028d86f.10
-rw-rw----. 2 root root         0 May  2 20:23
14215e87-1c82-45ec-9a34-4a8bb028d86f.11
-rw-rw----. 2 root root         0 May  2 20:24
14215e87-1c82-45ec-9a34-4a8bb028d86f.12
-rw-rw----. 2 root root 536870912 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.13
-rw-rw----. 2 root root  94326784 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.14
-rw-rw----. 2 root root         0 Apr 29 16:54
14215e87-1c82-45ec-9a34-4a8bb028d86f.15
-rw-rw----. 2 root root         0 Apr 29 16:55
14215e87-1c82-45ec-9a34-4a8bb028d86f.16
-rw-rw----. 2 root root         0 Apr 29 16:55
14215e87-1c82-45ec-9a34-4a8bb028d86f.17
-rw-rw----. 2 root root         0 May  2 20:26
14215e87-1c82-45ec-9a34-4a8bb028d86f.18
-rw-rw----. 2 root root         0 May  2 20:26
14215e87-1c82-45ec-9a34-4a8bb028d86f.19
-rw-rw----. 2 root root         0 May  2 20:20
14215e87-1c82-45ec-9a34-4a8bb028d86f.2
-rw-rw----. 2 root root         0 May  2 20:26
14215e87-1c82-45ec-9a34-4a8bb028d86f.20
-rw-rw----. 2 root root         0 May  2 20:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.21
-rw-rw----. 2 root root 499495424 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.22
-rw-rw----. 2 root root         0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.23
-rw-rw----. 2 root root         0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.24
-rw-rw----. 2 root root         0 Apr 26 19:28
14215e87-1c82-45ec-9a34-4a8bb028d86f.25
-rw-rw----. 2 root root         0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.26
-rw-rw----. 2 root root         0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.27
-rw-rw----. 2 root root         0 Apr 26 19:28
14215e87-1c82-45ec-9a34-4a8bb028d86f.28
-rw-rw----. 2 root root         0 Apr 26 19:27
14215e87-1c82-45ec-9a34-4a8bb028d86f.29
-rw-rw----. 2 root root         0 May  2 20:20
14215e87-1c82-45ec-9a34-4a8bb028d86f.3
-rw-rw----. 2 root root         0 Apr 26 19:28
14215e87-1c82-45ec-9a34-4a8bb028d86f.30
-rw-rw----. 2 root root 372343296 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.31
-rw-rw----. 2 root root         0 May  2 20:31
14215e87-1c82-45ec-9a34-4a8bb028d86f.32
-rw-rw----. 2 root root         0 May  2 20:31
14215e87-1c82-45ec-9a34-4a8bb028d86f.33
-rw-rw----. 2 root root         0 May  2 20:32
14215e87-1c82-45ec-9a34-4a8bb028d86f.34
-rw-rw----. 2 root root         0 May  2 20:32
14215e87-1c82-45ec-9a34-4a8bb028d86f.35
-rw-rw----. 2 root root         0 May  2 20:32
14215e87-1c82-45ec-9a34-4a8bb028d86f.36
-rw-rw----. 2 root root         0 May  2 20:33
14215e87-1c82-45ec-9a34-4a8bb028d86f.37
-rw-rw----. 2 root root         0 May  2 20:33
14215e87-1c82-45ec-9a34-4a8bb028d86f.38
-rw-rw----. 2 root root         0 Apr 13 11:15
14215e87-1c82-45ec-9a34-4a8bb028d86f.39
-rw-rw----. 2 root root 536869888 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.4
-rw-rw----. 2 root root 214117376 Apr 13 11:02
14215e87-1c82-45ec-9a34-4a8bb028d86f.5
-rw-rw----. 2 root root         0 Apr 29 19:24
14215e87-1c82-45ec-9a34-4a8bb028d86f.6
-rw-rw----. 2 root root         0 Apr 29 19:24
14215e87-1c82-45ec-9a34-4a8bb028d86f.7
-rw-rw----. 2 root root         0 May  2 20:22
14215e87-1c82-45ec-9a34-4a8bb028d86f.8
-rw-rw----. 2 root root         0 May  2 20:22
14215e87-1c82-45ec-9a34-4a8bb028d86f.9


I ran the command virsh -r list command to make sure that there are no other
vms running other than HostedEngine from all the nodes:

[root at sulphur ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------
 616   HostedEngine                   running


[root at tettnang ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------

[root at zod ~]# virsh -r list
 Id    Name                           State
----------------------------------------------------






Version-Release number of selected component (if applicable):
glusterfs-3.7.9-2.el7rhgs.x86_64

How reproducible:
Tried it once

Steps to Reproduce:
1. Create vms in HC setup using RHEV templates and Pools.
2. Now delete the pool so that all the vms are removed except HostedEngine
3.

Actual results:
I still see that there are some image files present inside the storage domain
plus entries under .shard folder for both data and vmstore volumes.

Expected results:
When all the virtual machines are removed ideally there should not be any
entries present in the .shard folder and in the storage domain. 

Additional info:


--- Additional comment from Pranith Kumar K on 2016-05-12 06:02:21 EDT ---

RCA:

Handling of hardlinks in sharding xlator:
Sharding translator performs hardlinks only on the base-file. When the last
link is deleted, it deletes all the shard files associated with that base-file.
So when an unlink fop comes to shard, it does a lookup (Equivalent of stat for
user applications) to find out the number of links. If the number of links is >
1 then it just deletes that link but doesn't shards. If the number of links is
1, then it goes ahead and deletes all the shards as well.

What happens in the rhev-setup on VM deletion:
Kasturi (QE) selected all the VMs at once and performed delete. All these vm
data will be moved to remove_me_xyz directory and host which is the owner of
that VM will start doing unlinks in the background.

It seems like there are hardlinks on the setup where VMs are stored, where
links are shared between different hosts.
[root at sulphur vmstore-brick1]# find . -type f | xargs getfattr -d -m. -e hex |
grep trusted.gfid | sort | uniq -c | grep -v " 2 "
     17 trusted.gfid=0x026de0b69bea4b148b6fb7261eb6c030
      1 trusted.gfid=0x122c529ce3ab440f87edc2cf9ef79051
      1 trusted.gfid=0x1383cf29af214231bbaafd3120fd4c27
     16 trusted.gfid=0x13fd032531be41d4a5e8cff2aff02aca
     16 trusted.gfid=0x14215e871c8245ec9a344a8bb028d86f
      1 trusted.gfid=0x14807a3c2c0b4f6b92cea4a042856971
     16 trusted.gfid=0x2b294ed3423c4e43afee404315a316ab
      1 trusted.gfid=0x445a3b7cfb444fb290352b9087dc105b
      1 trusted.gfid=0x4a21a35d946d4fe88283f3a2a29992cd
      1 trusted.gfid=0x4ae3729419894b3bbb08467e32a088e7
      1 trusted.gfid=0x655f5fdf9a59499bb3d329a7ab23ff5e
      1 trusted.gfid=0x710d1bbe94e54a60ac64d916b1df672c
      1 trusted.gfid=0x97abfcae67ff420797ad3014ce084e2e
      1 trusted.gfid=0xc5816599efb547cca48a4a58a9751844
     17 trusted.gfid=0xd939b5a7aa29453e814e32c0cf6e2608
      1 trusted.gfid=0xd94592fb54ef46dbbf3da881ac94e465
     17 trusted.gfid=0xdec2404ec7574494a78598ea4bd31381
      1 trusted.gfid=0xfff5f116ef9140078c6536f23197695f
[root at sulphur vmstore-brick1]# ls -lih
.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
469762591 -rw-rw----. 17 vdsm kvm 512M Apr 13 12:10
.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
[root at sulphur vmstore-brick1]# find . -inum 469762591
./.glusterfs/02/6d/026de0b6-9bea-4b14-8b6f-b7261eb6c030
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/8d0b02a5-d0f1-42e4-bc10-9baa9d0d5506/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5308d74e-28e2-4158-b2f9-af898e14e6d0/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/7ca57bac-c614-402e-9bb4-0d172d69c476/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/5ee9dc38-f4a5-41ec-bd95-878165c022b1/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/99f4c43a-dc2f-46f1-b4b4-2cd289f43757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/bedfd824-6560-454a-9ed1-6baf3e592428/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/9965825b-3e79-434a-8eec-b85994901ce5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/548b241a-7958-40f4-b84a-161628f4e0e5/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/394f07f9-baab-4299-ba76-a13b9d2a949f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/6e97619b-bbfb-4856-b63c-05727f84556f/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ff21f8ba-5038-42f6-9b29-d56d87d87b21/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/3f61feef-0eb3-4cee-875b-d2e22dc8e7e3/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/a96768b4-548c-4c9d-80ac-fb4552e97a94/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/ae9715b8-c6a7-4125-948f-877fba1a7757/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d
./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d


When all the VMs are deleted together, it can lead to different hosts trying to
delete hardlinks in parallel in their respective remove_me_xxx directory. i.e.
host1 can delete
"./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/280f77cd-63b8-4cc9-9449-b10d1c5c36e1/fe3a369e-567b-4d46-a82d-5dd309af0f5d"
where as host2 can delete
"./297a9b9c-4396-4b30-8bfe-976a67d49a74/images/f8de7123-c638-4ee7-9ce0-c87bad52141c/fe3a369e-567b-4d46-a82d-5dd309af0f5d".
which are hardlinks to each other. So at the time when shard queries how many
links the file has by doing lookup both of them will get a number greater than
1, so they both go ahead and delete the file. But one of those unlinks will be
the last one but shard has no clue about this because at the time when it
queried number of links is more than 1. This leads to shard not deleting
shards. This is the reason for the bug.

We have seen files with hardlink count '1' which suggests there is a bug even
in posix-xlator which is not deleting gfid-link thinking it is not the last
link.

Pranith

--- Additional comment from Vijay Bellur on 2016-05-13 08:07:18 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#1) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-16 06:47:42 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#2) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-18 02:18:20 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#3) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-20 02:56:54 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#4) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)

--- Additional comment from Vijay Bellur on 2016-05-20 03:38:32 EDT ---

REVIEW: http://review.gluster.org/14334 (features/shard: Get hard-link-count in
{unlink,rename}_cbk before deleting shards) posted (#5) for review on master by
Krutika Dhananjay (kdhananj at redhat.com)


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1333643
[Bug 1333643] Files present in the .shard folder even after deleting all
the vms from the UI
https://bugzilla.redhat.com/show_bug.cgi?id=1335858
[Bug 1335858] Files present in the .shard folder even after deleting all
the vms from the UI
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list