[Gluster-devel] Glusterfs meta data space consumption issue

Ashish Pandey aspandey at redhat.com
Fri Apr 7 08:01:00 UTC 2017


Are you sure that the bricks which you used for this volume was not having any previous data? 
Find out the total number of files and directories on your mount point (recursively) and then see the number of entries on .glusterfs/ 


----- Original Message -----

From: "ABHISHEK PALIWAL" <abhishpaliwal at gmail.com> 
To: "Gluster Devel" <gluster-devel at gluster.org>, "gluster-users" <gluster-users at gluster.org> 
Sent: Friday, April 7, 2017 12:15:22 PM 
Subject: Re: [Gluster-devel] Glusterfs meta data space consumption issue 



Is there any update ?? 

On Thu, Apr 6, 2017 at 12:45 PM, ABHISHEK PALIWAL < abhishpaliwal at gmail.com > wrote: 



Hi, 
We are currently experiencing a serious issue w.r.t volume space usage by glusterfs. 
In the below outputs, we can see that the size of the real data in /c (glusterfs volume) is nearly 1GB but the “.glusterfs” directory inside the brick (i.e., “/opt/lvmdir/c2/brick”) is consuming around 3.4 GB 
Can you tell us why the volume space is fully used by glusterfs even though the real data size is around 1GB itself ? 
# gluster peer status 
Number of Peers: 0 
# 
# 
# gluster volume status 
Status of volume: c_glusterfs 
Gluster process TCP Port RDMA Port Online Pid 
------------------------------------------------------------------------------ 
Brick 10.32.0.48:/opt/lvmdir/c2/brick 49152 0 Y 1507 
Task Status of Volume c_glusterfs 
------------------------------------------------------------------------------ 
There are no active volume tasks 
# gluster volume info 
Volume Name: c_glusterfs 
Type: Distribute 
Volume ID: d83b1b8c-bc37-4615-bf4b-529f56968ecc 
Status: Started 
Number of Bricks: 1 
Transport-type: tcp 
Bricks: 
Brick1: 10.32.0.48:/opt/lvmdir/c2/brick 
Options Reconfigured: 
nfs.disable: on 
network.ping-timeout: 4 
performance.readdir-ahead: on 
# 
# ls -a /c/ 
. .. .trashcan RNC_Exceptions configuration java license loadmodules loadmodules_norepl loadmodules_tftp logfiles lost+found node_id pm_data pmd public_html rnc security systemfiles tmp toplog.txt up usr xb 
# du -sh /c/.trashcan/ 
8.0K /c/.trashcan/ 
# du -sh /c/* 
11K /c/RNC_Exceptions 
5.5K /c/configuration 
62M /c/java 
138K /c/license 
609M /c/loadmodules 
90M /c/loadmodules_norepl 
246M /c/loadmodules_tftp 
4.1M /c/logfiles 
4.0K /c/lost+found 
5.0K /c/node_id 
8.0K /c/pm_data 
4.5K /c/pmd 
9.1M /c/public_html 
113K /c/rnc 
16K /c/security 
1.3M /c/systemfiles 
228K /c/tmp 
75K /c/toplog.txt 
1.5M /c/up 
4.0K /c/usr 
4.0K /c/xb 
# du -sh /c/ 
1022M /c/ 
# df -h /c/ 
Filesystem Size Used Avail Use% Mounted on 
10.32.0.48:c_glusterfs 3.6G 3.4G 0 100% /mnt/c 
# 
# 
# 
# du -sh /opt/lvmdir/c2/brick/ 
3.4G /opt/lvmdir/c2/brick/ 
# du -sh /opt/lvmdir/c2/brick/* 
112K /opt/lvmdir/c2/brick/RNC_Exceptions 
36K /opt/lvmdir/c2/brick/configuration 
63M /opt/lvmdir/c2/brick/java 
176K /opt/lvmdir/c2/brick/license 
610M /opt/lvmdir/c2/brick/loadmodules 
95M /opt/lvmdir/c2/brick/loadmodules_norepl 
246M /opt/lvmdir/c2/brick/loadmodules_tftp 
4.2M /opt/lvmdir/c2/brick/logfiles 
8.0K /opt/lvmdir/c2/brick/lost+found 
24K /opt/lvmdir/c2/brick/node_id 
16K /opt/lvmdir/c2/brick/pm_data 
16K /opt/lvmdir/c2/brick/pmd 
9.2M /opt/lvmdir/c2/brick/public_html 
268K /opt/lvmdir/c2/brick/rnc 
80K /opt/lvmdir/c2/brick/security 
1.4M /opt/lvmdir/c2/brick/systemfiles 
252K /opt/lvmdir/c2/brick/tmp 
80K /opt/lvmdir/c2/brick/toplog.txt 
1.5M /opt/lvmdir/c2/brick/up 
8.0K /opt/lvmdir/c2/brick/usr 
8.0K /opt/lvmdir/c2/brick/xb 
# du -sh /opt/lvmdir/c2/brick/.glusterfs/ 
3.4G /opt/lvmdir/c2/brick/.glusterfs/ 

Below are the statics of the below command " du -sh /opt/lvmdir/c2/brick/.glusterfs /*" 

# du -sh /opt/lvmdir/c2/brick/.glusterfs/* 
14M /opt/lvmdir/c2/brick/.glusterfs/00 
8.3M /opt/lvmdir/c2/brick/.glusterfs/01 
23M /opt/lvmdir/c2/brick/.glusterfs/02 
17M /opt/lvmdir/c2/brick/.glusterfs/03 
7.1M /opt/lvmdir/c2/brick/.glusterfs/04 
336K /opt/lvmdir/c2/brick/.glusterfs/05 
3.5M /opt/lvmdir/c2/brick/.glusterfs/06 
1.7M /opt/lvmdir/c2/brick/.glusterfs/07 
154M /opt/lvmdir/c2/brick/.glusterfs/08 
14M /opt/lvmdir/c2/brick/.glusterfs/09 
9.5M /opt/lvmdir/c2/brick/.glusterfs/0a 
5.5M /opt/lvmdir/c2/brick/.glusterfs/0b 
11M /opt/lvmdir/c2/brick/.glusterfs/0c 
764K /opt/lvmdir/c2/brick/.glusterfs/0d 
69M /opt/lvmdir/c2/brick/.glusterfs/0e 
3.7M /opt/lvmdir/c2/brick/.glusterfs/0f 
14M /opt/lvmdir/c2/brick/.glusterfs/10 
1.8M /opt/lvmdir/c2/brick/.glusterfs/11 
5.0M /opt/lvmdir/c2/brick/.glusterfs/12 
18M /opt/lvmdir/c2/brick/.glusterfs/13 
7.8M /opt/lvmdir/c2/brick/.glusterfs/14 
151M /opt/lvmdir/c2/brick/.glusterfs/15 
15M /opt/lvmdir/c2/brick/.glusterfs/16 
9.0M /opt/lvmdir/c2/brick/.glusterfs/17 
11M /opt/lvmdir/c2/brick/.glusterfs/18 
3.1M /opt/lvmdir/c2/brick/.glusterfs/19 
8.1M /opt/lvmdir/c2/brick/.glusterfs/1a 
82M /opt/lvmdir/c2/brick/.glusterfs/1b 
95M /opt/lvmdir/c2/brick/.glusterfs/1c 
2.1M /opt/lvmdir/c2/brick/.glusterfs/1d 
4.5M /opt/lvmdir/c2/brick/.glusterfs/1e 
9.7M /opt/lvmdir/c2/brick/.glusterfs/1f 
13M /opt/lvmdir/c2/brick/.glusterfs/20 
54M /opt/lvmdir/c2/brick/.glusterfs/21 
9.4M /opt/lvmdir/c2/brick/.glusterfs/22 
2.3M /opt/lvmdir/c2/brick/.glusterfs/23 
14M /opt/lvmdir/c2/brick/.glusterfs/24 
12M /opt/lvmdir/c2/brick/.glusterfs/25 
56M /opt/lvmdir/c2/brick/.glusterfs/26 
5.6M /opt/lvmdir/c2/brick/.glusterfs/27 
2.5M /opt/lvmdir/c2/brick/.glusterfs/28 
55M /opt/lvmdir/c2/brick/.glusterfs/29 
47M /opt/lvmdir/c2/brick/.glusterfs/2a 
6.1M /opt/lvmdir/c2/brick/.glusterfs/2b 
2.2M /opt/lvmdir/c2/brick/.glusterfs/2c 
4.7M /opt/lvmdir/c2/brick/.glusterfs/2d 
52M /opt/lvmdir/c2/brick/.glusterfs/2e 
16M /opt/lvmdir/c2/brick/.glusterfs/2f 
2.3M /opt/lvmdir/c2/brick/.glusterfs/30 
7.6M /opt/lvmdir/c2/brick/.glusterfs/31 
12M /opt/lvmdir/c2/brick/.glusterfs/32 
2.3M /opt/lvmdir/c2/brick/.glusterfs/33 
4.5M /opt/lvmdir/c2/brick/.glusterfs/34 
1.5M /opt/lvmdir/c2/brick/.glusterfs/35 
24M /opt/lvmdir/c2/brick/.glusterfs/36 
6.8M /opt/lvmdir/c2/brick/.glusterfs/37 
10M /opt/lvmdir/c2/brick/.glusterfs/38 
5.0M /opt/lvmdir/c2/brick/.glusterfs/39 
21M /opt/lvmdir/c2/brick/.glusterfs/3a 
9.6M /opt/lvmdir/c2/brick/.glusterfs/3b 
19M /opt/lvmdir/c2/brick/.glusterfs/3c 
8.0M /opt/lvmdir/c2/brick/.glusterfs/3d 
9.3M /opt/lvmdir/c2/brick/.glusterfs/3e 
9.9M /opt/lvmdir/c2/brick/.glusterfs/3f 
952K /opt/lvmdir/c2/brick/.glusterfs/40 
10M /opt/lvmdir/c2/brick/.glusterfs/41 
17M /opt/lvmdir/c2/brick/.glusterfs/42 
20M /opt/lvmdir/c2/brick/.glusterfs/43 
1.7M /opt/lvmdir/c2/brick/.glusterfs/44 
7.1M /opt/lvmdir/c2/brick/.glusterfs/45 
29M /opt/lvmdir/c2/brick/.glusterfs/46 
4.5M /opt/lvmdir/c2/brick/.glusterfs/47 
17M /opt/lvmdir/c2/brick/.glusterfs/48 
9.0M /opt/lvmdir/c2/brick/.glusterfs/49 
4.4M /opt/lvmdir/c2/brick/.glusterfs/4a 
9.7M /opt/lvmdir/c2/brick/.glusterfs/4b 
3.1M /opt/lvmdir/c2/brick/.glusterfs/4c 
1.9M /opt/lvmdir/c2/brick/.glusterfs/4d 
9.7M /opt/lvmdir/c2/brick/.glusterfs/4e 
648K /opt/lvmdir/c2/brick/.glusterfs/4f 
8.0M /opt/lvmdir/c2/brick/.glusterfs/50 
14M /opt/lvmdir/c2/brick/.glusterfs/51 
1.9M /opt/lvmdir/c2/brick/.glusterfs/52 
1.6M /opt/lvmdir/c2/brick/.glusterfs/53 
4.4M /opt/lvmdir/c2/brick/.glusterfs/54 
3.3M /opt/lvmdir/c2/brick/.glusterfs/55 
6.1M /opt/lvmdir/c2/brick/.glusterfs/56 
18M /opt/lvmdir/c2/brick/.glusterfs/57 
2.8M /opt/lvmdir/c2/brick/.glusterfs/58 
6.5M /opt/lvmdir/c2/brick/.glusterfs/59 
2.2M /opt/lvmdir/c2/brick/.glusterfs/5a 
380K /opt/lvmdir/c2/brick/.glusterfs/5b 
12M /opt/lvmdir/c2/brick/.glusterfs/5c 
2.4M /opt/lvmdir/c2/brick/.glusterfs/5d 
22M /opt/lvmdir/c2/brick/.glusterfs/5e 
5.6M /opt/lvmdir/c2/brick/.glusterfs/5f 
15M /opt/lvmdir/c2/brick/.glusterfs/60 
15M /opt/lvmdir/c2/brick/.glusterfs/61 
9.3M /opt/lvmdir/c2/brick/.glusterfs/62 
1.1M /opt/lvmdir/c2/brick/.glusterfs/63 
57M /opt/lvmdir/c2/brick/.glusterfs/64 
7.5M /opt/lvmdir/c2/brick/.glusterfs/65 
18M /opt/lvmdir/c2/brick/.glusterfs/66 
4.6M /opt/lvmdir/c2/brick/.glusterfs/67 
1.1M /opt/lvmdir/c2/brick/.glusterfs/68 
9.3M /opt/lvmdir/c2/brick/.glusterfs/69 
12M /opt/lvmdir/c2/brick/.glusterfs/6a 
18M /opt/lvmdir/c2/brick/.glusterfs/6b 
8.5M /opt/lvmdir/c2/brick/.glusterfs/6c 
1.7M /opt/lvmdir/c2/brick/.glusterfs/6d 
7.8M /opt/lvmdir/c2/brick/.glusterfs/6e 
5.4M /opt/lvmdir/c2/brick/.glusterfs/6f 
13M /opt/lvmdir/c2/brick/.glusterfs/70 
3.7M /opt/lvmdir/c2/brick/.glusterfs/71 
6.4M /opt/lvmdir/c2/brick/.glusterfs/72 
596K /opt/lvmdir/c2/brick/.glusterfs/73 
13M /opt/lvmdir/c2/brick/.glusterfs/74 
10M /opt/lvmdir/c2/brick/.glusterfs/75 
6.0M /opt/lvmdir/c2/brick/.glusterfs/76 
448K /opt/lvmdir/c2/brick/.glusterfs/77 
102M /opt/lvmdir/c2/brick/.glusterfs/78 
3.3M /opt/lvmdir/c2/brick/.glusterfs/79 
4.5M /opt/lvmdir/c2/brick/.glusterfs/7a 
5.2M /opt/lvmdir/c2/brick/.glusterfs/7b 
3.0M /opt/lvmdir/c2/brick/.glusterfs/7c 
6.7M /opt/lvmdir/c2/brick/.glusterfs/7d 
8.0M /opt/lvmdir/c2/brick/.glusterfs/7e 
10M /opt/lvmdir/c2/brick/.glusterfs/7f 
7.4M /opt/lvmdir/c2/brick/.glusterfs/80 
7.0M /opt/lvmdir/c2/brick/.glusterfs/81 
4.3M /opt/lvmdir/c2/brick/.glusterfs/82 
5.7M /opt/lvmdir/c2/brick/.glusterfs/83 
12M /opt/lvmdir/c2/brick/.glusterfs/84 
4.9M /opt/lvmdir/c2/brick/.glusterfs/85 
7.9M /opt/lvmdir/c2/brick/.glusterfs/86 
14M /opt/lvmdir/c2/brick/.glusterfs/87 
7.7M /opt/lvmdir/c2/brick/.glusterfs/88 
55M /opt/lvmdir/c2/brick/.glusterfs/89 
19M /opt/lvmdir/c2/brick/.glusterfs/8a 
7.4M /opt/lvmdir/c2/brick/.glusterfs/8b 
26M /opt/lvmdir/c2/brick/.glusterfs/8c 
2.0M /opt/lvmdir/c2/brick/.glusterfs/8d 
5.4M /opt/lvmdir/c2/brick/.glusterfs/8e 
13M /opt/lvmdir/c2/brick/.glusterfs/8f 
3.6M /opt/lvmdir/c2/brick/.glusterfs/90 
60M /opt/lvmdir/c2/brick/.glusterfs/91 
16M /opt/lvmdir/c2/brick/.glusterfs/92 
30M /opt/lvmdir/c2/brick/.glusterfs/93 
2.6M /opt/lvmdir/c2/brick/.glusterfs/94 
52M /opt/lvmdir/c2/brick/.glusterfs/95 
4.9M /opt/lvmdir/c2/brick/.glusterfs/96 
1.7M /opt/lvmdir/c2/brick/.glusterfs/97 
9.2M /opt/lvmdir/c2/brick/.glusterfs/98 
6.2M /opt/lvmdir/c2/brick/.glusterfs/99 
5.5M /opt/lvmdir/c2/brick/.glusterfs/9a 
4.2M /opt/lvmdir/c2/brick/.glusterfs/9b 
47M /opt/lvmdir/c2/brick/.glusterfs/9c 
2.2M /opt/lvmdir/c2/brick/.glusterfs/9d 
32M /opt/lvmdir/c2/brick/.glusterfs/9e 
13M /opt/lvmdir/c2/brick/.glusterfs/9f 
11M /opt/lvmdir/c2/brick/.glusterfs/a0 
5.6M /opt/lvmdir/c2/brick/.glusterfs/a1 
17M /opt/lvmdir/c2/brick/.glusterfs/a2 
3.3M /opt/lvmdir/c2/brick/.glusterfs/a3 
18M /opt/lvmdir/c2/brick/.glusterfs/a4 
21M /opt/lvmdir/c2/brick/.glusterfs/a5 
13M /opt/lvmdir/c2/brick/.glusterfs/a6 
18M /opt/lvmdir/c2/brick/.glusterfs/a7 
5.5M /opt/lvmdir/c2/brick/.glusterfs/a8 
4.5M /opt/lvmdir/c2/brick/.glusterfs/a9 
4.4M /opt/lvmdir/c2/brick/.glusterfs/aa 
21M /opt/lvmdir/c2/brick/.glusterfs/ab 
3.7M /opt/lvmdir/c2/brick/.glusterfs/ac 
13M /opt/lvmdir/c2/brick/.glusterfs/ad 
1.7M /opt/lvmdir/c2/brick/.glusterfs/ae 
9.1M /opt/lvmdir/c2/brick/.glusterfs/af 
50M /opt/lvmdir/c2/brick/.glusterfs/b0 
3.9M /opt/lvmdir/c2/brick/.glusterfs/b1 
8.8M /opt/lvmdir/c2/brick/.glusterfs/b2 
64M /opt/lvmdir/c2/brick/.glusterfs/b3 
13M /opt/lvmdir/c2/brick/.glusterfs/b4 
7.7M /opt/lvmdir/c2/brick/.glusterfs/b5 
3.0M /opt/lvmdir/c2/brick/.glusterfs/b6 
7.8M /opt/lvmdir/c2/brick/.glusterfs/b7 
2.8M /opt/lvmdir/c2/brick/.glusterfs/b8 
11M /opt/lvmdir/c2/brick/.glusterfs/b9 
4.6M /opt/lvmdir/c2/brick/.glusterfs/ba 
40M /opt/lvmdir/c2/brick/.glusterfs/bb 
17M /opt/lvmdir/c2/brick/.glusterfs/bc 
716K /opt/lvmdir/c2/brick/.glusterfs/bd 
2.2M /opt/lvmdir/c2/brick/.glusterfs/be 
14M /opt/lvmdir/c2/brick/.glusterfs/bf 
3.2M /opt/lvmdir/c2/brick/.glusterfs/c0 
11M /opt/lvmdir/c2/brick/.glusterfs/c1 
18M /opt/lvmdir/c2/brick/.glusterfs/c2 
8.7M /opt/lvmdir/c2/brick/.glusterfs/c3 
1.1M /opt/lvmdir/c2/brick/.glusterfs/c4 
4.3M /opt/lvmdir/c2/brick/.glusterfs/c5 
5.0M /opt/lvmdir/c2/brick/.glusterfs/c6 
44M /opt/lvmdir/c2/brick/.glusterfs/c7 
1.0M /opt/lvmdir/c2/brick/.glusterfs/c8 
4.5M /opt/lvmdir/c2/brick/.glusterfs/c9 
2.9M /opt/lvmdir/c2/brick/.glusterfs/ca 
8.4M /opt/lvmdir/c2/brick/.glusterfs/cb 
3.1M /opt/lvmdir/c2/brick/.glusterfs/cc 
14M /opt/lvmdir/c2/brick/.glusterfs/cd 
15M /opt/lvmdir/c2/brick/.glusterfs/ce 
4.7M /opt/lvmdir/c2/brick/.glusterfs/cf 
12K /opt/lvmdir/c2/brick/.glusterfs/changelogs 
36M /opt/lvmdir/c2/brick/.glusterfs/d0 
8.8M /opt/lvmdir/c2/brick/.glusterfs/d1 
6.8M /opt/lvmdir/c2/brick/.glusterfs/d2 
4.4M /opt/lvmdir/c2/brick/.glusterfs/d3 
2.3M /opt/lvmdir/c2/brick/.glusterfs/d4 
1.1M /opt/lvmdir/c2/brick/.glusterfs/d5 
16M /opt/lvmdir/c2/brick/.glusterfs/d6 
48M /opt/lvmdir/c2/brick/.glusterfs/d7 
6.8M /opt/lvmdir/c2/brick/.glusterfs/d8 
20M /opt/lvmdir/c2/brick/.glusterfs/d9 
5.7M /opt/lvmdir/c2/brick/.glusterfs/da 
3.9M /opt/lvmdir/c2/brick/.glusterfs/db 
788K /opt/lvmdir/c2/brick/.glusterfs/dc 
3.1M /opt/lvmdir/c2/brick/.glusterfs/dd 
3.5M /opt/lvmdir/c2/brick/.glusterfs/de 
45M /opt/lvmdir/c2/brick/.glusterfs/df 
5.2M /opt/lvmdir/c2/brick/.glusterfs/e0 
4.2M /opt/lvmdir/c2/brick/.glusterfs/e1 
9.2M /opt/lvmdir/c2/brick/.glusterfs/e2 
2.4M /opt/lvmdir/c2/brick/.glusterfs/e3 
11M /opt/lvmdir/c2/brick/.glusterfs/e4 
61M /opt/lvmdir/c2/brick/.glusterfs/e5 
12M /opt/lvmdir/c2/brick/.glusterfs/e6 
1.1M /opt/lvmdir/c2/brick/.glusterfs/e7 
5.9M /opt/lvmdir/c2/brick/.glusterfs/e8 
5.3M /opt/lvmdir/c2/brick/.glusterfs/e9 
1.6M /opt/lvmdir/c2/brick/.glusterfs/ea 
968K /opt/lvmdir/c2/brick/.glusterfs/eb 
9.5M /opt/lvmdir/c2/brick/.glusterfs/ec 
13M /opt/lvmdir/c2/brick/.glusterfs/ed 
15M /opt/lvmdir/c2/brick/.glusterfs/ee 
13M /opt/lvmdir/c2/brick/.glusterfs/ef 
4.1M /opt/lvmdir/c2/brick/.glusterfs/f0 
13M /opt/lvmdir/c2/brick/.glusterfs/f1 
32M /opt/lvmdir/c2/brick/.glusterfs/f2 
5.8M /opt/lvmdir/c2/brick/.glusterfs/f3 
5.2M /opt/lvmdir/c2/brick/.glusterfs/f4 
5.9M /opt/lvmdir/c2/brick/.glusterfs/f5 
14M /opt/lvmdir/c2/brick/.glusterfs/f6 
2.3M /opt/lvmdir/c2/brick/.glusterfs/f7 
2.5M /opt/lvmdir/c2/brick/.glusterfs/f8 
12M /opt/lvmdir/c2/brick/.glusterfs/f9 
432K /opt/lvmdir/c2/brick/.glusterfs/fa 
15M /opt/lvmdir/c2/brick/.glusterfs/fb 
37M /opt/lvmdir/c2/brick/.glusterfs/fc 
3.1M /opt/lvmdir/c2/brick/.glusterfs/fd 
1.9M /opt/lvmdir/c2/brick/.glusterfs/fe 
4.1M /opt/lvmdir/c2/brick/.glusterfs/ff 
4.0K /opt/lvmdir/c2/brick/.glusterfs/health_check 
8.0K /opt/lvmdir/c2/brick/.glusterfs/indices 
4.0K /opt/lvmdir/c2/brick/.glusterfs/landfill 
# 

In which it is clear that all the backup/metadata files created w.r.t original files in gluster mount point taking to much spaces. 

Please let us know the reason of this problem. 
-- 

Regards 
Abhishek Paliwal 






-- 




Regards 
Abhishek Paliwal 

_______________________________________________ 
Gluster-devel mailing list 
Gluster-devel at gluster.org 
http://lists.gluster.org/mailman/listinfo/gluster-devel 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20170407/1f08b961/attachment-0001.html>


More information about the Gluster-devel mailing list