[Gluster-users] Shared storage showing 100% used

[Pam Patterson]

Hello list,

I recently enabled shared storage on a working cluster with nfs-ganesha 
and am just storing my ganesha.conf file there so that all 4 nodes can 
access it(baby steps).  It was all working great for a couple of weeks 
until I was alerted that /run/gluster/shared_storage was full, see 
below.  There was no warning; it went from fine to critical overnight.

Filesystem                         Size  Used Avail Use% Mounted on
/dev/md125                          50G  102M   47G   1% /
devtmpfs                            32G     0   32G   0% /dev
tmpfs                               32G     0   32G   0% /dev/shm
tmpfs                               32G   17M   32G   1% /run
tmpfs                               32G     0   32G   0% /sys/fs/cgroup
/dev/md124                          59G  1.6G   55G   3% /usr
/dev/md153p2                        13T   34M   13T   1% /glusterfs/a4/b2
/dev/md151p1                        13T   34M   13T   1% /glusterfs/a2/b1
/dev/md151p2                        13T   34M   13T   1% /glusterfs/a2/b2
/dev/md152p1                        26T  4.4T   22T  17% /glusterfs/a3/b1
/dev/md122                          20G  6.1G   13G  33% /var
/dev/md126                         976M  233M  677M  26% /boot
/dev/md150p1                        13T  1.1T   12T   9% /glusterfs/a1/b1
/dev/md150p2                        13T  6.7T  6.2T  52% /glusterfs/a1/b2
/dev/md123                         1.7T   77M  1.6T   1% /home
/dev/md153p1                        13T  1.8T   11T  14% /glusterfs/a4/b1
localhost:/gluster_shared_storage   20G   20G     0 100% 
/run/gluster/shared_storage
tmpfs                              6.3G     0  6.3G   0% /run/user/1000

There is only one file there

# ls -la /run/gluster/shared_storage/nfs-ganesha/
total 20
drwxr-xr-x 2 root root  4096 Nov  7 13:57 .
drwxr-xr-x 4 root root  4096 Nov 12 13:44 ..
-rw-r--r-- 1 root root 11562 Nov  7 13:57 ganesha.conf

# du -sh /run/gluster/shared_storage
20K    /run/gluster/shared_storage

When I go look on the bricksthemselves, I see some other 
filesin.gluster, but certainlynot 20GB worth (412K, 20Kand 408K).  Does 
gluster think it is really full or is this just an artifactof the shared 
storage process?

More info:

CentOS7 fully patched, glusterfs-3.12.1-2.el7.x86_64

Volume Name: gluster_shared_storage
Type: Replicate
Volume ID: cc1fd307-a2bb-4901-a6f9-d92b0f52a65f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: ace-storage-4n3:/var/lib/glusterd/ss_brick
Brick2: ace-storage-4n4:/var/lib/glusterd/ss_brick
Brick3: ace-storage-4n1:/var/lib/glusterd/ss_brick
Options Reconfigured:
nfs.disable: ON
cluster.enable-shared-storage: enable

Thanks for any insights!

Pam