[Gluster-users] 3.6.6 healing issues?

Osborne, Paul (paul.osborne@canterbury.ac.uk) paul.osborne at canterbury.ac.uk
Fri Oct 16 08:00:00 UTC 2015


Just to add to the report:

gluster> volume heal kerberos info
Volume kerberos does not exist
Volume heal failed


Which has me rather concerned as it clearly does exist.


Anyway - since the reports I was getting indicated that there was a lack of space rather than a lack of write permissions, I have extended the size of the volume from 100MB to 500MB and the volume does now appear to work.

However a volume heal gives the same response as before, which leaves me with 2 questions:

1. why is volume heal reporting that the volume does not exist.
2. how much 'hidden' overhead does gluster actually use? 

Thanks

Paul

________________________________________
From: gluster-users-bounces at gluster.org <gluster-users-bounces at gluster.org> on behalf of Osborne, Paul (paul.osborne at canterbury.ac.uk) <paul.osborne at canterbury.ac.uk>
Sent: 15 October 2015 16:40
To: gluster-users at gluster.org
Subject: [Gluster-users] 3.6.6  healing issues?

HI,


I am seeing what I can best describe as an oddity where my monitoring is telling me that there is an issue (nagios touches a file and then removes it - to check read/write access available on the client mount point), gluster says that there is not an issue on the server mounting the file store there is allegedly a lack of space and I am not certain where to turn.:

# dpkg --list | grep gluster
ii  glusterfs-client                   3.6.6-1                           amd64        clustered file-system (client package)
ii  glusterfs-common                   3.6.6-1                           amd64        GlusterFS common libraries and translator modules
ii  glusterfs-server                   3.6.6-1                           amd64        clustered file-system (server package)


So these are packages straight out of the gluster.org repository.

# gluster volume status kerberos
Status of volume: kerberos
Gluster process                                         Port    Online  Pid
------------------------------------------------------------------------------
Brick gfsi-rh-01:/srv/hod/kerberos/gfs                  49171   Y       12863
Brick gfsi-isr-01:/srv/hod/kerberos/gfs                 49169   Y       37115
Brick gfsi-cant-01:/srv/hod/kerberos/gfs                49163   Y       49057
NFS Server on localhost                                 2049    Y       49069
Self-heal Daemon on localhost                           N/A     Y       49076
NFS Server on gfsi-isr-01.core.canterbury.ac.uk         2049    Y       37127
Self-heal Daemon on gfsi-isr-01.core.canterbury.ac.uk   N/A     Y       37134
NFS Server on gfsi-rh-01.core.canterbury.ac.uk          2049    Y       12875
Self-heal Daemon on gfsi-rh-01.core.canterbury.ac.uk    N/A     Y       12882

Task Status of Volume kerberos
------------------------------------------------------------------------------
There are no active volume tasks

# gluster volume info kerberos

Volume Name: kerberos
Type: Replicate
Volume ID: 89d63332-bb1e-4b47-8882-dfdb9af7f97d
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: gfsi-rh-01:/srv/hod/kerberos/gfs
Brick2: gfsi-isr-01:/srv/hod/kerberos/gfs
Brick3: gfsi-cant-01:/srv/hod/kerberos/gfs
Options Reconfigured:
cluster.server-quorum-ratio: 51


# gluster volume heal kerberos statistics | grep No  | grep -v 0


which to me looks good - 3 servers as a replica with qurom set.


BUT:

# mount | grep kerberos
gfsi-cant-01:/kerberos on /var/gfs/kerberos type nfs (rw,noatime,nodiratime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,nolock,proto=tcp,timeo=50,retrans=2,sec=sys,mountaddr=194.82.211.115,mountvers=3,mountport=38465,mountproto=tcp,local_lock=all,addr=194.82.211.115)

To be clear we use autofs failover mounting NFS  to our 3 bricks with a .5 second timeout

# df -h
Filesystem               Size  Used Avail Use% Mounted on
gfsi-cant-01:/kerberos    97M   31M   61M  34% /var/gfs/kerberos

Its a deliberately small volume as it only holds kerberos keytabs that need to be in sync across web servers.  Amusingly there is nothing like that much data being used:

/var/gfs/kerberos# ls -la
total 8
drwxr-xr-x  7 root   root   1024 Oct 15 16:03 .
drwxr-xr-x  5 root   root      0 Oct 15 16:03 ..
drwxr-xr-x  2 root   root   1024 Sep 16 12:09 HTTP_blog-mgmnt
drwxr-xr-x  2 root   root   1024 Jul 31  2014 HTTP_kerbtest
drwxr-xr-x  2 root   root   1024 May 14 15:48 HTTP_wiki-dev
-rw-r--r--+ 1 root   root   2546 Sep 16 11:24 krb5-keytab-autoupdate
drwxr-xr-x  2 nagios nagios 1024 Oct 15 16:18 .nagios
-rw-r--r--  1 nagios nagios   47 Apr 23 16:08 .nagioscheck

Rlampint-rh-01:/var/gfs/kerberos# du -hs *
5.0K    HTTP_blog-mgmnt
2.5K    HTTP_kerbtest
2.5K    HTTP_wiki-dev
2.5K    krb5-keytab-autoupdate

Rlampint-rh-01:/var/gfs/kerberos# du -hs .nagios
1.0K    .nagios


So I can only assume that the rest of the data is masked as Gluster metadata.

Rlampint-rh-01:/var/gfs/kerberos# ls -la >file
bash: file: No space left on device


which suggests that the volume is full - it clearly isn't.

In the logs:
/var/log/glusterfs# less glfsheal-kerberos.log
[2015-10-15 15:29:31.407041] E [glfs-mgmt.c:520:mgmt_getspec_cbk] 0-gfapi: failed to get the 'volume file' from server
[2015-10-15 15:29:31.407090] E [glfs-mgmt.c:599:mgmt_getspec_cbk] 0-glfs-mgmt: failed to fetch volume file (key:kerberos)
<Repeatedly>

I have stopped and restarted the volume, which has made no difference that I can see.

Other volumes configured and provisioned in a similar way on these 3 GFS servers are not reporting issues and they are rather well loaded compared to this one.

The nfs.log shows:
[2015-10-15 15:35:15.222424] W [nfs3.c:2370:nfs3svc_create_cbk] 0-nfs: 9b62284f: /.nagios/.nagiosrwtest.1444923315 => -1 (No space left on device)
[2015-10-15 15:35:28.081437] W [client-rpc-fops.c:2220:client3_3_create_cbk] 0-kerberos-client-2: remote operation failed: No space left on device. Path: /.nagios/.nagiosrwtest.1444923328
[2015-10-15 15:35:28.082000] W [client-rpc-fops.c:2220:client3_3_create_cbk] 0-kerberos-client-0: remote operation failed: No space left on device. Path: /.nagios/.nagiosrwtest.1444923328
[2015-10-15 15:35:28.082064] W [client-rpc-fops.c:2220:client3_3_create_cbk] 0-kerberos-client-1: remote operation failed: No space left on device. Path: /.nagios/.nagiosrwtest.1444923328
[2015-10-15 15:35:28.083019] W [nfs3.c:2370:nfs3svc_create_cbk] 0-nfs: 4b32485d: /.nagios/.nagiosrwtest.1444923328 => -1 (No space left on device)


At this point I am rather confused and not entirely certain where to turn next - as I can see that there is space although allegedly there isnt - the three bricks are comprised of 3 100MB volumes - in any case testing has shown previously that gluster will size for the smallest brick when using replicas.


Thoughts/comments are as always welcome.

Thanks

Paul





_______________________________________________
Gluster-users mailing list
Gluster-users at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-users


More information about the Gluster-users mailing list