[Gluster-users] Can't delete or move file off mounted brick

Tue Oct 18 06:01:34 UTC 2016

Hi gluster-users!

I’ve got an issue currently with my 2x brick replicate volume. The
volume is fine, and both bricks are replicating fine, but there is 1
file which refuses to be deleted or moved (and now can’t be viewed via a
glusterfs mount)

We just upgraded from |3.7.13-ubuntu1~trusty1| to
|3.7.16-ubuntu1~trusty1|, whether this has anything to do with it, I am
not sure.

The following tests have been performed on the mount points.

  * Doing a stat on the file works
  * Moving(renaming) the folder above works
  * Opening the file |vi file| and doing a |cat file| use to work before
    I did:
      o |mv /var/www/site/file /home/ubuntu/file|

This mv command never completed and I had to kill -9 it after an hour
waiting.

It seems whenever the file is opened now, it results in the process
halting on a stat command. The following is a stack trace when trying to
run the code (which calls the file, its php)

|<snip> 16:17:34.358528
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
{st_mode=S_IFREG|0664, st_size=11878, ...}) = 0 16:17:34.358613
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.358699
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.358810
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
{st_mode=S_IFREG|0664, st_size=11878, ...}) = 0 16:17:34.358870
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.358966
lstat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes",
{st_mode=S_IFDIR|0775, st_size=4096, ...}) = 0 16:17:34.359046
open("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
O_RDONLY) = 5 16:17:34.362692 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.362769 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.362848 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.362910 fstat(5, {st_mode=S_IFREG|0664,
st_size=11878, ...}) = 0 16:17:34.363008 mmap(NULL, 11878, PROT_READ,
MAP_SHARED, 5, 0) = 0x7eff135fd000 16:17:34.363102
stat("/var/www/site.com/wordpress/wp-content/plugins/gravityforms/includes/fields/class-gf-field-captcha.php",
{st_mode=S_IFREG|0664, st_size=11878, ...}) = 0 |

This is where the stack trace fails everytime.

------------------------------------------------------------------------

sudo gluster volume heal gv-www info
Brick demo-gluster01:/gluster
Status: Connected
Number of entries: 0

Brick demo-gluster02:/gluster
Status: Connected
Number of entries: 0

------------------------------------------------------------------------

sudo gluster volume info

Volume Name: gv-www
Type: Replicate
Volume ID: 6163f185-d58b-45f2-8bf0-eeb3a18142d9
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: demo-gluster01:/gluster
Brick2: demo-gluster02:/gluster
Options Reconfigured:
cluster.self-heal-daemon: enable
features.scrub: Active
features.bitrot: on
diagnostics.client-log-level: INFO
diagnostics.brick-log-level: DEBUG
performance.readdir-ahead: on

------------------------------------------------------------------------

I just enabled bitrot and scrubbing. (To help with these possible issues)

The only real error I can see in this timeframe is the following:

Brick1:

|/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.160099] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.174907] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.176708] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.178831] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.181356] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.183602] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.186835] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.189262] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain)
/var/log/glusterfs/var-www.log.1:[2016-10-17 23:29:12.191969] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
2ce27b6e-99aa-408b-b386-b70d3e09d228. (Possible split-brain) |

Brick2:

|/var/log/glusterfs/var-www.log.1:[2016-10-15 00:16:48.163353] W [MSGID:
108008] [afr-read-txn.c:244:afr_read_txn] 0-gv-www-replicate-0:
Unreadable subvolume -1 found with event generation 2 for gfid
6b0b2d6a-f6ea-4ab8-89fc-d776afc17c35. (Possible split-brain) |

But I cannot find any log relating to this on the gluster servers.

Also I can cat the file no problem if I login to the bricks. What could
be a reason of it not working on the mount point…

Any help would be greatly appreciated.

Cheers, Tim

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161018/ec38aefe/attachment-0001.html>