[Gluster-users] Unable to delete file - no such file or directory

Tue Jun 11 14:43:52 UTC 2013

Hello,

I have faced strange problem on one of the Gluster client/server nodes. File exist but cannot be deleted from the fuse mount point on one client node, but can be deleted from another

Attemt to delete:
root at SERVER:~# rm /var/lib/nova/instances/instance-0098a323/disk
rm: cannot remove `/var/lib/nova/instances/instance-0098a323/disk': No such file or directory

Although files  listed as existing in the volume:
root@ SERVER:~# ls -al /var/lib/nova/instances/instance-0098a323
total 6143442
drwxrwxr-x   2 nova nova        143 Jun 11 11:54 .
drwxr-xr-x 107 nova nova     409600 Jun 11 11:54 ..
-rw-r--r--   1 root root 6290472960 Jun 11 11:54 disk
root@ SERVER:~# stat /var/lib/nova/instances/instance-0098a323/disk
  File: `/var/lib/nova/instances/instance-0098a323/disk'
  Size: 6290472960      Blocks: 12286080   IO Block: 131072 regular file
Device: 1eh/30d Inode: 11831107392822693115  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-06-03 14:12:03.803403051 +0100
Modify: 2013-06-11 11:54:09.623013613 +0100
Change: 2013-06-11 11:54:12.355063759 +0100
Birth: -

File locartion information:
root at SERVER:~/openstack# getfattr -n trusted.glusterfs.pathinfo /var/lib/nova/instances/instance-0098a323/disk
getfattr: Removing leading '/' from absolute path names
# file: var/lib/nova/instances/instance-0098a323/disk
trusted.glusterfs.pathinfo="(<DISTRIBUTE:glustervmstore-dht> (<REPLICATE:glustervmstore-replicate-21> <POSIX(/mnt/vmstore/brick):BRICK_SERV1:/mnt/vmstore/brick/instance-0098a323/disk> <POSIX(/mnt/vmstore/brick): BRICK_SERV2:/mnt/vmstore/brick/instance-0098a323/disk>))"

Subvolume information:
volume glustervmstore-replicate-3
    type cluster/replicate
    subvolumes glustervmstore-client-6 glustervmstore-client-7
end-volume
volume glustervmstore-replicate-21
    type cluster/replicate
    subvolumes glustervmstore-client-42 glustervmstore-client-43
end-volume

It is also available on the replica bricks. However I WAS ABLE to delete the file from another node, which also mounts the same volume under: /var/lib/nova/instances

Log file excerpt (I presume that’s /var/log/gluster/var-lib-nova-instances.log). What is strange that  glustervmstore-client-6 and glustervmstore-client-6 does not represent actual brick servers where file is stored:
[2013-06-11 14:10:02.640244] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
[2013-06-11 14:10:51.769641] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:10:51.769737] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:10:51.769769] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204475960: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:18:15.350431] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:18:15.350513] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:18:15.350547] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204480095: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:26:07.418892] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:26:07.418980] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:26:07.419011] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204484135: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:27:12.510324] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
[2013-06-11 14:27:44.518849] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-6: remote operation failed: No such f
ile or directory
[2013-06-11 14:27:44.518937] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-7: remote operation failed: No such f
ile or directory
[2013-06-11 14:27:44.518967] W [fuse-bridge.c:1029:fuse_unlink_cbk] 0-glusterfs-fuse: 204485011: UNLINK() /instance-0098a323/disk => -1
(No such file or directory)
[2013-06-11 14:29:33.289316] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
[2013-06-11 14:38:13.487311] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-41: remote operation failed: No such
file or directory
[2013-06-11 14:38:13.487563] W [client3_1-fops.c:592:client3_1_unlink_cbk] 3-glustervmstore-client-40: remote operation failed: No such file or directory

The same happened for /instance-0098a327/disk  file although there are no similar line
[2013-06-11 14:27:12.510324] I [afr-self-heal-entry.c:2333:afr_sh_entry_fix] 3-glustervmstore-replicate-3: /instance-0098a323: Performin
g conservative merge
I was able to delete the file from another node.

There are no mention of the folder in the GlusterFS logs on the actual brick servers.
Thank you
______________________________________________________________________________

Elvinas Piliponis  I  UNIX Engineer  I  GTIS UNIX Engineering
Tel +370 5 251 1218, 7 2249 1218  I   Mobile +370 656 69249 I  Email  elvinas.piliponis at barclays.com<mailto:elvinas.piliponis at barclays.com>
Barclays, GreenHall 9th floor 09.E4.1, Upės g. 21, Vilnius, Lithuania LT-081218
barclays.com

This e-mail and any attachments are confidential and intended
solely for the addressee and may also be privileged or exempt from
disclosure under applicable law. If you are not the addressee, or
have received this e-mail in error, please notify the sender
immediately, delete it from your system and do not copy, disclose
or otherwise act upon any part of this e-mail or its attachments.

Internet communications are not guaranteed to be secure or
virus-free.
The Barclays Group does not accept responsibility for any loss
arising from unauthorised access to, or interference with, any
Internet communications by any third party, or from the
transmission of any viruses. Replies to this e-mail may be
monitored by the Barclays Group for operational or business
reasons.

Any opinion or other information in this e-mail or its attachments
that does not relate to the business of the Barclays Group is
personal to the sender and is not given or endorsed by the Barclays
Group.

Barclays Bank PLC. Registered in England and Wales (registered no.
1026167).
Registered Office: 1 Churchill Place, London, E14 5HP, United
Kingdom.

Barclays Bank PLC is authorised by the Prudential Regulation
Authority and regulated by the Financial Conduct Authority and the
Prudential Regulation Authority (Financial Services Register No.
122702).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130611/6fd9dbb5/attachment.html>