[Gluster-devel] file shown by ls but inaccessible (dht confusion?)

[Emmanuel Dreyfus]

Here is another problem with an inaccessible file. Consider this problem 
on a 2x2 replicaed/rdistributed setup:

client# ls /gfs/usr/src/tools/compat/obj/ |grep strlcat.lo
strlcat.lo
client# ls /gfs/usr/src/tools/compat/obj/strlcat.lo
ls: /gfs/usr/src/tools/compat/obj/strlcat.lo: Socket is not connected

client log:
[2011-07-27 06:14:37.684558] I [client3_1-fops.c:411:client3_1_stat_cbk] 
  0-gfs-client-1: remote operation failed: No such file or directory
[2011-07-27 06:14:37.684716] W [client3_1-fops.c:2656:client3_1_stat] 
  0-gfs-client-0: failed to send the fop Socket is not connected
[2011-07-27 06:14:37.690212] I [client3_1-fops.c:411:client3_1_stat_cbk] 
  0-gfs-client-1: remote operation failed: No such file or directory
[2011-07-27 06:14:37.690336] W [client3_1-fops.c:2656:client3_1_stat] 
  0-gfs-client-0: failed to send the fop Socket is not connected

The "Socket is not connected" message is there because one of the servers
is down. That should work anyway, right?

On the running server, log for brick /export/wd1a
[2011-07-27 06:14:37.685180] I [server3_1-fops.c:1492:server_stat_cbk] 
  0-gfs-server: 5989901: STAT /usr/src/tools/compat/obj/strlcat.lo (0) 
  ==> -1 (No such file or directory)
[2011-07-27 06:14:37.690954] I [server3_1-fops.c:1492:server_stat_cbk] 
  0-gfs-server: 5989902: STAT /usr/src/tools/compat/obj/strlcat.lo (0) 
  ==> -1 (No such file or directory)

The file exists on the server and the gfid is safe:

server# ls -l /export/*/usr/src/tools/compat/obj/strlcat.lo 
-rw-r--r--  1 root  wheel  595 Jul 22 09:20 
  /export/wd1a/usr/src/tools/compat/obj/strlcat.lo
---------T  1 root  wheel    0 Jul 22 09:20 
  /export/wd3a/usr/src/tools/compat/obj/strlcat.lo

server# getextattr -x trusted.gfid /export/*/usr/src/tools/compat/obj/strlcat.lo
/export/wd1a/usr/src/tools/compat/obj/strlcat.lo        
   000   56 e9 57 ce f4 9a 46 08 aa b0 55 fa 2f 5d 13 d3    V.W...F...U./]..
/export/wd3a/usr/src/tools/compat/obj/strlcat.lo        
   000   56 e9 57 ce f4 9a 46 08 aa b0 55 fa 2f 5d 13 d3    V.W...F...U./]..

Here is the kernel trace on the server for brick /export/wd1a
 490   4 glusterfsd CALL  __lstat30(0xb9fff760,0xb9fff6b4)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd RET   __lstat30 0
 490   4 glusterfsd CALL  lgetxattr(0xb9fff760,0xbba4c073,0xb9fff718,0x10)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd MISC  xattr-name:
        000             646966672e64657473757274  trusted.gfid
 490   4 glusterfsd MISC  xattr-val:
        000     d3135d2ffa55b0aa08469af4ce57e956  V.W...F...U./]..
 490   4 glusterfsd RET   lgetxattr 16/0x10
 490   4 glusterfsd CALL  __lstat30(0xb9fff760,0xb9fff608)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd RET   __lstat30 0
 490   4 glusterfsd CALL  lgetxattr(0xb9fff760,0xbba4c073,0xb9fff66c,0x10)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd MISC  xattr-name:
        000             646966672e64657473757274  trusted.gfid
 490   4 glusterfsd MISC  xattr-val:
        000     d3135d2ffa55b0aa08469af4ce57e956  V.W...F...U./]..
 490   4 glusterfsd RET   lgetxattr 16/0x10
 490   4 glusterfsd CALL  __lstat30(0xb9fff760,0xb9fff6cc)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd RET   __lstat30 0
 490   4 glusterfsd CALL  lgetxattr(0xb9fff760,0xbba4c073,0xb9fff66c,0x10)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd MISC  xattr-name:
        000             646966672e64657473757274  trusted.gfid
 490   4 glusterfsd MISC  xattr-val:
        000     d3135d2ffa55b0aa08469af4ce57e956  V.W...F...U./]..
 490   4 glusterfsd RET   lgetxattr 16/0x10
 490   4 glusterfsd CALL  lgetxattr(0xb9fff760,0xb97031a0,0,0)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd MISC  xattr-name:
        000     6c63615f7869736f702e6d6574737973  system.posix_acl
        010                     746c75616665645f  _default
 490   4 glusterfsd RET   lgetxattr -1 errno 93 Attribute not found
 490   4 glusterfsd CALL  lgetxattr(0xb9fff760,0xb9703100,0,0)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj/strlcat.lo"
 490   4 glusterfsd MISC  xattr-name:
        000     6c63615f7869736f702e6d6574737973  system.posix_acl
        010                       7373656363615f  _access
 490   4 glusterfsd RET   lgetxattr -1 errno 93 Attribute not found
 490   4 glusterfsd CALL  __lstat30(0xb970c040,0xb9fff6cc)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj"
 490   4 glusterfsd RET   __lstat30 0
 490   4 glusterfsd CALL  lgetxattr(0xb970c040,0xbba4c073,0xb9fff66c,0x10)
 490   4 glusterfsd NAMI  "/export/wd1a/usr/src/tools/compat/obj"
 490   4 glusterfsd MISC  xattr-name:
        000             646966672e64657473757274  trusted.gfid
 490   4 glusterfsd MISC  xattr-val:
        000     7eef14d5ffdb52a4aa46a593fbf50b80  ......F..R.....~
 490   4 glusterfsd RET   lgetxattr 16/0x10
   "
(... and then the logged message )

I tried removing the posix-acl xlator but it does not change anything. 
Any idea of what is wrong here?

-- 
Emmanuel Dreyfus
manu at netbsd.org