[Gluster-users] gluster 5.3: file or directory not read-/writeable, although it exists - cache?

[Hu Bert]

No, that conclusion was too early. Error happens on all clients, but
the directories are different, e.g.:

dir1 fails on client1, but works on all others
dir2 fails on client2, but works on all others
etc.

Interesting: when doing a 'ls' on the dir and TAB->autocomplete
(missing the last number), the directory appears:

ls /data/repository/shared/public/images/370/435/3704359
37043592/ 37043593/ 37043595/ 37043597/ 37043598/ 37043599/

and the can be opened and cd'ed into. Content appears:

ls -lah /shared/public/images/370/435/37043597/
total 1.6M
drwxr-xr-x  2 tomcat8 tomcat8 4.0K Feb 19 13:52 .
drwxr-xr-x 69 tomcat8 tomcat8 4.0K Feb 17 21:42 ..
-rw-r--r--  1 tomcat8 tomcat8 7.0K Feb 19 13:52 100x100f.jpg
-rw-r--r--  1 tomcat8 tomcat8 235K Feb 19 13:23 1080x610r.jpg
-rw-r--r--  1 tomcat8 tomcat8  96K Feb 18 10:46 1200x1200s.jpg
-rw-r--r--  1 tomcat8 tomcat8  14K Feb 17 23:03 150x150f.jpg
[...]

Some error message in
/var/log/glusterfs/data-repository-shared-public.log on one of the
clients saying that mkdir fails because the directory exists:

[2019-02-19 10:39:44.673279] W [fuse-bridge.c:582:fuse_entry_cbk]
0-glusterfs-fuse: 69757484: MKDIR() /images/370/435/37043597 => -1
(File exists)
[2019-02-19 11:14:58.625014] W [fuse-bridge.c:582:fuse_entry_cbk]
0-glusterfs-fuse: 69977210: MKDIR() /images/370/435/37043597 => -1
(File exists)
[2019-02-19 11:42:15.626959] W [fuse-bridge.c:582:fuse_entry_cbk]
0-glusterfs-fuse: 70148412: MKDIR() /images/370/435/37043597 => -1
(File exists)
[2019-02-19 12:26:01.065483] W [fuse-bridge.c:582:fuse_entry_cbk]
0-glusterfs-fuse: 70406931: MKDIR() /images/370/435/37043597 => -1
(File exists)

The directory exists -> warning is OK, but why doesn't it appear first?


Am Di., 19. Feb. 2019 um 14:10 Uhr schrieb Hu Bert <revirii at googlemail.com>:
>
> a little update: it seems to be only one client. Whenever there's a
> "no such file or directory" error in the logs, the directory can be
> read/opened on all the other clients. very strange...
>
> Nothing in glusterfs logs besides the zillion "dict is NULL [Invalid
> argument]" warnings. Must be something on the client itself.
>
> Am Di., 19. Feb. 2019 um 10:47 Uhr schrieb Hu Bert <revirii at googlemail.com>:
> >
> > Hello @ll,
> >
> > one of our backend developers told me that, in the tomcat logs, he
> > sees errors that directories on a glusterfs mount aren't readable.
> > Within tomcat the errors look like this:
> >
> > 2019-02-19 07:39:27,124 WARN  Path
> > /data/repository/shared/public/staticmap/370/626 is existed but it is
> > not directory
> > java.nio.file.FileAlreadyExistsException:
> > /data/repository/shared/public/staticmap/370/626
> >
> > But the basic directory does exist, has been created on 2019-02-18
> > (and is readable on other clients):
> >
> > ls -lah /data/repository/shared/public/staticmap/370/626/
> > total 36K
> > drwxr-xr-x   9 tomcat8 tomcat8 4.0K Feb 18 12:15 .
> > drwxr-xr-x 522 tomcat8 tomcat8 4.0K Feb 19 10:29 ..
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:45 37062632
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 19 09:29 37062647
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 12:15 37062663
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:18 37062668
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:36 37062681
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 16:53 37062682
> > drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 19 08:19 37062688
> >
> > gluster v5.3, debian stretch.
> > gluster volume info: https://pastebin.com/UBVWSUex
> > gluster volume status: https://pastebin.com/3guxFq5m
> >
> > mount options on client:
> > gluster1:/workdata /data/repository/shared/public glusterfs
> > defaults,_netdev,lru-limit=0,backup-volfile-servers=gluster2:gluster3
> > 0 0
> >
> > brick mount options:
> > /dev/md/4  /gluster/md4  xfs  inode64,noatime,nodiratime  0 0
> >
> > Hmm... problem with mount options? Or is some cache involved?
> >
> >
> > Best regards,
> > Hubert