[Gluster-users] gluster 5.3: file or directory not read-/writeable, although it exists - cache?

Wed Feb 20 08:47:30 UTC 2019

Hi Nithya,

apologies for the long mail... the backend developer told me that he
simply uses standard java file utils; test if a directory exists - if
not throw exception.

A little explanation to our setup: our frontend requests a few files,
and via round robin they get distributed to 3 tomcats, and they create
the directories and files in the glusterfs volume. It's possible that
one of the tomcats creates a directory and writes a file, and then
another tomcat wants to write another file but then can't - and throws
such a exception.

After comparing the timestamps in tomcat and in the filesystem there
are 2 cases: a directory already exists a couple of days or it was
freshly created.

Case 1: directory was freshly created. Timestamps say that the
directory and one file were created at 07:05:26, and at 07:05:27
there's an error message in tomcat:

2019-02-20 07:05:27,399 WARN  Path
/data/repository/shared/public/staticmap/271/362/2713628 is existed
but it is not directory [[fetch data pool thread
6 at de.alpstein.core.io.FileUtils:747]]
java.nio.file.FileAlreadyExistsException:
/data/repository/shared/public/staticmap/271/362/2713628

ls -lah --full-time /data/repository/shared/public/staticmap/271/362/2713628/
total 22K
drwxr-xr-x 2 tomcat8 tomcat8 4.0K 2019-02-20 07:05:26.420007137 +0100 .
drwxr-xr-x 7 tomcat8 tomcat8 4.0K 2019-02-20 07:05:26.411096237 +0100 ..
-rw-r--r-- 1 tomcat8 tomcat8  14K 2019-02-20 07:05:26.424007285 +0100
170x100_ed82ac740fe8b9bbaac9ca77aa47f573.jpeg

Case 2: directory is already a couple of days old. I just saw an error
message in one of the tomcats i immediately checked the 3 paths on the
3 servers:
server1: ls /data/repository/shared/public/staticmap/369/217/36921711/ works
server2: ls /data/repository/shared/public/staticmap/369/217/36921711/ works
server3: ls /data/repository/shared/public/staticmap/369/217/36921711/
ls: cannot access
'/data/repository/shared/public/staticmap/369/217/36921711/': No such
file or directory

Error message in tomcat:
2019-02-20 08:49:28,762 WARN  Path
/data/repository/shared/public/staticmap/369/217/36921711 is existed
but it is not directory [[fetch data pool thread
43 at de.alpstein.core.io.FileUtils:747]]
java.nio.file.FileAlreadyExistsException:
/data/repository/shared/public/staticmap/369/217/36921711

Timestamps of directory (is already ~7 days old):
ls -lah --full-time /data/repository/shared/public/staticmap/369/217/36921711/
total 197K
drwxr-xr-x 2 tomcat8 tomcat8 4.0K 2019-02-13 12:02:58.164004209 +0100 .
drwxr-xr-x 6 tomcat8 tomcat8 4.0K 2019-02-13 13:36:41.159642935 +0100 ..
-rw-r--r-- 1 tomcat8 tomcat8 123K 2019-02-13 12:02:58.172004504 +0100
404x475_d473f1d60fc6842dddc7a5460672722b.jpeg
-rw-r--r-- 1 tomcat8 tomcat8  66K 2019-02-13 11:58:50.218874380 +0100
420x237_1a7913cc18f4ec142ebc7c578856fb66.jpeg

In both cases the directory itself is OK on the other 2 servers,
there's an error message on only one of them, including the 'ls $dir
-> no such file or directory' error. Interestingly, in both cases, the
error gets fixed when either doing umount/mount of glusterfs volume or
doing this on the server with the error message:

ls /data/repository/shared/public/staticmap/369/217/36921711 -> TAB ->
wait 1-2 seconds -> last / appears -> Enter -> directory shows up.

First i thought that might be some caching problem, but that seems not
quite probable with a directory that's 7 days old.

Regards,
Hubert

Am Mi., 20. Feb. 2019 um 06:12 Uhr schrieb Nithya Balachandran
<nbalacha at redhat.com>:
>
>
>
> On Tue, 19 Feb 2019 at 15:18, Hu Bert <revirii at googlemail.com> wrote:
>>
>> Hello @ll,
>>
>> one of our backend developers told me that, in the tomcat logs, he
>> sees errors that directories on a glusterfs mount aren't readable.
>> Within tomcat the errors look like this:
>>
>> 2019-02-19 07:39:27,124 WARN  Path
>> /data/repository/shared/public/staticmap/370/626 is existed but it is
>> not directory
>> java.nio.file.FileAlreadyExistsException:
>> /data/repository/shared/public/staticmap/370/626
>
>
> Do you know which operation failed here?
> regards,
> Nithya
>>
>>
>> But the basic directory does exist, has been created on 2019-02-18
>> (and is readable on other clients):
>>
>> ls -lah /data/repository/shared/public/staticmap/370/626/
>> total 36K
>> drwxr-xr-x   9 tomcat8 tomcat8 4.0K Feb 18 12:15 .
>> drwxr-xr-x 522 tomcat8 tomcat8 4.0K Feb 19 10:29 ..
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:45 37062632
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 19 09:29 37062647
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 12:15 37062663
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:18 37062668
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 11:36 37062681
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 18 16:53 37062682
>> drwxr-xr-x   2 tomcat8 tomcat8 4.0K Feb 19 08:19 37062688
>>
>> gluster v5.3, debian stretch.
>> gluster volume info: https://pastebin.com/UBVWSUex
>> gluster volume status: https://pastebin.com/3guxFq5m
>>
>> mount options on client:
>> gluster1:/workdata /data/repository/shared/public glusterfs
>> defaults,_netdev,lru-limit=0,backup-volfile-servers=gluster2:gluster3
>> 0 0
>>
>> brick mount options:
>> /dev/md/4  /gluster/md4  xfs  inode64,noatime,nodiratime  0 0
>>
>> Hmm... problem with mount options? Or is some cache involved?
>>
>>
>> Best regards,
>> Hubert
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users