[Gluster-users] File not found errors

[cfeller at rocketmail.com]

Hi,

I've got an interesting issue with files not being found when accessed directly. 

When accessing a file like so:

  #ls /mnt/path/to/some/file/file.json
  ls: cannot access '/mnt/path/to/some/file/file.json': No such file or directory

I can wait, try again, and same result. I could put this in a loop with a delay between read attempts, same result.

However, if I do an "ls" of the directory first then everything works. That is:

  # ls /mnt/path/to/some/file/file.json
  ls: cannot access '/mnt/path/to/some/file/file.json': No such file or directory

  # ls /mnt/path/to/some/file/
  dir1  dir2  file1.txt  file2.txt  file3.txt  file.json

At this point I can successfully "ls" the file:

  # ls /mnt/path/to/some/file/file.json
  /mnt/path/to/some/file/file.json

It is curious, but seems to be isolated to certain directories/mount points.

Looking at the mount logs there is absolutely nothing printed to the mnt log on the client with regards to the failure or the eventual success.

The configuration:

  Distributed-Replicate
  4 x 2 = 8
  Gluster version 9.2

Clients are using native FUSE mounts (also version 9.2)

I did recently add the 4th brick, and the system is currently undergoing a rebalance (the fix-layout rebalance already completed). I am throttling the full rebalance as 'lazy'.

I looked at the bricks on the server and the files still exists on the older bricks, so I don't think the files in question got rebalanced (yet).

The system load isn't too high, and this doesn't seem to be a consistent error as I don't have this issue in other directories, but I consistently have the issue on this directory as well as a few others.

Adding some additional background/history:

  Cluster was setup a couple of years ago on gluster 6.x. 
  Did an expansion (two replica pairs to three replica pairs) and rebalance while on 6.x, no issues.
  Upgraded to gluster 7.x -> 8.x -> 9.x
  Ops version has been upgraded to 90000
  Doing another expansion (three replica pairs to four replica pairs) and rebalance now, and seeing some interesting issues, including this one.

There is 0 entries under "heal info" or "info split-brain".

I'm just wondering if anyone on this list has seen anything similar, or has any suggestions.