[Gluster-devel] rm -r problem on FreeBSD port

[Rick Macklem]

Sakshi Bansal wrote:
> 
> > Sometimes. When this occurs (I get something like this every time I do an
> > "rm -r")
> > I can sometimes go into the directory and do an explicit "rm <file>". At
> > that point
> > it will report that the file doesn't exist (I suspect because it sees 2 of
> > them and
> > the second one is gone once the first is removed).
> > Other times, the "rm" doesn't gte rid of the file. (I am not sure if
> > unmount/mounting
> > affects this?)
> When rm on file reports that file does not exist, can you check if by any
> chance the file is a linkto.
> 
Ok. It sounds like the files with mode == 0 and size == 0 are "linkto" files?

> 
> > Afraid not. I deleted the volume and the underlying bricks and created a
> > new one.
> Do you by any chance have the log files? If not, if you hit the issue again
> please do send us the log files.
> You can set the log level to DEBUG or TRACE for getting better logs
> http://www.gluster.org/community/documentation/index.php/Gluster_3.2:_Setting_Volume_Options
> 
> 
> > I'll admit I find the fact that there are files on both bricks under the
> > same name
> > when I am not using replication weird.
> > Do you know why this is the case?
> Looks like one of it is a linkto file and the other is a data file and hence
> you see files with same name on two
> separate bricks. You can easily confirm this, by doing an 'ls -l' on the
> files in the bricks. For example:
> Brick 1:
> ---------T 2 sabansal sabansal    0 Jan 18 15:48 file1  ---> linkto file
> Brick2:
> -rw-rw-r-- 2 sabansal sabansal    0 Jan 18 12:54 file1  ---> data file
> 
> 
> Just curious, do you have 'lookup-optimize' set on?
> 
If you are referring to cluster.lookup-optimize then, yes, it was on.
(Initially I was just using defaults for volume create.)
However, I turned both it plus cluster.readdir-optimize and performance.readdir-ahead
off and the problem still seemed to occur.
- I also tried doing the "rm -r" from a NFSv3 mount (with no fuse mount) and it also
  ran into problems.

I did just do a test run using a volume with a single brick and it worked fine.
(Didn't have a problem with "rm -r". Also, with 2 bricks the test would fail
 part way through when not all of the files in a directory were visible. This
 test completed, so it didn't have that problem either.)

I am not sure if there are multiple problems, but it seems that at least part
of the issue is that readdir sometimes doesn't return all the file names in
the directory. (Similar to the bug report where "ls" doesn't return any file
names, although sometimes the files on one of the two bricks were returned
by readdir.)
I suspect "lookup" is affected too, since the failure in the test was a
"not found" that wouldn't have been a readdir.
(If you care, the test consists of unrolling a tarball of a FreeBSD kernel
 source tree and then doing a build cycle on it. The "rm -r" is done on the
 newly built binaries in a subtree under the source tree.)

Since both fuse and NFSv3 see the problem, I'll snoop around in the
distribute translator. I will also capture the log files for a test
run that fails (using 2 bricks) and I can email you that at some point.
(I can set DEBUG but it will be huge. This test takes hours to run.)

Thanks, rick