[Gluster-users] file corruption on Gluster 3.5.1 and Ubuntu 14.04

[mike]

I upgraded the client to Gluster 3.5.2, but there is no difference.

The bug is almost certainly in the Fuse client. If I remount the filesystem with NFS, the problem is no longer observable.

I spent a little time looking through the xlator/fuse-bridge to see where the offsets are coming from, but I'm really not familiar enough with the code, so it is slow going.

Unfortunately, I'm still having trouble reproducing this in a python script that could be readily attached to a bug report.

I'll take a crack at that again, but I will a file a bug anyway for completeness.

On Sep 5, 2014, at 7:10 PM, mike <mike at luminatewireless.com> wrote:

> I have narrowed down the source of the bug. 
> 
> Here is an strace of glusterfsd http://fpaste.org/131455/40996378/
> 
> The first line represents a write that does *not* make it into the underlying file.
> 
> The last line is the write that stomps the earlier write.
> 
> As I said, the client file is opened in O_APPEND mode, but on the glusterfsd side, the file is just O_CREAT|O_WRONLY. The means the offsets to pwrite() need to be valid.
> 
> I correlated this to a tcpdump I took and I can see that in fact, the RPCs being sent have the wrong offset.  Interestingly, glusterfs.write-is-append = 0, which I wouldn't have expected.
> 
> I think the bug lies in the glusterfs fuse client.
> 
> As to your question about Gluster 3.5.2, I may be able to do that if I am unable to find the bug in the source.
> 
> -Mike
> 
> On Sep 5, 2014, at 6:16 PM, Justin Clift <justin at gluster.org> wrote:
> 
>> On 06/09/2014, at 12:10 AM, mike wrote:
>>> I have found that the O_APPEND flag is key to this failure - I had overlooked that flag when reading the strace and trying to cobble up a minimal reproduction.
>>> 
>>> I now have a small pair of python scripts that can reliably reproduce this failure.
>> 
>> 
>> As a thought, is there a reasonable way you can test this on GlusterFS 3.5.2?
>> 
>> There were some important bug fixes in 3.5.2 (from 3.5.1).
>> 
>> Note I'm not saying yours is one of them, I'm just asking if it's
>> easy to test and find out. :)
>> 
>> Regards and best wishes,
>> 
>> Justin Clift
>> 
>> --
>> GlusterFS - http://www.gluster.org
>> 
>> An open source, distributed file system scaling to several
>> petabytes, and handling thousands of clients.
>> 
>> My personal twitter: twitter.com/realjustinclift
>> 
>