[Gluster-users] file corruption on Gluster 3.5.1 and Ubuntu 14.04

[Pranith Kumar Karampuri]

Interestingly disabling write-behind prevents the bug.
CC Raghavendra G

Pranith
On 09/07/2014 01:27 PM, Anand Avati wrote:
> The only reason O_APPEND gets stripped on the server side, is because 
> of one of the following xlators:
>
> - stripe
> - quiesce
> - crypt
>
> If you have any of these, please try unloading/reconfiguring without 
> these features and try again.
>
> Thanks
>
>
> On Sat, Sep 6, 2014 at 3:31 PM, mike <mike at luminatewireless.com 
> <mailto:mike at luminatewireless.com>> wrote:
>
>     I was able to narrow it down to smallish python script.
>
>     I've attached that to the bug.
>
>     https://bugzilla.redhat.com/show_bug.cgi?id=1138970
>
>
>     On Sep 6, 2014, at 1:05 PM, Justin Clift <justin at gluster.org
>     <mailto:justin at gluster.org>> wrote:
>
>     > Thanks Mike, this is good stuff. :)
>     >
>     > + Justin
>     >
>     >
>     > On 06/09/2014, at 8:19 PM, mike wrote:
>     >> I upgraded the client to Gluster 3.5.2, but there is no difference.
>     >>
>     >> The bug is almost certainly in the Fuse client. If I remount
>     the filesystem with NFS, the problem is no longer observable.
>     >>
>     >> I spent a little time looking through the xlator/fuse-bridge to
>     see where the offsets are coming from, but I'm really not familiar
>     enough with the code, so it is slow going.
>     >>
>     >> Unfortunately, I'm still having trouble reproducing this in a
>     python script that could be readily attached to a bug report.
>     >>
>     >> I'll take a crack at that again, but I will a file a bug anyway
>     for completeness.
>     >>
>     >> On Sep 5, 2014, at 7:10 PM, mike <mike at luminatewireless.com
>     <mailto:mike at luminatewireless.com>> wrote:
>     >>
>     >>> I have narrowed down the source of the bug.
>     >>>
>     >>> Here is an strace of glusterfsd http://fpaste.org/131455/40996378/
>     >>>
>     >>> The first line represents a write that does *not* make it into
>     the underlying file.
>     >>>
>     >>> The last line is the write that stomps the earlier write.
>     >>>
>     >>> As I said, the client file is opened in O_APPEND mode, but on
>     the glusterfsd side, the file is just O_CREAT|O_WRONLY. The means
>     the offsets to pwrite() need to be valid.
>     >>>
>     >>> I correlated this to a tcpdump I took and I can see that in
>     fact, the RPCs being sent have the wrong offset.  Interestingly,
>     glusterfs.write-is-append = 0, which I wouldn't have expected.
>     >>>
>     >>> I think the bug lies in the glusterfs fuse client.
>     >>>
>     >>> As to your question about Gluster 3.5.2, I may be able to do
>     that if I am unable to find the bug in the source.
>     >>>
>     >>> -Mike
>     >>>
>     >>> On Sep 5, 2014, at 6:16 PM, Justin Clift <justin at gluster.org
>     <mailto:justin at gluster.org>> wrote:
>     >>>
>     >>>> On 06/09/2014, at 12:10 AM, mike wrote:
>     >>>>> I have found that the O_APPEND flag is key to this failure -
>     I had overlooked that flag when reading the strace and trying to
>     cobble up a minimal reproduction.
>     >>>>>
>     >>>>> I now have a small pair of python scripts that can reliably
>     reproduce this failure.
>     >>>>
>     >>>>
>     >>>> As a thought, is there a reasonable way you can test this on
>     GlusterFS 3.5.2?
>     >>>>
>     >>>> There were some important bug fixes in 3.5.2 (from 3.5.1).
>     >>>>
>     >>>> Note I'm not saying yours is one of them, I'm just asking if it's
>     >>>> easy to test and find out. :)
>     >>>>
>     >>>> Regards and best wishes,
>     >>>>
>     >>>> Justin Clift
>     >>>>
>     >>>> --
>     >>>> GlusterFS - http://www.gluster.org
>     >>>>
>     >>>> An open source, distributed file system scaling to several
>     >>>> petabytes, and handling thousands of clients.
>     >>>>
>     >>>> My personal twitter: twitter.com/realjustinclift
>     <http://twitter.com/realjustinclift>
>     >>>>
>     >>>
>     >>
>     >
>     > --
>     > GlusterFS - http://www.gluster.org
>     >
>     > An open source, distributed file system scaling to several
>     > petabytes, and handling thousands of clients.
>     >
>     > My personal twitter: twitter.com/realjustinclift
>     <http://twitter.com/realjustinclift>
>     >
>
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20140907/664dfb0c/attachment.html>