[Gluster-devel] Moratorium on new patch acceptance

Tue May 19 06:19:56 UTC 2015

----- Original Message -----
> From: "Raghavendra Gowdappa" <rgowdapp at redhat.com>
> To: "Shyam" <srangana at redhat.com>
> Cc: gluster-devel at gluster.org
> Sent: Tuesday, May 19, 2015 11:46:19 AM
> Subject: Re: [Gluster-devel] Moratorium on new patch acceptance
> 
> 
> 
> ----- Original Message -----
> > From: "Shyam" <srangana at redhat.com>
> > To: gluster-devel at gluster.org
> > Sent: Tuesday, May 19, 2015 6:13:06 AM
> > Subject: Re: [Gluster-devel] Moratorium on new patch acceptance
> > 
> > On 05/18/2015 07:05 PM, Shyam wrote:
> > > On 05/18/2015 03:49 PM, Shyam wrote:
> > >> On 05/18/2015 10:33 AM, Vijay Bellur wrote:
> > >>
> > >> The etherpad did not call out, ./tests/bugs/distribute/bug-1161156.t
> > >> which did not have an owner, and so I took a stab at it and below are
> > >> the results.
> > >>
> > >> I also think failure in ./tests/bugs/quota/bug-1038598.t is the same as
> > >> the observation below.
> > >>
> > >> NOTE: Anyone with better knowledge of Quota can possibly chip in as to
> > >> what should we expect in this case and how to correct the expectation
> > >> from these test cases.
> > >>
> > >> (Details of ./tests/bugs/distribute/bug-1161156.t)
> > >> 1) Failure is in TEST #20
> > >>     Failed line: TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=1k
> > >> count=10240 conv=fdatasync
> > >>
> > >> 2) The above line is expected to fail (i.e dd is expected to fail) as,
> > >> the set quota is 20MB and we are attempting to exceed it by another 5MB
> > >> at this point in the test case.
> > >>
> > >> 3) The failure is easily reproducible in my laptop, 2/10 times
> > >>
> > >> 4) On debugging, I see that when the above dd succeeds (or the test
> > >> fails, which means dd succeeded in writing more than the set quota),
> > >> there are no write errors from the bricks or any errors on the final
> > >> COMMIT RPC call to NFS.
> > >>
> > >> As a result the expectation of this test fails.
> > >>
> > >> NOTE: Sometimes there is a write failure from one of the bricks (the
> > >> above test uses AFR as well), but AFR self healing kicks in and fixes
> > >> the problem, as expected, as the write succeeded on one of the replicas.
> > >> I add this observation, as the failed regression run logs, has some
> > >> EDQUOT errors reported in the client xlator, but only from one of the
> > >> client bricks, and there are further AFR self heal logs noted in the
> > >> logs.
> > >>
> > >> 5) When the test case succeeds the writes fail with EDQUOT as expected.
> > >> There are times when the quota is exceeded by say 1MB - 4.8MB, but the
> > >> test case still passes. Which means that, if we were to try to exceed
> > >> the quota by 1MB (instead of the 5MB as in the test case), this test
> > >> case may fail always.
> > >
> > > Here is why I think this passes by quota sometime and not others making
> > > this and the other test case mentioned below spurious.
> > > - Each write is 256K from the client (that is what is sent over the wire)
> > > - If more IO was queued by io-threads after passing quota checks, which
> > > in this 5MB case requires >20 IOs to be queued (16 IOs could be active
> > > in io-threads itself), we could end up writing more than the quota amount
> > >
> > > So, if quota checks to see if a write is violating the quota, and let's
> > > it through, and updates on the UNWIND the space used for future checks,
> > > we could have more IO outstanding than what the quota allows, and as a
> > > result allow such a larger write to pass through, considering IO threads
> > > queue and active IOs as well. Would this be a fair assumption of how
> > > quota works?
> 
> Yes, this is a possible scenario. There is a finite time window between,
> 
> 1. Querying the size of a directory. In other words checking whether current
> write can be allowed
> 2. The "effect" of this write getting reflected in size of all the parent
> directories of a file till root
> 
> If 1 and 2 were atomic, another parallel write which could've exceed the
> quota-limit could not have slipped through. Unfortunately, in the current
> scheme of things they are not atomic. Now there can be parallel writes in
> this test case because of nfs-client and/or glusterfs write-back (though
> we've one single threaded application - dd - running). One way of testing
> this hypothesis is to disable nfs and glusterfs write-back and run the same
> (unmodified) test and the test should succeed always (dd should fail). To
> disable write-back in nfs you can use noac option while mounting.
> 
> The situation becomes worse in real-life scenarios because of parallelism
> involved at many layers:
> 
> 1. multiple applications, each possibly being multithreaded writing to
> possibly many/or single file(s) in a quota subtree
> 2. write-back in NFS-client and glusterfs
> 3. Multiple bricks holding files of a quota-subtree. Each brick processing
> simultaneously many write requests through io-threads.

4. Background accounting of directory sizes _after_ a write is complete.

> 
> I've tried in past to fix the issue, though unsuccessfully. It seems to me
> that one effective strategy is to make enforcement and updation of size of
> parents atomic. But if we do that we end up adding latency of accounting to
> latency of fop. Other options can be explored. But, our Quota functionality
> requirements allow a buffer of 10% while enforcing limits. So, this issue
> has not been high on our priority list till now. So, our tests should also
> expect failures allowing for this 10% buffer.
> 
> > >
> > > I believe this is what is happening in this case. Checking a fix on my
> > > machine, and will post the same if it proves to be help the situation.
> > 
> > Posted a patch to fix the problem: http://review.gluster.org/#/c/10811/
> > 
> > There are arguably other ways to fix/overcome the same, this seemed apt
> > for this test case though.
> > 
> > >
> > >>
> > >> 6) Note on dd with conv=fdatasync
> > >> As one of the fixes attempts to overcome this issue with the addition of
> > >> "conv=fdatasync", wanted to cover that behavior here.
> > >>
> > >> What the above parameter does is to send an NFS_COMMIT (which internally
> > >> becomes a flush FOP) at the end of writing the blocks to the NFS share.
> > >> This commit as a result triggers any pending writes for this file and
> > >> sends the flush to the brick, all of which succeeds at times, resulting
> > >> in the failure of the test case.
> > >>
> > >> NOTE: In the TC ./tests/bugs/quota/bug-1038598.t the failed line is
> > >> pretty much in the same context (LINE 26: TEST ! dd if=/dev/zero
> > >> of=$M0/test_dir/file1.txt bs=1024k count=15 (expecting hard limit to be
> > >> exceeded and there are no write failures in the logs (which should be
> > >> expected with EDQUOT (122))).
> > > _______________________________________________
> > > Gluster-devel mailing list
> > > Gluster-devel at gluster.org
> > > http://www.gluster.org/mailman/listinfo/gluster-devel
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-devel
> > 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-devel
> 