[Gluster-devel] Moratorium on new patch acceptance
Shyam
srangana at redhat.com
Thu May 21 14:54:56 UTC 2015
On 05/21/2015 09:33 AM, Vijaikumar M wrote:
>
>
> On Thursday 21 May 2015 06:48 PM, Shyam wrote:
>> On 05/21/2015 04:04 AM, Vijaikumar M wrote:
>>> On Tuesday 19 May 2015 09:50 PM, Shyam wrote:
>>>> On 05/19/2015 11:23 AM, Vijaikumar M wrote:
>>>>
>>>> Did that (in the attached script that I sent) and it still failed.
>>>>
>>>> Please note:
>>>> - This dd command passes (or fails with EDQUOT)
>>>> - dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=512 count=10240
>>>> oflag=append oflag=sync conv=fdatasync
>>>> - We can even drop append and fdatasync, as sync sends a commit per
>>>> block written which is better for the test and quota enforcement,
>>>> whereas fdatasync does one in the end and sometimes fails (with larger
>>>> block sizes, say 1M)
>>>> - We can change bs to [512 - 256k]
>>>>
>>>> - This dd command fails (or writes all the data)
>>>> - dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2 oflag=append
>>>> oflag=sync conv=fdatasync
>>>>
>>>> The reasoning is that when we write a larger block size, NFS sends in
>>>> multiple 256k chunks to write and then sends the commit before the
>>>> next block. As a result if we exceed quota in the *last block* that we
>>>> are writing, we *may* fail. If we exceed quota in the last but one
>>>> block we will pass.
>>>>
>>>> Hope this shorter version explains it better.
>>>>
>>>> (VijayM is educating me on quota (over IM), and it looks like the
>>>> quota update happens as a synctask in the background, so post the
>>>> flush (NFS commit) we may still have a race)
>>>>
>>>> Post education solution:
>>>> - Quota updates on disk xattr as a sync task, as a result if we
>>>> exceeded quota in the n-1th block there is no guarantee that the nth
>>>> block would fail, as the sync task may not have completed
>>>>
>>>> So I think we need to do the following for the quota based tests
>>>> (expanding on the provided patch,
>>>> http://review.gluster.org/#/c/10811/ )
>>>> - First dd that exceeds quota (with either oflag=sync or
>>>> conv=fdatasync so that we do not see any flush behind or write behind
>>>> effects) to be done without checks
>>>> - Next check in an EXPECT_WITHIN that quota is exceeded (maybe add
>>>> checks on the just created/appended file w.r.t its minimum size that
>>>> would make it exceed the quota)
>>>> - Then do a further dd to a new file or append to an existing file to
>>>> get the EDQUOT error
>>>> - Proceed with whatever the test case needs to do next
>>>>
>>>> Suggestions?
>>>>
>>>
>>> Here is my analysis on spurious failure with testcase:
>>> tests/bugs/distribute/bug-1161156.t
>>> In release-3.7, marker is re-factored to use synctask to do background
>>> accounting.
>>> I have done below tests with different combination and found that
>>> parallel writes is causing the spurious failure.
>>> I have filed a bug# 1223658 to track parallel write issue with quota.
>>
>> Agreed with the observations, tallies with mine. Just one addition,
>> when we write 256k or less, the writes become serial as NFS writes in
>> 256k chunks, and due to oflag=sync it follows up with a flush, correct?
>>
> Yes
>
>> Test (2) is interesting, even with marker foreground updates (which is
>> still in the UNWIND path), we observe failures. Do we know why? My
>> analysis/understanding of the same is that we have more in flight IOs
>> that passed quota enforcement (due to accounting on the UNWIND path),
>> does this bear any merit post your tests?
>>
> Yes, my understanding is same that it could be because of more in-flight
> IOs and there is not much impact if the marker is doing background updates.
jFYI, I ran the patch posted for the above, 10 times in my machine, and
it passed all times. One caveat is that at times the file size 0.5 MB
more (see attached). Just stating this so that as we set the rules for
quota tests, we should also add that the attempt should be to write
*more* than the quota limit, and what this *more* is should be at least
a few extra 256k blocks (say 16 at least?). This is to prevent future
tests ending up in a similar regression state.
>
>
>>>
>>>
>>> 1) Parallel writes and Marker background update (Test always fails)
>>> TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2
>>> conv=fdatasync oflag=sync oflag=append
>>>
>>> NFS client breaks 3M writes into multiple 256k chunks and does
>>> parallel writes
>>>
>>> 2) Parallel writes and Marker foreground update (Test always fails)
>>> TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2
>>> conv=fdatasync oflag=sync oflag=append
>>>
>>> Made a marker code change to account quota in foreground (without
>>> synctask)
>>>
>>> 3) Serial writes and Marker background update (Test passed 100/100
>>> times)
>>> TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=256k count=24
>>> conv=fdatasync oflag=sync oflag=append
>>>
>>> Using smaller block size (256k), so that NFS client reduces
>>> parallel writes
>>>
>>> 4) Serial writes and Marker foreground update (Test passed 100/100
>>> times)
>>> TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=256k count=24
>>> conv=fdatasync oflag=sync oflag=append
>>>
>>> Using smaller block size (256k), so that NFS client reduces
>>> parallel writes
>>> Made a marker code change to account quota in foreground (without
>>> synctask)
>>>
>>> 5) Parallel writes on release-3.6 (Test always fails)
>>> TEST ! dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=3M count=2
>>> conv=fdatasync oflag=sync oflag=append
>>> Moved marker xlator above IO-Threads in the graph.
>>>
>>> Thanks,
>>> Vijay
>
-------------- next part --------------
Test modified as follows to check the file size post dd,
# wait for write behind to complete.
EXPECT_WITHIN $MARKER_UPDATE_TIMEOUT "15.0MB" usage "/"
TEST ! quota_dd if=/dev/zero of=$N0/$mydir/newfile_2 bs=256k count=40
---> stat $N0/$mydir/newfile_2 > /dev/tty <---
# Test rename within a directory. It should pass even when the
# corresponding directory quota is filled.
TEST mv $N0/dir/file $N0/dir/newfile_3
Results:
--------
host# for i in {1..10}; do ./tests/bugs/distribute/bug-1161156.t; done
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 9671870521241932594 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:25:37.511565334 -0400
Modify: 2015-05-21 09:25:37.857565398 -0400
Change: 2015-05-21 09:25:37.857565398 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 10709475173786443596 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:26:39.816576835 -0400
Modify: 2015-05-21 09:26:40.139576895 -0400
Change: 2015-05-21 09:26:40.139576895 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 10363430981752982285 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:27:36.347586026 -0400
Modify: 2015-05-21 09:27:36.663585983 -0400
Change: 2015-05-21 09:27:36.663585983 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 11172862175538664493 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:28:33.815578272 -0400
Modify: 2015-05-21 09:28:34.139578229 -0400
Change: 2015-05-21 09:28:34.147578227 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 12608171046258204792 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:29:32.903570300 -0400
Modify: 2015-05-21 09:29:33.230570256 -0400
Change: 2015-05-21 09:29:33.230570256 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5505024 Blocks: 10240 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 9920617755958839421 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:30:30.724562499 -0400
Modify: 2015-05-21 09:30:31.017562460 -0400
Change: 2015-05-21 09:30:31.017562460 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 12864360325492940828 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:31:39.560553212 -0400
Modify: 2015-05-21 09:31:39.872553170 -0400
Change: 2015-05-21 09:31:39.872553170 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5505024 Blocks: 10240 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 11174885303917170550 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:33:11.187542186 -0400
Modify: 2015-05-21 09:33:11.495542150 -0400
Change: 2015-05-21 09:33:11.496542150 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5767168 Blocks: 10752 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 9914333059591019731 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:34:14.258534735 -0400
Modify: 2015-05-21 09:34:14.590534695 -0400
Change: 2015-05-21 09:34:14.590534695 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
1..24
ok 1
ok 2
No volumes present
ok 3
ok 4
ok 5
ok 6
ok 7
ok 8
ok 9
ok 10
ok 11
ok 12
ok 13
ok 14
ok 15
ok 16
ok 17
ok 18
ok 19
ok 20
File: ‘/mnt/nfs/0/dir/newfile_2’
Size: 5505024 Blocks: 10240 IO Block: 1048576 regular file
Device: 2ah/42d Inode: 13416120642134532771 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Context: system_u:object_r:nfs_t:s0
Access: 2015-05-21 09:35:13.752527706 -0400
Modify: 2015-05-21 09:35:14.063527669 -0400
Change: 2015-05-21 09:35:14.064527669 -0400
Birth: -
ok 21
ok 22
Y
ok 23
ok 24
More information about the Gluster-devel
mailing list