[Gluster-devel] How to solve the FSYNC() ERR

Krutika Dhananjay kdhananj at redhat.com
Mon Jul 11 10:51:02 UTC 2016 

What's the application you are running? Sounds like you're using a text
editor like vi(m) to test this out?
Is the application opening the files with O_DIRECT?
Do you have the strace output of the running application that confirms that
it is open()ing the file with O_DIRECT?
Also, what are the offsets and sizes of the writes on this file by this
application in the strace output?

-Krutika


On Mon, Jul 11, 2016 at 2:44 PM, Keiviw <keiviw at 163.com> wrote:

> I have checked the page-aligned, i.e. the file was larger than one page, a
> part of the file(one page size) was saved successfully, and the rest(more
> than one page but less than two pages) was lost.
>
>
>
>
>
>
> At 2016-07-11 12:53:32, "Pranith Kumar Karampuri" <pkarampu at redhat.com>
> wrote:
>
> Is it possible to share the test you are running? As per your volume,
> o-direct is not enabled on your volume, i.e. the file shouldn't be opened
> with o-direct but as per the logs it is giving Invalid Argument as if there
> is something wrong with the arguments when we do o-direct write with wrong
> size. so I would like to test out why exactly is it giving this problem.
> Please note that for o-direct write to succeed, both offset and size should
> be page-aligned, something like multiple of 512 is one way to check it.
>
> On Sun, Jul 10, 2016 at 5:19 PM, Keiviw <keiviw at 163.com> wrote:
>
>> My volume info:
>>
>>     Volume Name: test
>>     Type: Distribute
>>     Volume ID: 9294b122-d81e-4b12-9b5c-46e89ee0e40b
>>     Status: Started
>>     Number of Bricks: 2
>>     Transport-type: tcp
>>     Bricks:
>>     Brick1: compute2:/home/brick1
>>     Brick2: compute2:/home/brick2
>>     Options Reconfigured:
>>     performance.flush-behind: off
>>     storage.linux-aio: off
>> My brick logs(I have cleaned up the history log):
>>     [2016-07-10 11:42:50.577683] E [posix.c:2128:posix_writev]
>> 0-test-posix: write failed: offset 0, Invalid argument
>>     [2016-07-10 11:42:50.577735] I
>> [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569840: WRITEV 5
>> (526a3118-9994-429e-afc0-4aa063606bde) ==> -1 (Invalid argument)
>>     [2016-07-10 11:42:54.583038] E [posix.c:2128:posix_writev]
>> 0-test-posix: write failed: offset 0, Invalid argument
>>     [2016-07-10 11:42:54.583080] I
>> [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569870: WRITEV 5
>> (c3d28f34-8f43-446d-8d0b-80841ae8ec5b) ==> -1 (Invalid argument)
>> My mnt-test-.logs:
>>     [2016-07-10 11:42:50.577816] W
>> [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote
>> operation failed: Invalid argument
>>     [2016-07-10 11:42:50.578508] W [fuse-bridge.c:968:fuse_err_cbk]
>> 0-glusterfs-fuse: 12398282: FSYNC() ERR => -1 (Invalid argument)
>>     [2016-07-10 11:42:54.583156] W
>> [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote
>> operation failed: Invalid argument
>>     [2016-07-10 11:42:54.583762] W [fuse-bridge.c:968:fuse_err_cbk]
>> 0-glusterfs-fuse: 12398317: FSYNC() ERR => -1 (Invalid argument)
>>
>>
>>
>>
>>
>>
>> 在 2016-07-10 19:18:18，"Krutika Dhananjay" <kdhananj at redhat.com> 写道：
>>
>>
>> To me it looks like a case of a flush triggering a write() that was
>> cached by write-behind and because the write buffer
>> did not meet the page alignment requirement with o-direct write, it was
>> failed with EINVAL and the trigger fop - i.e., flush() was failed with the
>> 'Invalid argument' error code.
>>
>> Could you attach the brick logs as well, so that we can confirm the
>> theory?
>>
>> -Krutika
>>
>> On Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>> Pranith/Krutika,
>>>
>>> Your inputs please, IIRC we'd need to turn on some o_direct option here?
>>>
>>>
>>> On Saturday 9 July 2016, Keiviw <keiviw at 163.com> wrote:
>>>
>>>> The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag
>>>> in client protocol open() and create()! How to explain and solve the
>>>> problem?
>>>>
>>>> 发自 网易邮箱大师 <http://u.163.com/signature>
>>>> On 07/09/2016 17:58, Atin Mukherjee wrote:
>>>>
>>>> Any specific reason of using 3.3 given that its really quite old? We
>>>> are at 3.6, 3.7 & 3.8 supportability matrix now.
>>>>
>>>>
>>>> On Saturday 9 July 2016, Keiviw <keiviw at 163.com> wrote:
>>>>
>>>>> hi,
>>>>>     I have installed GlusterFS 3.3.0, and now I get Fsync failures
>>>>> when saving files with the O_DIRECT flag in open() and create().
>>>>>     1, I tried to save a flie in vi and got this error:
>>>>>         "test" E667: Fsync failed
>>>>>     2, I see this in the logs:
>>>>>         [2016-07-07 14:20:10.325400] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 102:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:20:13.930384] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 137:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:20:51.199448] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 174:         FLUSH() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:21:32.804738] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 206:         FLUSH() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:21:43.702146] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 276:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:21:51.296809] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 314:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:21:54.062687] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 349:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:22:54.678960] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 429:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:24:35.546980] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 505:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:24:48.696888] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 538:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:24:55.824752] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 575:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:25:04.364629] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 612:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:27:08.579052] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 713:         FSYNC() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:46:44.635169] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 859:         FLUSH() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:46:55.892041] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 989:         FLUSH() ERR
>>>>> => -1 (Invalid argument)
>>>>>         [2016-07-07 14:51:45.904233] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148747:         FLUSH()
>>>>> ERR => -1 (Invalid argument)
>>>>>         [2016-07-07 14:54:45.603557] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148986:         FLUSH()
>>>>> ERR => -1 (Invalid argument)
>>>>>         [2016-07-07 14:59:18.818972] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 292106:         FLUSH()
>>>>> ERR => -1 (Invalid argument)
>>>>>         [2016-07-07 15:48:32.708207] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 6753592:         FLUSH()
>>>>> ERR => -1 (Invalid argument)
>>>>>         [2016-07-07 16:24:10.476228] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         10733703:
>>>>> FLUSH() ERR => -1 (Invalid argument)
>>>>>         [2016-07-07 16:30:42.388220] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         11497698:
>>>>> FLUSH() ERR => -1 (Invalid argument)
>>>>>         [2016-07-07 16:44:18.247511] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         12037185:
>>>>> FLUSH() ERR => -1 (Invalid argument)
>>>>>         [2016-07-08 01:00:55.131141] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         12181783:
>>>>> FLUSH() ERR => -1 (Invalid argument)
>>>>>         [2016-07-08 06:54:13.418607] W
>>>>> [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         12397060:
>>>>> FLUSH() ERR => -1 (Invalid argument)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Atin
>>>> Sent from iPhone
>>>>
>>>>
>>>>
>>>>
>>>
>>> --
>>> Atin
>>> Sent from iPhone
>>>
>>
>>
>>
>>
>>
>
>
>
> --
> Pranith
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160711/ea4b2b37/attachment.html>

More information about the Gluster-devel mailing list