[Gluster-devel] How to solve the FSYNC() ERR

Keiviw keiviw at 163.com
Mon Jul 11 12:38:02 UTC 2016 

    Yes, I just used a text editor i.e. vi to test this out, but my real application is surveillance camera system. The video was stored in GlusterFS by FUSE and Samba. 
    Because of the surveillance camera system 7*24 working mode, it doesn't have to cache the files. So I added O_DIRECT flag in GlusterFS client protocol open() and create(). According my test, I wrote a video file into GlusterFS in windows, then I didn't find any cache in GlusterFS node, and I read a video file in windows, it also wasn't cached in GlusterFS node, but some frames in the video file(at the last) was lost, just like my vi test. 
    IIRC the page-alighed may be the reason of this question. How to solve this problem? Or in GlusterFS 3.6, 3.7, is there the option to enable io-direct mode to avoid the video cache in GlusterFS node?
    






At 2016-07-11 18:51:02, "Krutika Dhananjay" <kdhananj at redhat.com> wrote:

What's the application you are running? Sounds like you're using a text editor like vi(m) to test this out?
Is the application opening the files with O_DIRECT?

Do you have the strace output of the running application that confirms that it is open()ing the file with O_DIRECT?

Also, what are the offsets and sizes of the writes on this file by this application in the strace output?


-Krutika





On Mon, Jul 11, 2016 at 2:44 PM, Keiviw <keiviw at 163.com> wrote:

I have checked the page-aligned, i.e. the file was larger than one page, a part of the file(one page size) was saved successfully, and the rest(more than one page but less than two pages) was lost.







At 2016-07-11 12:53:32, "Pranith Kumar Karampuri" <pkarampu at redhat.com> wrote:

Is it possible to share the test you are running? As per your volume, o-direct is not enabled on your volume, i.e. the file shouldn't be opened with o-direct but as per the logs it is giving Invalid Argument as if there is something wrong with the arguments when we do o-direct write with wrong size. so I would like to test out why exactly is it giving this problem. Please note that for o-direct write to succeed, both offset and size should be page-aligned, something like multiple of 512 is one way to check it.



On Sun, Jul 10, 2016 at 5:19 PM, Keiviw <keiviw at 163.com> wrote:

My volume info:
    
    Volume Name: test
    Type: Distribute
    Volume ID: 9294b122-d81e-4b12-9b5c-46e89ee0e40b
    Status: Started
    Number of Bricks: 2
    Transport-type: tcp
    Bricks:
    Brick1: compute2:/home/brick1
    Brick2: compute2:/home/brick2
    Options Reconfigured:
    performance.flush-behind: off
    storage.linux-aio: off
My brick logs(I have cleaned up the history log):
    [2016-07-10 11:42:50.577683] E [posix.c:2128:posix_writev] 0-test-posix: write failed: offset 0, Invalid argument
    [2016-07-10 11:42:50.577735] I [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569840: WRITEV 5 (526a3118-9994-429e-afc0-4aa063606bde) ==> -1 (Invalid argument)
    [2016-07-10 11:42:54.583038] E [posix.c:2128:posix_writev] 0-test-posix: write failed: offset 0, Invalid argument
    [2016-07-10 11:42:54.583080] I [server3_1-fops.c:1414:server_writev_cbk] 0-test-server: 8569870: WRITEV 5 (c3d28f34-8f43-446d-8d0b-80841ae8ec5b) ==> -1 (Invalid argument)
My mnt-test-.logs:
    [2016-07-10 11:42:50.577816] W [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote operation failed: Invalid argument
    [2016-07-10 11:42:50.578508] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12398282: FSYNC() ERR => -1 (Invalid argument)
    [2016-07-10 11:42:54.583156] W [client3_1-fops.c:876:client3_1_writev_cbk] 0-test-client-1: remote operation failed: Invalid argument
    [2016-07-10 11:42:54.583762] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 12398317: FSYNC() ERR => -1 (Invalid argument)









在 2016-07-10 19:18:18，"Krutika Dhananjay" <kdhananj at redhat.com> 写道：


To me it looks like a case of a flush triggering a write() that was cached by write-behind and because the write buffer

did not meet the page alignment requirement with o-direct write, it was failed with EINVAL and the trigger fop - i.e., flush() was failed with the 'Invalid argument' error code.


Could you attach the brick logs as well, so that we can confirm the theory?



-Krutika


On Sat, Jul 9, 2016 at 9:31 PM, Atin Mukherjee <amukherj at redhat.com> wrote:
Pranith/Krutika,


Your inputs please, IIRC we'd need to turn on some o_direct option here?



On Saturday 9 July 2016, Keiviw <keiviw at 163.com> wrote:

The errors also occured in GlusterFS 3.6.7，I just add the O_DIRECT flag in client protocol open() and create()! How to explain and solve the problem?


发自 网易邮箱大师
On 07/09/2016 17:58, Atin Mukherjee wrote:
Any specific reason of using 3.3 given that its really quite old? We are at 3.6, 3.7 & 3.8 supportability matrix now.


On Saturday 9 July 2016, Keiviw <keiviw at 163.com> wrote:

hi,
    I have installed GlusterFS 3.3.0, and now I get Fsync failures when saving files with the O_DIRECT flag in open() and create().
    1, I tried to save a flie in vi and got this error:
        "test" E667: Fsync failed
    2, I see this in the logs:    
        [2016-07-07 14:20:10.325400] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 102:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:20:13.930384] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 137:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:20:51.199448] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 174:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 14:21:32.804738] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 206:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 14:21:43.702146] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 276:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:21:51.296809] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 314:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:21:54.062687] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 349:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:22:54.678960] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 429:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:24:35.546980] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 505:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:24:48.696888] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 538:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:24:55.824752] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 575:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:25:04.364629] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 612:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:27:08.579052] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 713:         FSYNC() ERR => -1 (Invalid argument)
        [2016-07-07 14:46:44.635169] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 859:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 14:46:55.892041] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 989:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 14:51:45.904233] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148747:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 14:54:45.603557] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 148986:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 14:59:18.818972] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 292106:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 15:48:32.708207] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse: 6753592:         FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 16:24:10.476228] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         10733703: FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 16:30:42.388220] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         11497698: FLUSH() ERR => -1 (Invalid argument)
        [2016-07-07 16:44:18.247511] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         12037185: FLUSH() ERR => -1 (Invalid argument)
        [2016-07-08 01:00:55.131141] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         12181783: FLUSH() ERR => -1 (Invalid argument)
        [2016-07-08 06:54:13.418607] W [fuse-bridge.c:968:fuse_err_cbk] 0-glusterfs-fuse:         12397060: FLUSH() ERR => -1 (Invalid argument)




 



--
Atin
Sent from iPhone






--
Atin
Sent from iPhone







 




--

Pranith





 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-devel/attachments/20160711/4f0033db/attachment-0001.html>

More information about the Gluster-devel mailing list