<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Mar 5, 2018 at 8:21 PM, Paul Anderson <span dir="ltr"><<a href="mailto:pha@umich.edu" target="_blank">pha@umich.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
tl;dr summary of below: flock() works, but what does it take to make<br>
sync()/fsync() work in a 3 node GFS cluster?<br>
<br>
I am under the impression that POSIX flock, POSIX<br>
fcntl(F_SETLK/F_GETLK,...), and POSIX read/write/sync/fsync are all<br>
supported in cluster operations, such that in theory, SQLite3 should<br>
be able to atomically lock the file (or a subset of page), modify<br>
pages, flush the pages to gluster, then release the lock, and thus<br>
satisfy the ACID property that SQLite3 appears to try to accomplish on<br>
a local filesystem.<br>
<br>
In a test we wrote that fires off 10 simple concurrernt SQL insert,<br>
read, update loops, we discovered that we at least need to use flock()<br>
around the SQLite3 db connection open/update/close to protect it.<br>
<br>
However, that is not enough - although from testing, it looks like<br>
flock() works as advertised across gluster mounted files, sync/fsync<br>
don't appear to, so we end up getting corruption in the SQLite3 file<br>
(pragma integrity_check generally will show a bunch of problems after<br>
a short test).<br>
<br>
Is what we're trying to do achievable? We're testing using the docker<br>
container gluster/gluster-centos as the three servers, with a php test<br>
inside of php-cli using filesystem mounts. If we mount the gluster FS<br>
via sapk/plugin-gluster into the php-cli containers using docker, we<br>
seem to have better success sometimes, but I haven't figured out why,<br>
yet.<br>
<br>
I did see that I needed to set the server volume parameter<br>
'performance.flush-behind off', otherwise it seems that flushes won't<br>
block as would be needed by SQLite3.<br></blockquote><div><br></div><div>If you are relying on fsync this shouldn't matter as fsync makes sure data is synced to disk.</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Does anyone have any suggestions? Any words of widsom would be much appreciated.<br></blockquote><div><br></div><div>Can you experiment with turning on/off various performance xlators? Based on earlier issues, its likely that there is stale metadata which might be causing the issue (not necessarily improper fsync behavior). I would suggest turning off all performance xlators. You can refer [1] for a related discussion. In theory the only perf xlator relevant for fsync is write-behind and I am not aware of any issues where fsync is not working. Does glusterfs log file has any messages complaining about writes or fsync failing? Does your application use O_DIRECT? If yes, please note that you need to turn the option performance.strict-o-direct on for write-behind to honour O_DIRECT<br></div><div><br></div><div>Also, is it possible to identify nature of corruption - Data or metadata? More detailed explanation will help to RCA the issue.</div><div><br></div><div>Also, is your application running on a single mount or from multiple mounts? Can you collect strace of your application (strace -ff -T -p <pid> -o <file>)? If possible can you also collect fuse-dump using option --dump-fuse while mounting glusterfs?<br></div><div><br></div><div>[1] <a href="http://lists.gluster.org/pipermail/gluster-users/2018-February/033503.html">http://lists.gluster.org/pipermail/gluster-users/2018-February/033503.html</a></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Thanks,<br>
<br>
Paul<br>
______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</blockquote></div><br></div></div>