<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 6, 2018 at 10:58 PM, Raghavendra Gowdappa <span dir="ltr"><<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On Tue, Mar 6, 2018 at 10:22 PM, Paul Anderson <span dir="ltr"><<a href="mailto:pha@umich.edu" target="_blank">pha@umich.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Raghavendra,<br>
<br>
I've commited my tests case to <a href="https://github.com/powool/gluster.git" rel="noreferrer" target="_blank">https://github.com/powool/glus<wbr>ter.git</a> -<br>
it's grungy, and a work in progress, but I am happy to take change<br>
suggestions, especially if it will save folks significant time.<br>
<br>
For the rest, I'll reply inline below...<br>
<span class="m_7964123702605126319gmail-"><br>
On Mon, Mar 5, 2018 at 10:39 PM, Raghavendra Gowdappa<br>
<<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br>
> +Csaba.<br>
><br>
> On Tue, Mar 6, 2018 at 2:52 AM, Paul Anderson <<a href="mailto:pha@umich.edu" target="_blank">pha@umich.edu</a>> wrote:<br>
>><br>
>> Raghavendra,<br>
>><br>
>> Thanks very much for your reply.<br>
>><br>
>> I fixed our data corruption problem by disabling the volume<br>
>> performance.write-behind flag as you suggested, and simultaneously<br>
>> disabling caching in my client side mount command.<br>
><br>
><br>
> Good to know it worked. Can you give us the output of<br>
> # gluster volume info<br>
<br>
</span>[root@node-1 /]# gluster volume info<br>
<br>
Volume Name: dockerstore<br>
Type: Replicate<br>
Volume ID: fb08b9f4-0784-4534-9ed3-e01ff7<wbr>1a0144<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 3 = 3<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: 172.18.0.4:/data/glusterfs/sto<wbr>re/dockerstore<br>
Brick2: 172.18.0.3:/data/glusterfs/sto<wbr>re/dockerstore<br>
Brick3: 172.18.0.2:/data/glusterfs/sto<wbr>re/dockerstore<br>
Options Reconfigured:<br>
performance.client-io-threads: off<br>
nfs.disable: on<br>
transport.address-family: inet<br>
locks.mandatory-locking: optimal<br>
performance.flush-behind: off<br>
performance.write-behind: off<br>
<span class="m_7964123702605126319gmail-"><br>
><br>
> We would like to debug the problem in write-behind. Some questions:<br>
><br>
> 1. What version of Glusterfs are you using?<br>
<br>
</span>On the server nodes:<br>
<br>
[root@node-1 /]# gluster --version<br>
glusterfs 3.13.2<br>
Repository revision: git://<a href="http://git.gluster.org/glusterfs.git" rel="noreferrer" target="_blank">git.gluster.org/glusterf<wbr>s.git</a><br>
<br>
On the docker container sqlite test node:<br>
<br>
root@b4055d8547d2:/# glusterfs --version<br>
glusterfs 3.8.8 built on Jan 11 2017 14:07:11<br></blockquote><div><br></div></div></div><div>I guess this is where client is mounted. If I am correct on where glusterfs client is mounted, client is running quite a old version. There have been significant number of fixes between 3.8.8 and current master. </div></div></div></div></blockquote><div><br></div><div>... significant number of fixes to write-behind...</div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>I would suggest to try out 3.13.2 patched with [1]. If you get a chance to try this out, please report back how did the tests go.<br></div></div></div></div></blockquote><div><br></div><div>I would suggest to try out 3.13.2 patched with [1] and run tests with write-behind turned on. <br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><br></div><div>[1] <a href="https://review.gluster.org/19673" target="_blank">https://review.gluster.org/<wbr>19673</a></div><div><div class="h5"><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I recognize that version skew could be an issue.<br>
<span class="m_7964123702605126319gmail-"><br>
> 2. Were you able to figure out whether its stale data or metadata that is<br>
> causing the issue?<br>
<br>
</span>I lean towards stale data based on the only real observation I have:<br>
<br>
While debugging, I put log messages in as to when the flock() is<br>
acquired, and when it is released. There is no instance where two<br>
different processes ever hold the same flock()'d file. From what I<br>
have read, the locks are considered metadata, and they appear to me to<br>
be working, so that's why I'm inclined to think stale data is the<br>
issue.<br>
<span class="m_7964123702605126319gmail-"><br>
><br>
> There have been patches merged in write-behind in recent past and one in the<br>
> works which address metadata consistency. Would like to understand whether<br>
> you've run into any of the already identified issues.<br>
<br>
</span>Agreed!<br>
<br>
Thanks,<br>
<br>
Paul<br>
<div class="m_7964123702605126319gmail-HOEnZb"><div class="m_7964123702605126319gmail-h5"><br>
><br>
> regards,<br>
> Raghavendra<br>
>><br>
>><br>
>> In very modest testing, the flock() case appears to me to work well -<br>
>> before it would corrupt the db within a few transactions.<br>
>><br>
>> Testing using built in sqlite3 locks is better (fcntl range locks),<br>
>> but has some behavioral issues (probably just requires query retry<br>
>> when the file is locked). I'll research this more, although the test<br>
>> case is not critical to our use case.<br>
>><br>
>> There are no signs of O_DIRECT use in the sqlite3 code that I can see.<br>
>><br>
>> I intend to set up tests that run much longer than a few minutes, to<br>
>> see if there are any longer term issues. Also, I want to experiment<br>
>> with data durability by killing various gluster server nodes during<br>
>> the tests.<br>
>><br>
>> If anyone would like our test scripts, I can either tar them up and<br>
>> email them or put them in github - either is fine with me. (they rely<br>
>> on current builds of docker and docker-compose)<br>
>><br>
>> Thanks again!!<br>
>><br>
>> Paul<br>
>><br>
>> On Mon, Mar 5, 2018 at 11:26 AM, Raghavendra Gowdappa<br>
>> <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br>
>> ><br>
>> ><br>
>> > On Mon, Mar 5, 2018 at 8:21 PM, Paul Anderson <<a href="mailto:pha@umich.edu" target="_blank">pha@umich.edu</a>> wrote:<br>
>> >><br>
>> >> Hi,<br>
>> >><br>
>> >> tl;dr summary of below: flock() works, but what does it take to make<br>
>> >> sync()/fsync() work in a 3 node GFS cluster?<br>
>> >><br>
>> >> I am under the impression that POSIX flock, POSIX<br>
>> >> fcntl(F_SETLK/F_GETLK,...), and POSIX read/write/sync/fsync are all<br>
>> >> supported in cluster operations, such that in theory, SQLite3 should<br>
>> >> be able to atomically lock the file (or a subset of page), modify<br>
>> >> pages, flush the pages to gluster, then release the lock, and thus<br>
>> >> satisfy the ACID property that SQLite3 appears to try to accomplish on<br>
>> >> a local filesystem.<br>
>> >><br>
>> >> In a test we wrote that fires off 10 simple concurrernt SQL insert,<br>
>> >> read, update loops, we discovered that we at least need to use flock()<br>
>> >> around the SQLite3 db connection open/update/close to protect it.<br>
>> >><br>
>> >> However, that is not enough - although from testing, it looks like<br>
>> >> flock() works as advertised across gluster mounted files, sync/fsync<br>
>> >> don't appear to, so we end up getting corruption in the SQLite3 file<br>
>> >> (pragma integrity_check generally will show a bunch of problems after<br>
>> >> a short test).<br>
>> >><br>
>> >> Is what we're trying to do achievable? We're testing using the docker<br>
>> >> container gluster/gluster-centos as the three servers, with a php test<br>
>> >> inside of php-cli using filesystem mounts. If we mount the gluster FS<br>
>> >> via sapk/plugin-gluster into the php-cli containers using docker, we<br>
>> >> seem to have better success sometimes, but I haven't figured out why,<br>
>> >> yet.<br>
>> >><br>
>> >> I did see that I needed to set the server volume parameter<br>
>> >> 'performance.flush-behind off', otherwise it seems that flushes won't<br>
>> >> block as would be needed by SQLite3.<br>
>> ><br>
>> ><br>
>> > If you are relying on fsync this shouldn't matter as fsync makes sure<br>
>> > data<br>
>> > is synced to disk.<br>
>> ><br>
>> >><br>
>> >> Does anyone have any suggestions? Any words of widsom would be much<br>
>> >> appreciated.<br>
>> ><br>
>> ><br>
>> > Can you experiment with turning on/off various performance xlators?<br>
>> > Based on<br>
>> > earlier issues, its likely that there is stale metadata which might be<br>
>> > causing the issue (not necessarily improper fsync behavior). I would<br>
>> > suggest<br>
>> > turning off all performance xlators. You can refer [1] for a related<br>
>> > discussion. In theory the only perf xlator relevant for fsync is<br>
>> > write-behind and I am not aware of any issues where fsync is not<br>
>> > working.<br>
>> > Does glusterfs log file has any messages complaining about writes or<br>
>> > fsync<br>
>> > failing? Does your application use O_DIRECT? If yes, please note that<br>
>> > you<br>
>> > need to turn the option performance.strict-o-direct on for write-behind<br>
>> > to<br>
>> > honour O_DIRECT<br>
>> ><br>
>> > Also, is it possible to identify nature of corruption - Data or<br>
>> > metadata?<br>
>> > More detailed explanation will help to RCA the issue.<br>
>> ><br>
>> > Also, is your application running on a single mount or from multiple<br>
>> > mounts?<br>
>> > Can you collect strace of your application (strace -ff -T -p <pid> -o<br>
>> > <file>)? If possible can you also collect fuse-dump using option<br>
>> > --dump-fuse<br>
>> > while mounting glusterfs?<br>
>> ><br>
>> > [1]<br>
>> ><br>
>> > <a href="http://lists.gluster.org/pipermail/gluster-users/2018-February/033503.html" rel="noreferrer" target="_blank">http://lists.gluster.org/piper<wbr>mail/gluster-users/2018-Februa<wbr>ry/033503.html</a><br>
>> ><br>
>> >><br>
>> >> Thanks,<br>
>> >><br>
>> >> Paul<br>
>> >> ______________________________<wbr>_________________<br>
>> >> Gluster-users mailing list<br>
>> >> <a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
>> >> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br>
>> ><br>
>> ><br>
><br>
><br>
</div></div></blockquote></div></div></div><br></div></div>
</blockquote></div><br></div></div>