<div dir="ltr"><div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Dec 28, 2018 at 3:13 AM Dmitry Isakbayev <<a href="mailto:isakdim@gmail.com">isakdim@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Raghavendra,</div><div dir="ltr"><br></div><div>Thank for the suggestion. </div><div><br></div><div><br></div><div>I am suing </div><div><br></div><div dir="ltr">[root@jl-fanexoss1p glusterfs]# gluster --version</div><div dir="ltr">glusterfs 5.0</div><div dir="ltr"><br></div><div>On </div><div><div>[root@jl-fanexoss1p glusterfs]# hostnamectl</div><div> Icon name: computer-vm<br></div><div> Chassis: vm</div><div> Machine ID: e44b8478ef7a467d98363614f4e50535</div><div> Boot ID: eed98992fdda4c88bdd459a89101766b</div><div> Virtualization: vmware</div><div> Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo)</div><div> CPE OS Name: cpe:/o:redhat:enterprise_linux:7.5:GA:server</div><div> Kernel: Linux 3.10.0-862.14.4.el7.x86_64</div><div> Architecture: x86-64</div></div><div><br></div><div dir="ltr"><br></div><div>I have configured the following options</div><div><br></div><div dir="ltr">[root@jl-fanexoss1p glusterfs]# gluster volume info</div><div dir="ltr">Volume Name: gv0</div><div dir="ltr">Type: Replicate</div><div dir="ltr">Volume ID: 5ffbda09-c5e2-4abc-b89e-79b5d8a40824</div><div dir="ltr">Status: Started</div><div dir="ltr">Snapshot Count: 0</div><div dir="ltr">Number of Bricks: 1 x 3 = 3</div><div dir="ltr">Transport-type: tcp</div><div dir="ltr">Bricks:</div><div dir="ltr">Brick1: jl-fanexoss1p.cspire.net:/data/brick1/gv0</div><div dir="ltr">Brick2: sl-fanexoss2p.cspire.net:/data/brick1/gv0</div><div dir="ltr">Brick3: nxquorum1p.cspire.net:/data/brick1/gv0</div><div dir="ltr">Options Reconfigured:</div><div dir="ltr">performance.io-cache: off</div><div dir="ltr">performance.stat-prefetch: off</div><div dir="ltr">performance.quick-read: off</div><div dir="ltr">performance.parallel-readdir: off</div><div dir="ltr">performance.readdir-ahead: off</div><div dir="ltr">performance.write-behind: off</div><div dir="ltr">performance.read-ahead: off</div><div dir="ltr">performance.client-io-threads: off</div><div dir="ltr">nfs.disable: on</div><div dir="ltr">transport.address-family: inet</div><div><br></div><div>I don't know if it is related, but I am seeing a lot of </div><div><div>[2018-12-27 20:19:23.776080] W [MSGID: 114031] [client-rpc-fops_v2.c:1932:client4_0_seek_cbk] 2-gv0-client-0: remote operation failed [No such device or address]</div><div>[2018-12-27 20:19:47.735190] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 2-epoll: Failed to dispatch handler<br></div></div></div></div></div></div></blockquote><div><br></div><div>These msgs were introduced by patch [1]. To the best of my knowledge they are benign. We'll be sending a patch to fix these msgs though.<br><br></div><div><a class="gmail_plusreply" id="plusReplyChip-1" href="mailto:moagrawa@redhat.com" tabindex="-1">+Mohit Agrawal</a> <a class="gmail_plusreply" id="plusReplyChip-2" href="mailto:mchangir@redhat.com" tabindex="-1">+Milind Changire</a> . Can you try to identify why we are seeing these messages? If possible please send a patch to fix this.<br></div><div><br></div><div>[1] <a href="https://review.gluster.org/r/I578c3fc67713f4234bd3abbec5d3fbba19059ea5">https://review.gluster.org/r/I578c3fc67713f4234bd3abbec5d3fbba19059ea5</a><br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><div></div><div><br></div><div>And <a href="http://java.io" target="_blank">java.io</a> exceptions trying to rename files.</div></div></div></div></div></div></blockquote><div><br></div><div>When you see the errors is it possible to collect,<br></div><div>* strace of the java application (strace -ff -v ...)<br></div><div>* fuse-dump of the glusterfs mount (use option --dump-fuse while mounting)?<br><br></div><div>I also need another favour from you. By trail and error, can you point out which of the many performance xlators you've turned off is causing the issue?<br><br></div><div>The above two data-points will help us to fix the problem.<br><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><br class="gmail-m_-2992392695285668653gmail-Apple-interchange-newline"></div><div>Thank You,</div><div>Dmitry</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Dec 27, 2018 at 3:48 PM Raghavendra Gowdappa <<a href="mailto:rgowdapp@redhat.com" target="_blank">rgowdapp@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>What version of glusterfs are you using? It might be either<br>* a stale metadata issue. <br></div>* inconsistent ctime issue.<br><br><div>Can you try turning off all performance xlators? If the issue is 1, that should help.<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Dec 28, 2018 at 1:51 AM Dmitry Isakbayev <<a href="mailto:isakdim@gmail.com" target="_blank">isakdim@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Attempted to set <span style="color:rgb(36,41,46);font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px">'performance.read-ahead off` according to </span><a rel="nofollow" href="https://jira.apache.org/jira/browse/AMQ-7041" style="box-sizing:border-box;color:rgb(3,102,214);outline-width:0px;font-family:-apple-system,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";font-size:14px" target="_blank">https://jira.apache.org/jira/browse/AMQ-7041</a><div>That did not help.</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 24, 2018 at 2:11 PM Dmitry Isakbayev <<a href="mailto:isakdim@gmail.com" target="_blank">isakdim@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">The core file generated by JVM suggests that it happens because the file is changing while it is being read - <a href="https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8186557" target="_blank">https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8186557</a>.</div><div>The application reads in the zipfile and goes through the zip entries, then reloads the file and goes the zip entries again. It does so 3 times. The application never crushes on the 1st cycle but sometimes crushes on the 2nd or 3rd cycle.</div><div>The zip file is generated about 20 seconds prior to it being used and is not updated or even used by any other application. I have never seen this problem on a plain file system.</div><div><br></div><div>I would appreciate any suggestions on how to go debugging this issue. I can change the source code of the java application.</div><div><br></div><div>Regards,</div><div>Dmitry</div><div><br></div><div dir="ltr"><br></div></div>
</blockquote></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div>
</blockquote></div></div></div></div>
</blockquote></div></div></div>