<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 9, 2017 at 3:37 PM, ABHISHEK PALIWAL <span dir="ltr"><<a href="mailto:abhishpaliwal@gmail.com" target="_blank">abhishpaliwal@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">+ Muthu-vingeshwaran<br></div><div class="gmail_extra"><div><div class="gmail-h5"><br><div class="gmail_quote">On Tue, May 9, 2017 at 11:30 AM, ABHISHEK PALIWAL <span dir="ltr"><<a href="mailto:abhishpaliwal@gmail.com" target="_blank">abhishpaliwal@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Hi Atin/Team,<br><br></div>We are using gluster-3.7.6 with setup of two brick and while restart of system I have seen that the glusterd daemon is getting failed from start.<br><br><br></div><div>At the time of analyzing the logs from etc-glusterfs.......log file I have received the below logs<br></div><div><div><div><div><div><br><br><span lang="sv"><div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font size="2" face="Calibri,sans-serif"><span style="font-size:11pt">[2017-05-06 03:33:39.798087] I [MSGID: 100030] [glusterfsd.c:2348:main] 0-/usr/sbin/glusterd:
Started running /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.807859] I [MSGID: 106478] [glusterd.c:1350:init] 0-management:
Maximum allowed open file descriptors set to 65536</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.807974] I [MSGID: 106479] [glusterd.c:1399:init] 0-management:
Using /system/glusterd as working directory</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.826833] I [MSGID: 106513] [glusterd-store.c:2047:gluster<wbr>d_restore_op_version]
0-glusterd: retrieved op-version: 30706</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font color="red" size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.827515] E [MSGID: 106206] [glusterd-store.c:2562:gluster<wbr>d_store_update_volinfo]
0-management: Failed to get next store iter</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font color="red" size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.827563] E [MSGID: 106207] [glusterd-store.c:2844:gluster<wbr>d_store_retrieve_volume]
0-management: Failed to update volinfo for c_glusterfs volume</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font color="red" size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.827625] E [MSGID: 106201] [glusterd-store.c:3042:gluster<wbr>d_store_retrieve_volumes]
0-management: Unable to restore volume: c_glusterfs</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font color="red" size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.827722] E [MSGID: 101019] [xlator.c:428:xlator_init]
0-management: Initialization of volume 'management' failed, review your volfile again</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font color="red" size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.827762] E [graph.c:322:glusterfs_graph_i<wbr>nit]
0-management: initializing translator failed</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font color="red" size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06 03:33:39.827784] E [graph.c:661:glusterfs_graph_a<wbr>ctivate]
0-graph: init failed</span></font></span></font></div>
<div style="margin:0px"><font size="3" face="Times New Roman,serif"><span style="font-size:12pt"><font size="2" face="Calibri,sans-serif"><span style="font-size:11pt" lang="en-US">[2017-05-06
03:33:39.828396] W [glusterfsd.c:1238:cleanup_and<wbr>_exit]
(-->/usr/sbin/glusterd(gluster<wbr>fs_volumes_init-0x1b0b8)
[0x1000a648] -->/usr/sbin/glusterd(glusterf<wbr>s_process_volfp-0x1b210)
[0x1000a4d8] -->/usr/sbin/glusterd(cleanup_<wbr>and_exit-0x1beac)
[0x100097ac] ) 0-: received signum (0), shutting down</span></font></span></font></div></span></div></div></div></div></div></div></blockquote></div></div></div></div></blockquote><div><br></div><div>Abhishek,<br><br>This patch needs to be thoroughly reviewed to ensure that it doesn't cause any regression given this touches on the core store management functionality of glusterd. AFAICT, we get into an empty info file only when volume set operation is executed and in parallel one of the glusterd instance in other nodes have been brought down and whole sequence of operation happens in a loop. The test case through which you can get into this situation is not something you'd hit in production. Please help me to understand the urgency here.<br><br></div><div>Also in one of the earlier thread, I did mention the workaround of this issue back to Xin through <a href="http://lists.gluster.org/pipermail/gluster-users/2017-January/029600.html">http://lists.gluster.org/pipermail/gluster-users/2017-January/029600.html</a><br><pre>"If you end up in having a 0 byte info file you'd need to copy the same info file from other node and put it there and restart glusterd"<br><br></pre></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail_extra"><div><div class="gmail-h5"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div><div><br><br></div><div>I have found one of the existing case is there and also solution patch is available but the status of that patch in "cannot merge". Also the "info" file is empty and "info.tmp" file present in "lib/glusterd/vol" directory.<br><span lang="sv"></span></div><div><div class="gmail-m_6061362542201391447m_-4566318753540493684gmail_signature"><div dir="ltr"><br>Below is the link of the existing case.<br><br><a href="https://review.gluster.org/#/c/16279/5" target="_blank">https://review.gluster.org/#/c<wbr>/16279/5</a><br><br></div><div>please let me know what is the plan of community to provide the solution of this problem and in which version.<br></div><div dir="ltr"><br>Regards<span class="gmail-m_6061362542201391447HOEnZb"><font color="#888888"><br>
Abhishek Paliwal<br>
</font></span></div></div>
</div></div></div></div></div></div>
</blockquote></div><br><br clear="all"><br></div></div><span class="gmail-HOEnZb"><font color="#888888">-- <br><div class="gmail-m_6061362542201391447gmail_signature"><div dir="ltr"><br><br><br><br>Regards<br>
Abhishek Paliwal<br>
</div></div>
</font></span></div>
</blockquote></div><br></div></div>