<div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Hi <span style="line-height: 23.8px;">Atin,</span></div><div><span style="line-height: 23.8px;"><br></span></div><div><span style="line-height: 23.8px;">Thank your for your reply.</span></div><div><span style="line-height: 23.8px;">It is a question from my customer.</span></div><div><span style="line-height: 23.8px;"><br></span></div><div><span style="line-height: 23.8px;">And I have question about in code snippets as below in the patch of </span><span style="line-height: 23.8px;"> </span><a href="http://review.gluster.org/#/c/16279" target="_blank" style="line-height: 23.8px;">http://review.gluster.org/#/c/<wbr></wbr>16279</a>.</div><div><br></div><div><span style="line-height: 23.8px;">...</span></div><div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>if (!unlink)</div><div><span class="Apple-tab-span" style="white-space: pre;"> </span>ret = sys_unlink (brickpath);</div></div><div><span style="line-height: 23.8px;">...</span></div><div><br></div><div><span style="line-height: 23.8px;">In my understand the sys_unlink should be call when unlink is true, as t</span><span style="line-height: 23.8px;">he variable name is "unlink".</span></div><div><span style="line-height: 23.8px;">And it is appropriate that</span><span style="line-height: 23.8px;"> the </span><span style="line-height: 23.8px;">variable name</span><span style="line-height: 23.8px;"> is no_unlink.</span></div><div><span style="line-height: 23.8px;"><br></span></div><div><span style="line-height: 23.8px;">Why the logic is reverse？</span></div><div><span style="line-height: 23.8px;"><br></span></div><div><span style="line-height: 23.8px;">Thanks, </span></div><div><span style="line-height: 23.8px;">xin</span></div><br><br><br><br><div style="position:relative;zoom:1"></div><div id="divNeteaseMailCard"></div><br>At 2017-01-17 15:53:49, "Atin Mukherjee" <amukherj@redhat.com> wrote:<br> <blockquote id="isReplyContent" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><div dir="ltr">We do use few static code analyser tools like coverity, cppcheck. Vijay would have more details on how frequently we run them. Is there any specific issue you want to report about or this is just a generic query you wanted to put up.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 17, 2017 at 7:42 AM, songxin <span dir="ltr"><<a href="mailto:songxin_1980@126.com" target="_blank">songxin_1980@126.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><br><div>Hi Atin,</div><div>Thank your for your reply.It is very helpful for me.</div><div>And I have a question for you.</div><div><br></div><div>I want to know that if gluster community ever run code static tool, such as valgrind, to check the glusterfs.</div><div><br></div><div>Thanks,</div><div>Xin</div><br><br><br><div style="zoom:1"></div><div id="m_-1029199758057587740divNeteaseMailCard"></div><br>At 2017-01-10 12:02:12, "Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> wrote:<br> <blockquote id="m_-1029199758057587740isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div dir="ltr"><div>Xin,<br><br></div><div>There is a patch [1] attempted to handle this case which is under review.<br><br></div><div>[1] <a href="http://review.gluster.org/#/c/16279" target="_blank">http://review.gluster.org/#/c/<wbr></wbr>16279</a><br></div><div><div><div><br><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 10, 2017 at 7:15 AM, songxin <span dir="ltr"><<a href="mailto:songxin_1980@126.com" target="_blank">songxin_1980@126.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Hi <span style="line-height:23.8px">Atin,</span></div><div><span style="line-height:23.8px"><br></span></div><div>Have you fix this issue?</div><br><div>Thanks,</div><div>Xin</div><div><div class="m_-1029199758057587740h5"><br><br><div style="zoom:1"></div><div id="m_-1029199758057587740m_-2194467258279258135divNeteaseMailCard"></div><br>在 2016-11-25 15:46:25，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> 写道：<br> <blockquote id="m_-1029199758057587740m_-2194467258279258135isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 25, 2016 at 1:14 PM, songxin <span dir="ltr"><<a href="mailto:songxin_1980@126.com" target="_blank">songxin_1980@126.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Hi Atin,</div><div>It seems that this workaround should be done by manual.</div><div>Is that right?</div><div>And even the files in bricks/* may be empty too.</div></div></blockquote><div><br></div><div>Yes, that's right<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div><br></div><div>Do you have a workaround, which is implemented in glusterfs code？</div></div></blockquote><div><br></div><div>Workaround is by nature manual and anything to be done through code should be considered as fix not work around :)<br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div><br></div><div>Thanks,</div><div>Xin</div><div><div class="m_-1029199758057587740m_-2194467258279258135h5"><br><br><br><br><div style="zoom:1"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675divNeteaseMailCard"></div><br>在 2016-11-25 15:36:29，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> 写道：<br> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Nov 25, 2016 at 12:06 PM, songxin <span dir="ltr"><<a href="mailto:songxin_1980@126.com" target="_blank">songxin_1980@126.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div>Hi Atin,</div><div>Do you mean that you have the workaround applicable now？</div><div>Or it will take time to design the workaround？</div><div><br></div><div>If you have <span style="line-height:23.8px">workaround now, could you share it to me ?</span></div></div></blockquote><div><br></div><div>If you end up in having a 0 byte info file you'd need to copy the same info file from other node and put it there and restart glusterd.<br> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial"><div><br></div><div>Thanks,</div><div>Xin,</div><span><br><br><br><br><div style="zoom:1"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627divNeteaseMailCard"></div><br>在 2016-11-24 19:12:07，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" target="_blank">amukherj@redhat.com</a>> 写道：<br> </span><div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675h5"><blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div style="white-space:pre-wrap">Xin - I appreciate your patience. I'd need some more time to pick this item up from my backlog. I believe we have a workaround applicable here too.</div><br><div class="gmail_quote"><div dir="ltr">On Thu, 24 Nov 2016 at 14:24, songxin <<a href="mailto:songxin_1980@126.com" target="_blank">songxin_1980@126.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi <span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Atin,</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Actually, the glusterfs is used in my project.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And our test team find this issue.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">So I want to make sure that whether you plan to fix it.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">if you have plan I will wait you because your method shoud be better than mine.</span></div></div><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="zoom:1" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-21 10:00:36，"Atin Mukherjee" <<a href="mailto:atin.mukherjee83@gmail.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">atin.mukherjee83@gmail.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </div><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi Xin,<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I've not got a chance to look into it yet. delete stale volume function is in place to take care of wiping off volume configuration data which has been deleted from the cluster. However we need to revisit this code to see if this function is anymore needed given we recently added a validation to fail delete request if one of the glusterd is down. I'll get back to you on this.</div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Mon, 21 Nov 2016 at 07:24, songxin <<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi Atin,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thank you for your support.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And any conclusions about this issue?</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div></div><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="zoom:1" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-16 20:59:05，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Tue, Nov 15, 2016 at 1:53 PM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">ok, thank you.<div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="zoom:1" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-15 16:12:34，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Tue, Nov 15, 2016 at 12:47 PM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi Atin,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I think the root cause is in the function glusterd_import_friend_volume as below. </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">int32_t </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_import_friend_volume (dict_t *peer_data, size_t count) </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">{ </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">... </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> ret = glusterd_volinfo_find (new_volinfo->volname, &old_volinfo); </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> if (0 == ret) { </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> (void) gd_check_and_update_rebalance_<wbr></wbr>info (old_volinfo, </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <wbr></wbr> n<wbr></wbr>ew_volinfo); </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> (void) glusterd_delete_stale_volume (old_volinfo, new_volinfo); </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> } </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">... </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> ret = glusterd_store_volinfo (new_volinfo, GLUSTERD_VOLINFO_VER_AC_NONE);<wbr></wbr> </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> if (ret) { </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> gf_msg (this->name, GF_LOG_ERROR, 0, </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> GD_MSG<wbr></wbr>_VOLINFO_STORE_FAIL, "Failed to store " </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> "volin<wbr></wbr>fo for volume %s", new_volinfo->volname); </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> goto out; </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> } </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">... </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">} </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_delete_stale_volume will remove the info and bricks/* and the glusterd_store_volinfo will create the new one. </span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">But if glusterd is killed before rename the info will is empty. </span></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And glusterd will start failed because the infois empty in the next time you start the glusterd.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Any idea， Atin？</div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Give me some time, will check it out, but reading at this analysis looks very well possible if a volume is changed when the glusterd was done on node a and when the same comes up during peer handshake we update the volinfo and during that time glusterd goes down once again. I'll confirm it by tomorrow.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div></blockquote></div></div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I checked the code and it does look like you have got the right RCA for the issue which you simulated through those two scripts. However this can happen even when you try to create a fresh volume and while glusterd tries to write the content into the store and goes down before renaming the info.tmp file you get into the same situation.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I'd really need to think through if this can be fixed. Suggestions are always appreciated.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">BTW, excellent work Xin!<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="line-height:1.7;color:#000000;font-size:14px;font-family:Arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="zoom:1" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-15 12:07:05，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222isReplyContent" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Tue, Nov 15, 2016 at 8:58 AM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi Atin,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I have some clues about this issue.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I could reproduce this issue use the scrip that mentioned in <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1308487" style="color:rgb(59,115,175);font-family:arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">https://bugzilla.redhat.com<wbr></wbr>/show_bug.cgi?id=1308487</a><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px;background-color:rgb(245,245,245)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> .</span></div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I really appreciate your help in trying to nail down this issue. While I am at your email and going through the code to figure out the possible cause for it, unfortunately I don't see any script in the attachment of the bug. Could you please cross check?<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">After I added some debug print,which like below, in glusterd-store.c and I found that the /var/lib/glusterd/vols/xxx<wbr></wbr>/info and <span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">/var/lib/glusterd/vols/xxx<wbr></wbr>/bricks/* are removed. </span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">But other files in </span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">/var/lib/glusterd/vols/xxx/ will not be remove.</span></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">int32_t</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_store_volinfo (glusterd_volinfo_t *volinfo, glusterd_volinfo_ver_ac_t ac)</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">{</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> int32_t ret = -1;</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> GF_ASSERT (volinfo)</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> ret = access("/var/lib/glusterd/vols<wbr></wbr>/gv0/info", F_OK);</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> if(ret < 0)</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> {</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> gf_msg (THIS->name, GF_LOG_ERROR, 0, 0, "info is not exit(%d)", errno);</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> }</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> else</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> {</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> ret = stat("/var/lib/glusterd/vols/g<wbr></wbr>v0/info", &buf);</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> if(ret < 0)</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> {</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> gf_msg (THIS->name, GF_LOG_ERROR, 0, 0, "stat info error");</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> }</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> else</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> {</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> gf_msg (THIS->name, GF_LOG_ERROR, 0, 0, "info size is %lu, inode num is %lu", buf.st_size, buf.st_ino);</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> }</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> }</div></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> glusterd_perform_volinfo_versi<wbr></wbr>on_action (volinfo, ac);</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> ret = glusterd_store_create_volume_d<wbr></wbr>ir (volinfo);</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> if (ret)</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> goto out;</div></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">...</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">}</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">So it is easy to understand why the info or <span style="font-family:arial;line-height:23.8px;white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">10.32.1.144.-opt-lvmdir-c2-<wbr></wbr>bri</span><span style="font-family:arial;line-height:23.8px;white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">ck </span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">sometimes</span><span style="font-family:arial;line-height:23.8px;white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> is empty.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">It is becaue the info file is not exist, and it will be create by “fd = open (path, O_RDWR | O_CREAT | O_APPEND, 0600);</span></font><span style="white-space:pre-wrap;font-family:arial;line-height:1.7" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">” in function </span><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">gf_store_handle_new.</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And the info file is empty before rename.</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">So the info file is empty if glusterd shutdown before rename.</span></font><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></blockquote><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">My question is following.</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="font-family:arial;line-height:23.8px;white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">1.I did not find the point the info is removed.Could you tell me the point where the info and </span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">/bricks/* are removed?</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">2.why the file info and bricks/* is removed?But other files in </span></font><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">var/lib/glusterd/vols/xxx/ are not be removed?</span></div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">AFAIK, we never delete the info file and hence this file is opened with
O_APPEND flag. As I said I will go back and cross check the code once
again.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="arial"><span style="white-space:pre-wrap" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-11 20:34:05，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438isReplyContent" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Fri, Nov 11, 2016 at 4:00 PM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi </span><span style="line-height:23.8px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Atin,</span><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thank you for your support.</div><div style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Sincerely wait for your reply.</div><div style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">By the way, could you make sure that the issue, file info is empty, cause by rename is interrupted in kernel?</div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">As per my RCA on that bug, it looked to be.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-11 15:49:02，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358isReplyContent" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Fri, Nov 11, 2016 at 1:15 PM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi <span style="font-family:arial;line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Atin,</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="font-family:arial;line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thank you for your reply.</span></div></span><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="font-family:arial;line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Actually it is very difficult to reproduce because I don't know when </span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">there was an ongoing commit happening.</span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">It is just a coincidence.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">But I want to make sure the root cause.</span></div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I'll give it a another try and see if this situation can be simulated/reproduced and will keep you posted.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">So I would be grateful if you could answer my questions below.</div><span class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div style="font-family:arial;line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="white-space:pre-wrap;line-height:1.7" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">You said that "This issue is hit at part of the negative testing where while gluster volume set was executed at the same point of time glusterd in another instance was brought down. In the faulty node we could see /var/lib/glusterd/vols/<volnam<wbr></wbr>e>info file been empty whereas the info.tmp file has the correct contents." in comment</span><span style="white-space:pre-wrap;line-height:1.7" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">.</span></div><div style="font-family:arial;line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><pre class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_wrap_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137comment_text_9">I have two questions for you.
1.Could you reproduce this issue by gluster volume set glusterd which was brought down?
2.Could you be certain that this issue is cause by rename is interrupted in kernel?</pre><pre class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_wrap_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137comment_text_9">In my case there are two files, info and 10.32.1.144.-opt-lvmdir-c2-bri<wbr></wbr>ck, are both empty.
But in my view only one rename can be running at the same time because of the big lock.
Why there are two files are empty?
Could rename("info.tmp", "info") and rename("xxx-brick.tmp", "xxx-brick") be running in two thread?</pre></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-11 15:27:03，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467isReplyContent" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Fri, Nov 11, 2016 at 12:38 PM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail- m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi <span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Atin,</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thank you for your reply.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div></span><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">As you said that the info file can only be changed in the </span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_store_volinfo() sequentially because of the big lock.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I have found the similar issue as below that you mentioned. </span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><a href="https://bugzilla.redhat.com/show_bug.cgi?id=1308487" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">https://bugzilla.redhat.com/sh<wbr></wbr>ow_bug.cgi?id=1308487</a></div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Great, so this is what I was actually trying to refer in my first email that I saw a similar issue. Have you got a chance to look at <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1308487#c4" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">https://bugzilla.redhat.com/sh<wbr></wbr>ow_bug.cgi?id=1308487#c4</a> ? But in your case, did you try to bring down glusterd when there was an ongoing commit happening?<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="font-family:arial;white-space:pre-wrap;line-height:1.7" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="font-family:arial;white-space:pre-wrap;line-height:1.7" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">You said that "This issue is hit at part of the negative testing where while gluster volume set was executed at the same point of time glusterd in another instance was brought down. In the faulty node we could see /var/lib/glusterd/vols/<volnam<wbr></wbr>e>info file been empty whereas the info.tmp file has the correct contents." in comment</span><span style="font-family:arial;white-space:pre-wrap;line-height:1.7" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><pre class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_wrap_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137comment_text_9">I have two questions for you.
1.Could you reproduce this issue by gluster volume set glusterd which was brought down?
2.Could you be certain that this issue is cause by rename is interrupted in kernel?</pre><pre class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137bz_wrap_comment_text m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137comment_text_9">In my case there are two files, info and 10.32.1.144.-opt-lvmdir-c2-bri<wbr></wbr>ck, are both empty.
But in my view only one rename can be running at the same time because of the big lock.
Why there are two files are empty?
Could rename("info.tmp", "info") and rename("xxx-brick.tmp", "xxx-brick") be running in two thread?
Thanks,
Xin</pre></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">在 2016-11-11 14:36:40，"Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> 写道：<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137isReplyContent" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Fri, Nov 11, 2016 at 8:33 AM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi Atin,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thank you for your reply.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I have two questions for you.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">1.Are the two files info and info.tmp are only to be created or changed in function glusterd_store_volinfo()? I did not find other point in which the two file are changed.</div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">If we are talking about info file volume then yes, the mentioned function actually takes care of it.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">2.I found that <span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_store_volinfo() will be call in many point by glusterd.Is there a problem of thread synchronization?If so, one thread may open a same file info.tmp using </span><span style="color:rgb(128,0,128);font-family:微软雅黑,"microsoft yahei",arial,helvetica,sans-serif;line-height:22.4px;background-color:rgb(238,238,238)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">O_TRUNC </span><span style="line-height:23.8px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">flag when another thread is writing the info,tmp.Could this case happen?</span></div></div></blockquote><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> In glusterd threads are big lock protected and I don't see a possibility (theoretically) to have two glusterd_store_volinfo () calls at a given point of time.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137m_3281658264091363472h5 m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137m_3281658264091363472m_-3349385561517349848divNeteaseMailCard" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">At 2016-11-10 21:41:06, "Atin Mukherjee" <<a href="mailto:amukherj@redhat.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">amukherj@redhat.com</a>> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> <blockquote id="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137m_3281658264091363472m_-3349385561517349848isReplyContent" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Did you run out of disk space by any chance? AFAIK, the code is like we write new stuffs to .tmp file and rename it back to the original file. In case of a disk space issue I expect both the files to be of non zero size. But having said that I vaguely remember a similar issue (in the form of a bug or an email) landed up once but we couldn't reproduce it, so something is wrong with the atomic update here is what I guess. I'll be glad if you have a reproducer for the same and then we can dig into it further.<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="gmail_extra m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">On Thu, Nov 10, 2016 at 1:32 PM, songxin <span dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><<a href="mailto:songxin_1980@126.com" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">songxin_1980@126.com</a>></span> wrote:<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><blockquote class="gmail_quote m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div style="line-height:1.7;color:rgb(0,0,0);font-size:14px;font-family:arial" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Hi,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">When I start the glusterd some error happened.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And the log is following.</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:34.989365] I [MSGID: 100030] [glusterfsd.c:2318:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.7.6 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO) </span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:34.998356] I [MSGID: 106478] [glusterd.c:1350:init] 0-management: Maximum allowed open file descriptors set to 65536 </span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.000667] I [MSGID: 106479] [glusterd.c:1399:init] 0-management: Using /system/glusterd as working directory</span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.024508] I [MSGID: 106514] [glusterd-store.c:2075:gluster<wbr></wbr>d_restore_op_version] 0-management: Upgrade detected. Setting op-version to minimum : 1 </span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><b class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.025356] E [MSGID: 106206] [glusterd-store.c:2562:gluster<wbr></wbr>d_store_update_volinfo] 0-management: Failed to get next store iter </b></span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><b class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.025401] E [MSGID: 106207] [glusterd-store.c:2844:gluster<wbr></wbr>d_store_retrieve_volume] 0-management: Failed to update volinfo for c_glusterfs volume </b></span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><b class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.025463] E [MSGID: 106201] [glusterd-store.c:3042:gluster<wbr></wbr>d_store_retrieve_volumes] 0-management: Unable to restore volume: c_glusterfs </b></span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><b class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.025544] E [MSGID: 101019] [xlator.c:428:xlator_init] 0-management: Initialization of volume 'management' failed, review your volfile again </b></span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><b class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.025582] E [graph.c:322:glusterfs_graph_i<wbr></wbr>nit] 0-management: initializing translator failed </b></span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><b class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.025629] E [graph.c:661:glusterfs_graph_a<wbr></wbr>ctivate] 0-graph: init failed </b></span><br style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">[2016-11-08 07:58:35.026109] W [glusterfsd.c:1236:cleanup_and<wbr></wbr>_exit] (-->/usr/sbin/glusterd(gluster<wbr></wbr>fs_volumes_init-0x1b260) [0x1000a718] -->/usr/sbin/glusterd(glusterf<wbr></wbr>s_process_volfp-0x1b3b8) [0x1000a5a8] -->/usr/sbin/glusterd(cleanup_<wbr></wbr>and_exit-0x1c02c) [0x100098bc] ) 0-: received signum (0), shutting down </span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And then I found that the size of vols/volume_name/info is 0.It cause glusterd shutdown.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">But I found that vols/volume_name_info.tmp is not 0.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">And I found that there is a brick file </span><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">vols/volume_name/bricks/x<wbr></wbr>xxx.brick is 0, but </span><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">vols/volume_name/bricks/xx<wbr></wbr>xx.brick.tmp is not 0.</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I read the function code </span><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="Arial, sans-serif" color="#333333"><span style="line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_store_volinfo ()</span></font><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> in glusterd-store.c .</span></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span style="color:rgb(51,51,51);font-family:arial,sans-serif;line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">I know that the info.tmp will be rename to info in function </span><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="Arial, sans-serif" color="#333333"><span style="line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">glusterd_store_volume<wbr></wbr>_atomic_update().</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="Arial, sans-serif" color="#333333"><span style="line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><font class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" face="Arial, sans-serif" color="#333333"><span style="line-height:20px" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">But my question is that why the info file is 0 but info.tmp is not 0.</span></font></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Thanks,</div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">Xin</div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">______________________________<wbr></wbr>_________________<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">
Gluster-users mailing list<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">
<a href="mailto:Gluster-users@gluster.org" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">Gluster-users@gluster.org</a><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">http://www.gluster.org/mailman<wbr></wbr>/listinfo/gluster-users</a><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137m_3281658264091363472m_-3349385561517349848gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail-m_-2053880103813163137m_3281658264091363472gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358m_1097009954043043467gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438m_4708060938337307358gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail-m_-7426951264015316438gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435m_-7556161270989908222gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409m_-2874832610256538435gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" data-smartmail="gmail_signature"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div></div></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span></blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" clear="all"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627m_4630254397014683099m_5921458609669446409gmail_signature m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" data-smartmail="gmail_signature"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">~ Atin (atinm)<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div></div></div></div>
</div></div>
</blockquote></div><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><span title="neteasefooter" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"><p class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"> </p></span>______________________________<wbr></wbr>_________________<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">
Gluster-users mailing list<br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">
<a href="mailto:Gluster-users@gluster.org" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">Gluster-users@gluster.org</a><br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">
<a href="http://www.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg" target="_blank">http://www.gluster.org/mailman<wbr></wbr>/listinfo/gluster-users</a></blockquote></div><div dir="ltr" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">-- <br class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg"></div><div data-smartmail="gmail_signature" class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675m_8315869254904041627gmail_msg">--Atin</div>
</blockquote></div></blockquote></div><div dir="ltr">-- <br></div><div data-smartmail="gmail_signature">- Atin (atinm)</div>
</blockquote></div></div></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-1029199758057587740m_-2194467258279258135m_5136503642986701675gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</div></div>
</blockquote></div></div></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br><br clear="all"><span class="HOEnZb"><font color="#888888"><br>-- <br><div class="m_-1029199758057587740m_-2194467258279258135gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div></div></div><span class="HOEnZb"><font color="#888888"><br><br><span title="neteasefooter"><p> </p></span></font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-1029199758057587740gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</font></span></div>
</blockquote></div><br><br><span title="neteasefooter"><p> </p></span></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><br></div><div>~ Atin (atinm)<br></div></div></div></div>
</div>
</blockquote></div><br><br><span title="neteasefooter"><p> </p></span>