<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin:0cm;
margin-bottom:.0001pt;
text-indent:21.0pt;
font-size:12.0pt;
font-family:宋体;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:DengXian;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:DengXian;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1799252312;
mso-list-type:hybrid;
mso-list-template-ids:169237442 -1328508680 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Hi, GlusterFS expert,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">We have encounter a coredump of client process “glusterfs” issue recently, and it could be reproduced more easy when the IO load and CPU/memory load is highly during stability
testing.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Our glusterfs release is 3.12.2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">I have copy the call trace of core dump as the below, and have some question, wish can get help from you.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">Do you have encounter related issue? From the call trace, we could see the fd variable seems abnoram with the field “refcount” and “inode”,<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">For wb_inode->this, it become invalid value with
</span><span lang="EN-US">0xffffffffffffff00, did the value “0xffffffffffffff00”is some meaningful value? Because every coredump occurred, the value of inode->this is same with “0xffffffffffffff00”<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:0cm"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian">When I check the source code, in function wb_enqueue_common, could find function __wb_request_unref used instead of wb_request_unref, and we could see though function
wb_request_unref is defined, but never used! Firstly it seems some strange, secondly, in wb_request_unref, there are lock mechanism to avoid race condition, but __wb_request_unref without those mechanism, and we could see there are more occurrence called
from __wb_request_unref, will it lead to race issue?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.5pt;font-family:DengXian"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">[Current thread is 1 (Thread 0x7f54e82a3700 (LWP 6078))]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) bt<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#0 0x00007f54e623197c in <b>wb_fulfill</b> (wb_inode=0x7f54d4066bd0, liabilities=0x7f54d0824440) at write-behind.c:1155<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f54e6233662 in <b>wb_process_queue</b> (wb_inode=0x7f54d4066bd0) at write-behind.c:1728<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f54e6234039 in <b>wb_writev </b>
(frame=0x7f54d406d6c0, this=0x7f54e0014b10, fd=0x7f54d8019d70, vector=0x7f54d0018000, count=1, offset=33554431, flags=32770, iobref=0x7f54d021ec20, xdata=0x0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> at write-behind.c:1842<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#3 0x00007f54e6026fcb in du_writev_resume (ret=0, frame=0x7f54d0002260, opaque=0x7f54d0002260) at disk-usage.c:490<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#4 0x00007f54ece07160 in synctask_wrap () at syncop.c:377<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#5 0x00007f54eb3a2660 in ?? () from /lib64/libc.so.6<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#6 0x0000000000000000 in ?? ()<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p wb_inode<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$6 = (wb_inode_t *) <b>0x7f54d4066bd0</b><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p wb_inode->this<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$1 = (xlator_t *) 0xffffffffffffff00<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) frame 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#1 0x00007f54e6233662 in wb_process_queue (wb_inode=0x7f54d4066bd0) at write-behind.c:1728<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1728 in write-behind.c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p wind_failure<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$2 = 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p *wb_inode<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$3 = {window_conf = 35840637416824320, window_current = 35840643167805440, transit = 35839681019027968, all = {next = 0xb000, prev = 0x7f54d4066bd000}, todo = {next = 0x7f54deadc0de00,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> prev = 0x7f54e00489e000}, liability = {next = 0x7f54000000a200, prev = 0xb000}, temptation = {next = 0x7f54d4066bd000, prev = 0x7f54deadc0de00}, wip = {next = 0x7f54e00489e000, prev = 0x7f54000000a200},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> gen = 45056, size = 35840591659782144, lock = {spinlock = 0, mutex = {__data = {__lock = 0, __count = 8344798, __owner = 0, __nusers = 8344799, __kind = 41472, __spins = 21504, __elision = 127, __list = {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __prev = 0xb000, __next = 0x7f54d4066bd000}},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __size = "\000\000\000\000\336T\177\000\000\000\000\000\337T\177\000\000\242\000\000\000T\177\000\000\260\000\000\000\000\000\000\000\320k\006\324T\177", __align = 35840634501726208}},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> this = 0xffffffffffffff00, dontsync = -1}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) frame 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">#2 0x00007f54e6234039 in wb_writev (frame=0x7f54d406d6c0, this=0x7f54e0014b10, fd=0x7f54d8019d70, vector=0x7f54d0018000, count=1, offset=33554431, flags=32770, iobref=0x7f54d021ec20, xdata=0x0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> at write-behind.c:1842<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">1842 in write-behind.c<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p fd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$4 = (fd_t *)<span style="color:red"> 0x7f54d8019d70</span><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb) p *fd<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">$5 = {pid = 140002378149040, flags = -670836240, refcount = 32596, inode_list = {next = 0x7f54d8019d80, prev = 0x7f54d8019d80}, inode = 0x0, lock = {spinlock = -536740032, mutex = {__data = {<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __lock = -536740032, __count = 32596, __owner = -453505333, __nusers = 32596, __kind = 0, __spins = 0, __elision = 0, __list = {__prev = 0x0, __next = 0x0}},
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> __size = "@\377\001\340T\177\000\000\313\016\370\344T\177", '\000' <repeats 25 times>, __align = 140002512207680}}, _ctx = 0xffffffff, xl_count = 0, lk_ctx = 0x0, anonymous = (unknown: 3623984496)}<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">(gdb)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks & Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">George<o:p></o:p></span></p>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<div>
<div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
</blockquote>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</body>
</html>