<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:DengXian;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@DengXian";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
font-size:10.5pt;
font-family:DengXian;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:DengXian;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:DengXian;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:DengXian;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FI" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">yes, in test code there was such issue. The production code that occasionally has issues with first time accessing the file however
has proper parameters in use when opening a file.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">In production the issue is now gone when the same file being read/written is no longer locked with another file descriptor usin gflock/fcntl
but the access prevention is done with a separate lock file, so avoiding dual handle to same file seems to fix the problem. Perhaps I'll need to tune the production code still a bit so that further information could be obtained why the file does not contain
valid data right after writing. <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US">br, sami<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" align="left" style="text-align:left"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Lian, George (NSB - CN/Hangzhou)
<br>
<b>Sent:</b> Thursday, December 28, 2017 8:21 AM<br>
<b>To:</b> Gluster-devel@gluster.org; Venetjoki, Sami (Nokia - FI/Espoo) <sami.venetjoki@nokia.com><br>
<b>Cc:</b> Zhong, Hua (NSB - CN/Hangzhou) <hua.zhong@nokia-sbell.com>; Li, Deqian (NSB - CN/Hangzhou) <deqian.li@nokia-sbell.com>; I_MBB_HETRAN_SW3_TEAM_MATRIX_GMS <I_MBB_HETRAN_SW3_TEAM_MATRIX@internal.nsn.com><br>
<b>Subject:</b> RE: Sometimes read failed just after write before file closed when try 100 times in different new files<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" align="left" style="text-align:left"><o:p> </o:p></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">Hi, Gluster Expert,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">Sorry for wrong code
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">open(argv[1], O_CREAT|O_RDWR|S_IRUSR|S_IWUSR)
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">in previous mail, after correct it , the issues seems GONE.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">But I still confuse why wrong code not failed in EXT4 file system.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">After I check tcpdump packet via network, it show that in CREATE process, the mode of file sometime became to
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN"> Mode: 0101000,
<span style="background:yellow;mso-highlight:yellow">S_ISVTX, Reserved</span> </span>
<span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">, with this mode, the file could be seen with
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">ls</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">”</span><span lang="EN-US" style="mso-fareast-language:ZH-CN"> command,
but can</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">’</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">t seen by
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">hexdump</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">”</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">command
even with root privilege, </span><span lang="EN-US" style="font-family:"Calibri",sans-serif;mso-fareast-language:ZH-CN"> </span><span lang="EN-US" style="mso-fareast-language:ZH-CN">but if run this wrong program in ext4 file system, this MODE can</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">’</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">t
be occurred.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">So I suggest maybe Gluster could do some inprovment for this MODE , so that can avoid thiss can</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">’</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">t
access file with wrong API called.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">@Sami,
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">After I check the manual of open API, the correct format should be
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Calibri",sans-serif;mso-fareast-language:ZH-CN"> </span><span lang="EN-US" style="mso-fareast-language:ZH-CN">int open(const char *pathname, int flags);<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-family:"Calibri",sans-serif;mso-fareast-language:ZH-CN"> </span><span lang="EN-US" style="mso-fareast-language:ZH-CN"> int open(const char *pathname, int flags, mode_t mode);<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">for your test code, I suppose the correct call should be
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">open(argv[1], O_CREAT|O_RDWR<span style="background:yellow;mso-highlight:yellow">,</span>S_IRUSR|S_IWUSR)
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">after I correct it , c and C++ code all PASS in 3.6 and 3.12.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">George<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal" align="left" style="text-align:left"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:ZH-CN">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:ZH-CN">
Lian, George (NSB - CN/Hangzhou) <br>
<b>Sent:</b> Wednesday, December 27, 2017 5:23 PM<br>
<b>To:</b> <a href="mailto:Gluster-devel@gluster.org">Gluster-devel@gluster.org</a><br>
<b>Cc:</b> Zhong, Hua (NSB - CN/Hangzhou) <<a href="mailto:hua.zhong@nokia-sbell.com">hua.zhong@nokia-sbell.com</a>>; Venetjoki, Sami (Nokia - FI/Espoo) <<a href="mailto:sami.venetjoki@nokia.com">sami.venetjoki@nokia.com</a>>; Li, Deqian (NSB - CN/Hangzhou)
<<a href="mailto:deqian.li@nokia-sbell.com">deqian.li@nokia-sbell.com</a>><br>
<b>Subject:</b> Sometimes read failed just after write before file closed when try 100 times in different new files<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal" align="left" style="text-align:left"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">Hi, Gluster expert,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">I has just raise a bug with
<a href="https://bugzilla.redhat.com/show_bug.cgi?id=1529237">https://bugzilla.redhat.com/show_bug.cgi?id=1529237</a> on gluster community.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">I also attached the reproduce source code and shell script here FYI.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">From my investigation , I suppose the issue maybe caused by
</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">“</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">cache in client is invalid but the file still not been persistented in server brick</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">”</span><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">But I can</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">’</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">t find the root cause from the debug log which attached
in <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1529237">https://bugzilla.redhat.com/show_bug.cgi?id=1529237</a>.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">From the following log I abstract client log from the attachment in the Bugzilla, I highlight it with
<span style="background:yellow;mso-highlight:yellow">YELLOW</span> which in read phase, you can see
<span style="background:yellow;mso-highlight:yellow">both volume(test-client-0 and test-client-1) are failed</span>
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">I just suppose it maybe some abnormal when it in read phase, but I can</span><span lang="ZH-CN" style="mso-fareast-language:ZH-CN">’</span><span lang="EN-US" style="mso-fareast-language:ZH-CN">t
any usefully log in server brick log from myself, so your helps is highly appreciated!<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332524] T [MSGID: 0] [afr-inode-read.c:286:afr_fstat_cbk] 12-stack-trace: stack-address: 0x7fb5ec010680, test-replicate-0 returned 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332724] T [MSGID: 0] [syncop.c:1715:syncop_fgetxattr] 12-stack-trace: stack-address: 0x7fb5fc0618d0, winding from test-dht to test-replicate-0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332768] D [MSGID: 0] [afr-read-txn.c:220:afr_read_txn] 12-test-replicate-0: 70e0f7ee-2f0a-4364-a38e-96e7031eee7b: generation now vs cached: 2, 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332791] T [MSGID: 0] [afr-inode-read.c:1728:afr_fgetxattr_wind] 12-stack-trace: stack-address: 0x7fb5fc0618d0, winding from test-replicate-0 to test-client-0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332817] T [rpc-clnt.c:1496:rpc_clnt_record] 12-test-client-0: Auth Info: pid: 5257, uid: 0, gid: 0, owner: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332837] T [rpc-clnt.c:1353:rpc_clnt_record_build_header] 12-rpc-clnt: Request fraglen 140, payload: 64, rpc hdr: 76<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.332862] T [rpc-clnt.c:1699:rpc_clnt_submit] 12-rpc-clnt: submitted request (XID: 0x2681 Program: GlusterFS 3.3, ProgVers: 330, Proc: 35) to rpc-transport (test-client-0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.333943] T [rpc-clnt.c:675:rpc_clnt_reply_init] 12-test-client-0: received rpc message (RPC XID: 0x2681 Program: GlusterFS 3.3, ProgVers: 330, Proc: 35) from rpc-transport
(test-client-0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334013] D [MSGID: 0] [<span style="background:yellow;mso-highlight:yellow">client-rpc-fops.c:1143:client3_3_fgetxattr_cbk] 12-test-client-0: remote operation failed:
No data available</span><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334036] D [MSGID: 0] [client-rpc-fops.c:1151:client3_3_fgetxattr_cbk] 12-stack-trace: stack-address: 0x7fb5fc0618d0, test-client-0 returned -1 error:
<span style="background:yellow;mso-highlight:yellow">No data available [No data available]</span><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334114] T [MSGID: 0] [afr-common.c:1306:afr_inode_refresh_subvol_with_fstat] 12-stack-trace: stack-address: 0x7fb5fc0618d0, winding from test-replicate-0 to test-client-0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334174] T [rpc-clnt.c:1496:rpc_clnt_record] 12-test-client-0: Auth Info: pid: 5257, uid: 0, gid: 0, owner: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334196] T [rpc-clnt.c:1353:rpc_clnt_record_build_header] 12-rpc-clnt: Request fraglen 336, payload: 260, rpc hdr: 76<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334229] T [rpc-clnt.c:1699:rpc_clnt_submit] 12-rpc-clnt: submitted request (XID: 0x2682 Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (test-client-0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334280] T [MSGID: 0] [afr-common.c:1306:afr_inode_refresh_subvol_with_fstat] 12-stack-trace: stack-address: 0x7fb5fc0618d0, winding from test-replicate-0 to test-client-1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334311] T [rpc-clnt.c:1496:rpc_clnt_record] 12-test-client-1: Auth Info: pid: 5257, uid: 0, gid: 0, owner: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334330] T [rpc-clnt.c:1353:rpc_clnt_record_build_header] 12-rpc-clnt: Request fraglen 336, payload: 260, rpc hdr: 76<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334354] T [rpc-clnt.c:1699:rpc_clnt_submit] 12-rpc-clnt: submitted request (XID: 0x252c Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) to rpc-transport (test-client-1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334904] T [rpc-clnt.c:675:rpc_clnt_reply_init] 12-test-client-1: received rpc message (RPC XID: 0x252c Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport
(test-client-1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.334983] T [MSGID: 0] [client-rpc-fops.c:1462:client3_3_fstat_cbk] 12-stack-trace: stack-address: 0x7fb5fc0618d0, test-client-1 returned 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335128] T [rpc-clnt.c:675:rpc_clnt_reply_init] 12-test-client-0: received rpc message (RPC XID: 0x2682 Program: GlusterFS 3.3, ProgVers: 330, Proc: 25) from rpc-transport
(test-client-0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335165] T [MSGID: 0] [client-rpc-fops.c:1462:client3_3_fstat_cbk] 12-stack-trace: stack-address: 0x7fb5fc0618d0, test-client-0 returned 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335193] T [MSGID: 0] [afr-inode-read.c:1728:afr_fgetxattr_wind] 12-stack-trace: stack-address: 0x7fb5fc0618d0, winding from test-replicate-0 to test-client-1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335224] T [rpc-clnt.c:1496:rpc_clnt_record] 12-test-client-1: Auth Info: pid: 5257, uid: 0, gid: 0, owner: 0000000000000000<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335253] T [rpc-clnt.c:1353:rpc_clnt_record_build_header] 12-rpc-clnt: Request fraglen 140, payload: 64, rpc hdr: 76<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335284] T [rpc-clnt.c:1699:rpc_clnt_submit] 12-rpc-clnt: submitted request (XID: 0x252d Program: GlusterFS 3.3, ProgVers: 330, Proc: 35) to rpc-transport (test-client-1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335870] T [rpc-clnt.c:675:rpc_clnt_reply_init] 12-test-client-1: received rpc message (RPC XID: 0x252d Program: GlusterFS 3.3, ProgVers: 330, Proc: 35) from rpc-transport
(test-client-1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="background:yellow;mso-highlight:yellow;mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335902] D [MSGID: 0] [client-rpc-fops.c:1143:client3_3_fgetxattr_cbk] 12-test-client-1: remote operation failed: No data available<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="background:yellow;mso-highlight:yellow;mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335922] D [MSGID: 0] [client-rpc-fops.c:1151:client3_3_fgetxattr_cbk] 12-stack-trace: stack-address: 0x7fb5fc0618d0, test-client-1
returned -1 error: No data available [No data available]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="background:yellow;mso-highlight:yellow;mso-fareast-language:ZH-CN">[2017-12-27 06:21:04.335950] D [MSGID: 0] [afr-inode-read.c:1720:afr_fgetxattr_wind] 12-stack-trace: stack-address: 0x7fb5fc0618d0, test-replicate-0
returned -1 error: No data available [No data available]</span><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">Best Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="mso-fareast-language:ZH-CN">George<o:p></o:p></span></p>
</div>
</body>
</html>