<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hello
Raghavendra,</div><div><br></div><div>I can not give you the output of the gluster commands because I repaired the system already. But beside of this this errors occurs randomly. I am sure that only one copy of the file was corrupted because it is part of a test and I corrupt one copy of the file manually on brick level and after this I check if it is still readable. During this conversation the error occurs again.</div><div>Here is the Log from brick of node1:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>[2019-02-06 14:15:09.524638] E [MSGID: 115070] [server-rpc-fops_v2.c:1503:server4_open_cbk] 0-<br>archive1-server: 32: OPEN /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86), client: CTX_<br>ID:e3871169-af62-44aa-a990-fa4248283c08-GRAPH_ID:0-PID:31830-HOST:fs-lrunning-c1-n1-PC_NAME:ar<br>chive1-client-0-RECON_NO:-0, error-xlator: archive1-bitrot-stub [Input/output error]<br>[2019-02-06 14:15:09.535587] E [MSGID: 115070] [server-rpc-fops_v2.c:1503:server4_open_cbk] 0-<br>archive1-server: 56: OPEN /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86), client: CTX_<br>ID:e3871169-af62-44aa-a990-fa4248283c08-GRAPH_ID:0-PID:31830-HOST:fs-lrunning-c1-n1-PC_NAME:ar<br>chive1-client-0-RECON_NO:-0, error-xlator: archive1-bitrot-stub [Input/output error]<br>The message "E [MSGID: 116020] [bit-rot-stub.c:647:br_stub_check_bad_object] 0-archive1-bitrot<br>-stub: 23b623cb-7256-4fe6-85b0-1026b1531a86 is a bad object. Returning" repeated 2 times betwe<br>en [2019-02-06 14:15:09.524599] and [2019-02-06 14:15:09.549409]<br>[2019-02-06 14:15:09.549427] E [MSGID: 115070] [server-rpc-fops_v2.c:1503:server4_open_cbk] 0-<br>archive1-server: 70: OPEN /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86), client: CTX_<br>ID:e3871169-af62-44aa-a990-fa4248283c08-GRAPH_ID:0-PID:31830-HOST:fs-lrunning-c1-n1-PC_NAME:ar<br>chive1-client-0-RECON_NO:-0, error-xlator: archive1-bitrot-stub [Input/output error]<br>[2019-02-06 14:15:09.561450] I [MSGID: 115036] [server.c:469:server_rpc_notify] 0-archive1-ser<br>ver: disconnecting connection from CTX_ID:e3871169-af62-44aa-a990-fa4248283c08-GRAPH_ID:0-PID:<br>31830-HOST:fs-lrunning-c1-n1-PC_NAME:archive1-client-0-RECON_NO:-0<br>[2019-02-06 14:15:09.561568] I [MSGID: 101055] [client_t.c:435:gf_client_unref] 0-archive1-ser<br>ver: Shutting down connection CTX_ID:e3871169-af62-44aa-a990-fa4248283c08-GRAPH_ID:0-PID:31830-HOST:fs-lrunning-c1-n1-PC_NAME:archive1-client-0-RECON_NO:-0<br>[2019-02-06 14:15:10.188406] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:10.201029] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>[2019-02-06 14:15:10.514721] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:10.526216] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>The message "E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler" repeated 79 times between [2019-02-06 14:15:09.499105] and [2019-02-06 14:15:10.682592]<br>[2019-02-06 14:15:10.684204] E [MSGID: 116020] [bit-rot-stub.c:647:br_stub_check_bad_object] 0-archive1-bitrot-stub: 23b623cb-7256-4fe6-85b0-1026b1531a86 is a bad object. Returning<br>[2019-02-06 14:15:10.684262] E [MSGID: 115070] [server-rpc-fops_v2.c:1503:server4_open_cbk] 0-archive1-server: 2146148: OPEN /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86), client: CTX_ID:0545b52c-2843-4833-a5fc-b11e062a72d3-GRAPH_ID:0-PID:2458-HOST:fs-lrunning-c1-n1-PC_NAME:archive1-client-0-RECON_NO:-3, error-xlator: archive1-bitrot-stub [Input/output error]<br>[2019-02-06 14:15:10.684949] E [MSGID: 116020] [bit-rot-stub.c:647:br_stub_check_bad_object] 0-archive1-bitrot-stub: 23b623cb-7256-4fe6-85b0-1026b1531a86 is a bad object. Returning<br>[2019-02-06 14:15:10.684982] E [MSGID: 115070] [server-rpc-fops_v2.c:1503:server4_open_cbk] 0-archive1-server: 2146149: OPEN /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86), client: CTX_ID:0545b52c-2843-4833-a5fc-b11e062a72d3-GRAPH_ID:0-PID:2458-HOST:fs-lrunning-c1-n1-PC_NAME:archive1-client-0-RECON_NO:-3, error-xlator: archive1-bitrot-stub [Input/output error]<br>[2019-02-06 14:15:10.686566] E [MSGID: 116020] [bit-rot-stub.c:647:br_stub_check_bad_object] 0-archive1-bitrot-stub: 23b623cb-7256-4fe6-85b0-1026b1531a86 is a bad object. Returning<br>[2019-02-06 14:15:10.686600] E [MSGID: 115070] [server-rpc-fops_v2.c:1503:server4_open_cbk] 0-archive1-server: 2146150: OPEN /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86), client: CTX_ID:0545b52c-2843-4833-a5fc-b11e062a72d3-GRAPH_ID:0-PID:2458-HOST:fs-lrunning-c1-n1-PC_NAME:archive1-client-0-RECON_NO:-3, error-xlator: archive1-bitrot-stub [Input/output error]<br>[2019-02-06 14:15:11.189361] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:11.207835] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:11.220763] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br><br></div></blockquote><div><br></div><div>One can see that there is bitrot file on brick of node1. This seems to be correct.</div><div>Here the Log of the FUSE Mount Node1:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>[2019-02-06 14:15:10.684387] E [MSGID: 114031] [client-rpc-fops_v2.c:281:client4_0_open_cbk] 0-archive1-client-0: remote operation failed. Path: /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86) [Input/output error]<br>[2019-02-06 14:15:10.684556] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7feff4bde329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7feff4defaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7feffcecf218] ) 0-dict: dict is NULL [Invalid argument]<br>[2019-02-06 14:15:10.685122] E [MSGID: 114031] [client-rpc-fops_v2.c:281:client4_0_open_cbk] 0-archive1-client-0: remote operation failed. Path: /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86) [Input/output error]<br>[2019-02-06 14:15:10.685127] E [MSGID: 108009] [afr-open.c:220:afr_openfd_fix_open_cbk] 0-archive1-replicate-0: Failed to open /data/file1.txt on subvolume archive1-client-0 [Input/output error]<br>[2019-02-06 14:15:10.686207] W [fuse-bridge.c:2371:fuse_readv_cbk] 0-glusterfs-fuse: 4623583: READ => -1 gfid=23b623cb-7256-4fe6-85b0-1026b1531a86 fd=0x7fefa8c5d618 (Transport endpoint is not connected)<br>[2019-02-06 14:15:10.686306] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7feff4bde329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7feff4defaf5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7feffcecf218] ) 0-dict: dict is NULL [Invalid argument]<br>[2019-02-06 14:15:10.686690] E [MSGID: 114031] [client-rpc-fops_v2.c:281:client4_0_open_cbk] 0-archive1-client-0: remote operation failed. Path: /data/file1.txt (23b623cb-7256-4fe6-85b0-1026b1531a86) [Input/output error]<br>[2019-02-06 14:15:10.686714] E [MSGID: 108009] [afr-open.c:220:afr_openfd_fix_open_cbk] 0-archive1-replicate-0: Failed to open /data/file1.txt on subvolume archive1-client-0 [Input/output error]<br>[2019-02-06 14:15:10.686877] W [fuse-bridge.c:2371:fuse_readv_cbk] 0-glusterfs-fuse: 4623584: READ => -1 gfid=23b623cb-7256-4fe6-85b0-1026b1531a86 fd=0x7fefa8c5d618 (Transport endpoint is not connected)<br>[2019-02-06 14:15:10.687500] W [MSGID: 114028] [client-lk.c:347:delete_granted_locks_owner] 0-archive1-client-0: fdctx not valid [Invalid argument]<br><br></div></blockquote><div><br></div><div>One can see an "Input/output error" because of the corrupted file from brick of node1. At this time the <b>brick on node 2 was really down</b> but on Node 3, 4 they were up. So still 2 good copies are reachable. Or not?</div><div>The Log of the bricks from node 3,4 has no entry for this "file1.txt". It seems to be that the Client Stack did no requests to this bricks.</div><div>Example Log of brick from node 3:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div>[2019-02-06 14:15:09.561650] E [MSGID: 101191] [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to dispatch handler<br>[2019-02-06 14:15:10.220218] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:10.236379] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>[2019-02-06 14:15:10.541472] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:10.556125] I [glusterfsd-mgmt.c:2005:mgmt_getspec_cbk] 0-glusterfs: No change in volfile,continuing<br>[2019-02-06 14:15:11.248253] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:11.264428] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br>[2019-02-06 14:15:11.277016] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt: Volume file changed<br><br></div></blockquote><div> </div><div>Is there a hidden quorum active? I have a 4-way Replica Volume, so 2 of 4 Copies are good and reachable<br></div><div><br></div><div>Regards</div><div>David<br></div><div><br></div></div></div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Di., 5. Feb. 2019 um 21:06 Uhr schrieb FNU Raghavendra Manjunath <<a href="mailto:rabhat@redhat.com">rabhat@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div>Hi David,</div><div><br></div><div>Do you have any bricks down? Can you please share the output of the following commands and also the logs of the server and the client nodes?</div><div><br></div><div>1) gluster volume info</div><div>2) gluster volume status</div><div>3) gluster volume bitrot <volume name> scrub status</div><div><br></div><div>Few more questions</div><div><br></div><div>1) How many copies of the file were corrupted? (All? Or Just one?)</div><div><br></div><div>2 things I am trying to understand</div><div><br></div><div>A) IIUC, if only one copy is corrupted, then the replication module from the gluster client should serve the data from the </div><div> remaining good copy</div><div>B) If all the copies were corrupted (or say more than quorum copies were corrupted which means 2 in case of 3 way replication)</div><div> then there will be an error to the application. But the error to be reported should 'Input/Output Error'. Not 'Transport endpoint not connected'</div><div> 'Transport endpoint not connected' error usually comes when a brick where the operation is being directed to is not connected to the client.</div><div><br></div><div><br></div><div><br></div><div>Regards,</div><div>Raghavendra</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 4, 2019 at 6:02 AM David Spisla <<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hello Amar,</div><div>sounds good. Until now this patch is only merged into master. I think it should be part of the next v5.x patch release!</div><div><br></div><div>Regards</div><div>David<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Am Mo., 4. Feb. 2019 um 09:58 Uhr schrieb Amar Tumballi Suryanarayan <<a href="mailto:atumball@redhat.com" target="_blank">atumball@redhat.com</a>>:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hi David, <div><br></div><div>I guess <a href="https://review.gluster.org/#/c/glusterfs/+/21996/" target="_blank">https://review.gluster.org/#/c/glusterfs/+/21996/</a> helps to fix the issue. I will leave it to Raghavendra Bhat to reconfirm.</div><div><br></div><div>Regards,</div><div>Amar</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Feb 1, 2019 at 8:45 PM David Spisla <<a href="mailto:spisla80@gmail.com" target="_blank">spisla80@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hello Gluster Community,</div><div>I have got a 4 Node Cluster with a Replica 4 Volume, so each node has a brick with a copy of a file. Now I tried out the bitrot functionality and corrupt the copy on the brick of node1. After this I scrub ondemand and the file is marked correctly as corrupted. <br></div><div><br></div><div>No I try to read that file from FUSE on node1 (with corrupt copy):</div><div>$ cat file1.txt <br>cat: file1.txt: Transport endpoint is not connected</div>FUSE log says:</div><div dir="ltr"><br></div><div dir="ltr"><b>[2019-02-01 15:02:19.191984] E [MSGID: 114031] [client-rpc-fops_v2.c:281:client4_0_open_cbk] 0-archive1-client-0: remote operation failed. Path: /data/file1.txt (b432c1d6-ece2-42f2-8749-b11e058c4be3) [Input/output error]</b><br>[2019-02-01 15:02:19.192269] W [dict.c:761:dict_ref] (-->/usr/lib64/glusterfs/5.3/xlator/performance/quick-read.so(+0x7329) [0x7fc642471329] -->/usr/lib64/glusterfs/5.3/xlator/performance/io-cache.so(+0xaaf5) [0x7fc642682af5] -->/usr/lib64/libglusterfs.so.0(dict_ref+0x58) [0x7fc64a78d218] ) 0-dict: dict is NULL [Invalid argument]<br>[2019-02-01 15:02:19.192714] E [MSGID: 108009] [afr-open.c:220:afr_openfd_fix_open_cbk] 0-archive1-replicate-0: Failed to open /data/file1.txt on subvolume archive1-client-0 [Input/output error]<br><b>[2019-02-01 15:02:19.193009] W [fuse-bridge.c:2371:fuse_readv_cbk] 0-glusterfs-fuse: 147733: READ => -1 gfid=b432c1d6-ece2-42f2-8749-b11e058c4be3 fd=0x7fc60408bbb8 (Transport endpoint is not connected)</b><br>[2019-02-01 15:02:19.193653] W [MSGID: 114028] [client-lk.c:347:delete_granted_locks_owner] 0-archive1-client-0: fdctx not valid [Invalid argument]<br><br></div><div dir="ltr"><div>And from FUSE on node2 (with heal copy):</div><div>$ cat file1.txt <br>file1<br></div><div><br></div><div>It seems to be that node1 wants to get the file from its own brick, but the copy there is broken. Node2 gets the file from its own brick with a heal copy, so reading the file succeed.</div><div>But I am wondering myself because sometimes reading the file from node1 with the broken copy succeed</div><div><br></div><div>What is the expected behaviour here? Is it possibly to read files with a corrupted copy from any client access?</div><div><br></div><div>Regards</div><div>David Spisla<br></div><div><br></div><div><br></div></div></div></div></div></div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail-m_-7358655793243076686gmail-m_1325824890776359779gmail-m_-3012308099005136219gmail-m_1634570276701474471gmail-m_807676680040787958gmail_signature"><div dir="ltr"><div><div dir="ltr"><div>Amar Tumballi (amarts)<br></div></div></div></div></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>