[Gluster-users] total outage - almost

[Bernhard Dübi]

Hi,

we use a bunch of replicated gluster volumes as a backend for our
backup. Yesterday I noticed that some synthetic backups failed because
of I/O errors.

Today I ran "find /gluster_vol -type f | xargs md5sum" and got loads
of I/O errors.
The brick log file shows the below errors

[2017-06-19 13:42:33.554875] E [MSGID: 116020]
[bit-rot-stub.c:566:br_stub_check_bad_object]
0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba
is a bad object. Returning
[2017-06-19 13:42:33.554923] E [MSGID: 116020]
[bit-rot-stub.c:566:br_stub_check_bad_object]
0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba
is a bad object. Returning
[2017-06-19 13:42:33.554931] E [MSGID: 115081]
[server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server:
21461: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==>
(Input/output error) [Input/output error]
[2017-06-19 13:42:33.554940] E [MSGID: 115081]
[server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server:
21462: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==>
(Input/output error) [Input/output error]
[2017-06-19 13:42:33.555655] E [MSGID: 116020]
[bit-rot-stub.c:566:br_stub_check_bad_object]
0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba
is a bad object. Returning
[2017-06-19 13:42:33.555697] E [MSGID: 115081]
[server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server:
21463: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==>
(Input/output error) [Input/output error]
[2017-06-19 13:42:33.555950] E [MSGID: 116020]
[bit-rot-stub.c:566:br_stub_check_bad_object]
0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba
is a bad object. Returning
[2017-06-19 13:42:33.555983] E [MSGID: 115081]
[server-rpc-fops.c:1201:server_fstat_cbk] 0-Server_Standard_05-server:
21464: FSTAT -2 (c75016a9-95c1-4819-b24a-e5d77107c4ba) ==>
(Input/output error) [Input/output error]
[2017-06-19 13:42:33.556604] E [MSGID: 116020]
[bit-rot-stub.c:566:br_stub_check_bad_object]
0-Server_Standard_05-bitrot-stub: c75016a9-95c1-4819-b24a-e5d77107c4ba
is a bad object. Returning




Any idea what's wrong?


BTW: I'm running gluster 3.8.12 on Ubuntu 16.04 - 4.4.0-79

many thanks for your help
Bernhard