[Gluster-users] 答复: read error: Input/output error but not in brain split stat
likun
kun.li at ucarinc.com
Wed Jan 4 05:55:46 UTC 2017
The “java.io.FileNotFound” error happened at 0:00 Jan 1st 2017, That’s
Beijing time, while glusterfs use UTC time，the time should be 2017-01-01
16:00:00.
Here is the log of three server at that time:
ab09 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log
…
[2017-01-01 16:00:00.778962] E [MSGID: 113039] [posix.c:3011:posix_open]
0-gvol0-posix: open on
/mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbea,
flags: 1025 [No such file or directory]
[2017-01-01 16:00:04.864163] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link
/mnt/brick3/vol/gps/data/2017-01/02/84.txt ->
/mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed
[File exists]
The message "E [MSGID: 113039] [posix.c:3011:posix_open] 0-gvol0-posix: open
on /mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbea,
flags: 1025 [No such file or directory]" repeated 5 times between
[2017-01-01 16:00:00.778962] and [2017-01-01 16:00:00.779308]
The message "W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard]
0-gvol0-posix: link /mnt/brick3/vol/gps/data/2017-01/02/84.txt ->
/mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed
[File exists]" repeated 4 times between [2017-01-01 16:00:04.864163] and
[2017-01-01 16:00:04.864286]
…
ac08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log
…
[2017-01-01 16:00:00.371922] W [MSGID: 113096]
[posix-handle.c:828:posix_handle_soft] 0-gvol0-posix: symlink
../../e1/b5/e1b5fbec-f861-4c6a-99e8-8d3c28cce76a/02 ->
/mnt/brick3/vol/.glusterfs/05/9b/059b889e-5e7f-429c-a208-922226b329a2 failed
[File exists]
[2017-01-01 16:00:00.778274] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link
/mnt/brick3/vol/gps/up_data/2017-01/02/4.txt ->
/mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbeafailed
[File exists]
[2017-01-01 16:00:01.264791] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link
/mnt/brick3/vol/gps/down_data/2017-01/02/8.txt ->
/mnt/brick3/vol/.glusterfs/f4/58/f458e364-1772-4047-8edd-4aa48a096126failed
[File exists]
[2017-01-01 16:00:04.834138] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link
/mnt/brick3/vol/gps/data/2017-01/02/84.txt ->
/mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed
[File exists]
The message "W [MSGID: 113096] [posix-handle.c:828:posix_handle_soft]
0-gvol0-posix: symlink ../../e1/b5/e1b5fbec-f861-4c6a-99e8-8d3c28cce76a/02
-> /mnt/brick3/vol/.glusterfs/05/9b/059b889e-5e7f-429c-a208-922226b329a2
failed [File exists]" repeated 5 times between[2017-01-01 16:00:00.371922]
and [2017-01-01 16:00:00.372030]
The message "W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard]
0-gvol0-posix: link /mnt/brick3/vol/gps/up_data/2017-01/02/4.txt ->
/mnt/brick3/vol/.glusterfs/a6/1d/a61d7b2f-d53f-4eb9-be7a-656b0915dbeafailed
[File exists]" repeated 4 times between [2017-01-01 16:00:00.778274] and
[2017-01-01 16:00:00.778341]
The message "W [MSGID: 113096] [posix-handle.c:761:posix_handle_hard]
0-gvol0-posix: link /mnt/brick3/vol/gps/data/2017-01/02/84.txt ->
/mnt/brick3/vol/.glusterfs/02/a6/02a67715-47ba-4552-ae84-35cd660f38e5failed
[File exists]" repeated 4 times between [2017-01-01 16:00:04.834138] and
[2017-01-01 16:00:04.834314]
…
ad08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log
…
[2017-01-01 16:00:00.399017] W [MSGID: 113096]
[posix-handle.c:828:posix_handle_soft] 0-gvol0-posix: symlink
../../92/c6/92c699d6-2670-4547-9845-ec0450107a2e/02 ->
/mnt/brick3/vol/.glusterfs/36/3d/363d3c2d-5ccb-497e-9ecc-6eaf5f435026 failed
[File exists]
[2017-01-01 16:00:02.057190] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link
/mnt/brick3/vol/gps/up_data/2017-01/02/46.txt ->
/mnt/brick3/vol/.glusterfs/93/ea/93ea8b72-e208-468d-812c-88d7f80bc204failed
[File exists]
[2017-01-01 16:00:02.057221] E [MSGID: 113020] [posix.c:2895:posix_create]
0-gvol0-posix: setting gfid on /mnt/brick3/vol/gps/up_data/2017-01/02/46.txt
failedThe message "W [MSGID: 113096] [posix-handle.c:828:posix_handle_soft]
0-gvol0-posix: symlink ../../92/c6/92c699d6-2670-4547-9845-ec0450107a2e/02
-> /mnt/brick3/vol/.glusterfs/36/3d/363d3c2d-5ccb-497e-9ecc-6eaf5f435026
failed [File exists]" repeated 5 times between [2017-01-01 16:00:00.399017]
and [2017-01-01 16:00:00.399097]
…
And the “cat: read error: Input/output error” happened at about 2017-0103
10:30, 2:30 at UTC time. Here is the log:
ab09 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log
…
[2017-01-03 02:30:57.684122] I [login.c:76:gf_auth] 0-auth/login: allowed
user names: 94bedfd1-619d-402a-9826-67dab7600f43
[2017-01-03 02:30:57.684147] I [MSGID: 115029]
[server-handshake.c:692:server_setvolume] 0-gvol0-server: accepted client
from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-51-0-0
(version: 3.8.5)
[2017-01-03 02:30:58.368750] I [MSGID: 115036]
[server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection
from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-51-0-0
[2017-01-03 02:30:58.368797] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection
aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-51-0-0
[2017-01-03 02:46:29.361991] W [MSGID: 113096]
[posix-handle.c:761:posix_handle_hard] 0-gvol0-posix: link
/mnt/brick3/vol/gps/up_data/2017-01/03/91.txt ->
/mnt/brick3/vol/.glusterfs/e6/82/e6823e82-04c5-4e81-a3a8-6534d809d3f1failed
[File exists]
[2017-01-03 02:48:55.302428] I [MSGID: 115036]
[server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection
from ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-51-0-0
[2017-01-03 02:48:55.302487] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection
ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-51-0-0
…
ac08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log
…
[2017-01-03 02:30:57.684317] I [login.c:76:gf_auth] 0-auth/login: allowed
user names: 94bedfd1-619d-402a-9826-67dab7600f43
[2017-01-03 02:30:57.684338] I [MSGID: 115029]
[server-handshake.c:692:server_setvolume] 0-gvol0-server: accepted client
from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-52-0-0
(version: 3.8.5)
[2017-01-03 02:30:58.368991] I [MSGID: 115036]
[server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection
from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-52-0-0
[2017-01-03 02:30:58.369016] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection
aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-52-0-0
[2017-01-03 02:48:55.302622] I [MSGID: 115036]
[server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection
from ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-52-0-0
[2017-01-03 02:48:55.302657] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection
ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-52-0-0
…
ad08 /var/log/glusterfs/bricks # cat mnt-brick3-vol.log
…
[2017-01-03 02:30:57.693935] I [login.c:76:gf_auth] 0-auth/login: allowed
user names: 94bedfd1-619d-402a-9826-67dab7600f43
[2017-01-03 02:30:57.693956] I [MSGID: 115029]
[server-handshake.c:692:server_setvolume] 0-gvol0-server: accepted client
from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-53-0-0
(version: 3.8.5)
[2017-01-03 02:30:58.369055] I [MSGID: 115036]
[server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection
from aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-53-0-0
[2017-01-03 02:30:58.369074] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection
aa08.pek.prod.com-16474-2017/01/03-02:30:57:280485-gvol0-client-53-0-0
[2017-01-03 02:48:55.302720] I [MSGID: 115036]
[server.c:548:server_rpc_notify] 0-gvol0-server: disconnecting connection
from ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-53-0-0
[2017-01-03 02:48:55.302773] I [MSGID: 101055]
[client_t.c:415:gf_client_unref] 0-gvol0-server: Shutting down connection
ad02.pek.prod.com-120059-2016/12/30-06:43:02:91882-gvol0-client-53-0-0
…
Likun
发件人: Ravishankar N [mailto:ravishankar at redhat.com]
发送时间: 2017年1月4日 10:30
收件人: likun; gluster-users at gluster.org
主题: Re: [Gluster-users] read error: Input/output error but not in brain
split stat
On 01/03/2017 09:03 AM, likun wrote:
Then I tried to access this file:
# cd /mnt/glusterfs/gps/data/2017-01/02
# more 4.txt
# cat 4.txt
cat: read error: Input/output error
I supposed there was something wrong with glusterfs, so I logined into
glusterfs system, found nothing wrong, the output of ‘gluster volume
status|info’ and ‘gluster volume heal gvol0 info’ command appended at the
end. So, question is what’s wrong?
What did the mount log say at this time? Also were there errors/warnings on
the 3 brick logs where the file resides at the same time?
Regards,
Ravi
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20170104/ad1b3b42/attachment.html>
More information about the Gluster-users
mailing list