[Gluster-users] 3.8.5 replica 3 volumes: I/O error on file on fuse mounts
Alastair Neil
ajneil.tech at gmail.com
Wed Dec 21 17:38:31 UTC 2016
Would apprecaite any insight into this issue:
replica 3 volume, it is showing a number of files on two of the bricks as
needing healed, when you examine the files on the fuse mounts they generate
I/O errors.
No files listed in split brain, but if I look at one of the files it looks
to me like they have been updated on gluster-2 and gluster0 but not on
gluster1 (see below).
I see errors in /va/log/gluster/glustershd.log
-Thanks Alastair
[2016-12-20 07:25:06.018829] I [MSGID: 101190]
> [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index 1
> [2016-12-20 07:25:06.018901] E [socket.c:2309:socket_connect_finish]
> 0-glusterfs: connection to ::1:24007 failed (Connection refused)
> [2016-12-20 07:25:06.018944] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify]
> 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport
> endpoint is not connected)
> [2016-12-20 07:25:07.187710] W [glusterfsd.c:1327:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fd93f669dc5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fd940cfbcd5]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x7fd940cfbb4b] ) 0-:
> received signum (15), shutting down
> [2016-12-20 07:25:08.197959] I [MSGID: 100030] [glusterfsd.c:2454:main]
> 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.5
> (args: /usr/sbin/glusterfs -s localhost --volfile-id gluster/glustershd -p
> /var/lib/glusterd/glustershd/run/glustershd.pid -l
> /var/log/glusterfs/glustershd.log -S
> /var/run/gluster/3fe0b238bd46c38a95636f25cb5b9d8a.socket --xlator-option
> *replicate*.node-uuid=bcff5245-ea86-4384-a1bf-9219c8be8001)
> [2016-12-20 07:25:08.216336] I [MSGID: 101190]
> [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index 1
> [2016-12-20 07:25:08.216419] E [socket.c:2309:socket_connect_finish]
> 0-glusterfs: connection to ::1:24007 failed (Connection refused)
> [2016-12-20 07:25:08.216464] E [glusterfsd-mgmt.c:1902:mgmt_rpc_notify]
> 0-glusterfsd-mgmt: failed to connect with remote-host: localhost (Transport
> endpoint is not connected)
> [2016-12-20 07:25:12.208092] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-digitalcorpora-replicate-0: adding
> option 'node-uuid' for volume 'digitalcorpora-replicate-0' with value
> 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 07:25:12.208122] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-gluster_shared_storage-replicate-0:
> adding option 'node-uuid' for volume 'gluster_shared_storage-replicate-0'
> with value 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 07:25:12.208140] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-homes-replicate-0: adding option
> 'node-uuid' for volume 'homes-replicate-0' with value
> 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 07:25:12.208155] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-public-replicate-0: adding option
> 'node-uuid' for volume 'public-replicate-0' with value
> 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 07:25:12.208173] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-static-web-replicate-0: adding
> option 'node-uuid' for volume 'static-web-replicate-0' with value
> 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 07:25:12.208199] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-tmp-replicate-0: adding option
> 'node-uuid' for volume 'tmp-replicate-0' with value
> 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 07:25:12.208215] I [MSGID: 101173]
> [graph.c:269:gf_add_cmdline_options] 0-usr-local-replicate-0: adding option
> 'node-uuid' for volume 'usr-local-replicate-0' with value
> 'bcff5245-ea86-4384-a1bf-9219c8be8001'
> [2016-12-20 18:32:06.121734] E [client-common.c:526:client_pre_getxattr]
> (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8)
> [0x7f6bc4ba65d8]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd)
> [0x7f6bc4bc1ebd]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3)
> [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0
> [2016-12-20 18:32:06.121809] E [client-common.c:587:client_pre_opendir]
> (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5)
> [0x7f6bc4ba59d5]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65)
> [0x7f6bc4bc0a65]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7)
> [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0
> [2016-12-20 18:46:51.764776] E [client-common.c:526:client_pre_getxattr]
> (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8)
> [0x7f6bc4ba65d8]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd)
> [0x7f6bc4bc1ebd]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3)
> [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0
> [2016-12-20 18:46:51.764850] E [client-common.c:587:client_pre_opendir]
> (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5)
> [0x7f6bc4ba59d5]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65)
> [0x7f6bc4bc0a65]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7)
> [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0
> [2016-12-20 18:49:29.657568] E [client-common.c:526:client_pre_getxattr]
> (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xb5d8)
> [0x7f6bc4ba65d8]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x26ebd)
> [0x7f6bc4bc1ebd]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x393e3)
> [0x7f6bc4bd43e3] ) 0-: Assertion failed: 0
> [2016-12-20 18:49:29.657645] E [client-common.c:587:client_pre_opendir]
> (-->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0xa9d5)
> [0x7f6bc4ba59d5]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x25a65)
> [0x7f6bc4bc0a65]
> -->/usr/lib64/glusterfs/3.8.5/xlator/protocol/client.so(+0x396b7)
> [0x7f6bc4bd46b7] ) 0-: Assertion failed: 0
>
gluster2:
# getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.homes-client-5=0x000000020000000100000000
trusted.bit-rot.version=0x020000000000000058589e6b0005bdac
trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5
gluster1:
# getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.bit-rot.version=0x0200000000000000583f45c20008d152
trusted.gfid=0x6c278b5c94ae436bb669b5f5dd21777e
gluster0:
# getfattr -d -m. -e hex /export/brick2/home/a/j/ajn/.Xauthority
getfattr: Removing leading '/' from absolute path names
# file: export/brick2/home/a/j/ajn/.Xauthority
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.homes-client-5=0x000000020000000100000000
trusted.bit-rot.version=0x0200000000000000583f3fbb000b5b01
trusted.gfid=0xb8b156b764304fd1bf7e692649bcecc5
[root at gluster0 Project3]# glv heal homes info
> Brick gluster-2:/export/brick2/home
> /s/a/sadams25/pp2.txt
> /s/a/sadams25/.viminfo
> /a/v/avakil/.Xauthority
> /j/m/jmurra17/fork
> /c/f/cferris2/.viminfo
> /c/s/cs367/bomblab/S001/log-status.txt
> /c/s/cs367/bomblab/S001/bomblab-scoreboard.html
> /c/s/cs367/bomblab/S001/scores.txt
> /c/s/cs367/bomblab/S003/bomblab-scoreboard.html
> /c/s/cs367/bomblab/S003/scores.txt
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o
>
> /j/m/jmurra17/fork/fork.c
> /j/m/jmurra17/.viminfo
> /a/j/ajn/.Xauthority
> /a/v/avakil/source_code/rm_setup/common_setup.tcl
> /a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl
> /a/v/avakil/source_code/rm_setup/dc_setup.tcl
> /j/d/jdenton3/.viminfo
> /s/a/sadams25/x.txt
> /j/d/jdenton3/Project3/Project3.c
> /j/m/jmurra17/fork/fork
> /j/d/jdenton3/Project3/p5
> Status: Connected
> Number of entries: 27
>
> Brick gluster1.vsnet.gmu.edu:/export/brick2/home
> Status: Connected
> Number of entries: 0
>
> Brick gluster0:/export/brick2/home
> /s/a/sadams25/pp2.txt
> /s/a/sadams25/.viminfo
> /c/s/cs367/bomblab/S003/scores.txt
> /a/v/avakil/.Xauthority
> /c/s/cs367/bomblab/S001/scores.txt
> /c/f/cferris2/.viminfo
> /c/s/cs367/bomblab/S001/log-status.txt
> /c/s/cs367/bomblab/S003/tmpwebpage.14635
> /c/s/cs367/bomblab/S001/bomblab-scoreboard.html
> /c/s/cs367/bomblab/S003/bomblab-scoreboard.html
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/libsupport.a
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/Makefile
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.c
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.h
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/caching.c
>
> /w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/memory_system.o
>
> /j/m/jmurra17/fork
> <gfid:310211c2-aeec-4906-894f-023d0ad7d5cc>/#
> affiliate.nagios.com/settings.sol
> /a/v/avakil/source_code/rm_setup/common_setup.tcl
> /a/j/ajn/.Xauthority
> /j/m/jmurra17/.viminfo
> /a/v/avakil/source_code/rm_setup/dc_setup.tcl
> /j/m/jmurra17/fork/fork.c
> /a/v/avakil/source_code/rm_setup/dc_setup_filenames.tcl
> /j/d/jdenton3/Project3/Project3.c
> /j/d/jdenton3/.viminfo
> /s/a/sadams25/x.txt
> /j/m/jmurra17/fork/fork
> /j/d/jdenton3/Project3/p5
> Status: Connected
> Number of entries: 29
>
> [
> [root at gluster0 .bad]# cd
> /mnt/home/w/h/white/Semesters/Fall16/Lab4/Lab4/.bad/mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh/
> [root at gluster0 mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh]# ls
> -al
> ls: cannot access libsupport.a: Input/output error
> ls: cannot access Makefile: Input/output error
> ls: cannot access memory_system.c: Input/output error
> ls: cannot access memory_system.h: Input/output error
> ls: cannot access caching.c: Input/output error
> ls: cannot access memory_system.o: Input/output error
> total 626
> drwxrwxr-x 2 1735 users 4096 Dec 20 11:38 .
> drwxr-xr-x 3 root root 4096 Dec 20 13:53 ..
> -????????? ? ? ? ? ? caching.c
> -rw-rw-r-- 1 1735 users 9056 Dec 20 11:36 caching.o
> -rwxrwxr-x 1 1735 users 147855 Dec 20 11:36 lab4
> -rw-r--r-- 1 1735 users 307200 Dec 13 07:04 Lab 4 - 12
> 9_mchehreh_attempt_2016-12-10-00-26-11_lab4_mchehreh.tar
> -rw-rw-r-- 1 1735 users 8254 Dec 20 11:38 lab4_logfile
> -rw-r--r-- 1 1735 users 153600 Dec 20 11:32 lab4_mchehreh.tar
> -????????? ? ? ? ? ? libsupport.a
> -????????? ? ? ? ? ? Makefile
> -????????? ? ? ? ? ? memory_system.c
> -????????? ? ? ? ? ? memory_system.h
> -????????? ? ? ? ? ? memory_system.o
> -rw-rw-r-- 1 1735 users 449 Dec 20 11:38 t1
> -rw-rw-r-- 1 1735 users 453 Dec 20 11:38 t2
> -rw-rw-r-- 1 1735 users 2185 Dec 20 11:38 t3
> -rw-rw-r-- 1 1735 users 2195 Dec 20 11:38 t4
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161221/1451bc66/attachment.html>
More information about the Gluster-users
mailing list