[Gluster-users] Failure after update

Tue Feb 16 09:31:01 UTC 2016

On Tue, Feb 16, 2016 at 08:35:05AM +0000, ousmane sanogo wrote:
> Hello i .update my gluster node yesterday
> I am using cinder openstack with gluster
>  And i have this warning after update :
> 
> warning:
> /var/lib/glusterd/vols/vol_cinder/vol_cinder.172.16.10.2.glusterfs-cinder.vol
> saved as
> /var/lib/glusterd/vols/vol_cinder/vol_cinder.172.16.10.2.glusterfs-cinder.vol.rpmsave\nwarning:
> /var/lib/glusterd/vols/vol_cinder/vol_cinder.tcp-fuse.vol saved as
> /var/lib/glusterd/vols/vol_cinder/vol_cinder.tcp-fuse.vol.rpmsave\nwarning:
> /var/lib/glusterd/vols/vol_cinder/vol_cinder.172.16.10.3.glusterfs-cinder.vol
> saved as
> /var/lib/glusterd/vols/vol_cinder/vol_cinder.172.16.10.3.glusterfs-cinder.vol.rpmsave\nwarning:
> /var/lib/glusterd/vols/vol_cinder/trusted-vol_cinder.tcp-fuse.vol saved as
> /var/lib/glusterd/vols/vol_cinder/trusted-vol_cinder.tcp-fuse.vol.rpmsave\nwarning:
> /var/lib/glusterd/vols/vol_instances/vol_instances.172.16.10.2.glusterfs-instances.vol
> saved as
> /var/lib/glusterd/vols/vol_instances/vol_instances.172.16.10.2.glusterfs-instances.vol.rpmsave
> Warning: glusterd.service changed on disk. Run 'systemctl daemon-reload' to
> reload units.
> 
> I run "systemctl daemon-reload" on node.
> 
> On node 1:
> [root at compute1 ~]# tail
> /var/log/glusterfs/var-lib-nova-mnt-7e2fea33428149438b876dd122157f27.log -f
> [2016-02-15 19:56:44.459473] I [MSGID: 114057]
> [client-handshake.c:1437:select_server_supported_programs]
> 0-vol_cinder-client-0: Using Program GlusterFS 3.3, Num (1298437), Version
> (330)
> [2016-02-15 19:56:44.459644] I [MSGID: 114046]
> [client-handshake.c:1213:client_setvolume_cbk] 0-vol_cinder-client-0:
> Connected to vol_cinder-client-0, attached to remote volume
> '/glusterfs-cinder'.
> [2016-02-15 19:56:44.459658] I [MSGID: 114047]
> [client-handshake.c:1224:client_setvolume_cbk] 0-vol_cinder-client-0:
> Server and Client lk-version numbers are not same, reopening the fds
> [2016-02-15 19:56:44.459666] I [MSGID: 114042]
> [client-handshake.c:1056:client_post_handshake] 0-vol_cinder-client-0: 1
> fds open - Delaying child_up until they are re-opened
> [2016-02-15 19:56:44.459882] I [MSGID: 114041]
> [client-handshake.c:678:client_child_up_reopen_done] 0-vol_cinder-client-0:
> last fd open'd/lock-self-heal'd - notifying CHILD-UP
> [2016-02-15 19:56:44.459944] I [MSGID: 114035]
> [client-handshake.c:193:client_set_lk_version_cbk] 0-vol_cinder-client-0:
> Server lk version = 1
> [2016-02-15 19:57:32.625556] I [MSGID: 108031]
> [afr-common.c:1782:afr_local_discovery_cbk] 0-vol_cinder-replicate-0:
> selecting local read_child vol_cinder-client-0
> [2016-02-15 20:08:23.876756] I [fuse-bridge.c:4984:fuse_thread_proc]
> 0-fuse: unmounting /var/lib/nova/mnt/7e2fea33428149438b876dd122157f27
> [2016-02-15 20:08:23.918397] W [glusterfsd.c:1236:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7fee5b957dc5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7fee5cfc2855]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7fee5cfc26d9] ) 0-:
> received signum (15), shutting down
> [2016-02-15 20:08:23.918424] I [fuse-bridge.c:5683:fini] 0-fuse: Unmounting
> '/var/lib/nova/mnt/7e2fea33428149438b876dd122157f27'.
> 
> 
> [root at compute1 ~]# tail /var/log/glusterfs/var-lib-nova-instances.log -f
> [2016-02-16 08:32:32.292641] W [fuse-bridge.c:2292:fuse_writev_cbk]
> 0-glusterfs-fuse: 62606120: WRITE => -1 (Noeud final de transport n'est pas
> connecté)
> [2016-02-16 08:32:32.292700] W [fuse-bridge.c:2292:fuse_writev_cbk]
> 0-glusterfs-fuse: 62606122: WRITE => -1 (Noeud final de transport n'est pas
> connecté)
> [2016-02-16 08:32:32.292756] W [fuse-bridge.c:2292:fuse_writev_cbk]
> 0-glusterfs-fuse: 62606124: WRITE => -1 (Noeud final de transport n'est pas
> connecté)
> 
> On node 2
> 
> [root at compute2 ~]# tail
> /var/log/glusterfs/var-lib-nova-mnt-7e2fea33428149438b876dd122157f27.log -f
> [2016-02-15 19:56:47.042442] W [fuse-bridge.c:2292:fuse_writev_cbk]
> 0-glusterfs-fuse: 15109968: WRITE => -1 (Transport endpoint is not
> connected)
> [2016-02-15 19:56:47.047263] W [fuse-bridge.c:2292:fuse_writev_cbk]
> 0-glusterfs-fuse: 15109970: WRITE => -1 (Transport endpoint is not
> connected)
> [2016-02-15 19:56:47.047339] W [fuse-bridge.c:2292:fuse_writev_cbk]
> 0-glusterfs-fuse: 15109972: WRITE => -1 (Transport endpoint is not
> connected)
> [2016-02-15 19:57:03.118138] I [MSGID: 108031]
> [afr-common.c:1782:afr_local_discovery_cbk] 0-vol_cinder-replicate-0:
> selecting local read_child vol_cinder-client-1
> [2016-02-15 20:07:19.303007] W [fuse-bridge.c:1282:fuse_err_cbk]
> 0-glusterfs-fuse: 15109995: FSYNC() ERR => -1 (Transport endpoint is not
> connected)
> [2016-02-15 20:07:19.318493] W [fuse-bridge.c:1282:fuse_err_cbk]
> 0-glusterfs-fuse: 15109996: FSYNC() ERR => -1 (Transport endpoint is not
> connected)
> [2016-02-15 20:07:19.318601] W [fuse-bridge.c:1282:fuse_err_cbk]
> 0-glusterfs-fuse: 15109997: FLUSH() ERR => -1 (Transport endpoint is not
> connected)
> [2016-02-15 20:07:20.264111] I [fuse-bridge.c:4984:fuse_thread_proc]
> 0-fuse: unmounting /var/lib/nova/mnt/7e2fea33428149438b876dd122157f27
> [2016-02-15 20:07:20.264361] W [glusterfsd.c:1236:cleanup_and_exit]
> (-->/lib64/libpthread.so.0(+0x7dc5) [0x7f853d29fdc5]
> -->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x7f853e90a855]
> -->/usr/sbin/glusterfs(cleanup_and_exit+0x69) [0x7f853e90a6d9] ) 0-:
> received signum (15), shutting down
> [2016-02-15 20:07:20.264381] I [fuse-bridge.c:5683:fini] 0-fuse: Unmounting
> '/var/lib/nova/mnt/7e2fea33428149438b876dd122157f27'.
> 
> 
> I restart glusterd, glusterfsd but i can't mount
>   /var/lib/nova/mnt/7e2fea33428149438b876dd122157f27

Some things that are missing:
- What version of glusterfs packages were upgraded to what version?
- Which OS/distribution?
- does 'gluster volume status' show any missing processes?

If not all brick processes are running, you probably should check the
logs of those processes (/var/log/glusterfs/bricks/*.log).

The messages that you posted suggest that some (or all?) of the bricks
are not reachable. This causes the mounting to fail.

HTH,
Niels
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160216/dcfc241b/attachment.sig>