[Gluster-users] Fedora upgrade to f24 installed 3.8.0 client and broke mounting

Alastair Neil ajneil.tech at gmail.com
Fri Jun 24 18:12:11 UTC 2016 

I upgraded my fedora 23 system to f24 a couple of days ago, now I am unable
to mount my gluster cluster.

The update installed:

glusterfs-3.8.0-1.fc24.x86_64
glusterfs-libs-3.8.0-1.fc24.x86_64
glusterfs-fuse-3.8.0-1.fc24.x86_64
glusterfs-client-xlators-3.8.0-1.fc24.x86_64

the gluster is running 3.7.11

The volume is replica 3

I see these errors in the mount log:

[2016-06-24 17:55:34.016462] I [MSGID: 100030] [glusterfsd.c:2408:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.8.0
(args: /usr/sbin/glusterfs --volfile-server=gluster1 --volfile-id=homes
/mnt/homes)
[2016-06-24 17:55:34.094345] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2016-06-24 17:55:34.240135] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 2
[2016-06-24 17:55:34.240130] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 4
[2016-06-24 17:55:34.240130] I [MSGID: 101190]
[event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 3
[2016-06-24 17:55:34.241499] I [MSGID: 114020] [client.c:2356:notify]
0-homes-client-2: parent translators are ready, attempting connect on
transport
[2016-06-24 17:55:34.249172] I [MSGID: 114020] [client.c:2356:notify]
0-homes-client-5: parent translators are ready, attempting connect on
transport
[2016-06-24 17:55:34.250186] I [rpc-clnt.c:1855:rpc_clnt_reconfig]
0-homes-client-2: changing port to 49171 (from 0)
[2016-06-24 17:55:34.253347] I [MSGID: 114020] [client.c:2356:notify]
0-homes-client-6: parent translators are ready, attempting connect on
transport
[2016-06-24 17:55:34.254213] I [rpc-clnt.c:1855:rpc_clnt_reconfig]
0-homes-client-5: changing port to 49154 (from 0)
[2016-06-24 17:55:34.255115] I [MSGID: 114057]
[client-handshake.c:1441:select_server_supported_programs]
0-homes-client-2: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-06-24 17:55:34.255861] W [MSGID: 114007]
[client-handshake.c:1176:client_setvolume_cbk] 0-homes-client-2: failed to
find key 'child_up' in the options
[2016-06-24 17:55:34.259097] I [MSGID: 114057]
[client-handshake.c:1441:select_server_supported_programs]
0-homes-client-5: Using Program GlusterFS 3.3, Num (1298437), Version (330)
Final graph:
+------------------------------------------------------------------------------+
  1: volume homes-client-2
  2:     type protocol/client
  3:     option clnt-lk-version 1
  4:     option volfile-checksum 0
  5:     option volfile-key homes
  6:     option client-version 3.8.0
  7:     option process-uuid
Island-29185-2016/06/24-17:55:34:10054-homes-client-2-0-0
  8:     option fops-version 1298437
  9:     option ping-timeout 20
 10:     option remote-host gluster-2
 11:     option remote-subvolume /export/brick2/home
 12:     option transport-type socket
 13:     option event-threads 4
 14:     option send-gids true
 15: end-volume
 16:
 17: volume homes-client-5
 18:     type protocol/client
 19:     option clnt-lk-version 1
 20:     option volfile-checksum 0
 21:     option volfile-key homes
 22:     option client-version 3.8.0
 23:     option process-uuid
Island-29185-2016/06/24-17:55:34:10054-homes-client-5-0-0
 24:     option fops-version 1298437
 25:     option ping-timeout 20
 26:     option remote-host gluster1.vsnet.gmu.edu
 27:     option remote-subvolume /export/brick2/home
 28:     option transport-type socket
 29:     option event-threads 4
 30:     option send-gids true
 31: end-volume
 32:
 33: volume homes-client-6
 34:     type protocol/client
 35:     option ping-timeout 20
 36:     option remote-host gluster0
 37:     option remote-subvolume /export/brick2/home
 38:     option transport-type socket
 39:     option event-threads 4
 40:     option send-gids true
 41: end-volume
 42:
 43: volume homes-replicate-0
 44:     type cluster/replicate
 45:     option background-self-heal-count 20
 46:     option metadata-self-heal on
 47:     option data-self-heal off
 48:     option entry-self-heal on
 49:     option data-self-heal-window-size 8
 50:     option data-self-heal-algorithm diff
 51:     option eager-lock on
 52:     option quorum-type auto
 53:     option self-heal-readdir-size 64KB
 54:     subvolumes homes-client-2 homes-client-5 homes-client-6
 55: end-volume
 56:
 57: volume homes-dht
 58:     type cluster/distribute
 59:     option min-free-disk 5%
 60:     option rebalance-stats on
 61:     option readdir-optimize on
 62:     subvolumes homes-replicate-0
 63: end-volume
 64:
 65: volume homes-read-ahead
 66:     type performance/read-ahead
 67:     subvolumes homes-dht
 68: end-volume
 69:
 70: volume homes-io-cache
 71:     type performance/io-cache
 72:     subvolumes homes-read-ahead
 73: end-volume
 74:
 75: volume homes-quick-read
 76:     type performance/quick-read
 77:     subvolumes homes-io-cache
 78: end-volume
 79:
 80: volume homes-open-behind
 81:     type performance/open-behind
 82:     subvolumes homes-quick-read
 83: end-volume
 84:
 85: volume homes-md-cache
 86:     type performance/md-cache
 87:     subvolumes homes-open-behind
 88: end-volume
 89:
 90: volume homes
 91:     type debug/io-stats
 92:     option log-level INFO
 93:     option latency-measurement off
 94:     option count-fop-hits on
 95:     subvolumes homes-md-cache
 96: end-volume
 97:
 98: volume meta-autoload
 99:     type meta
100:     subvolumes homes
101: end-volume
102:
+------------------------------------------------------------------------------+
[2016-06-24 17:55:34.261219] I [rpc-clnt.c:1855:rpc_clnt_reconfig]
0-homes-client-6: changing port to 49153 (from 0)
[2016-06-24 17:55:34.266096] I [MSGID: 114057]
[client-handshake.c:1441:select_server_supported_programs]
0-homes-client-6: Using Program GlusterFS 3.3, Num (1298437), Version (330)
[2016-06-24 17:55:34.266905] W [MSGID: 114007]
[client-handshake.c:1176:client_setvolume_cbk] 0-homes-client-6: failed to
find key 'child_up' in the options
[2016-06-24 17:55:34.273618] W [MSGID: 114007]
[client-handshake.c:1176:client_setvolume_cbk] 0-homes-client-5: failed to
find key 'child_up' in the options
[2016-06-24 17:55:45.071822] I [fuse-bridge.c:5241:fuse_graph_setup]
0-fuse: switched to graph 0
[2016-06-24 17:55:45.072098] I [fuse-bridge.c:4153:fuse_init]
0-glusterfs-fuse: FUSE inited with protocol versions: glusterfs 7.24 kernel
7.24
[2016-06-24 17:55:45.072229] I [MSGID: 108006]
[afr-common.c:4268:afr_local_init] 0-homes-replicate-0: no subvolumes up
[2016-06-24 17:55:45.072537] E
[dht-helper.c:1600:dht_inode_ctx_time_update]
(-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a)
[0x7fd0e9e10b4a]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746)
[0x7fd0e9b74746]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4)
[0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]
[2016-06-24 17:55:45.072773] E
[dht-helper.c:1600:dht_inode_ctx_time_update]
(-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a)
[0x7fd0e9e10b4a]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746)
[0x7fd0e9b74746]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4)
[0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]
[2016-06-24 17:55:45.072817] W [fuse-bridge.c:767:fuse_attr_cbk]
0-glusterfs-fuse: 2: LOOKUP() / => -1 (Transport endpoint is not connected)
[2016-06-24 17:55:45.072968] E
[dht-helper.c:1600:dht_inode_ctx_time_update]
(-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a)
[0x7fd0e9e10b4a]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746)
[0x7fd0e9b74746]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4)
[0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]
[2016-06-24 17:55:45.076081] E
[dht-helper.c:1600:dht_inode_ctx_time_update]
(-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a)
[0x7fd0e9e10b4a]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746)
[0x7fd0e9b74746]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4)
[0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]
[2016-06-24 17:55:45.076285] E
[dht-helper.c:1600:dht_inode_ctx_time_update]
(-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/replicate.so(+0x49b4a)
[0x7fd0e9e10b4a]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x32746)
[0x7fd0e9b74746]
-->/usr/lib64/glusterfs/3.8.0/xlator/cluster/distribute.so(+0x99e4)
[0x7fd0e9b4b9e4] ) 0-homes-dht: invalid argument: inode [Invalid argument]
[2016-06-24 17:55:45.090890] I [fuse-bridge.c:5082:fuse_thread_proc]
0-fuse: unmounting /mnt/homes
The message "I [MSGID: 108006] [afr-common.c:4268:afr_local_init]
0-homes-replicate-0: no subvolumes up" repeated 4 times between [2016-06-24
17:55:45.072229] and [2016-06-24 17:55:45.076259]
[2016-06-24 17:55:45.091051] W [glusterfsd.c:1286:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x75ca) [0x7fd0f77495ca]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xf5) [0x5647b219d565]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x54) [0x5647b219d3c4] ) 0-:
received signum (15), shutting down
[2016-06-24 17:55:45.091070] I [fuse-bridge.c:5793:fini] 0-fuse: Unmounting
'/mnt/homes'.



I checked the release notes for 3.8.0 but I did not see any caveats or
compatibility warnings.

Anyone else seeing issues with 3.8 clients mounting 3.7 volumes?


-Alastair
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160624/06e02f43/attachment.html>

More information about the Gluster-users mailing list