[Gluster-users] Ip based peer probe volume create error
Ercan Aydoğan
ercan.aydogan at gmail.com
Wed Feb 7 23:28:31 UTC 2018
After fresh installation ip based peer probe same error message. Staging of operation 'Volume Create' failed on localhost : Host 51.15.77.14 is not in 'Peer in Cluster' state
node 1
root at pri:/var/log/glusterfs# cat glusterd.log
[2018-02-07 23:06:34.526018] I [MSGID: 100030] [glusterfsd.c:2556:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.13.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-02-07 23:06:34.536120] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2018-02-07 23:06:34.536172] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2018-02-07 23:06:34.536195] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2018-02-07 23:06:34.544353] W [MSGID: 103071] [rdma.c:4631:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2018-02-07 23:06:34.544408] W [MSGID: 103055] [rdma.c:4940:init] 0-rdma.management: Failed to initialize IB Device
[2018-02-07 23:06:34.544430] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2018-02-07 23:06:34.544511] W [rpcsvc.c:1770:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2018-02-07 23:06:34.544535] E [MSGID: 106243] [glusterd.c:1769:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2018-02-07 23:06:37.885430] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:06:37.885511] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:06:37.885514] I [MSGID: 106514] [glusterd-store.c:2263:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 31302
[2018-02-07 23:06:37.885570] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory]
[2018-02-07 23:06:37.890034] I [MSGID: 106194] [glusterd-store.c:3831:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.listen-backlog 10
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2018-02-07 23:06:37.890658] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-02-07 23:07:12.455572] I [MSGID: 106487] [glusterd-handler.c:1243:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 163.172.151.120 24007
[2018-02-07 23:07:12.456285] I [MSGID: 106129] [glusterd-handler.c:3624:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 163.172.151.120 (24007)
[2018-02-07 23:07:12.461909] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-02-07 23:07:12.461957] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-07 23:07:12.462126] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2018-02-07 23:07:12.462796] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-02-07 23:07:12.462914] I [MSGID: 106498] [glusterd-handler.c:3550:glusterd_friend_add] 0-management: connect returned 0
[2018-02-07 23:07:12.490489] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:07:12.490632] I [MSGID: 106477] [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:12.584639] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b, host: 163.172.151.120
[2018-02-07 23:07:12.584690] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
[2018-02-07 23:07:12.629513] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b, host: 163.172.151.120, port: 0
[2018-02-07 23:07:12.653271] I [MSGID: 106163] [glusterd-handshake.c:1361:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-02-07 23:07:12.674036] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:14.037145] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to 163.172.151.120, op_ret: 0, op_errno: 0, ret: 0
[2018-02-07 23:07:14.050712] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:14.054458] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 163.172.151.120 (0), ret: 0, op_ret: 0
[2018-02-07 23:07:14.067810] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:14.067884] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:14.070706] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:27.876367] I [MSGID: 106487] [glusterd-handler.c:1243:__glusterd_handle_cli_probe] 0-glusterd: Received CLI probe req 51.15.90.60 24007
[2018-02-07 23:07:27.877159] I [MSGID: 106129] [glusterd-handler.c:3624:glusterd_probe_begin] 0-glusterd: Unable to find peerinfo for host: 51.15.90.60 (24007)
[2018-02-07 23:07:27.882767] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-02-07 23:07:27.882814] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-07 23:07:27.882975] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2018-02-07 23:07:27.883335] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-02-07 23:07:27.883432] I [MSGID: 106498] [glusterd-handler.c:3550:glusterd_friend_add] 0-management: connect returned 0
[2018-02-07 23:07:27.941707] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a, host: 51.15.90.60
[2018-02-07 23:07:27.941760] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
[2018-02-07 23:07:27.982678] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a, host: 51.15.90.60, port: 0
[2018-02-07 23:07:27.992001] I [MSGID: 106163] [glusterd-handshake.c:1361:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-02-07 23:07:28.000650] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.081112] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to 51.15.90.60, op_ret: 0, op_errno: 0, ret: 0
[2018-02-07 23:07:28.081698] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.085086] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 51.15.90.60 (0), ret: 0, op_ret: 0
[2018-02-07 23:07:28.092362] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.092440] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:28.095256] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.108036] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:48.384034] E [MSGID: 106429] [glusterd-utils.c:1268:glusterd_brickinfo_new_from_brick] 0-management: Failed to convert hostname 51.15.77.14 to uuid
[2018-02-07 23:07:48.384088] E [MSGID: 106301] [glusterd-syncop.c:1352:gd_stage_op_phase] 0-management: Staging of operation 'Volume Create' failed on localhost : Host 51.15.77.14 is not in 'Peer in Cluster' state
[2018-02-07 23:10:15.952540] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
root at pri:/var/log/glusterfs# cat cmd_history.log
[2018-02-07 23:07:12.629562] : peer probe 163.172.151.120 : SUCCESS
[2018-02-07 23:07:27.982728] : peer probe 51.15.90.60 : SUCCESS
[2018-02-07 23:07:48.403504] : volume create myvol1 replica 3 transport tcp 51.15.77.14:/gluster/brick1/mpoint1 51.15.90.60:/gluster/brick1/mpoint1 163.172.151.120:/gluster/brick1/mpoint1 force : FAILED : Host 51.15.77.14 is not in 'Peer in Cluster' state
root at sec:/var/log/glusterfs# cat glusterd.log
[2018-02-07 23:06:40.122276] I [MSGID: 100030] [glusterfsd.c:2556:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.13.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-02-07 23:06:40.141034] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2018-02-07 23:06:40.141084] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2018-02-07 23:06:40.141108] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2018-02-07 23:06:40.148754] W [MSGID: 103071] [rdma.c:4631:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2018-02-07 23:06:40.148816] W [MSGID: 103055] [rdma.c:4940:init] 0-rdma.management: Failed to initialize IB Device
[2018-02-07 23:06:40.148838] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2018-02-07 23:06:40.148919] W [rpcsvc.c:1770:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2018-02-07 23:06:40.148943] E [MSGID: 106243] [glusterd.c:1769:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2018-02-07 23:06:43.533480] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:06:43.533561] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:06:43.533565] I [MSGID: 106514] [glusterd-store.c:2263:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 31302
[2018-02-07 23:06:43.533626] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory]
[2018-02-07 23:06:43.537948] I [MSGID: 106194] [glusterd-store.c:3831:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.listen-backlog 10
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2018-02-07 23:06:43.538578] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-02-07 23:07:27.891335] I [MSGID: 106163] [glusterd-handshake.c:1361:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-02-07 23:07:27.891397] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:07:27.891516] I [MSGID: 106477] [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:27.901193] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:27.934639] I [MSGID: 106129] [glusterd-handler.c:2926:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 51.15.77.14 (24007)
[2018-02-07 23:07:27.939591] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-02-07 23:07:27.939643] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-07 23:07:27.939815] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2018-02-07 23:07:27.940274] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-02-07 23:07:27.940392] I [MSGID: 106498] [glusterd-handler.c:3550:glusterd_friend_add] 0-management: connect returned 0
[2018-02-07 23:07:27.940459] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to 51.15.77.14, op_ret: 0, op_errno: 0, ret: 0
[2018-02-07 23:07:27.942425] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:27.981420] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 51.15.77.14 (0), ret: 0, op_ret: 0
[2018-02-07 23:07:28.080313] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25, host: 51.15.77.14
[2018-02-07 23:07:28.080362] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
[2018-02-07 23:07:28.084580] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25, host: 51.15.77.14, port: 0
[2018-02-07 23:07:28.088394] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:28.093284] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-02-07 23:07:28.093332] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-07 23:07:28.093493] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2018-02-07 23:07:28.093816] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-02-07 23:07:28.093909] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2018-02-07 23:07:28.093940] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:28.094046] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:28.150270] I [MSGID: 106163] [glusterd-handshake.c:1361:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-02-07 23:07:28.185322] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b, host: 163.172.151.120, port: 0
[2018-02-07 23:07:28.197155] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:28.230262] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 163.172.151.120 (0), ret: 0, op_ret: 0
[2018-02-07 23:07:28.236846] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:28.243655] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:28.247163] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:28.248146] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:10:30.149826] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
root at third:/var/log/glusterfs# cat glusterd.log
[2018-02-07 23:06:43.461573] I [MSGID: 100030] [glusterfsd.c:2556:main] 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 3.13.2 (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)
[2018-02-07 23:06:43.483471] I [MSGID: 106478] [glusterd.c:1423:init] 0-management: Maximum allowed open file descriptors set to 65536
[2018-02-07 23:06:43.483518] I [MSGID: 106479] [glusterd.c:1481:init] 0-management: Using /var/lib/glusterd as working directory
[2018-02-07 23:06:43.483542] I [MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster as pid file working directory
[2018-02-07 23:06:43.492137] W [MSGID: 103071] [rdma.c:4631:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event channel creation failed [No such device]
[2018-02-07 23:06:43.492195] W [MSGID: 103055] [rdma.c:4940:init] 0-rdma.management: Failed to initialize IB Device
[2018-02-07 23:06:43.492218] W [rpc-transport.c:350:rpc_transport_load] 0-rpc-transport: 'rdma' initialization failed
[2018-02-07 23:06:43.492303] W [rpcsvc.c:1770:rpcsvc_create_listener] 0-rpc-service: cannot create listener, initing the transport failed
[2018-02-07 23:06:43.492327] E [MSGID: 106243] [glusterd.c:1769:init] 0-management: creation of 1 listeners failed, continuing with succeeded transport
[2018-02-07 23:06:47.073543] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:06:47.073633] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:06:47.073636] I [MSGID: 106514] [glusterd-store.c:2263:glusterd_restore_op_version] 0-management: Detected new install. Setting op-version to maximum : 31302
[2018-02-07 23:06:47.073695] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/options. [No such file or directory]
[2018-02-07 23:06:47.077544] I [MSGID: 106194] [glusterd-store.c:3831:glusterd_store_retrieve_missed_snaps_list] 0-management: No missed snaps list.
Final graph:
+------------------------------------------------------------------------------+
1: volume management
2: type mgmt/glusterd
3: option rpc-auth.auth-glusterfs on
4: option rpc-auth.auth-unix on
5: option rpc-auth.auth-null on
6: option rpc-auth-allow-insecure on
7: option transport.listen-backlog 10
8: option event-threads 1
9: option ping-timeout 0
10: option transport.socket.read-fail-log off
11: option transport.socket.keepalive-interval 2
12: option transport.socket.keepalive-time 10
13: option transport-type rdma
14: option working-directory /var/lib/glusterd
15: end-volume
16:
+------------------------------------------------------------------------------+
[2018-02-07 23:06:47.078154] I [MSGID: 101190] [event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1
[2018-02-07 23:07:12.515797] I [MSGID: 106163] [glusterd-handshake.c:1361:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-02-07 23:07:12.515869] E [MSGID: 101032] [store.c:441:gf_store_handle_retrieve] 0-: Path corresponding to /var/lib/glusterd/glusterd.info. [No such file or directory]
[2018-02-07 23:07:12.516013] I [MSGID: 106477] [glusterd.c:190:glusterd_uuid_generate_save] 0-management: generated UUID: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
[2018-02-07 23:07:12.539761] I [MSGID: 106490] [glusterd-handler.c:2891:__glusterd_handle_probe_query] 0-glusterd: Received probe from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:12.571076] I [MSGID: 106129] [glusterd-handler.c:2926:__glusterd_handle_probe_query] 0-glusterd: Unable to find peerinfo for host: 51.15.77.14 (24007)
[2018-02-07 23:07:12.576726] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-02-07 23:07:12.576781] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-07 23:07:12.576952] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2018-02-07 23:07:12.577467] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-02-07 23:07:12.577592] I [MSGID: 106498] [glusterd-handler.c:3550:glusterd_friend_add] 0-management: connect returned 0
[2018-02-07 23:07:12.577661] I [MSGID: 106493] [glusterd-handler.c:2954:__glusterd_handle_probe_query] 0-glusterd: Responded to 51.15.77.14, op_ret: 0, op_errno: 0, ret: 0
[2018-02-07 23:07:12.591119] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:12.622531] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 51.15.77.14 (0), ret: 0, op_ret: 0
[2018-02-07 23:07:14.041482] I [MSGID: 106511] [glusterd-rpc-ops.c:262:__glusterd_probe_cbk] 0-management: Received probe resp from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25, host: 51.15.77.14
[2018-02-07 23:07:14.041537] I [MSGID: 106511] [glusterd-rpc-ops.c:422:__glusterd_probe_cbk] 0-glusterd: Received resp to probe req
[2018-02-07 23:07:14.058621] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25, host: 51.15.77.14, port: 0
[2018-02-07 23:07:14.063567] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:14.063628] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:14.072598] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:28.094810] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
[2018-02-07 23:07:28.094890] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:28.100295] W [MSGID: 106062] [glusterd-handler.c:3400:glusterd_transport_inet_options_build] 0-glusterd: Failed to get tcp-user-timeout
[2018-02-07 23:07:28.100343] I [rpc-clnt.c:1044:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2018-02-07 23:07:28.100505] W [MSGID: 101002] [options.c:995:xl_opt_validate] 0-management: option 'address-family' is deprecated, preferred is 'transport.address-family', continuing with correction
[2018-02-07 23:07:28.100829] W [socket.c:3216:socket_connect] 0-management: Error disabling sockopt IPV6_V6ONLY: "Protocol not available"
[2018-02-07 23:07:28.100941] I [MSGID: 106498] [glusterd-handler.c:3603:glusterd_friend_add_from_peerinfo] 0-management: connect returned 0
[2018-02-07 23:07:28.132112] I [MSGID: 106163] [glusterd-handshake.c:1361:__glusterd_mgmt_hndsk_versions_ack] 0-management: using the op-version 31302
[2018-02-07 23:07:28.146325] I [MSGID: 106490] [glusterd-handler.c:2540:__glusterd_handle_incoming_friend_req] 0-glusterd: Received probe from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.179890] I [MSGID: 106493] [glusterd-handler.c:3800:glusterd_xfer_friend_add_resp] 0-glusterd: Responded to 51.15.90.60 (0), ret: 0, op_ret: 0
[2018-02-07 23:07:28.192778] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.195524] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:28.235528] I [MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a, host: 51.15.90.60, port: 0
[2018-02-07 23:07:28.239135] I [MSGID: 106492] [glusterd-handler.c:2718:__glusterd_handle_friend_update] 0-glusterd: Received friend update from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:07:28.242708] I [MSGID: 106502] [glusterd-handler.c:2763:__glusterd_handle_friend_update] 0-management: Received my uuid as Friend
[2018-02-07 23:07:28.252522] I [MSGID: 106493] [glusterd-rpc-ops.c:701:__glusterd_friend_update_cbk] 0-management: Received ACC from uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
[2018-02-07 23:10:36.230769] I [MSGID: 106487] [glusterd-handler.c:1485:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
root at pri:/var/log/glusterfs# gluster peer status
Number of Peers: 2
Hostname: 163.172.151.120
Uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
State: Peer in Cluster (Connected)
Hostname: 51.15.90.60
Uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
State: Peer in Cluster (Connected)
root at sec:/var/log/glusterfs# gluster peer status
Number of Peers: 2
Hostname: 51.15.77.14
Uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
State: Peer in Cluster (Connected)
Hostname: 163.172.151.120
Uuid: 92d0bd97-ca85-4836-ba7d-3377a8f9417b
State: Peer in Cluster (Connected)
root at third:/var/log/glusterfs# gluster peer status
Number of Peers: 2
Hostname: 51.15.77.14
Uuid: 71064698-39f8-4b3d-840b-306b92ff8e25
State: Peer in Cluster (Connected)
Hostname: 51.15.90.60
Uuid: 089abf08-fe7a-4f17-a3b8-311abace6f5a
State: Peer in Cluster (Connected)
> On 8 Feb 2018, at 01:13, Lindsay Mathieson <lindsay.mathieson at gmail.com> wrote:
>
> On 8/02/2018 4:45 AM, Gaurav Yadav wrote:
>> After seeing command history, I could see that you have 3 nodes, and firstly you are peer probing 51.15.90.60 and 163.172.151.120 from 51.15.77.14
>> So here itself you have 3 node cluster, after all this you are going on node 2 and again peer probing 51.15.77.14.
>> Ideally it should work, with above steps, but due to some address resolution volume create is getting failed, and from glusterd logs its not clear.
>
> Given 163.172.151.120 is on a different subnet I'd guess routing issues. Can it be pinged from both the other nodes?
>
> --
> Lindsay
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180208/7079299c/attachment.html>
More information about the Gluster-users
mailing list