[Gluster-users] Q'apla brick does not come online with gluster 5.0, even with fresh install
Computerisms Corporation
bob at computerisms.ca
Thu Nov 1 04:37:32 UTC 2018
My troubleshooting took me to confirming that all my package versions
were lined up and I came to realized that I had gotten version 5.0 from
the debian repos instead of the repo at download.gluster.org. I
downgraded everything to 4.1.5-1 from gluster.org, rebooted, and messed
around a bit, and my gluster is back online.
On 2018-10-31 10:32 a.m., Computerisms Corporation wrote:
> forgot to add output of glusterd console when starting the volume:
>
> [2018-10-31 17:31:33.887923] D [MSGID: 0]
> [glusterd-volume-ops.c:572:__glusterd_handle_cli_start_volume]
> 0-management: Received start vol req for volume moogle-gluster
> [2018-10-31 17:31:33.887976] D [MSGID: 0]
> [glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to
> acquire lock of vol moogle-gluster for
> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
> [2018-10-31 17:31:33.888171] D [MSGID: 0]
> [glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for vol
> moogle-gluster successfully held by bb8c61eb-f321-4485-8a8d-ddc369ac2203
> [2018-10-31 17:31:33.888189] D [MSGID: 0]
> [glusterd-locks.c:519:glusterd_multiple_mgmt_v3_lock] 0-management:
> Returning 0
> [2018-10-31 17:31:33.888204] D [MSGID: 0]
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
> moogle-gluster found
> [2018-10-31 17:31:33.888213] D [MSGID: 0]
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888229] D [MSGID: 0]
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
> moogle-gluster found
> [2018-10-31 17:31:33.888237] D [MSGID: 0]
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888247] D [MSGID: 0]
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
> moogle-gluster found
> [2018-10-31 17:31:33.888256] D [MSGID: 0]
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888269] D [MSGID: 0]
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
> moogle-gluster found
> [2018-10-31 17:31:33.888277] D [MSGID: 0]
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888294] D [MSGID: 0]
> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
> [2018-10-31 17:31:33.888318] D [MSGID: 0]
> [glusterd-mgmt.c:223:gd_mgmt_v3_pre_validate_fn] 0-management: OP = 5.
> Returning 0
> [2018-10-31 17:31:33.888668] D [MSGID: 0]
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
> moogle-gluster found
> [2018-10-31 17:31:33.888682] D [MSGID: 0]
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.888719] E [MSGID: 101012]
> [common-utils.c:4070:gf_is_service_running] 0-: Unable to read pidfile:
> /var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>
> [2018-10-31 17:31:33.888757] I
> [glusterd-utils.c:6300:glusterd_brick_start] 0-management: starting a
> fresh brick process for brick /var/GlusterBrick/moogle-gluster
> [2018-10-31 17:31:33.898943] D [logging.c:1998:_gf_msg_internal]
> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
> About to flush least recently used log message to disk
> [2018-10-31 17:31:33.888780] E [MSGID: 101012]
> [common-utils.c:4070:gf_is_service_running] 0-: Unable to read pidfile:
> /var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>
> [2018-10-31 17:31:33.898942] E [MSGID: 106005]
> [glusterd-utils.c:6305:glusterd_brick_start] 0-management: Unable to
> start brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
> [2018-10-31 17:31:33.899068] D [MSGID: 0]
> [glusterd-utils.c:6315:glusterd_brick_start] 0-management: returning -107
> [2018-10-31 17:31:33.899088] E [MSGID: 106122]
> [glusterd-mgmt.c:308:gd_mgmt_v3_commit_fn] 0-management: Volume start
> commit failed.
> [2018-10-31 17:31:33.899100] D [MSGID: 0]
> [glusterd-mgmt.c:392:gd_mgmt_v3_commit_fn] 0-management: OP = 5.
> Returning -107
> [2018-10-31 17:31:33.899114] E [MSGID: 106122]
> [glusterd-mgmt.c:1557:glusterd_mgmt_v3_commit] 0-management: Commit
> failed for operation Start on local node
> [2018-10-31 17:31:33.899128] D [MSGID: 0]
> [glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx
> modification not required
> [2018-10-31 17:31:33.899140] E [MSGID: 106122]
> [glusterd-mgmt.c:2160:glusterd_mgmt_v3_initiate_all_phases]
> 0-management: Commit Op Failed
> [2018-10-31 17:31:33.899168] D [MSGID: 0]
> [glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to
> release lock of vol moogle-gluster for
> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
> [2018-10-31 17:31:33.899195] D [MSGID: 0]
> [glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for
> vol moogle-gluster successfully released
> [2018-10-31 17:31:33.899211] D [MSGID: 0]
> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
> moogle-gluster found
> [2018-10-31 17:31:33.899221] D [MSGID: 0]
> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
> [2018-10-31 17:31:33.899232] D [MSGID: 0]
> [glusterd-locks.c:464:glusterd_multiple_mgmt_v3_unlock] 0-management:
> Returning 0
> [2018-10-31 17:31:33.899314] D [MSGID: 0]
> [glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management:
> Returning 0
> [2018-10-31 17:31:33.900750] D [socket.c:2927:socket_event_handler]
> 0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
> [2018-10-31 17:31:33.900809] E [MSGID: 101191]
> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to
> dispatch handler
>
>
> On 2018-10-31 10:19 a.m., Computerisms Corporation wrote:
>> Hi,
>>
>> it occurs maybe the previous email was too many words and not enough
>> data. so will try to display the issue differently.
>>
>> gluster created (single brick volume following advice from
>> https://lists.gluster.org/pipermail/gluster-users/2016-October/028821.html):
>>
>>
>> root at sand1lian:~# gluster volume create moogle-gluster
>> sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>>
>> Gluster was started from cli with --debug, console reports the
>> following with creation of the volume:
>>
>> [2018-10-31 17:00:51.555918] D [MSGID: 0]
>> [glusterd-volume-ops.c:328:__glusterd_handle_create_volume]
>> 0-management: Received create volume req
>> [2018-10-31 17:00:51.555963] D [MSGID: 0]
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
>> [2018-10-31 17:00:51.556072] D [MSGID: 0]
>> [glusterd-op-sm.c:209:glusterd_generate_txn_id] 0-management:
>> Transaction_id = 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.556090] D [MSGID: 0]
>> [glusterd-op-sm.c:302:glusterd_set_txn_opinfo] 0-management:
>> Successfully set opinfo for transaction ID :
>> 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.556099] D [MSGID: 0]
>> [glusterd-op-sm.c:309:glusterd_set_txn_opinfo] 0-management: Returning 0
>> [2018-10-31 17:00:51.556108] D [MSGID: 0]
>> [glusterd-syncop.c:1809:gd_sync_task_begin] 0-management: Transaction
>> ID : 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.556127] D [MSGID: 0]
>> [glusterd-locks.c:573:glusterd_mgmt_v3_lock] 0-management: Trying to
>> acquire lock of vol moogle-gluster for
>> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
>> [2018-10-31 17:00:51.556293] D [MSGID: 0]
>> [glusterd-locks.c:657:glusterd_mgmt_v3_lock] 0-management: Lock for
>> vol moogle-gluster successfully held by
>> bb8c61eb-f321-4485-8a8d-ddc369ac2203
>> [2018-10-31 17:00:51.556333] D [MSGID: 0]
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
>> [2018-10-31 17:00:51.556368] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> [2018-10-31 17:00:51.556345] D [MSGID: 0]
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning -1
>> [2018-10-31 17:00:51.556368] D [MSGID: 0]
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556608] D [MSGID: 0]
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.556656] D [MSGID: 0]
>> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556669] D [MSGID: 0]
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556681] D [MSGID: 0]
>> [glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.556690] D [MSGID: 0]
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556699] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-:
>> Returning 0" repeated 3 times between [2018-10-31 17:00:51.556690] and
>> [2018-10-31 17:00:51.556698]
>> [2018-10-31 17:00:51.556699] D [MSGID: 0]
>> [glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning 0
>> [2018-10-31 17:00:51.556728] D [MSGID: 0]
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556738] D [MSGID: 0]
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.556752] D [MSGID: 0]
>> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556764] D [MSGID: 0]
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556772] D [MSGID: 0]
>> [glusterd-utils.c:990:glusterd_volume_brickinfos_delete] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.556781] D [MSGID: 0]
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.556791] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:473:gf_store_handle_destroy] 0-:
>> Returning 0" repeated 3 times between [2018-10-31 17:00:51.556781] and
>> [2018-10-31 17:00:51.556790]
>> [2018-10-31 17:00:51.556791] D [MSGID: 0]
>> [glusterd-utils.c:1042:glusterd_volinfo_delete] 0-management: Returning 0
>> [2018-10-31 17:00:51.556818] D [MSGID: 0]
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.556955] D [MSGID: 0]
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname]
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.557033] D [MSGID: 0]
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.557140] D [MSGID: 0]
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52
>> is local address at interface eno1
>> [2018-10-31 17:00:51.557154] D [MSGID: 0]
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.557172] D [MSGID: 0]
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.557183] D [MSGID: 0]
>> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
>> [2018-10-31 17:00:51.557198] D [MSGID: 0]
>> [glusterd-utils.c:7558:glusterd_new_brick_validate] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.557207] D [MSGID: 0]
>> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
>> [2018-10-31 17:00:51.557392] D [MSGID: 0]
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname]
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.557468] D [MSGID: 0]
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.557542] D [MSGID: 0]
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52
>> is local address at interface eno1
>> [2018-10-31 17:00:51.557554] D [MSGID: 0]
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.557573] D [MSGID: 0]
>> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
>> [2018-10-31 17:00:51.557586] D [MSGID: 0]
>> [glusterd-volume-ops.c:1467:glusterd_op_stage_create_volume]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.557595] D [MSGID: 0]
>> [glusterd-op-sm.c:6014:glusterd_op_stage_validate] 0-management: OP =
>> 1. Returning 0
>> [2018-10-31 17:00:51.557610] D [MSGID: 0]
>> [glusterd-op-sm.c:7659:glusterd_op_bricks_select] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.557620] D [MSGID: 0]
>> [glusterd-syncop.c:1751:gd_brick_op_phase] 0-management: Sent op req
>> to 0 bricks
>> [2018-10-31 17:00:51.557663] D [MSGID: 0]
>> [glusterd-utils.c:678:glusterd_volinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.557693] D [MSGID: 0]
>> [glusterd-utils.c:1094:glusterd_brickinfo_new] 0-management: Returning 0
>> [2018-10-31 17:00:51.557771] D [MSGID: 0]
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname]
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.557844] D [MSGID: 0]
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.557917] D [MSGID: 0]
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52
>> is local address at interface eno1
>> [2018-10-31 17:00:51.557931] D [MSGID: 0]
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.557947] D [MSGID: 0]
>> [glusterd-utils.c:1308:glusterd_brickinfo_new_from_brick]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.557957] D [MSGID: 0]
>> [glusterd-utils.c:1142:glusterd_resolve_brick] 0-management: Returning 0
>> [2018-10-31 17:00:51.558393] D [MSGID: 0]
>> [xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning 0
>> [2018-10-31 17:00:51.558409] D [MSGID: 0]
>> [glusterd-volgen.c:3140:_get_xlator_opt_key_from_vme] 0-glusterd:
>> Returning 0
>> [2018-10-31 17:00:51.558495] W [MSGID: 101095]
>> [xlator.c:180:xlator_volopt_dynload] 0-xlator:
>> /usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/nfs/server.so: cannot
>> open shared object file: No such file or directory
>> [2018-10-31 17:00:51.558509] D [MSGID: 0]
>> [xlator.c:218:xlator_volopt_dynload] 0-xlator: Returning -1
>> [2018-10-31 17:00:51.558566] D [MSGID: 0]
>> [glusterd-store.c:1107:glusterd_store_create_volume_dir] 0-management:
>> Returning with 0
>> [2018-10-31 17:00:51.558593] D [MSGID: 0]
>> [glusterd-store.c:1125:glusterd_store_create_volume_run_dir]
>> 0-management: Returning with 0
>> [2018-10-31 17:00:51.899586] D [MSGID: 0]
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.930562] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> [2018-10-31 17:00:51.930485] D [MSGID: 0]
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.930561] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.932563] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:386:gf_store_save_value]
>> 0-management: returning: 0" repeated 19 times between [2018-10-31
>> 17:00:51.930561] and [2018-10-31 17:00:51.930794]
>> [2018-10-31 17:00:51.932562] D [MSGID: 0]
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.932688] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.932709] D [MSGID: 0]
>> [glusterd-store.c:457:glusterd_store_snapd_write] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.935196] D [MSGID: 0]
>> [glusterd-store.c:521:glusterd_store_perform_snapd_store]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.935226] D [MSGID: 0]
>> [glusterd-store.c:585:glusterd_store_snapd_info] 0-management:
>> Returning with 0
>> [2018-10-31 17:00:51.935251] D [MSGID: 0]
>> [glusterd-store.c:788:_storeopts] 0-management: Storing in
>> volinfo:key= transport.address-family, val=inet
>> [2018-10-31 17:00:51.935290] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.935314] D [MSGID: 0]
>> [glusterd-store.c:788:_storeopts] 0-management: Storing in
>> volinfo:key= nfs.disable, val=on
>> [2018-10-31 17:00:51.935344] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.935360] D [MSGID: 0]
>> [glusterd-store.c:1174:glusterd_store_volinfo_write] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.935382] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.936584] D [MSGID: 0]
>> [store.c:432:gf_store_handle_new] 0-: Returning 0
>> [2018-10-31 17:00:51.936685] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.936807] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:386:gf_store_save_value]
>> 0-management: returning: 0" repeated 10 times between [2018-10-31
>> 17:00:51.936685] and [2018-10-31 17:00:51.936806]
>> [2018-10-31 17:00:51.936807] D [MSGID: 0]
>> [glusterd-store.c:430:glusterd_store_brickinfo_write] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.936833] D [MSGID: 0]
>> [glusterd-store.c:481:glusterd_store_perform_brick_store]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.936841] D [MSGID: 0]
>> [glusterd-store.c:550:glusterd_store_brickinfo] 0-management:
>> Returning with 0
>> [2018-10-31 17:00:51.936848] D [MSGID: 0]
>> [glusterd-store.c:1394:glusterd_store_brickinfos] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.936856] D [MSGID: 0]
>> [glusterd-store.c:1620:glusterd_store_perform_volume_store]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.958353] D [MSGID: 0]
>> [store.c:386:gf_store_save_value] 0-management: returning: 0
>> [2018-10-31 17:00:51.958494] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> The message "D [MSGID: 0] [store.c:386:gf_store_save_value]
>> 0-management: returning: 0" repeated 9 times between [2018-10-31
>> 17:00:51.958353] and [2018-10-31 17:00:51.958493]
>> [2018-10-31 17:00:51.958493] D [MSGID: 0]
>> [glusterd-store.c:1558:glusterd_store_node_state_write] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.960449] D [MSGID: 0]
>> [glusterd-store.c:1592:glusterd_store_perform_node_state_store]
>> 0-management: Returning 0
>> [2018-10-31 17:00:51.960683] D [MSGID: 0]
>> [glusterd-utils.c:2840:glusterd_volume_compute_cksum] 0-management:
>> Returning with 0
>> [2018-10-31 17:00:51.960699] D [MSGID: 0]
>> [glusterd-store.c:1832:glusterd_store_volinfo] 0-management: Returning 0
>> [2018-10-31 17:00:51.960797] D [MSGID: 0]
>> [glusterd-utils.c:181:_brick_for_each] 0-management: Found a brick -
>> sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>> [2018-10-31 17:00:51.961200] D [MSGID: 0]
>> [glusterd-volgen.c:1309:server_check_marker_off] 0-glusterd: Returning 0
>> [2018-10-31 17:00:51.961529] D [MSGID: 0]
>> [glusterd-volgen.c:5816:generate_brick_volfiles] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.961681] D [MSGID: 0]
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname]
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.961756] D [MSGID: 0]
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.961832] D [MSGID: 0]
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52
>> is local address at interface eno1
>> [2018-10-31 17:00:51.961846] D [MSGID: 0]
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.961855] D [MSGID: 0]
>> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management:
>> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>> in volume moogle-gluster
>> [2018-10-31 17:00:51.961864] D [MSGID: 0]
>> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.963126] D [MSGID: 0]
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname]
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.963203] D [MSGID: 0]
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.963280] D [MSGID: 0]
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52
>> is local address at interface eno1
>> [2018-10-31 17:00:51.963298] D [MSGID: 0]
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.963308] D [MSGID: 0]
>> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management:
>> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>> in volume moogle-gluster
>> [2018-10-31 17:00:51.963316] D [MSGID: 0]
>> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.964038] D [MSGID: 0]
>> [glusterd-peer-utils.c:130:glusterd_peerinfo_find_by_hostname]
>> 0-management: Unable to find friend: sand1lian.computerisms.ca
>> [2018-10-31 17:00:51.964112] D [MSGID: 0]
>> [common-utils.c:3590:gf_is_local_addr] 0-management: 192.168.25.52
>> [2018-10-31 17:00:51.964186] D [MSGID: 0]
>> [common-utils.c:3478:gf_interface_search] 0-management: 192.168.25.52
>> is local address at interface eno1
>> [2018-10-31 17:00:51.964200] D [MSGID: 0]
>> [glusterd-peer-utils.c:165:glusterd_hostname_to_uuid] 0-management:
>> returning 0
>> [2018-10-31 17:00:51.964211] D [MSGID: 0]
>> [glusterd-utils.c:1668:glusterd_volume_brickinfo_get] 0-management:
>> Found brick sand1lian.computerisms.ca:/var/GlusterBrick/moogle-gluster
>> in volume moogle-gluster
>> [2018-10-31 17:00:51.964226] D [MSGID: 0]
>> [glusterd-utils.c:1677:glusterd_volume_brickinfo_get] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.965159] D [MSGID: 0]
>> [glusterd-op-sm.c:6150:glusterd_op_commit_perform] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.965177] D [MSGID: 0]
>> [glusterd-utils.c:9664:glusterd_aggr_brick_mount_dirs] 0-management:
>> No brick_count present
>> [2018-10-31 17:00:51.965193] D [MSGID: 0]
>> [glusterd-op-sm.c:5109:glusterd_op_modify_op_ctx] 0-management: op_ctx
>> modification not required
>> [2018-10-31 17:00:51.965219] D [MSGID: 0]
>> [glusterd-locks.c:785:glusterd_mgmt_v3_unlock] 0-management: Trying to
>> release lock of vol moogle-gluster for
>> bb8c61eb-f321-4485-8a8d-ddc369ac2203 as moogle-gluster_vol
>> [2018-10-31 17:00:51.966350] D [MSGID: 0]
>> [glusterd-locks.c:834:glusterd_mgmt_v3_unlock] 0-management: Lock for
>> vol moogle-gluster successfully released
>> [2018-10-31 17:00:51.966462] D [MSGID: 0]
>> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
>> moogle-gluster found
>> [2018-10-31 17:00:51.966479] D [MSGID: 0]
>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management: Returning 0
>> [2018-10-31 17:00:51.966509] D [MSGID: 0]
>> [glusterd-op-sm.c:248:glusterd_get_txn_opinfo] 0-management:
>> Successfully got opinfo for transaction ID :
>> 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.966532] D [MSGID: 0]
>> [glusterd-op-sm.c:252:glusterd_get_txn_opinfo] 0-management: Returning 0
>> [2018-10-31 17:00:51.966551] D [MSGID: 0]
>> [glusterd-op-sm.c:352:glusterd_clear_txn_opinfo] 0-management:
>> Successfully cleared opinfo for transaction ID :
>> 3f5d14c9-ee08-493c-afac-d04d53c12aad
>> [2018-10-31 17:00:51.966668] D [logging.c:1998:_gf_msg_internal]
>> 0-logging-infra: Buffer overflow of a buffer whose size limit is 5.
>> About to flush least recently used log message to disk
>> [2018-10-31 17:00:51.966561] D [MSGID: 0]
>> [glusterd-op-sm.c:356:glusterd_clear_txn_opinfo] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.966667] D [MSGID: 0]
>> [glusterd-rpc-ops.c:199:glusterd_op_send_cli_response] 0-management:
>> Returning 0
>> [2018-10-31 17:00:51.968134] D [socket.c:2927:socket_event_handler]
>> 0-transport: EPOLLERR - disconnecting (sock:7) (non-SSL)
>> [2018-10-31 17:00:51.968183] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to
>> dispatch handler
>> grep: /var/lib/glusterd/vols/moogle-gluster/bricks/*: No such file or
>> directory
>> [2018-10-31 17:00:51.975661] I [run.c:242:runner_log]
>> (-->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe0dbe)
>> [0x7f3f248dbdbe]
>> -->/usr/lib/x86_64-linux-gnu/glusterfs/5.0/xlator/mgmt/glusterd.so(+0xe07fe)
>> [0x7f3f248db7fe]
>> -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(runner_log+0x105)
>> [0x7f3f28ac35a5] ) 0-management: Ran script:
>> /var/lib/glusterd/hooks/1/create/post/S10selinux-label-brick.sh
>> --volname=moogle-gluster
>> [2018-10-31 17:01:12.466614] D
>> [logging.c:1871:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer
>> timed out. About to flush outstanding messages if present
>> [2018-10-31 17:01:12.466667] D
>> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting
>> timer now. Timeout = 120, current buf size = 5
>> [2018-10-31 17:03:12.492414] D
>> [logging.c:1871:gf_log_flush_timeout_cbk] 0-logging-infra: Log timer
>> timed out. About to flush outstanding messages if present
>> [2018-10-31 17:03:12.492447] D
>> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting
>> timer now. Timeout = 120, current buf size = 5
>>
>> Not sure about the unable to find friend message:
>>
>> root at sand1lian:~# dig +short sand1lian.computerisms.ca
>> 192.168.25.52
>>
>> start the volume:
>>
>> root at sand1lian:~# gluster v start moogle-gluster
>> volume start: moogle-gluster: failed: Commit failed on localhost.
>> Please check log file for details.
>>
>> output of cli.log while issuing start command:
>>
>> [2018-10-31 17:08:49.019079] I [cli.c:764:main] 0-cli: Started running
>> gluster with version 5.0
>> [2018-10-31 17:08:49.021694] W [socket.c:3365:socket_connect]
>> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not
>> supported"
>> [2018-10-31 17:08:49.021924] W [socket.c:3365:socket_connect]
>> 0-glusterfs: Error disabling sockopt IPV6_V6ONLY: "Operation not
>> supported"
>> [2018-10-31 17:08:49.101120] I [MSGID: 101190]
>> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 1
>> [2018-10-31 17:08:49.101231] E [MSGID: 101191]
>> [event-epoll.c:671:event_dispatch_epoll_worker] 0-epoll: Failed to
>> dispatch handler
>> [2018-10-31 17:08:49.113485] I
>> [cli-rpc-ops.c:1419:gf_cli_start_volume_cbk] 0-cli: Received resp to
>> start volume
>> [2018-10-31 17:08:49.113626] I [input.c:31:cli_batch] 0-: Exiting
>> with: -1
>>
>> and output of brick log while starting volume:
>>
>> [2018-10-31 17:08:49.107966] I [MSGID: 100030]
>> [glusterfsd.c:2691:main] 0-/usr/sbin/glusterfsd: Started running
>> /usr/sbin/glusterfsd version 5.0 (args: /usr/sbin/glusterfsd -s
>> sand1lian.computerisms.ca --volfile-id
>> moogle-gluster.sand1lian.computerisms.ca.var-GlusterBrick-moogle-gluster
>> -p
>> /var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>> -S /var/run/gluster/f41bfcfaf40deb7d.socket --brick-name
>> /var/GlusterBrick/moogle-gluster -l
>> /var/log/glusterfs/bricks/var-GlusterBrick-moogle-gluster.log
>> --xlator-option
>> *-posix.glusterd-uuid=bb8c61eb-f321-4485-8a8d-ddc369ac2203
>> --process-name brick --brick-port 49157 --xlator-option
>> moogle-gluster-server.listen-port=49157)
>> [2018-10-31 17:08:49.112123] E [socket.c:3466:socket_connect]
>> 0-glusterfs: connection attempt on failed, (Invalid argument)
>> [2018-10-31 17:08:49.112293] I [MSGID: 101190]
>> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 1
>> [2018-10-31 17:08:49.112374] I
>> [glusterfsd-mgmt.c:2424:mgmt_rpc_notify] 0-glusterfsd-mgmt:
>> disconnected from remote-host: sand1lian.computerisms.ca
>> [2018-10-31 17:08:49.112399] I
>> [glusterfsd-mgmt.c:2444:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted
>> all volfile servers
>> [2018-10-31 17:08:49.112656] W [glusterfsd.c:1481:cleanup_and_exit]
>> (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(+0xf023) [0x7f3466c12023]
>> -->/usr/sbin/glusterfsd(+0x1273e) [0x557f4ea6373e]
>> -->/usr/sbin/glusterfsd(cleanup_and_exit+0x54) [0x557f4ea5be94] ) 0-:
>> received signum (1), shutting down
>> [2018-10-31 17:08:49.112973] E [socket.c:3466:socket_connect]
>> 0-glusterfs: connection attempt on failed, (Invalid argument)
>> [2018-10-31 17:08:49.112996] W [rpc-clnt.c:1683:rpc_clnt_submit]
>> 0-glusterfs: error returned while attempting to connect to
>> host:(null), port:0
>> [2018-10-31 17:08:49.113007] I
>> [socket.c:3710:socket_submit_outgoing_msg] 0-glusterfs: not connected
>> (priv->connected = 0)
>> [2018-10-31 17:08:49.113016] W [rpc-clnt.c:1695:rpc_clnt_submit]
>> 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x2
>> Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport
>> (glusterfs)
>>
>>
>> still seeing the empty pid file and the connection attempt on failed,
>> (Invalid argument) as the mostly likely culprits, but have read
>> everything of relevance I have found on google and not discovered a
>> solution yet...
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On 2018-10-30 9:15 p.m., Computerisms Corporation wrote:
>>> Hi,
>>>
>>> Fortunately I am playing in a sandbox right now, but I am good and
>>> stuck and hoping someone can point me in the right direction.
>>>
>>> I have been playing for about 3 months with a gluster that currently
>>> has one brick. The idea is that I have a server with data, I need to
>>> migrate that server onto the new gluster-capable server, then I can
>>> use the original server to make a 2nd brick, then I will be able to
>>> make some room on a 3rd server for an arbiter brick. So I am
>>> building and testing to be sure it all works before I try it in
>>> production.
>>>
>>> Yesterday morning I was plugging away at figuring out how to make
>>> stuff work on the new gluster server when I ran into an issue trying
>>> to rm -rf a directory and it telling me it wasn't empty when ls -al
>>> showed that it was. This has happened to me before, and what I did
>>> to fix it before was unmount the Glusterfs, go into the brick, delete
>>> the files, and remount the Glusterfs. I did that and it appeared to
>>> mount fine, but when I tried to access the gluster mount, it gave me
>>> an error that there were too many levels of symlinks.
>>>
>>> I spent my day yesterday trying pretty much everything I could find
>>> on google and a few things I couldn't. In the past when stuff has
>>> gone funny with gluster on this box, I have always shut everything
>>> down and checked if there was a new version of gluster, and indeed
>>> there was version 5.0 available. So I did the upgrade quite early in
>>> the day. Sadly it didn't fix my problem, but it did give me an error
>>> that led me to modifying my hosts file to be ipv6 resolvable. Also
>>> after that, the only time the gluster would mount was at reboot, but
>>> always with the symlinks error, and it wasn't really mounted as
>>> reported by mount, but the directory could be unmounted.
>>>
>>> Having struck out completely yesterday, today I decided to start with
>>> a new machine. I kept a history of the commands I had used to build
>>> the gluster a few months back and pasted them all in. Found that the
>>> 5.0 package does not enable systemd, found that I needed the ipv6
>>> entries in the hosts file again, and also found the same problem in
>>> that the glusterfs would not mount, the symlinks error at reboot, and
>>> the same log entries.
>>>
>>> I am still pretty new with gluster, so my best may not be that good,
>>> but as best as I can tell the issue is that the brick will not start,
>>> even with the force option. I think the problem boils down to one or
>>> both of two lines in the logs. In the glusterd.log I have a line:
>>>
>>> 0-: Unable to read pidfile:
>>> /var/run/gluster/vols/moogle-gluster/sand1lian.computerisms.ca-var-GlusterBrick-moogle-gluster.pid
>>>
>>>
>>> The file exists, and I can't see anything wrong with permissions on
>>> the file or the file tree leading to it, but it is a zero-bit file,
>>> so I am thinking the problem is not the file, but that it can't read
>>> the contents of the file because there aren't any.
>>>
>>> The other log entry is in the brick log:
>>>
>>> 0-glusterfs: connection attempt on failed, (Invalid argument)
>>>
>>> When I looked this up, it seems in my case there should be an attempt
>>> to connect on 127.0.0.1, but given the double space I am thinking the
>>> host argument is null, hence the invalid argument. It occurs that
>>> maybe I still need some other entry in my hosts file to satisfy this,
>>> but I can't think what it would be. I have created DNS entries; dig
>>> works, and both hostname and FQDN resolve.
>>>
>>> I have tried to change a lot of things today, so probably things are
>>> buggered up beyond hope right now so even if I do find the solution
>>> maybe it won't work. will wipe the new machine and start over again
>>> tomorrow.
>>>
>>> I realize the post is kinda long, sorry for that, but I want to make
>>> sure I get every thing important. In fairness, though, I could
>>> easily double the length of this post with possibly relevant things
>>> (if you are interested). If you are still reading, thank you so
>>> much, I would appreciate anything, even a wild guess, as to how to
>>> move forward on this?
>>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
More information about the Gluster-users
mailing list