[Gluster-devel] 0-management: Commit failed for operation Start on local node
TomK
tomkcpr at mdevsys.com
Wed Sep 25 11:17:07 UTC 2019
This issue looked nearly identical to:
https://bugzilla.redhat.com/show_bug.cgi?id=1702316
so tried:
option transport.socket.listen-port 24007
And it worked:
[root at mdskvm-p01 glusterfs]# systemctl stop glusterd
[root at mdskvm-p01 glusterfs]# history|grep server-quorum
3149 gluster volume set mdsgv01 cluster.server-quorum-type none
3186 history|grep server-quorum
[root at mdskvm-p01 glusterfs]# gluster volume set mdsgv01
transport.socket.listen-port 24007
Connection failed. Please check if gluster daemon is operational.
[root at mdskvm-p01 glusterfs]# systemctl start glusterd
[root at mdskvm-p01 glusterfs]# gluster volume set mdsgv01
transport.socket.listen-port 24007
volume set: failed: option : transport.socket.listen-port does not exist
Did you mean transport.keepalive or ...listen-backlog?
[root at mdskvm-p01 glusterfs]#
[root at mdskvm-p01 glusterfs]# netstat -pnltu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address
State PID/Program name
tcp 0 0 0.0.0.0:16514 0.0.0.0:*
LISTEN 4562/libvirtd
tcp 0 0 0.0.0.0:24007 0.0.0.0:*
LISTEN 24193/glusterd
tcp 0 0 0.0.0.0:2223 0.0.0.0:*
LISTEN 4277/sshd
tcp 0 0 0.0.0.0:111 0.0.0.0:*
LISTEN 1/systemd
tcp 0 0 0.0.0.0:51760 0.0.0.0:*
LISTEN 4479/rpc.statd
tcp 0 0 0.0.0.0:54322 0.0.0.0:*
LISTEN 13229/python
tcp 0 0 0.0.0.0:22 0.0.0.0:*
LISTEN 4279/sshd
tcp6 0 0 :::54811 :::*
LISTEN 4479/rpc.statd
tcp6 0 0 :::16514 :::*
LISTEN 4562/libvirtd
tcp6 0 0 :::2223 :::*
LISTEN 4277/sshd
tcp6 0 0 :::111 :::*
LISTEN 3357/rpcbind
tcp6 0 0 :::54321 :::*
LISTEN 13225/python2
tcp6 0 0 :::22 :::*
LISTEN 4279/sshd
udp 0 0 0.0.0.0:24009 0.0.0.0:*
4281/python2
udp 0 0 0.0.0.0:38873 0.0.0.0:*
4479/rpc.statd
udp 0 0 0.0.0.0:111 0.0.0.0:*
1/systemd
udp 0 0 127.0.0.1:323 0.0.0.0:*
3361/chronyd
udp 0 0 127.0.0.1:839 0.0.0.0:*
4479/rpc.statd
udp 0 0 0.0.0.0:935 0.0.0.0:*
3357/rpcbind
udp6 0 0 :::46947 :::*
4479/rpc.statd
udp6 0 0 :::111 :::*
3357/rpcbind
udp6 0 0 ::1:323 :::*
3361/chronyd
udp6 0 0 :::935 :::*
3357/rpcbind
[root at mdskvm-p01 glusterfs]# gluster volume start mdsgv01
volume start: mdsgv01: success
[root at mdskvm-p01 glusterfs]# gluster volume info
Volume Name: mdsgv01
Type: Replicate
Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/glusterv02
Brick2: mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
Options Reconfigured:
storage.owner-gid: 36
cluster.data-self-heal-algorithm: full
performance.low-prio-threads: 32
features.shard-block-size: 512MB
features.shard: on
storage.owner-uid: 36
cluster.server-quorum-type: none
cluster.quorum-type: none
server.event-threads: 8
client.event-threads: 8
performance.write-behind-window-size: 8MB
performance.io-thread-count: 16
performance.cache-size: 1GB
nfs.trusted-sync: on
server.allow-insecure: on
performance.readdir-ahead: on
diagnostics.brick-log-level: DEBUG
diagnostics.brick-sys-log-level: INFO
diagnostics.client-log-level: DEBUG
[root at mdskvm-p01 glusterfs]# gluster volume status
Status of volume: mdsgv01
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/g
lusterv01 49152 0 Y
24487
NFS Server on localhost N/A N/A N N/A
Self-heal Daemon on localhost N/A N/A Y
24515
Task Status of Volume mdsgv01
------------------------------------------------------------------------------
There are no active volume tasks
[root at mdskvm-p01 glusterfs]# cat /etc/glusterfs/glusterd.vol
volume management
type mgmt/glusterd
option working-directory /var/lib/glusterd
option transport-type socket,rdma
option transport.socket.keepalive-time 10
option transport.socket.keepalive-interval 2
option transport.socket.read-fail-log off
option ping-timeout 0
option event-threads 1
option rpc-auth-allow-insecure on
option cluster.server-quorum-type none
option cluster.quorum-type none
# option cluster.server-quorum-type server
# option cluster.quorum-type auto
option server.event-threads 8
option client.event-threads 8
option performance.write-behind-window-size 8MB
option performance.io-thread-count 16
option performance.cache-size 1GB
option nfs.trusted-sync on
option storage.owner-uid 36
option storage.owner-uid 36
option cluster.data-self-heal-algorithm full
option performance.low-prio-threads 32
option features.shard-block-size 512MB
option features.shard on
option transport.socket.listen-port 24007
end-volume
[root at mdskvm-p01 glusterfs]#
Cheers,
TK
On 9/25/2019 7:05 AM, TomK wrote:
> Mind you, I just upgraded from 3.12 to 6.X.
>
> On 9/25/2019 6:56 AM, TomK wrote:
>>
>>
>> Brick log for specific gluster start command attempt (full log attached):
>>
>> [2019-09-25 10:53:37.847426] I [MSGID: 100030]
>> [glusterfsd.c:2847:main] 0-/usr/sbin/glusterfsd: Started running
>> /usr/sbin/glusterfsd version 6.5 (args: /usr/sbin/glusterfsd -s
>> mdskvm-p01.nix.mds.xyz --volfile-id
>> mdsgv01.mdskvm-p01.nix.mds.xyz.mnt-p01-d01-glusterv01 -p
>> /var/run/gluster/vols/mdsgv01/mdskvm-p01.nix.mds.xyz-mnt-p01-d01-glusterv01.pid
>> -S /var/run/gluster/defbdb699838d53b.socket --brick-name
>> /mnt/p01-d01/glusterv01 -l
>> /var/log/glusterfs/bricks/mnt-p01-d01-glusterv01.log --xlator-option
>> *-posix.glusterd-uuid=f7336db6-22b4-497d-8c2f-04c833a28546
>> --process-name brick --brick-port 49155 --xlator-option
>> mdsgv01-server.listen-port=49155)
>> [2019-09-25 10:53:37.848508] I [glusterfsd.c:2556:daemonize]
>> 0-glusterfs: Pid of current running process is 23133
>> [2019-09-25 10:53:37.858381] I [socket.c:902:__socket_server_bind]
>> 0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
>> [2019-09-25 10:53:37.865940] I [MSGID: 101190]
>> [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 0
>> [2019-09-25 10:53:37.866054] I
>> [glusterfsd-mgmt.c:2443:mgmt_rpc_notify] 0-glusterfsd-mgmt:
>> disconnected from remote-host: mdskvm-p01.nix.mds.xyz
>> [2019-09-25 10:53:37.866043] I [MSGID: 101190]
>> [event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started
>> thread with index 1
>> [2019-09-25 10:53:37.866083] I
>> [glusterfsd-mgmt.c:2463:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted
>> all volfile servers
>> [2019-09-25 10:53:37.866454] W [glusterfsd.c:1570:cleanup_and_exit]
>> (-->/lib64/libgfrpc.so.0(+0xf1d3) [0x7f9680ee91d3]
>> -->/usr/sbin/glusterfsd(+0x12fef) [0x55ca25710fef]
>> -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55ca2570901b] ) 0-:
>> received signum (1), shutting down
>> [2019-09-25 10:53:37.872399] I
>> [socket.c:3754:socket_submit_outgoing_msg] 0-glusterfs: not connected
>> (priv->connected = 0)
>> [2019-09-25 10:53:37.872445] W [rpc-clnt.c:1704:rpc_clnt_submit]
>> 0-glusterfs: failed to submit rpc-request (unique: 0, XID: 0x2
>> Program: Gluster Portmap, ProgVers: 1, Proc: 5) to rpc-transport
>> (glusterfs)
>> [2019-09-25 10:53:37.872534] W [glusterfsd.c:1570:cleanup_and_exit]
>> (-->/lib64/libgfrpc.so.0(+0xf1d3) [0x7f9680ee91d3]
>> -->/usr/sbin/glusterfsd(+0x12fef) [0x55ca25710fef]
>> -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55ca2570901b] ) 0-:
>> received signum (1), shutting down
>>
>>
>>
>>
>>
>> On 9/25/2019 6:48 AM, TomK wrote:
>>> Attached.
>>>
>>>
>>> On 9/25/2019 5:08 AM, Sanju Rakonde wrote:
>>>> Hi, The below errors indicate that brick process is failed to start.
>>>> Please attach brick log.
>>>>
>>>> [glusterd-utils.c:6312:glusterd_brick_start] 0-management: starting a
>>>> fresh brick process for brick /mnt/p01-d01/glusterv01
>>>> [2019-09-25 05:17:26.722717] E [MSGID: 106005]
>>>> [glusterd-utils.c:6317:glusterd_brick_start] 0-management: Unable to
>>>> start brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
>>>> [2019-09-25 05:17:26.722960] D [MSGID: 0]
>>>> [glusterd-utils.c:6327:glusterd_brick_start] 0-management: returning
>>>> -107
>>>> [2019-09-25 05:17:26.723006] E [MSGID: 106122]
>>>> [glusterd-mgmt.c:341:gd_mgmt_v3_commit_fn] 0-management: Volume start
>>>> commit failed.
>>>>
>>>> On Wed, Sep 25, 2019 at 11:00 AM TomK <tomkcpr at mdevsys.com
>>>> <mailto:tomkcpr at mdevsys.com>> wrote:
>>>>
>>>> Hey All,
>>>>
>>>> I'm getting the below error when trying to start a 2 node Gluster
>>>> cluster.
>>>>
>>>> I had the quorum enabled when I was at version 3.12 . However with
>>>> this
>>>> version it needed the quorum disabled. So I did so however now
>>>> see the
>>>> subject error.
>>>>
>>>> Any ideas what I could try next?
>>>>
>>>> -- Thx,
>>>> TK.
>>>>
>>>>
>>>> [2019-09-25 05:17:26.615203] D [MSGID: 0]
>>>> [glusterd-utils.c:1136:glusterd_resolve_brick] 0-management:
>>>> Returning 0
>>>> [2019-09-25 05:17:26.615555] D [MSGID: 0]
>>>> [glusterd-mgmt.c:243:gd_mgmt_v3_pre_validate_fn] 0-management:
>>>> OP = 5.
>>>> Returning 0
>>>> [2019-09-25 05:17:26.616271] D [MSGID: 0]
>>>> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
>>>> mdsgv01 found
>>>> [2019-09-25 05:17:26.616305] D [MSGID: 0]
>>>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management:
>>>> Returning 0
>>>> [2019-09-25 05:17:26.616327] D [MSGID: 0]
>>>> [glusterd-utils.c:6327:glusterd_brick_start] 0-management:
>>>> returning 0
>>>> [2019-09-25 05:17:26.617056] I
>>>> [glusterd-utils.c:6312:glusterd_brick_start] 0-management:
>>>> starting a
>>>> fresh brick process for brick /mnt/p01-d01/glusterv01
>>>> [2019-09-25 05:17:26.722717] E [MSGID: 106005]
>>>> [glusterd-utils.c:6317:glusterd_brick_start] 0-management:
>>>> Unable to
>>>> start brick mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
>>>> [2019-09-25 05:17:26.722960] D [MSGID: 0]
>>>> [glusterd-utils.c:6327:glusterd_brick_start] 0-management:
>>>> returning
>>>> -107
>>>> [2019-09-25 05:17:26.723006] E [MSGID: 106122]
>>>> [glusterd-mgmt.c:341:gd_mgmt_v3_commit_fn] 0-management: Volume
>>>> start
>>>> commit failed.
>>>> [2019-09-25 05:17:26.723027] D [MSGID: 0]
>>>> [glusterd-mgmt.c:444:gd_mgmt_v3_commit_fn] 0-management: OP = 5.
>>>> Returning -107
>>>> [2019-09-25 05:17:26.723045] E [MSGID: 106122]
>>>> [glusterd-mgmt.c:1696:glusterd_mgmt_v3_commit] 0-management: Commit
>>>> failed for operation Start on local node
>>>> [2019-09-25 05:17:26.723073] D [MSGID: 0]
>>>> [glusterd-op-sm.c:5106:glusterd_op_modify_op_ctx] 0-management:
>>>> op_ctx
>>>> modification not required
>>>> [2019-09-25 05:17:26.723141] E [MSGID: 106122]
>>>> [glusterd-mgmt.c:2466:glusterd_mgmt_v3_initiate_all_phases]
>>>> 0-management: Commit Op Failed
>>>> [2019-09-25 05:17:26.723204] D [MSGID: 0]
>>>> [glusterd-locks.c:797:glusterd_mgmt_v3_unlock] 0-management:
>>>> Trying to
>>>> release lock of vol mdsgv01 for
>>>> f7336db6-22b4-497d-8c2f-04c833a28546 as
>>>> mdsgv01_vol
>>>> [2019-09-25 05:17:26.723239] D [MSGID: 0]
>>>> [glusterd-locks.c:846:glusterd_mgmt_v3_unlock] 0-management:
>>>> Lock for
>>>> vol mdsgv01 successfully released
>>>> [2019-09-25 05:17:26.723273] D [MSGID: 0]
>>>> [glusterd-utils.c:1767:glusterd_volinfo_find] 0-management: Volume
>>>> mdsgv01 found
>>>> [2019-09-25 05:17:26.723326] D [MSGID: 0]
>>>> [glusterd-utils.c:1774:glusterd_volinfo_find] 0-management:
>>>> Returning 0
>>>> [2019-09-25 05:17:26.723360] D [MSGID: 0]
>>>> [glusterd-locks.c:464:glusterd_multiple_mgmt_v3_unlock]
>>>> 0-management:
>>>> Returning 0
>>>>
>>>> ==> /var/log/glusterfs/cmd_history.log <==
>>>> [2019-09-25 05:17:26.723390] : volume start mdsgv01 : FAILED :
>>>> Commit
>>>> failed on localhost. Please check log file for details.
>>>>
>>>> ==> /var/log/glusterfs/glusterd.log <==
>>>> [2019-09-25 05:17:26.723479] D [MSGID: 0]
>>>> [glusterd-rpc-ops.c:199:glusterd_op_send_cli_response]
>>>> 0-management:
>>>> Returning 0
>>>>
>>>>
>>>>
>>>> [root at mdskvm-p01 glusterfs]# cat /etc/glusterfs/glusterd.vol
>>>> volume management
>>>> type mgmt/glusterd
>>>> option working-directory /var/lib/glusterd
>>>> option transport-type socket,rdma
>>>> option transport.socket.keepalive-time 10
>>>> option transport.socket.keepalive-interval 2
>>>> option transport.socket.read-fail-log off
>>>> option ping-timeout 0
>>>> option event-threads 1
>>>> option rpc-auth-allow-insecure on
>>>> # option cluster.server-quorum-type server
>>>> # option cluster.quorum-type auto
>>>> option server.event-threads 8
>>>> option client.event-threads 8
>>>> option performance.write-behind-window-size 8MB
>>>> option performance.io-thread-count 16
>>>> option performance.cache-size 1GB
>>>> option nfs.trusted-sync on
>>>> option storage.owner-uid 36
>>>> option storage.owner-uid 36
>>>> option cluster.data-self-heal-algorithm full
>>>> option performance.low-prio-threads 32
>>>> option features.shard-block-size 512MB
>>>> option features.shard on
>>>> end-volume
>>>> [root at mdskvm-p01 glusterfs]#
>>>>
>>>>
>>>> [root at mdskvm-p01 glusterfs]# gluster volume info
>>>>
>>>> Volume Name: mdsgv01
>>>> Type: Replicate
>>>> Volume ID: f5b57076-dbd4-4d77-ae13-c1f3ee3adbe0
>>>> Status: Stopped
>>>> Snapshot Count: 0
>>>> Number of Bricks: 1 x 2 = 2
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: mdskvm-p02.nix.mds.xyz:/mnt/p02-d01/glusterv02
>>>> Brick2: mdskvm-p01.nix.mds.xyz:/mnt/p01-d01/glusterv01
>>>> Options Reconfigured:
>>>> storage.owner-gid: 36
>>>> cluster.data-self-heal-algorithm: full
>>>> performance.low-prio-threads: 32
>>>> features.shard-block-size: 512MB
>>>> features.shard: on
>>>> storage.owner-uid: 36
>>>> cluster.server-quorum-type: none
>>>> cluster.quorum-type: none
>>>> server.event-threads: 8
>>>> client.event-threads: 8
>>>> performance.write-behind-window-size: 8MB
>>>> performance.io-thread-count: 16
>>>> performance.cache-size: 1GB
>>>> nfs.trusted-sync: on
>>>> server.allow-insecure: on
>>>> performance.readdir-ahead: on
>>>> diagnostics.brick-log-level: DEBUG
>>>> diagnostics.brick-sys-log-level: INFO
>>>> diagnostics.client-log-level: DEBUG
>>>> [root at mdskvm-p01 glusterfs]#
>>>>
>>>>
>>>> _______________________________________________
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> APAC Schedule -
>>>> Every 2nd and 4th Tuesday at 11:30 AM IST
>>>> Bridge: https://bluejeans.com/118564314
>>>>
>>>> NA/EMEA Schedule -
>>>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>>>> Bridge: https://bluejeans.com/118564314
>>>>
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>>>
>>>> --
>>>> Thanks,
>>>> Sanju
>>>
>>>
>>>
>>> _______________________________________________
>>>
>>> Community Meeting Calendar:
>>>
>>> APAC Schedule -
>>> Every 2nd and 4th Tuesday at 11:30 AM IST
>>> Bridge: https://bluejeans.com/118564314
>>>
>>> NA/EMEA Schedule -
>>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>>> Bridge: https://bluejeans.com/118564314
>>>
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>
>>
>>
>> _______________________________________________
>>
>> Community Meeting Calendar:
>>
>> APAC Schedule -
>> Every 2nd and 4th Tuesday at 11:30 AM IST
>> Bridge: https://bluejeans.com/118564314
>>
>> NA/EMEA Schedule -
>> Every 1st and 3rd Tuesday at 01:00 PM EDT
>> Bridge: https://bluejeans.com/118564314
>>
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>
>
--
Thx,
TK.
More information about the Gluster-devel
mailing list