[Bugs] [Bug 1687051] gluster volume heal failed when online upgrading from 3.12 to 5.x and when rolling back online upgrade from 4.1.4 to 3.12.15
bugzilla at redhat.com
bugzilla at redhat.com
Tue Mar 12 14:32:31 UTC 2019
https://bugzilla.redhat.com/show_bug.cgi?id=1687051
Amgad <amgad.saleh at nokia.com> changed:
What |Removed |Added
----------------------------------------------------------------------------
Flags|needinfo?(srangana at redhat.c |
|om) |
|needinfo?(amgad.saleh at nokia |
|.com) |
--- Comment #8 from Amgad <amgad.saleh at nokia.com> ---
Case 1) online upgrade from 3.12.15 to 5.3
A) I have a cluster of 3 replicas: gfs-1, gfs-2, gfs-3new running 3.12.15. When
online upgraded gfs-1 from 3.12.15, here are the outputs:
(notice that bricks on gfs-1 are offline - both glusterd and glusterfsd are
active and running)
[root at gfs-1 ~]# gluster volume info
Volume Name: glustervol1
Type: Replicate
Volume ID: 28b16639-7c58-4f28-975b-5ea17274e87b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.76.153.206:/mnt/data1/1
Brick2: 10.76.153.213:/mnt/data1/1
Brick3: 10.76.153.207:/mnt/data1/1
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
Volume Name: glustervol2
Type: Replicate
Volume ID: 8637eee7-20b7-4a88-b497-192b4626093d
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.76.153.206:/mnt/data2/2
Brick2: 10.76.153.213:/mnt/data2/2
Brick3: 10.76.153.207:/mnt/data2/2
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
Volume Name: glustervol3
Type: Replicate
Volume ID: f8c21e8c-0a9a-40ba-b098-931a4219de0f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.76.153.206:/mnt/data3/3
Brick2: 10.76.153.213:/mnt/data3/3
Brick3: 10.76.153.207:/mnt/data3/3
Options Reconfigured:
performance.client-io-threads: off
nfs.disable: on
transport.address-family: inet
---
[root at gfs-1 ~]# gluster volume status
Status of volume: glustervol1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.76.153.206:/mnt/data1/1 N/A N/A N N/A
Brick 10.76.153.213:/mnt/data1/1 49152 0 Y 24733
Brick 10.76.153.207:/mnt/data1/1 49152 0 Y 7790
Self-heal Daemon on localhost N/A N/A Y 14928
Self-heal Daemon on 10.76.153.207 N/A N/A Y 7780
Self-heal Daemon on 10.76.153.213 N/A N/A Y 24723
Task Status of Volume glustervol1
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: glustervol2
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.76.153.206:/mnt/data2/2 N/A N/A N N/A
Brick 10.76.153.213:/mnt/data2/2 49153 0 Y 24742
Brick 10.76.153.207:/mnt/data2/2 49153 0 Y 7800
Self-heal Daemon on localhost N/A N/A Y 14928
Self-heal Daemon on 10.76.153.207 N/A N/A Y 7780
Self-heal Daemon on 10.76.153.213 N/A N/A Y 24723
Task Status of Volume glustervol2
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: glustervol3
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.76.153.206:/mnt/data3/3 N/A N/A N N/A
Brick 10.76.153.213:/mnt/data3/3 49154 0 Y 24751
Brick 10.76.153.207:/mnt/data3/3 49154 0 Y 7809
Self-heal Daemon on localhost N/A N/A Y 14928
Self-heal Daemon on 10.76.153.207 N/A N/A Y 7780
Self-heal Daemon on 10.76.153.213 N/A N/A Y 24723
Task Status of Volume glustervol3
------------------------------------------------------------------------------
There are no active volume tasks
[root at gfs-1 ~]#
======
Running "gluster volume heal" ==> unsuccessful
[root at gfs-1 ~]# for i in `gluster volume list`; do gluster volume heal $i; done
Launching heal operation to perform index self heal on volume glustervol1 has
been unsuccessful:
Glusterd Syncop Mgmt brick op 'Heal' failed. Please check glustershd log file
for details.
Launching heal operation to perform index self heal on volume glustervol2 has
been unsuccessful:
Glusterd Syncop Mgmt brick op 'Heal' failed. Please check glustershd log file
for details.
Launching heal operation to perform index self heal on volume glustervol3 has
been unsuccessful:
Glusterd Syncop Mgmt brick op 'Heal' failed. Please check glustershd log file
for details.
[root at gfs-1 ~]#
B) Reverting gfs-1 back to 3.12.15, bricks are on line and heal is successfull
[root at gfs-1 log]# gluster volume info
Volume Name: glustervol1
Type: Replicate
Volume ID: 28b16639-7c58-4f28-975b-5ea17274e87b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.76.153.206:/mnt/data1/1
Brick2: 10.76.153.213:/mnt/data1/1
Brick3: 10.76.153.207:/mnt/data1/1
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Volume Name: glustervol2
Type: Replicate
Volume ID: 8637eee7-20b7-4a88-b497-192b4626093d
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.76.153.206:/mnt/data2/2
Brick2: 10.76.153.213:/mnt/data2/2
Brick3: 10.76.153.207:/mnt/data2/2
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Volume Name: glustervol3
Type: Replicate
Volume ID: f8c21e8c-0a9a-40ba-b098-931a4219de0f
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: 10.76.153.206:/mnt/data3/3
Brick2: 10.76.153.213:/mnt/data3/3
Brick3: 10.76.153.207:/mnt/data3/3
Options Reconfigured:
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root at gfs-1 log]#
[root at gfs-1 log]# gluster volume status
Status of volume: glustervol1
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.76.153.206:/mnt/data1/1 49152 0 Y 16029
Brick 10.76.153.213:/mnt/data1/1 49152 0 Y 24733
Brick 10.76.153.207:/mnt/data1/1 49152 0 Y 7790
Self-heal Daemon on localhost N/A N/A Y 16019
Self-heal Daemon on 10.76.153.207 N/A N/A Y 7780
Self-heal Daemon on 10.76.153.213 N/A N/A Y 24723
Task Status of Volume glustervol1
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: glustervol2
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.76.153.206:/mnt/data2/2 49153 0 Y 16038
Brick 10.76.153.213:/mnt/data2/2 49153 0 Y 24742
Brick 10.76.153.207:/mnt/data2/2 49153 0 Y 7800
Self-heal Daemon on localhost N/A N/A Y 16019
Self-heal Daemon on 10.76.153.207 N/A N/A Y 7780
Self-heal Daemon on 10.76.153.213 N/A N/A Y 24723
Task Status of Volume glustervol2
------------------------------------------------------------------------------
There are no active volume tasks
Status of volume: glustervol3
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.76.153.206:/mnt/data3/3 49154 0 Y 16047
Brick 10.76.153.213:/mnt/data3/3 49154 0 Y 24751
Brick 10.76.153.207:/mnt/data3/3 49154 0 Y 7809
Self-heal Daemon on localhost N/A N/A Y 16019
Self-heal Daemon on 10.76.153.213 N/A N/A Y 24723
Self-heal Daemon on 10.76.153.207 N/A N/A Y 7780
Task Status of Volume glustervol3
------------------------------------------------------------------------------
There are no active volume tasks
[root at gfs-1 log]#
[root at gfs-1 log]# for i in `gluster volume list`; do gluster volume heal $i;
done
Launching heal operation to perform index self heal on volume glustervol1 has
been successful
Use heal info commands to check status.
Launching heal operation to perform index self heal on volume glustervol2 has
been successful
Use heal info commands to check status.
Launching heal operation to perform index self heal on volume glustervol3 has
been successful
Use heal info commands to check status.
[root at gfs-1 log]#
Uploading /var/log/glusterfs:
- when upgraded gfs-1 to 5.3: gfs-1-logs.tgz, gfs-2-logs.tgz, and
gfs-3new-logs.tgz
- when reverted back to 3.12.15: gfs-1-logs-3.12.15.tgz,
gfs-2-logs-3.12.15.tgz, and gfs-3new-logs-3.12.15.tgz
Next comment will have the 2nd case upgrade 3.12.15 -to- 4.1.4 and rollback
--
You are receiving this mail because:
You are on the CC list for the bug.
More information about the Bugs
mailing list