[Gluster-users] Broken after 3.7.8 upgrade from 3.7.6
Anuradha Talur
atalur at redhat.com
Wed Mar 2 09:17:35 UTC 2016
----- Original Message -----
> From: "Alan Millar" <grunthos503 at yahoo.com>
> To: "Anuradha Talur" <atalur at redhat.com>
> Cc: gluster-users at gluster.org
> Sent: Wednesday, March 2, 2016 2:00:49 AM
> Subject: Re: [Gluster-users] Broken after 3.7.8 upgrade from 3.7.6
>
> >> I’m still trying to figure out why the self-heal-daemon doesn’t seem to be
> >> working, and what “unable to get index-dir” means. Any advice on what to
> >> look at would be appreciated. Thanks!
>
>
> >At any point did you have one node with 3.7.6 and another in 3.7.8 version?
>
> Yes. I upgraded each server in turn by stopping all gluster server and
> client processes on a machine, installing the update, and restarting
> everything on that machine. Then I waited for "heal info" on all volumes to
> say it had nothing to work on before proceeding to the next server.
>
Thanks for the information.
The "unable to get index-dir on .." messages you saw in log are not harmful in this scenario.
A simple explanation : when you have 1 new node and 2 old nodes, the self-heal-deamon and
heal commands run on the new node are expecting that the index-dir
"<brickpath>/.glusterfs/xattrop/dirty" exists, to process entries from it.
But the mentioned directory doesn't exist on the old bricks. (it was introduced as part of
3.7.7). As a result you see these logs.
But this doesn't explain why heal didn't happen on replacing the brick. :-/
After replacing, did you check volume heal info to know the status of heal?
>
> >I couldn't find information about the setup (number of nodes, vol info etc).
>
>
> I wasn't sure how much to spam the list :-)
>
> There are 3 servers in the gluster pool.
>
>
> amillar at chunxy:~$ sudo gluster pool list
> UUID Hostname State
> c1531143-229e-44a8-9fbb-089769ec999d pve4 Connected
> e82b0dc0-a490-47e4-bd11-6dcfcd36fd62 pve3 Connected
> 0422e779-7219-4392-ad8d-6263af4372fa localhost Connected
>
>
>
> Example volume:
>
>
>
> amillar at chunxy:~$ sudo gluster vol info public
>
> Volume Name: public
> Type: Replicate
> Volume ID: f75bbc7b-5db7-4497-b14c-c0433a84bcd9
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: chunxy:/data/glfs23/public
> Brick2: pve4:/data/glfs240/public
> Brick3: pve3:/data/glfs830/public
> Options Reconfigured:
> cluster.metadata-self-heal: on
> cluster.data-self-heal: on
> cluster.entry-self-heal: on
> cluster.self-heal-daemon: enable
> performance.readdir-ahead: on
>
>
>
> amillar at chunxy:~$ sudo gluster vol status public
> Status of volume: public
> Gluster process TCP Port RDMA Port Online Pid
> ------------------------------------------------------------------------------
> Brick chunxy:/data/glfs23/public 49184 0 Y
> 13119
> Brick pve4:/data/glfs240/public 49168 0 Y 837
> Brick pve3:/data/glfs830/public 49167 0 Y
> 21074
> NFS Server on localhost 2049 0 Y
> 10237
> Self-heal Daemon on localhost N/A N/A Y
> 10243
> NFS Server on pve3 2049 0 Y
> 30177
> Self-heal Daemon on pve3 N/A N/A Y
> 30183
> NFS Server on pve4 2049 0 Y
> 13070
> Self-heal Daemon on pve4 N/A N/A Y
> 13069
>
> Task Status of Volume public
> ------------------------------------------------------------------------------
> There are no active volume tasks
>
>
>
> amillar at chunxy:~$ sudo gluster vol heal public info
> Brick chunxy:/data/glfs23/public
> Number of entries: 0
>
> Brick pve4:/data/glfs240/public
> Number of entries: 0
>
> Brick pve3:/data/glfs830/public
> Number of entries: 0
>
>
>
> amillar at chunxy:~$ sudo grep public /var/log/glusterfs/glustershd.log |tail
> -10
> [2016-03-01 18:38:41.006543] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 18:48:42.001580] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 18:58:43.001631] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:08:43.002149] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:18:44.001444] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:28:44.001444] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:38:44.001445] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:48:44.001555] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 19:58:44.001405] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
> [2016-03-01 20:08:44.001444] W [MSGID: 108034]
> [afr-self-heald.c:445:afr_shd_index_sweep] 0-public-replicate-0: unable to
> get index-dir on public-client-1
>
>
>
> in glustershd.log:
> 39: volume public-client-1
> 40: type protocol/client
> 41: option clnt-lk-version 1
> 42: option volfile-checksum 0
> 43: option volfile-key gluster/glustershd
> 44: option client-version 3.7.8
> 45: option process-uuid
> chunxy-10236-2016/03/01-15:38:17:614674-public-client-1-0-0
> 46: option fops-version 1298437
> 47: option ping-timeout 42
> 48: option remote-host chunxy
> 49: option remote-subvolume /data/glfs23/public
> 50: option transport-type socket
> 51: option username 1aafe2d7-79ea-46b8-9d3e-XXXXXXXXXXXX
> 52: option password 7aaa0ce0-2a56-45e1-bc0e-YYYYYYYYYYYY
> 53: end-volume
>
>
>
> What other information would help? Thanks!
>
--
Thanks,
Anuradha.
More information about the Gluster-users
mailing list