[Gluster-users] Rebalance times in 3.2.5 vs 3.4.2
Viktor Villafuerte
viktor.villafuerte at optusnet.com.au
Fri Feb 28 03:56:37 UTC 2014
I just got this error
[2014-02-28 03:11:26.311077] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
after I umount-ed and stopped glusterd on a replicated setup. 2 bricks
replicated, one server umount the volume (still ok), 'gluster volume
stop' (still ok), after 'service glusterd stop' this error comes up on the
other brick (server).
complete log:
[root at gluster07.uat ~]# service glusterd stop [ OK ]
==> log on gluster08.uat
==> /var/log/glusterfs/etc-glusterfs-glusterd.vol.log <==
[2014-02-28 03:14:11.416945] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[2014-02-28 03:14:11.417160] W
[socket.c:1962:__socket_proto_state_machine] 0-management: reading from
socket failed. Error (No data available), peer (10.116.126.31:24007)
[2014-02-28 03:14:12.989556] E [socket.c:2157:socket_connect_finish]
0-management: connection to 10.116.126.31:24007 failed (Connection
refused)
[2014-02-28 03:14:12.989702] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[2014-02-28 03:14:15.003420] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[2014-02-28 03:14:18.020311] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[2014-02-28 03:14:21.036216] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[root at gluster07.uat ~]# service glusterd start
Starting glusterd: [ OK ]
==> log on gluster08.uat (continue)
[2014-02-28 03:15:54.595801] W [socket.c:514:__socket_rwv] 0-management:
readv failed (No data available)
[2014-02-28 03:15:56.140337] I
[glusterd-handshake.c:557:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 2
[2014-02-28 03:15:56.170983] I
[glusterd-handler.c:1956:__glusterd_handle_incoming_friend_req]
0-glusterd: Received probe from uuid:
79a76cb4-1163-464f-84c4-19f2a39deee9
[2014-02-28 03:15:57.613644] I
[glusterd-handler.c:2987:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to 10.116.126.32 (0), ret: 0
[2014-02-28 03:15:57.635688] I
[glusterd-sm.c:494:glusterd_ac_send_friend_update] 0-: Added uuid:
79a76cb4-1163-464f-84c4-19f2a39deee9, host: gluster07.uat
[2014-02-28 03:15:57.656542] I
[glusterd-rpc-ops.c:542:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 79a76cb4-1163-464f-84c4-19f2a39deee9
[2014-02-28 03:15:57.687594] I
[glusterd-rpc-ops.c:345:__glusterd_friend_add_cbk] 0-glusterd: Received
ACC from uuid: 79a76cb4-1163-464f-84c4-19f2a39deee9, host:
gluster07.uat, port: 0
[2014-02-28 03:15:57.712743] I
[glusterd-handler.c:2118:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid: 79a76cb4-1163-464f-84c4-19f2a39deee9
[2014-02-28 03:15:57.713470] I
[glusterd-handler.c:2163:__glusterd_handle_friend_update] 0-: Received
uuid: c1d10b71-d118-4f4a-adc2-3cfbea13fd54, hostname:10.116.126.32
[2014-02-28 03:15:57.713564] I
[glusterd-handler.c:2172:__glusterd_handle_friend_update] 0-: Received
my uuid as Friend
no error after this point as long as glusterd is running on both bricks.
Even when everything else is stopped..
On Thu 27 Feb 2014 21:54:46, Matt Edwards wrote:
> Hi Viktor,
>
> Thanks for the tips.
Everything I say here is more of a comment then a 'tip' :) as I'm still
learning pretty much everything about Gluster.
> I'm a bit confused, since the clients mount the share
> fine, and "gluster peer status" and "gluster volume status all detail" are
> happy.
Rebalance is more difficult for the bricks. I've had situation before
when I had files in '-T' state after rebalance completed (on the bricks).
This is clearly wrong to me, but the mount was ok. The file still existed
on the inital replica..
>
> What is the expected output of "rebalance status" for just a fix-layout
> run? I believe the last time I did that, the status was always 0s (which
> makes some sense, as files aren't moving) and the log was empty, but the
> operation seemed to complete successfully. Does a file rebalance first
> require a fix-layout operation internally, and is it possible that my
> volume is still in that phase? Or I making up an overly optimistic
> scenario?
I've just tried 'fix-layout' only and you're right the result is all
'0's. But the the status is 'completed' and 'success'
[root at gluster08.uat ~]# gluster volume rebalance cdn-uat status
Node Rebalanced-files size scanned failures skipped status run time in secs
--------- ----------- ----------- ----------- ----------- ----------- ------------ --------------
localhost 0 0Bytes 0 0 0 completed 18.00
gluster07.uat 0 0Bytes 0 0 0 completed 18.00
gluster01.uat 0 0Bytes 0 0 0 completed 19.00
gluster02.uat 0 0Bytes 0 0 0 completed 18.00
volume rebalance: cdn-uat: success:
[root at gluster08.uat ~]#
when you do rebalance it does 'fix-layout' first, no matter if you
did fix-layout explicitly before. I've never delt with 75T before
and so I cannot tell how long it should take, however, I recon that
4 days is too much. Fix-layout it much quicker then the actuall
rebalance of files.
v
>
> Thanks,
>
> Matt
>
>
> On Thu, Feb 27, 2014 at 8:33 PM, Viktor Villafuerte <
> viktor.villafuerte at optusnet.com.au> wrote:
>
> > Hi Matt,
> >
> > if the 'status' says 0 for everything that's not good. Normally when I
> > do rebalance the numbers should change (up). Also the rebalance log
> > should show files being moved around.
> >
> > For the errors - my (limited) experience with Gluster is that the 'W'
> > are normally harmless and they show up quite a bit. For the actuall
> > error 'E' you could try to play with 'auth.allow' as suggested here
> >
> > http://gluster.org/pipermail/gluster-users/2011-November/009094.html
> >
> >
> > Normally when rebalancing I do count of files on the bricks and the
> > Gluster mount to make sure they eventually add up. Also I grep and count
> > '-T' and see how the count goes down and 'rw' count goes up.
> >
> > v
> >
> >
> >
> >
> > On Thu 27 Feb 2014 00:57:28, Matt Edwards wrote:
> > > Hopefully I'm not derailing this thread too far, but I have a related
> > > rebalance progress/speed issue.
> > >
> > > I have a rebalance process started that's been running for 3-4 days. Is
> > > there a good way to see if it's running successfully, or might this be a
> > > sign of some problem?
> > >
> > > This is on a 4-node distribute setup with v3.4.2 and 45T of data.
> > >
> > > The *-rebalance.log has been silent since some informational messages
> > when
> > > the rebalance started. There were a few initial warnings and errors
> > that I
> > > observed, though:
> > >
> > >
> > > E [client-handshake.c:1397:client_setvolume_cbk] 0-cluster2-client-0:
> > > SETVOLUME on remote-host failed: Authentication failed
> > >
> > > W [client-handshake.c:1365:client_setvolume_cbk] 0-cluster2-client-4:
> > > failed to set the volume (Permission denied)
> > >
> > > W [client-handshake.c:1391:client_setvolume_cbk] 0-cluster2-client-4:
> > > failed to get 'process-uuid' from reply dict
> > >
> > > W [socket.c:514:__socket_rwv] 0-cluster2-client-3: readv failed (No data
> > > available)
> > >
> > >
> > > "gluster volume status" reports that the rebalance is in progress, the
> > > process listed in vols/<volname>/rebalance/<hash>.pid is still running on
> > > the server, but "gluster volume rebalance <volname> status" reports 0 for
> > > everything (files scanned or rebalanced, failures, run time).
> > >
> > > Thanks,
> > >
> > > Matt
> > >
> > >
> > > On Thu, Feb 27, 2014 at 12:39 AM, Shylesh Kumar <shmohan at redhat.com>
> > wrote:
> > >
> > > > Hi Viktor,
> > > >
> > > > Lots of optimizations and improvements went in for 3.4 so it should be
> > > > faster than 3.2.
> > > > Just to make sure what's happening could you please check rebalance
> > logs
> > > > which will be in
> > > > /var/log/glusterfs/<volname>-rebalance.log and check is there any
> > > > progress ?
> > > >
> > > > Thanks,
> > > > Shylesh
> > > >
> > > >
> > > > Viktor Villafuerte wrote:
> > > >
> > > >> Anybody can confirm/dispute that this is normal/abnormal?
> > > >>
> > > >> v
> > > >>
> > > >>
> > > >> On Tue 25 Feb 2014 15:21:40, Viktor Villafuerte wrote:
> > > >>
> > > >>> Hi all,
> > > >>>
> > > >>> I have distributed replicated set with 2 servers (replicas) and am
> > > >>> trying to add another set of replicas: 1 x (1x1) => 2 x (1x1)
> > > >>>
> > > >>> I have about 23G of data which I copy onto the first replica, check
> > > >>> everything and then add the other set of replicas and eventually
> > > >>> rebalance fix-layout, migrate-data.
> > > >>>
> > > >>> Now on
> > > >>>
> > > >>> Gluster v3.2.5 this took about 30 mins (to rebalance + migrate-data)
> > > >>>
> > > >>> on
> > > >>>
> > > >>> Gluster v3.4.2 this has been running for almost 4 hours and it's
> > still
> > > >>> not finished
> > > >>>
> > > >>>
> > > >>> As I may have to do this in production, where the amount of data is
> > > >>> significantly larger than 23G, I'm looking at about three weeks of
> > wait
> > > >>> to rebalance :)
> > > >>>
> > > >>> Now my question is if this is as it's meant to be? I can see that
> > v3.4.2
> > > >>> gives me more info about the rebalance process etc, but that surely
> > > >>> cannot justify the enormous time difference.
> > > >>>
> > > >>> Is this normal/expected behaviour? If so I will have to stick with
> > the
> > > >>> v3.2.5 as it seems way quicker.
> > > >>>
> > > >>> Please, let me know if there is any 'well known' option/way/secret to
> > > >>> speed the rebalance up on v3.4.2.
> > > >>>
> > > >>>
> > > >>> thanks
> > > >>>
> > > >>>
> > > >>>
> > > >>> --
> > > >>> Regards
> > > >>>
> > > >>> Viktor Villafuerte
> > > >>> Optus Internet Engineering
> > > >>> t: 02 808-25265
> > > >>> _______________________________________________
> > > >>> Gluster-users mailing list
> > > >>> Gluster-users at gluster.org
> > > >>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > > >>>
> > > >>
> > > > _______________________________________________
> > > > Gluster-users mailing list
> > > > Gluster-users at gluster.org
> > > > http://supercolony.gluster.org/mailman/listinfo/gluster-users
> > > >
> >
> > --
> > Regards
> >
> > Viktor Villafuerte
> > Optus Internet Engineering
> > t: 02 808-25265
> >
--
Regards
Viktor Villafuerte
Optus Internet Engineering
t: 02 808-25265
More information about the Gluster-users
mailing list