[Gluster-users] Gluster upgrade planning

[Dietmar Putz]

Hi Jamie,

we are running a historical grown distributed replicated gluster-volume 
on Ubuntu 14.04.4 LTS in a productive environment.
The current Volume size is 55 TB, about 44 TB in use, ~ 6.000.000 files 
in ~ 450.000 directories on 6 nodes.
The volume is geo-replicated to another distributed replicated 
gluster-volume with the same configuration as the master volume.

since update from 3.4.7 to 3.5.x to 3.6.x to 3.7.8 we had trouble with 
the new distributed geo-replication.
since update to 3.7.9 the behavior of the distributed geo-replication 
significantly improved and since update to 3.7.11 the geo-replication is 
working again as expected.

the crawl status reached the 'changelog crawl' some days ago and 
meanwhile i observed 2 failover from an active node to a passive node 
because of 'less' significant errors...but the geo-replication survived 
and never gets into faulty state.
i compared all files/gfid's between master and slave volume and came to 
the conclusion...yes, it's working.

btw....many thanks to the developers.


[ 11:55:10 ] - root at gluster-ger-ber-08 
/var/log/glusterfs/geo-replication/ger-ber-01 $gluster volume 
geo-replication ger-ber-01 gluster-wien-02::wien-01 status

MASTER NODE           MASTER VOL    MASTER BRICK SLAVE USER    
SLAVE                       SLAVE NODE STATUS     CRAWL STATUS       
LAST_SYNCED
------------------------------------------------------------------------------------------------------------------------------------------------------------------------
gluster-ger-ber-08    ger-ber-01    /gluster-export root          
gluster-wien-02::wien-01    gluster-wien-04-int Passive    
N/A                N/A
gluster-ger-ber-10    ger-ber-01    /gluster-export root          
gluster-wien-02::wien-01    gluster-wien-02-int Active     Changelog 
Crawl    2016-05-13 11:55:05
gluster-ger-ber-09    ger-ber-01    /gluster-export root          
gluster-wien-02::wien-01    gluster-wien-05-int Passive    
N/A                N/A
gluster-ger-ber-11    ger-ber-01    /gluster-export root          
gluster-wien-02::wien-01    gluster-wien-03-int Active     Changelog 
Crawl    2016-05-13 11:55:43
gluster-ger-ber-12    ger-ber-01    /gluster-export root          
gluster-wien-02::wien-01    gluster-wien-06-int Passive    
N/A                N/A
gluster-ger-ber-07    ger-ber-01    /gluster-export root          
gluster-wien-02::wien-01    gluster-wien-07-int Active     Changelog 
Crawl    2016-05-13 11:55:16
[ 11:56:01 ] - root at gluster-ger-ber-08 
/var/log/glusterfs/geo-replication/ger-ber-01 $


after a lot of updates on live systems and test systems i would strongly 
recommend to plan a short downtime during upgrade.
even i was able to update one gluster node after the other, lastly it 
was necessary to stop the gluster volume, restart all gluster processes 
and start the volume again.
as far as i know the official recommendation is to stop all gluster 
volume(s), stop / kill all gluster processes, update glusterfs and 
restart processes / volume(s)...this worked always without any trouble.

best regards
dietmar






On 11.05.2016 19:21, Jamie Lawrence wrote:
>> On May 10, 2016, at 3:27 PM, Lindsay Mathieson <lindsay.mathieson at gmail.com> wrote:
>>
>> On 11/05/2016 4:40 AM, Jamie Lawrence wrote:
>>>   - Is anyone currently running Gluster on Debian or Ubuntu in production? We would prefer to get off RHEL-flavored hosts (currently Centos 7), thus I’m hoping I’m wrong, but I’ve seen little evidence that trying this for production would be a safe endeavor.[1]
>> I'm running gluster 3.7.11 on our proxmox cluster, hosting VM's. Based on debian Jessie, no issues. gluster.org  always has uptodate debian packages.
> Thanks, Lindsay. Good to know.
>
> -j
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users

-- 
Dietmar Putz
3Q Medien GmbH
Wetzlarer Str. 86
D-14482 Potsdam
  
Telefax:  +49 (0)331 / 2797 866 - 1
Telefon:  +49 (0)331 / 2792 866 - 8
Mobile:   +49 171 / 90 160 39
Mail:     putz at 3qmedien.net