[Gluster-devel] HA, GlusterFS server protocol and LDirectorD

[Geoff Kassel]

Hi all,
   Just thought I'd reply to myself, since I've worked out something of a 
solution to my GlusterFS HA issues.

   The solution turned out to be abandoning Heartbeat and LDirectorD, and 
using Keepalived with a few heartbeat-like scripts. This gives server 
load-balancing, two-second failover and some limited recovery ability, 
without filling up log files with peer EOF messages. 

   While the two-second failover is not friendly to already running processes 
requiring continuous filesystem access (i.e. find, du, copy operations, 
databases etc), this should be reasonably adequate for use with web and mail 
servers where a few seconds downtime should not be of much issue. (At least, 
I hope so, since I'll be deploying web and mail services based around this 
configuration shortly. Wish me luck.)

   For the curious, I've attached the associated Keepalived configuration file 
and Perl scripts. The exact same Keepalived configuration should be in use 
across all hosts running GlusterFS servers. To use this with GlusterFS 
clients, just set the remote host in the GlusterFS client translator to the 
virtual server IP address you're using in keepalived.conf.

   Scripts and configuration files are released under the LGPL and the FDL, 
respectively. Enjoy!

Kind regards,

Geoff Kassel.

On Tue, 11 Sep 2007, Geoff Kassel wrote:
> Hi all,
>    I'm trying to set up LDirectorD (through Heartbeat) to load-balance and
> failover client connections to GlusterFS server instances over TCP.
>
>    First of all, I'm curious to find out if anyone else has attempted this,
> as I've had no luck with maintaining client continuity with round-robin DNS
> in /etc/hosts and client timeouts, as advised in previous posts and
> tutorials. The clients just go dead with 'Transport endpoint is not
> connected' messages.
>
>    My main problem is that LDirectorD doesn't seem to recognize that a
> GlusterFS server is functional through the connection test method, so I
> can't detect if a server goes down. While LDirectorD does a
> request-response method of liveness detection, the GlusterFS protocol is
> unfortunately too lengthy to use in the configuration files. (It needs to
> be a request that can fit on a single line, it seems.)
>
>    I'm wondering if there's a simple request-response connection test I
> haven't found yet that I can use to check for liveness of a server over
> TCP. If there isn't... could I make a feature request for such? Anything
> that can be done manually over a telnet connection to the port would be
> perfect.
>
>    Thank you for GlusterFS, and thanks in advance for your time and effort
> in answering my question.
>
> Kind regards,
>
> Geoff Kassel.
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel


-------------- next part --------------
! Configuration File for keepalived

# Distributed storage - GlusterFS

vrrp_instance INT_GLUSTERFSD {
    interface eth0
    state MASTER
    virtual_router_id 1
    priority 10
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass *secret*
    }
    virtual_ipaddress {
        192.168.0.3
    }
}

virtual_server 192.168.0.3 6996 {
    delay_loop 1
    lb_algo wrr
    lb_kind DR
    persistence_granularity 255.255.255.255
    protocol TCP

    real_server 127.0.0.1 6996 {
        weight 10
        notify_down "/etc/init.d/glusterfsd stop ; /etc/init.d/glusterfsd start"
        MISC_CHECK {
            misc_path "/scripts/check-glusterfsd.pl 127.0.0.1 6996 1"
        }
    }

    real_server 192.168.0.1 6996 {
        weight 1
        MISC_CHECK {
            misc_path "/scripts/check-glusterfsd.pl 192.168.0.1 6996 1"
        }
    }

    real_server 192.168.0.2 6996 {
        weight 1
        MISC_CHECK {
            misc_path "/scripts/check-glusterfsd.pl 192.168.0.2 6996 1"
        }
    }
}