[Gluster-users] WAN Challenge

Burnash, James jburnash at knight.com
Fri Apr 9 17:17:48 UTC 2010


I'm no expert on this - more like a somewhat informed amateur - but your configuration sounds perhaps more suitable for a grid architecture using each location as a node in the grid. Grids often communicate via WANs like you have, and are tolerant of the slow bandwidth and possible loss of connections, which is not a strength of any parallel file systems that I know of. The disadvantage of the grid architecture is that it is not real-time. I guess your solution will depend on what latency (measure in seconds to minutes) you can accommodate in syncing the data.

More info and leads can be found here: http://www.isgtw.org/?pid=1002049


-----Original Message-----
From: gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] On Behalf Of Count Zero
Sent: Friday, April 09, 2010 12:56 PM
To: Tejas N. Bhise
Subject: [Gluster-users] WAN Challenge

Hi Guys, I've sent this once but I did not even get it myself from the mailing list so i'm not sure it was even received correctly so I am re-posting. My apologies if this is a re-post.

I have an interesting situation, and I'm wondering if there's a solution for it in the glusterfs realm or if I will have to resort to other solutions that complement glusterfs (such as rsync or unison).

I have 9 servers in 3 locations on the internet (3 servers per location). Unfortunately, the network distance between them is such that setting up a Distribute or NUFA cluster between them all is difficult (I'm not saying impossible, because it may be possible and I just don't know how to pull it off).

There are 3 servers in each data center, and they are all clustered via NUFA:

-+ NUFA-Cluster
---+ SRV-A1
---+ SRV-A2
---+ SRV-A3

DC-B ( >> rsync from A)
-+ NUFA-Cluster
---+ SRV-B1
---+ SRV-B2
---+ SRV-B3

DC-C ( >> rsync from B)
-+ NUFA-Cluster
---+ SRV-C1
---+ SRV-C2
---+ SRV-C3

The reason I did it like this, so far:

1) I needed file reads to be fast on each local node, so I have the "option local-volume-name `hostname`" trick in my glusterfs.vol file (like in the cookbook).

2) Bandwidth between DC-A and DC-B and DC-C is kinda low... and since glusterfs waits for the last server to finish, this severely slows down the entire cluster for any operation, including just listing the files in a directory.

Is there a better way to implement this? All the examples I find are about 4 node replication, etc.

What about inter-continent replication of data between NUFA Clusters?
Any advice would be greatly appreciated :-)

At the moment, out of lack of options, I plan to sync between the 3 NUFA clusters with "INOSYNC".

Count Zero

P.S. Below is my configuration file, from /etc/glusterfs/glusterfs.vol:


volume posix
type storage/posix
option directory /data/export

volume locks
type features/locks
subvolumes posix

volume brick
type performance/io-threads
subvolumes locks

volume server
type protocol/server
option transport-type tcp
option auth.addr.brick.allow *
subvolumes brick

volume srv-a1
type protocol/client
option transport-type tcp
option remote-host srv-a1
option remote-subvolume brick

volume srv-a2
type protocol/client
option transport-type tcp
option remote-host srv-a2
option remote-subvolume brick

volume srv-a3
type protocol/client
option transport-type tcp
option remote-host srv-a3
option remote-subvolume brick

volume nufa
type cluster/nufa
option local-volume-name `hostname`
subvolumes srv-a1 srv-a2 srv-a3

volume writebehind
type performance/write-behind
option cache-size 1MB
subvolumes nufa

volume cache
type performance/io-cache
option cache-size 512MB
subvolumes writebehind

Gluster-users mailing list
Gluster-users at gluster.org

This e-mail, and any attachments thereto, is intended only for use by the addressee(s) named herein and may contain legally privileged and/or confidential information. If you are not the intended recipient of this e-mail, you are hereby notified that any dissemination, distribution or copying of this e-mail, and any attachments thereto, is strictly prohibited. If you have received this in error, please immediately notify me and permanently delete the original and any copy of any e-mail and any printout thereof. E-mail transmission cannot be guaranteed to be secure or error-free. The sender therefore does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission.
NOTICE REGARDING PRIVACY AND CONFIDENTIALITY Knight Capital Group may, at its discretion, monitor and review the content of all e-mail communications. http://www.knight.com

More information about the Gluster-users mailing list