[Gluster-users] Inter-Continent Challenge

[Count Zero]

Hi Guys,

I have an interesting situation, and I'm wondering if there's a solution for it in the glusterfs realm or if I will have to resort to other solutions that complement glusterfs (such as rsync or unison).

I have 9 servers in 3 locations on the internet (3 servers per location). Unfortunately, the network distance between them is such that setting up a Distribute or NUFA cluster between them all is difficult (I'm not saying impossible, because it may be possible and I just don't know how to pull it off).

There are 3 servers in each data center, and they are all clustered via NUFA:

DC-A
-+ NUFA-Cluster
---+ SRV-A1
---+ SRV-A2
---+ SRV-A3

DC-B ( >> rsync from A)
-+ NUFA-Cluster
---+ SRV-B1
---+ SRV-B2
---+ SRV-B3

DC-C ( >> rsync from B)
-+ NUFA-Cluster
---+ SRV-C1
---+ SRV-C2
---+ SRV-C3

The reason I did it like this, so far:

1) I needed file reads to be fast on each local node, so I have the "option local-volume-name `hostname`" trick in my glusterfs.vol file (like in the cookbook).

2) Bandwidth between DC-A and DC-B and DC-C is kinda low... and since glusterfs waits for the last server to finish, this severely slows down the entire cluster for any operation, including just listing the files in a directory.


Question 1:

Is there a better way to implement this? All the examples I find are about 4 node replication, etc.
What about inter-continent replication of data between NUFA Clusters?
Any advice would be greatly appreciated :-)
At the moment, out of lack of options, I plan to sync between the 3 NUFA clusters with Unison or Rsync.


Question 2:

Sometimes when I mount this cluster, I get two sets of dot files, like so:

root at srv-a1:/mnt/gfs# ls -la
total 188
drwxr-xr-x 12 root        root        12288 2010-04-08 06:45 .
drwxr-xr-x 12 root        root        12288 2010-04-08 06:45 .
drwxr-xr-x  4 root        root         4096 2010-04-08 06:14 ..
drwxr-xr-x  4 root        root         4096 2010-04-08 06:14 ..

A reboot of the machine fixes it... but I'm wondering why this happens and how to avoid it?


Thanks,
Count Zero

P.S. Below is my configuration file, from /etc/glusterfs/glusterfs.vol:

---------------------8<--------------------8<------------------

volume posix
 type storage/posix
 option directory /data/export
end-volume

volume locks
  type features/locks
  subvolumes posix
end-volume

volume brick
 type performance/io-threads
 subvolumes locks
end-volume

volume server
 type protocol/server
 option transport-type tcp
 option auth.addr.brick.allow *
 subvolumes brick
end-volume

volume srv-a1
 type protocol/client
 option transport-type tcp
 option remote-host srv-a1
 option remote-subvolume brick
end-volume

volume srv-a2
 type protocol/client
 option transport-type tcp
 option remote-host srv-a2
 option remote-subvolume brick
end-volume

volume srv-a3
 type protocol/client
 option transport-type tcp
 option remote-host srv-a3
 option remote-subvolume brick
end-volume

volume nufa
 type cluster/nufa
 option local-volume-name `hostname`
 subvolumes srv-a1 srv-a2 srv-a3
end-volume

volume writebehind
  type performance/write-behind
  option cache-size 1MB
  subvolumes nufa
end-volume

volume cache
  type performance/io-cache
  option cache-size 512MB
  subvolumes writebehind
end-volume

--------------------->8-------------------->8------------------