[Gluster-users] Switch recommendations

[Dan Bretherton]

Hello Harald,
Thanks for taking an interest.  Answers are in line.

On 01/31/2012 02:12 AM, Harald Stürzebecher wrote:
> Are there some network admins or someone with networking experience
> you could talk to?
Only in the university's over stretched central IT Services department.  
I daren't distract them from doing various other things I'm waiting for 
them to do for me.

> Do you use the management functions that your current switches support?
Not at the moment.  I bought managed switches in case I had to set up a 
VLAN, but that turned out not to be necessary.
> How many server and clients do you have now?.
The 8 compute nodes in my Rocks cluster are the important ones.

>
> Do you plan to increase the numbers in the near future?
I might be buying another 4 compute nodes some time this year

> In that case I'd suggest to get stackable switches for easier expansion.
>
I agree that stackable switches are a good idea even if there is no 
immediate need for extra switches.
> How are the servers and clients distributed to the switches now?
The clients are all connected to one switch but some of the servers are 
connected to switches in adjacent racks.
>
> How are the switches connected to each other?
Single cables.  There aren't enough spare ports to connect the other 
rack switches via a LAG, so I am planning to buy a 48-port switch and 
connect all the clients and servers to that.
>
> Can you tell where your bottleneck is?
> Is it the connection between your switches or is it something else?
I'm pretty sure it's the connection between the switches.  I tested the 
application that has been causing the most concern on a conventional NFS 
server, which was connected to the main Rocks compute cluster switch.  I 
then connected the NFS server to a different switch and the application 
ran 5-6 times more slowly, even more slowly that when its data is on a 
GlusterFS volume in fact (when it only runs 3 times more slowly than 
conventional NFS...).  I don't think all the applications that run on 
the cluster are affected so severely, but this is the one my boss has 
heard about...
>
> Could you plug all the servers and some of the clients into one switch
> and the rest of the clients into the other switch(es) for a short
> period of time? There should be a big difference in speed between the
> two groups of clients. How does that compare to the speed you have
> now?
Good idea; I'll give that a try.

> What happens if one switch fails? Can the remaining equipment do some
> useful work until a replacement arrives?
> Depending on the answers it might be better to have two or more
> smaller stackable switches instead of one big switch, even if the big
> one might be cheaper.
I hadn't thought about redundancy I must admit, but buying two stackable 
24 port switches instead of one 48 port switch is an interesting idea. I 
would have to connect one set of servers to one switch and the GlusterFS 
replica servers to the other.  The clients would be distributed across 
both switches, and half of them would be able to connect to half the 
servers in the event of a switch failure.  One thing that worries me 
about this scenario is the time it would take to self-heal all the 
volumes after running with the replica servers missing for a day or 
two.  In theory GlusterFS should be able to cope with this, but there is 
a possibility of a server failing during the mammoth self-heal, and if 
were one of the up-to-date servers that was connected to the live switch 
when the other switch failed then the users would find themselves 
looking at old data.  The only way to avoid this would be to have two 
stackable 48-port switches.  I think I'll have to put this on my "nice 
to have" list.
>
> I don't have much experience with network administration so I cannot
> recommend a brand or type of switch.
>
> I just looked at the Netgear website and googled for prices:
> Two Netgear GS724TS stackable switches seem to cost nearly the same as
> one GS748TS, both are supposed to have "a 20 Gbps, dual-ring, highly
> redundant stacking bus".
I'll have a look at those.
>
>
>
> Regards,
>
> Harald
Thanks again.
Regards,
Dan.