[Gluster-users] TCP port usage

[JF Le Fillâtre]

Hello,

I see the same things on my nodes:

stor104: /var/lib/glusterd/vols/live/bricks/stor104:-zfs-brick0-brick:listen-port=49170
stor104: /var/lib/glusterd/vols/live/bricks/stor104:-zfs-brick1-brick:listen-port=49171
stor104: /var/lib/glusterd/vols/live/bricks/stor104:-zfs-brick2-brick:listen-port=49172
stor104: /var/lib/glusterd/vols/live/bricks/stor106:-zfs-brick0-brick:listen-port=0
stor104: /var/lib/glusterd/vols/live/bricks/stor106:-zfs-brick1-brick:listen-port=0
stor104: /var/lib/glusterd/vols/live/bricks/stor106:-zfs-brick2-brick:listen-port=0

stor106: /var/lib/glusterd/vols/live/bricks/stor104:-zfs-brick0-brick:listen-port=0
stor106: /var/lib/glusterd/vols/live/bricks/stor104:-zfs-brick1-brick:listen-port=0
stor106: /var/lib/glusterd/vols/live/bricks/stor104:-zfs-brick2-brick:listen-port=0
stor106: /var/lib/glusterd/vols/live/bricks/stor106:-zfs-brick0-brick:listen-port=49162
stor106: /var/lib/glusterd/vols/live/bricks/stor106:-zfs-brick1-brick:listen-port=49163
stor106: /var/lib/glusterd/vols/live/bricks/stor106:-zfs-brick2-brick:listen-port=49164

So yes, on a given server you only see the ports for the bricks of that server. From that I can deduce that the glusterd server running on port 24007 provides the local port numbers to all other machines (servers and clients).

And it seems that brick port numbers are unique pool-wise, rather than incrementing from a same number on each machine.

Compare that with:
http://www.gluster.org/community/documentation/index.php/Basic_Gluster_Troubleshooting

It states:
"One TCP port for each brick in a volume. So, for example, if you have 4 bricks in a volume, port [...] 49152 - 49155 from GlusterFS 3.4 & later."

It seems that the starting port isn't set in stone, and that the uniqueness of the port numbers takes precedence over a linear port number sequence on a given server.

Thanks,
JF



On 19/03/15 09:10, Melkor Lord wrote:
> Hi,
> 
> To my understanding, Gluster starts volumes TCP services at port 49152
> and then increases the port number with every new volume right? I have a
> 3 replica test environment with only one volume "TEST"
> 
> This is what I expected :
> 
> server0 : glusterd 24007 + glusterfsd 49152
> server1 : glusterd 24007 + glusterfsd 49152
> server2 : glusterd 24007 + glusterfsd 49152
> 
> and this is what I really get :
> 
> server0 : glusterd 24007 + glusterfsd 49156
> server1 : glusterd 24007 + glusterfsd 49152
> server2 : glusterd 24007 + glusterfsd 49155
> 
> I launch all my commands (gluster volume start/stop/status/info) from
> server1.
> 
> I checked the configuration files under
> /var/lib/gluster/glusterd/vols/TEST/bricks and the TCP is "hardcoded"
> for some servers but not for others.
> 
> server0 :
>   server0/listen-port=49156
>   server1/listen-port=0
>   server2/listen-port=0
> 
> server1 :
>   server0/listen-port=0
>   server1/listen-port=49155
>   server2/listen-port=0
> 
> server2 :
>   server0/listen-port=0
>   server1/listen-port=0
>   server2/listen-port=49152
> 
> What's the cause of this?
> 
> The "brick" logs on all servers say something specific :
> 
> On "server0" :
> I [graph.c:269:gf_add_cmdline_options] 0-TEST-server: adding option
> 'listen-port' for
> volume 'TEST-server' with value '49156'
> 
> On "server1" :
> I [graph.c:269:gf_add_cmdline_options] 0-TEST-server: adding option
> 'listen-port' for
> volume 'TEST-server' with value '49155'
> 
> On "server2" :
> I [graph.c:269:gf_add_cmdline_options] 0-TEST-server: adding option
> 'listen-port' for
> volume 'TEST-server' with value '49152'
> 
> I stopped the volume, I edited each brick config file on each server to
> set the port to "0" and even changed "listen-port" to
> "transport.socket.listen-port" (see below). Starting the volume again
> did reset everything the way it was before my changes, even
> "listen-port", so I guess that these "brick" files are dynamically
> created upon volume start.
> 
> Did I understand wrong about the port assignment or is there something
> off with my test setup?
> 
> BTW, they all complain about the same thing :
> 
> W [options.c:898:xl_opt_validate] 0-TEST-server: option 'listen-port' is
> deprecated, preferred is 'transport.socket.listen-port', continuing with
> correction
> 
> -- 
> Unix _IS_ user friendly, it's just selective about who its friends are.
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
> 

-- 

 Jean-François Le Fillâtre
 -------------------------------
 HPC Systems Administrator
 LCSB - University of Luxembourg
 -------------------------------
 PGP KeyID 0x134657C6