[GEDI] Unexpected libgfapi behaviour

[Niels de Vos]

On Mon, Oct 30, 2017 at 02:58:48PM +0100, Denis Chaplygin wrote:
...
> > The connection to glusterd is only for the management part. Once
> > glfs_init() is called, the volume layout is fetched from glusterd, and
> > based on those details, the connections to the bricks are made. You can
> > check the logs from the bricks to see if connections go established.
> > Capturing a tcpdump on the system where the gfapi application is running
> > and loading it in wireshark can show more whats happening on the network
> > side (wireshark knows most of the Gluster protocol).
> >
> >
> Looks like it loads volinfo and everything stops here. I attached my dump
> file, so you make take a look on it if you like.

In Wireshark, you can filter with "rpc" to show all the Gluster traffic.
The GETSPEC call is indeed returning the .vol file with the volume
layout.

Bricks are normally listening on ports 49000-50000 (I dont remember the
exact range), so incomming connections can be shown with this filter:

  tcp.dstport > 49000 and tcp.dstport < 50000

The output from that suggests that from frame 58 a connection to the
brick on 10.37.138.244:49144 is made. There is some communication
happening, but it is not clear what the processes are exchanging. The
data in those packets is not getting decoded, see for yourself in frame
58 and others, a few of them have a "data" part:

  tcp && frame.number >= 58

Upon closer inspection, the data in the tcp packets does not look like
RPC procedures. Maybe some encryption is enabled?

You could turn the log-level for the bricks higher as well. Maybe they
will show what is going on. Use the 'gluster' command to do this:

  # gluster volume set testiso diagnostics.brick-log-level DEBUG
  (no restart is needed, just run your test again)

HTH,
Niels