[Gluster-users] Gluster high RPC calls and reply

[Niels de Vos]

On Sun, Jul 06, 2014 at 11:28:51PM +1000, Gurdeep Singh (Guru) wrote:
> Hello,
> 
> I have setup gluster in replicate type and its working fine.
> 
> I am seeing a constant chatting between the hosts for lookup call and 
> lookup reply. I am trying to understand as to why this traffic is 
> being initiated constantly. Please look at the attached image. This 
> traffic is using around 200KB/s of constant bandwidth and is 
> exhausting our allocated monthly bandwidth on our 2 VPS.

You can use Wireshark to identify which process does the LOOKUP calls.  
For this, do the following:

1. select a LOOKUP Call
2. enable the 'packet details' pane (found in the main menu, 'view')
3. expand the 'Transmission Control Protocol' tree
4. check the 'Source port' of the LOOKUP Call

Together with the 'Source' and the 'Source port' you can go to the 
server that matches the 'Source' address. A command like this would give 
you the PID of the process in the right column:

  # netstat -tpn | grep $SOURCE_PORT

And with 'ps -v $PID' you can check which process is responsible for the 
LOOKUP. This process can be a fuse-mount, self-heal-daemon or any other 
glusterfs-client. Depending on the type of client, you maybe can tune 
the workload or other options a little.

In Wireshark you can also check what filename is LOOKUP'd, just expand 
the 'GlusterFS' part in the 'packet details' and check the 'Basename'.  
Maybe this filename (without directory structure) does give you any 
ideas of which activity is causing the LOOKUPs.

HTH,
Niels

> 
> The configuration I have for Gluster is:
> 
> [guru at srv1 ~]$ sudo gluster volume info
> [sudo] password for guru: 
>  
> Volume Name: gv0
> Type: Replicate
> Volume ID: dc8dc3f2-f5bd-4047-9101-acad04695442
> Status: Started
> Number of Bricks: 1 x 2 = 2
> Transport-type: tcp
> Bricks:
> Brick1: srv1:/root/gluster-vol0
> Brick2: srv2:/root/gluster-vol0
> Options Reconfigured:
> cluster.lookup-unhashed: on
> performance.cache-refresh-timeout: 60
> performance.cache-size: 1GB
> storage.health-check-interval: 30
> 
> 
> 
> Please suggest how to fine tune the RPC calls/reply.