[Gluster-users] NFS-Ganesha lo traffic

[Soumya Koduri]

On 08/09/2016 09:06 PM, Mahdi Adnan wrote:
> Hi,
> Thank you for your reply.
>
> The traffic is related to GlusterFS;
>
> 18:31:20.419056 IP 192.168.208.134.49058 > 192.168.208.134.49153: Flags
> [.], ack 3876, win 24576, options [nop,nop,TS val 247718812 ecr
> 247718772], length 0
> 18:31:20.419080 IP 192.168.208.134.49056 > 192.168.208.134.49154: Flags
> [.], ack 11625, win 24576, options [nop,nop,TS val 247718812 ecr
> 247718772], length 0
> 18:31:20.419084 IP 192.168.208.134.49060 > 192.168.208.134.49152: Flags
> [.], ack 9861, win 24576, options [nop,nop,TS val 247718812 ecr
> 247718772], length 0
> 18:31:20.419088 IP 192.168.208.134.49054 > 192.168.208.134.49155: Flags
> [.], ack 4393, win 24568, options [nop,nop,TS val 247718812 ecr
> 247718772], length 0
> 18:31:20.420084 IP 192.168.208.134.49052 > 192.168.208.134.49156: Flags
> [.], ack 5525, win 24576, options [nop,nop,TS val 247718813 ecr
> 247718773], length 0
> 18:31:20.420092 IP 192.168.208.134.49049 > 192.168.208.134.49158: Flags
> [.], ack 6657, win 24576, options [nop,nop,TS val 247718813 ecr
> 247718773], length 0
> 18:31:20.421065 IP 192.168.208.134.49050 > 192.168.208.134.49157: Flags
> [.], ack 4729, win 24570, options [nop,nop,TS val 247718814 ecr
> 247718774], length 0
>

Looks like that is the traffic coming to the bricks local to that node 
(>4915* ports are used by glusterfs brick processes). It could be from 
nfs-ganesha or any other glusterfs client processes (like self-heal 
daemon etc). Do you see this traffic even when there is no active I/O 
from the nfs-client? If so, it could be from the self-heal daemon then. 
Verify if there are any files/directories to be healed.

> Screenshot from wireshark can be found in the attachments.
> 208.134 is the server IP address, and it's looks like it talking to
> itself via the lo interface, im wondering if this is a normal behavior
> or not.
yes. It is the expected behavior when there are clients actively 
accessing the volumes.

> and regarding the Ganesha server logs, how can i debug it to find why
> the servers not responding to the requests on time ?

I suggest again to take tcpdump. Sometimes nfs-ganesha server (glusterfs 
client) may have to communicate with all the bricks over the network 
(like LOOKUP) and that may result in delay if there are lots of bricks 
involved. Try capturing packets from the node where the nfs-ganesha 
server is running and examine the packets between any of the NFS-client 
request and its corresponding reply packet.

I usually use below cmd to capture the packets on all the interfaces -
#tcpdump -i any -s 0 -w /var/tmp/nfs.pcap tcp and not port 22

Thanks,
Soumya
>
>
> --
>
> Respectfully*
> **Mahdi A. Mahdi*
>
>
>
>> Subject: Re: [Gluster-users] NFS-Ganesha lo traffic
>> To: mahdi.adnan at outlook.com
>> From: skoduri at redhat.com
>> CC: gluster-users at gluster.org; nfs-ganesha-devel at lists.sourceforge.net
>> Date: Tue, 9 Aug 2016 18:02:01 +0530
>>
>>
>>
>> On 08/09/2016 03:33 PM, Mahdi Adnan wrote:
>> > Hi,
>> >
>> > Im using NFS-Ganesha to access my volume, it's working fine for now but
>> > im seeing lots of traffic on the Loopback interface, in fact it's the
>> > same amount of traffic on the bonding interface, can anyone please
>> > explain to me why is this happening ?
>>
>> Could you please capture packets on those interfaces using tcpdump and
>> examine the traffic?
>>
>> > also, i got the following error in the ganesha log file;
>> >
>> > 09/08/2016 11:35:54 : epoch 57a5da0c : gfs04 :
>> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
>> > status is unhealthy. Not sending heartbeat
>> > 09/08/2016 11:46:04 : epoch 57a5da0c : gfs04 :
>> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
>> > status is unhealthy. Not sending heartbeat
>> > 09/08/2016 11:54:39 : epoch 57a5da0c : gfs04 :
>> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
>> > status is unhealthy. Not sending heartbeat
>> > 09/08/2016 12:06:04 : epoch 57a5da0c : gfs04 :
>> > ganesha.nfsd-1646[dbus_heartbeat] dbus_heartbeat_cb :DBUS :WARN :Health
>> > status is unhealthy. Not sending heartbeat
>> >
>> > is it something i should care about ?
>>
>> Above warnings are thrown when the outstanding rpc request queue count
>> doesn't change within two heartbeats, in other words the server may be
>> taking a while to process the requests and responding slowly to its
> clients.
>>
>> Thanks,
>> Soumya
>>
>> >
>> > My ganesha config is the following;
>> >
>> >
>> > EXPORT{
>> > Export_Id = 1 ;
>> > Path = "/vlm02";
>> >
>> > FSAL {
>> > name = GLUSTER;
>> > hostname = "gfs04";
>> > volume = "vlm02";
>> > }
>> >
>> > Access_type = RW;
>> > Disable_ACL = TRUE;
>> > Squash = No_root_squash;
>> > Protocols = "3" ;
>> > Transports = "TCP";
>> > }
>> >
>> >
>> > Im accessing it via a floating ip assigned by CTDB.
>> >
>> >
>> > Thank you.
>> > --
>> >
>> > Respectfully*
>> > **Mahdi A. Mahdi*
>> >
>> >
>> >
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > http://www.gluster.org/mailman/listinfo/gluster-users
>> >