[Gluster-infra] review.gluster.org is really laggy today

Tue May 5 14:12:16 UTC 2015

There seems to be problem with the network at the host. Ran an mtr
command for google.com which gave the following output,
```
[root at dev ~]# mtr -c 50 --report google.com
dev.gluster.org                   Snt: 50    Loss%  Last   Avg  Best  Wrst StDev
10.3.129.1                                   82.0%   3.5   6.7   2.6  10.3   3.1
72.55.168.2                                  98.0%  17.2  17.2  17.2  17.2   0.0
te7-2.cr3.mtl.iweb.com                       96.0%  11.2  14.9  11.2  18.7   5.3
ae0.er1.ny.iweb.com                          98.0%  18.2  18.2  18.2  18.2   0.0
core1-0-2-0.lga.net.google.com               94.0%  18.1  15.5  13.9  18.1   2.3
209.85.248.178                               94.0%  18.1  17.8  13.7  21.5   3.9
209.85.252.242                               98.0%  14.6  14.6  14.6  14.6   0.0
209.85.143.120                               98.0%  25.6  25.6  25.6  25.6   0.0
216.239.43.66                                98.0%  25.4  25.4  25.4  25.4   0.0
???                                          100.0   0.0   0.0   0.0   0.0   0.0
???                                          100.0   0.0   0.0   0.0   0.0   0.0
???                                          100.0   0.0   0.0   0.0   0.0   0.0
???                                          100.0   0.0   0.0   0.0   0.0   0.0
iad23s24-in-f0.1e100.net                     97.9%  28.2  28.2  28.2  28.2   0.0
```

We seem to be getting incoming requests okay enough, but a lot of the
responses are being lost. I'm almost sure that there is nothing wrong
with our gerrit installation.

~kaushal

On Tue, May 5, 2015 at 4:22 PM, Kaushal M <kshlmster at gmail.com> wrote:
> The site seems to be a better after the reboot. Pages are loading
> quicker now. Should be okay for now.
>
> Thanks for looking into this Justin. Take care and get well soon.
>
> ~kaushal
>
> On Tue, May 5, 2015 at 4:19 PM, Justin Clift <justin at gluster.org> wrote:
>> k, so it's a problem with more than just the Gerrit VM. :(
>>
>> On 5 May 2015, at 11:45, Kaushal M <kshlmster at gmail.com> wrote:
>>> Slightly less for the host machine, ~20%.
>>>
>>>
>>> On Tue, May 5, 2015 at 4:11 PM, Justin Clift <justin at gluster.org> wrote:
>>>> Do you get packet loss to the host machine as well?
>>>>
>>>> + Justin
>>>>
>>>>
>>>> On 5 May 2015, at 11:39, Kaushal M <kshlmster at gmail.com> wrote:
>>>>> Might have been a spike at that moment. The loss is back to ~30%.
>>>>>
>>>>> On Tue, May 5, 2015 at 4:06 PM, Kaushal M <kshlmster at gmail.com> wrote:
>>>>>> It's acutally worse now, 70-80% loss!
>>>>>>
>>>>>> On Tue, May 5, 2015 at 4:04 PM, Justin Clift <justin at gluster.org> wrote:
>>>>>>> Rebooted it.  Seems a little bit better from a Gerrit-isn't-quite-as-slow
>>>>>>> point of view.
>>>>>>>
>>>>>>> How's the packet loss looking from your point now?
>>>>>>>
>>>>>>> + Justin
>>>>>>>
>>>>>>>
>>>>>>> On 5 May 2015, at 11:26, Justin Clift <justin at gluster.org> wrote:
>>>>>>>> Hmmmm.  I'll reboot the VM and see if that helps.
>>>>>>>>
>>>>>>>> Packet loss indicates it's not just Gerrit inside the VM having a problem.
>>>>>>>>
>>>>>>>> + Justin
>>>>>>>>
>>>>>>>>
>>>>>>>> On 5 May 2015, at 07:47, Kaushal M <kshlmster at gmail.com> wrote:
>>>>>>>>> Michael, Justin,
>>>>>>>>>
>>>>>>>>> Can one of you help here? It's becoming really painful to review, and
>>>>>>>>> we need to review more than ever.
>>>>>>>>>
>>>>>>>>> ~kaushal
>>>>>>>>>
>>>>>>>>> On Mon, May 4, 2015 at 11:16 AM, Kaushal M <kshlmster at gmail.com> wrote:
>>>>>>>>>> review.gluster.org is really laggy today, with a lot of packet loss.
>>>>>>>>>> Could someone look into it?
>>>>>>>>>>
>>>>>>>>>> ```
>>>>>>>>>> --- review.gluster.org ping statistics ---
>>>>>>>>>> 33 packets transmitted, 17 received, 48% packet loss, time 32075ms
>>>>>>>>>> rtt min/avg/max/mdev = 235.347/252.873/321.057/18.161 ms
>>>>>>>>>> ```
>>>>>>>>
>>>>>>>> --
>>>>>>>> GlusterFS - http://www.gluster.org
>>>>>>>>
>>>>>>>> An open source, distributed file system scaling to several
>>>>>>>> petabytes, and handling thousands of clients.
>>>>>>>>
>>>>>>>> My personal twitter: twitter.com/realjustinclift
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> GlusterFS - http://www.gluster.org
>>>>>>>
>>>>>>> An open source, distributed file system scaling to several
>>>>>>> petabytes, and handling thousands of clients.
>>>>>>>
>>>>>>> My personal twitter: twitter.com/realjustinclift
>>>>>>>
>>>>
>>>> --
>>>> GlusterFS - http://www.gluster.org
>>>>
>>>> An open source, distributed file system scaling to several
>>>> petabytes, and handling thousands of clients.
>>>>
>>>> My personal twitter: twitter.com/realjustinclift
>>>>
>>
>> --
>> GlusterFS - http://www.gluster.org
>>
>> An open source, distributed file system scaling to several
>> petabytes, and handling thousands of clients.
>>
>> My personal twitter: twitter.com/realjustinclift
>>