[Gluster-users] random disconnects of peers

[dpgluster at posteo.de]

I have made extensive load tests in the last few days and figured out 
it's definitely a network related issue. I changed from jumbo frames 
(mtu 9000) to default mtu of 1500. With a mtu of 1500 the problem 
doesn't occur. I'm able to bump the io-wait of our gluster storage 
servers to the max possible values of the disks without any error or 
connection loss between the hypervisors or the storage nodes.

As mentioned in multiple gluster best practices it's recommended to use 
jumbo frames in gluster setups for better performance. So I would like 
to use jumbo frames in my datacenter.

What could be the issue here?


Am 19.08.2022 07:47 schrieb Strahil Nikolov:
> You can check the max op version and if only the oVirt nodes are using
> it -> bump it to the maximum.
> 
> I upgraded my 4.4 while preserving the Gluster storage - just back up
> the /etc/glusterfs & /var/lib/glusterd . Keep in mind that if you use
> VDO you need to backup it's config too.
> 
> Best Regards,
> Strahil Nikolov
> 
>> Yes, the firmware update of the network adapters is planned for the
>> next
>> week.
>> The tcpdump is currently running and i will share the result with
>> you.
>> The update to ovirt 4.4 (and to 4.5) is quite a big deal because of
>> the
>> switch to CentOS stream where a full reinstall is required and there
>> is
>> no possibility to preserve local storage on standalone hypervisors.
>> :P
>> 
>> Gluster opversion is 60000.
>> 
>> Am 18.08.2022 23:46 schrieb Strahil Nikolov:
>>> Usually I start with firmware updates/OS updates.
>>> You can be surprised how many times bad firmware (or dieing NIC)
>> has
>>> left me puzzled.
>>> 
>>> I also support the tcpdump - let it run on all nodes and it might
>> give
>>> a clue what is causing it.
>>> 
>>> I think there is no need to remind you that you should update to
>> oVirt
>>> 4.4 and then to 4.5 ;)
>>> 
>>> By the way, what is your cluster OP version ?
>>> 
>>> Best Regards,
>>> Strahil Nikolov
>>> 
>>>> On Thu, Aug 18, 2022 at 14:27, Péter Károly JUHÁSZ
>>>> <juhasz.peter.karoly at gmail.com> wrote:
>>>> ________
>>>> 
>>>> Community Meeting Calendar:
>>>> 
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://meet.google.com/cpu-eiue-hvk [1][1]
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users [2][2]
>>> 
>>> 
>>> Links:
>>> ------
>>> [1] https://meet.google.com/cpu-eiue-hvk [1]
>> 
>>> [2] https://lists.gluster.org/mailman/listinfo/gluster-users [2]
> 
> 
> Links:
> ------
> [1] https://meet.google.com/cpu-eiue-hvk
> [2] https://lists.gluster.org/mailman/listinfo/gluster-users