From archon810 at gmail.com  Sat Aug  1 04:45:43 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Fri, 31 Jul 2020 21:45:43 -0700
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
In-Reply-To: <CAERh03rWRpp87LCJ9AEu+w3Qw6eTbFbWca69NhtPAMChSCvaVw@mail.gmail.com>
References: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
	<CAD+dzQep-zAiEW3GpKgF138SJ7EavM0t5Ck=A3+TVKdazSxrJA@mail.gmail.com>
	<CAERh03rWRpp87LCJ9AEu+w3Qw6eTbFbWca69NhtPAMChSCvaVw@mail.gmail.com>
Message-ID: <CAD+dzQcr=-wjvZQwiQb95SfLrKRxBVAQiL6ZeJ_mRN2WjT16VA@mail.gmail.com>

Got it, thanks. Already started upgrading the fleet to 15.2, so we'll be
able to upgrade from 7.6 soon.

On Thu, Jul 30, 2020, 10:39 PM Shwetha Acharya <sacharya at redhat.com> wrote:

> Hi Artem,
>
> As per current Tentative plans for community packages
> <https://github.com/gluster/glusterdocs/blob/master/docs/Install-Guide/Community_Packages.md> we
> are supporting Leap15.2 only.
>
> Regards,
> Shwetha
>
> On Fri, Jul 31, 2020 at 1:03 AM Artem Russakovskii <archon810 at gmail.com>
> wrote:
>
>> Hi,
>>
>>
>> https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.1-7/openSUSE_Leap_15.1/x86_64/
>> is still missing 7.7. Is there an ETA please?
>>
>> Thanks.
>>
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>>
>>
>> On Wed, Jul 22, 2020 at 9:27 AM Rinku Kothiya <rkothiya at redhat.com>
>> wrote:
>>
>>> Hi,
>>>
>>> The Gluster community is pleased to announce the release of Gluster7.7
>>> (packages available at [1]).
>>> Release notes for the release can be found at [2].
>>>
>>> Major changes, features and limitations addressed in this release:
>>> None
>>>
>>> Please Note: Some of the packages are unavailable and we are working on
>>> it. We will release them soon.
>>>
>>> Thanks,
>>> Gluster community
>>>
>>> References:
>>>
>>> [1] Packages for 7.7:
>>> https://download.gluster.org/pub/gluster/glusterfs/7/7.7/
>>>
>>> [2] Release notes for 7.7:
>>> https://docs.gluster.org/en/latest/release-notes/7.7/
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200731/8e04d8f9/attachment.html>

From spalai at redhat.com  Mon Aug  3 05:46:36 2020
From: spalai at redhat.com (Susant Palai)
Date: Mon, 3 Aug 2020 11:16:36 +0530
Subject: [Gluster-users] Rebalance improvement.
Message-ID: <90AD956E-EB56-4A00-AB8F-C44D3A1BE0E1@redhat.com>

Hi,
    Recently, we have pushed some performance improvements for Rebalance Crawl which used to consume a significant amount of time, out of the entire rebalance process.


The patch [1] is recently merged in upstream and may land as an experimental feature in the upcoming upstream release.

The improvement currently works only for pure-distribute Volume. (which can be expanded).


Things to look forward to in future :
 - Parallel Crawl in Rebalance
 - Global Layout

Once these improvements are in place, we would be able to reduce the overall rebalance time by a significant time.

Would request our community to try out the feature and give us feedback.

More information regarding the same will follow.


Thanks & Regards,
Susant Palai


[1] https://review.gluster.org/#/c/glusterfs/+/24443/ <https://review.gluster.org/#/c/glusterfs/+/24443/>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/bb04dd78/attachment.html>

From revirii at googlemail.com  Mon Aug  3 06:46:21 2020
From: revirii at googlemail.com (Hu Bert)
Date: Mon, 3 Aug 2020 08:46:21 +0200
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
In-Reply-To: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
References: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
Message-ID: <CAAV-989ZQf6oaStynwvmLpn3XheytO+5kw1Hw1h2utXWE=_Anw@mail.gmail.com>

Hi there,

just wanted to say thanks to all the developers, maintainers etc. This
release (7) has brought us a small but nice performance improvement.
Utilization and IOs per disk decreased, latency dropped. See attached
images.

I read the release notes but couldn't identify the specific
changes/features for this improvement. Maybe someone could point to
them - but no hurry... :-)


Best regards,
Hubert

Am Mi., 22. Juli 2020 um 18:27 Uhr schrieb Rinku Kothiya <rkothiya at redhat.com>:
>
> Hi,
>
> The Gluster community is pleased to announce the release of Gluster7.7 (packages available at [1]).
> Release notes for the release can be found at [2].
>
> Major changes, features and limitations addressed in this release:
> None
>
> Please Note: Some of the packages are unavailable and we are working on it. We will release them soon.
>
> Thanks,
> Gluster community
>
> References:
>
> [1] Packages for 7.7:
> https://download.gluster.org/pub/gluster/glusterfs/7/7.7/
>
> [2] Release notes for 7.7:
> https://docs.gluster.org/en/latest/release-notes/7.7/
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diskstats_iops-week.png
Type: image/png
Size: 68055 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/782dc9d9/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diskstats_utilization-week.png
Type: image/png
Size: 61232 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/782dc9d9/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diskstats_latency-week.png
Type: image/png
Size: 54543 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/782dc9d9/attachment-0002.png>

From spalai at redhat.com  Mon Aug  3 07:17:01 2020
From: spalai at redhat.com (Susant Palai)
Date: Mon, 3 Aug 2020 12:47:01 +0530
Subject: [Gluster-users] Rebalance improvement.
In-Reply-To: <90AD956E-EB56-4A00-AB8F-C44D3A1BE0E1@redhat.com>
References: <90AD956E-EB56-4A00-AB8F-C44D3A1BE0E1@redhat.com>
Message-ID: <CA+xDNC2Jz0hgqy+6D5qguuoekR_ZSOTRi8Jzw6gbTy=0as2bcQ@mail.gmail.com>

Centos Users can add the following repo and install the build from the
master branch to try out the feature. [Testing purpose only, not ready for
consumption in production env.]

[gluster-nightly-master] baseurl=
http://artifacts.ci.centos.org/gluster/nightly/master/7/x86_64/ gpgcheck=0
keepalive=1 enabled=1 repo_gpgcheck = 0 name=Gluster Nightly builds (master
branch)

A summary of perf numbers from our test lab :

DirSize - 1Million Old New %diff Depth - 100 (Run 1) 353 74 +377% Depth -
100 (Run 2) 348 72 +377~% Depth - 50 246 122 +100% Depth - 3 174 114 +52%

Susant


On Mon, Aug 3, 2020 at 11:16 AM Susant Palai <spalai at redhat.com> wrote:

> Hi,
>     Recently, we have pushed some performance improvements for Rebalance
> Crawl which used to consume a significant amount of time, out of the entire
> rebalance process.
>
>
> The patch [1] is recently merged in upstream and may land as an
> experimental feature in the upcoming upstream release.
>
> The improvement currently works only for pure-distribute Volume. (which
> can be expanded).
>
>
> Things to look forward to in future :
>  - Parallel Crawl in Rebalance
>  - Global Layout
>
> Once these improvements are in place, we would be able to reduce the
> overall rebalance time by a significant time.
>
> Would request our community to try out the feature and give us feedback.
>
> More information regarding the same will follow.
>
>
> Thanks & Regards,
> Susant Palai
>
>
> [1] https://review.gluster.org/#/c/glusterfs/+/24443/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/282a8887/attachment.html>

From aravinda at kadalu.io  Mon Aug  3 08:28:40 2020
From: aravinda at kadalu.io (Aravinda VK)
Date: Mon, 3 Aug 2020 13:58:40 +0530
Subject: [Gluster-users] Rebalance improvement.
In-Reply-To: <CA+xDNC2Jz0hgqy+6D5qguuoekR_ZSOTRi8Jzw6gbTy=0as2bcQ@mail.gmail.com>
References: <90AD956E-EB56-4A00-AB8F-C44D3A1BE0E1@redhat.com>
	<CA+xDNC2Jz0hgqy+6D5qguuoekR_ZSOTRi8Jzw6gbTy=0as2bcQ@mail.gmail.com>
Message-ID: <36E38592-A906-48A5-B437-84C7C37057F2@kadalu.io>

Interesting numbers. Thanks for the effort.

What is the unit of old/new numbers? seconds? 

> On 03-Aug-2020, at 12:47 PM, Susant Palai <spalai at redhat.com> wrote:
> 
> Centos Users can add the following repo and install the build from the master branch to try out the feature. [Testing purpose only, not ready for consumption in production env.]
> 
> [gluster-nightly-master]
> baseurl=http://artifacts.ci.centos.org/gluster/nightly/master/7/x86_64/ <http://artifacts.ci.centos.org/gluster/nightly/master/7/x86_64/>
> gpgcheck=0
> keepalive=1
> enabled=1
> repo_gpgcheck = 0
> name=Gluster Nightly builds (master branch)
> 
> A summary of perf numbers from our test lab :
> 
> DirSize - 1Million	Old	New	%diff
> Depth 	- 100 (Run 1)	353	74	+377%
> Depth 	- 100 (Run 2)	348	72	+377~%
> Depth 	- 50		246	122	+100%
> Depth 	- 3		174	114	+52%
> 
> Susant
> 
> 
> On Mon, Aug 3, 2020 at 11:16 AM Susant Palai <spalai at redhat.com <mailto:spalai at redhat.com>> wrote:
> Hi,
>     Recently, we have pushed some performance improvements for Rebalance Crawl which used to consume a significant amount of time, out of the entire rebalance process.
> 
> 
> The patch [1] is recently merged in upstream and may land as an experimental feature in the upcoming upstream release.
> 
> The improvement currently works only for pure-distribute Volume. (which can be expanded).
> 
> 
> Things to look forward to in future :
>  - Parallel Crawl in Rebalance
>  - Global Layout
> 
> Once these improvements are in place, we would be able to reduce the overall rebalance time by a significant time.
> 
> Would request our community to try out the feature and give us feedback.
> 
> More information regarding the same will follow.
> 
> 
> Thanks & Regards,
> Susant Palai
> 
> 
> [1] https://review.gluster.org/#/c/glusterfs/+/24443/ <https://review.gluster.org/#/c/glusterfs/+/24443/>________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

Aravinda Vishwanathapura
https://kadalu.io



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/ac34a9fc/attachment.html>

From spalai at redhat.com  Mon Aug  3 08:39:37 2020
From: spalai at redhat.com (Susant Palai)
Date: Mon, 3 Aug 2020 14:09:37 +0530
Subject: [Gluster-users] Rebalance improvement.
In-Reply-To: <36E38592-A906-48A5-B437-84C7C37057F2@kadalu.io>
References: <90AD956E-EB56-4A00-AB8F-C44D3A1BE0E1@redhat.com>
	<CA+xDNC2Jz0hgqy+6D5qguuoekR_ZSOTRi8Jzw6gbTy=0as2bcQ@mail.gmail.com>
	<36E38592-A906-48A5-B437-84C7C37057F2@kadalu.io>
Message-ID: <F8321A9C-28F9-416C-A0FC-CEDF8EDF54B3@redhat.com>



> On 03-Aug-2020, at 13:58, Aravinda VK <aravinda at kadalu.io> wrote:
> 
> Interesting numbers. Thanks for the effort.
> 
> What is the unit of old/new numbers? seconds? 

Minutes. 

> 
>> On 03-Aug-2020, at 12:47 PM, Susant Palai <spalai at redhat.com <mailto:spalai at redhat.com>> wrote:
>> 
>> Centos Users can add the following repo and install the build from the master branch to try out the feature. [Testing purpose only, not ready for consumption in production env.]
>> 
>> [gluster-nightly-master]
>> baseurl=http://artifacts.ci.centos.org/gluster/nightly/master/7/x86_64/ <http://artifacts.ci.centos.org/gluster/nightly/master/7/x86_64/>
>> gpgcheck=0
>> keepalive=1
>> enabled=1
>> repo_gpgcheck = 0
>> name=Gluster Nightly builds (master branch)
>> 
>> A summary of perf numbers from our test lab :
>> 
>> DirSize - 1Million	Old	New	%diff
>> Depth 	- 100 (Run 1)	353	74	+377%
>> Depth 	- 100 (Run 2)	348	72	+377~%
>> Depth 	- 50		246	122	+100%
>> Depth 	- 3		174	114	+52%
>> 
>> Susant
>> 
>> 
>> On Mon, Aug 3, 2020 at 11:16 AM Susant Palai <spalai at redhat.com <mailto:spalai at redhat.com>> wrote:
>> Hi,
>>     Recently, we have pushed some performance improvements for Rebalance Crawl which used to consume a significant amount of time, out of the entire rebalance process.
>> 
>> 
>> The patch [1] is recently merged in upstream and may land as an experimental feature in the upcoming upstream release.
>> 
>> The improvement currently works only for pure-distribute Volume. (which can be expanded).
>> 
>> 
>> Things to look forward to in future :
>>  - Parallel Crawl in Rebalance
>>  - Global Layout
>> 
>> Once these improvements are in place, we would be able to reduce the overall rebalance time by a significant time.
>> 
>> Would request our community to try out the feature and give us feedback.
>> 
>> More information regarding the same will follow.
>> 
>> 
>> Thanks & Regards,
>> Susant Palai
>> 
>> 
>> [1] https://review.gluster.org/#/c/glusterfs/+/24443/ <https://review.gluster.org/#/c/glusterfs/+/24443/>________
>> 
>> 
>> 
>> Community Meeting Calendar:
>> 
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968 <https://bluejeans.com/441850968>
>> 
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> https://lists.gluster.org/mailman/listinfo/gluster-users
> 
> Aravinda Vishwanathapura
> https://kadalu.io <https://kadalu.io/>
> 
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/3ed78250/attachment.html>

From archon810 at gmail.com  Mon Aug  3 17:54:24 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Mon, 3 Aug 2020 10:54:24 -0700
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <ab84b036-4f0b-4583-6d7a-54bc5772bfd5@redhat.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
	<CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
	<CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>
	<CADkK8gvKdO6EapmAA12VzCGj8m=BAvnEbW32QEaB-tTnrf6gQQ@mail.gmail.com>
	<CADkK8gs1pDrAmMA_ONnc06kGAX8uxr5A7CarGtkYMFfTg=1sTA@mail.gmail.com>
	<CAD+dzQdYk2xvDd2XifwFroO7nEYMJzDs-3ZOx+_-=g8czJNV-Q@mail.gmail.com>
	<ab84b036-4f0b-4583-6d7a-54bc5772bfd5@redhat.com>
Message-ID: <CAD+dzQeSp-iNpD2EPq5drs8wcvH-FJDtnqzr2Z5ozzn6RH6jxQ@mail.gmail.com>

>
> Do you kill all gluster processes (not just glusterd but even the brick
> processes) before issuing reboot? This is necessary to prevent I/O stalls.
> There is stop-all-gluster-processes.sh which should be available as a part
> of the gluster installation (maybe in /usr/share/glusterfs/scripts/) which
> you can use.  Can you check if this helps?
>
A reboot shuts down gracefully, so those processes are shut down before the
reboot begins.

We've moved on to discussing this matter in the gluster slack, there's a
lot more info there now about the above. The gist is heavy xfs
fragmentation when bricks are almost full (95-96%) made healing as well as
disk accesses a lot more expensive and slow, and prone to hanging.

What's still not clear is why a slowdown of one brick/gluster instance
affects similarly affects all bricks/gluster instances, on other servers,
and how that can be optimized/mitigated.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Thu, Jul 30, 2020 at 8:21 PM Ravishankar N <ravishankar at redhat.com>
wrote:

>
> On 25/07/20 4:35 am, Artem Russakovskii wrote:
>
> Speaking of fio, could the gluster team please help me understand
> something?
>
> We've been having lots of performance issues related to gluster using
> attached block storage on Linode. At some point, I figured out that Linode
> has a cap of 500 IOPS on their block storage
> <https://www.linode.com/community/questions/19437/does-a-dedicated-cpu-or-high-memory-plan-improve-disk-io-performance#answer-72142>
> (with spikes to 1500 IOPS). The block storage we use is formatted xfs with
> 4KB bsize (block size).
>
> I then ran a bunch of fio tests on the block storage itself (not the
> gluster fuse mount), which performed horribly when the bs parameter was set
> to 4k:
> fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1
> --name=test --filename=test --bs=4k --iodepth=64 --size=4G
> --readwrite=randwrite --ramp_time=4
> During these tests, fio ETA crawled to over an hour, at some point dropped
> to 45min and I did see 500-1500 IOPS flash by briefly, then it went back
> down to 0. I/O seems majorly choked for some reason, likely because gluster
> is using some of it. Transfer speed with such 4k block size is 2 MB/s with
> spikes to 6MB/s. This causes the load on the server to spike up to 100+ and
> brings down all our servers.
>
> Jobs: 1 (f=1): [w(1)][20.3%][r=0KiB/s,w=5908KiB/s][r=0,w=1477 IOPS][eta 43m:00s]    Jobs: 1 (f=1): [w(1)][21.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 44m:54s]
>
> xfs_info /mnt/citadel_block1
> meta-data=/dev/sdc               isize=512    agcount=103, agsize=26214400 blks
>          =                       sectsz=512   attr=2, projid32bit=1
>          =                       crc=1        finobt=1, sparse=0, rmapbt=0
>          =                       reflink=0
> data     =                       bsize=4096   blocks=2684354560, imaxpct=25
>          =                       sunit=0      swidth=0 blks
> naming   =version 2              bsize=4096   ascii-ci=0, ftype=1log      =internal log           bsize=4096   blocks=51200, version=2
>          =                       sectsz=512   sunit=0 blks, lazy-count=1
> realtime =none                   extsz=4096   blocks=0, rtextents=0
>
> When I increase the --bs param to fio from 4k to, say, 64k, transfer speed
> goes up significantly and is more like 50MB/s, and at 256k, it's 200MB/s.
>
> So what I'm trying to understand is:
>
>    1. How does the xfs block size (4KB) relate to the block size in fio
>    tests? If we're limited by IOPS, and xfs block size is 4KB, how can fio
>    produce better results with varying --bs param?
>    2. Would increasing the xfs data block size to something like 64-256KB
>    help with our issue of choking IO and skyrocketing load?
>
> I have experienced similar behavior when running fio tests with bs=4k on a
> gluster volume backed by XFS with a high load (numjobs=32) . When I
> observed the strace of the brick processes (fsync -f -T -p $PID), I saw
> fysnc system calls taking around 2500 seconds which is insane. I'm not sure
> if this is specific to the way fio does its i/o pattern and the way XFS
> handles it. When I used 64k block sizes, the fio tests completed just fine.
>
>
>    1. The worst hangs and load spikes happen when we reboot one of the
>    gluster servers, but not when it's down - when it comes back online. Even
>    with gluster not showing anything pending heal, my guess is it's still
>    trying to do lots of IO between the 4 nodes for some reason, but I don't
>    understand why.
>
> Do you kill all gluster processes (not just glusterd but even the brick
> processes) before issuing reboot? This is necessary to prevent I/O stalls.
> There is stop-all-gluster-processes.sh which should be available as a part
> of the gluster installation (maybe in /usr/share/glusterfs/scripts/) which
> you can use.  Can you check if this helps?
>
> Regards,
>
> Ravi
>
> I've been banging my head on the wall with this problem for months.
> Appreciate any feedback here.
>
> Thank you.
>
> gluster volume info below
>
> Volume Name: SNIP_data1
> Type: Replicate
> Volume ID: SNIP
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x 4 = 4
> Transport-type: tcp
> Bricks:
> Brick1: nexus2:/mnt/SNIP_block1/SNIP_data1
> Brick2: forge:/mnt/SNIP_block1/SNIP_data1
> Brick3: hive:/mnt/SNIP_block1/SNIP_data1
> Brick4: citadel:/mnt/SNIP_block1/SNIP_data1
> Options Reconfigured:
> cluster.quorum-count: 1
> cluster.quorum-type: fixed
> network.ping-timeout: 5
> network.remote-dio: enable
> performance.rda-cache-limit: 256MB
> performance.readdir-ahead: on
> performance.parallel-readdir: on
> network.inode-lru-limit: 500000
> performance.md-cache-timeout: 600
> performance.cache-invalidation: on
> performance.stat-prefetch: on
> features.cache-invalidation-timeout: 600
> features.cache-invalidation: on
> cluster.readdir-optimize: on
> performance.io-thread-count: 32
> server.event-threads: 4
> client.event-threads: 4
> performance.read-ahead: off
> cluster.lookup-optimize: on
> performance.cache-size: 1GB
> cluster.self-heal-daemon: enable
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: on
> cluster.granular-entry-heal: enable
> cluster.data-self-heal-algorithm: full
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Thu, Jul 23, 2020 at 12:08 AM Qing Wang <qw at g.clemson.edu> wrote:
>
>> Hi,
>>
>> I have one more question about the Gluster linear scale-out performance
>> regarding the "write-behind off" case specifically -- when "write-behind"
>> is off, and still the stripe volumes and other settings as early thread
>> posted, the storage I/O seems not to relate to the number of storage
>> nodes. In my experiment, no matter I have 2 brick server nodes or 8 brick
>> server nodes, the aggregated gluster I/O performance is ~100MB/sec. And fio
>> benchmark measurement gives the same result. If "write behind" is on, then
>> the storage performance is linear scale-out along with the # of brick
>> server nodes increasing.
>>
>> No matter the write behind option is on/off, I thought the gluster I/O
>> performance should be pulled and aggregated together as a whole. If that is
>> the case, why do I get a consistent gluster performance (~100MB/sec) when
>> "write behind" is off? Please advise me if I misunderstood something.
>>
>> Thanks,
>> Qing
>>
>>
>>
>>
>> On Tue, Jul 21, 2020 at 7:29 PM Qing Wang <qw at g.clemson.edu> wrote:
>>
>>> fio gives me the correct linear scale-out results, and you're right, the
>>> storage cache is the root cause that makes the dd measurement results not
>>> accurate at all.
>>>
>>> Thanks,
>>> Qing
>>>
>>>
>>> On Tue, Jul 21, 2020 at 2:53 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Tue, 21 Jul 2020, 21:43 Qing Wang <qw at g.clemson.edu> wrote:
>>>>
>>>>> Hi Yaniv,
>>>>>
>>>>> Thanks for the quick response. I forget to mention I am testing the
>>>>> writing performance, not reading. In this case, would the client cache hit
>>>>> rate still be a big issue?
>>>>>
>>>>
>>>> It's not hitting the storage directly. Since it's also single threaded,
>>>> it may also not saturate it. I highly recommend testing properly.
>>>> Y.
>>>>
>>>>
>>>>> I'll use fio to run my test once again, thanks for the suggestion.
>>>>>
>>>>> Thanks,
>>>>> Qing
>>>>>
>>>>> On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I am trying to test Gluster linear scale-out performance by adding
>>>>>>> more storage server/bricks, and measure the storage I/O performance. To
>>>>>>> vary the storage server number, I create several "stripe" volumes that
>>>>>>> contain 2 brick servers, 3 brick servers, 4 brick servers, and so on. On
>>>>>>> gluster client side, I used "dd if=/dev/zero
>>>>>>> of=/mnt/glusterfs/dns_test_data_26g bs=1M count=26000" to create 26G data
>>>>>>> (or larger size), and those data will be distributed to the corresponding
>>>>>>> gluster servers (each has gluster brick on it) and "dd" returns the final
>>>>>>> I/O throughput. The Internet is 40G infiniband, although I didn't do any
>>>>>>> specific configurations to use advanced features.
>>>>>>>
>>>>>>
>>>>>> Your dd command is inaccurate, as it'll hit the client cache. It is
>>>>>> also single threaded. I suggest switching to fio.
>>>>>> Y.
>>>>>>
>>>>>>
>>>>>>> What confuses me is that the storage I/O seems not to relate to the
>>>>>>> number of storage nodes, but Gluster documents said it should be linear
>>>>>>> scaling. For example, when "write-behind" is on, and when Infiniband "jumbo
>>>>>>> frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no
>>>>>>> matter I have 2 brick servers or 8 brick servers -- for 2 server case, each
>>>>>>> server can have ~400 MB/sec; for 4 server case, each server can have
>>>>>>> ~200MB/sec. That said, each server I/O does aggregate to the final storage
>>>>>>> I/O (800 MB/sec), but this is not "linear scale-out".
>>>>>>>
>>>>>>> Can somebody help me to understand why this is the case? I certainly
>>>>>>> can have some misunderstanding/misconfiguration here. Please correct me if
>>>>>>> I do, thanks!
>>>>>>>
>>>>>>> Best,
>>>>>>> Qing
>>>>>>> ________
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Community Meeting Calendar:
>>>>>>>
>>>>>>> Schedule -
>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/c854e0e1/attachment.html>

From bob at computerisms.ca  Tue Aug  4 03:01:17 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Mon, 3 Aug 2020 20:01:17 -0700
Subject: [Gluster-users] performance
Message-ID: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>

Hi Gurus,

I have been trying to wrap my head around performance improvements on my 
gluster setup, and I don't seem to be making any progress.  I mean 
forward progress.  making it worse takes practically no effort at all.

My gluster is distributed-replicated across 6 bricks and 2 servers, with 
an arbiter on each server.  I designed it like this so I have an 
expansion path to more servers in the future (like the staggered arbiter 
diagram in the red hat documentation).  gluster v info output is below. 
I have compiled gluster 7.6 from sources on both servers.

Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit 
network connections.  They are running debian, and are being used as 
redundant web servers.  There is some 3Million files on the Gluster 
Storage averaging 130KB/file.  Currently only one of the two servers is 
serving web services.  There are well over 100 sites, and apache 
server-status claims around 5 hits per second, depending on time of day, 
so a fair bit of logging going on.  The gluster is only holding website 
data and config files that will be common between the two servers, no 
databases or anything like that on the Gluster.

When the serving server is under load load average is consistently 
12-20.  glusterfs is always at the top with 150%-250% cpu, and each of 3 
bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores. 
apache processes will easily eat up all the rest of the cpus after that. 
  And web page response time is underwhelming at best.

Interestingly, mostly because it is not something I have ever 
experienced before, software interrupts sit between 1 and 5 on each 
core, but the last core is usually sitting around 20.  Have never 
encountered a high load average where the si number was ever 
significant.  I have googled the crap out of that (as well as gluster 
performance in general), there are nearly limitless posts about what it 
is, but have yet to see one thing to explain what to do about it.  Sadly 
I can't really shut down the gluster process to confirm if that is the 
cause, but it's a pretty good bet, I think.

When the system is not under load, glusterfs will be running at around 
100% with each of the 3 bricks around 35%, so using 2 cores when doing 
not much of anything.

nload shows the network cards rarely climb above 300 Mbps unless I am 
doing a direct file transfer between the servers, in which case it gets 
right up to the 1Gbps limit.  RAM is never above 15GB unless I am 
causing it to happen.  atop show a disk busy percentage, it is often 
above 50% and sometimes will hit 100%, and is no where near as 
consistently showing excessive usage like the cpu cores are.  The cpu 
definitely seems to be the bottleneck.

When I found out about the groups directory, I figured one of those must 
be useful to me, but as best as I can tell they are not.  But I am 
really hoping that someone has configured a system like mine and has a 
good group file they might share for this situation, or a peak at their 
volume info output?

or maybe this is really just about as good as I should expect?  Maybe 
the fix is that I need more/faster cores?  I hope not, as that isn't 
really an option.

Anyway, here is my volume info as promised.

root at mooglian:/Computerisms/sites/computerisms.ca/log# gluster v info

Volume Name: webisms
Type: Distributed-Replicate
Volume ID: 261901e7-60b4-4760-897d-0163beed356e
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb 
(arbiter)
Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb 
(arbiter)
Options Reconfigured:
auth.allow: xxxx
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
performance.stat-prefetch: on
network.inode-lru-limit: 200000
performance.write-behind-window-size: 4MB
performance.readdir-ahead: on
performance.io-thread-count: 64
performance.cache-size: 8GB
server.event-threads: 8
client.event-threads: 8
performance.nl-cache-timeout: 600


-- 
Bob Miller
Cell: 867-334-7117
Office: 867-633-3760
Office: 867-322-0362
www.computerisms.ca

From hunter86_bg at yahoo.com  Tue Aug  4 04:00:06 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Tue, 04 Aug 2020 07:00:06 +0300
Subject: [Gluster-users] performance
In-Reply-To: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
Message-ID: <7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>



?? 4 ?????? 2020 ?. 6:01:17 GMT+03:00, Computerisms Corporation <bob at computerisms.ca> ??????:
>Hi Gurus,
>
>I have been trying to wrap my head around performance improvements on
>my 
>gluster setup, and I don't seem to be making any progress.  I mean 
>forward progress.  making it worse takes practically no effort at all.
>
>My gluster is distributed-replicated across 6 bricks and 2 servers,
>with 
>an arbiter on each server.  I designed it like this so I have an 
>expansion path to more servers in the future (like the staggered
>arbiter 
>diagram in the red hat documentation).  gluster v info output is below.
>
>I have compiled gluster 7.6 from sources on both servers.

There  is a 7.7 version which is fixing somw stuff. Why do you have to compile it from source ?

>Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit 
>network connections.  They are running debian, and are being used as 
>redundant web servers.  There is some 3Million files on the Gluster 
>Storage averaging 130KB/file.  

This type of workload is called 'metadata-intensive'.
There are some recommendations for this type of workload:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements

Keep an eye on the section that mentions dirty-ratio?= 5 &dirty-background-ration?= 2. 


>Currently only one of the two servers is
>
>serving web services.  There are well over 100 sites, and apache 
>server-status claims around 5 hits per second, depending on time of
>day, 
>so a fair bit of logging going on.  The gluster is only holding website
>
>data and config files that will be common between the two servers, no 
>databases or anything like that on the Gluster.
>
>When the serving server is under load load average is consistently 
>12-20.  glusterfs is always at the top with 150%-250% cpu, and each of
>3 
>bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores. 
>apache processes will easily eat up all the rest of the cpus after
>that. 
>  And web page response time is underwhelming at best.
>
>Interestingly, mostly because it is not something I have ever 
>experienced before, software interrupts sit between 1 and 5 on each 
>core, but the last core is usually sitting around 20.  Have never 
>encountered a high load average where the si number was ever 
>significant.  I have googled the crap out of that (as well as gluster 
>performance in general), there are nearly limitless posts about what it
>
>is, but have yet to see one thing to explain what to do about it. 

There is an explanation  about that in the link I provided above:

Configuring a higher event threads value than the available processing units could again cause context switches on these threads. As a result reducing the number deduced from the previous step to a number that is less that the available processing units is recommended.


>Sadly 
>I can't really shut down the gluster process to confirm if that is the 
>cause, but it's a pretty good bet, I think.
>
>When the system is not under load, glusterfs will be running at around 
>100% with each of the 3 bricks around 35%, so using 2 cores when doing 
>not much of anything.
>
>nload shows the network cards rarely climb above 300 Mbps unless I am 
>doing a direct file transfer between the servers, in which case it gets
>
>right up to the 1Gbps limit.  RAM is never above 15GB unless I am 
>causing it to happen.  atop show a disk busy percentage, it is often 
>above 50% and sometimes will hit 100%, and is no where near as 
>consistently showing excessive usage like the cpu cores are.  The cpu 
>definitely seems to be the bottleneck.
>When I found out about the groups directory, I figured one of those
>must 
>be useful to me, but as best as I can tell they are not.  But I am 
>really hoping that someone has configured a system like mine and has a 
>good group file they might share for this situation, or a peak at their
>
>volume info output?
>
>or maybe this is really just about as good as I should expect?  Maybe 
>the fix is that I need more/faster cores?  I hope not, as that isn't 
>really an option.
>
>Anyway, here is my volume info as promised.
>
>root at mooglian:/Computerisms/sites/computerisms.ca/log# gluster v info
>
>Volume Name: webisms
>Type: Distributed-Replicate
>Volume ID: 261901e7-60b4-4760-897d-0163beed356e
>Status: Started
>Snapshot Count: 0
>Number of Bricks: 2 x (2 + 1) = 6
>Transport-type: tcp
>Bricks:
>Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
>Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
>Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb 
>(arbiter)
>Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
>Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
>Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb 
>(arbiter)
>Options Reconfigured:
>auth.allow: xxxx
>performance.client-io-threads: off
>nfs.disable: on
>storage.fips-mode-rchecksum: on
>transport.address-family: inet
>performance.stat-prefetch: on
>network.inode-lru-limit: 200000
>performance.write-behind-window-size: 4MB
>performance.readdir-ahead: on
>performance.io-thread-count: 64
>performance.cache-size: 8GB
>server.event-threads: 8
>client.event-threads: 8
>performance.nl-cache-timeout: 600

As 'storage.fips-mode-rchecksum' is using sha256, you can try to disable it - which should use the less cpu intensive md5. Yet, I have never played with that option ...


Check the RH page about the tunings and try different values  for the event threads.


Best Regards,
Strahil Nikolov

From archon810 at gmail.com  Tue Aug  4 04:42:45 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Mon, 3 Aug 2020 21:42:45 -0700
Subject: [Gluster-users] performance
In-Reply-To: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
Message-ID: <CAD+dzQeQCUf3tSgW6DZ24HeqA=RUHNmL1PteN1CAQmLHHGCzHg@mail.gmail.com>

I tried putting all web files (specifically WordPress php and static files
as well as various cache files) on gluster before, and the results were
miserable on a busy site - our usual ~8-10 load quickly turned into 100+
and killed everything.

I had to go back to running just the user uploads (which are static files
in the Wordpress uploads/ dir) on gluster and using rsync (via lsyncd) for
the frequently executed php / cache.

I'd love to figure this out as well and tune gluster for heavy reads and
moderate writes, but I haven't cracked that recipe yet.

On Mon, Aug 3, 2020, 8:08 PM Computerisms Corporation <bob at computerisms.ca>
wrote:

> Hi Gurus,
>
> I have been trying to wrap my head around performance improvements on my
> gluster setup, and I don't seem to be making any progress.  I mean
> forward progress.  making it worse takes practically no effort at all.
>
> My gluster is distributed-replicated across 6 bricks and 2 servers, with
> an arbiter on each server.  I designed it like this so I have an
> expansion path to more servers in the future (like the staggered arbiter
> diagram in the red hat documentation).  gluster v info output is below.
> I have compiled gluster 7.6 from sources on both servers.
>
> Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
> network connections.  They are running debian, and are being used as
> redundant web servers.  There is some 3Million files on the Gluster
> Storage averaging 130KB/file.  Currently only one of the two servers is
> serving web services.  There are well over 100 sites, and apache
> server-status claims around 5 hits per second, depending on time of day,
> so a fair bit of logging going on.  The gluster is only holding website
> data and config files that will be common between the two servers, no
> databases or anything like that on the Gluster.
>
> When the serving server is under load load average is consistently
> 12-20.  glusterfs is always at the top with 150%-250% cpu, and each of 3
> bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores.
> apache processes will easily eat up all the rest of the cpus after that.
>   And web page response time is underwhelming at best.
>
> Interestingly, mostly because it is not something I have ever
> experienced before, software interrupts sit between 1 and 5 on each
> core, but the last core is usually sitting around 20.  Have never
> encountered a high load average where the si number was ever
> significant.  I have googled the crap out of that (as well as gluster
> performance in general), there are nearly limitless posts about what it
> is, but have yet to see one thing to explain what to do about it.  Sadly
> I can't really shut down the gluster process to confirm if that is the
> cause, but it's a pretty good bet, I think.
>
> When the system is not under load, glusterfs will be running at around
> 100% with each of the 3 bricks around 35%, so using 2 cores when doing
> not much of anything.
>
> nload shows the network cards rarely climb above 300 Mbps unless I am
> doing a direct file transfer between the servers, in which case it gets
> right up to the 1Gbps limit.  RAM is never above 15GB unless I am
> causing it to happen.  atop show a disk busy percentage, it is often
> above 50% and sometimes will hit 100%, and is no where near as
> consistently showing excessive usage like the cpu cores are.  The cpu
> definitely seems to be the bottleneck.
>
> When I found out about the groups directory, I figured one of those must
> be useful to me, but as best as I can tell they are not.  But I am
> really hoping that someone has configured a system like mine and has a
> good group file they might share for this situation, or a peak at their
> volume info output?
>
> or maybe this is really just about as good as I should expect?  Maybe
> the fix is that I need more/faster cores?  I hope not, as that isn't
> really an option.
>
> Anyway, here is my volume info as promised.
>
> root at mooglian:/Computerisms/sites/computerisms.ca/log# gluster v info
>
> Volume Name: webisms
> Type: Distributed-Replicate
> Volume ID: 261901e7-60b4-4760-897d-0163beed356e
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x (2 + 1) = 6
> Transport-type: tcp
> Bricks:
> Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
> Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
> Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb
> (arbiter)
> Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
> Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
> Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb
> (arbiter)
> Options Reconfigured:
> auth.allow: xxxx
> performance.client-io-threads: off
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> performance.stat-prefetch: on
> network.inode-lru-limit: 200000
> performance.write-behind-window-size: 4MB
> performance.readdir-ahead: on
> performance.io-thread-count: 64
> performance.cache-size: 8GB
> server.event-threads: 8
> client.event-threads: 8
> performance.nl-cache-timeout: 600
>
>
> --
> Bob Miller
> Cell: 867-334-7117
> Office: 867-633-3760
> Office: 867-322-0362
> www.computerisms.ca
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200803/81d3aa66/attachment.html>

From bob at computerisms.ca  Tue Aug  4 19:47:44 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Tue, 4 Aug 2020 12:47:44 -0700
Subject: [Gluster-users] performance
In-Reply-To: <7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
Message-ID: <345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>

Hi Strahil, thanks for your response.

>>
>> I have compiled gluster 7.6 from sources on both servers.
> 
> There  is a 7.7 version which is fixing somw stuff. Why do you have to compile it from source ?

Because I have often found with other stuff in the past compiling from 
source makes a bunch of problems go away.  software generally works the 
way the developers expect it to if you use the sources, so they are 
better able to help if required.  so now I generally compile most of my 
center-piece softwares and use packages for all the supporting stuff.

> 
>> Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
>> network connections.  They are running debian, and are being used as
>> redundant web servers.  There is some 3Million files on the Gluster
>> Storage averaging 130KB/file.
> 
> This type of workload is called 'metadata-intensive'.

does this mean the metadata-cache group file would be a good one to 
enable?  will try.

waited 10 minutes, no change that I can see.

> There are some recommendations for this type of workload:
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements
> 
> Keep an eye on the section that mentions dirty-ratio?= 5 &dirty-background-ration?= 2.

I have actually read that whole manual, and specifically that page 
several times.  And also this one:

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/small_file_performance_enhancements

Perhaps I am not understanding it correctly.  I tried these suggestions 
before and it got worse, not better.  so I have been operating under the 
assumption that maybe these guidelines are not appropriate for newer 
versions.

But will try again.  adjusting the dirty ratios.

Load average went from around 15 to 35 in about 2-3 minutes, but 20 
minutes later, it is back down to 20.  It may be having a minimal 
positive impact on cpu, though, I haven't see the main glusterfs go over 
200% since I changed this, an the brick processes are hovering just 
below 50%  where they were consistently above 50% before.  Might just be 
time of day with the system not as busy.

after watching for 30 minutes, load average is fluctuating between 10 
and 30, but cpu idle appears marginally better on average than it was.

>> Interestingly, mostly because it is not something I have ever
>> experienced before, software interrupts sit between 1 and 5 on each
>> core, but the last core is usually sitting around 20.  Have never
>> encountered a high load average where the si number was ever
>> significant.  I have googled the crap out of that (as well as gluster
>> performance in general), there are nearly limitless posts about what it
>>
>> is, but have yet to see one thing to explain what to do about it.
> 
> There is an explanation  about that in the link I provided above:
> 
> Configuring a higher event threads value than the available processing units could again cause context switches on these threads. As a result reducing the number deduced from the previous step to a number that is less that the available processing units is recommended.

Okay, again, have played with these numbers before and it did not pan 
out as expected.  if I understand it correctly, I have 3 brick processes 
(glusterfsd), so the "deduced" number should be 3, and I should set it 
lower than that, so 2.  but it also says "If a specific thread consumes 
more number of CPU cycles than needed, increasing the event thread count 
would enhance the performance of the Red Hat Storage Server."  which is 
why I had it at 8.

but will set it to 2 now.  load average is at 17 to start, waiting a 
while to see what happens.

so 15 minutes later, load average is currently 12, but is fluctuating 
between 10 and 20, have seen no significant change in cpu usage or 
anything else in top.

now try also changing server.outstanding-rpc-limit to 256 and wait.

15 minutes later; load has been above 30 but is currently back down to 
12.  no significant change in cpu.  try increasing to 512 and wait.

15 minutes later, load average is 50.  no signficant difference in cpu. 
Software interrupts remain around where they were.  wa from top remains 
about where it was.  not sure why load average is climbing so high. 
changing rpc-limit to 128.

ugh.  10 minutes later, load average just popped over 100.  resetting 
rpc-limit.

now trying cluster.lookup-optimize on, lazy rebalancing (probably a bad 
idea on the live system, but how much worse can it get?)  Ya, bad idea, 
80 hours estimated to complete, load is over 50 and server is crawling. 
disabling rebalance and turning lookup-optimize off, for now.

right now the only suggested parameter I haven't played with is the 
performance.io-thread-count, which I currently have at 64.

sigh.  an hour later load average is 80 and climbing.  apache processes 
are numbering in the hundreds and I am constantly having to restart it. 
this brings load average down to 5, but as apache processes climb and 
are held open load average gets up to over 100 again with 3-4 minutes, 
and system starts going non-responsive.  rinse and repeat.

so followed all the recommendations, maybe the dirty settings had a 
small positive impact, but overall system is most definitely worse for 
having made the changes.

I have returned the configs back to how they were except the dirty 
settings and the metadata-cache group.  increased performance.cache-size 
to 16GB for now, because that is the one thing that seems to help when I 
"tune" (aka make worse) the system.  have had to restart apache a couple 
dozen times or more, but after another 30 minutes or so system has 
pretty much settled back to how it was before I started.  cpu is like I 
originally stated, all 6 cores maxed out most of the time, software 
interrupts still have all cpus running around 5 with the last one 
consistently sitting around 20-25.  Disk is busy but not usually maxed 
out.  RAM is about half used.  network load peaks at about 1/3 capacity. 
  load average is between 10 and 20.  sites are responding, but sluggish.

so am I not reading these recommendations and following the instructions 
correctly?  am I not waiting long enough after each implementation, 
should I be making 1 change per day instead of thinking 15 minutes 
should be enough for the system to catch up?  I have read the full red 
hat documentation and the significant majority of the gluster docs, 
maybe I am missing something else there?  should these settings have had 
a different effect than they did?

For what it's worth, I am running ext4 as my underlying fs and I have 
read a few times that XFS might have been a better choice.  But that is 
not a trivial experiment to make at this time with the system in 
production.  It's one thing (and still a bad thing to be sure) to 
semi-bork the system for an hour or two while I play with 
configurations, but would take a day or so offline to reformat and 
restore the data.

> 
> As 'storage.fips-mode-rchecksum' is using sha256, you can try to disable it - which should use the less cpu intensive md5. Yet, I have never played with that option ...

Done.  no signficant difference than I can see.

> Check the RH page about the tunings and try different values  for the event threads.

in the past I have tried 2, 4, 8, 16, and 32.  Playing with just those I 
never noticed that any of them made any difference.  Though I might have 
some different options now than I did then, so might try these again 
throughout the day...

Thanks again for your time Strahil, if you have any more thoughts would 
love to hear them.

> 
> 
> Best Regards,
> Strahil Nikolov
> 

From bob at computerisms.ca  Tue Aug  4 19:48:51 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Tue, 4 Aug 2020 12:48:51 -0700
Subject: [Gluster-users] performance
In-Reply-To: <CAD+dzQeQCUf3tSgW6DZ24HeqA=RUHNmL1PteN1CAQmLHHGCzHg@mail.gmail.com>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<CAD+dzQeQCUf3tSgW6DZ24HeqA=RUHNmL1PteN1CAQmLHHGCzHg@mail.gmail.com>
Message-ID: <4b507ee0-1028-f006-2fb7-461a6bc0a3ef@computerisms.ca>

Hi Artem,

would also like this recipe.  If you have any comments on my answer to 
Strahil, would love to hear them...

On 2020-08-03 9:42 p.m., Artem Russakovskii wrote:
> I tried putting all web files (specifically WordPress php and static 
> files as well as various cache files) on gluster before, and the results 
> were miserable on a busy site - our usual ~8-10 load quickly turned into 
> 100+ and killed everything.
> 
> I had to go back to running just the user uploads (which are static 
> files in the Wordpress uploads/ dir) on gluster and using rsync (via 
> lsyncd) for the frequently executed php / cache.
> 
> I'd love to figure this out as well and tune gluster for heavy reads and 
> moderate writes, but I haven't cracked that recipe yet.
> 
> On Mon, Aug 3, 2020, 8:08 PM Computerisms Corporation 
> <bob at computerisms.ca <mailto:bob at computerisms.ca>> wrote:
> 
>     Hi Gurus,
> 
>     I have been trying to wrap my head around performance improvements
>     on my
>     gluster setup, and I don't seem to be making any progress.? I mean
>     forward progress.? making it worse takes practically no effort at all.
> 
>     My gluster is distributed-replicated across 6 bricks and 2 servers,
>     with
>     an arbiter on each server.? I designed it like this so I have an
>     expansion path to more servers in the future (like the staggered
>     arbiter
>     diagram in the red hat documentation).? gluster v info output is below.
>     I have compiled gluster 7.6 from sources on both servers.
> 
>     Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and gigabit
>     network connections.? They are running debian, and are being used as
>     redundant web servers.? There is some 3Million files on the Gluster
>     Storage averaging 130KB/file.? Currently only one of the two servers is
>     serving web services.? There are well over 100 sites, and apache
>     server-status claims around 5 hits per second, depending on time of
>     day,
>     so a fair bit of logging going on.? The gluster is only holding website
>     data and config files that will be common between the two servers, no
>     databases or anything like that on the Gluster.
> 
>     When the serving server is under load load average is consistently
>     12-20.? glusterfs is always at the top with 150%-250% cpu, and each
>     of 3
>     bricks at roughly 50-70%, so consistently pegging 4 of the 6 cores.
>     apache processes will easily eat up all the rest of the cpus after
>     that.
>      ? And web page response time is underwhelming at best.
> 
>     Interestingly, mostly because it is not something I have ever
>     experienced before, software interrupts sit between 1 and 5 on each
>     core, but the last core is usually sitting around 20.? Have never
>     encountered a high load average where the si number was ever
>     significant.? I have googled the crap out of that (as well as gluster
>     performance in general), there are nearly limitless posts about what it
>     is, but have yet to see one thing to explain what to do about it. 
>     Sadly
>     I can't really shut down the gluster process to confirm if that is the
>     cause, but it's a pretty good bet, I think.
> 
>     When the system is not under load, glusterfs will be running at around
>     100% with each of the 3 bricks around 35%, so using 2 cores when doing
>     not much of anything.
> 
>     nload shows the network cards rarely climb above 300 Mbps unless I am
>     doing a direct file transfer between the servers, in which case it gets
>     right up to the 1Gbps limit.? RAM is never above 15GB unless I am
>     causing it to happen.? atop show a disk busy percentage, it is often
>     above 50% and sometimes will hit 100%, and is no where near as
>     consistently showing excessive usage like the cpu cores are.? The cpu
>     definitely seems to be the bottleneck.
> 
>     When I found out about the groups directory, I figured one of those
>     must
>     be useful to me, but as best as I can tell they are not.? But I am
>     really hoping that someone has configured a system like mine and has a
>     good group file they might share for this situation, or a peak at their
>     volume info output?
> 
>     or maybe this is really just about as good as I should expect?? Maybe
>     the fix is that I need more/faster cores?? I hope not, as that isn't
>     really an option.
> 
>     Anyway, here is my volume info as promised.
> 
>     root at mooglian:/Computerisms/sites/computerisms.ca/log#
>     <http://computerisms.ca/log#> gluster v info
> 
>     Volume Name: webisms
>     Type: Distributed-Replicate
>     Volume ID: 261901e7-60b4-4760-897d-0163beed356e
>     Status: Started
>     Snapshot Count: 0
>     Number of Bricks: 2 x (2 + 1) = 6
>     Transport-type: tcp
>     Bricks:
>     Brick1: mooglian:/var/GlusterBrick/replset-0/webisms-replset-0
>     Brick2: moogle:/var/GlusterBrick/replset-0/webisms-replset-0
>     Brick3: moogle:/var/GlusterBrick/replset-0-arb/webisms-replset-0-arb
>     (arbiter)
>     Brick4: moogle:/var/GlusterBrick/replset-1/webisms-replset-1
>     Brick5: mooglian:/var/GlusterBrick/replset-1/webisms-replset-1
>     Brick6: mooglian:/var/GlusterBrick/replset-1-arb/webisms-replset-1-arb
>     (arbiter)
>     Options Reconfigured:
>     auth.allow: xxxx
>     performance.client-io-threads: off
>     nfs.disable: on
>     storage.fips-mode-rchecksum: on
>     transport.address-family: inet
>     performance.stat-prefetch: on
>     network.inode-lru-limit: 200000
>     performance.write-behind-window-size: 4MB
>     performance.readdir-ahead: on
>     performance.io-thread-count: 64
>     performance.cache-size: 8GB
>     server.event-threads: 8
>     client.event-threads: 8
>     performance.nl-cache-timeout: 600
> 
> 
>     -- 
>     Bob Miller
>     Cell: 867-334-7117
>     Office: 867-633-3760
>     Office: 867-322-0362
>     www.computerisms.ca <http://www.computerisms.ca>
>     ________
> 
> 
> 
>     Community Meeting Calendar:
> 
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
> 
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
> 

From hunter86_bg at yahoo.com  Tue Aug  4 22:51:59 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Wed, 05 Aug 2020 01:51:59 +0300
Subject: [Gluster-users] performance
In-Reply-To: <345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
	<345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
Message-ID: <2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>



?? 4 ?????? 2020 ?. 22:47:44 GMT+03:00, Computerisms Corporation <bob at computerisms.ca> ??????:
>Hi Strahil, thanks for your response.
>
>>>
>>> I have compiled gluster 7.6 from sources on both servers.
>> 
>> There  is a 7.7 version which is fixing somw stuff. Why do you have
>to compile it from source ?
>
>Because I have often found with other stuff in the past compiling from 
>source makes a bunch of problems go away.  software generally works the
>
>way the developers expect it to if you use the sources, so they are 
>better able to help if required.  so now I generally compile most of my
>
>center-piece softwares and use packages for all the supporting stuff.

Hm...  OK. I guess you can try 7.7 whenever it's possible.

>> 
>>> Servers are 6core/3.4Ghz with 32 GB RAM, no swap, and SSD and
>gigabit
>>> network connections.  They are running debian, and are being used as
>>> redundant web servers.  There is some 3Million files on the Gluster
>>> Storage averaging 130KB/file.
>> 
>> This type of workload is called 'metadata-intensive'.
>
>does this mean the metadata-cache group file would be a good one to 
>enable?  will try.
>
>waited 10 minutes, no change that I can see.
>
>> There are some recommendations for this type of workload:
>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/small_file_performance_enhancements
>> 
>> Keep an eye on the section that mentions dirty-ratio?= 5
>&dirty-background-ration?= 2.
>
>I have actually read that whole manual, and specifically that page 
>several times.  And also this one:
>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.1/html/administration_guide/small_file_performance_enhancements
>
>Perhaps I am not understanding it correctly.  I tried these suggestions
>
>before and it got worse, not better.  so I have been operating under
>the 
>assumption that maybe these guidelines are not appropriate for newer 
>versions.

Actually, the  settings are not changed  much,  so they should work for you.

>But will try again.  adjusting the dirty ratios.
>
>Load average went from around 15 to 35 in about 2-3 minutes, but 20 
>minutes later, it is back down to 20.  It may be having a minimal 
>positive impact on cpu, though, I haven't see the main glusterfs go
>over 
>200% since I changed this, an the brick processes are hovering just 
>below 50%  where they were consistently above 50% before.  Might just
>be 
>time of day with the system not as busy.
>
>after watching for 30 minutes, load average is fluctuating between 10 
>and 30, but cpu idle appears marginally better on average than it was.
>
>>> Interestingly, mostly because it is not something I have ever
>>> experienced before, software interrupts sit between 1 and 5 on each
>>> core, but the last core is usually sitting around 20.  Have never
>>> encountered a high load average where the si number was ever
>>> significant.  I have googled the crap out of that (as well as
>gluster
>>> performance in general), there are nearly limitless posts about what
>it
>>>
>>> is, but have yet to see one thing to explain what to do about it.

This is happening on all nodes ?
I got a similar situation caused by bad NIC  (si in top was way high), but the chance for bad NIC on all servers is very low.
You can still patch OS + Firmware on your next maintenance.

>> There is an explanation  about that in the link I provided above:
>> 
>> Configuring a higher event threads value than the available
>processing units could again cause context switches on these threads.
>As a result reducing the number deduced from the previous step to a
>number that is less that the available processing units is recommended.
>
>Okay, again, have played with these numbers before and it did not pan 
>out as expected.  if I understand it correctly, I have 3 brick
>processes 
>(glusterfsd), so the "deduced" number should be 3, and I should set it 
>lower than that, so 2.  but it also says "If a specific thread consumes
>
>more number of CPU cycles than needed, increasing the event thread
>count 
>would enhance the performance of the Red Hat Storage Server."  which is
>
>why I had it at 8.

Yeah, but you got only 6 cores  and they are not dedicated for gluster only. I think that you need to test with lower values.

>but will set it to 2 now.  load average is at 17 to start, waiting a 
>while to see what happens.
>
>so 15 minutes later, load average is currently 12, but is fluctuating 
>between 10 and 20, have seen no significant change in cpu usage or 
>anything else in top.
>
>now try also changing server.outstanding-rpc-limit to 256 and wait.
>
>15 minutes later; load has been above 30 but is currently back down to 
>12.  no significant change in cpu.  try increasing to 512 and wait.
>
>15 minutes later, load average is 50.  no signficant difference in cpu.
>
>Software interrupts remain around where they were.  wa from top remains
>
>about where it was.  not sure why load average is climbing so high. 
>changing rpc-limit to 128.
>
>ugh.  10 minutes later, load average just popped over 100.  resetting 
>rpc-limit.
>
>now trying cluster.lookup-optimize on, lazy rebalancing (probably a bad
>
>idea on the live system, but how much worse can it get?)  Ya, bad idea,
>
>80 hours estimated to complete, load is over 50 and server is crawling.
>
>disabling rebalance and turning lookup-optimize off, for now.
>
>right now the only suggested parameter I haven't played with is the 
>performance.io-thread-count, which I currently have at 64.

I think that as you have SSDs only,  you might have some results by changing this one.


>sigh.  an hour later load average is 80 and climbing.  apache processes
>
>are numbering in the hundreds and I am constantly having to restart it.
>
>this brings load average down to 5, but as apache processes climb and 
>are held open load average gets up to over 100 again with 3-4 minutes, 
>and system starts going non-responsive.  rinse and repeat.
>
>so followed all the recommendations, maybe the dirty settings had a 
>small positive impact, but overall system is most definitely worse for 
>having made the changes.
>
>I have returned the configs back to how they were except the dirty 
>settings and the metadata-cache group.  increased
>performance.cache-size 
>to 16GB for now, because that is the one thing that seems to help when
>I 
>"tune" (aka make worse) the system.  have had to restart apache a
>couple 
>dozen times or more, but after another 30 minutes or so system has 
>pretty much settled back to how it was before I started.  cpu is like I
>
>originally stated, all 6 cores maxed out most of the time, software 
>interrupts still have all cpus running around 5 with the last one 
>consistently sitting around 20-25.  Disk is busy but not usually maxed 
>out.  RAM is about half used.  network load peaks at about 1/3
>capacity. 
>load average is between 10 and 20.  sites are responding, but sluggish.
>
>so am I not reading these recommendations and following the
>instructions 
>correctly?  am I not waiting long enough after each implementation, 
>should I be making 1 change per day instead of thinking 15 minutes 
>should be enough for the system to catch up?  I have read the full red 
>hat documentation and the significant majority of the gluster docs, 
>maybe I am missing something else there?  should these settings have
>had 
>a different effect than they did?
>
>For what it's worth, I am running ext4 as my underlying fs and I have 
>read a few times that XFS might have been a better choice.  But that is
>
>not a trivial experiment to make at this time with the system in 
>production.  It's one thing (and still a bad thing to be sure) to 
>semi-bork the system for an hour or two while I play with 
>configurations, but would take a day or so offline to reformat and 
>restore the data.

XFS  should bring better performance, but if the issue is not in FS ->  it won't make  a change...
What I/O scheduler are you using for the SSDs (you can check via 'cat /sys/block/sdX/queue/scheduler)?
>> 
>> As 'storage.fips-mode-rchecksum' is using sha256, you can try to
>disable it - which should use the less cpu intensive md5. Yet, I have
>never played with that option ...
>
>Done.  no signficant difference than I can see.
>
>> Check the RH page about the tunings and try different values  for the
>event threads.
>
>in the past I have tried 2, 4, 8, 16, and 32.  Playing with just those
>I 
>never noticed that any of them made any difference.  Though I might
>have 
>some different options now than I did then, so might try these again 
>throughout the day...

Are you talking about server or client event threads (or both)?

>Thanks again for your time Strahil, if you have any more thoughts would
>
>love to hear them.

Can you check if you use 'noatime' for the bricks ? It won't bring any effect on the CPU side, but it might help with the I/O.

I see that your indicator for high load  is loadavg,  but have you actually checked how many processes are in 'R' or 'D' state ?
Some  monitoring checks can raise loadavg artificially.

Also,  are you using software mirroring (either mdadm or striped/mirrored LVs )?

>> 
>> 
>> Best Regards,
>> Strahil Nikolov
>> 
>________
>
>
>
>Community Meeting Calendar:
>
>Schedule -
>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users

From bob at computerisms.ca  Wed Aug  5 01:53:34 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Tue, 4 Aug 2020 18:53:34 -0700
Subject: [Gluster-users] performance
In-Reply-To: <2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
	<345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
	<2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>
Message-ID: <64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>

Hi Strahil,

thanks again for sticking with me on this.
> Hm...  OK. I guess you can try 7.7 whenever it's possible.

Acknowledged.

>> Perhaps I am not understanding it correctly.  I tried these suggestions
>>
>> before and it got worse, not better.  so I have been operating under
>> the
>> assumption that maybe these guidelines are not appropriate for newer
>> versions.
> 
> Actually, the  settings are not changed  much,  so they should work for you.

Okay, then maybe I am doing something incorrectly, or not understanding 
some fundamental piece of things that I should be.

>>>> Interestingly, mostly because it is not something I have ever
>>>> experienced before, software interrupts sit between 1 and 5 on each
>>>> core, but the last core is usually sitting around 20.  Have never
>>>> encountered a high load average where the si number was ever
>>>> significant.  I have googled the crap out of that (as well as
>> gluster
>>>> performance in general), there are nearly limitless posts about what
>> it
>>>>
>>>> is, but have yet to see one thing to explain what to do about it.
> 
> This is happening on all nodes ?
> I got a similar situation caused by bad NIC  (si in top was way high), but the chance for bad NIC on all servers is very low.
> You can still patch OS + Firmware on your next maintenance.

Yes, but it's not to the same extreme.  The other node is currently not 
actually serving anything to the internet, so right now it's only 
function is replicated gluster and databases.  On the 2nd node there is 
also one core, the first one in this case as opposed to the last one on 
the main node, but it sits between 10 and 15 instead of 20 and 25, and 
the remaining cores will be between 0 and 2 instead of 1 and 5.

I have no evidence of any bad hardware, and these servers were both 
commissioned only within the last couple of months.  But will still poke 
around on this path.

>> more number of CPU cycles than needed, increasing the event thread
>> count
>> would enhance the performance of the Red Hat Storage Server."  which is
>>
>> why I had it at 8.
> 
> Yeah, but you got only 6 cores  and they are not dedicated for gluster only. I think that you need to test with lower values.

Okay, I will change these values a few times over the next couple of 
hours and see what happens.

>> right now the only suggested parameter I haven't played with is the
>> performance.io-thread-count, which I currently have at 64.
> 
> I think that as you have SSDs only,  you might have some results by changing this one.

Okay, will also modify this incrementally.  do you think it can go 
higher?  I think I got this number from a thread on this list, but I am 
not really sure what would be a reasonable value for my system.

>>
>> For what it's worth, I am running ext4 as my underlying fs and I have
>> read a few times that XFS might have been a better choice.  But that is
>>
>> not a trivial experiment to make at this time with the system in
>> production.  It's one thing (and still a bad thing to be sure) to
>> semi-bork the system for an hour or two while I play with
>> configurations, but would take a day or so offline to reformat and
>> restore the data.
> 
> XFS  should bring better performance, but if the issue is not in FS ->  it won't make  a change...
> What I/O scheduler are you using for the SSDs (you can check via 'cat /sys/block/sdX/queue/scheduler)?

# cat /sys/block/vda/queue/scheduler
[mq-deadline] none

>> in the past I have tried 2, 4, 8, 16, and 32.  Playing with just those
>> I
>> never noticed that any of them made any difference.  Though I might
>> have
>> some different options now than I did then, so might try these again
>> throughout the day...
> 
> Are you talking about server or client event threads (or both)?

It never occurred to me to set them to different values.  so far when I 
set one I set the other to the same value.

> 
>> Thanks again for your time Strahil, if you have any more thoughts would
>>
>> love to hear them.
> 
> Can you check if you use 'noatime' for the bricks ? It won't bring any effect on the CPU side, but it might help with the I/O.

I checked into this, and I have nodiratime set, but not noatime.  from 
what I can gather, it should provide nearly the same benefit performance 
wise while leaving the atime attribute on the files.  Never know, I may 
decide I want those at some point in the future.

> I see that your indicator for high load  is loadavg,  but have you actually checked how many processes are in 'R' or 'D' state ?
> Some  monitoring checks can raise loadavg artificially.

occasionally a batch of processes will be in R state, and I see the D 
state show up from time to time, but mostly everything is S.

> Also,  are you using software mirroring (either mdadm or striped/mirrored LVs )?

No, single disk.  And I opted to not put the gluster on a thinLVM, as I 
don't see myself using the lvm snapshots in this scenario.

So, we just moved into a quieter time of the day, but maybe I just 
stumbled onto something.  I was trying to figure out if/how I could 
throw more RAM at the problem.  gluster docs says write behind is not a 
cache unless flush-behind is on.  So seems that is a way to throw ram to 
it?  I put performance.write-behind-window-size: 512MB and 
performance.flush-behind: on and the whole system calmed down pretty 
much immediately.  could be just timing, though, will have to see 
tomorrow during business hours whether the system stays at a reasonable 
load.

I will still test the other options you suggested tonight, though, this 
is probably too good to be true.

Can't thank you enough for your input, Strahil, your help is truly 
appreciated!






> 
>>>
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users

From gilberto.nunes32 at gmail.com  Wed Aug  5 02:00:52 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Tue, 4 Aug 2020 23:00:52 -0300
Subject: [Gluster-users] Two VMS as arbiter...
Message-ID: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>

Hi there.
I have two physical servers deployed as replica 2 and, obviously, I got a
split-brain.
So I am thinking in use two virtual machines,each one in physical
servers....
Then this two VMS act as a artiber of gluster set....

Is this doable?

Thanks
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200804/49a735f1/attachment.html>

From bob at computerisms.ca  Wed Aug  5 02:47:05 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Tue, 4 Aug 2020 19:47:05 -0700
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
Message-ID: <4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>

Hi Gilberto,

My understanding is there can only be one arbiter per replicated set.  I 
don't have a lot of practice with gluster, so this could be bad advice, 
but the way I dealt with it on my two servers was to use 6 bricks as 
distributed-replicated (this is also relatively easy to migrate to 3 
servers if that happens for you in the future):

Server1     Server2
brick1      brick1.5
arbiter1.5  brick2
brick2.5    arbiter2.5

On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
> Hi there.
> I have two physical servers deployed as replica 2 and, obviously, I got 
> a split-brain.
> So I am thinking in use two virtual machines,each one in physical 
> servers....
> Then this two VMS act as a artiber of gluster set....
> 
> Is this doable?
> 
> Thanks
> 
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
> 

From gilberto.nunes32 at gmail.com  Wed Aug  5 03:25:07 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Wed, 5 Aug 2020 00:25:07 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
Message-ID: <CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>

Hi Bob!

Could you, please, send me more detail about this configuration?
I will appreciate that!

Thank you
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation <
bob at computerisms.ca> escreveu:

> Hi Gilberto,
>
> My understanding is there can only be one arbiter per replicated set.  I
> don't have a lot of practice with gluster, so this could be bad advice,
> but the way I dealt with it on my two servers was to use 6 bricks as
> distributed-replicated (this is also relatively easy to migrate to 3
> servers if that happens for you in the future):
>
> Server1     Server2
> brick1      brick1.5
> arbiter1.5  brick2
> brick2.5    arbiter2.5
>
> On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
> > Hi there.
> > I have two physical servers deployed as replica 2 and, obviously, I got
> > a split-brain.
> > So I am thinking in use two virtual machines,each one in physical
> > servers....
> > Then this two VMS act as a artiber of gluster set....
> >
> > Is this doable?
> >
> > Thanks
> >
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> >
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/8bdc4fab/attachment.html>

From bob at computerisms.ca  Wed Aug  5 05:14:28 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Tue, 4 Aug 2020 22:14:28 -0700
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
Message-ID: <6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>

check the example of the chained configuration on this page:

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes

and apply it to two servers...

On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
> Hi Bob!
> 
> Could you, please, send me more detail about this configuration?
> I will appreciate that!
> 
> Thank you
> ---
> Gilberto Nunes Ferreira
> 
> (47) 3025-5907
> **
> (47) 99676-7530 - Whatsapp / Telegram
> 
> Skype: gilberto.nunes36
> 
> 
> 
> 
> 
> Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation 
> <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
> 
>     Hi Gilberto,
> 
>     My understanding is there can only be one arbiter per replicated
>     set.? I
>     don't have a lot of practice with gluster, so this could be bad advice,
>     but the way I dealt with it on my two servers was to use 6 bricks as
>     distributed-replicated (this is also relatively easy to migrate to 3
>     servers if that happens for you in the future):
> 
>     Server1? ? ?Server2
>     brick1? ? ? brick1.5
>     arbiter1.5? brick2
>     brick2.5? ? arbiter2.5
> 
>     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>      > Hi there.
>      > I have two physical servers deployed as replica 2 and, obviously,
>     I got
>      > a split-brain.
>      > So I am thinking in use two virtual machines,each one in physical
>      > servers....
>      > Then this two VMS act as a artiber of gluster set....
>      >
>      > Is this doable?
>      >
>      > Thanks
>      >
>      > ________
>      >
>      >
>      >
>      > Community Meeting Calendar:
>      >
>      > Schedule -
>      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>      > Bridge: https://bluejeans.com/441850968
>      >
>      > Gluster-users mailing list
>      > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>      > https://lists.gluster.org/mailman/listinfo/gluster-users
>      >
>     ________
> 
> 
> 
>     Community Meeting Calendar:
> 
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
> 
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
> 

From gilberto.nunes32 at gmail.com  Wed Aug  5 11:57:10 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Wed, 5 Aug 2020 08:57:10 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
Message-ID: <CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>

hum I see... like this:
[image: image.png]
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
bob at computerisms.ca> escreveu:

> check the example of the chained configuration on this page:
>
>
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>
> and apply it to two servers...
>
> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
> > Hi Bob!
> >
> > Could you, please, send me more detail about this configuration?
> > I will appreciate that!
> >
> > Thank you
> > ---
> > Gilberto Nunes Ferreira
> >
> > (47) 3025-5907
> > **
> > (47) 99676-7530 - Whatsapp / Telegram
> >
> > Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
> >
> >     Hi Gilberto,
> >
> >     My understanding is there can only be one arbiter per replicated
> >     set.  I
> >     don't have a lot of practice with gluster, so this could be bad
> advice,
> >     but the way I dealt with it on my two servers was to use 6 bricks as
> >     distributed-replicated (this is also relatively easy to migrate to 3
> >     servers if that happens for you in the future):
> >
> >     Server1     Server2
> >     brick1      brick1.5
> >     arbiter1.5  brick2
> >     brick2.5    arbiter2.5
> >
> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
> >      > Hi there.
> >      > I have two physical servers deployed as replica 2 and, obviously,
> >     I got
> >      > a split-brain.
> >      > So I am thinking in use two virtual machines,each one in physical
> >      > servers....
> >      > Then this two VMS act as a artiber of gluster set....
> >      >
> >      > Is this doable?
> >      >
> >      > Thanks
> >      >
> >      > ________
> >      >
> >      >
> >      >
> >      > Community Meeting Calendar:
> >      >
> >      > Schedule -
> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >      > Bridge: https://bluejeans.com/441850968
> >      >
> >      > Gluster-users mailing list
> >      > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
> >      >
> >     ________
> >
> >
> >
> >     Community Meeting Calendar:
> >
> >     Schedule -
> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >     Bridge: https://bluejeans.com/441850968
> >
> >     Gluster-users mailing list
> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >     https://lists.gluster.org/mailman/listinfo/gluster-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/cba8de5e/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 54749 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/cba8de5e/attachment.png>

From mathias.waack at seim-partner.de  Wed Aug  5 13:48:52 2020
From: mathias.waack at seim-partner.de (Mathias Waack)
Date: Wed, 5 Aug 2020 15:48:52 +0200
Subject: [Gluster-users] Repair after accident
Message-ID: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>

Hi all,

we are running a gluster setup with two nodes:

Status of volume: gvol
Gluster process???????????????????????????? TCP Port? RDMA Port Online? Pid
------------------------------------------------------------------------------
Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y?????? 13350
Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y?????? 5965
Self-heal Daemon on localhost?????????????? N/A?????? N/A Y?????? 14188
Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y?????? 6003

Task Status of Volume gvol
------------------------------------------------------------------------------
There are no active volume tasks

The glusterfs hosts a bunch of containers with its data volumes. The 
underlying fs is zfs. Few days ago one of the containers created a lot 
of files in one of its data volumes, and at the end it completely filled 
up the space of the glusterfs volume. But this happened only on one 
host, on the other host there was still enough space. We finally were 
able to identify this container and found out, the sizes of the data on 
/zbrick were different on both hosts for this container. Now we made the 
big mistake to delete these files on both hosts in the /zbrick volume, 
not on the mounted glusterfs volume.

Later we found the reason for this behavior: the network driver on the 
second node partially crashed (which means we ware able to login on the 
node, so we assumed the network was running, but the card was already 
dropping packets at this time) at the same time, as the failed container 
started to fill up the gluster volume. After rebooting the second node? 
the gluster became available again.

Now the glusterfs volume is running again- but it is still (nearly) 
full: the files created by the container are not visible, but they still 
count into amount of free space. How can we fix this?

In addition there are some files which are no longer accessible since 
this accident:

tail access.log.old
tail: cannot open 'access.log.old' for reading: Input/output error

Looks like affected by this error are files which have been changed 
during the accident. Is there a way to fix this too?

Thanks
 ??? Mathias



From gilberto.nunes32 at gmail.com  Wed Aug  5 14:07:10 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Wed, 5 Aug 2020 11:07:10 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
Message-ID: <CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>

Well...
I do the follow:

gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force

And now I have:
gluster vol info

Volume Name: VMS
Type: Distributed-Replicate
Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
Status: Started
Snapshot Count: 0
Number of Bricks: 2 x (2 + 1) = 6
Transport-type: tcp
Bricks:
Brick1: pve01:/DATA/brick1
Brick2: pve02:/DATA/brick1.5
Brick3: pve01:/DATA/arbiter1.5 (arbiter)
Brick4: pve02:/DATA/brick2
Brick5: pve01:/DATA/brick2.5
Brick6: pve02:/DATA/arbiter2.5 (arbiter)
Options Reconfigured:
cluster.quorum-count: 1
cluster.quorum-reads: false
cluster.self-heal-daemon: enable
cluster.heal-timeout: 10
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

This values I have put it myself, in order to see if could improve the time
to make the volume available, when pve01 goes down with ifupdown
cluster.quorum-count: 1
cluster.quorum-reads: false
cluster.self-heal-daemon: enable
cluster.heal-timeout: 10

Nevertheless, it took more than 1 minutes to the volume VMS available in
the other host (pve02).
Is there any trick to reduce this time ?

Thanks

---
Gilberto Nunes Ferreira






Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> hum I see... like this:
> [image: image.png]
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
> bob at computerisms.ca> escreveu:
>
>> check the example of the chained configuration on this page:
>>
>>
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>>
>> and apply it to two servers...
>>
>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>> > Hi Bob!
>> >
>> > Could you, please, send me more detail about this configuration?
>> > I will appreciate that!
>> >
>> > Thank you
>> > ---
>> > Gilberto Nunes Ferreira
>> >
>> > (47) 3025-5907
>> > **
>> > (47) 99676-7530 - Whatsapp / Telegram
>> >
>> > Skype: gilberto.nunes36
>> >
>> >
>> >
>> >
>> >
>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
>> >
>> >     Hi Gilberto,
>> >
>> >     My understanding is there can only be one arbiter per replicated
>> >     set.  I
>> >     don't have a lot of practice with gluster, so this could be bad
>> advice,
>> >     but the way I dealt with it on my two servers was to use 6 bricks as
>> >     distributed-replicated (this is also relatively easy to migrate to 3
>> >     servers if that happens for you in the future):
>> >
>> >     Server1     Server2
>> >     brick1      brick1.5
>> >     arbiter1.5  brick2
>> >     brick2.5    arbiter2.5
>> >
>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>> >      > Hi there.
>> >      > I have two physical servers deployed as replica 2 and, obviously,
>> >     I got
>> >      > a split-brain.
>> >      > So I am thinking in use two virtual machines,each one in physical
>> >      > servers....
>> >      > Then this two VMS act as a artiber of gluster set....
>> >      >
>> >      > Is this doable?
>> >      >
>> >      > Thanks
>> >      >
>> >      > ________
>> >      >
>> >      >
>> >      >
>> >      > Community Meeting Calendar:
>> >      >
>> >      > Schedule -
>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >      > Bridge: https://bluejeans.com/441850968
>> >      >
>> >      > Gluster-users mailing list
>> >      > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
>> >      >
>> >     ________
>> >
>> >
>> >
>> >     Community Meeting Calendar:
>> >
>> >     Schedule -
>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >     Bridge: https://bluejeans.com/441850968
>> >
>> >     Gluster-users mailing list
>> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
>> >
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/edf300fe/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 54749 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/edf300fe/attachment.png>

From bob at computerisms.ca  Wed Aug  5 16:44:28 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Wed, 5 Aug 2020 09:44:28 -0700
Subject: [Gluster-users] performance
In-Reply-To: <64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
	<345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
	<2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>
	<64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>
Message-ID: <d3ac7f18-858c-f46b-994f-24ab8f20f648@computerisms.ca>

Hi List,

> So, we just moved into a quieter time of the day, but maybe I just 
> stumbled onto something.? I was trying to figure out if/how I could 
> throw more RAM at the problem.? gluster docs says write behind is not a 
> cache unless flush-behind is on.? So seems that is a way to throw ram to 
> it?? I put performance.write-behind-window-size: 512MB and 
> performance.flush-behind: on and the whole system calmed down pretty 
> much immediately.? could be just timing, though, will have to see 
> tomorrow during business hours whether the system stays at a reasonable 
> load.

so reporting back that this seems to have definitely had a significant 
positive effect.

So far today I have not seen the load average climb over 13 with the 
15minute average hovering around 7.  cpus are still spiking from time to 
time, but they are not staying maxed out all the time, and frequently I 
am seeing brief periods of up to 80% idle.  glusterfs process still 
spiking up to 180% or so, but consistently running around 70%, and the 
brick processes still spiking up to 70-80%, but consistently running 
around 20%.  Disk has only been above 50% in atop once so far today when 
it spiked up to 92%, and still lots of RAM left over.  So far nload even 
seems indicates I could get away with a 100Mbit network connection. 
Websites are snappy relative to what they were, still a bit sluggish on 
the first page of any given site, but tolerable or close to.  Apache 
processes are opening and closing right away, instead of stacking up.

Overall, system is performing pretty much like I would expect it to 
without gluster.  I haven't played with any of the other settings yet, 
just going to leave it like this for a day.

I have to admit I am a little bit suspicious.  I have been arguing with 
Gluster for a very long time, and I have never known it to play this 
nice.  kind feels like when your girl tells you she is "fine"; 
conversation has stopped, but you aren't really sure if it's done...

> 
> I will still test the other options you suggested tonight, though, this 
> is probably too good to be true.
> 
> Can't thank you enough for your input, Strahil, your help is truly 
> appreciated!
> 
> 
> 
> 
> 
> 
>>
>>>>
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

From gilberto.nunes32 at gmail.com  Wed Aug  5 20:41:57 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Wed, 5 Aug 2020 17:41:57 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
Message-ID: <CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>

I'm in trouble here.
When I shutdown the pve01 server, the shared folder over glusterfs is EMPTY!
It's supposed to be a qcow2 file inside it.
The content is show right, just after I power on pve01 backup...

Some advice?


Thanks

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> Well...
> I do the follow:
>
> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
>
> And now I have:
> gluster vol info
>
> Volume Name: VMS
> Type: Distributed-Replicate
> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 2 x (2 + 1) = 6
> Transport-type: tcp
> Bricks:
> Brick1: pve01:/DATA/brick1
> Brick2: pve02:/DATA/brick1.5
> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
> Brick4: pve02:/DATA/brick2
> Brick5: pve01:/DATA/brick2.5
> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
> Options Reconfigured:
> cluster.quorum-count: 1
> cluster.quorum-reads: false
> cluster.self-heal-daemon: enable
> cluster.heal-timeout: 10
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> nfs.disable: on
> performance.client-io-threads: off
>
> This values I have put it myself, in order to see if could improve the
> time to make the volume available, when pve01 goes down with ifupdown
> cluster.quorum-count: 1
> cluster.quorum-reads: false
> cluster.self-heal-daemon: enable
> cluster.heal-timeout: 10
>
> Nevertheless, it took more than 1 minutes to the volume VMS available in
> the other host (pve02).
> Is there any trick to reduce this time ?
>
> Thanks
>
> ---
> Gilberto Nunes Ferreira
>
>
>
>
>
>
> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> escreveu:
>
>> hum I see... like this:
>> [image: image.png]
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
>> bob at computerisms.ca> escreveu:
>>
>>> check the example of the chained configuration on this page:
>>>
>>>
>>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>>>
>>> and apply it to two servers...
>>>
>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>>> > Hi Bob!
>>> >
>>> > Could you, please, send me more detail about this configuration?
>>> > I will appreciate that!
>>> >
>>> > Thank you
>>> > ---
>>> > Gilberto Nunes Ferreira
>>> >
>>> > (47) 3025-5907
>>> > **
>>> > (47) 99676-7530 - Whatsapp / Telegram
>>> >
>>> > Skype: gilberto.nunes36
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
>>> >
>>> >     Hi Gilberto,
>>> >
>>> >     My understanding is there can only be one arbiter per replicated
>>> >     set.  I
>>> >     don't have a lot of practice with gluster, so this could be bad
>>> advice,
>>> >     but the way I dealt with it on my two servers was to use 6 bricks
>>> as
>>> >     distributed-replicated (this is also relatively easy to migrate to
>>> 3
>>> >     servers if that happens for you in the future):
>>> >
>>> >     Server1     Server2
>>> >     brick1      brick1.5
>>> >     arbiter1.5  brick2
>>> >     brick2.5    arbiter2.5
>>> >
>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>>> >      > Hi there.
>>> >      > I have two physical servers deployed as replica 2 and,
>>> obviously,
>>> >     I got
>>> >      > a split-brain.
>>> >      > So I am thinking in use two virtual machines,each one in
>>> physical
>>> >      > servers....
>>> >      > Then this two VMS act as a artiber of gluster set....
>>> >      >
>>> >      > Is this doable?
>>> >      >
>>> >      > Thanks
>>> >      >
>>> >      > ________
>>> >      >
>>> >      >
>>> >      >
>>> >      > Community Meeting Calendar:
>>> >      >
>>> >      > Schedule -
>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >      > Bridge: https://bluejeans.com/441850968
>>> >      >
>>> >      > Gluster-users mailing list
>>> >      > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >      >
>>> >     ________
>>> >
>>> >
>>> >
>>> >     Community Meeting Calendar:
>>> >
>>> >     Schedule -
>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >     Bridge: https://bluejeans.com/441850968
>>> >
>>> >     Gluster-users mailing list
>>> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/746ac4bd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image.png
Type: image/png
Size: 54749 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/746ac4bd/attachment.png>

From hunter86_bg at yahoo.com  Wed Aug  5 20:54:37 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Wed, 05 Aug 2020 23:54:37 +0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
Message-ID: <1EF050E2-FDB5-42DE-BF6B-4AA08997CB4B@yahoo.com>

If I understood you correctly, you are looking for this:
Option: network.ping-timeout                    Default Value: 42                               Description: Time duration for which the client waits to check if the server is responsive.

Best Regards,
Strahil Nikolov


?? 5 ?????? 2020 ?. 17:07:10 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>Well...
>I do the follow:
>
>gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
>pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
>e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
>
>And now I have:
>gluster vol info
>
>Volume Name: VMS
>Type: Distributed-Replicate
>Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
>Status: Started
>Snapshot Count: 0
>Number of Bricks: 2 x (2 + 1) = 6
>Transport-type: tcp
>Bricks:
>Brick1: pve01:/DATA/brick1
>Brick2: pve02:/DATA/brick1.5
>Brick3: pve01:/DATA/arbiter1.5 (arbiter)
>Brick4: pve02:/DATA/brick2
>Brick5: pve01:/DATA/brick2.5
>Brick6: pve02:/DATA/arbiter2.5 (arbiter)
>Options Reconfigured:
>cluster.quorum-count: 1
>cluster.quorum-reads: false
>cluster.self-heal-daemon: enable
>cluster.heal-timeout: 10
>storage.fips-mode-rchecksum: on
>transport.address-family: inet
>nfs.disable: on
>performance.client-io-threads: off
>
>This values I have put it myself, in order to see if could improve the
>time
>to make the volume available, when pve01 goes down with ifupdown
>cluster.quorum-count: 1
>cluster.quorum-reads: false
>cluster.self-heal-daemon: enable
>cluster.heal-timeout: 10
>
>Nevertheless, it took more than 1 minutes to the volume VMS available
>in
>the other host (pve02).
>Is there any trick to reduce this time ?
>
>Thanks
>
>---
>Gilberto Nunes Ferreira
>
>
>
>
>
>
>Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
>gilberto.nunes32 at gmail.com> escreveu:
>
>> hum I see... like this:
>> [image: image.png]
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
>> bob at computerisms.ca> escreveu:
>>
>>> check the example of the chained configuration on this page:
>>>
>>>
>>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>>>
>>> and apply it to two servers...
>>>
>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>>> > Hi Bob!
>>> >
>>> > Could you, please, send me more detail about this configuration?
>>> > I will appreciate that!
>>> >
>>> > Thank you
>>> > ---
>>> > Gilberto Nunes Ferreira
>>> >
>>> > (47) 3025-5907
>>> > **
>>> > (47) 99676-7530 - Whatsapp / Telegram
>>> >
>>> > Skype: gilberto.nunes36
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
>>> >
>>> >     Hi Gilberto,
>>> >
>>> >     My understanding is there can only be one arbiter per
>replicated
>>> >     set.  I
>>> >     don't have a lot of practice with gluster, so this could be
>bad
>>> advice,
>>> >     but the way I dealt with it on my two servers was to use 6
>bricks as
>>> >     distributed-replicated (this is also relatively easy to
>migrate to 3
>>> >     servers if that happens for you in the future):
>>> >
>>> >     Server1     Server2
>>> >     brick1      brick1.5
>>> >     arbiter1.5  brick2
>>> >     brick2.5    arbiter2.5
>>> >
>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>>> >      > Hi there.
>>> >      > I have two physical servers deployed as replica 2 and,
>obviously,
>>> >     I got
>>> >      > a split-brain.
>>> >      > So I am thinking in use two virtual machines,each one in
>physical
>>> >      > servers....
>>> >      > Then this two VMS act as a artiber of gluster set....
>>> >      >
>>> >      > Is this doable?
>>> >      >
>>> >      > Thanks
>>> >      >
>>> >      > ________
>>> >      >
>>> >      >
>>> >      >
>>> >      > Community Meeting Calendar:
>>> >      >
>>> >      > Schedule -
>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >      > Bridge: https://bluejeans.com/441850968
>>> >      >
>>> >      > Gluster-users mailing list
>>> >      > Gluster-users at gluster.org
><mailto:Gluster-users at gluster.org>
>>> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >      >
>>> >     ________
>>> >
>>> >
>>> >
>>> >     Community Meeting Calendar:
>>> >
>>> >     Schedule -
>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >     Bridge: https://bluejeans.com/441850968
>>> >
>>> >     Gluster-users mailing list
>>> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >
>>>
>>

From hunter86_bg at yahoo.com  Wed Aug  5 21:07:53 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 06 Aug 2020 00:07:53 +0300
Subject: [Gluster-users] performance
In-Reply-To: <64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
	<345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
	<2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>
	<64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>
Message-ID: <68274322-B514-4555-A236-D159B16D42FC@yahoo.com>



?? 5 ?????? 2020 ?. 4:53:34 GMT+03:00, Computerisms Corporation <bob at computerisms.ca> ??????:
>Hi Strahil,
>
>thanks again for sticking with me on this.
>> Hm...  OK. I guess you can try 7.7 whenever it's possible.
>
>Acknowledged.
>
>>> Perhaps I am not understanding it correctly.  I tried these
>suggestions
>>>
>>> before and it got worse, not better.  so I have been operating under
>>> the
>>> assumption that maybe these guidelines are not appropriate for newer
>>> versions.
>> 
>> Actually, the  settings are not changed  much,  so they should work
>for you.
>
>Okay, then maybe I am doing something incorrectly, or not understanding
>
>some fundamental piece of things that I should be.

To be honest, the documentation seems pretty useless to me.

>>>>> Interestingly, mostly because it is not something I have ever
>>>>> experienced before, software interrupts sit between 1 and 5 on
>each
>>>>> core, but the last core is usually sitting around 20.  Have never
>>>>> encountered a high load average where the si number was ever
>>>>> significant.  I have googled the crap out of that (as well as
>>> gluster
>>>>> performance in general), there are nearly limitless posts about
>what
>>> it
>>>>>
>>>>> is, but have yet to see one thing to explain what to do about it.
>> 
>> This is happening on all nodes ?
>> I got a similar situation caused by bad NIC  (si in top was way
>high), but the chance for bad NIC on all servers is very low.
>> You can still patch OS + Firmware on your next maintenance.
>
>Yes, but it's not to the same extreme.  The other node is currently not
>
>actually serving anything to the internet, so right now it's only 
>function is replicated gluster and databases.  On the 2nd node there is
>
>also one core, the first one in this case as opposed to the last one on
>
>the main node, but it sits between 10 and 15 instead of 20 and 25, and 
>the remaining cores will be between 0 and 2 instead of 1 and 5.


>I have no evidence of any bad hardware, and these servers were both 
>commissioned only within the last couple of months.  But will still
>poke 
>around on this path.

It could be a bad firmware also. If you get the opportunity,  flash the firmware and bump the OS to the max.


>>> more number of CPU cycles than needed, increasing the event thread
>>> count
>>> would enhance the performance of the Red Hat Storage Server."  which
>is
>>>
>>> why I had it at 8.
>> 
>> Yeah, but you got only 6 cores  and they are not dedicated for
>gluster only. I think that you need to test with lower values.
>
>Okay, I will change these values a few times over the next couple of 
>hours and see what happens.
>
>>> right now the only suggested parameter I haven't played with is the
>>> performance.io-thread-count, which I currently have at 64.
>> 
>> I think that as you have SSDs only,  you might have some results by
>changing this one.
>
>Okay, will also modify this incrementally.  do you think it can go 
>higher?  I think I got this number from a thread on this list, but I am
>
>not really sure what would be a reasonable value for my system.

I guess you can try to increase it a little bit and check how is it going.

>>>
>>> For what it's worth, I am running ext4 as my underlying fs and I
>have
>>> read a few times that XFS might have been a better choice.  But that
>is
>>>
>>> not a trivial experiment to make at this time with the system in
>>> production.  It's one thing (and still a bad thing to be sure) to
>>> semi-bork the system for an hour or two while I play with
>>> configurations, but would take a day or so offline to reformat and
>>> restore the data.
>> 
>> XFS  should bring better performance, but if the issue is not in FS
>->  it won't make  a change...
>> What I/O scheduler are you using for the SSDs (you can check via 'cat
>/sys/block/sdX/queue/scheduler)?
>
># cat /sys/block/vda/queue/scheduler
>[mq-deadline] none

Deadline prioritizes  reads in a 2:1 ratio /default tunings/ . You can consider testing 'none' if your SSDs are good.

I see vda , please share details on the infra as this is very important. Virtual disks have their limitations and if you are on a VM,  then there might be chance to increase the CPU count.
If you are on a VM, I would recommend you to use more (in numbers)  and smaller disks in stripe sets (either raid0  via mdadm,  or pure striped LV).
Also, if you are  on a VM -> there is no reason to reorder  your I/O requests  in the VM, just to do  it again on the Hypervisour. In such case 'none' can bring better performance,  but this varies on the workload.



>>> in the past I have tried 2, 4, 8, 16, and 32.  Playing with just
>those
>>> I
>>> never noticed that any of them made any difference.  Though I might
>>> have
>>> some different options now than I did then, so might try these again
>>> throughout the day...
>> 
>> Are you talking about server or client event threads (or both)?
>
>It never occurred to me to set them to different values.  so far when I
>
>set one I set the other to the same value.

Yeah, this makes sense.

>> 
>>> Thanks again for your time Strahil, if you have any more thoughts
>would
>>>
>>> love to hear them.
>> 
>> Can you check if you use 'noatime' for the bricks ? It won't bring
>any effect on the CPU side, but it might help with the I/O.
>
>I checked into this, and I have nodiratime set, but not noatime.  from 
>what I can gather, it should provide nearly the same benefit
>performance 
>wise while leaving the atime attribute on the files.  Never know, I may
>
>decide I want those at some point in the future.

All necessary data is in the file attributes on the brick. I doubt you will need to have access times on the brick itself. Another possibility is to use 'relatime'.

>> I see that your indicator for high load  is loadavg,  but have you
>actually checked how many processes are in 'R' or 'D' state ?
>> Some  monitoring checks can raise loadavg artificially.
>
>occasionally a batch of processes will be in R state, and I see the D 
>state show up from time to time, but mostly everything is S.
>
>> Also,  are you using software mirroring (either mdadm or
>striped/mirrored LVs )?
>
>No, single disk.  And I opted to not put the gluster on a thinLVM, as I
>
>don't see myself using the lvm snapshots in this scenario.
>
>So, we just moved into a quieter time of the day, but maybe I just 
>stumbled onto something.  I was trying to figure out if/how I could 
>throw more RAM at the problem.  gluster docs says write behind is not a
>
>cache unless flush-behind is on.  So seems that is a way to throw ram
>to 
>it?  I put performance.write-behind-window-size: 512MB and 
>performance.flush-behind: on and the whole system calmed down pretty 
>much immediately.  could be just timing, though, will have to see 
>tomorrow during business hours whether the system stays at a reasonable
>
>load.
>
>I will still test the other options you suggested tonight, though, this
>
>is probably too good to be true.
>
>Can't thank you enough for your input, Strahil, your help is truly 
>appreciated!
>
>
>
>
>
>
>> 
>>>>
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>________
>
>
>
>Community Meeting Calendar:
>
>Schedule -
>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users

From hunter86_bg at yahoo.com  Wed Aug  5 21:15:23 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 06 Aug 2020 00:15:23 +0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
Message-ID: <77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>

This could happen if you have pending heals. Did you reboot that node recently ?
Did you set automatic unsplit-brain ?

Check for pending heals and files in splitbrain.

If not, you can check https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/ (look at point 5).

Best Regards,
Strahil Nikolov

?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>I'm in trouble here.
>When I shutdown the pve01 server, the shared folder over glusterfs is
>EMPTY!
>It's supposed to be a qcow2 file inside it.
>The content is show right, just after I power on pve01 backup...
>
>Some advice?
>
>
>Thanks
>
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
>gilberto.nunes32 at gmail.com> escreveu:
>
>> Well...
>> I do the follow:
>>
>> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
>> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
>> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
>>
>> And now I have:
>> gluster vol info
>>
>> Volume Name: VMS
>> Type: Distributed-Replicate
>> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 2 x (2 + 1) = 6
>> Transport-type: tcp
>> Bricks:
>> Brick1: pve01:/DATA/brick1
>> Brick2: pve02:/DATA/brick1.5
>> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
>> Brick4: pve02:/DATA/brick2
>> Brick5: pve01:/DATA/brick2.5
>> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
>> Options Reconfigured:
>> cluster.quorum-count: 1
>> cluster.quorum-reads: false
>> cluster.self-heal-daemon: enable
>> cluster.heal-timeout: 10
>> storage.fips-mode-rchecksum: on
>> transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: off
>>
>> This values I have put it myself, in order to see if could improve
>the
>> time to make the volume available, when pve01 goes down with ifupdown
>> cluster.quorum-count: 1
>> cluster.quorum-reads: false
>> cluster.self-heal-daemon: enable
>> cluster.heal-timeout: 10
>>
>> Nevertheless, it took more than 1 minutes to the volume VMS available
>in
>> the other host (pve02).
>> Is there any trick to reduce this time ?
>>
>> Thanks
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>>
>>
>>
>>
>>
>> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> escreveu:
>>
>>> hum I see... like this:
>>> [image: image.png]
>>> ---
>>> Gilberto Nunes Ferreira
>>>
>>> (47) 3025-5907
>>> (47) 99676-7530 - Whatsapp / Telegram
>>>
>>> Skype: gilberto.nunes36
>>>
>>>
>>>
>>>
>>>
>>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
>>> bob at computerisms.ca> escreveu:
>>>
>>>> check the example of the chained configuration on this page:
>>>>
>>>>
>>>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>>>>
>>>> and apply it to two servers...
>>>>
>>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>>>> > Hi Bob!
>>>> >
>>>> > Could you, please, send me more detail about this configuration?
>>>> > I will appreciate that!
>>>> >
>>>> > Thank you
>>>> > ---
>>>> > Gilberto Nunes Ferreira
>>>> >
>>>> > (47) 3025-5907
>>>> > **
>>>> > (47) 99676-7530 - Whatsapp / Telegram
>>>> >
>>>> > Skype: gilberto.nunes36
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
>>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
>>>> >
>>>> >     Hi Gilberto,
>>>> >
>>>> >     My understanding is there can only be one arbiter per
>replicated
>>>> >     set.  I
>>>> >     don't have a lot of practice with gluster, so this could be
>bad
>>>> advice,
>>>> >     but the way I dealt with it on my two servers was to use 6
>bricks
>>>> as
>>>> >     distributed-replicated (this is also relatively easy to
>migrate to
>>>> 3
>>>> >     servers if that happens for you in the future):
>>>> >
>>>> >     Server1     Server2
>>>> >     brick1      brick1.5
>>>> >     arbiter1.5  brick2
>>>> >     brick2.5    arbiter2.5
>>>> >
>>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>>>> >      > Hi there.
>>>> >      > I have two physical servers deployed as replica 2 and,
>>>> obviously,
>>>> >     I got
>>>> >      > a split-brain.
>>>> >      > So I am thinking in use two virtual machines,each one in
>>>> physical
>>>> >      > servers....
>>>> >      > Then this two VMS act as a artiber of gluster set....
>>>> >      >
>>>> >      > Is this doable?
>>>> >      >
>>>> >      > Thanks
>>>> >      >
>>>> >      > ________
>>>> >      >
>>>> >      >
>>>> >      >
>>>> >      > Community Meeting Calendar:
>>>> >      >
>>>> >      > Schedule -
>>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> >      > Bridge: https://bluejeans.com/441850968
>>>> >      >
>>>> >      > Gluster-users mailing list
>>>> >      > Gluster-users at gluster.org
><mailto:Gluster-users at gluster.org>
>>>> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
>>>> >      >
>>>> >     ________
>>>> >
>>>> >
>>>> >
>>>> >     Community Meeting Calendar:
>>>> >
>>>> >     Schedule -
>>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> >     Bridge: https://bluejeans.com/441850968
>>>> >
>>>> >     Gluster-users mailing list
>>>> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
>>>> >
>>>>
>>>

From gilberto.nunes32 at gmail.com  Wed Aug  5 22:56:58 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Wed, 5 Aug 2020 19:56:58 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
	<77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
Message-ID: <CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>

Ok...Thanks a lot Strahil

This gluster volume set VMS cluster.favorite-child-policy size do the trick
to me here!

Cheers
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qua., 5 de ago. de 2020 ?s 18:15, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> This could happen if you have pending heals. Did you reboot that node
> recently ?
> Did you set automatic unsplit-brain ?
>
> Check for pending heals and files in splitbrain.
>
> If not, you can check
> https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
> (look at point 5).
>
> Best Regards,
> Strahil Nikolov
>
> ?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >I'm in trouble here.
> >When I shutdown the pve01 server, the shared folder over glusterfs is
> >EMPTY!
> >It's supposed to be a qcow2 file inside it.
> >The content is show right, just after I power on pve01 backup...
> >
> >Some advice?
> >
> >
> >Thanks
> >
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
> >gilberto.nunes32 at gmail.com> escreveu:
> >
> >> Well...
> >> I do the follow:
> >>
> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
> >>
> >> And now I have:
> >> gluster vol info
> >>
> >> Volume Name: VMS
> >> Type: Distributed-Replicate
> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
> >> Status: Started
> >> Snapshot Count: 0
> >> Number of Bricks: 2 x (2 + 1) = 6
> >> Transport-type: tcp
> >> Bricks:
> >> Brick1: pve01:/DATA/brick1
> >> Brick2: pve02:/DATA/brick1.5
> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
> >> Brick4: pve02:/DATA/brick2
> >> Brick5: pve01:/DATA/brick2.5
> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
> >> Options Reconfigured:
> >> cluster.quorum-count: 1
> >> cluster.quorum-reads: false
> >> cluster.self-heal-daemon: enable
> >> cluster.heal-timeout: 10
> >> storage.fips-mode-rchecksum: on
> >> transport.address-family: inet
> >> nfs.disable: on
> >> performance.client-io-threads: off
> >>
> >> This values I have put it myself, in order to see if could improve
> >the
> >> time to make the volume available, when pve01 goes down with ifupdown
> >> cluster.quorum-count: 1
> >> cluster.quorum-reads: false
> >> cluster.self-heal-daemon: enable
> >> cluster.heal-timeout: 10
> >>
> >> Nevertheless, it took more than 1 minutes to the volume VMS available
> >in
> >> the other host (pve02).
> >> Is there any trick to reduce this time ?
> >>
> >> Thanks
> >>
> >> ---
> >> Gilberto Nunes Ferreira
> >>
> >>
> >>
> >>
> >>
> >>
> >> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
> >> gilberto.nunes32 at gmail.com> escreveu:
> >>
> >>> hum I see... like this:
> >>> [image: image.png]
> >>> ---
> >>> Gilberto Nunes Ferreira
> >>>
> >>> (47) 3025-5907
> >>> (47) 99676-7530 - Whatsapp / Telegram
> >>>
> >>> Skype: gilberto.nunes36
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
> >>> bob at computerisms.ca> escreveu:
> >>>
> >>>> check the example of the chained configuration on this page:
> >>>>
> >>>>
> >>>>
> >
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
> >>>>
> >>>> and apply it to two servers...
> >>>>
> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
> >>>> > Hi Bob!
> >>>> >
> >>>> > Could you, please, send me more detail about this configuration?
> >>>> > I will appreciate that!
> >>>> >
> >>>> > Thank you
> >>>> > ---
> >>>> > Gilberto Nunes Ferreira
> >>>> >
> >>>> > (47) 3025-5907
> >>>> > **
> >>>> > (47) 99676-7530 - Whatsapp / Telegram
> >>>> >
> >>>> > Skype: gilberto.nunes36
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
> >>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
> >>>> >
> >>>> >     Hi Gilberto,
> >>>> >
> >>>> >     My understanding is there can only be one arbiter per
> >replicated
> >>>> >     set.  I
> >>>> >     don't have a lot of practice with gluster, so this could be
> >bad
> >>>> advice,
> >>>> >     but the way I dealt with it on my two servers was to use 6
> >bricks
> >>>> as
> >>>> >     distributed-replicated (this is also relatively easy to
> >migrate to
> >>>> 3
> >>>> >     servers if that happens for you in the future):
> >>>> >
> >>>> >     Server1     Server2
> >>>> >     brick1      brick1.5
> >>>> >     arbiter1.5  brick2
> >>>> >     brick2.5    arbiter2.5
> >>>> >
> >>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
> >>>> >      > Hi there.
> >>>> >      > I have two physical servers deployed as replica 2 and,
> >>>> obviously,
> >>>> >     I got
> >>>> >      > a split-brain.
> >>>> >      > So I am thinking in use two virtual machines,each one in
> >>>> physical
> >>>> >      > servers....
> >>>> >      > Then this two VMS act as a artiber of gluster set....
> >>>> >      >
> >>>> >      > Is this doable?
> >>>> >      >
> >>>> >      > Thanks
> >>>> >      >
> >>>> >      > ________
> >>>> >      >
> >>>> >      >
> >>>> >      >
> >>>> >      > Community Meeting Calendar:
> >>>> >      >
> >>>> >      > Schedule -
> >>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>>> >      > Bridge: https://bluejeans.com/441850968
> >>>> >      >
> >>>> >      > Gluster-users mailing list
> >>>> >      > Gluster-users at gluster.org
> ><mailto:Gluster-users at gluster.org>
> >>>> >      > https://lists.gluster.org/mailman/listinfo/gluster-users
> >>>> >      >
> >>>> >     ________
> >>>> >
> >>>> >
> >>>> >
> >>>> >     Community Meeting Calendar:
> >>>> >
> >>>> >     Schedule -
> >>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>>> >     Bridge: https://bluejeans.com/441850968
> >>>> >
> >>>> >     Gluster-users mailing list
> >>>> >     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
> >>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
> >>>> >
> >>>>
> >>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/ff3596ed/attachment.html>

From archon810 at gmail.com  Thu Aug  6 00:28:44 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Wed, 5 Aug 2020 17:28:44 -0700
Subject: [Gluster-users] performance
In-Reply-To: <d3ac7f18-858c-f46b-994f-24ab8f20f648@computerisms.ca>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
	<345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
	<2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>
	<64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>
	<d3ac7f18-858c-f46b-994f-24ab8f20f648@computerisms.ca>
Message-ID: <CAD+dzQc4b9qe+=5jNksw-u4B1VZtFPzBCkA=3AkdwSBp998Zzw@mail.gmail.com>

I'm very curious whether these improvements hold up over the next few days.
Please report back.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation <bob at computerisms.ca>
wrote:

> Hi List,
>
> > So, we just moved into a quieter time of the day, but maybe I just
> > stumbled onto something.  I was trying to figure out if/how I could
> > throw more RAM at the problem.  gluster docs says write behind is not a
> > cache unless flush-behind is on.  So seems that is a way to throw ram to
> > it?  I put performance.write-behind-window-size: 512MB and
> > performance.flush-behind: on and the whole system calmed down pretty
> > much immediately.  could be just timing, though, will have to see
> > tomorrow during business hours whether the system stays at a reasonable
> > load.
>
> so reporting back that this seems to have definitely had a significant
> positive effect.
>
> So far today I have not seen the load average climb over 13 with the
> 15minute average hovering around 7.  cpus are still spiking from time to
> time, but they are not staying maxed out all the time, and frequently I
> am seeing brief periods of up to 80% idle.  glusterfs process still
> spiking up to 180% or so, but consistently running around 70%, and the
> brick processes still spiking up to 70-80%, but consistently running
> around 20%.  Disk has only been above 50% in atop once so far today when
> it spiked up to 92%, and still lots of RAM left over.  So far nload even
> seems indicates I could get away with a 100Mbit network connection.
> Websites are snappy relative to what they were, still a bit sluggish on
> the first page of any given site, but tolerable or close to.  Apache
> processes are opening and closing right away, instead of stacking up.
>
> Overall, system is performing pretty much like I would expect it to
> without gluster.  I haven't played with any of the other settings yet,
> just going to leave it like this for a day.
>
> I have to admit I am a little bit suspicious.  I have been arguing with
> Gluster for a very long time, and I have never known it to play this
> nice.  kind feels like when your girl tells you she is "fine";
> conversation has stopped, but you aren't really sure if it's done...
>
> >
> > I will still test the other options you suggested tonight, though, this
> > is probably too good to be true.
> >
> > Can't thank you enough for your input, Strahil, your help is truly
> > appreciated!
> >
> >
> >
> >
> >
> >
> >>
> >>>>
> >>>>
> >>>> Best Regards,
> >>>> Strahil Nikolov
> >>>>
> >>> ________
> >>>
> >>>
> >>>
> >>> Community Meeting Calendar:
> >>>
> >>> Schedule -
> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> Bridge: https://bluejeans.com/441850968
> >>>
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200805/1346beb6/attachment.html>

From hunter86_bg at yahoo.com  Thu Aug  6 04:08:51 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 06 Aug 2020 07:08:51 +0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
	<77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
	<CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>
Message-ID: <EA7AFBC2-38CF-4000-827B-8E0E31797FAC@yahoo.com>

As  you mentioned qcow2  files,  check the virt group (/var/lib/glusterfs/group or something like that). It has optimal setttins for VMs and is used by oVirt.

WARNING: If you decide to enable the group, which will also enable sharding, NEVER EVER DISABLE SHARDING -> ONCE ENABLED STAYS ENABLED !!!
Sharding helps reduce loocking during replica heals.

WARNING2: As virt group uses sharding (fixes the size of file into shard size),  you should consider cluster.favorite-child-policy with value ctime/mtime.

Best Regards,
Strahil Nikolov

?? 6 ?????? 2020 ?. 1:56:58 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>Ok...Thanks a lot Strahil
>
>This gluster volume set VMS cluster.favorite-child-policy size do the
>trick
>to me here!
>
>Cheers
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qua., 5 de ago. de 2020 ?s 18:15, Strahil Nikolov
><hunter86_bg at yahoo.com>
>escreveu:
>
>> This could happen if you have pending heals. Did you reboot that node
>> recently ?
>> Did you set automatic unsplit-brain ?
>>
>> Check for pending heals and files in splitbrain.
>>
>> If not, you can check
>>
>https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
>> (look at point 5).
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >I'm in trouble here.
>> >When I shutdown the pve01 server, the shared folder over glusterfs
>is
>> >EMPTY!
>> >It's supposed to be a qcow2 file inside it.
>> >The content is show right, just after I power on pve01 backup...
>> >
>> >Some advice?
>> >
>> >
>> >Thanks
>> >
>> >---
>> >Gilberto Nunes Ferreira
>> >
>> >(47) 3025-5907
>> >(47) 99676-7530 - Whatsapp / Telegram
>> >
>> >Skype: gilberto.nunes36
>> >
>> >
>> >
>> >
>> >
>> >Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
>> >gilberto.nunes32 at gmail.com> escreveu:
>> >
>> >> Well...
>> >> I do the follow:
>> >>
>> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
>> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
>> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
>> >>
>> >> And now I have:
>> >> gluster vol info
>> >>
>> >> Volume Name: VMS
>> >> Type: Distributed-Replicate
>> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
>> >> Status: Started
>> >> Snapshot Count: 0
>> >> Number of Bricks: 2 x (2 + 1) = 6
>> >> Transport-type: tcp
>> >> Bricks:
>> >> Brick1: pve01:/DATA/brick1
>> >> Brick2: pve02:/DATA/brick1.5
>> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
>> >> Brick4: pve02:/DATA/brick2
>> >> Brick5: pve01:/DATA/brick2.5
>> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
>> >> Options Reconfigured:
>> >> cluster.quorum-count: 1
>> >> cluster.quorum-reads: false
>> >> cluster.self-heal-daemon: enable
>> >> cluster.heal-timeout: 10
>> >> storage.fips-mode-rchecksum: on
>> >> transport.address-family: inet
>> >> nfs.disable: on
>> >> performance.client-io-threads: off
>> >>
>> >> This values I have put it myself, in order to see if could improve
>> >the
>> >> time to make the volume available, when pve01 goes down with
>ifupdown
>> >> cluster.quorum-count: 1
>> >> cluster.quorum-reads: false
>> >> cluster.self-heal-daemon: enable
>> >> cluster.heal-timeout: 10
>> >>
>> >> Nevertheless, it took more than 1 minutes to the volume VMS
>available
>> >in
>> >> the other host (pve02).
>> >> Is there any trick to reduce this time ?
>> >>
>> >> Thanks
>> >>
>> >> ---
>> >> Gilberto Nunes Ferreira
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
>> >> gilberto.nunes32 at gmail.com> escreveu:
>> >>
>> >>> hum I see... like this:
>> >>> [image: image.png]
>> >>> ---
>> >>> Gilberto Nunes Ferreira
>> >>>
>> >>> (47) 3025-5907
>> >>> (47) 99676-7530 - Whatsapp / Telegram
>> >>>
>> >>> Skype: gilberto.nunes36
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
>> >>> bob at computerisms.ca> escreveu:
>> >>>
>> >>>> check the example of the chained configuration on this page:
>> >>>>
>> >>>>
>> >>>>
>> >
>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>> >>>>
>> >>>> and apply it to two servers...
>> >>>>
>> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>> >>>> > Hi Bob!
>> >>>> >
>> >>>> > Could you, please, send me more detail about this
>configuration?
>> >>>> > I will appreciate that!
>> >>>> >
>> >>>> > Thank you
>> >>>> > ---
>> >>>> > Gilberto Nunes Ferreira
>> >>>> >
>> >>>> > (47) 3025-5907
>> >>>> > **
>> >>>> > (47) 99676-7530 - Whatsapp / Telegram
>> >>>> >
>> >>>> > Skype: gilberto.nunes36
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
>> >>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
>> >>>> >
>> >>>> >     Hi Gilberto,
>> >>>> >
>> >>>> >     My understanding is there can only be one arbiter per
>> >replicated
>> >>>> >     set.  I
>> >>>> >     don't have a lot of practice with gluster, so this could
>be
>> >bad
>> >>>> advice,
>> >>>> >     but the way I dealt with it on my two servers was to use 6
>> >bricks
>> >>>> as
>> >>>> >     distributed-replicated (this is also relatively easy to
>> >migrate to
>> >>>> 3
>> >>>> >     servers if that happens for you in the future):
>> >>>> >
>> >>>> >     Server1     Server2
>> >>>> >     brick1      brick1.5
>> >>>> >     arbiter1.5  brick2
>> >>>> >     brick2.5    arbiter2.5
>> >>>> >
>> >>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>> >>>> >      > Hi there.
>> >>>> >      > I have two physical servers deployed as replica 2 and,
>> >>>> obviously,
>> >>>> >     I got
>> >>>> >      > a split-brain.
>> >>>> >      > So I am thinking in use two virtual machines,each one
>in
>> >>>> physical
>> >>>> >      > servers....
>> >>>> >      > Then this two VMS act as a artiber of gluster set....
>> >>>> >      >
>> >>>> >      > Is this doable?
>> >>>> >      >
>> >>>> >      > Thanks
>> >>>> >      >
>> >>>> >      > ________
>> >>>> >      >
>> >>>> >      >
>> >>>> >      >
>> >>>> >      > Community Meeting Calendar:
>> >>>> >      >
>> >>>> >      > Schedule -
>> >>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >>>> >      > Bridge: https://bluejeans.com/441850968
>> >>>> >      >
>> >>>> >      > Gluster-users mailing list
>> >>>> >      > Gluster-users at gluster.org
>> ><mailto:Gluster-users at gluster.org>
>> >>>> >      >
>https://lists.gluster.org/mailman/listinfo/gluster-users
>> >>>> >      >
>> >>>> >     ________
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >     Community Meeting Calendar:
>> >>>> >
>> >>>> >     Schedule -
>> >>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >>>> >     Bridge: https://bluejeans.com/441850968
>> >>>> >
>> >>>> >     Gluster-users mailing list
>> >>>> >     Gluster-users at gluster.org
><mailto:Gluster-users at gluster.org>
>> >>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
>> >>>> >
>> >>>>
>> >>>
>>

From gilberto.nunes32 at gmail.com  Thu Aug  6 12:32:29 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 6 Aug 2020 09:32:29 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <EA7AFBC2-38CF-4000-827B-8E0E31797FAC@yahoo.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
	<77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
	<CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>
	<EA7AFBC2-38CF-4000-827B-8E0E31797FAC@yahoo.com>
Message-ID: <CAOKSTBtdybdWw0i-CvsKSLeGGuhW9yyXau3vEAdJNbBSL4Rqsw@mail.gmail.com>

What do you mean "sharding"? Do you mean sharing folders between two
servers to host qcow2 or raw vm images?
Here I am using Proxmox which uses qemu but not virsh.

Thanks
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 6 de ago. de 2020 ?s 01:09, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> As  you mentioned qcow2  files,  check the virt group
> (/var/lib/glusterfs/group or something like that). It has optimal setttins
> for VMs and is used by oVirt.
>
> WARNING: If you decide to enable the group, which will also enable
> sharding, NEVER EVER DISABLE SHARDING -> ONCE ENABLED STAYS ENABLED !!!
> Sharding helps reduce loocking during replica heals.
>
> WARNING2: As virt group uses sharding (fixes the size of file into shard
> size),  you should consider cluster.favorite-child-policy with value
> ctime/mtime.
>
> Best Regards,
> Strahil Nikolov
>
> ?? 6 ?????? 2020 ?. 1:56:58 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >Ok...Thanks a lot Strahil
> >
> >This gluster volume set VMS cluster.favorite-child-policy size do the
> >trick
> >to me here!
> >
> >Cheers
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em qua., 5 de ago. de 2020 ?s 18:15, Strahil Nikolov
> ><hunter86_bg at yahoo.com>
> >escreveu:
> >
> >> This could happen if you have pending heals. Did you reboot that node
> >> recently ?
> >> Did you set automatic unsplit-brain ?
> >>
> >> Check for pending heals and files in splitbrain.
> >>
> >> If not, you can check
> >>
> >https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
> >> (look at point 5).
> >>
> >> Best Regards,
> >> Strahil Nikolov
> >>
> >> ?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <
> >> gilberto.nunes32 at gmail.com> ??????:
> >> >I'm in trouble here.
> >> >When I shutdown the pve01 server, the shared folder over glusterfs
> >is
> >> >EMPTY!
> >> >It's supposed to be a qcow2 file inside it.
> >> >The content is show right, just after I power on pve01 backup...
> >> >
> >> >Some advice?
> >> >
> >> >
> >> >Thanks
> >> >
> >> >---
> >> >Gilberto Nunes Ferreira
> >> >
> >> >(47) 3025-5907
> >> >(47) 99676-7530 - Whatsapp / Telegram
> >> >
> >> >Skype: gilberto.nunes36
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
> >> >gilberto.nunes32 at gmail.com> escreveu:
> >> >
> >> >> Well...
> >> >> I do the follow:
> >> >>
> >> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
> >> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
> >> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
> >> >>
> >> >> And now I have:
> >> >> gluster vol info
> >> >>
> >> >> Volume Name: VMS
> >> >> Type: Distributed-Replicate
> >> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
> >> >> Status: Started
> >> >> Snapshot Count: 0
> >> >> Number of Bricks: 2 x (2 + 1) = 6
> >> >> Transport-type: tcp
> >> >> Bricks:
> >> >> Brick1: pve01:/DATA/brick1
> >> >> Brick2: pve02:/DATA/brick1.5
> >> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
> >> >> Brick4: pve02:/DATA/brick2
> >> >> Brick5: pve01:/DATA/brick2.5
> >> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
> >> >> Options Reconfigured:
> >> >> cluster.quorum-count: 1
> >> >> cluster.quorum-reads: false
> >> >> cluster.self-heal-daemon: enable
> >> >> cluster.heal-timeout: 10
> >> >> storage.fips-mode-rchecksum: on
> >> >> transport.address-family: inet
> >> >> nfs.disable: on
> >> >> performance.client-io-threads: off
> >> >>
> >> >> This values I have put it myself, in order to see if could improve
> >> >the
> >> >> time to make the volume available, when pve01 goes down with
> >ifupdown
> >> >> cluster.quorum-count: 1
> >> >> cluster.quorum-reads: false
> >> >> cluster.self-heal-daemon: enable
> >> >> cluster.heal-timeout: 10
> >> >>
> >> >> Nevertheless, it took more than 1 minutes to the volume VMS
> >available
> >> >in
> >> >> the other host (pve02).
> >> >> Is there any trick to reduce this time ?
> >> >>
> >> >> Thanks
> >> >>
> >> >> ---
> >> >> Gilberto Nunes Ferreira
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
> >> >> gilberto.nunes32 at gmail.com> escreveu:
> >> >>
> >> >>> hum I see... like this:
> >> >>> [image: image.png]
> >> >>> ---
> >> >>> Gilberto Nunes Ferreira
> >> >>>
> >> >>> (47) 3025-5907
> >> >>> (47) 99676-7530 - Whatsapp / Telegram
> >> >>>
> >> >>> Skype: gilberto.nunes36
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>>
> >> >>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
> >> >>> bob at computerisms.ca> escreveu:
> >> >>>
> >> >>>> check the example of the chained configuration on this page:
> >> >>>>
> >> >>>>
> >> >>>>
> >> >
> >>
> >
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
> >> >>>>
> >> >>>> and apply it to two servers...
> >> >>>>
> >> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
> >> >>>> > Hi Bob!
> >> >>>> >
> >> >>>> > Could you, please, send me more detail about this
> >configuration?
> >> >>>> > I will appreciate that!
> >> >>>> >
> >> >>>> > Thank you
> >> >>>> > ---
> >> >>>> > Gilberto Nunes Ferreira
> >> >>>> >
> >> >>>> > (47) 3025-5907
> >> >>>> > **
> >> >>>> > (47) 99676-7530 - Whatsapp / Telegram
> >> >>>> >
> >> >>>> > Skype: gilberto.nunes36
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
> >> >>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
> >> >>>> >
> >> >>>> >     Hi Gilberto,
> >> >>>> >
> >> >>>> >     My understanding is there can only be one arbiter per
> >> >replicated
> >> >>>> >     set.  I
> >> >>>> >     don't have a lot of practice with gluster, so this could
> >be
> >> >bad
> >> >>>> advice,
> >> >>>> >     but the way I dealt with it on my two servers was to use 6
> >> >bricks
> >> >>>> as
> >> >>>> >     distributed-replicated (this is also relatively easy to
> >> >migrate to
> >> >>>> 3
> >> >>>> >     servers if that happens for you in the future):
> >> >>>> >
> >> >>>> >     Server1     Server2
> >> >>>> >     brick1      brick1.5
> >> >>>> >     arbiter1.5  brick2
> >> >>>> >     brick2.5    arbiter2.5
> >> >>>> >
> >> >>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
> >> >>>> >      > Hi there.
> >> >>>> >      > I have two physical servers deployed as replica 2 and,
> >> >>>> obviously,
> >> >>>> >     I got
> >> >>>> >      > a split-brain.
> >> >>>> >      > So I am thinking in use two virtual machines,each one
> >in
> >> >>>> physical
> >> >>>> >      > servers....
> >> >>>> >      > Then this two VMS act as a artiber of gluster set....
> >> >>>> >      >
> >> >>>> >      > Is this doable?
> >> >>>> >      >
> >> >>>> >      > Thanks
> >> >>>> >      >
> >> >>>> >      > ________
> >> >>>> >      >
> >> >>>> >      >
> >> >>>> >      >
> >> >>>> >      > Community Meeting Calendar:
> >> >>>> >      >
> >> >>>> >      > Schedule -
> >> >>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >> >>>> >      > Bridge: https://bluejeans.com/441850968
> >> >>>> >      >
> >> >>>> >      > Gluster-users mailing list
> >> >>>> >      > Gluster-users at gluster.org
> >> ><mailto:Gluster-users at gluster.org>
> >> >>>> >      >
> >https://lists.gluster.org/mailman/listinfo/gluster-users
> >> >>>> >      >
> >> >>>> >     ________
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> >     Community Meeting Calendar:
> >> >>>> >
> >> >>>> >     Schedule -
> >> >>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >> >>>> >     Bridge: https://bluejeans.com/441850968
> >> >>>> >
> >> >>>> >     Gluster-users mailing list
> >> >>>> >     Gluster-users at gluster.org
> ><mailto:Gluster-users at gluster.org>
> >> >>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
> >> >>>> >
> >> >>>>
> >> >>>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200806/a7a992ee/attachment.html>

From gilberto.nunes32 at gmail.com  Thu Aug  6 13:37:07 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 6 Aug 2020 10:37:07 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBtdybdWw0i-CvsKSLeGGuhW9yyXau3vEAdJNbBSL4Rqsw@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
	<77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
	<CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>
	<EA7AFBC2-38CF-4000-827B-8E0E31797FAC@yahoo.com>
	<CAOKSTBtdybdWw0i-CvsKSLeGGuhW9yyXau3vEAdJNbBSL4Rqsw@mail.gmail.com>
Message-ID: <CAOKSTBvd3njJpYhWkZGO+W_Gh2-JKoMfa5z_+5OQ=n7-+Eh-aA@mail.gmail.com>

Oh I see... I was confused because the terms... Now I read this and
everything becomes clear...

https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/

https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/configuring_red_hat_virtualization_with_red_hat_gluster_storage/chap-hosting_virtual_machine_images_on_red_hat_storage_volumes


Should I use cluster.granular-entrey-heal-enable too, since I am working
with big files?

Thanks

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 6 de ago. de 2020 ?s 09:32, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> What do you mean "sharding"? Do you mean sharing folders between two
> servers to host qcow2 or raw vm images?
> Here I am using Proxmox which uses qemu but not virsh.
>
> Thanks
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qui., 6 de ago. de 2020 ?s 01:09, Strahil Nikolov <
> hunter86_bg at yahoo.com> escreveu:
>
>> As  you mentioned qcow2  files,  check the virt group
>> (/var/lib/glusterfs/group or something like that). It has optimal setttins
>> for VMs and is used by oVirt.
>>
>> WARNING: If you decide to enable the group, which will also enable
>> sharding, NEVER EVER DISABLE SHARDING -> ONCE ENABLED STAYS ENABLED !!!
>> Sharding helps reduce loocking during replica heals.
>>
>> WARNING2: As virt group uses sharding (fixes the size of file into shard
>> size),  you should consider cluster.favorite-child-policy with value
>> ctime/mtime.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 6 ?????? 2020 ?. 1:56:58 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >Ok...Thanks a lot Strahil
>> >
>> >This gluster volume set VMS cluster.favorite-child-policy size do the
>> >trick
>> >to me here!
>> >
>> >Cheers
>> >---
>> >Gilberto Nunes Ferreira
>> >
>> >(47) 3025-5907
>> >(47) 99676-7530 - Whatsapp / Telegram
>> >
>> >Skype: gilberto.nunes36
>> >
>> >
>> >
>> >
>> >
>> >Em qua., 5 de ago. de 2020 ?s 18:15, Strahil Nikolov
>> ><hunter86_bg at yahoo.com>
>> >escreveu:
>> >
>> >> This could happen if you have pending heals. Did you reboot that node
>> >> recently ?
>> >> Did you set automatic unsplit-brain ?
>> >>
>> >> Check for pending heals and files in splitbrain.
>> >>
>> >> If not, you can check
>> >>
>> >https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
>> >> (look at point 5).
>> >>
>> >> Best Regards,
>> >> Strahil Nikolov
>> >>
>> >> ?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <
>> >> gilberto.nunes32 at gmail.com> ??????:
>> >> >I'm in trouble here.
>> >> >When I shutdown the pve01 server, the shared folder over glusterfs
>> >is
>> >> >EMPTY!
>> >> >It's supposed to be a qcow2 file inside it.
>> >> >The content is show right, just after I power on pve01 backup...
>> >> >
>> >> >Some advice?
>> >> >
>> >> >
>> >> >Thanks
>> >> >
>> >> >---
>> >> >Gilberto Nunes Ferreira
>> >> >
>> >> >(47) 3025-5907
>> >> >(47) 99676-7530 - Whatsapp / Telegram
>> >> >
>> >> >Skype: gilberto.nunes36
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >
>> >> >Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
>> >> >gilberto.nunes32 at gmail.com> escreveu:
>> >> >
>> >> >> Well...
>> >> >> I do the follow:
>> >> >>
>> >> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
>> >> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2 pv
>> >> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
>> >> >>
>> >> >> And now I have:
>> >> >> gluster vol info
>> >> >>
>> >> >> Volume Name: VMS
>> >> >> Type: Distributed-Replicate
>> >> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
>> >> >> Status: Started
>> >> >> Snapshot Count: 0
>> >> >> Number of Bricks: 2 x (2 + 1) = 6
>> >> >> Transport-type: tcp
>> >> >> Bricks:
>> >> >> Brick1: pve01:/DATA/brick1
>> >> >> Brick2: pve02:/DATA/brick1.5
>> >> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
>> >> >> Brick4: pve02:/DATA/brick2
>> >> >> Brick5: pve01:/DATA/brick2.5
>> >> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
>> >> >> Options Reconfigured:
>> >> >> cluster.quorum-count: 1
>> >> >> cluster.quorum-reads: false
>> >> >> cluster.self-heal-daemon: enable
>> >> >> cluster.heal-timeout: 10
>> >> >> storage.fips-mode-rchecksum: on
>> >> >> transport.address-family: inet
>> >> >> nfs.disable: on
>> >> >> performance.client-io-threads: off
>> >> >>
>> >> >> This values I have put it myself, in order to see if could improve
>> >> >the
>> >> >> time to make the volume available, when pve01 goes down with
>> >ifupdown
>> >> >> cluster.quorum-count: 1
>> >> >> cluster.quorum-reads: false
>> >> >> cluster.self-heal-daemon: enable
>> >> >> cluster.heal-timeout: 10
>> >> >>
>> >> >> Nevertheless, it took more than 1 minutes to the volume VMS
>> >available
>> >> >in
>> >> >> the other host (pve02).
>> >> >> Is there any trick to reduce this time ?
>> >> >>
>> >> >> Thanks
>> >> >>
>> >> >> ---
>> >> >> Gilberto Nunes Ferreira
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
>> >> >> gilberto.nunes32 at gmail.com> escreveu:
>> >> >>
>> >> >>> hum I see... like this:
>> >> >>> [image: image.png]
>> >> >>> ---
>> >> >>> Gilberto Nunes Ferreira
>> >> >>>
>> >> >>> (47) 3025-5907
>> >> >>> (47) 99676-7530 - Whatsapp / Telegram
>> >> >>>
>> >> >>> Skype: gilberto.nunes36
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>>
>> >> >>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation <
>> >> >>> bob at computerisms.ca> escreveu:
>> >> >>>
>> >> >>>> check the example of the chained configuration on this page:
>> >> >>>>
>> >> >>>>
>> >> >>>>
>> >> >
>> >>
>> >
>> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>> >> >>>>
>> >> >>>> and apply it to two servers...
>> >> >>>>
>> >> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>> >> >>>> > Hi Bob!
>> >> >>>> >
>> >> >>>> > Could you, please, send me more detail about this
>> >configuration?
>> >> >>>> > I will appreciate that!
>> >> >>>> >
>> >> >>>> > Thank you
>> >> >>>> > ---
>> >> >>>> > Gilberto Nunes Ferreira
>> >> >>>> >
>> >> >>>> > (47) 3025-5907
>> >> >>>> > **
>> >> >>>> > (47) 99676-7530 - Whatsapp / Telegram
>> >> >>>> >
>> >> >>>> > Skype: gilberto.nunes36
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms Corporation
>> >> >>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>> escreveu:
>> >> >>>> >
>> >> >>>> >     Hi Gilberto,
>> >> >>>> >
>> >> >>>> >     My understanding is there can only be one arbiter per
>> >> >replicated
>> >> >>>> >     set.  I
>> >> >>>> >     don't have a lot of practice with gluster, so this could
>> >be
>> >> >bad
>> >> >>>> advice,
>> >> >>>> >     but the way I dealt with it on my two servers was to use 6
>> >> >bricks
>> >> >>>> as
>> >> >>>> >     distributed-replicated (this is also relatively easy to
>> >> >migrate to
>> >> >>>> 3
>> >> >>>> >     servers if that happens for you in the future):
>> >> >>>> >
>> >> >>>> >     Server1     Server2
>> >> >>>> >     brick1      brick1.5
>> >> >>>> >     arbiter1.5  brick2
>> >> >>>> >     brick2.5    arbiter2.5
>> >> >>>> >
>> >> >>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>> >> >>>> >      > Hi there.
>> >> >>>> >      > I have two physical servers deployed as replica 2 and,
>> >> >>>> obviously,
>> >> >>>> >     I got
>> >> >>>> >      > a split-brain.
>> >> >>>> >      > So I am thinking in use two virtual machines,each one
>> >in
>> >> >>>> physical
>> >> >>>> >      > servers....
>> >> >>>> >      > Then this two VMS act as a artiber of gluster set....
>> >> >>>> >      >
>> >> >>>> >      > Is this doable?
>> >> >>>> >      >
>> >> >>>> >      > Thanks
>> >> >>>> >      >
>> >> >>>> >      > ________
>> >> >>>> >      >
>> >> >>>> >      >
>> >> >>>> >      >
>> >> >>>> >      > Community Meeting Calendar:
>> >> >>>> >      >
>> >> >>>> >      > Schedule -
>> >> >>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >> >>>> >      > Bridge: https://bluejeans.com/441850968
>> >> >>>> >      >
>> >> >>>> >      > Gluster-users mailing list
>> >> >>>> >      > Gluster-users at gluster.org
>> >> ><mailto:Gluster-users at gluster.org>
>> >> >>>> >      >
>> >https://lists.gluster.org/mailman/listinfo/gluster-users
>> >> >>>> >      >
>> >> >>>> >     ________
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >
>> >> >>>> >     Community Meeting Calendar:
>> >> >>>> >
>> >> >>>> >     Schedule -
>> >> >>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >> >>>> >     Bridge: https://bluejeans.com/441850968
>> >> >>>> >
>> >> >>>> >     Gluster-users mailing list
>> >> >>>> >     Gluster-users at gluster.org
>> ><mailto:Gluster-users at gluster.org>
>> >> >>>> >     https://lists.gluster.org/mailman/listinfo/gluster-users
>> >> >>>> >
>> >> >>>>
>> >> >>>
>> >>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200806/8ae6bbe4/attachment.html>

From hunter86_bg at yahoo.com  Thu Aug  6 17:14:42 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 06 Aug 2020 20:14:42 +0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <CAOKSTBvd3njJpYhWkZGO+W_Gh2-JKoMfa5z_+5OQ=n7-+Eh-aA@mail.gmail.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
	<77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
	<CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>
	<EA7AFBC2-38CF-4000-827B-8E0E31797FAC@yahoo.com>
	<CAOKSTBtdybdWw0i-CvsKSLeGGuhW9yyXau3vEAdJNbBSL4Rqsw@mail.gmail.com>
	<CAOKSTBvd3njJpYhWkZGO+W_Gh2-JKoMfa5z_+5OQ=n7-+Eh-aA@mail.gmail.com>
Message-ID: <7A2FF09A-AE59-4C89-B596-968D55BD5521@yahoo.com>

The settings  I got in my group is:
[root at ovirt1 ~]# cat /var/lib/glusterd/groups/virt
performance.quick-read=off
performance.read-ahead=off
performance.io-cache=off
performance.low-prio-threads=32
network.remote-dio=enable
cluster.eager-lock=enable
cluster.quorum-type=auto
cluster.server-quorum-type=server
cluster.data-self-heal-algorithm=full
cluster.locking-scheme=granular
cluster.shd-max-threads=8
cluster.shd-wait-qlength=10000
features.shard=on
user.cifs=off
cluster.choose-local=off
client.event-threads=4
server.event-threads=4
performance.client-io-threads=on

I'm not  sure that sharded  files are  treated  as  big or not.If your  brick disks are faster than your network bandwidth, you can enable 'cluster.choose-local' .

Keep in mind that some  users report issues  with sparse  qcow2  images  during intensive writes (suspected shard  xlator cannot create fast enough the shards -> default shard size (64MB) is way smaller than the RedHat's  supported size  which is 512MB)  and  I would recommend you  to use  preallocated  qcow2  disks as  much as  possible or to bump the shard size.

Sharding was  developed especially for Virt usage.

Consider  using another cluster.favorite-child-policy  , as  all  shards  have the same size.

Best Regards,
Strahil Nikolov



?? 6 ?????? 2020 ?. 16:37:07 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>Oh I see... I was confused because the terms... Now I read this and
>everything becomes clear...
>
>https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/
>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/configuring_red_hat_virtualization_with_red_hat_gluster_storage/chap-hosting_virtual_machine_images_on_red_hat_storage_volumes
>
>
>Should I use cluster.granular-entrey-heal-enable too, since I am
>working
>with big files?
>
>Thanks
>
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qui., 6 de ago. de 2020 ?s 09:32, Gilberto Nunes <
>gilberto.nunes32 at gmail.com> escreveu:
>
>> What do you mean "sharding"? Do you mean sharing folders between two
>> servers to host qcow2 or raw vm images?
>> Here I am using Proxmox which uses qemu but not virsh.
>>
>> Thanks
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qui., 6 de ago. de 2020 ?s 01:09, Strahil Nikolov <
>> hunter86_bg at yahoo.com> escreveu:
>>
>>> As  you mentioned qcow2  files,  check the virt group
>>> (/var/lib/glusterfs/group or something like that). It has optimal
>setttins
>>> for VMs and is used by oVirt.
>>>
>>> WARNING: If you decide to enable the group, which will also enable
>>> sharding, NEVER EVER DISABLE SHARDING -> ONCE ENABLED STAYS ENABLED
>!!!
>>> Sharding helps reduce loocking during replica heals.
>>>
>>> WARNING2: As virt group uses sharding (fixes the size of file into
>shard
>>> size),  you should consider cluster.favorite-child-policy with value
>>> ctime/mtime.
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> ?? 6 ?????? 2020 ?. 1:56:58 GMT+03:00, Gilberto Nunes <
>>> gilberto.nunes32 at gmail.com> ??????:
>>> >Ok...Thanks a lot Strahil
>>> >
>>> >This gluster volume set VMS cluster.favorite-child-policy size do
>the
>>> >trick
>>> >to me here!
>>> >
>>> >Cheers
>>> >---
>>> >Gilberto Nunes Ferreira
>>> >
>>> >(47) 3025-5907
>>> >(47) 99676-7530 - Whatsapp / Telegram
>>> >
>>> >Skype: gilberto.nunes36
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >Em qua., 5 de ago. de 2020 ?s 18:15, Strahil Nikolov
>>> ><hunter86_bg at yahoo.com>
>>> >escreveu:
>>> >
>>> >> This could happen if you have pending heals. Did you reboot that
>node
>>> >> recently ?
>>> >> Did you set automatic unsplit-brain ?
>>> >>
>>> >> Check for pending heals and files in splitbrain.
>>> >>
>>> >> If not, you can check
>>> >>
>>>
>>https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
>>> >> (look at point 5).
>>> >>
>>> >> Best Regards,
>>> >> Strahil Nikolov
>>> >>
>>> >> ?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <
>>> >> gilberto.nunes32 at gmail.com> ??????:
>>> >> >I'm in trouble here.
>>> >> >When I shutdown the pve01 server, the shared folder over
>glusterfs
>>> >is
>>> >> >EMPTY!
>>> >> >It's supposed to be a qcow2 file inside it.
>>> >> >The content is show right, just after I power on pve01 backup...
>>> >> >
>>> >> >Some advice?
>>> >> >
>>> >> >
>>> >> >Thanks
>>> >> >
>>> >> >---
>>> >> >Gilberto Nunes Ferreira
>>> >> >
>>> >> >(47) 3025-5907
>>> >> >(47) 99676-7530 - Whatsapp / Telegram
>>> >> >
>>> >> >Skype: gilberto.nunes36
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >
>>> >> >Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
>>> >> >gilberto.nunes32 at gmail.com> escreveu:
>>> >> >
>>> >> >> Well...
>>> >> >> I do the follow:
>>> >> >>
>>> >> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
>>> >> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2
>pv
>>> >> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
>>> >> >>
>>> >> >> And now I have:
>>> >> >> gluster vol info
>>> >> >>
>>> >> >> Volume Name: VMS
>>> >> >> Type: Distributed-Replicate
>>> >> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
>>> >> >> Status: Started
>>> >> >> Snapshot Count: 0
>>> >> >> Number of Bricks: 2 x (2 + 1) = 6
>>> >> >> Transport-type: tcp
>>> >> >> Bricks:
>>> >> >> Brick1: pve01:/DATA/brick1
>>> >> >> Brick2: pve02:/DATA/brick1.5
>>> >> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
>>> >> >> Brick4: pve02:/DATA/brick2
>>> >> >> Brick5: pve01:/DATA/brick2.5
>>> >> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
>>> >> >> Options Reconfigured:
>>> >> >> cluster.quorum-count: 1
>>> >> >> cluster.quorum-reads: false
>>> >> >> cluster.self-heal-daemon: enable
>>> >> >> cluster.heal-timeout: 10
>>> >> >> storage.fips-mode-rchecksum: on
>>> >> >> transport.address-family: inet
>>> >> >> nfs.disable: on
>>> >> >> performance.client-io-threads: off
>>> >> >>
>>> >> >> This values I have put it myself, in order to see if could
>improve
>>> >> >the
>>> >> >> time to make the volume available, when pve01 goes down with
>>> >ifupdown
>>> >> >> cluster.quorum-count: 1
>>> >> >> cluster.quorum-reads: false
>>> >> >> cluster.self-heal-daemon: enable
>>> >> >> cluster.heal-timeout: 10
>>> >> >>
>>> >> >> Nevertheless, it took more than 1 minutes to the volume VMS
>>> >available
>>> >> >in
>>> >> >> the other host (pve02).
>>> >> >> Is there any trick to reduce this time ?
>>> >> >>
>>> >> >> Thanks
>>> >> >>
>>> >> >> ---
>>> >> >> Gilberto Nunes Ferreira
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
>>> >> >> gilberto.nunes32 at gmail.com> escreveu:
>>> >> >>
>>> >> >>> hum I see... like this:
>>> >> >>> [image: image.png]
>>> >> >>> ---
>>> >> >>> Gilberto Nunes Ferreira
>>> >> >>>
>>> >> >>> (47) 3025-5907
>>> >> >>> (47) 99676-7530 - Whatsapp / Telegram
>>> >> >>>
>>> >> >>> Skype: gilberto.nunes36
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>>
>>> >> >>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation
><
>>> >> >>> bob at computerisms.ca> escreveu:
>>> >> >>>
>>> >> >>>> check the example of the chained configuration on this page:
>>> >> >>>>
>>> >> >>>>
>>> >> >>>>
>>> >> >
>>> >>
>>> >
>>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
>>> >> >>>>
>>> >> >>>> and apply it to two servers...
>>> >> >>>>
>>> >> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
>>> >> >>>> > Hi Bob!
>>> >> >>>> >
>>> >> >>>> > Could you, please, send me more detail about this
>>> >configuration?
>>> >> >>>> > I will appreciate that!
>>> >> >>>> >
>>> >> >>>> > Thank you
>>> >> >>>> > ---
>>> >> >>>> > Gilberto Nunes Ferreira
>>> >> >>>> >
>>> >> >>>> > (47) 3025-5907
>>> >> >>>> > **
>>> >> >>>> > (47) 99676-7530 - Whatsapp / Telegram
>>> >> >>>> >
>>> >> >>>> > Skype: gilberto.nunes36
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms
>Corporation
>>> >> >>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>>
>escreveu:
>>> >> >>>> >
>>> >> >>>> >     Hi Gilberto,
>>> >> >>>> >
>>> >> >>>> >     My understanding is there can only be one arbiter per
>>> >> >replicated
>>> >> >>>> >     set.  I
>>> >> >>>> >     don't have a lot of practice with gluster, so this
>could
>>> >be
>>> >> >bad
>>> >> >>>> advice,
>>> >> >>>> >     but the way I dealt with it on my two servers was to
>use 6
>>> >> >bricks
>>> >> >>>> as
>>> >> >>>> >     distributed-replicated (this is also relatively easy
>to
>>> >> >migrate to
>>> >> >>>> 3
>>> >> >>>> >     servers if that happens for you in the future):
>>> >> >>>> >
>>> >> >>>> >     Server1     Server2
>>> >> >>>> >     brick1      brick1.5
>>> >> >>>> >     arbiter1.5  brick2
>>> >> >>>> >     brick2.5    arbiter2.5
>>> >> >>>> >
>>> >> >>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
>>> >> >>>> >      > Hi there.
>>> >> >>>> >      > I have two physical servers deployed as replica 2
>and,
>>> >> >>>> obviously,
>>> >> >>>> >     I got
>>> >> >>>> >      > a split-brain.
>>> >> >>>> >      > So I am thinking in use two virtual machines,each
>one
>>> >in
>>> >> >>>> physical
>>> >> >>>> >      > servers....
>>> >> >>>> >      > Then this two VMS act as a artiber of gluster
>set....
>>> >> >>>> >      >
>>> >> >>>> >      > Is this doable?
>>> >> >>>> >      >
>>> >> >>>> >      > Thanks
>>> >> >>>> >      >
>>> >> >>>> >      > ________
>>> >> >>>> >      >
>>> >> >>>> >      >
>>> >> >>>> >      >
>>> >> >>>> >      > Community Meeting Calendar:
>>> >> >>>> >      >
>>> >> >>>> >      > Schedule -
>>> >> >>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >> >>>> >      > Bridge: https://bluejeans.com/441850968
>>> >> >>>> >      >
>>> >> >>>> >      > Gluster-users mailing list
>>> >> >>>> >      > Gluster-users at gluster.org
>>> >> ><mailto:Gluster-users at gluster.org>
>>> >> >>>> >      >
>>> >https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >> >>>> >      >
>>> >> >>>> >     ________
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >
>>> >> >>>> >     Community Meeting Calendar:
>>> >> >>>> >
>>> >> >>>> >     Schedule -
>>> >> >>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >> >>>> >     Bridge: https://bluejeans.com/441850968
>>> >> >>>> >
>>> >> >>>> >     Gluster-users mailing list
>>> >> >>>> >     Gluster-users at gluster.org
>>> ><mailto:Gluster-users at gluster.org>
>>> >> >>>> >    
>https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >> >>>> >
>>> >> >>>>
>>> >> >>>
>>> >>
>>>
>>

From gilberto.nunes32 at gmail.com  Thu Aug  6 18:15:33 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 6 Aug 2020 15:15:33 -0300
Subject: [Gluster-users] Two VMS as arbiter...
In-Reply-To: <7A2FF09A-AE59-4C89-B596-968D55BD5521@yahoo.com>
References: <CAOKSTBu1O=bX77XL7+W+VkjCv11DeDpUPXi2RUmKUtt7CjwXTw@mail.gmail.com>
	<4610d2cc-eafa-6a5b-d778-797e6ce7e994@computerisms.ca>
	<CAOKSTBv8v7C0jkP_d4vrJA7rUmcQKfRiyCwAsTE04u3kyXTCvw@mail.gmail.com>
	<6496d212-9ffa-5112-fc14-aee578b25f01@computerisms.ca>
	<CAOKSTBu0DE7gaaK8_XLZdKY7-v2-q9-cpuCH=3BbasFWXykTrg@mail.gmail.com>
	<CAOKSTBs+vd9BPmuLSsoP0+h8KCLEYd0pQeOi9guP78Jwwff3BA@mail.gmail.com>
	<CAOKSTBtm_k1VKBh_9QzSgdM+Ck7tu3CoKyM526j+VhjEEbzm7w@mail.gmail.com>
	<77292D07-DE1E-4797-A54C-13086317763C@yahoo.com>
	<CAOKSTBvf8me3yz85Dp=e6vWbDvtgzVrgkz0_DjN835ZYMn_DOw@mail.gmail.com>
	<EA7AFBC2-38CF-4000-827B-8E0E31797FAC@yahoo.com>
	<CAOKSTBtdybdWw0i-CvsKSLeGGuhW9yyXau3vEAdJNbBSL4Rqsw@mail.gmail.com>
	<CAOKSTBvd3njJpYhWkZGO+W_Gh2-JKoMfa5z_+5OQ=n7-+Eh-aA@mail.gmail.com>
	<7A2FF09A-AE59-4C89-B596-968D55BD5521@yahoo.com>
Message-ID: <CAOKSTBvu6yueVYZ_TXVeq=rH0q8dugpi9852dg+aBC54Yf-wJA@mail.gmail.com>

The options that worked best in my tests were as follows, to avoid
split-brain

gluster vol set VMS cluster.heal-timeout 20
gluster volume heal VMS enable
gluster vol set VMS cluster.quorum-reads false
gluster vol set VMS cluster.quorum-count 1
gluster vol set VMS network.ping-timeout 2
gluster volume set VMS cluster.favorite-child-policy mtime
gluster volume heal VMS granular-entry-heal enable
gluster volume set VMS cluster.data-self-heal-algorithm full

Here
gluster volume set VMS cluster.favorite-child-policy mtime
I used "size" but I read in several places that mtime is better ...

I did several and exhaustive tests ... power off hosts, migrating vm,
creating folders and files inside the vm ... activating HA etc ...
After the "crash" ie after the host that was restarted / shutdown comes
back, the volume looks like this
Brick pve02: / DATA / brick
/images/100/vm-100-disk-0.qcow2 - Possibly undergoing heal
Status: Connected
Number of entries: 1

Indicating that healing is taking place ...
After a few minutes / hours depending on the hardware speed, "possibly
undergoing" disappears ...

But at no time was there data loss ...
While possibly undergoing heals I migrate the vm from one side to another
also without problems ...

Here in the tests I performed, the healing of a 10G VM HD, having 4G busy,
took 30 minutes ...
Remembering that I'm using a virtualbox with 2 vms in it with 2 G of ram
each, each vm being a proxmox.
In a real environment this time is much less and also depends on the size
of the VM's HD!

Cheers

---
Gilberto Nunes Ferreira


Em qui., 6 de ago. de 2020 ?s 14:14, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> The settings  I got in my group is:
> [root at ovirt1 ~]# cat /var/lib/glusterd/groups/virt
> performance.quick-read=off
> performance.read-ahead=off
> performance.io-cache=off
> performance.low-prio-threads=32
> network.remote-dio=enable
> cluster.eager-lock=enable
> cluster.quorum-type=auto
> cluster.server-quorum-type=server
> cluster.data-self-heal-algorithm=full
> cluster.locking-scheme=granular
> cluster.shd-max-threads=8
> cluster.shd-wait-qlength=10000
> features.shard=on
> user.cifs=off
> cluster.choose-local=off
> client.event-threads=4
> server.event-threads=4
> performance.client-io-threads=on
>
> I'm not  sure that sharded  files are  treated  as  big or not.If your
> brick disks are faster than your network bandwidth, you can enable
> 'cluster.choose-local' .
>
> Keep in mind that some  users report issues  with sparse  qcow2  images
> during intensive writes (suspected shard  xlator cannot create fast enough
> the shards -> default shard size (64MB) is way smaller than the RedHat's
> supported size  which is 512MB)  and  I would recommend you  to use
> preallocated  qcow2  disks as  much as  possible or to bump the shard size.
>
> Sharding was  developed especially for Virt usage.
>
> Consider  using another cluster.favorite-child-policy  , as  all  shards
> have the same size.
>
> Best Regards,
> Strahil Nikolov
>
>
>
> ?? 6 ?????? 2020 ?. 16:37:07 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >Oh I see... I was confused because the terms... Now I read this and
> >everything becomes clear...
> >
> >
> https://staged-gluster-docs.readthedocs.io/en/release3.7.0beta1/Features/shard/
> >
> >
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/configuring_red_hat_virtualization_with_red_hat_gluster_storage/chap-hosting_virtual_machine_images_on_red_hat_storage_volumes
> >
> >
> >Should I use cluster.granular-entrey-heal-enable too, since I am
> >working
> >with big files?
> >
> >Thanks
> >
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em qui., 6 de ago. de 2020 ?s 09:32, Gilberto Nunes <
> >gilberto.nunes32 at gmail.com> escreveu:
> >
> >> What do you mean "sharding"? Do you mean sharing folders between two
> >> servers to host qcow2 or raw vm images?
> >> Here I am using Proxmox which uses qemu but not virsh.
> >>
> >> Thanks
> >> ---
> >> Gilberto Nunes Ferreira
> >>
> >> (47) 3025-5907
> >> (47) 99676-7530 - Whatsapp / Telegram
> >>
> >> Skype: gilberto.nunes36
> >>
> >>
> >>
> >>
> >>
> >> Em qui., 6 de ago. de 2020 ?s 01:09, Strahil Nikolov <
> >> hunter86_bg at yahoo.com> escreveu:
> >>
> >>> As  you mentioned qcow2  files,  check the virt group
> >>> (/var/lib/glusterfs/group or something like that). It has optimal
> >setttins
> >>> for VMs and is used by oVirt.
> >>>
> >>> WARNING: If you decide to enable the group, which will also enable
> >>> sharding, NEVER EVER DISABLE SHARDING -> ONCE ENABLED STAYS ENABLED
> >!!!
> >>> Sharding helps reduce loocking during replica heals.
> >>>
> >>> WARNING2: As virt group uses sharding (fixes the size of file into
> >shard
> >>> size),  you should consider cluster.favorite-child-policy with value
> >>> ctime/mtime.
> >>>
> >>> Best Regards,
> >>> Strahil Nikolov
> >>>
> >>> ?? 6 ?????? 2020 ?. 1:56:58 GMT+03:00, Gilberto Nunes <
> >>> gilberto.nunes32 at gmail.com> ??????:
> >>> >Ok...Thanks a lot Strahil
> >>> >
> >>> >This gluster volume set VMS cluster.favorite-child-policy size do
> >the
> >>> >trick
> >>> >to me here!
> >>> >
> >>> >Cheers
> >>> >---
> >>> >Gilberto Nunes Ferreira
> >>> >
> >>> >(47) 3025-5907
> >>> >(47) 99676-7530 - Whatsapp / Telegram
> >>> >
> >>> >Skype: gilberto.nunes36
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >Em qua., 5 de ago. de 2020 ?s 18:15, Strahil Nikolov
> >>> ><hunter86_bg at yahoo.com>
> >>> >escreveu:
> >>> >
> >>> >> This could happen if you have pending heals. Did you reboot that
> >node
> >>> >> recently ?
> >>> >> Did you set automatic unsplit-brain ?
> >>> >>
> >>> >> Check for pending heals and files in splitbrain.
> >>> >>
> >>> >> If not, you can check
> >>> >>
> >>>
> >>https://docs.gluster.org/en/latest/Troubleshooting/resolving-splitbrain/
> >>> >> (look at point 5).
> >>> >>
> >>> >> Best Regards,
> >>> >> Strahil Nikolov
> >>> >>
> >>> >> ?? 5 ?????? 2020 ?. 23:41:57 GMT+03:00, Gilberto Nunes <
> >>> >> gilberto.nunes32 at gmail.com> ??????:
> >>> >> >I'm in trouble here.
> >>> >> >When I shutdown the pve01 server, the shared folder over
> >glusterfs
> >>> >is
> >>> >> >EMPTY!
> >>> >> >It's supposed to be a qcow2 file inside it.
> >>> >> >The content is show right, just after I power on pve01 backup...
> >>> >> >
> >>> >> >Some advice?
> >>> >> >
> >>> >> >
> >>> >> >Thanks
> >>> >> >
> >>> >> >---
> >>> >> >Gilberto Nunes Ferreira
> >>> >> >
> >>> >> >(47) 3025-5907
> >>> >> >(47) 99676-7530 - Whatsapp / Telegram
> >>> >> >
> >>> >> >Skype: gilberto.nunes36
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >
> >>> >> >Em qua., 5 de ago. de 2020 ?s 11:07, Gilberto Nunes <
> >>> >> >gilberto.nunes32 at gmail.com> escreveu:
> >>> >> >
> >>> >> >> Well...
> >>> >> >> I do the follow:
> >>> >> >>
> >>> >> >> gluster vol create VMS replica 3 arbiter 1 pve01:/DATA/brick1
> >>> >> >> pve02:/DATA/brick1.5 pve01:/DATA/arbiter1.5 pve02:/DATA/brick2
> >pv
> >>> >> >> e01:/DATA/brick2.5 pve02:/DATA/arbiter2.5 force
> >>> >> >>
> >>> >> >> And now I have:
> >>> >> >> gluster vol info
> >>> >> >>
> >>> >> >> Volume Name: VMS
> >>> >> >> Type: Distributed-Replicate
> >>> >> >> Volume ID: 1bd712f5-ccb9-4322-8275-abe363d1ffdd
> >>> >> >> Status: Started
> >>> >> >> Snapshot Count: 0
> >>> >> >> Number of Bricks: 2 x (2 + 1) = 6
> >>> >> >> Transport-type: tcp
> >>> >> >> Bricks:
> >>> >> >> Brick1: pve01:/DATA/brick1
> >>> >> >> Brick2: pve02:/DATA/brick1.5
> >>> >> >> Brick3: pve01:/DATA/arbiter1.5 (arbiter)
> >>> >> >> Brick4: pve02:/DATA/brick2
> >>> >> >> Brick5: pve01:/DATA/brick2.5
> >>> >> >> Brick6: pve02:/DATA/arbiter2.5 (arbiter)
> >>> >> >> Options Reconfigured:
> >>> >> >> cluster.quorum-count: 1
> >>> >> >> cluster.quorum-reads: false
> >>> >> >> cluster.self-heal-daemon: enable
> >>> >> >> cluster.heal-timeout: 10
> >>> >> >> storage.fips-mode-rchecksum: on
> >>> >> >> transport.address-family: inet
> >>> >> >> nfs.disable: on
> >>> >> >> performance.client-io-threads: off
> >>> >> >>
> >>> >> >> This values I have put it myself, in order to see if could
> >improve
> >>> >> >the
> >>> >> >> time to make the volume available, when pve01 goes down with
> >>> >ifupdown
> >>> >> >> cluster.quorum-count: 1
> >>> >> >> cluster.quorum-reads: false
> >>> >> >> cluster.self-heal-daemon: enable
> >>> >> >> cluster.heal-timeout: 10
> >>> >> >>
> >>> >> >> Nevertheless, it took more than 1 minutes to the volume VMS
> >>> >available
> >>> >> >in
> >>> >> >> the other host (pve02).
> >>> >> >> Is there any trick to reduce this time ?
> >>> >> >>
> >>> >> >> Thanks
> >>> >> >>
> >>> >> >> ---
> >>> >> >> Gilberto Nunes Ferreira
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >>
> >>> >> >> Em qua., 5 de ago. de 2020 ?s 08:57, Gilberto Nunes <
> >>> >> >> gilberto.nunes32 at gmail.com> escreveu:
> >>> >> >>
> >>> >> >>> hum I see... like this:
> >>> >> >>> [image: image.png]
> >>> >> >>> ---
> >>> >> >>> Gilberto Nunes Ferreira
> >>> >> >>>
> >>> >> >>> (47) 3025-5907
> >>> >> >>> (47) 99676-7530 - Whatsapp / Telegram
> >>> >> >>>
> >>> >> >>> Skype: gilberto.nunes36
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>>
> >>> >> >>> Em qua., 5 de ago. de 2020 ?s 02:14, Computerisms Corporation
> ><
> >>> >> >>> bob at computerisms.ca> escreveu:
> >>> >> >>>
> >>> >> >>>> check the example of the chained configuration on this page:
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >>>>
> >>> >> >
> >>> >>
> >>> >
> >>>
> >
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.3/html/administration_guide/creating_arbitrated_replicated_volumes
> >>> >> >>>>
> >>> >> >>>> and apply it to two servers...
> >>> >> >>>>
> >>> >> >>>> On 2020-08-04 8:25 p.m., Gilberto Nunes wrote:
> >>> >> >>>> > Hi Bob!
> >>> >> >>>> >
> >>> >> >>>> > Could you, please, send me more detail about this
> >>> >configuration?
> >>> >> >>>> > I will appreciate that!
> >>> >> >>>> >
> >>> >> >>>> > Thank you
> >>> >> >>>> > ---
> >>> >> >>>> > Gilberto Nunes Ferreira
> >>> >> >>>> >
> >>> >> >>>> > (47) 3025-5907
> >>> >> >>>> > **
> >>> >> >>>> > (47) 99676-7530 - Whatsapp / Telegram
> >>> >> >>>> >
> >>> >> >>>> > Skype: gilberto.nunes36
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> > Em ter., 4 de ago. de 2020 ?s 23:47, Computerisms
> >Corporation
> >>> >> >>>> > <bob at computerisms.ca <mailto:bob at computerisms.ca>>
> >escreveu:
> >>> >> >>>> >
> >>> >> >>>> >     Hi Gilberto,
> >>> >> >>>> >
> >>> >> >>>> >     My understanding is there can only be one arbiter per
> >>> >> >replicated
> >>> >> >>>> >     set.  I
> >>> >> >>>> >     don't have a lot of practice with gluster, so this
> >could
> >>> >be
> >>> >> >bad
> >>> >> >>>> advice,
> >>> >> >>>> >     but the way I dealt with it on my two servers was to
> >use 6
> >>> >> >bricks
> >>> >> >>>> as
> >>> >> >>>> >     distributed-replicated (this is also relatively easy
> >to
> >>> >> >migrate to
> >>> >> >>>> 3
> >>> >> >>>> >     servers if that happens for you in the future):
> >>> >> >>>> >
> >>> >> >>>> >     Server1     Server2
> >>> >> >>>> >     brick1      brick1.5
> >>> >> >>>> >     arbiter1.5  brick2
> >>> >> >>>> >     brick2.5    arbiter2.5
> >>> >> >>>> >
> >>> >> >>>> >     On 2020-08-04 7:00 p.m., Gilberto Nunes wrote:
> >>> >> >>>> >      > Hi there.
> >>> >> >>>> >      > I have two physical servers deployed as replica 2
> >and,
> >>> >> >>>> obviously,
> >>> >> >>>> >     I got
> >>> >> >>>> >      > a split-brain.
> >>> >> >>>> >      > So I am thinking in use two virtual machines,each
> >one
> >>> >in
> >>> >> >>>> physical
> >>> >> >>>> >      > servers....
> >>> >> >>>> >      > Then this two VMS act as a artiber of gluster
> >set....
> >>> >> >>>> >      >
> >>> >> >>>> >      > Is this doable?
> >>> >> >>>> >      >
> >>> >> >>>> >      > Thanks
> >>> >> >>>> >      >
> >>> >> >>>> >      > ________
> >>> >> >>>> >      >
> >>> >> >>>> >      >
> >>> >> >>>> >      >
> >>> >> >>>> >      > Community Meeting Calendar:
> >>> >> >>>> >      >
> >>> >> >>>> >      > Schedule -
> >>> >> >>>> >      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> >> >>>> >      > Bridge: https://bluejeans.com/441850968
> >>> >> >>>> >      >
> >>> >> >>>> >      > Gluster-users mailing list
> >>> >> >>>> >      > Gluster-users at gluster.org
> >>> >> ><mailto:Gluster-users at gluster.org>
> >>> >> >>>> >      >
> >>> >https://lists.gluster.org/mailman/listinfo/gluster-users
> >>> >> >>>> >      >
> >>> >> >>>> >     ________
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> >
> >>> >> >>>> >     Community Meeting Calendar:
> >>> >> >>>> >
> >>> >> >>>> >     Schedule -
> >>> >> >>>> >     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> >> >>>> >     Bridge: https://bluejeans.com/441850968
> >>> >> >>>> >
> >>> >> >>>> >     Gluster-users mailing list
> >>> >> >>>> >     Gluster-users at gluster.org
> >>> ><mailto:Gluster-users at gluster.org>
> >>> >> >>>> >
> >https://lists.gluster.org/mailman/listinfo/gluster-users
> >>> >> >>>> >
> >>> >> >>>>
> >>> >> >>>
> >>> >>
> >>>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200806/669c511b/attachment.html>

From archon810 at gmail.com  Thu Aug  6 18:39:20 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Thu, 6 Aug 2020 11:39:20 -0700
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
In-Reply-To: <CAAV-989ZQf6oaStynwvmLpn3XheytO+5kw1Hw1h2utXWE=_Anw@mail.gmail.com>
References: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
	<CAAV-989ZQf6oaStynwvmLpn3XheytO+5kw1Hw1h2utXWE=_Anw@mail.gmail.com>
Message-ID: <CAD+dzQfccyjt0A5BOjwu0AZQ0WQq+t5_WB_uqHuoCxfTnWEcmA@mail.gmail.com>

Looks like someone built gluster 7.7 for OpenSUSE 15.1 after all. Yay.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Sun, Aug 2, 2020 at 11:46 PM Hu Bert <revirii at googlemail.com> wrote:

> Hi there,
>
> just wanted to say thanks to all the developers, maintainers etc. This
> release (7) has brought us a small but nice performance improvement.
> Utilization and IOs per disk decreased, latency dropped. See attached
> images.
>
> I read the release notes but couldn't identify the specific
> changes/features for this improvement. Maybe someone could point to
> them - but no hurry... :-)
>
>
> Best regards,
> Hubert
>
> Am Mi., 22. Juli 2020 um 18:27 Uhr schrieb Rinku Kothiya <
> rkothiya at redhat.com>:
> >
> > Hi,
> >
> > The Gluster community is pleased to announce the release of Gluster7.7
> (packages available at [1]).
> > Release notes for the release can be found at [2].
> >
> > Major changes, features and limitations addressed in this release:
> > None
> >
> > Please Note: Some of the packages are unavailable and we are working on
> it. We will release them soon.
> >
> > Thanks,
> > Gluster community
> >
> > References:
> >
> > [1] Packages for 7.7:
> > https://download.gluster.org/pub/gluster/glusterfs/7/7.7/
> >
> > [2] Release notes for 7.7:
> > https://docs.gluster.org/en/latest/release-notes/7.7/
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200806/e30e5c76/attachment.html>

From mathias.waack at seim-partner.de  Fri Aug  7 07:24:38 2020
From: mathias.waack at seim-partner.de (Mathias Waack)
Date: Fri, 7 Aug 2020 09:24:38 +0200
Subject: [Gluster-users] Repair after accident
In-Reply-To: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
Message-ID: <f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>

Hi all,

maybe I should add some more information:

The container which filled up the space was running on node x, which 
still shows a nearly filled fs:

192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster

nearly the same situation on the underlying brick partition on node x:

zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick

On node y the network card crashed, glusterfs shows the same values:

192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster

but different values on the brick:

zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick

I think this happened because glusterfs still has hardlinks to the 
deleted files on node x? So I can find these files with:

find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '

But now I am lost. How can I verify these files really belongs to the 
right container? Or can I just delete this files because there is no way 
to access it? Or offers glusterfs a way to solve this situation?

Mathias

On 05.08.20 15:48, Mathias Waack wrote:
> Hi all,
>
> we are running a gluster setup with two nodes:
>
> Status of volume: gvol
> Gluster process???????????????????????????? TCP Port? RDMA Port 
> Online? Pid
> ------------------------------------------------------------------------------ 
>
> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>
> Task Status of Volume gvol
> ------------------------------------------------------------------------------ 
>
> There are no active volume tasks
>
> The glusterfs hosts a bunch of containers with its data volumes. The 
> underlying fs is zfs. Few days ago one of the containers created a lot 
> of files in one of its data volumes, and at the end it completely 
> filled up the space of the glusterfs volume. But this happened only on 
> one host, on the other host there was still enough space. We finally 
> were able to identify this container and found out, the sizes of the 
> data on /zbrick were different on both hosts for this container. Now 
> we made the big mistake to delete these files on both hosts in the 
> /zbrick volume, not on the mounted glusterfs volume.
>
> Later we found the reason for this behavior: the network driver on the 
> second node partially crashed (which means we ware able to login on 
> the node, so we assumed the network was running, but the card was 
> already dropping packets at this time) at the same time, as the failed 
> container started to fill up the gluster volume. After rebooting the 
> second node? the gluster became available again.
>
> Now the glusterfs volume is running again- but it is still (nearly) 
> full: the files created by the container are not visible, but they 
> still count into amount of free space. How can we fix this?
>
> In addition there are some files which are no longer accessible since 
> this accident:
>
> tail access.log.old
> tail: cannot open 'access.log.old' for reading: Input/output error
>
> Looks like affected by this error are files which have been changed 
> during the accident. Is there a way to fix this too?
>
> Thanks
> ??? Mathias
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

From hunter86_bg at yahoo.com  Fri Aug  7 12:32:46 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Fri, 07 Aug 2020 15:32:46 +0300
Subject: [Gluster-users] Repair after accident
In-Reply-To: <f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
	<f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
Message-ID: <333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>

Have you tried to gluster heal and check if the files are back into their place?

I always thought that those hard links are used  by the healing mechanism  and if that is true - gluster should restore the files to their original location and then wiping the correct files from FUSE will be easy.

Best Regards,
Strahil Nikolov

?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>Hi all,
>
>maybe I should add some more information:
>
>The container which filled up the space was running on node x, which 
>still shows a nearly filled fs:
>
>192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>
>nearly the same situation on the underlying brick partition on node x:
>
>zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>
>On node y the network card crashed, glusterfs shows the same values:
>
>192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>
>but different values on the brick:
>
>zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>
>I think this happened because glusterfs still has hardlinks to the 
>deleted files on node x? So I can find these files with:
>
>find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>
>But now I am lost. How can I verify these files really belongs to the 
>right container? Or can I just delete this files because there is no
>way 
>to access it? Or offers glusterfs a way to solve this situation?
>
>Mathias
>
>On 05.08.20 15:48, Mathias Waack wrote:
>> Hi all,
>>
>> we are running a gluster setup with two nodes:
>>
>> Status of volume: gvol
>> Gluster process???????????????????????????? TCP Port? RDMA Port 
>> Online? Pid
>>
>------------------------------------------------------------------------------
>
>>
>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>>
>> Task Status of Volume gvol
>>
>------------------------------------------------------------------------------
>
>>
>> There are no active volume tasks
>>
>> The glusterfs hosts a bunch of containers with its data volumes. The 
>> underlying fs is zfs. Few days ago one of the containers created a
>lot 
>> of files in one of its data volumes, and at the end it completely 
>> filled up the space of the glusterfs volume. But this happened only
>on 
>> one host, on the other host there was still enough space. We finally 
>> were able to identify this container and found out, the sizes of the 
>> data on /zbrick were different on both hosts for this container. Now 
>> we made the big mistake to delete these files on both hosts in the 
>> /zbrick volume, not on the mounted glusterfs volume.
>>
>> Later we found the reason for this behavior: the network driver on
>the 
>> second node partially crashed (which means we ware able to login on 
>> the node, so we assumed the network was running, but the card was 
>> already dropping packets at this time) at the same time, as the
>failed 
>> container started to fill up the gluster volume. After rebooting the 
>> second node? the gluster became available again.
>>
>> Now the glusterfs volume is running again- but it is still (nearly) 
>> full: the files created by the container are not visible, but they 
>> still count into amount of free space. How can we fix this?
>>
>> In addition there are some files which are no longer accessible since
>
>> this accident:
>>
>> tail access.log.old
>> tail: cannot open 'access.log.old' for reading: Input/output error
>>
>> Looks like affected by this error are files which have been changed 
>> during the accident. Is there a way to fix this too?
>>
>> Thanks
>> ??? Mathias
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>________
>
>
>
>Community Meeting Calendar:
>
>Schedule -
>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users

From crl.langlois at gmail.com  Fri Aug  7 17:14:07 2020
From: crl.langlois at gmail.com (carl langlois)
Date: Fri, 7 Aug 2020 13:14:07 -0400
Subject: [Gluster-users] Keep having unsync entries
Message-ID: <CAMqdCPEFbcKvgS=-6AvCzLY8EQY__aFxVL9rbfA7bgTmPPgvQw@mail.gmail.com>

Hi all,

I am currently upgrading my ovirt cluster and after doing the upgrade on
one node i end up having unsync entries that heal by the headl command.
My setup is a 2+1  with 4 volume.
here is a snapshot of one a volume info
Volume Name: data
Type: Replicate
Volume ID: 71c999a4-b769-471f-8169-a1a66b28f9b0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ovhost1:/gluster_bricks/data/data
Brick2: ovhost2:/gluster_bricks/data/data
Brick3: ovhost3:/gluster_bricks/data/data (arbiter)
Options Reconfigured:
server.allow-insecure: on
nfs.disable: on
transport.address-family: inet
performance.quick-read: off
performance.read-ahead: off
performance.io-cache: off
performance.low-prio-threads: 32
network.remote-dio: enable
cluster.eager-lock: enable
cluster.quorum-type: auto
cluster.server-quorum-type: server
cluster.data-self-heal-algorithm: full
cluster.locking-scheme: granular
cluster.shd-max-threads: 8
cluster.shd-wait-qlength: 10000
features.shard: on
user.cifs: off
storage.owner-uid: 36
storage.owner-gid: 36
network.ping-timeout: 30
performance.strict-o-direct: on
cluster.granular-entry-heal: enable
features.shard-block-size: 64MB

Also the output of v headl data info

gluster> v heal data info
Brick ovhost1:/gluster_bricks/data/data
/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 2

Brick ovhost2:/gluster_bricks/data/data
Status: Connected
Number of entries: 0

Brick ovhost3:/gluster_bricks/data/data
/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
/__DIRECT_IO_TEST__
Status: Connected
Number of entries: 2

does not seem to be a split brain also.
gluster> v heal data info split-brain
Brick ovhost1:/gluster_bricks/data/data
Status: Connected
Number of entries in split-brain: 0

Brick ovhost2:/gluster_bricks/data/data
Status: Connected
Number of entries in split-brain: 0

Brick ovhost3:/gluster_bricks/data/data
Status: Connected
Number of entries in split-brain: 0

not sure how to resolve this issue.
gluster version is 3.2.15

Regards

Carl
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200807/cd3db634/attachment.html>

From hunter86_bg at yahoo.com  Fri Aug  7 18:00:15 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Fri, 07 Aug 2020 21:00:15 +0300
Subject: [Gluster-users] Keep having unsync entries
In-Reply-To: <CAMqdCPEFbcKvgS=-6AvCzLY8EQY__aFxVL9rbfA7bgTmPPgvQw@mail.gmail.com>
References: <CAMqdCPEFbcKvgS=-6AvCzLY8EQY__aFxVL9rbfA7bgTmPPgvQw@mail.gmail.com>
Message-ID: <6EA80B60-FB4B-4B68-883C-E81BC1A95FFC@yahoo.com>

I think Ravi made a change to prevent that in gluster v6.6

You can  rsync the  2 files from  ovhost1 and run a full heal (I don't know why heal without 'full' doesn't clean up the entries).

Anyways, ovirt can live without these 2 , but as you don't want to risk any downtimes  - just rsync them from ovhost1 and run a 'gluster volume heal data full'.

By the way , which version of ovirt do you use ? Gluster v3 was  used  in 4.2.X 

Best Regards,
Strahil Nikolov



?? 7 ?????? 2020 ?. 20:14:07 GMT+03:00, carl langlois <crl.langlois at gmail.com> ??????:
>Hi all,
>
>I am currently upgrading my ovirt cluster and after doing the upgrade
>on
>one node i end up having unsync entries that heal by the headl command.
>My setup is a 2+1  with 4 volume.
>here is a snapshot of one a volume info
>Volume Name: data
>Type: Replicate
>Volume ID: 71c999a4-b769-471f-8169-a1a66b28f9b0
>Status: Started
>Snapshot Count: 0
>Number of Bricks: 1 x (2 + 1) = 3
>Transport-type: tcp
>Bricks:
>Brick1: ovhost1:/gluster_bricks/data/data
>Brick2: ovhost2:/gluster_bricks/data/data
>Brick3: ovhost3:/gluster_bricks/data/data (arbiter)
>Options Reconfigured:
>server.allow-insecure: on
>nfs.disable: on
>transport.address-family: inet
>performance.quick-read: off
>performance.read-ahead: off
>performance.io-cache: off
>performance.low-prio-threads: 32
>network.remote-dio: enable
>cluster.eager-lock: enable
>cluster.quorum-type: auto
>cluster.server-quorum-type: server
>cluster.data-self-heal-algorithm: full
>cluster.locking-scheme: granular
>cluster.shd-max-threads: 8
>cluster.shd-wait-qlength: 10000
>features.shard: on
>user.cifs: off
>storage.owner-uid: 36
>storage.owner-gid: 36
>network.ping-timeout: 30
>performance.strict-o-direct: on
>cluster.granular-entry-heal: enable
>features.shard-block-size: 64MB
>
>Also the output of v headl data info
>
>gluster> v heal data info
>Brick ovhost1:/gluster_bricks/data/data
>/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
>/__DIRECT_IO_TEST__
>Status: Connected
>Number of entries: 2
>
>Brick ovhost2:/gluster_bricks/data/data
>Status: Connected
>Number of entries: 0
>
>Brick ovhost3:/gluster_bricks/data/data
>/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
>/__DIRECT_IO_TEST__
>Status: Connected
>Number of entries: 2
>
>does not seem to be a split brain also.
>gluster> v heal data info split-brain
>Brick ovhost1:/gluster_bricks/data/data
>Status: Connected
>Number of entries in split-brain: 0
>
>Brick ovhost2:/gluster_bricks/data/data
>Status: Connected
>Number of entries in split-brain: 0
>
>Brick ovhost3:/gluster_bricks/data/data
>Status: Connected
>Number of entries in split-brain: 0
>
>not sure how to resolve this issue.
>gluster version is 3.2.15
>
>Regards
>
>Carl

From gilberto.nunes32 at gmail.com  Fri Aug  7 18:03:06 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Fri, 7 Aug 2020 15:03:06 -0300
Subject: [Gluster-users] Pending healing...
Message-ID: <CAOKSTBt1vfY+i=5Ah_ijw5eY_m=SKFBgYiFsbW9syyS-3VqD8g@mail.gmail.com>

Hi

I have a pending entry like this

gluster vol heal VMS info summary
Brick glusterfs01:/DATA/vms
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick glusterfs02:/DATA/vms
Status: Connected
Total Number of entries: 1
Number of entries in heal pending: 1
Number of entries in split-brain: 0
Number of entries possibly healing: 0

How can I solve this?
Should I follow this?

https://icicimov.github.io/blog/high-availability/GlusterFS-metadata-split-brain-recovery/



---
Gilberto Nunes Ferreira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200807/0cf14af2/attachment.html>

From bob at computerisms.ca  Fri Aug  7 18:27:56 2020
From: bob at computerisms.ca (Computerisms Corporation)
Date: Fri, 7 Aug 2020 11:27:56 -0700
Subject: [Gluster-users] performance
In-Reply-To: <CAD+dzQc4b9qe+=5jNksw-u4B1VZtFPzBCkA=3AkdwSBp998Zzw@mail.gmail.com>
References: <696b3c28-519b-c3e3-ce5d-e60d2f194d4c@computerisms.ca>
	<7991483E-5365-4C87-89FA-C871AED18062@yahoo.com>
	<345b06c4-5996-9aa3-f846-0944c60ee398@computerisms.ca>
	<2CD68ED2-199F-407D-B0CC-385793BA16FD@yahoo.com>
	<64ee1b88-42d6-75d2-05ff-4703d168cc25@computerisms.ca>
	<d3ac7f18-858c-f46b-994f-24ab8f20f648@computerisms.ca>
	<CAD+dzQc4b9qe+=5jNksw-u4B1VZtFPzBCkA=3AkdwSBp998Zzw@mail.gmail.com>
Message-ID: <40dab403-411b-aa6a-83e5-7a45021ac866@computerisms.ca>

Hi Artem and others,

Happy to report the system has been relatively stable for the remainder 
of the week.  I have one wordpress site that seems to get hung processes 
when someone logs in with an incorrect password.  Since it is only one, 
and reliably reproduceable, I am not sure if the issue is to do with 
Gluster or Wordpress itself, but afaik it was not doing it some months 
back before the system was using Gluster so I am guessing some combo of 
both.

Regardless, that is the one and only time apache processes stacked up to 
over 150, and that still only brought the load average up to just under 
25; the system did go a bit sluggish, but remained fairly responsive 
throughout until I restarted apache.  Otherwise 15 minute load average 
consistently runs between 8 and 11 during peak hours and between 4 and 7 
during off hours, and other than the one time I have not seen the 
one-minute load average go over 15.  all resources still spike to full 
capacity from time to time, but it never remains that way for long like 
it did before.

For site responsiveness, first visit to any given site is quite slow, 
like 3-5 seconds on straight html pages, 10-15 seconds for some of the 
more bloated WP themes, but clicking links within the site after the 
first page is loaded is relatively quick, like 1 second on straight html 
pages, and ~5-6 seconds on the bloated themes.  Again, not sure if that 
is a Gluster related thing or something else.

So, still holding my breath a bit, but seems this solution is working, 
at least for me.  I haven't played with any of the other settings yet to 
see if I can improve it further, probably will next week.  thinking to 
increase the write behind window size further to see what happens, as 
well as play with the settings suggested by Strahil.

On 2020-08-05 5:28 p.m., Artem Russakovskii wrote:
> I'm very curious whether these improvements hold up over the next few 
> days. Please report back.
> 
> Sincerely,
> Artem
> 
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror 
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
> 
> 
> On Wed, Aug 5, 2020 at 9:44 AM Computerisms Corporation 
> <bob at computerisms.ca <mailto:bob at computerisms.ca>> wrote:
> 
>     Hi List,
> 
>      > So, we just moved into a quieter time of the day, but maybe I just
>      > stumbled onto something.? I was trying to figure out if/how I could
>      > throw more RAM at the problem.? gluster docs says write behind is
>     not a
>      > cache unless flush-behind is on.? So seems that is a way to throw
>     ram to
>      > it?? I put performance.write-behind-window-size: 512MB and
>      > performance.flush-behind: on and the whole system calmed down pretty
>      > much immediately.? could be just timing, though, will have to see
>      > tomorrow during business hours whether the system stays at a
>     reasonable
>      > load.
> 
>     so reporting back that this seems to have definitely had a significant
>     positive effect.
> 
>     So far today I have not seen the load average climb over 13 with the
>     15minute average hovering around 7.? cpus are still spiking from
>     time to
>     time, but they are not staying maxed out all the time, and frequently I
>     am seeing brief periods of up to 80% idle.? glusterfs process still
>     spiking up to 180% or so, but consistently running around 70%, and the
>     brick processes still spiking up to 70-80%, but consistently running
>     around 20%.? Disk has only been above 50% in atop once so far today
>     when
>     it spiked up to 92%, and still lots of RAM left over.? So far nload
>     even
>     seems indicates I could get away with a 100Mbit network connection.
>     Websites are snappy relative to what they were, still a bit sluggish on
>     the first page of any given site, but tolerable or close to.? Apache
>     processes are opening and closing right away, instead of stacking up.
> 
>     Overall, system is performing pretty much like I would expect it to
>     without gluster.? I haven't played with any of the other settings yet,
>     just going to leave it like this for a day.
> 
>     I have to admit I am a little bit suspicious.? I have been arguing with
>     Gluster for a very long time, and I have never known it to play this
>     nice.? kind feels like when your girl tells you she is "fine";
>     conversation has stopped, but you aren't really sure if it's done...
> 
>      >
>      > I will still test the other options you suggested tonight,
>     though, this
>      > is probably too good to be true.
>      >
>      > Can't thank you enough for your input, Strahil, your help is truly
>      > appreciated!
>      >
>      >
>      >
>      >
>      >
>      >
>      >>
>      >>>>
>      >>>>
>      >>>> Best Regards,
>      >>>> Strahil Nikolov
>      >>>>
>      >>> ________
>      >>>
>      >>>
>      >>>
>      >>> Community Meeting Calendar:
>      >>>
>      >>> Schedule -
>      >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>      >>> Bridge: https://bluejeans.com/441850968
>      >>>
>      >>> Gluster-users mailing list
>      >>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>      >>> https://lists.gluster.org/mailman/listinfo/gluster-users
>      > ________
>      >
>      >
>      >
>      > Community Meeting Calendar:
>      >
>      > Schedule -
>      > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>      > Bridge: https://bluejeans.com/441850968
>      >
>      > Gluster-users mailing list
>      > Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>      > https://lists.gluster.org/mailman/listinfo/gluster-users
>     ________
> 
> 
> 
>     Community Meeting Calendar:
> 
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
> 
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
> 

From crl.langlois at gmail.com  Fri Aug  7 18:34:27 2020
From: crl.langlois at gmail.com (carl langlois)
Date: Fri, 7 Aug 2020 14:34:27 -0400
Subject: [Gluster-users] Keep having unsync entries
In-Reply-To: <6EA80B60-FB4B-4B68-883C-E81BC1A95FFC@yahoo.com>
References: <CAMqdCPEFbcKvgS=-6AvCzLY8EQY__aFxVL9rbfA7bgTmPPgvQw@mail.gmail.com>
	<6EA80B60-FB4B-4B68-883C-E81BC1A95FFC@yahoo.com>
Message-ID: <CAMqdCPHauF-v+xiBvJo6i_3auTH0C84rPc9W2sE8hWZBO=bo2w@mail.gmail.com>

Hi Strahil

Thanks for the quick answer. I will try to rsync them manually like you
suggested.
I am still on 4.2.x. I am in the process of moving my cluster to 4.3 but
need to move to 4.2.8 first. But moving to 4.2.8 is not an easy task since
I need to pin the base os to 7.6 before moving to 4.2.8.
Hope moving to 4.3 will be easy :-) ... I suspect 4.4 to be a pain to
upgrade since there is no upgrade path from 7.8 -> 8 ... :-(
Anyway thanks for the hints.

Regards
Carl


On Fri, Aug 7, 2020 at 2:00 PM Strahil Nikolov <hunter86_bg at yahoo.com>
wrote:

> I think Ravi made a change to prevent that in gluster v6.6
>
> You can  rsync the  2 files from  ovhost1 and run a full heal (I don't
> know why heal without 'full' doesn't clean up the entries).
>
> Anyways, ovirt can live without these 2 , but as you don't want to risk
> any downtimes  - just rsync them from ovhost1 and run a 'gluster volume
> heal data full'.
>
> By the way , which version of ovirt do you use ? Gluster v3 was  used  in
> 4.2.X
>
> Best Regards,
> Strahil Nikolov
>
>
>
> ?? 7 ?????? 2020 ?. 20:14:07 GMT+03:00, carl langlois <
> crl.langlois at gmail.com> ??????:
> >Hi all,
> >
> >I am currently upgrading my ovirt cluster and after doing the upgrade
> >on
> >one node i end up having unsync entries that heal by the headl command.
> >My setup is a 2+1  with 4 volume.
> >here is a snapshot of one a volume info
> >Volume Name: data
> >Type: Replicate
> >Volume ID: 71c999a4-b769-471f-8169-a1a66b28f9b0
> >Status: Started
> >Snapshot Count: 0
> >Number of Bricks: 1 x (2 + 1) = 3
> >Transport-type: tcp
> >Bricks:
> >Brick1: ovhost1:/gluster_bricks/data/data
> >Brick2: ovhost2:/gluster_bricks/data/data
> >Brick3: ovhost3:/gluster_bricks/data/data (arbiter)
> >Options Reconfigured:
> >server.allow-insecure: on
> >nfs.disable: on
> >transport.address-family: inet
> >performance.quick-read: off
> >performance.read-ahead: off
> >performance.io-cache: off
> >performance.low-prio-threads: 32
> >network.remote-dio: enable
> >cluster.eager-lock: enable
> >cluster.quorum-type: auto
> >cluster.server-quorum-type: server
> >cluster.data-self-heal-algorithm: full
> >cluster.locking-scheme: granular
> >cluster.shd-max-threads: 8
> >cluster.shd-wait-qlength: 10000
> >features.shard: on
> >user.cifs: off
> >storage.owner-uid: 36
> >storage.owner-gid: 36
> >network.ping-timeout: 30
> >performance.strict-o-direct: on
> >cluster.granular-entry-heal: enable
> >features.shard-block-size: 64MB
> >
> >Also the output of v headl data info
> >
> >gluster> v heal data info
> >Brick ovhost1:/gluster_bricks/data/data
> >/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
> >/__DIRECT_IO_TEST__
> >Status: Connected
> >Number of entries: 2
> >
> >Brick ovhost2:/gluster_bricks/data/data
> >Status: Connected
> >Number of entries: 0
> >
> >Brick ovhost3:/gluster_bricks/data/data
> >/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
> >/__DIRECT_IO_TEST__
> >Status: Connected
> >Number of entries: 2
> >
> >does not seem to be a split brain also.
> >gluster> v heal data info split-brain
> >Brick ovhost1:/gluster_bricks/data/data
> >Status: Connected
> >Number of entries in split-brain: 0
> >
> >Brick ovhost2:/gluster_bricks/data/data
> >Status: Connected
> >Number of entries in split-brain: 0
> >
> >Brick ovhost3:/gluster_bricks/data/data
> >Status: Connected
> >Number of entries in split-brain: 0
> >
> >not sure how to resolve this issue.
> >gluster version is 3.2.15
> >
> >Regards
> >
> >Carl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200807/29194d5f/attachment.html>

From mathias.waack at seim-partner.de  Fri Aug  7 18:39:59 2020
From: mathias.waack at seim-partner.de (Mathias Waack)
Date: Fri, 7 Aug 2020 20:39:59 +0200
Subject: [Gluster-users] Repair after accident
In-Reply-To: <333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
	<f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
	<333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>
Message-ID: <151fd00d-94d1-1053-7202-bcd60735c1fc@seim-partner.de>

Hi Strahil,

but I cannot find these files in the heal info:

find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
...
7443397? 132463 -rw-------?? 1 999????? docker?? 1073741824 Aug? 3 10:35 
/zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983

Now looking for this file in the heal infos:

gluster volume heal gvol info | grep b53c8e46-068b-4286-94a6-7cf54f711983

shows nothing.

So I do not know, what I have to heal...

Mathias

On 07.08.20 14:32, Strahil Nikolov wrote:
> Have you tried to gluster heal and check if the files are back into their place?
>
> I always thought that those hard links are used  by the healing mechanism  and if that is true - gluster should restore the files to their original location and then wiping the correct files from FUSE will be easy.
>
> Best Regards,
> Strahil Nikolov
>
> ?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>> Hi all,
>>
>> maybe I should add some more information:
>>
>> The container which filled up the space was running on node x, which
>> still shows a nearly filled fs:
>>
>> 192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>
>> nearly the same situation on the underlying brick partition on node x:
>>
>> zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>>
>> On node y the network card crashed, glusterfs shows the same values:
>>
>> 192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>
>> but different values on the brick:
>>
>> zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>>
>> I think this happened because glusterfs still has hardlinks to the
>> deleted files on node x? So I can find these files with:
>>
>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>
>> But now I am lost. How can I verify these files really belongs to the
>> right container? Or can I just delete this files because there is no
>> way
>> to access it? Or offers glusterfs a way to solve this situation?
>>
>> Mathias
>>
>> On 05.08.20 15:48, Mathias Waack wrote:
>>> Hi all,
>>>
>>> we are running a gluster setup with two nodes:
>>>
>>> Status of volume: gvol
>>> Gluster process???????????????????????????? TCP Port? RDMA Port
>>> Online? Pid
>>>
>> ------------------------------------------------------------------------------
>>
>>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
>>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>>>
>>> Task Status of Volume gvol
>>>
>> ------------------------------------------------------------------------------
>>
>>> There are no active volume tasks
>>>
>>> The glusterfs hosts a bunch of containers with its data volumes. The
>>> underlying fs is zfs. Few days ago one of the containers created a
>> lot
>>> of files in one of its data volumes, and at the end it completely
>>> filled up the space of the glusterfs volume. But this happened only
>> on
>>> one host, on the other host there was still enough space. We finally
>>> were able to identify this container and found out, the sizes of the
>>> data on /zbrick were different on both hosts for this container. Now
>>> we made the big mistake to delete these files on both hosts in the
>>> /zbrick volume, not on the mounted glusterfs volume.
>>>
>>> Later we found the reason for this behavior: the network driver on
>> the
>>> second node partially crashed (which means we ware able to login on
>>> the node, so we assumed the network was running, but the card was
>>> already dropping packets at this time) at the same time, as the
>> failed
>>> container started to fill up the gluster volume. After rebooting the
>>> second node? the gluster became available again.
>>>
>>> Now the glusterfs volume is running again- but it is still (nearly)
>>> full: the files created by the container are not visible, but they
>>> still count into amount of free space. How can we fix this?
>>>
>>> In addition there are some files which are no longer accessible since
>>> this accident:
>>>
>>> tail access.log.old
>>> tail: cannot open 'access.log.old' for reading: Input/output error
>>>
>>> Looks like affected by this error are files which have been changed
>>> during the accident. Is there a way to fix this too?
>>>
>>> Thanks
>>>  ??? Mathias
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users

From gilberto.nunes32 at gmail.com  Sat Aug  8 04:27:14 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Sat, 8 Aug 2020 01:27:14 -0300
Subject: [Gluster-users] Monitoring tools for GlusterFS
Message-ID: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>

Hi guys...  I miss some tools that could be used in order to monitor
healing for example, and others things like resources used...  What do you
recommend?
Tools that could be used in CLI but that shows a percentage as healing is
under way would be nice!

Thanks.

---
Gilberto Nunes Ferreira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200808/bec17564/attachment.html>

From adrianquintero at gmail.com  Sat Aug  8 05:09:38 2020
From: adrianquintero at gmail.com (Adrian Quintero)
Date: Sat, 8 Aug 2020 01:09:38 -0400
Subject: [Gluster-users] Monitoring tools for GlusterFS
In-Reply-To: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>
References: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>
Message-ID: <CADWatvdQ5tNn0SzAR3R+NrZSrv-A==1amWQDXvMhbST-9ksFqw@mail.gmail.com>

Hello Gilberto,
I've had the same questions and some of the community friends were kind
enough to send me a few things to look for

However as you have stated it would be interesting to know exactly how to
monitor the healing process.
A friend mentioned some things we should monitor from a
Gluster's perspective but not limited to:
- Thin LVM (pool should never get full)
- Number of snapshots
- Quotas (both inodes and total size)
- GeoRep  Status
- Gluster brick status (2  out of 3 down and it's an outage for ' replica
3' volumes)
- Pending Heals
- Errors in Gluster brick logs can indicate FS issues
-  Errors  in other  Gluster  logs


As for the resources used please look at
https://docs.gluster.org/en/latest/Administrator%20Guide/Monitoring%20Workload/
it has helped me quite a bit.

If you find out about the healing process and how to monitor it , please
let me know.


Regards,

Adrian Quintero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200808/4a7793c1/attachment.html>

From hunter86_bg at yahoo.com  Sat Aug  8 06:57:16 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sat, 8 Aug 2020 06:57:16 +0000 (UTC)
Subject: [Gluster-users] Keep having unsync entries
In-Reply-To: <CAMqdCPHauF-v+xiBvJo6i_3auTH0C84rPc9W2sE8hWZBO=bo2w@mail.gmail.com>
References: <CAMqdCPEFbcKvgS=-6AvCzLY8EQY__aFxVL9rbfA7bgTmPPgvQw@mail.gmail.com>
	<6EA80B60-FB4B-4B68-883C-E81BC1A95FFC@yahoo.com>
	<CAMqdCPHauF-v+xiBvJo6i_3auTH0C84rPc9W2sE8hWZBO=bo2w@mail.gmail.com>
Message-ID: <855486521.1248366.1596869836761@mail.yahoo.com>

Keep in mind that 4.3 is using Gluster v6 .
I'm on latest 4.3.10 but with Gluster v7.

I was hit by a very rare ACL bug (reported by some other guys here and in oVirt ML) and thus I will recommend you to test functionality after every gluster major upgrade (start,stop,create snapshot,remove snapshot, etc).

In my case 6.6+ and 7.1+ were problematic, but you got no way to skip them.


Best Regards,
Strahil Nikolov








? ?????, 7 ?????? 2020 ?., 21:34:39 ???????+3, carl langlois <crl.langlois at gmail.com> ??????: 





Hi?Strahil

Thanks for the quick answer. I will try to rsync them manually like you suggested.??
I am still on 4.2.x. I am in the process of moving my cluster to 4.3 but need to move to 4.2.8 first. But moving to 4.2.8 is not an easy task since I need to pin the base os to 7.6 before moving to 4.2.8.?
Hope moving to 4.3 will be easy :-) ... I suspect 4.4 to be a pain to upgrade since there is no upgrade path from 7.8 -> 8 ... :-(
Anyway thanks for the hints.

Regards
Carl


On Fri, Aug 7, 2020 at 2:00 PM Strahil Nikolov <hunter86_bg at yahoo.com> wrote:
> I think Ravi made a change to prevent that in gluster v6.6
> 
> You can? rsync the? 2 files from? ovhost1 and run a full heal (I don't know why heal without 'full' doesn't clean up the entries).
> 
> Anyways, ovirt can live without these 2 , but as you don't want to risk any downtimes? - just rsync them from ovhost1 and run a 'gluster volume heal data full'.
> 
> By the way , which version of ovirt do you use ? Gluster v3 was? used? in 4.2.X 
> 
> Best Regards,
> Strahil Nikolov
> 
> 
> 
> ?? 7 ?????? 2020 ?. 20:14:07 GMT+03:00, carl langlois <crl.langlois at gmail.com> ??????:
>>Hi all,
>>
>>I am currently upgrading my ovirt cluster and after doing the upgrade
>>on
>>one node i end up having unsync entries that heal by the headl command.
>>My setup is a 2+1? with 4 volume.
>>here is a snapshot of one a volume info
>>Volume Name: data
>>Type: Replicate
>>Volume ID: 71c999a4-b769-471f-8169-a1a66b28f9b0
>>Status: Started
>>Snapshot Count: 0
>>Number of Bricks: 1 x (2 + 1) = 3
>>Transport-type: tcp
>>Bricks:
>>Brick1: ovhost1:/gluster_bricks/data/data
>>Brick2: ovhost2:/gluster_bricks/data/data
>>Brick3: ovhost3:/gluster_bricks/data/data (arbiter)
>>Options Reconfigured:
>>server.allow-insecure: on
>>nfs.disable: on
>>transport.address-family: inet
>>performance.quick-read: off
>>performance.read-ahead: off
>>performance.io-cache: off
>>performance.low-prio-threads: 32
>>network.remote-dio: enable
>>cluster.eager-lock: enable
>>cluster.quorum-type: auto
>>cluster.server-quorum-type: server
>>cluster.data-self-heal-algorithm: full
>>cluster.locking-scheme: granular
>>cluster.shd-max-threads: 8
>>cluster.shd-wait-qlength: 10000
>>features.shard: on
>>user.cifs: off
>>storage.owner-uid: 36
>>storage.owner-gid: 36
>>network.ping-timeout: 30
>>performance.strict-o-direct: on
>>cluster.granular-entry-heal: enable
>>features.shard-block-size: 64MB
>>
>>Also the output of v headl data info
>>
>>gluster> v heal data info
>>Brick ovhost1:/gluster_bricks/data/data
>>/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
>>/__DIRECT_IO_TEST__
>>Status: Connected
>>Number of entries: 2
>>
>>Brick ovhost2:/gluster_bricks/data/data
>>Status: Connected
>>Number of entries: 0
>>
>>Brick ovhost3:/gluster_bricks/data/data
>>/4e59777c-5b7b-4bf1-8463-1c818067955e/dom_md/ids
>>/__DIRECT_IO_TEST__
>>Status: Connected
>>Number of entries: 2
>>
>>does not seem to be a split brain also.
>>gluster> v heal data info split-brain
>>Brick ovhost1:/gluster_bricks/data/data
>>Status: Connected
>>Number of entries in split-brain: 0
>>
>>Brick ovhost2:/gluster_bricks/data/data
>>Status: Connected
>>Number of entries in split-brain: 0
>>
>>Brick ovhost3:/gluster_bricks/data/data
>>Status: Connected
>>Number of entries in split-brain: 0
>>
>>not sure how to resolve this issue.
>>gluster version is 3.2.15
>>
>>Regards
>>
>>Carl
> 

From hunter86_bg at yahoo.com  Sat Aug  8 07:00:55 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sat, 8 Aug 2020 07:00:55 +0000 (UTC)
Subject: [Gluster-users] Repair after accident
In-Reply-To: <151fd00d-94d1-1053-7202-bcd60735c1fc@seim-partner.de>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
	<f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
	<333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>
	<151fd00d-94d1-1053-7202-bcd60735c1fc@seim-partner.de>
Message-ID: <1181804313.1257349.1596870055548@mail.yahoo.com>

In glusterfs the long string is called "gfid" and does not represent the name.

Best Regards,
Strahil Nikolov






? ?????, 7 ?????? 2020 ?., 21:40:11 ???????+3, Mathias Waack <mathias.waack at seim-partner.de> ??????: 





Hi Strahil,

but I cannot find these files in the heal info:

find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
...
7443397? 132463 -rw-------?? 1 999????? docker?? 1073741824 Aug? 3 10:35 
/zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983

Now looking for this file in the heal infos:

gluster volume heal gvol info | grep b53c8e46-068b-4286-94a6-7cf54f711983

shows nothing.

So I do not know, what I have to heal...

Mathias

On 07.08.20 14:32, Strahil Nikolov wrote:
> Have you tried to gluster heal and check if the files are back into their place?
>
> I always thought that those hard links are used? by the healing mechanism? and if that is true - gluster should restore the files to their original location and then wiping the correct files from FUSE will be easy.
>
> Best Regards,
> Strahil Nikolov
>
> ?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>> Hi all,
>>
>> maybe I should add some more information:
>>
>> The container which filled up the space was running on node x, which
>> still shows a nearly filled fs:
>>
>> 192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>
>> nearly the same situation on the underlying brick partition on node x:
>>
>> zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>>
>> On node y the network card crashed, glusterfs shows the same values:
>>
>> 192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>
>> but different values on the brick:
>>
>> zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>>
>> I think this happened because glusterfs still has hardlinks to the
>> deleted files on node x? So I can find these files with:
>>
>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>
>> But now I am lost. How can I verify these files really belongs to the
>> right container? Or can I just delete this files because there is no
>> way
>> to access it? Or offers glusterfs a way to solve this situation?
>>
>> Mathias
>>
>> On 05.08.20 15:48, Mathias Waack wrote:
>>> Hi all,
>>>
>>> we are running a gluster setup with two nodes:
>>>
>>> Status of volume: gvol
>>> Gluster process???????????????????????????? TCP Port? RDMA Port
>>> Online? Pid
>>>
>> ------------------------------------------------------------------------------
>>
>>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
>>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>>>
>>> Task Status of Volume gvol
>>>
>> ------------------------------------------------------------------------------
>>
>>> There are no active volume tasks
>>>
>>> The glusterfs hosts a bunch of containers with its data volumes. The
>>> underlying fs is zfs. Few days ago one of the containers created a
>> lot
>>> of files in one of its data volumes, and at the end it completely
>>> filled up the space of the glusterfs volume. But this happened only
>> on
>>> one host, on the other host there was still enough space. We finally
>>> were able to identify this container and found out, the sizes of the
>>> data on /zbrick were different on both hosts for this container. Now
>>> we made the big mistake to delete these files on both hosts in the
>>> /zbrick volume, not on the mounted glusterfs volume.
>>>
>>> Later we found the reason for this behavior: the network driver on
>> the
>>> second node partially crashed (which means we ware able to login on
>>> the node, so we assumed the network was running, but the card was
>>> already dropping packets at this time) at the same time, as the
>> failed
>>> container started to fill up the gluster volume. After rebooting the
>>> second node? the gluster became available again.
>>>
>>> Now the glusterfs volume is running again- but it is still (nearly)
>>> full: the files created by the container are not visible, but they
>>> still count into amount of free space. How can we fix this?
>>>
>>> In addition there are some files which are no longer accessible since
>>> this accident:
>>>
>>> tail access.log.old
>>> tail: cannot open 'access.log.old' for reading: Input/output error
>>>
>>> Looks like affected by this error are files which have been changed
>>> during the accident. Is there a way to fix this too?
>>>
>>> Thanks
>>>? ??? Mathias
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users

>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
________



Community Meeting Calendar:

Schedule -
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
Bridge: https://bluejeans.com/441850968

Gluster-users mailing list
Gluster-users at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

From mathias.waack at seim-partner.de  Sat Aug  8 15:02:10 2020
From: mathias.waack at seim-partner.de (Mathias Waack)
Date: Sat, 8 Aug 2020 17:02:10 +0200
Subject: [Gluster-users] Repair after accident
In-Reply-To: <1181804313.1257349.1596870055548@mail.yahoo.com>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
	<f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
	<333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>
	<151fd00d-94d1-1053-7202-bcd60735c1fc@seim-partner.de>
	<1181804313.1257349.1596870055548@mail.yahoo.com>
Message-ID: <235f1f25-c9dd-cb75-3f49-b4a82bffb4d6@seim-partner.de>

So b53c8e46-068b-4286-94a6-7cf54f711983 is not a gfid? What else is it?

Mathias

On 08.08.20 09:00, Strahil Nikolov wrote:
> In glusterfs the long string is called "gfid" and does not represent the name.
>
> Best Regards,
> Strahil Nikolov
>
>
>
>
>
>
> ? ?????, 7 ?????? 2020 ?., 21:40:11 ???????+3, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>
>
>
>
>
> Hi Strahil,
>
> but I cannot find these files in the heal info:
>
> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
> ...
> 7443397? 132463 -rw-------?? 1 999????? docker?? 1073741824 Aug? 3 10:35
> /zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983
>
> Now looking for this file in the heal infos:
>
> gluster volume heal gvol info | grep b53c8e46-068b-4286-94a6-7cf54f711983
>
> shows nothing.
>
> So I do not know, what I have to heal...
>
> Mathias
>
> On 07.08.20 14:32, Strahil Nikolov wrote:
>> Have you tried to gluster heal and check if the files are back into their place?
>>
>> I always thought that those hard links are used? by the healing mechanism? and if that is true - gluster should restore the files to their original location and then wiping the correct files from FUSE will be easy.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>>> Hi all,
>>>
>>> maybe I should add some more information:
>>>
>>> The container which filled up the space was running on node x, which
>>> still shows a nearly filled fs:
>>>
>>> 192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>
>>> nearly the same situation on the underlying brick partition on node x:
>>>
>>> zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>>>
>>> On node y the network card crashed, glusterfs shows the same values:
>>>
>>> 192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>
>>> but different values on the brick:
>>>
>>> zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>>>
>>> I think this happened because glusterfs still has hardlinks to the
>>> deleted files on node x? So I can find these files with:
>>>
>>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>>
>>> But now I am lost. How can I verify these files really belongs to the
>>> right container? Or can I just delete this files because there is no
>>> way
>>> to access it? Or offers glusterfs a way to solve this situation?
>>>
>>> Mathias
>>>
>>> On 05.08.20 15:48, Mathias Waack wrote:
>>>> Hi all,
>>>>
>>>> we are running a gluster setup with two nodes:
>>>>
>>>> Status of volume: gvol
>>>> Gluster process???????????????????????????? TCP Port? RDMA Port
>>>> Online? Pid
>>>>
>>> ------------------------------------------------------------------------------
>>>
>>>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>>>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>>>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
>>>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>>>>
>>>> Task Status of Volume gvol
>>>>
>>> ------------------------------------------------------------------------------
>>>
>>>> There are no active volume tasks
>>>>
>>>> The glusterfs hosts a bunch of containers with its data volumes. The
>>>> underlying fs is zfs. Few days ago one of the containers created a
>>> lot
>>>> of files in one of its data volumes, and at the end it completely
>>>> filled up the space of the glusterfs volume. But this happened only
>>> on
>>>> one host, on the other host there was still enough space. We finally
>>>> were able to identify this container and found out, the sizes of the
>>>> data on /zbrick were different on both hosts for this container. Now
>>>> we made the big mistake to delete these files on both hosts in the
>>>> /zbrick volume, not on the mounted glusterfs volume.
>>>>
>>>> Later we found the reason for this behavior: the network driver on
>>> the
>>>> second node partially crashed (which means we ware able to login on
>>>> the node, so we assumed the network was running, but the card was
>>>> already dropping packets at this time) at the same time, as the
>>> failed
>>>> container started to fill up the gluster volume. After rebooting the
>>>> second node? the gluster became available again.
>>>>
>>>> Now the glusterfs volume is running again- but it is still (nearly)
>>>> full: the files created by the container are not visible, but they
>>>> still count into amount of free space. How can we fix this?
>>>>
>>>> In addition there are some files which are no longer accessible since
>>>> this accident:
>>>>
>>>> tail access.log.old
>>>> tail: cannot open 'access.log.old' for reading: Input/output error
>>>>
>>>> Looks like affected by this error are files which have been changed
>>>> during the accident. Is there a way to fix this too?
>>>>
>>>> Thanks
>>>>  ? ??? Mathias
>>>>
>>>>
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

From hunter86_bg at yahoo.com  Sat Aug  8 16:05:23 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sat, 08 Aug 2020 19:05:23 +0300
Subject: [Gluster-users] Repair after accident
In-Reply-To: <235f1f25-c9dd-cb75-3f49-b4a82bffb4d6@seim-partner.de>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
	<f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
	<333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>
	<151fd00d-94d1-1053-7202-bcd60735c1fc@seim-partner.de>
	<1181804313.1257349.1596870055548@mail.yahoo.com>
	<235f1f25-c9dd-cb75-3f49-b4a82bffb4d6@seim-partner.de>
Message-ID: <785C240A-515B-4AD7-B30E-BA00F2DF2E06@yahoo.com>

If you read my previous email, you will see that i noted that the string IS GFID and not the name of the file :)


You can find the name  following  the procedure at: https://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/


Of course,  that will be slow  for all entries in .glusterfs and you will need  to create a script to match all gfids to brick path.


I guess the fastest way to find the deleted files (As far as  I understood they were  deleted on the brick directly and entries  in .glusterfs were left) is  to create a script that:

0.Create a ramfs for the files:
findmnt /mnt || mount -t ramfs -o size=128MB - /mnt

1. Get  all  inodes
ionice -c 2 -n 7 nice -n 15 find /full/path/to/brick -type f -exec ls -i {} \; >/mnt/data

2. Get only the inodes:
nice  -n 15 awk '{print $1}'  /mnt/data >  /mnt/inode_only
3. Now the fun starts now-> find  inodes that are  not duplicate:

nice -n 15 uniq -u /mnt/inode_only >  /mnt/gfid-only

4. Once  you have the inodes, you can verify that they do exists only in .gluster dir
for  i in $(cat /mnt/gfid-only); do ionice  -c 2 -n 7 nice  -n 15 find /path/to/.glusterfs -inum $i ; echo;echo; done

5. If it's OK  ->  delete

for i in $(cat  /mnt/gfid-only); do ionice -c 2 -n 7 nice -n 15 find /path/to/brick  -inum $i -delete ; done


Last ,  repeat  on all bricks
Good  luck!


P.S.:  Consider  creating a gluster snapshot before that - just in case...  Better safe than sorry.

P.S: If you think that you got enough resources,  you can remove ionice/nice  stuff . They are just to guarantee you won't eat too many resources.


Best Regards,
Strahil  Nikolov




?? 8 ?????? 2020 ?. 18:02:10 GMT+03:00, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>So b53c8e46-068b-4286-94a6-7cf54f711983 is not a gfid? What else is it?
>
>Mathias
>
>On 08.08.20 09:00, Strahil Nikolov wrote:
>> In glusterfs the long string is called "gfid" and does not represent
>the name.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>>
>>
>>
>>
>>
>> ? ?????, 7 ?????? 2020 ?., 21:40:11 ???????+3, Mathias Waack
><mathias.waack at seim-partner.de> ??????:
>>
>>
>>
>>
>>
>> Hi Strahil,
>>
>> but I cannot find these files in the heal info:
>>
>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>> ...
>> 7443397? 132463 -rw-------?? 1 999????? docker?? 1073741824 Aug? 3
>10:35
>> /zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983
>>
>> Now looking for this file in the heal infos:
>>
>> gluster volume heal gvol info | grep
>b53c8e46-068b-4286-94a6-7cf54f711983
>>
>> shows nothing.
>>
>> So I do not know, what I have to heal...
>>
>> Mathias
>>
>> On 07.08.20 14:32, Strahil Nikolov wrote:
>>> Have you tried to gluster heal and check if the files are back into
>their place?
>>>
>>> I always thought that those hard links are used? by the healing
>mechanism? and if that is true - gluster should restore the files to
>their original location and then wiping the correct files from FUSE
>will be easy.
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> ?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack
><mathias.waack at seim-partner.de> ??????:
>>>> Hi all,
>>>>
>>>> maybe I should add some more information:
>>>>
>>>> The container which filled up the space was running on node x,
>which
>>>> still shows a nearly filled fs:
>>>>
>>>> 192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>>
>>>> nearly the same situation on the underlying brick partition on node
>x:
>>>>
>>>> zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>>>>
>>>> On node y the network card crashed, glusterfs shows the same
>values:
>>>>
>>>> 192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>>
>>>> but different values on the brick:
>>>>
>>>> zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>>>>
>>>> I think this happened because glusterfs still has hardlinks to the
>>>> deleted files on node x? So I can find these files with:
>>>>
>>>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>>>
>>>> But now I am lost. How can I verify these files really belongs to
>the
>>>> right container? Or can I just delete this files because there is
>no
>>>> way
>>>> to access it? Or offers glusterfs a way to solve this situation?
>>>>
>>>> Mathias
>>>>
>>>> On 05.08.20 15:48, Mathias Waack wrote:
>>>>> Hi all,
>>>>>
>>>>> we are running a gluster setup with two nodes:
>>>>>
>>>>> Status of volume: gvol
>>>>> Gluster process???????????????????????????? TCP Port? RDMA Port
>>>>> Online? Pid
>>>>>
>>>>
>------------------------------------------------------------------------------
>>>>
>>>>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>>>>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>>>>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
>>>>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>>>>>
>>>>> Task Status of Volume gvol
>>>>>
>>>>
>------------------------------------------------------------------------------
>>>>
>>>>> There are no active volume tasks
>>>>>
>>>>> The glusterfs hosts a bunch of containers with its data volumes.
>The
>>>>> underlying fs is zfs. Few days ago one of the containers created a
>>>> lot
>>>>> of files in one of its data volumes, and at the end it completely
>>>>> filled up the space of the glusterfs volume. But this happened
>only
>>>> on
>>>>> one host, on the other host there was still enough space. We
>finally
>>>>> were able to identify this container and found out, the sizes of
>the
>>>>> data on /zbrick were different on both hosts for this container.
>Now
>>>>> we made the big mistake to delete these files on both hosts in the
>>>>> /zbrick volume, not on the mounted glusterfs volume.
>>>>>
>>>>> Later we found the reason for this behavior: the network driver on
>>>> the
>>>>> second node partially crashed (which means we ware able to login
>on
>>>>> the node, so we assumed the network was running, but the card was
>>>>> already dropping packets at this time) at the same time, as the
>>>> failed
>>>>> container started to fill up the gluster volume. After rebooting
>the
>>>>> second node? the gluster became available again.
>>>>>
>>>>> Now the glusterfs volume is running again- but it is still
>(nearly)
>>>>> full: the files created by the container are not visible, but they
>>>>> still count into amount of free space. How can we fix this?
>>>>>
>>>>> In addition there are some files which are no longer accessible
>since
>>>>> this accident:
>>>>>
>>>>> tail access.log.old
>>>>> tail: cannot open 'access.log.old' for reading: Input/output error
>>>>>
>>>>> Looks like affected by this error are files which have been
>changed
>>>>> during the accident. Is there a way to fix this too?
>>>>>
>>>>> Thanks
>>>>>  ? ??? Mathias
>>>>>
>>>>>
>>>>> ________
>>>>>
>>>>>
>>>>>
>>>>> Community Meeting Calendar:
>>>>>
>>>>> Schedule -
>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>> Bridge: https://bluejeans.com/441850968
>>>>>
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>________
>
>
>
>Community Meeting Calendar:
>
>Schedule -
>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users

From mathias.waack at seim-partner.de  Sat Aug  8 17:05:29 2020
From: mathias.waack at seim-partner.de (Mathias Waack)
Date: Sat, 8 Aug 2020 19:05:29 +0200
Subject: [Gluster-users] Repair after accident
In-Reply-To: <785C240A-515B-4AD7-B30E-BA00F2DF2E06@yahoo.com>
References: <5822bb92-432e-e08e-d230-7adbf57127ce@seim-partner.de>
	<f054c67f-b831-4c11-a4e3-47abb7401e6b@seim-partner.de>
	<333712BC-D10B-4759-AED1-7793F1C17AC6@yahoo.com>
	<151fd00d-94d1-1053-7202-bcd60735c1fc@seim-partner.de>
	<1181804313.1257349.1596870055548@mail.yahoo.com>
	<235f1f25-c9dd-cb75-3f49-b4a82bffb4d6@seim-partner.de>
	<785C240A-515B-4AD7-B30E-BA00F2DF2E06@yahoo.com>
Message-ID: <bc6e0c8b-8823-fb9d-b407-8309c240e7c3@seim-partner.de>

Oh I see, I got you wrong. Now I am going to start to understand the 
whole thing. Thank you for the comprehensive explanation. For good luck 
it is weekend, so I can start digging into this...

Thanks
 ??? Mathias

On 08.08.20 18:05, Strahil Nikolov wrote:
> If you read my previous email, you will see that i noted that the string IS GFID and not the name of the file :)
>
>
> You can find the name  following  the procedure at: https://docs.gluster.org/en/latest/Troubleshooting/gfid-to-path/
>
>
> Of course,  that will be slow  for all entries in .glusterfs and you will need  to create a script to match all gfids to brick path.
>
>
> I guess the fastest way to find the deleted files (As far as  I understood they were  deleted on the brick directly and entries  in .glusterfs were left) is  to create a script that:
>
> 0.Create a ramfs for the files:
> findmnt /mnt || mount -t ramfs -o size=128MB - /mnt
>
> 1. Get  all  inodes
> ionice -c 2 -n 7 nice -n 15 find /full/path/to/brick -type f -exec ls -i {} \; >/mnt/data
>
> 2. Get only the inodes:
> nice  -n 15 awk '{print $1}'  /mnt/data >  /mnt/inode_only
> 3. Now the fun starts now-> find  inodes that are  not duplicate:
>
> nice -n 15 uniq -u /mnt/inode_only >  /mnt/gfid-only
>
> 4. Once  you have the inodes, you can verify that they do exists only in .gluster dir
> for  i in $(cat /mnt/gfid-only); do ionice  -c 2 -n 7 nice  -n 15 find /path/to/.glusterfs -inum $i ; echo;echo; done
>
> 5. If it's OK  ->  delete
>
> for i in $(cat  /mnt/gfid-only); do ionice -c 2 -n 7 nice -n 15 find /path/to/brick  -inum $i -delete ; done
>
>
> Last ,  repeat  on all bricks
> Good  luck!
>
>
> P.S.:  Consider  creating a gluster snapshot before that - just in case...  Better safe than sorry.
>
> P.S: If you think that you got enough resources,  you can remove ionice/nice  stuff . They are just to guarantee you won't eat too many resources.
>
>
> Best Regards,
> Strahil  Nikolov
>
>
>
>
> ?? 8 ?????? 2020 ?. 18:02:10 GMT+03:00, Mathias Waack <mathias.waack at seim-partner.de> ??????:
>> So b53c8e46-068b-4286-94a6-7cf54f711983 is not a gfid? What else is it?
>>
>> Mathias
>>
>> On 08.08.20 09:00, Strahil Nikolov wrote:
>>> In glusterfs the long string is called "gfid" and does not represent
>> the name.
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>>
>>>
>>>
>>>
>>>
>>> ? ?????, 7 ?????? 2020 ?., 21:40:11 ???????+3, Mathias Waack
>> <mathias.waack at seim-partner.de> ??????:
>>>
>>>
>>>
>>>
>>> Hi Strahil,
>>>
>>> but I cannot find these files in the heal info:
>>>
>>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>> ...
>>> 7443397? 132463 -rw-------?? 1 999????? docker?? 1073741824 Aug? 3
>> 10:35
>>> /zbrick/.glusterfs/b5/3c/b53c8e46-068b-4286-94a6-7cf54f711983
>>>
>>> Now looking for this file in the heal infos:
>>>
>>> gluster volume heal gvol info | grep
>> b53c8e46-068b-4286-94a6-7cf54f711983
>>> shows nothing.
>>>
>>> So I do not know, what I have to heal...
>>>
>>> Mathias
>>>
>>> On 07.08.20 14:32, Strahil Nikolov wrote:
>>>> Have you tried to gluster heal and check if the files are back into
>> their place?
>>>> I always thought that those hard links are used? by the healing
>> mechanism? and if that is true - gluster should restore the files to
>> their original location and then wiping the correct files from FUSE
>> will be easy.
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>>> ?? 7 ?????? 2020 ?. 10:24:38 GMT+03:00, Mathias Waack
>> <mathias.waack at seim-partner.de> ??????:
>>>>> Hi all,
>>>>>
>>>>> maybe I should add some more information:
>>>>>
>>>>> The container which filled up the space was running on node x,
>> which
>>>>> still shows a nearly filled fs:
>>>>>
>>>>> 192.168.1.x:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>>>
>>>>> nearly the same situation on the underlying brick partition on node
>> x:
>>>>> zdata/brick???? 2.6T? 2.4T? 176G? 94% /zbrick
>>>>>
>>>>> On node y the network card crashed, glusterfs shows the same
>> values:
>>>>> 192.168.1.y:/gvol? 2.6T? 2.5T? 149G? 95% /gluster
>>>>>
>>>>> but different values on the brick:
>>>>>
>>>>> zdata/brick???? 2.9T? 1.6T? 1.4T? 54% /zbrick
>>>>>
>>>>> I think this happened because glusterfs still has hardlinks to the
>>>>> deleted files on node x? So I can find these files with:
>>>>>
>>>>> find /zbrick/.glusterfs -links 1 -ls | grep -v ' -> '
>>>>>
>>>>> But now I am lost. How can I verify these files really belongs to
>> the
>>>>> right container? Or can I just delete this files because there is
>> no
>>>>> way
>>>>> to access it? Or offers glusterfs a way to solve this situation?
>>>>>
>>>>> Mathias
>>>>>
>>>>> On 05.08.20 15:48, Mathias Waack wrote:
>>>>>> Hi all,
>>>>>>
>>>>>> we are running a gluster setup with two nodes:
>>>>>>
>>>>>> Status of volume: gvol
>>>>>> Gluster process???????????????????????????? TCP Port? RDMA Port
>>>>>> Online? Pid
>>>>>>
>> ------------------------------------------------------------------------------
>>>>>> Brick 192.168.1.x:/zbrick????????????????? 49152???? 0 Y 13350
>>>>>> Brick 192.168.1.y:/zbrick????????????????? 49152???? 0 Y 5965
>>>>>> Self-heal Daemon on localhost?????????????? N/A?????? N/A Y 14188
>>>>>> Self-heal Daemon on 192.168.1.93??????????? N/A?????? N/A Y 6003
>>>>>>
>>>>>> Task Status of Volume gvol
>>>>>>
>> ------------------------------------------------------------------------------
>>>>>> There are no active volume tasks
>>>>>>
>>>>>> The glusterfs hosts a bunch of containers with its data volumes.
>> The
>>>>>> underlying fs is zfs. Few days ago one of the containers created a
>>>>> lot
>>>>>> of files in one of its data volumes, and at the end it completely
>>>>>> filled up the space of the glusterfs volume. But this happened
>> only
>>>>> on
>>>>>> one host, on the other host there was still enough space. We
>> finally
>>>>>> were able to identify this container and found out, the sizes of
>> the
>>>>>> data on /zbrick were different on both hosts for this container.
>> Now
>>>>>> we made the big mistake to delete these files on both hosts in the
>>>>>> /zbrick volume, not on the mounted glusterfs volume.
>>>>>>
>>>>>> Later we found the reason for this behavior: the network driver on
>>>>> the
>>>>>> second node partially crashed (which means we ware able to login
>> on
>>>>>> the node, so we assumed the network was running, but the card was
>>>>>> already dropping packets at this time) at the same time, as the
>>>>> failed
>>>>>> container started to fill up the gluster volume. After rebooting
>> the
>>>>>> second node? the gluster became available again.
>>>>>>
>>>>>> Now the glusterfs volume is running again- but it is still
>> (nearly)
>>>>>> full: the files created by the container are not visible, but they
>>>>>> still count into amount of free space. How can we fix this?
>>>>>>
>>>>>> In addition there are some files which are no longer accessible
>> since
>>>>>> this accident:
>>>>>>
>>>>>> tail access.log.old
>>>>>> tail: cannot open 'access.log.old' for reading: Input/output error
>>>>>>
>>>>>> Looks like affected by this error are files which have been
>> changed
>>>>>> during the accident. Is there a way to fix this too?
>>>>>>
>>>>>> Thanks
>>>>>>   ? ??? Mathias
>>>>>>
>>>>>>
>>>>>> ________
>>>>>>
>>>>>>
>>>>>>
>>>>>> Community Meeting Calendar:
>>>>>>
>>>>>> Schedule -
>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>> ________
>>>>>
>>>>>
>>>>>
>>>>> Community Meeting Calendar:
>>>>>
>>>>> Schedule -
>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>> Bridge: https://bluejeans.com/441850968
>>>>>
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users

From gilberto.nunes32 at gmail.com  Sun Aug  9 15:16:57 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Sun, 9 Aug 2020 12:16:57 -0300
Subject: [Gluster-users] Monitoring tools for GlusterFS
In-Reply-To: <58f109a7-6d62-4814-425d-7728ea4f8338@fischer-ka.de>
References: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>
	<58f109a7-6d62-4814-425d-7728ea4f8338@fischer-ka.de>
Message-ID: <CAOKSTBvsbRWnr2dtG+KQaj-v4DvDC4yxZ6h+Fg0_HmDFPmPzoQ@mail.gmail.com>

I try it but get the error bellow

./gstatus.py -v

Traceback (most recent call last):
  File "./gstatus.py", line 245, in <module>
    main()
  File "./gstatus.py", line 133, in main
    cluster.initialise()
  File "/root/gstatus/gstatus/libgluster/cluster.py", line 97, in initialise
    self.define_nodes()
  File "/root/gstatus/gstatus/libgluster/cluster.py", line 170, in
define_nodes
    local_ip_list = get_ipv4_addr()  # Grab all IP's
  File "/root/gstatus/gstatus/libutils/network.py", line 130, in
get_ipv4_addr
    namestr = names.tobytes()
AttributeError: 'array.array' object has no attribute 'tobytes'


---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em s?b., 8 de ago. de 2020 ?s 18:28, Ingo Fischer <ingo at fischer-ka.de>
escreveu:

> Hey,
>
> I use gstatus https://github.com/gluster/gstatus
>
> Ingo
>
> Am 08.08.20 um 06:27 schrieb Gilberto Nunes:
> > Hi guys...  I miss some tools that could be used in order to monitor
> > healing for example, and others things like resources used...  What do
> > you recommend?
> > Tools that could be used in CLI but that shows a percentage as healing
> > is under way would be nice!
> >
> > Thanks.
> >
> > ---
> > Gilberto Nunes Ferreira
> >
> >
> >
> >
> > ________
> >
> >
> >
> > Community Meeting Calendar:
> >
> > Schedule -
> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> > Bridge: https://bluejeans.com/441850968
> >
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200809/5b7be543/attachment.html>

From hunter86_bg at yahoo.com  Sun Aug  9 15:27:22 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sun, 09 Aug 2020 18:27:22 +0300
Subject: [Gluster-users] Monitoring tools for GlusterFS
In-Reply-To: <CAOKSTBvsbRWnr2dtG+KQaj-v4DvDC4yxZ6h+Fg0_HmDFPmPzoQ@mail.gmail.com>
References: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>
	<58f109a7-6d62-4814-425d-7728ea4f8338@fischer-ka.de>
	<CAOKSTBvsbRWnr2dtG+KQaj-v4DvDC4yxZ6h+Fg0_HmDFPmPzoQ@mail.gmail.com>
Message-ID: <DE633E94-8DAC-47CD-B1C5-F70ABF02261A@yahoo.com>

You got a problem with your gstatus.
How did you deploy it ? 
What is your gluster version ?Mine is working quite fine with 7.7

Best Regards,
Strahil Nikolov

?? 9 ?????? 2020 ?. 18:16:57 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>I try it but get the error bellow
>
>./gstatus.py -v
>
>Traceback (most recent call last):
>  File "./gstatus.py", line 245, in <module>
>    main()
>  File "./gstatus.py", line 133, in main
>    cluster.initialise()
>File "/root/gstatus/gstatus/libgluster/cluster.py", line 97, in
>initialise
>    self.define_nodes()
>  File "/root/gstatus/gstatus/libgluster/cluster.py", line 170, in
>define_nodes
>    local_ip_list = get_ipv4_addr()  # Grab all IP's
>  File "/root/gstatus/gstatus/libutils/network.py", line 130, in
>get_ipv4_addr
>    namestr = names.tobytes()
>AttributeError: 'array.array' object has no attribute 'tobytes'
>
>
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em s?b., 8 de ago. de 2020 ?s 18:28, Ingo Fischer <ingo at fischer-ka.de>
>escreveu:
>
>> Hey,
>>
>> I use gstatus https://github.com/gluster/gstatus
>>
>> Ingo
>>
>> Am 08.08.20 um 06:27 schrieb Gilberto Nunes:
>> > Hi guys...  I miss some tools that could be used in order to
>monitor
>> > healing for example, and others things like resources used...  What
>do
>> > you recommend?
>> > Tools that could be used in CLI but that shows a percentage as
>healing
>> > is under way would be nice!
>> >
>> > Thanks.
>> >
>> > ---
>> > Gilberto Nunes Ferreira
>> >
>> >
>> >
>> >
>> > ________
>> >
>> >
>> >
>> > Community Meeting Calendar:
>> >
>> > Schedule -
>> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> > Bridge: https://bluejeans.com/441850968
>> >
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>> >
>>

From gilberto.nunes32 at gmail.com  Sun Aug  9 17:12:35 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Sun, 9 Aug 2020 14:12:35 -0300
Subject: [Gluster-users] Monitoring tools for GlusterFS
In-Reply-To: <DE633E94-8DAC-47CD-B1C5-F70ABF02261A@yahoo.com>
References: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>
	<58f109a7-6d62-4814-425d-7728ea4f8338@fischer-ka.de>
	<CAOKSTBvsbRWnr2dtG+KQaj-v4DvDC4yxZ6h+Fg0_HmDFPmPzoQ@mail.gmail.com>
	<DE633E94-8DAC-47CD-B1C5-F70ABF02261A@yahoo.com>
Message-ID: <CAOKSTBv=AgmLZQxtnrNS9ppdDRnnXkGQku=snatMDvcrOh3aXQ@mail.gmail.com>

How did you deploy it ? - git clone, ./gstatus.py, and python gstatus.py
install then gstatus

What is your gluster version ? Latest stable to Debian Buster (v8)



---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em dom., 9 de ago. de 2020 ?s 12:28, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> You got a problem with your gstatus.
> How did you deploy it ?
> What is your gluster version ?Mine is working quite fine with 7.7
>
> Best Regards,
> Strahil Nikolov
>
> ?? 9 ?????? 2020 ?. 18:16:57 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >I try it but get the error bellow
> >
> >./gstatus.py -v
> >
> >Traceback (most recent call last):
> >  File "./gstatus.py", line 245, in <module>
> >    main()
> >  File "./gstatus.py", line 133, in main
> >    cluster.initialise()
> >File "/root/gstatus/gstatus/libgluster/cluster.py", line 97, in
> >initialise
> >    self.define_nodes()
> >  File "/root/gstatus/gstatus/libgluster/cluster.py", line 170, in
> >define_nodes
> >    local_ip_list = get_ipv4_addr()  # Grab all IP's
> >  File "/root/gstatus/gstatus/libutils/network.py", line 130, in
> >get_ipv4_addr
> >    namestr = names.tobytes()
> >AttributeError: 'array.array' object has no attribute 'tobytes'
> >
> >
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em s?b., 8 de ago. de 2020 ?s 18:28, Ingo Fischer <ingo at fischer-ka.de>
> >escreveu:
> >
> >> Hey,
> >>
> >> I use gstatus https://github.com/gluster/gstatus
> >>
> >> Ingo
> >>
> >> Am 08.08.20 um 06:27 schrieb Gilberto Nunes:
> >> > Hi guys...  I miss some tools that could be used in order to
> >monitor
> >> > healing for example, and others things like resources used...  What
> >do
> >> > you recommend?
> >> > Tools that could be used in CLI but that shows a percentage as
> >healing
> >> > is under way would be nice!
> >> >
> >> > Thanks.
> >> >
> >> > ---
> >> > Gilberto Nunes Ferreira
> >> >
> >> >
> >> >
> >> >
> >> > ________
> >> >
> >> >
> >> >
> >> > Community Meeting Calendar:
> >> >
> >> > Schedule -
> >> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >> > Bridge: https://bluejeans.com/441850968
> >> >
> >> > Gluster-users mailing list
> >> > Gluster-users at gluster.org
> >> > https://lists.gluster.org/mailman/listinfo/gluster-users
> >> >
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200809/6d030b81/attachment.html>

From hunter86_bg at yahoo.com  Sun Aug  9 18:37:35 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Sun, 9 Aug 2020 18:37:35 +0000 (UTC)
Subject: [Gluster-users] Monitoring tools for GlusterFS
In-Reply-To: <CAOKSTBv=AgmLZQxtnrNS9ppdDRnnXkGQku=snatMDvcrOh3aXQ@mail.gmail.com>
References: <CAOKSTBvt8A52=5VAVxd8nssDCu4bitzqurYdszSSy9Q0jNZO3Q@mail.gmail.com>
	<58f109a7-6d62-4814-425d-7728ea4f8338@fischer-ka.de>
	<CAOKSTBvsbRWnr2dtG+KQaj-v4DvDC4yxZ6h+Fg0_HmDFPmPzoQ@mail.gmail.com>
	<DE633E94-8DAC-47CD-B1C5-F70ABF02261A@yahoo.com>
	<CAOKSTBv=AgmLZQxtnrNS9ppdDRnnXkGQku=snatMDvcrOh3aXQ@mail.gmail.com>
Message-ID: <292755878.1582157.1596998255554@mail.yahoo.com>

Hi Gilberto,

I just tested latest master branch on CentOS8 -> total failure (both python2 and python3).

I have opened a github issue at?https://github.com/gluster/gstatus/issues/30?

I guess, you can add details for OS, packages and gluster version (maybe a traceback also).


The more interesing part is that the old zip I got is working on CentOS8 (with python2.7) without issues:


[root at glustera gstatus-master]# python2 gstatus.py ?
?
????Product: Community ?????????Capacity: ?22.00 GiB(raw bricks)
?????Status: HEALTHY ?????????????????????520.00 MiB(raw used)
??Glusterfs: 8.0 ???????????????????????????6.00 GiB(usable from volumes)
?OverCommit: No ???????????????Snapshots: ??0


If you wish, I can try to upload it somewhere for you.

Best Regards,
Strahil Nikolov






? ??????, 9 ?????? 2020 ?., 20:13:13 ???????+3, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????: 





How did you deploy it ? - git clone, ./gstatus.py, and python gstatus.py install then gstatus

What is your gluster version ? Latest?stable to Debian Buster (v8)



---
Gilberto Nunes Ferreira

(47) 3025-5907

(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36




Em dom., 9 de ago. de 2020 ?s 12:28, Strahil Nikolov <hunter86_bg at yahoo.com> escreveu:
> You got a problem with your gstatus.
> How did you deploy it ? 
> What is your gluster version ?Mine is working quite fine with 7.7
> 
> Best Regards,
> Strahil Nikolov
> 
> ?? 9 ?????? 2020 ?. 18:16:57 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>>I try it but get the error bellow
>>
>>./gstatus.py -v
>>
>>Traceback (most recent call last):
>>? File "./gstatus.py", line 245, in <module>
>>? ? main()
>>? File "./gstatus.py", line 133, in main
>>? ? cluster.initialise()
>>File "/root/gstatus/gstatus/libgluster/cluster.py", line 97, in
>>initialise
>>? ? self.define_nodes()
>>? File "/root/gstatus/gstatus/libgluster/cluster.py", line 170, in
>>define_nodes
>>? ? local_ip_list = get_ipv4_addr()? # Grab all IP's
>>? File "/root/gstatus/gstatus/libutils/network.py", line 130, in
>>get_ipv4_addr
>>? ? namestr = names.tobytes()
>>AttributeError: 'array.array' object has no attribute 'tobytes'
>>
>>
>>---
>>Gilberto Nunes Ferreira
>>
>>(47) 3025-5907
>>(47) 99676-7530 - Whatsapp / Telegram
>>
>>Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>>Em s?b., 8 de ago. de 2020 ?s 18:28, Ingo Fischer <ingo at fischer-ka.de>
>>escreveu:
>>
>>> Hey,
>>>
>>> I use gstatus https://github.com/gluster/gstatus
>>>
>>> Ingo
>>>
>>> Am 08.08.20 um 06:27 schrieb Gilberto Nunes:
>>> > Hi guys...? I miss some tools that could be used in order to
>>monitor
>>> > healing for example, and others things like resources used...? What
>>do
>>> > you recommend?
>>> > Tools that could be used in CLI but that shows a percentage as
>>healing
>>> > is under way would be nice!
>>> >
>>> > Thanks.
>>> >
>>> > ---
>>> > Gilberto Nunes Ferreira
>>> >
>>> >
>>> >
>>> >
>>> > ________
>>> >
>>> >
>>> >
>>> > Community Meeting Calendar:
>>> >
>>> > Schedule -
>>> > Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> > Bridge: https://bluejeans.com/441850968
>>> >
>>> > Gluster-users mailing list
>>> > Gluster-users at gluster.org
>>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >
>>>
> 

From rkothiya at redhat.com  Mon Aug 10 19:17:40 2020
From: rkothiya at redhat.com (Rinku Kothiya)
Date: Tue, 11 Aug 2020 00:47:40 +0530
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 6.10
Message-ID: <CAMX-mG3wwEESyjJ-=nFGTpmWkT7qgq_RXgUW4VtC7n1d1giyEQ@mail.gmail.com>

Hi,

The Gluster community is pleased to announce the release of Gluster6.10
(packages available at [1]).
Release notes for the release can be found at [2].

This is the last minor release of 6. Users are highly encouraged to upgrade
to newer releases of GlusterFS.

Please Note: Some of the packages are unavailable and we are working on it.
We will release them soon.

Thanks,
Gluster community

References:

[1] Packages for 6.10:
https://download.gluster.org/pub/gluster/glusterfs/6/6.10/

[2] Release notes for 6.10:
https://docs.gluster.org/en/latest/release-notes/6.10/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200811/396599ff/attachment.html>