From peljasz at yahoo.co.uk  Wed Jul  1 14:46:11 2020
From: peljasz at yahoo.co.uk (lejeczek)
Date: Wed, 1 Jul 2020 15:46:11 +0100
Subject: [Gluster-users] volume process does not start - glusterfs is
 happy with it?
In-Reply-To: <CAFx2kQ11KuoRewCmAQrWmmjxf1Wee5Sgbi3y1LdmnR1tO4tyog@mail.gmail.com>
References: <a71bcf34-0660-1ab5-76e5-8010c85636c6.ref@yahoo.co.uk>
	<a71bcf34-0660-1ab5-76e5-8010c85636c6@yahoo.co.uk>
	<CAFx2kQ11KuoRewCmAQrWmmjxf1Wee5Sgbi3y1LdmnR1tO4tyog@mail.gmail.com>
Message-ID: <13c2171b-ea34-1e8b-c060-7ebed1e016ee@yahoo.co.uk>



On 30/06/2020 11:31, Barak Sason Rofman wrote:
> Greetings,
>
> I'm not sure if that's directly related to your problem,
> but on a general level, AFAIK, replica-2 vols are not
> recommended due to split brain possibility:
> https://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
>
> It's recommended to either use replica-3 or arbiter Arbiter.
>
> Regards,
>
> On Tue, Jun 30, 2020 at 1:14 PM lejeczek
> <peljasz at yahoo.co.uk <mailto:peljasz at yahoo.co.uk>> wrote:
>
>     Hi everybody.
>
>     I have two peers in the cluster and a 2-replica volume
>     which seems okey if it was not for one weird bit -
>     when a peer reboots then on that peer after a reboot I
>     see:
>
>     $ gluster volume status USERs
>     Status of volume: USERs
>     Gluster process???????????????????????????? TCP Port?
>     RDMA Port? Online? Pid
>     ------------------------------------------------------------------------------
>     Brick swir.direct:/00.STORAGE/2/0-GLUSTER-U
>     SERs??????????????????????????????????????? N/A??????
>     N/A??????? N?????? N/A?
>     Brick dzien.direct:/00.STORAGE/2/0-GLUSTER-
>     USERs?????????????????????????????????????? 49152????
>     0????????? Y?????? 57338
>     Self-heal Daemon on localhost?????????????? N/A??????
>     N/A??????? Y?????? 4302
>     Self-heal Daemon on dzien.direct??????????? N/A??????
>     N/A??????? Y?????? 57359
>     ?
>     Task Status of Volume USERs
>     ------------------------------------------------------------------------------
>     There are no active volume tasks
>
>     I do not suppose it's expected.
>     On such rebooted node I see:
>     $ systemctl status -l glusterd
>     ? glusterd.service - GlusterFS, a clustered
>     file-system server
>     ?? Loaded: loaded
>     (/usr/lib/systemd/system/glusterd.service; enabled;
>     vendor preset: enabled)
>     ? Drop-In: /etc/systemd/system/glusterd.service.d
>     ?????????? ??override.conf
>     ?? Active: active (running) since Mon 2020-06-29
>     21:37:36 BST; 13h ago
>     ???? Docs: man:glusterd(8)
>     ? Process: 4071 ExecStart=/usr/sbin/glusterd -p
>     /var/run/glusterd.pid --log-level $LOG_LEVEL
>     $GLUSTERD_OPTIONS (code=exited, status>
>     ?Main PID: 4086 (glusterd)
>     ??? Tasks: 20 (limit: 101792)
>     ?? Memory: 28.9M
>     ?? CGroup: /system.slice/glusterd.service
>     ?????????? ??4086 /usr/sbin/glusterd -p
>     /var/run/glusterd.pid --log-level INFO
>     ?????????? ??4302 /usr/sbin/glusterfs -s localhost
>     --volfile-id shd/USERs -p
>     /var/run/gluster/shd/USERs/USERs-shd.pid -l /var/log/g>
>
>     Jun 29 21:37:36 swir.private.pawel systemd[1]:
>     Starting GlusterFS, a clustered file-system server...
>     Jun 29 21:37:36 swir.private.pawel systemd[1]: Started
>     GlusterFS, a clustered file-system server.
>
>     And I do not see any other apparent problems nor errors.
>     On that node I manually:
>     $ systemctl restart glusterd.service
>     and...
>
>     $ gluster volume status USERs
>     Status of volume: USERs
>     Gluster process???????????????????????????? TCP Port?
>     RDMA Port? Online? Pid
>     ------------------------------------------------------------------------------
>     Brick swir.direct:/00.STORAGE/2/0-GLUSTER-U
>     SERs??????????????????????????????????????? 49152????
>     0????????? Y?????? 103225
>     Brick dzien.direct:/00.STORAGE/2/0-GLUSTER-
>     USERs?????????????????????????????????????? 49152????
>     0????????? Y?????? 57338
>     Self-heal Daemon on localhost?????????????? N/A??????
>     N/A??????? Y?????? 103270
>     Self-heal Daemon on dzien.direct??????????? N/A??????
>     N/A??????? Y?????? 57359
>
>     Is not a puzzle??? I'm on glusterfs-7.6-1.el8.x86_64
>     I hope somebody can share some thoughts.
>     many thanks, L.
>
That cannot be it!? If the root cause of this problem is
2-replica volume then it would be a massive cock-up! Then
2-volume replica should be banned and forbidden.

I hope some can suggest a way to troubleshoot it.

ps. we all, I presume all, know problems of 2-replica volumes.

many thanks, L.


>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
>
>     Gluster-users mailing list
>     Gluster-users at gluster.org
>     <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> -- 
> *Barak Sason Rofman*
>
> Gluster Storage?Development
>
> Red Hat?Israel <https://www.redhat.com/>
>
> 34 Jerusalem rd. Ra'anana, 43501
>
> bsasonro at redhat.com <mailto:adi at redhat.com>?
> ??T:?_+972-9-7692304_
> M:?_+972-52-4326355_
>
> @RedHat <https://twitter.com/redhat>???Red Hat
> <https://www.linkedin.com/company/red-hat>??Red Hat
> <https://www.facebook.com/redhat.il/>
> <https://red.ht/sig>
>


From felix.koelzow at gmx.de  Wed Jul  1 17:57:22 2020
From: felix.koelzow at gmx.de (=?UTF-8?Q?Felix_K=c3=b6lzow?=)
Date: Wed, 1 Jul 2020 19:57:22 +0200
Subject: [Gluster-users] volume process does not start - glusterfs is
 happy with it?
In-Reply-To: <13c2171b-ea34-1e8b-c060-7ebed1e016ee@yahoo.co.uk>
References: <a71bcf34-0660-1ab5-76e5-8010c85636c6.ref@yahoo.co.uk>
	<a71bcf34-0660-1ab5-76e5-8010c85636c6@yahoo.co.uk>
	<CAFx2kQ11KuoRewCmAQrWmmjxf1Wee5Sgbi3y1LdmnR1tO4tyog@mail.gmail.com>
	<13c2171b-ea34-1e8b-c060-7ebed1e016ee@yahoo.co.uk>
Message-ID: <05676387-1bf2-b4c9-bf60-0106c94be047@gmx.de>

Hey,


what about the device mapper? Everything was mount properly during reboot?

It happens to me if the lvm device mapper got a timeout during the reboot

process while mounting the brick itself.


Regards,

Felix

On 01/07/2020 16:46, lejeczek wrote:
>
> On 30/06/2020 11:31, Barak Sason Rofman wrote:
>> Greetings,
>>
>> I'm not sure if that's directly related to your problem,
>> but on a general level, AFAIK, replica-2 vols are not
>> recommended due to split brain possibility:
>> https://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
>>
>> It's recommended to either use replica-3 or arbiter Arbiter.
>>
>> Regards,
>>
>> On Tue, Jun 30, 2020 at 1:14 PM lejeczek
>> <peljasz at yahoo.co.uk <mailto:peljasz at yahoo.co.uk>> wrote:
>>
>>      Hi everybody.
>>
>>      I have two peers in the cluster and a 2-replica volume
>>      which seems okey if it was not for one weird bit -
>>      when a peer reboots then on that peer after a reboot I
>>      see:
>>
>>      $ gluster volume status USERs
>>      Status of volume: USERs
>>      Gluster process???????????????????????????? TCP Port
>>      RDMA Port? Online? Pid
>>      ------------------------------------------------------------------------------
>>      Brick swir.direct:/00.STORAGE/2/0-GLUSTER-U
>>      SERs??????????????????????????????????????? N/A
>>      N/A??????? N?????? N/A
>>      Brick dzien.direct:/00.STORAGE/2/0-GLUSTER-
>>      USERs?????????????????????????????????????? 49152
>>      0????????? Y?????? 57338
>>      Self-heal Daemon on localhost?????????????? N/A
>>      N/A??????? Y?????? 4302
>>      Self-heal Daemon on dzien.direct??????????? N/A
>>      N/A??????? Y?????? 57359
>>
>>      Task Status of Volume USERs
>>      ------------------------------------------------------------------------------
>>      There are no active volume tasks
>>
>>      I do not suppose it's expected.
>>      On such rebooted node I see:
>>      $ systemctl status -l glusterd
>>      ? glusterd.service - GlusterFS, a clustered
>>      file-system server
>>      ?? Loaded: loaded
>>      (/usr/lib/systemd/system/glusterd.service; enabled;
>>      vendor preset: enabled)
>>      ? Drop-In: /etc/systemd/system/glusterd.service.d
>>      ?????????? ??override.conf
>>      ?? Active: active (running) since Mon 2020-06-29
>>      21:37:36 BST; 13h ago
>>      ???? Docs: man:glusterd(8)
>>      ? Process: 4071 ExecStart=/usr/sbin/glusterd -p
>>      /var/run/glusterd.pid --log-level $LOG_LEVEL
>>      $GLUSTERD_OPTIONS (code=exited, status>
>>      ?Main PID: 4086 (glusterd)
>>      ??? Tasks: 20 (limit: 101792)
>>      ?? Memory: 28.9M
>>      ?? CGroup: /system.slice/glusterd.service
>>      ?????????? ??4086 /usr/sbin/glusterd -p
>>      /var/run/glusterd.pid --log-level INFO
>>      ?????????? ??4302 /usr/sbin/glusterfs -s localhost
>>      --volfile-id shd/USERs -p
>>      /var/run/gluster/shd/USERs/USERs-shd.pid -l /var/log/g>
>>
>>      Jun 29 21:37:36 swir.private.pawel systemd[1]:
>>      Starting GlusterFS, a clustered file-system server...
>>      Jun 29 21:37:36 swir.private.pawel systemd[1]: Started
>>      GlusterFS, a clustered file-system server.
>>
>>      And I do not see any other apparent problems nor errors.
>>      On that node I manually:
>>      $ systemctl restart glusterd.service
>>      and...
>>
>>      $ gluster volume status USERs
>>      Status of volume: USERs
>>      Gluster process???????????????????????????? TCP Port
>>      RDMA Port? Online? Pid
>>      ------------------------------------------------------------------------------
>>      Brick swir.direct:/00.STORAGE/2/0-GLUSTER-U
>>      SERs??????????????????????????????????????? 49152
>>      0????????? Y?????? 103225
>>      Brick dzien.direct:/00.STORAGE/2/0-GLUSTER-
>>      USERs?????????????????????????????????????? 49152
>>      0????????? Y?????? 57338
>>      Self-heal Daemon on localhost?????????????? N/A
>>      N/A??????? Y?????? 103270
>>      Self-heal Daemon on dzien.direct??????????? N/A
>>      N/A??????? Y?????? 57359
>>
>>      Is not a puzzle??? I'm on glusterfs-7.6-1.el8.x86_64
>>      I hope somebody can share some thoughts.
>>      many thanks, L.
>>
> That cannot be it!? If the root cause of this problem is
> 2-replica volume then it would be a massive cock-up! Then
> 2-volume replica should be banned and forbidden.
>
> I hope some can suggest a way to troubleshoot it.
>
> ps. we all, I presume all, know problems of 2-replica volumes.
>
> many thanks, L.
>
>
>>      ________
>>
>>
>>
>>      Community Meeting Calendar:
>>
>>      Schedule -
>>      Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>      Bridge: https://bluejeans.com/441850968
>>
>>      Gluster-users mailing list
>>      Gluster-users at gluster.org
>>      <mailto:Gluster-users at gluster.org>
>>      https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage?Development
>>
>> Red Hat?Israel <https://www.redhat.com/>
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsasonro at redhat.com <mailto:adi at redhat.com>
>>  ??T:?_+972-9-7692304_
>> M:?_+972-52-4326355_
>>
>> @RedHat <https://twitter.com/redhat>???Red Hat
>> <https://www.linkedin.com/company/red-hat>??Red Hat
>> <https://www.facebook.com/redhat.il/>
>> <https://red.ht/sig>
>>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

From hunter86_bg at yahoo.com  Wed Jul  1 18:33:27 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Wed, 01 Jul 2020 21:33:27 +0300
Subject: [Gluster-users] volume process does not start - glusterfs is
	happy with it?
In-Reply-To: <05676387-1bf2-b4c9-bf60-0106c94be047@gmx.de>
References: <a71bcf34-0660-1ab5-76e5-8010c85636c6.ref@yahoo.co.uk>
	<a71bcf34-0660-1ab5-76e5-8010c85636c6@yahoo.co.uk>
	<CAFx2kQ11KuoRewCmAQrWmmjxf1Wee5Sgbi3y1LdmnR1tO4tyog@mail.gmail.com>
	<13c2171b-ea34-1e8b-c060-7ebed1e016ee@yahoo.co.uk>
	<05676387-1bf2-b4c9-bf60-0106c94be047@gmx.de>
Message-ID: <E37BEDFC-CED2-4361-AD33-46E9A6CCF1E5@yahoo.com>

Sometimes the brick  comes  up slower than glusterd service (which starts the brick processes).
The problem is that if you leave glusterd to depend on both  bricks and a brick fails (for example  FS problem)  then the other brick will not come up too.

After a  system crash,  the VDO service  replay log was  taking too much and the glusterd  failed (as bricks were not ready yet),  so I just created  an override like this one:

# /etc/systemd/system/glusterd.service.d/01-dependencies.conf
[Unit]
[root at ovirt1 ~]# cat /etc/systemd/system/glusterd.service.d/01-dependencies.conf
[Unit]
Description=GlusterFS, a clustered file-system server
After=network.target rpcbind.service gluster_bricks-engine.mount gluster_bricks-data.mount gluster_bricks-fast1.mount gluster_bricks-fast2.mount gluster_bricks-fast3.mount gluster_bricks-fast4.mount                                          
Before=network-online.target

I have created systemd mount units, due to VDO , but most probably the  local-fs.target will generate the  mount units for you from the fstab.

Best Regards,

Strahil Nikolov

?? 1 ??? 2020 ?. 20:57:22 GMT+03:00, "Felix K?lzow" <felix.koelzow at gmx.de> ??????:
>Hey,
>
>
>what about the device mapper? Everything was mount properly during
>reboot?
>
>It happens to me if the lvm device mapper got a timeout during the
>reboot
>
>process while mounting the brick itself.
>
>
>Regards,
>
>Felix
>
>On 01/07/2020 16:46, lejeczek wrote:
>>
>> On 30/06/2020 11:31, Barak Sason Rofman wrote:
>>> Greetings,
>>>
>>> I'm not sure if that's directly related to your problem,
>>> but on a general level, AFAIK, replica-2 vols are not
>>> recommended due to split brain possibility:
>>>
>https://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
>>>
>>> It's recommended to either use replica-3 or arbiter Arbiter.
>>>
>>> Regards,
>>>
>>> On Tue, Jun 30, 2020 at 1:14 PM lejeczek
>>> <peljasz at yahoo.co.uk <mailto:peljasz at yahoo.co.uk>> wrote:
>>>
>>>      Hi everybody.
>>>
>>>      I have two peers in the cluster and a 2-replica volume
>>>      which seems okey if it was not for one weird bit -
>>>      when a peer reboots then on that peer after a reboot I
>>>      see:
>>>
>>>      $ gluster volume status USERs
>>>      Status of volume: USERs
>>>      Gluster process???????????????????????????? TCP Port
>>>      RDMA Port? Online? Pid
>>>     
>------------------------------------------------------------------------------
>>>      Brick swir.direct:/00.STORAGE/2/0-GLUSTER-U
>>>      SERs??????????????????????????????????????? N/A
>>>      N/A??????? N?????? N/A
>>>      Brick dzien.direct:/00.STORAGE/2/0-GLUSTER-
>>>      USERs?????????????????????????????????????? 49152
>>>      0????????? Y?????? 57338
>>>      Self-heal Daemon on localhost?????????????? N/A
>>>      N/A??????? Y?????? 4302
>>>      Self-heal Daemon on dzien.direct??????????? N/A
>>>      N/A??????? Y?????? 57359
>>>
>>>      Task Status of Volume USERs
>>>     
>------------------------------------------------------------------------------
>>>      There are no active volume tasks
>>>
>>>      I do not suppose it's expected.
>>>      On such rebooted node I see:
>>>      $ systemctl status -l glusterd
>>>      ? glusterd.service - GlusterFS, a clustered
>>>      file-system server
>>>      ?? Loaded: loaded
>>>      (/usr/lib/systemd/system/glusterd.service; enabled;
>>>      vendor preset: enabled)
>>>      ? Drop-In: /etc/systemd/system/glusterd.service.d
>>>      ?????????? ??override.conf
>>>      ?? Active: active (running) since Mon 2020-06-29
>>>      21:37:36 BST; 13h ago
>>>      ???? Docs: man:glusterd(8)
>>>      ? Process: 4071 ExecStart=/usr/sbin/glusterd -p
>>>      /var/run/glusterd.pid --log-level $LOG_LEVEL
>>>      $GLUSTERD_OPTIONS (code=exited, status>
>>>      ?Main PID: 4086 (glusterd)
>>>      ??? Tasks: 20 (limit: 101792)
>>>      ?? Memory: 28.9M
>>>      ?? CGroup: /system.slice/glusterd.service
>>>      ?????????? ??4086 /usr/sbin/glusterd -p
>>>      /var/run/glusterd.pid --log-level INFO
>>>      ?????????? ??4302 /usr/sbin/glusterfs -s localhost
>>>      --volfile-id shd/USERs -p
>>>      /var/run/gluster/shd/USERs/USERs-shd.pid -l /var/log/g>
>>>
>>>      Jun 29 21:37:36 swir.private.pawel systemd[1]:
>>>      Starting GlusterFS, a clustered file-system server...
>>>      Jun 29 21:37:36 swir.private.pawel systemd[1]: Started
>>>      GlusterFS, a clustered file-system server.
>>>
>>>      And I do not see any other apparent problems nor errors.
>>>      On that node I manually:
>>>      $ systemctl restart glusterd.service
>>>      and...
>>>
>>>      $ gluster volume status USERs
>>>      Status of volume: USERs
>>>      Gluster process???????????????????????????? TCP Port
>>>      RDMA Port? Online? Pid
>>>     
>------------------------------------------------------------------------------
>>>      Brick swir.direct:/00.STORAGE/2/0-GLUSTER-U
>>>      SERs??????????????????????????????????????? 49152
>>>      0????????? Y?????? 103225
>>>      Brick dzien.direct:/00.STORAGE/2/0-GLUSTER-
>>>      USERs?????????????????????????????????????? 49152
>>>      0????????? Y?????? 57338
>>>      Self-heal Daemon on localhost?????????????? N/A
>>>      N/A??????? Y?????? 103270
>>>      Self-heal Daemon on dzien.direct??????????? N/A
>>>      N/A??????? Y?????? 57359
>>>
>>>      Is not a puzzle??? I'm on glusterfs-7.6-1.el8.x86_64
>>>      I hope somebody can share some thoughts.
>>>      many thanks, L.
>>>
>> That cannot be it!? If the root cause of this problem is
>> 2-replica volume then it would be a massive cock-up! Then
>> 2-volume replica should be banned and forbidden.
>>
>> I hope some can suggest a way to troubleshoot it.
>>
>> ps. we all, I presume all, know problems of 2-replica volumes.
>>
>> many thanks, L.
>>
>>
>>>      ________
>>>
>>>
>>>
>>>      Community Meeting Calendar:
>>>
>>>      Schedule -
>>>      Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>      Bridge: https://bluejeans.com/441850968
>>>
>>>      Gluster-users mailing list
>>>      Gluster-users at gluster.org
>>>      <mailto:Gluster-users at gluster.org>
>>>      https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>> --
>>> *Barak Sason Rofman*
>>>
>>> Gluster Storage?Development
>>>
>>> Red Hat?Israel <https://www.redhat.com/>
>>>
>>> 34 Jerusalem rd. Ra'anana, 43501
>>>
>>> bsasonro at redhat.com <mailto:adi at redhat.com>
>>>  ??T:?_+972-9-7692304_
>>> M:?_+972-52-4326355_
>>>
>>> @RedHat <https://twitter.com/redhat>???Red Hat
>>> <https://www.linkedin.com/company/red-hat>??Red Hat
>>> <https://www.facebook.com/redhat.il/>
>>> <https://red.ht/sig>
>>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>________
>
>
>
>Community Meeting Calendar:
>
>Schedule -
>Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>Bridge: https://bluejeans.com/441850968
>
>Gluster-users mailing list
>Gluster-users at gluster.org
>https://lists.gluster.org/mailman/listinfo/gluster-users

From peljasz at yahoo.co.uk  Thu Jul  2 13:32:21 2020
From: peljasz at yahoo.co.uk (lejeczek)
Date: Thu, 2 Jul 2020 14:32:21 +0100
Subject: [Gluster-users] gluster cmd output - how to format
References: <87c430eb-56b6-57d2-ef2a-f974b69cc5fe.ref@yahoo.co.uk>
Message-ID: <87c430eb-56b6-57d2-ef2a-f974b69cc5fe@yahoo.co.uk>

hi guys

Would you know if it's possible to format gluster cmd output?
What frustrates me personally is "forced" line wrapping, as
example of:

$ gluster volume status

many thanks, L.


From evilmf at gmail.com  Thu Jul  2 13:33:51 2020
From: evilmf at gmail.com (Marco Fais)
Date: Thu, 2 Jul 2020 14:33:51 +0100
Subject: [Gluster-users] Problems with qemu and disperse volumes (live
	merge)
In-Reply-To: <93D3EE3B-B5B3-4689-BF66-C1442A03971E@yahoo.com>
References: <CAC7CbKH2pgyAKLNfdPvSZSvTL2Dc5USngubNr0BMD573c_VgBQ@mail.gmail.com>
	<F1022CAA-A397-483D-AA12-98F6A20FD591@yahoo.com>
	<CAC7CbKHG4U5sCyru8jN7UgYzb4BbERuJnhsj=mOe15EH78kiJg@mail.gmail.com>
	<93D3EE3B-B5B3-4689-BF66-C1442A03971E@yahoo.com>
Message-ID: <CAC7CbKHc24RzVgQhRJiKpjjxCLGDjf1rhK-FEHUu4JbYyHjSJw@mail.gmail.com>

Hi Strahil,

WARNING: As you enabled sharding - NEVER DISABLE SHARDING, EVER !
>

Thanks -- good to be reminded :)


> >When you say they will not be optimal are you referring mainly to
> >performance considerations? We did plenty of testing, and in terms of
> >performance didn't have issues even with I/O intensive workloads (using
> >SSDs, I had issues with spinning disks).
>
> Yes, the client side has  to connect to 6 bricks (4+2) at a time  and
> calculate the data in order to obtain the necessary information.Same is
> valid for writing.
> If you need to conserve space, you can test VDO without compression (of
> even with it).
>

Understood -- will explore VDO. Storage usage efficiency is less important
than fault tolerance or performance for us -- disperse volumes seemed to
tick all the boxes so we looked at them primarily.
But clearly I had missed that they are not used as mainstream VM storage
for oVirt (I did know they weren't supported, but as explained thought was
more on the management side).


>
> Also  with replica  volumes,  you can use 'choose-local'  /in case you
> have faster than the network storage (like  NVMe)/ and increase the read
> speed. Of course  this feature is useful for Hyperconverged setup (gluster
> + ovirt on the same node).
>

Will explore this option as well, thanks for the suggestion.


> If you were using ovirt 4.3 ,  I  would  recommend you to focus  on
> gluster. Yet,  you  use  oVirt 4.4 which is quite  newer and it needs  some
> polishing.
>

Ovirt 4.3.9 (using the older Centos 7 qemu/libvirt) unfortunately had
similar issues with the disperse volumes. Not sure if exactly the same, as
never looked deeper into it, but the results were similar.
Ovirt 4.4.0 has some issues with snapshot deletion that are independent
from Gluster (I have raised the issue here,
https://bugzilla.redhat.com/show_bug.cgi?id=1840414, should be sorted with
4.4.2 I guess), so at the moment it only works with the "testing" AV repo.


> Check ovirt  engine  logs (on the HostedEngine VM or your standalone
> engine) ,  vdsm logs  on the host that was running the VM  and next - check
> the brick  logs.
>

Will do.

Thanks,
Marco
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200702/91030b72/attachment.html>

From shreyansh.shah at alpha-grep.com  Thu Jul  2 14:39:25 2020
From: shreyansh.shah at alpha-grep.com (Shreyansh Shah)
Date: Thu, 2 Jul 2020 20:09:25 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
Message-ID: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>

Hi All,

*We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.*

We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
on each node, 7 nodes in total. We added new bricks yesterday to the
existing setup.
Post that we did a rebalance fix-layout and then a rebalance (which is
currently still in progress). The status shows "failed" on certain bricks
but "in progress" for others. Adding output for gluster rebalance status
below.

The glusterfs client logs are flooded with "Mismatching layouts for
<directory>,gfid = <id>"
The performance too seems to have degraded due to this, even basic commands
like `cd` and `ls` are taking more than a minute compared to sub-second
number before brick addition.
Apart from that we also experienced many binaries and files giving error
stale file handle error even though the files were present.


*gluster rebalance status :*

Node Rebalanced-files          size       scanned      failures
skipped               status  run time in h:m:s
---------      -----------   -----------   -----------   -----------
-----------         ------------     --------------
localhost              176         3.5GB         12790             0
   8552          in progress       21:36:01
10.132.0.72             8232       394.8GB         19995            21
       26               failed       14:50:30
10.132.0.44            12625         1.0TB         50023             1
    10202          in progress       21:36:00
10.132.0.3            21982       956.8GB         79145             1
  34571          in progress       21:36:00
10.132.0.9             7975       355.8GB         20157             6
   1522               failed       14:51:45
10.132.0.73             6293       394.5GB         26414           151
     8085               failed       14:51:45
10.132.0.70             6480       477.1GB         21058            27
     1787               failed       14:50:32
Estimated time left for rebalance to complete :      130:56:28


*Logs from one of the clients below:*

[2020-07-02 12:30:14.971916] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:14.971935] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032013] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.032059] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.032107] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.032153] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
[2020-07-02 12:30:15.093329] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
layout - 2454306944 - 2761060379 - 4159036738
[2020-07-02 12:30:15.093373] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.093460] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
layout - 2761060380 - 3067813815 - 4159036738
[2020-07-02 12:30:15.093515] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151063] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
layout - 3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:15.151108] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.151149] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:15.151162] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
[2020-07-02 12:30:15.424321] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
layout - 920400036 - 1227153471 - 4159036738
[2020-07-02 12:30:15.424380] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424456] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
layout - 1840730208 - 2147483643 - 4159036738
[2020-07-02 12:30:15.424484] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424525] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
layout - 1533976772 - 1840730207 - 4159036738
[2020-07-02 12:30:15.424542] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:15.424596] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
layout - 613646600 - 920400035 - 4159036738
[2020-07-02 12:30:15.424607] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
[2020-07-02 12:30:16.004482] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
data-client-7 (hashed subvol is data-client-17)
[2020-07-02 12:30:16.005523] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_DATA.dat
[2020-07-02 12:30:16.531047] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:16.532086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_1_METADATA.dat
[2020-07-02 12:30:18.733229] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:18.734421] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_DATA.dat
[2020-07-02 12:30:19.171930] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:19.172901] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_CDS_2_METADATA.dat
[2020-07-02 12:30:21.028495] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
data-client-6 (hashed subvol is data-client-15)
[2020-07-02 12:30:21.029836] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_DATA.dat
[2020-07-02 12:30:21.127648] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:21.128713] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_2_METADATA.dat
[2020-07-02 12:30:21.201126] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.201928] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_DATA.dat
[2020-07-02 12:30:21.566158] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:21.567123] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_3_METADATA.dat
[2020-07-02 12:30:21.649357] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:21.661381] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_DATA.dat
[2020-07-02 12:30:21.748937] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
on data-client-15 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.749481] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_4_METADATA.dat
[2020-07-02 12:30:21.898593] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
data-client-14 (hashed subvol is data-client-7)
[2020-07-02 12:30:21.899442] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_DATA.dat
[2020-07-02 12:30:22.039337] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:22.040086] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/BSE_EQ_6_METADATA.dat
[2020-07-02 12:30:22.501877] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
on data-client-15 (hashed subvol is data-client-8)
[2020-07-02 12:30:22.502712] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_DATA.dat
[2020-07-02 12:30:22.782577] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
(hashed subvol is data-client-6)
[2020-07-02 12:30:22.783777] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECDS1_METADATA.dat
[2020-07-02 12:30:23.146847] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
data-client-17 (hashed subvol is data-client-9)
[2020-07-02 12:30:23.148009] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_DATA.dat
[2020-07-02 12:30:23.229290] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
subvol is data-client-6)
[2020-07-02 12:30:23.230151] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM1_METADATA.dat
[2020-07-02 12:30:23.889520] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:23.896618] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_DATA.dat
[2020-07-02 12:30:24.093017] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:24.094117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM2_METADATA.dat
[2020-07-02 12:30:24.345257] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:24.346234] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_DATA.dat
[2020-07-02 12:30:24.425835] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:24.426880] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSECM3_METADATA.dat
[2020-07-02 12:30:25.158718] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
on data-client-9 (hashed subvol is data-client-19)
[2020-07-02 12:30:25.159619] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_DATA.dat
[2020-07-02 12:30:25.531479] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
subvol is data-client-10)
[2020-07-02 12:30:25.540569] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO1_METADATA.dat
[2020-07-02 12:30:25.771692] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
on data-client-11 (hashed subvol is data-client-3)
[2020-07-02 12:30:25.772610] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_DATA.dat
[2020-07-02 12:30:25.866118] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
(hashed subvol is data-client-8)
[2020-07-02 12:30:25.866917] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO2_METADATA.dat
[2020-07-02 12:30:26.424386] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
on data-client-9 (hashed subvol is data-client-18)
[2020-07-02 12:30:26.425309] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_DATA.dat
[2020-07-02 12:30:26.818852] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
(hashed subvol is data-client-2)
[2020-07-02 12:30:26.819890] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO3_METADATA.dat
[2020-07-02 12:30:27.352405] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
on data-client-10 (hashed subvol is data-client-2)
[2020-07-02 12:30:27.352914] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_DATA.dat
[2020-07-02 12:30:27.521286] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
subvol is data-client-18)
[2020-07-02 12:30:27.522325] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO4_METADATA.dat
[2020-07-02 12:30:28.566634] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
on data-client-2 (hashed subvol is data-client-11)
[2020-07-02 12:30:28.579295] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO5_DATA.dat
[2020-07-02 12:30:28.958028] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
on data-client-7 (hashed subvol is data-client-16)
[2020-07-02 12:30:28.959102] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_DATA.dat
[2020-07-02 12:30:29.012429] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
subvol is data-client-15)
[2020-07-02 12:30:29.013416] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/MCASTNSEFNO6_METADATA.dat
[2020-07-02 12:30:29.396716] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
data-client-17 (hashed subvol is data-client-10)
[2020-07-02 12:30:29.397740] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSDATA.dat
[2020-07-02 12:30:29.556312] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
subvol is data-client-18)
[2020-07-02 12:30:29.557197] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
[2020-07-02 12:30:30.605354] I [MSGID: 109045]
[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
deletion of stale linkfile
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
(hashed subvol is data-client-19)
[2020-07-02 12:30:30.606117] I [MSGID: 109069]
[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
returned with op_ret -> 0 and op-errno -> 0 for
/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
[2020-07-02 12:30:31.559206] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
613576736 - 920330171 - 4159036738
[2020-07-02 12:30:31.559255] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.569025] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
920330172 - 1227083607 - 4159036738
[2020-07-02 12:30:31.569067] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
[2020-07-02 12:30:31.701849] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.701895] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid =
fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.738464] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
3681390552 - 3988143987 - 4159036738
[2020-07-02 12:30:31.738507] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX, gfid =
fff324f2-f855-4881-b77c-81e856522373
[2020-07-02 12:30:31.857102] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
layout - 3067883680 - 3374637115 - 4159036738
[2020-07-02 12:30:31.857147] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.857180] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
layout - 3374637116 - 3681390551 - 4159036738
[2020-07-02 12:30:31.857197] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917705] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
- 306753435 - 4159036738
[2020-07-02 12:30:31.917781] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:31.917855] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
layout - 3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:31.917874] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
f8447150-4801-4188-add9-ea295bb88729
[2020-07-02 12:30:32.390945] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
3681460416 - 3988213851 - 4159036738
[2020-07-02 12:30:32.390998] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:30:32.391056] I [MSGID: 109064]
[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
3988213852 - 4294967295 - 4159036738
[2020-07-02 12:30:32.391075] I [MSGID: 109018]
[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
b2d4deb7-c58c-4046-b6f2-7c7f44d71311
[2020-07-02 12:33:50.915279] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
(2cb54500-814d-4e85-83e7-e33d9440b18d)
(hash=data-client-6/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
(hash=data-client-6/cache=<nul>)
[2020-07-02 12:34:09.799586] I [MSGID: 109066]
[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
/raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
(99938ee6-6986-4123-9d72-ec09e2310b4f)
(hash=data-client-17/cache=data-client-18) =>
/raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
(hash=data-client-17/cache=<nul>)
....


Please look into this at top-priority if possible.
Let me know if anything else is required.


-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200702/350ba536/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul  2 14:45:38 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 02 Jul 2020 17:45:38 +0300
Subject: [Gluster-users] Problems with qemu and disperse volumes (live
	merge)
In-Reply-To: <CAC7CbKHc24RzVgQhRJiKpjjxCLGDjf1rhK-FEHUu4JbYyHjSJw@mail.gmail.com>
References: <CAC7CbKH2pgyAKLNfdPvSZSvTL2Dc5USngubNr0BMD573c_VgBQ@mail.gmail.com>
	<F1022CAA-A397-483D-AA12-98F6A20FD591@yahoo.com>
	<CAC7CbKHG4U5sCyru8jN7UgYzb4BbERuJnhsj=mOe15EH78kiJg@mail.gmail.com>
	<93D3EE3B-B5B3-4689-BF66-C1442A03971E@yahoo.com>
	<CAC7CbKHc24RzVgQhRJiKpjjxCLGDjf1rhK-FEHUu4JbYyHjSJw@mail.gmail.com>
Message-ID: <CCF9A9E3-151F-4D9B-A510-7E2675BD7BF4@yahoo.com>



?? 2 ??? 2020 ?. 16:33:51 GMT+03:00, Marco Fais <evilmf at gmail.com> ??????:
>Hi Strahil,
>
>WARNING: As you enabled sharding - NEVER DISABLE SHARDING, EVER !
>>
>
>Thanks -- good to be reminded :)
>
>
>> >When you say they will not be optimal are you referring mainly to
>> >performance considerations? We did plenty of testing, and in terms
>of
>> >performance didn't have issues even with I/O intensive workloads
>(using
>> >SSDs, I had issues with spinning disks).
>>
>> Yes, the client side has  to connect to 6 bricks (4+2) at a time  and
>> calculate the data in order to obtain the necessary information.Same
>is
>> valid for writing.
>> If you need to conserve space, you can test VDO without compression
>(of
>> even with it).
>>
>
>Understood -- will explore VDO. Storage usage efficiency is less
>important
>than fault tolerance or performance for us -- disperse volumes seemed
>to
>tick all the boxes so we looked at them primarily.
>But clearly I had missed that they are not used as mainstream VM
>storage
>for oVirt (I did know they weren't supported, but as explained thought
>was
>more on the management side).
>
>
>>
>> Also  with replica  volumes,  you can use 'choose-local'  /in case
>you
>> have faster than the network storage (like  NVMe)/ and increase the
>read
>> speed. Of course  this feature is useful for Hyperconverged setup
>(gluster
>> + ovirt on the same node).
>>
>
>Will explore this option as well, thanks for the suggestion.
>
>
>> If you were using ovirt 4.3 ,  I  would  recommend you to focus  on
>> gluster. Yet,  you  use  oVirt 4.4 which is quite  newer and it needs
> some
>> polishing.
>>
>
>Ovirt 4.3.9 (using the older Centos 7 qemu/libvirt) unfortunately had
>similar issues with the disperse volumes. Not sure if exactly the same,
>as
>never looked deeper into it, but the results were similar.
>Ovirt 4.4.0 has some issues with snapshot deletion that are independent
>from Gluster (I have raised the issue here,
>https://bugzilla.redhat.com/show_bug.cgi?id=1840414, should be sorted
>with
>4.4.2 I guess), so at the moment it only works with the "testing" AV
>repo.



In such case I can recommend you to:
1. Ensure you have enough space on all bricks for the logs (/var/log/gluster). Several gigs should be OK
2. Enable all logs to 'TRACE' . Red Hat's documentation on the topic is quite good:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level
3. Reproduce the issue on a fresh VM (never done snapshot deletion)
4. Disable (switch to info)  all logs as per the link in point 2

The logs will be spread among all nodes. If you have remote logging available, you can also use it for analysis of the logs.

Most probably the brick logs can provide useful information.


>
>> Check ovirt  engine  logs (on the HostedEngine VM or your standalone
>> engine) ,  vdsm logs  on the host that was running the VM  and next -
>check
>> the brick  logs.
>>
>
>Will do.
>
>Thanks,
>Marco


About VDO - it might require some tuning and even afterwards it won't be very performant, so it depends on your needs.

Best Regards,
Strahil Nikolov

From hunter86_bg at yahoo.com  Thu Jul  2 20:21:32 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 02 Jul 2020 23:21:32 +0300
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
	after new brick addition and rebalance
In-Reply-To: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
Message-ID: <F9DE30B5-F48C-4E0C-BDCC-A621B878F696@yahoo.com>

Hi Shreyansh,

have you checked  the gluster logs on the failed nodes for any clues -> for example 10.132.0.9 ?

Sadly I don't have much experience with pure distributed  volumes, but I do think that everything will be back to normal when the rebalance  is complete.
Yet, based on the output - it won't be soon.

Best Regards,
Strahil Nikolov

?? 2 ??? 2020 ?. 17:39:25 GMT+03:00, Shreyansh Shah <shreyansh.shah at alpha-grep.com> ??????:
>Hi All,
>
>*We are facing "Mismatching layouts for <directory>,gfid = <id>"
>errors.*
>
>We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB
>each)
>on each node, 7 nodes in total. We added new bricks yesterday to the
>existing setup.
>Post that we did a rebalance fix-layout and then a rebalance (which is
>currently still in progress). The status shows "failed" on certain
>bricks
>but "in progress" for others. Adding output for gluster rebalance
>status
>below.
>
>The glusterfs client logs are flooded with "Mismatching layouts for
><directory>,gfid = <id>"
>The performance too seems to have degraded due to this, even basic
>commands
>like `cd` and `ls` are taking more than a minute compared to sub-second
>number before brick addition.
>Apart from that we also experienced many binaries and files giving
>error
>stale file handle error even though the files were present.
>
>
>*gluster rebalance status :*
>
>Node Rebalanced-files          size       scanned      failures
>skipped               status  run time in h:m:s
>---------      -----------   -----------   -----------   -----------
>-----------         ------------     --------------
>localhost              176         3.5GB         12790             0
>   8552          in progress       21:36:01
>10.132.0.72             8232       394.8GB         19995            21
>       26               failed       14:50:30
>10.132.0.44            12625         1.0TB         50023             1
>    10202          in progress       21:36:00
>10.132.0.3            21982       956.8GB         79145             1
>  34571          in progress       21:36:00
>10.132.0.9             7975       355.8GB         20157             6
>   1522               failed       14:51:45
>10.132.0.73             6293       394.5GB         26414           151
>     8085               failed       14:51:45
>10.132.0.70             6480       477.1GB         21058            27
>     1787               failed       14:50:32
>Estimated time left for rebalance to complete :      130:56:28
>
>
>*Logs from one of the clients below:*
>
>[2020-07-02 12:30:14.971916] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641;
>disk
>layout - 2761060380 - 3067813815 - 4159036738
>[2020-07-02 12:30:14.971935] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>[2020-07-02 12:30:15.032013] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641;
>disk
>layout - 3681390552 - 3988143987 - 4159036738
>[2020-07-02 12:30:15.032059] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>[2020-07-02 12:30:15.032107] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641;
>disk
>layout - 3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:15.032153] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>[2020-07-02 12:30:15.093329] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794;
>disk
>layout - 2454306944 - 2761060379 - 4159036738
>[2020-07-02 12:30:15.093373] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.093460] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794;
>disk
>layout - 2761060380 - 3067813815 - 4159036738
>[2020-07-02 12:30:15.093515] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.151063] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794;
>disk
>layout - 3681390552 - 3988143987 - 4159036738
>[2020-07-02 12:30:15.151108] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.151149] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794;
>disk
>layout - 3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:15.151162] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>[2020-07-02 12:30:15.424321] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-11; inode layout - 920400036 - 1227153471 - 3997647794;
>disk
>layout - 920400036 - 1227153471 - 4159036738
>[2020-07-02 12:30:15.424380] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:15.424456] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794;
>disk
>layout - 1840730208 - 2147483643 - 4159036738
>[2020-07-02 12:30:15.424484] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:15.424525] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794;
>disk
>layout - 1533976772 - 1840730207 - 4159036738
>[2020-07-02 12:30:15.424542] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:15.424596] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>layout - 613646600 - 920400035 - 4159036738
>[2020-07-02 12:30:15.424607] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>[2020-07-02 12:30:16.004482] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat
>on
>data-client-7 (hashed subvol is data-client-17)
>[2020-07-02 12:30:16.005523] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_1_DATA.dat
>[2020-07-02 12:30:16.531047] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_1_METADATA.dat
>on data-client-9 (hashed subvol is data-client-19)
>[2020-07-02 12:30:16.532086] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_1_METADATA.dat
>[2020-07-02 12:30:18.733229] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat
>on
>data-client-17 (hashed subvol is data-client-9)
>[2020-07-02 12:30:18.734421] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_2_DATA.dat
>[2020-07-02 12:30:19.171930] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_2_METADATA.dat
>on data-client-9 (hashed subvol is data-client-18)
>[2020-07-02 12:30:19.172901] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_CDS_2_METADATA.dat
>[2020-07-02 12:30:21.028495] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat
>on
>data-client-6 (hashed subvol is data-client-15)
>[2020-07-02 12:30:21.029836] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_2_DATA.dat
>[2020-07-02 12:30:21.127648] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_2_METADATA.dat
>on data-client-11 (hashed subvol is data-client-3)
>[2020-07-02 12:30:21.128713] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_2_METADATA.dat
>[2020-07-02 12:30:21.201126] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat
>on
>data-client-15 (hashed subvol is data-client-7)
>[2020-07-02 12:30:21.201928] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_3_DATA.dat
>[2020-07-02 12:30:21.566158] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_3_METADATA.dat
>on data-client-7 (hashed subvol is data-client-16)
>[2020-07-02 12:30:21.567123] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_3_METADATA.dat
>[2020-07-02 12:30:21.649357] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat
>on
>data-client-2 (hashed subvol is data-client-11)
>[2020-07-02 12:30:21.661381] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_4_DATA.dat
>[2020-07-02 12:30:21.748937] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_4_METADATA.dat
>on data-client-15 (hashed subvol is data-client-7)
>[2020-07-02 12:30:21.749481] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_4_METADATA.dat
>[2020-07-02 12:30:21.898593] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat
>on
>data-client-14 (hashed subvol is data-client-7)
>[2020-07-02 12:30:21.899442] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_6_DATA.dat
>[2020-07-02 12:30:22.039337] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_6_METADATA.dat
>on data-client-10 (hashed subvol is data-client-2)
>[2020-07-02 12:30:22.040086] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/BSE_EQ_6_METADATA.dat
>[2020-07-02 12:30:22.501877] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECDS1_DATA.dat
>on data-client-15 (hashed subvol is data-client-8)
>[2020-07-02 12:30:22.502712] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECDS1_DATA.dat
>[2020-07-02 12:30:22.782577] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>(hashed subvol is data-client-6)
>[2020-07-02 12:30:22.783777] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECDS1_METADATA.dat
>[2020-07-02 12:30:23.146847] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM1_DATA.dat on
>data-client-17 (hashed subvol is data-client-9)
>[2020-07-02 12:30:23.148009] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM1_DATA.dat
>[2020-07-02 12:30:23.229290] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14
>(hashed
>subvol is data-client-6)
>[2020-07-02 12:30:23.230151] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM1_METADATA.dat
>[2020-07-02 12:30:23.889520] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM2_DATA.dat on
>data-client-2 (hashed subvol is data-client-11)
>[2020-07-02 12:30:23.896618] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM2_DATA.dat
>[2020-07-02 12:30:24.093017] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6
>(hashed
>subvol is data-client-15)
>[2020-07-02 12:30:24.094117] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM2_METADATA.dat
>[2020-07-02 12:30:24.345257] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM3_DATA.dat on
>data-client-17 (hashed subvol is data-client-10)
>[2020-07-02 12:30:24.346234] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM3_DATA.dat
>[2020-07-02 12:30:24.425835] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6
>(hashed
>subvol is data-client-15)
>[2020-07-02 12:30:24.426880] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSECM3_METADATA.dat
>[2020-07-02 12:30:25.158718] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO1_DATA.dat
>on data-client-9 (hashed subvol is data-client-19)
>[2020-07-02 12:30:25.159619] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO1_DATA.dat
>[2020-07-02 12:30:25.531479] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2
>(hashed
>subvol is data-client-10)
>[2020-07-02 12:30:25.540569] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>[2020-07-02 12:30:25.771692] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO2_DATA.dat
>on data-client-11 (hashed subvol is data-client-3)
>[2020-07-02 12:30:25.772610] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO2_DATA.dat
>[2020-07-02 12:30:25.866118] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>(hashed subvol is data-client-8)
>[2020-07-02 12:30:25.866917] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>[2020-07-02 12:30:26.424386] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO3_DATA.dat
>on data-client-9 (hashed subvol is data-client-18)
>[2020-07-02 12:30:26.425309] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO3_DATA.dat
>[2020-07-02 12:30:26.818852] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>(hashed subvol is data-client-2)
>[2020-07-02 12:30:26.819890] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>[2020-07-02 12:30:27.352405] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO4_DATA.dat
>on data-client-10 (hashed subvol is data-client-2)
>[2020-07-02 12:30:27.352914] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO4_DATA.dat
>[2020-07-02 12:30:27.521286] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8
>(hashed
>subvol is data-client-18)
>[2020-07-02 12:30:27.522325] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>[2020-07-02 12:30:28.566634] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO5_DATA.dat
>on data-client-2 (hashed subvol is data-client-11)
>[2020-07-02 12:30:28.579295] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO5_DATA.dat
>[2020-07-02 12:30:28.958028] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO6_DATA.dat
>on data-client-7 (hashed subvol is data-client-16)
>[2020-07-02 12:30:28.959102] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO6_DATA.dat
>[2020-07-02 12:30:29.012429] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6
>(hashed
>subvol is data-client-15)
>[2020-07-02 12:30:29.013416] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>[2020-07-02 12:30:29.396716] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>data-client-17 (hashed subvol is data-client-10)
>[2020-07-02 12:30:29.397740] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/NSEFO_BSE_TSDATA.dat
>[2020-07-02 12:30:29.556312] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9
>(hashed
>subvol is data-client-18)
>[2020-07-02 12:30:29.557197] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>[2020-07-02 12:30:30.605354] I [MSGID: 109045]
>[dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>deletion of stale linkfile
>/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on
>data-client-9
>(hashed subvol is data-client-19)
>[2020-07-02 12:30:30.606117] I [MSGID: 109069]
>[dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>returned with op_ret -> 0 and op-errno -> 0 for
>/processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>[2020-07-02 12:30:31.559206] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>613576736 - 920330171 - 4159036738
>[2020-07-02 12:30:31.559255] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>[2020-07-02 12:30:31.569025] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout
>-
>920330172 - 1227083607 - 4159036738
>[2020-07-02 12:30:31.569067] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>[2020-07-02 12:30:31.701849] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout
>-
>3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:31.701895] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX, gfid =
>fff324f2-f855-4881-b77c-81e856522373
>[2020-07-02 12:30:31.738464] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout
>-
>3681390552 - 3988143987 - 4159036738
>[2020-07-02 12:30:31.738507] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX, gfid =
>fff324f2-f855-4881-b77c-81e856522373
>[2020-07-02 12:30:31.857102] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641;
>disk
>layout - 3067883680 - 3374637115 - 4159036738
>[2020-07-02 12:30:31.857147] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:31.857180] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641;
>disk
>layout - 3374637116 - 3681390551 - 4159036738
>[2020-07-02 12:30:31.857197] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:31.917705] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout
>- 0
>- 306753435 - 4159036738
>[2020-07-02 12:30:31.917781] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:31.917855] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641;
>disk
>layout - 3988213852 - 4294967295 - 4159036738
>[2020-07-02 12:30:31.917874] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>f8447150-4801-4188-add9-ea295bb88729
>[2020-07-02 12:30:32.390945] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout
>-
>3681460416 - 3988213851 - 4159036738
>[2020-07-02 12:30:32.390998] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>[2020-07-02 12:30:32.391056] I [MSGID: 109064]
>[dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout
>-
>3988213852 - 4294967295 - 4159036738
>[2020-07-02 12:30:32.391075] I [MSGID: 109018]
>[dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts
>for
>/processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>[2020-07-02 12:33:50.915279] I [MSGID: 109066]
>[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>/raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>(2cb54500-814d-4e85-83e7-e33d9440b18d)
>(hash=data-client-6/cache=data-client-18) =>
>/raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>(hash=data-client-6/cache=<nul>)
>[2020-07-02 12:34:09.799586] I [MSGID: 109066]
>[dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>/raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>(99938ee6-6986-4123-9d72-ec09e2310b4f)
>(hash=data-client-17/cache=data-client-18) =>
>/raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>(hash=data-client-17/cache=<nul>)
>....
>
>
>Please look into this at top-priority if possible.
>Let me know if anything else is required.

From felix.koelzow at gmx.de  Fri Jul  3 08:16:30 2020
From: felix.koelzow at gmx.de (=?UTF-8?Q?Felix_K=c3=b6lzow?=)
Date: Fri, 3 Jul 2020 10:16:30 +0200
Subject: [Gluster-users] Geo-replication completely broken
In-Reply-To: <CAERh03pZY-xMfMtpVxk2uZwJwjsEMgZwMVte+SmuDVOGxKUxDQ@mail.gmail.com>
References: <DB7PR03MB372103FFFCF435FBC352C645FD920@DB7PR03MB3721.eurprd03.prod.outlook.com>
	<31c26aca-2dbd-e798-27f0-e8c33afe7f21@gmx.de>
	<CAERh03pZY-xMfMtpVxk2uZwJwjsEMgZwMVte+SmuDVOGxKUxDQ@mail.gmail.com>
Message-ID: <3c8c2b2b-b539-f8f7-0597-d96d2be1fd74@gmx.de>

Dear Users,
the geo-replication is still broken. This is not really a comfortable
situation.
Does any user has had the same experience and is able to share a
possible workaround?
We are actually running gluster v6.0
Regards,

Felix


On 25/06/2020 10:04, Shwetha Acharya wrote:
> Hi Rob and Felix,
>
> Please share the *-changes.log files and brick logs, which will help
> in analysis of the issue.
>
> Regards,
> Shwetha
>
> On Thu, Jun 25, 2020 at 1:26 PM Felix K?lzow <felix.koelzow at gmx.de
> <mailto:felix.koelzow at gmx.de>> wrote:
>
>     Hey Rob,
>
>
>     same issue for our third volume. Have a look at the logs just from
>     right now (below).
>
>     Question: You removed the htime files and the old changelogs. Just
>     rm the files or is there something to pay more attention
>
>     before removing the changelog files and the htime file.
>
>     Regards,
>
>     Felix
>
>     [2020-06-25 07:51:53.795430] I [resource(worker
>     /gluster/vg00/dispersed_fuse1024/brick):1435:connect_remote] SSH:
>     SSH connection between master and slave established.???
>     duration=1.2341
>     [2020-06-25 07:51:53.795639] I [resource(worker
>     /gluster/vg00/dispersed_fuse1024/brick):1105:connect] GLUSTER:
>     Mounting gluster volume locally...
>     [2020-06-25 07:51:54.520601] I [monitor(monitor):280:monitor]
>     Monitor: worker died in startup phase
>     brick=/gluster/vg01/dispersed_fuse1024/brick
>     [2020-06-25 07:51:54.535809] I
>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
>     Status Change??? status=Faulty
>     [2020-06-25 07:51:54.882143] I [resource(worker
>     /gluster/vg00/dispersed_fuse1024/brick):1128:connect] GLUSTER:
>     Mounted gluster volume??? duration=1.0864
>     [2020-06-25 07:51:54.882388] I [subcmds(worker
>     /gluster/vg00/dispersed_fuse1024/brick):84:subcmd_worker] <top>:
>     Worker spawn successful. Acknowledging back to monitor
>     [2020-06-25 07:51:56.911412] E [repce(agent
>     /gluster/vg00/dispersed_fuse1024/brick):121:worker] <top>: call
>     failed:
>     Traceback (most recent call last):
>     ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line
>     117, in worker
>     ??? res = getattr(self.obj, rmeth)(*in_data[2:])
>     ? File
>     "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py", line
>     40, in register
>     ??? return Changes.cl_register(cl_brick, cl_dir, cl_log, cl_level,
>     retries)
>     ? File
>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line
>     46, in cl_register
>     ??? cls.raise_changelog_err()
>     ? File
>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py", line
>     30, in raise_changelog_err
>     ??? raise ChangelogException(errn, os.strerror(errn))
>     ChangelogException: [Errno 2] No such file or directory
>     [2020-06-25 07:51:56.912056] E [repce(worker
>     /gluster/vg00/dispersed_fuse1024/brick):213:__call__] RepceClient:
>     call failed call=75086:140098349655872:1593071514.91
>     method=register??? error=ChangelogException
>     [2020-06-25 07:51:56.912396] E [resource(worker
>     /gluster/vg00/dispersed_fuse1024/brick):1286:service_loop]
>     GLUSTER: Changelog register failed??? error=[Errno 2] No such file
>     or directory
>     [2020-06-25 07:51:56.928031] I [repce(agent
>     /gluster/vg00/dispersed_fuse1024/brick):96:service_loop]
>     RepceServer: terminating on reaching EOF.
>     [2020-06-25 07:51:57.886126] I [monitor(monitor):280:monitor]
>     Monitor: worker died in startup phase
>     brick=/gluster/vg00/dispersed_fuse1024/brick
>     [2020-06-25 07:51:57.895920] I
>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus: Worker
>     Status Change??? status=Faulty
>     [2020-06-25 07:51:58.607405] I [gsyncdstatus(worker
>     /gluster/vg00/dispersed_fuse1024/brick):287:set_passive]
>     GeorepStatus: Worker Status Change??? status=Passive
>     [2020-06-25 07:51:58.607768] I [gsyncdstatus(worker
>     /gluster/vg01/dispersed_fuse1024/brick):287:set_passive]
>     GeorepStatus: Worker Status Change??? status=Passive
>     [2020-06-25 07:51:58.608004] I [gsyncdstatus(worker
>     /gluster/vg00/dispersed_fuse1024/brick):281:set_active]
>     GeorepStatus: Worker Status Change??? status=Active
>
>
>     On 25/06/2020 09:15, Rob.Quagliozzi at rabobank.com
>     <mailto:Rob.Quagliozzi at rabobank.com> wrote:
>>
>>     Hi All,
>>
>>     We?ve got two six node RHEL 7.8 clusters and geo-replication
>>     would appear to be completely broken between them. I?ve deleted
>>     the session, removed & recreated pem files, old changlogs/htime
>>     (after removing relevant options from volume) and completely set
>>     up geo-rep from scratch, but the new session comes up as
>>     Initializing, then goes faulty, and starts looping. Volume (on
>>     both sides) is a 4 x 2 disperse, running Gluster v6 (RH latest).?
>>     Gsyncd reports:
>>
>>     [2020-06-25 07:07:14.701423] I
>>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
>>     Worker Status Change status=Initializing...
>>
>>     [2020-06-25 07:07:14.701744] I [monitor(monitor):159:monitor]
>>     Monitor: starting gsyncd worker?? brick=/rhgs/brick20/brick
>>     slave_node=bxts470194.eu.rabonet.com
>>     <http://bxts470194.eu.rabonet.com>
>>
>>     [2020-06-25 07:07:14.707997] D [monitor(monitor):230:monitor]
>>     Monitor: Worker would mount volume privately
>>
>>     [2020-06-25 07:07:14.757181] I [gsyncd(agent
>>     /rhgs/brick20/brick):318:main] <top>: Using session config file
>>     path=/var/lib/glusterd/geo-replication/prd_mx_intvol_bxts470190_prd_mx_intvol/gsyncd.conf
>>
>>     [2020-06-25 07:07:14.758126] D [subcmds(agent
>>     /rhgs/brick20/brick):107:subcmd_agent] <top>: RPC FD?????
>>     rpc_fd='5,12,11,10'
>>
>>     [2020-06-25 07:07:14.758627] I [changelogagent(agent
>>     /rhgs/brick20/brick):72:__init__] ChangelogAgent: Agent listining...
>>
>>     [2020-06-25 07:07:14.764234] I [gsyncd(worker
>>     /rhgs/brick20/brick):318:main] <top>: Using session config file
>>     path=/var/lib/glusterd/geo-replication/prd_mx_intvol_bxts470190_prd_mx_intvol/gsyncd.conf
>>
>>     [2020-06-25 07:07:14.779409] I [resource(worker
>>     /rhgs/brick20/brick):1386:connect_remote] SSH: Initializing SSH
>>     connection between master and slave...
>>
>>     [2020-06-25 07:07:14.841793] D [repce(worker
>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>     6799:140380783982400:1593068834.84 __repce_version__() ...
>>
>>     [2020-06-25 07:07:16.148725] D [repce(worker
>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>     6799:140380783982400:1593068834.84 __repce_version__ -> 1.0
>>
>>     [2020-06-25 07:07:16.148911] D [repce(worker
>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>     6799:140380783982400:1593068836.15 version() ...
>>
>>     [2020-06-25 07:07:16.149574] D [repce(worker
>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>     6799:140380783982400:1593068836.15 version -> 1.0
>>
>>     [2020-06-25 07:07:16.149735] D [repce(worker
>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>     6799:140380783982400:1593068836.15 pid() ...
>>
>>     [2020-06-25 07:07:16.150588] D [repce(worker
>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>     6799:140380783982400:1593068836.15 pid -> 30703
>>
>>     [2020-06-25 07:07:16.150747] I [resource(worker
>>     /rhgs/brick20/brick):1435:connect_remote] SSH: SSH connection
>>     between master and slave established. duration=1.3712
>>
>>     [2020-06-25 07:07:16.150819] I [resource(worker
>>     /rhgs/brick20/brick):1105:connect] GLUSTER: Mounting gluster
>>     volume locally...
>>
>>     [2020-06-25 07:07:16.265860] D [resource(worker
>>     /rhgs/brick20/brick):879:inhibit] DirectMounter: auxiliary
>>     glusterfs mount in place
>>
>>     [2020-06-25 07:07:17.272511] D [resource(worker
>>     /rhgs/brick20/brick):953:inhibit] DirectMounter: auxiliary
>>     glusterfs mount prepared
>>
>>     [2020-06-25 07:07:17.272708] I [resource(worker
>>     /rhgs/brick20/brick):1128:connect] GLUSTER: Mounted gluster
>>     volume????? duration=1.1218
>>
>>     [2020-06-25 07:07:17.272794] I [subcmds(worker
>>     /rhgs/brick20/brick):84:subcmd_worker] <top>: Worker spawn
>>     successful. Acknowledging back to monitor
>>
>>     [2020-06-25 07:07:17.272973] D [master(worker
>>     /rhgs/brick20/brick):104:gmaster_builder] <top>: setting up
>>     change detection mode mode=xsync
>>
>>     [2020-06-25 07:07:17.273063] D [monitor(monitor):273:monitor]
>>     Monitor: worker(/rhgs/brick20/brick) connected
>>
>>     [2020-06-25 07:07:17.273678] D [master(worker
>>     /rhgs/brick20/brick):104:gmaster_builder] <top>: setting up
>>     change detection mode mode=changelog
>>
>>     [2020-06-25 07:07:17.274224] D [master(worker
>>     /rhgs/brick20/brick):104:gmaster_builder] <top>: setting up
>>     change detection mode mode=changeloghistory
>>
>>     [2020-06-25 07:07:17.276484] D [repce(worker
>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>     6799:140380783982400:1593068837.28 version() ...
>>
>>     [2020-06-25 07:07:17.276916] D [repce(worker
>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>     6799:140380783982400:1593068837.28 version -> 1.0
>>
>>     [2020-06-25 07:07:17.277009] D [master(worker
>>     /rhgs/brick20/brick):777:setup_working_dir] _GMaster: changelog
>>     working dir
>>     /var/lib/misc/gluster/gsyncd/prd_mx_intvol_bxts470190_prd_mx_intvol/rhgs-brick20-brick
>>
>>     [2020-06-25 07:07:17.277098] D [repce(worker
>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>     6799:140380783982400:1593068837.28 init() ...
>>
>>     [2020-06-25 07:07:17.292944] D [repce(worker
>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>     6799:140380783982400:1593068837.28 init -> None
>>
>>     [2020-06-25 07:07:17.293097] D [repce(worker
>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>     6799:140380783982400:1593068837.29
>>     register('/rhgs/brick20/brick',
>>     '/var/lib/misc/gluster/gsyncd/prd_mx_intvol_bxts470190_prd_mx_intvol/rhgs-brick20-brick',
>>     '/var/log/glusterfs/geo-replication/prd_mx_intvol_bxts470190_prd_mx_intvol/changes-rhgs-brick20-brick.log',
>>     8, 5) ...
>>
>>     [2020-06-25 07:07:19.296294] E [repce(agent
>>     /rhgs/brick20/brick):121:worker] <top>: call failed:
>>
>>     Traceback (most recent call last):
>>
>>     ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line
>>     117, in worker
>>
>>     ??? res = getattr(self.obj, rmeth)(*in_data[2:])
>>
>>     ? File
>>     "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py",
>>     line 40, in register
>>
>>     ??? return Changes.cl_register(cl_brick, cl_dir, cl_log,
>>     cl_level, retries)
>>
>>     ? File
>>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
>>     line 46, in cl_register
>>
>>     ??? cls.raise_changelog_err()
>>
>>     ? File
>>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
>>     line 30, in raise_changelog_err
>>
>>     ??? raise ChangelogException(errn, os.strerror(errn))
>>
>>     ChangelogException: [Errno 2] No such file or directory
>>
>>     [2020-06-25 07:07:19.297161] E [repce(worker
>>     /rhgs/brick20/brick):213:__call__] RepceClient: call failed
>>     call=6799:140380783982400:1593068837.29 method=register
>>     error=ChangelogException
>>
>>     [2020-06-25 07:07:19.297338] E [resource(worker
>>     /rhgs/brick20/brick):1286:service_loop] GLUSTER: Changelog
>>     register failed????? error=[Errno 2] No such file or directory
>>
>>     [2020-06-25 07:07:19.315074] I [repce(agent
>>     /rhgs/brick20/brick):96:service_loop] RepceServer: terminating on
>>     reaching EOF.
>>
>>     [2020-06-25 07:07:20.275701] I [monitor(monitor):280:monitor]
>>     Monitor: worker died in startup phase???? brick=/rhgs/brick20/brick
>>
>>     [2020-06-25 07:07:20.277383] I
>>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
>>     Worker Status Change status=Faulty
>>
>>     We?ve done everything we can think of, including an ?strace ?f?
>>     on the pid, and we can?t really find anything. I?m about to lose
>>     the last of my hair over this, so does anyone have any ideas at
>>     all? We?ve even removed the entire slave vol and rebuilt it.
>>
>>     Thanks
>>
>>     Rob
>>
>>     *Rob Quagliozzi*
>>
>>     *Specialised Application Support*
>>
>>
>>
>>     ------------------------------------------------------------------------
>>     This email (including any attachments to it) is confidential,
>>     legally privileged, subject to copyright and is sent for the
>>     personal attention of the intended recipient only. If you have
>>     received this email in error, please advise us immediately and
>>     delete it. You are notified that disclosing, copying,
>>     distributing or taking any action in reliance on the contents of
>>     this information is strictly prohibited. Although we have taken
>>     reasonable precautions to ensure no viruses are present in this
>>     email, we cannot accept responsibility for any loss or damage
>>     arising from the viruses in this email or attachments. We exclude
>>     any liability for the content of this email, or for the
>>     consequences of any actions taken on the basis of the information
>>     provided in this email or its attachments, unless that
>>     information is subsequently confirmed in writing. <#rbnl#1898i>
>>     ------------------------------------------------------------------------
>>
>>
>>     ________
>>
>>
>>
>>     Community Meeting Calendar:
>>
>>     Schedule -
>>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>     Bridge:https://bluejeans.com/441850968
>>
>>     Gluster-users mailing list
>>     Gluster-users at gluster.org  <mailto:Gluster-users at gluster.org>
>>     https://lists.gluster.org/mailman/listinfo/gluster-users
>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
>
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200703/993e8f43/attachment.html>

From peljasz at yahoo.co.uk  Fri Jul  3 10:46:33 2020
From: peljasz at yahoo.co.uk (lejeczek)
Date: Fri, 3 Jul 2020 11:46:33 +0100
Subject: [Gluster-users] Official Bugzilla?
References: <bea778af-c1e1-dde5-b0e3-9d01ae3b6185.ref@yahoo.co.uk>
Message-ID: <bea778af-c1e1-dde5-b0e3-9d01ae3b6185@yahoo.co.uk>

hi guys,

where those of use who run gluster from(via) EPEL repo
should go to report bugs?

many thanks, L.


From hunter86_bg at yahoo.com  Fri Jul  3 12:54:52 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Fri, 03 Jul 2020 15:54:52 +0300
Subject: [Gluster-users] Geo-replication completely broken
In-Reply-To: <3c8c2b2b-b539-f8f7-0597-d96d2be1fd74@gmx.de>
References: <DB7PR03MB372103FFFCF435FBC352C645FD920@DB7PR03MB3721.eurprd03.prod.outlook.com>
	<31c26aca-2dbd-e798-27f0-e8c33afe7f21@gmx.de>
	<CAERh03pZY-xMfMtpVxk2uZwJwjsEMgZwMVte+SmuDVOGxKUxDQ@mail.gmail.com>
	<3c8c2b2b-b539-f8f7-0597-d96d2be1fd74@gmx.de>
Message-ID: <D83F540F-E9D0-416C-8480-7CC8E083274D@yahoo.com>

Hi Felix,

It seems I missed your reply with the change log that Shwetha requested.

Best Regards,
Strahil Nikolov

?? 3 ??? 2020 ?. 11:16:30 GMT+03:00, "Felix K?lzow" <felix.koelzow at gmx.de> ??????:
>Dear Users,
>the geo-replication is still broken. This is not really a comfortable
>situation.
>Does any user has had the same experience and is able to share a
>possible workaround?
>We are actually running gluster v6.0
>Regards,
>
>Felix
>
>
>On 25/06/2020 10:04, Shwetha Acharya wrote:
>> Hi Rob and Felix,
>>
>> Please share the *-changes.log files and brick logs, which will help
>> in analysis of the issue.
>>
>> Regards,
>> Shwetha
>>
>> On Thu, Jun 25, 2020 at 1:26 PM Felix K?lzow <felix.koelzow at gmx.de
>> <mailto:felix.koelzow at gmx.de>> wrote:
>>
>>     Hey Rob,
>>
>>
>>     same issue for our third volume. Have a look at the logs just
>from
>>     right now (below).
>>
>>     Question: You removed the htime files and the old changelogs.
>Just
>>     rm the files or is there something to pay more attention
>>
>>     before removing the changelog files and the htime file.
>>
>>     Regards,
>>
>>     Felix
>>
>>     [2020-06-25 07:51:53.795430] I [resource(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):1435:connect_remote] SSH:
>>     SSH connection between master and slave established.???
>>     duration=1.2341
>>     [2020-06-25 07:51:53.795639] I [resource(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):1105:connect] GLUSTER:
>>     Mounting gluster volume locally...
>>     [2020-06-25 07:51:54.520601] I [monitor(monitor):280:monitor]
>>     Monitor: worker died in startup phase
>>     brick=/gluster/vg01/dispersed_fuse1024/brick
>>     [2020-06-25 07:51:54.535809] I
>>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
>Worker
>>     Status Change??? status=Faulty
>>     [2020-06-25 07:51:54.882143] I [resource(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):1128:connect] GLUSTER:
>>     Mounted gluster volume??? duration=1.0864
>>     [2020-06-25 07:51:54.882388] I [subcmds(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):84:subcmd_worker] <top>:
>>     Worker spawn successful. Acknowledging back to monitor
>>     [2020-06-25 07:51:56.911412] E [repce(agent
>>     /gluster/vg00/dispersed_fuse1024/brick):121:worker] <top>: call
>>     failed:
>>     Traceback (most recent call last):
>>     ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line
>>     117, in worker
>>     ??? res = getattr(self.obj, rmeth)(*in_data[2:])
>>     ? File
>>     "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py",
>line
>>     40, in register
>>     ??? return Changes.cl_register(cl_brick, cl_dir, cl_log,
>cl_level,
>>     retries)
>>     ? File
>>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
>line
>>     46, in cl_register
>>     ??? cls.raise_changelog_err()
>>     ? File
>>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
>line
>>     30, in raise_changelog_err
>>     ??? raise ChangelogException(errn, os.strerror(errn))
>>     ChangelogException: [Errno 2] No such file or directory
>>     [2020-06-25 07:51:56.912056] E [repce(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):213:__call__]
>RepceClient:
>>     call failed call=75086:140098349655872:1593071514.91
>>     method=register??? error=ChangelogException
>>     [2020-06-25 07:51:56.912396] E [resource(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):1286:service_loop]
>>     GLUSTER: Changelog register failed??? error=[Errno 2] No such
>file
>>     or directory
>>     [2020-06-25 07:51:56.928031] I [repce(agent
>>     /gluster/vg00/dispersed_fuse1024/brick):96:service_loop]
>>     RepceServer: terminating on reaching EOF.
>>     [2020-06-25 07:51:57.886126] I [monitor(monitor):280:monitor]
>>     Monitor: worker died in startup phase
>>     brick=/gluster/vg00/dispersed_fuse1024/brick
>>     [2020-06-25 07:51:57.895920] I
>>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
>Worker
>>     Status Change??? status=Faulty
>>     [2020-06-25 07:51:58.607405] I [gsyncdstatus(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):287:set_passive]
>>     GeorepStatus: Worker Status Change??? status=Passive
>>     [2020-06-25 07:51:58.607768] I [gsyncdstatus(worker
>>     /gluster/vg01/dispersed_fuse1024/brick):287:set_passive]
>>     GeorepStatus: Worker Status Change??? status=Passive
>>     [2020-06-25 07:51:58.608004] I [gsyncdstatus(worker
>>     /gluster/vg00/dispersed_fuse1024/brick):281:set_active]
>>     GeorepStatus: Worker Status Change??? status=Active
>>
>>
>>     On 25/06/2020 09:15, Rob.Quagliozzi at rabobank.com
>>     <mailto:Rob.Quagliozzi at rabobank.com> wrote:
>>>
>>>     Hi All,
>>>
>>>     We?ve got two six node RHEL 7.8 clusters and geo-replication
>>>     would appear to be completely broken between them. I?ve deleted
>>>     the session, removed & recreated pem files, old changlogs/htime
>>>     (after removing relevant options from volume) and completely set
>>>     up geo-rep from scratch, but the new session comes up as
>>>     Initializing, then goes faulty, and starts looping. Volume (on
>>>     both sides) is a 4 x 2 disperse, running Gluster v6 (RH
>latest).?
>>>     Gsyncd reports:
>>>
>>>     [2020-06-25 07:07:14.701423] I
>>>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
>>>     Worker Status Change status=Initializing...
>>>
>>>     [2020-06-25 07:07:14.701744] I [monitor(monitor):159:monitor]
>>>     Monitor: starting gsyncd worker?? brick=/rhgs/brick20/brick
>>>     slave_node=bxts470194.eu.rabonet.com
>>>     <http://bxts470194.eu.rabonet.com>
>>>
>>>     [2020-06-25 07:07:14.707997] D [monitor(monitor):230:monitor]
>>>     Monitor: Worker would mount volume privately
>>>
>>>     [2020-06-25 07:07:14.757181] I [gsyncd(agent
>>>     /rhgs/brick20/brick):318:main] <top>: Using session config file
>>>    
>path=/var/lib/glusterd/geo-replication/prd_mx_intvol_bxts470190_prd_mx_intvol/gsyncd.conf
>>>
>>>     [2020-06-25 07:07:14.758126] D [subcmds(agent
>>>     /rhgs/brick20/brick):107:subcmd_agent] <top>: RPC FD?????
>>>     rpc_fd='5,12,11,10'
>>>
>>>     [2020-06-25 07:07:14.758627] I [changelogagent(agent
>>>     /rhgs/brick20/brick):72:__init__] ChangelogAgent: Agent
>listining...
>>>
>>>     [2020-06-25 07:07:14.764234] I [gsyncd(worker
>>>     /rhgs/brick20/brick):318:main] <top>: Using session config file
>>>    
>path=/var/lib/glusterd/geo-replication/prd_mx_intvol_bxts470190_prd_mx_intvol/gsyncd.conf
>>>
>>>     [2020-06-25 07:07:14.779409] I [resource(worker
>>>     /rhgs/brick20/brick):1386:connect_remote] SSH: Initializing SSH
>>>     connection between master and slave...
>>>
>>>     [2020-06-25 07:07:14.841793] D [repce(worker
>>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>>     6799:140380783982400:1593068834.84 __repce_version__() ...
>>>
>>>     [2020-06-25 07:07:16.148725] D [repce(worker
>>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>>     6799:140380783982400:1593068834.84 __repce_version__ -> 1.0
>>>
>>>     [2020-06-25 07:07:16.148911] D [repce(worker
>>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>>     6799:140380783982400:1593068836.15 version() ...
>>>
>>>     [2020-06-25 07:07:16.149574] D [repce(worker
>>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>>     6799:140380783982400:1593068836.15 version -> 1.0
>>>
>>>     [2020-06-25 07:07:16.149735] D [repce(worker
>>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>>     6799:140380783982400:1593068836.15 pid() ...
>>>
>>>     [2020-06-25 07:07:16.150588] D [repce(worker
>>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>>     6799:140380783982400:1593068836.15 pid -> 30703
>>>
>>>     [2020-06-25 07:07:16.150747] I [resource(worker
>>>     /rhgs/brick20/brick):1435:connect_remote] SSH: SSH connection
>>>     between master and slave established. duration=1.3712
>>>
>>>     [2020-06-25 07:07:16.150819] I [resource(worker
>>>     /rhgs/brick20/brick):1105:connect] GLUSTER: Mounting gluster
>>>     volume locally...
>>>
>>>     [2020-06-25 07:07:16.265860] D [resource(worker
>>>     /rhgs/brick20/brick):879:inhibit] DirectMounter: auxiliary
>>>     glusterfs mount in place
>>>
>>>     [2020-06-25 07:07:17.272511] D [resource(worker
>>>     /rhgs/brick20/brick):953:inhibit] DirectMounter: auxiliary
>>>     glusterfs mount prepared
>>>
>>>     [2020-06-25 07:07:17.272708] I [resource(worker
>>>     /rhgs/brick20/brick):1128:connect] GLUSTER: Mounted gluster
>>>     volume????? duration=1.1218
>>>
>>>     [2020-06-25 07:07:17.272794] I [subcmds(worker
>>>     /rhgs/brick20/brick):84:subcmd_worker] <top>: Worker spawn
>>>     successful. Acknowledging back to monitor
>>>
>>>     [2020-06-25 07:07:17.272973] D [master(worker
>>>     /rhgs/brick20/brick):104:gmaster_builder] <top>: setting up
>>>     change detection mode mode=xsync
>>>
>>>     [2020-06-25 07:07:17.273063] D [monitor(monitor):273:monitor]
>>>     Monitor: worker(/rhgs/brick20/brick) connected
>>>
>>>     [2020-06-25 07:07:17.273678] D [master(worker
>>>     /rhgs/brick20/brick):104:gmaster_builder] <top>: setting up
>>>     change detection mode mode=changelog
>>>
>>>     [2020-06-25 07:07:17.274224] D [master(worker
>>>     /rhgs/brick20/brick):104:gmaster_builder] <top>: setting up
>>>     change detection mode mode=changeloghistory
>>>
>>>     [2020-06-25 07:07:17.276484] D [repce(worker
>>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>>     6799:140380783982400:1593068837.28 version() ...
>>>
>>>     [2020-06-25 07:07:17.276916] D [repce(worker
>>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>>     6799:140380783982400:1593068837.28 version -> 1.0
>>>
>>>     [2020-06-25 07:07:17.277009] D [master(worker
>>>     /rhgs/brick20/brick):777:setup_working_dir] _GMaster: changelog
>>>     working dir
>>>    
>/var/lib/misc/gluster/gsyncd/prd_mx_intvol_bxts470190_prd_mx_intvol/rhgs-brick20-brick
>>>
>>>     [2020-06-25 07:07:17.277098] D [repce(worker
>>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>>     6799:140380783982400:1593068837.28 init() ...
>>>
>>>     [2020-06-25 07:07:17.292944] D [repce(worker
>>>     /rhgs/brick20/brick):215:__call__] RepceClient: call
>>>     6799:140380783982400:1593068837.28 init -> None
>>>
>>>     [2020-06-25 07:07:17.293097] D [repce(worker
>>>     /rhgs/brick20/brick):195:push] RepceClient: call
>>>     6799:140380783982400:1593068837.29
>>>     register('/rhgs/brick20/brick',
>>>    
>'/var/lib/misc/gluster/gsyncd/prd_mx_intvol_bxts470190_prd_mx_intvol/rhgs-brick20-brick',
>>>    
>'/var/log/glusterfs/geo-replication/prd_mx_intvol_bxts470190_prd_mx_intvol/changes-rhgs-brick20-brick.log',
>>>     8, 5) ...
>>>
>>>     [2020-06-25 07:07:19.296294] E [repce(agent
>>>     /rhgs/brick20/brick):121:worker] <top>: call failed:
>>>
>>>     Traceback (most recent call last):
>>>
>>>     ? File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line
>>>     117, in worker
>>>
>>>     ??? res = getattr(self.obj, rmeth)(*in_data[2:])
>>>
>>>     ? File
>>>     "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py",
>>>     line 40, in register
>>>
>>>     ??? return Changes.cl_register(cl_brick, cl_dir, cl_log,
>>>     cl_level, retries)
>>>
>>>     ? File
>>>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
>>>     line 46, in cl_register
>>>
>>>     ??? cls.raise_changelog_err()
>>>
>>>     ? File
>>>     "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
>>>     line 30, in raise_changelog_err
>>>
>>>     ??? raise ChangelogException(errn, os.strerror(errn))
>>>
>>>     ChangelogException: [Errno 2] No such file or directory
>>>
>>>     [2020-06-25 07:07:19.297161] E [repce(worker
>>>     /rhgs/brick20/brick):213:__call__] RepceClient: call failed
>>>     call=6799:140380783982400:1593068837.29 method=register
>>>     error=ChangelogException
>>>
>>>     [2020-06-25 07:07:19.297338] E [resource(worker
>>>     /rhgs/brick20/brick):1286:service_loop] GLUSTER: Changelog
>>>     register failed????? error=[Errno 2] No such file or directory
>>>
>>>     [2020-06-25 07:07:19.315074] I [repce(agent
>>>     /rhgs/brick20/brick):96:service_loop] RepceServer: terminating
>on
>>>     reaching EOF.
>>>
>>>     [2020-06-25 07:07:20.275701] I [monitor(monitor):280:monitor]
>>>     Monitor: worker died in startup phase????
>brick=/rhgs/brick20/brick
>>>
>>>     [2020-06-25 07:07:20.277383] I
>>>     [gsyncdstatus(monitor):248:set_worker_status] GeorepStatus:
>>>     Worker Status Change status=Faulty
>>>
>>>     We?ve done everything we can think of, including an ?strace ?f?
>>>     on the pid, and we can?t really find anything. I?m about to lose
>>>     the last of my hair over this, so does anyone have any ideas at
>>>     all? We?ve even removed the entire slave vol and rebuilt it.
>>>
>>>     Thanks
>>>
>>>     Rob
>>>
>>>     *Rob Quagliozzi*
>>>
>>>     *Specialised Application Support*
>>>
>>>
>>>
>>>    
>------------------------------------------------------------------------
>>>     This email (including any attachments to it) is confidential,
>>>     legally privileged, subject to copyright and is sent for the
>>>     personal attention of the intended recipient only. If you have
>>>     received this email in error, please advise us immediately and
>>>     delete it. You are notified that disclosing, copying,
>>>     distributing or taking any action in reliance on the contents of
>>>     this information is strictly prohibited. Although we have taken
>>>     reasonable precautions to ensure no viruses are present in this
>>>     email, we cannot accept responsibility for any loss or damage
>>>     arising from the viruses in this email or attachments. We
>exclude
>>>     any liability for the content of this email, or for the
>>>     consequences of any actions taken on the basis of the
>information
>>>     provided in this email or its attachments, unless that
>>>     information is subsequently confirmed in writing. <#rbnl#1898i>
>>>    
>------------------------------------------------------------------------
>>>
>>>
>>>     ________
>>>
>>>
>>>
>>>     Community Meeting Calendar:
>>>
>>>     Schedule -
>>>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>     Bridge:https://bluejeans.com/441850968
>>>
>>>     Gluster-users mailing list
>>>     Gluster-users at gluster.org  <mailto:Gluster-users at gluster.org>
>>>     https://lists.gluster.org/mailman/listinfo/gluster-users
>>     ________
>>
>>
>>
>>     Community Meeting Calendar:
>>
>>     Schedule -
>>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>     Bridge: https://bluejeans.com/441850968
>>
>>     Gluster-users mailing list
>>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>     https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users

From kkeithle at redhat.com  Fri Jul  3 13:03:45 2020
From: kkeithle at redhat.com (Kaleb Keithley)
Date: Fri, 3 Jul 2020 09:03:45 -0400
Subject: [Gluster-users] Official Bugzilla?
In-Reply-To: <bea778af-c1e1-dde5-b0e3-9d01ae3b6185@yahoo.co.uk>
References: <bea778af-c1e1-dde5-b0e3-9d01ae3b6185.ref@yahoo.co.uk>
	<bea778af-c1e1-dde5-b0e3-9d01ae3b6185@yahoo.co.uk>
Message-ID: <CAC+Jd5DP5qX8xju8QRQtuzFr__ZN8gfG2gaab32kEu7PzaQCTg@mail.gmail.com>

On Fri, Jul 3, 2020 at 6:46 AM lejeczek <peljasz at yahoo.co.uk> wrote:

> hi guys,
>
> where those of use who run gluster from(via) EPEL repo
>

glusterfs rpms aren't in EPEL, and haven't been for something like six or
seven years.

Perhaps you meant from the CentOS Storage SIG repo?


> should go to report bugs?
>

https://github.com/gluster/glusterfs/issues

If you're really still running glusterfs from old EPEL RPMs then you're
running a very very old version of gluster and you should upgrade to
something more recent. I can pretty much guarantee any bugs in something
that old have been fixed in a newer version.




>
> many thanks, L.
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200703/a1b48e2a/attachment.html>

From shreyansh.shah at alpha-grep.com  Mon Jul  6 08:41:04 2020
From: shreyansh.shah at alpha-grep.com (Shreyansh Shah)
Date: Mon, 6 Jul 2020 14:11:04 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
Message-ID: <CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>

Hi,
Did anyone get a chance to look into this?

On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <shreyansh.shah at alpha-grep.com>
wrote:

> Hi All,
>
> *We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.*
>
> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
> on each node, 7 nodes in total. We added new bricks yesterday to the
> existing setup.
> Post that we did a rebalance fix-layout and then a rebalance (which is
> currently still in progress). The status shows "failed" on certain bricks
> but "in progress" for others. Adding output for gluster rebalance status
> below.
>
> The glusterfs client logs are flooded with "Mismatching layouts for
> <directory>,gfid = <id>"
> The performance too seems to have degraded due to this, even basic
> commands like `cd` and `ls` are taking more than a minute compared to
> sub-second number before brick addition.
> Apart from that we also experienced many binaries and files giving error
> stale file handle error even though the files were present.
>
>
> *gluster rebalance status :*
>
> Node Rebalanced-files          size       scanned      failures
> skipped               status  run time in h:m:s
> ---------      -----------   -----------   -----------   -----------
> -----------         ------------     --------------
> localhost              176         3.5GB         12790             0
>    8552          in progress       21:36:01
> 10.132.0.72             8232       394.8GB         19995            21
>        26               failed       14:50:30
> 10.132.0.44            12625         1.0TB         50023             1
>     10202          in progress       21:36:00
> 10.132.0.3            21982       956.8GB         79145             1
>     34571          in progress       21:36:00
> 10.132.0.9             7975       355.8GB         20157             6
>      1522               failed       14:51:45
> 10.132.0.73             6293       394.5GB         26414           151
>      8085               failed       14:51:45
> 10.132.0.70             6480       477.1GB         21058            27
>      1787               failed       14:50:32
> Estimated time left for rebalance to complete :      130:56:28
>
>
> *Logs from one of the clients below:*
>
> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
> layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
> layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
> layout - 2454306944 - 2761060379 - 4159036738
> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
> layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
> layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
> layout - 920400036 - 1227153471 - 4159036738
> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
> layout - 1840730208 - 2147483643 - 4159036738
> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
> layout - 1533976772 - 1840730207 - 4159036738
> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
> layout - 613646600 - 920400035 - 4159036738
> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
> data-client-7 (hashed subvol is data-client-17)
> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_1_DATA.dat
> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
> on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_1_METADATA.dat
> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
> data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_2_DATA.dat
> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
> on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_2_METADATA.dat
> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
> data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_2_DATA.dat
> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
> on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_2_METADATA.dat
> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
> data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_3_DATA.dat
> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
> on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_3_METADATA.dat
> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
> data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_4_DATA.dat
> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
> on data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_4_METADATA.dat
> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
> data-client-14 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_6_DATA.dat
> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
> on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_6_METADATA.dat
> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
> on data-client-15 (hashed subvol is data-client-8)
> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECDS1_DATA.dat
> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
> (hashed subvol is data-client-6)
> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
> data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM1_DATA.dat
> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
> subvol is data-client-6)
> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM1_METADATA.dat
> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
> data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM2_DATA.dat
> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM2_METADATA.dat
> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
> data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM3_DATA.dat
> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM3_METADATA.dat
> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
> subvol is data-client-10)
> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
> (hashed subvol is data-client-8)
> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
> (hashed subvol is data-client-2)
> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
> subvol is data-client-18)
> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> on data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
> data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
> subvol is data-client-18)
> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
> (hashed subvol is data-client-19)
> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
> 613576736 - 920330171 - 4159036738
> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
> 920330172 - 1227083607 - 4159036738
> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
> 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX, gfid =
> fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
> 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX, gfid =
> fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
> layout - 3067883680 - 3374637115 - 4159036738
> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
> - 306753435 - 4159036738
> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
> layout - 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
> 3681460416 - 3988213851 - 4159036738
> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
> 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
> (2cb54500-814d-4e85-83e7-e33d9440b18d)
> (hash=data-client-6/cache=data-client-18) =>
> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
> (hash=data-client-6/cache=<nul>)
> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
> (99938ee6-6986-4123-9d72-ec09e2310b4f)
> (hash=data-client-17/cache=data-client-18) =>
> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
> (hash=data-client-17/cache=<nul>)
> ....
>
>
> Please look into this at top-priority if possible.
> Let me know if anything else is required.
>
>
> --
> Regards,
> Shreyansh Shah
>


-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200706/3f8adf21/attachment.html>

From bsasonro at redhat.com  Mon Jul  6 17:13:34 2020
From: bsasonro at redhat.com (Barak Sason Rofman)
Date: Mon, 6 Jul 2020 20:13:34 +0300
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
Message-ID: <CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>

Greetings Shreyansh,

Off-hand I can't come up with a reason for these failures.
In order to start looking into this, access to the full rebalance logs is
required (possibly brick logs as well).
Can you provide those?

My regards,


On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
shreyansh.shah at alpha-grep.com> wrote:

> Hi,
> Did anyone get a chance to look into this?
>
> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
> shreyansh.shah at alpha-grep.com> wrote:
>
>> Hi All,
>>
>> *We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.*
>>
>> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
>> on each node, 7 nodes in total. We added new bricks yesterday to the
>> existing setup.
>> Post that we did a rebalance fix-layout and then a rebalance (which is
>> currently still in progress). The status shows "failed" on certain bricks
>> but "in progress" for others. Adding output for gluster rebalance status
>> below.
>>
>> The glusterfs client logs are flooded with "Mismatching layouts for
>> <directory>,gfid = <id>"
>> The performance too seems to have degraded due to this, even basic
>> commands like `cd` and `ls` are taking more than a minute compared to
>> sub-second number before brick addition.
>> Apart from that we also experienced many binaries and files giving error
>> stale file handle error even though the files were present.
>>
>>
>> *gluster rebalance status :*
>>
>> Node Rebalanced-files          size       scanned      failures
>> skipped               status  run time in h:m:s
>> ---------      -----------   -----------   -----------   -----------
>> -----------         ------------     --------------
>> localhost              176         3.5GB         12790             0
>>      8552          in progress       21:36:01
>> 10.132.0.72             8232       394.8GB         19995            21
>>          26               failed       14:50:30
>> 10.132.0.44            12625         1.0TB         50023             1
>>       10202          in progress       21:36:00
>> 10.132.0.3            21982       956.8GB         79145             1
>>     34571          in progress       21:36:00
>> 10.132.0.9             7975       355.8GB         20157             6
>>      1522               failed       14:51:45
>> 10.132.0.73             6293       394.5GB         26414           151
>>        8085               failed       14:51:45
>> 10.132.0.70             6480       477.1GB         21058            27
>>        1787               failed       14:50:32
>> Estimated time left for rebalance to complete :      130:56:28
>>
>>
>> *Logs from one of the clients below:*
>>
>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>> layout - 2761060380 - 3067813815 - 4159036738
>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>> layout - 3681390552 - 3988143987 - 4159036738
>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>> layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>> layout - 2454306944 - 2761060379 - 4159036738
>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>> layout - 2761060380 - 3067813815 - 4159036738
>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>> layout - 3681390552 - 3988143987 - 4159036738
>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>> layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>> layout - 920400036 - 1227153471 - 4159036738
>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>> layout - 1840730208 - 2147483643 - 4159036738
>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>> layout - 1533976772 - 1840730207 - 4159036738
>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>> layout - 613646600 - 920400035 - 4159036738
>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>> data-client-7 (hashed subvol is data-client-17)
>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>> on data-client-9 (hashed subvol is data-client-19)
>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>> data-client-17 (hashed subvol is data-client-9)
>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>> on data-client-9 (hashed subvol is data-client-18)
>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>> data-client-6 (hashed subvol is data-client-15)
>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>> on data-client-11 (hashed subvol is data-client-3)
>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>> data-client-15 (hashed subvol is data-client-7)
>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>> on data-client-7 (hashed subvol is data-client-16)
>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>> data-client-2 (hashed subvol is data-client-11)
>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>> on data-client-15 (hashed subvol is data-client-7)
>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>> data-client-14 (hashed subvol is data-client-7)
>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>> on data-client-10 (hashed subvol is data-client-2)
>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>> on data-client-15 (hashed subvol is data-client-8)
>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>> (hashed subvol is data-client-6)
>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>> data-client-17 (hashed subvol is data-client-9)
>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>> subvol is data-client-6)
>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>> data-client-2 (hashed subvol is data-client-11)
>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>> subvol is data-client-15)
>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>> data-client-17 (hashed subvol is data-client-10)
>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>> subvol is data-client-15)
>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>> on data-client-9 (hashed subvol is data-client-19)
>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>> subvol is data-client-10)
>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>> on data-client-11 (hashed subvol is data-client-3)
>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>> (hashed subvol is data-client-8)
>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>> on data-client-9 (hashed subvol is data-client-18)
>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>> (hashed subvol is data-client-2)
>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>> on data-client-10 (hashed subvol is data-client-2)
>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>> subvol is data-client-18)
>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>> on data-client-2 (hashed subvol is data-client-11)
>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>> on data-client-7 (hashed subvol is data-client-16)
>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>> subvol is data-client-15)
>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>> data-client-17 (hashed subvol is data-client-10)
>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>> subvol is data-client-18)
>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>> deletion of stale linkfile
>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>> (hashed subvol is data-client-19)
>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>> returned with op_ret -> 0 and op-errno -> 0 for
>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>> 613576736 - 920330171 - 4159036738
>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>> 920330172 - 1227083607 - 4159036738
>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>> 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX, gfid =
>> fff324f2-f855-4881-b77c-81e856522373
>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>> 3681390552 - 3988143987 - 4159036738
>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX, gfid =
>> fff324f2-f855-4881-b77c-81e856522373
>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>> layout - 3067883680 - 3374637115 - 4159036738
>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>> f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>> layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>> f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>> - 306753435 - 4159036738
>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>> f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>> layout - 3988213852 - 4294967295 - 4159036738
>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>> f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>> 3681460416 - 3988213851 - 4159036738
>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>> 3988213852 - 4294967295 - 4159036738
>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>> (hash=data-client-6/cache=data-client-18) =>
>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>> (hash=data-client-6/cache=<nul>)
>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>> (hash=data-client-17/cache=data-client-18) =>
>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>> (hash=data-client-17/cache=<nul>)
>> ....
>>
>>
>> Please look into this at top-priority if possible.
>> Let me know if anything else is required.
>>
>>
>> --
>> Regards,
>> Shreyansh Shah
>>
>
>
> --
> Regards,
> Shreyansh Shah
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>


-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel <https://www.redhat.com/>

34 Jerusalem rd. Ra'anana, 43501

bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
M: *+972-52-4326355*
@RedHat <https://twitter.com/redhat>   Red Hat
<https://www.linkedin.com/company/red-hat>  Red Hat
<https://www.facebook.com/redhat.il/>
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200706/48371bd6/attachment.html>

From bsasonro at redhat.com  Mon Jul  6 17:38:16 2020
From: bsasonro at redhat.com (Barak Sason Rofman)
Date: Mon, 6 Jul 2020 20:38:16 +0300
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
	<CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>
	<CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
Message-ID: <CAFx2kQ3H9A4fbKbLhd1iCcJ-k8M4QGj+tyGumPC1sx-VBkzADA@mail.gmail.com>

I think it would be best.
As I can't say at this point where the problem is originating from, brick
logs might also be necessary (I assume I would have a better picture once I
have the rebalance logs).

Cheers,

On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <shreyansh.shah at alpha-grep.com>
wrote:

> Hi Barak,
> Can provide the rebalance logs. Do you require all the brick logs (14 in
> total)?
>
> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <bsasonro at redhat.com>
> wrote:
>
>> Greetings Shreyansh,
>>
>> Off-hand I can't come up with a reason for these failures.
>> In order to start looking into this, access to the full rebalance logs is
>> required (possibly brick logs as well).
>> Can you provide those?
>>
>> My regards,
>>
>>
>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>> shreyansh.shah at alpha-grep.com> wrote:
>>
>>> Hi,
>>> Did anyone get a chance to look into this?
>>>
>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>> shreyansh.shah at alpha-grep.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>"
>>>> errors.*
>>>>
>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB
>>>> each) on each node, 7 nodes in total. We added new bricks yesterday to the
>>>> existing setup.
>>>> Post that we did a rebalance fix-layout and then a rebalance (which is
>>>> currently still in progress). The status shows "failed" on certain bricks
>>>> but "in progress" for others. Adding output for gluster rebalance status
>>>> below.
>>>>
>>>> The glusterfs client logs are flooded with "Mismatching layouts for
>>>> <directory>,gfid = <id>"
>>>> The performance too seems to have degraded due to this, even basic
>>>> commands like `cd` and `ls` are taking more than a minute compared to
>>>> sub-second number before brick addition.
>>>> Apart from that we also experienced many binaries and files giving
>>>> error stale file handle error even though the files were present.
>>>>
>>>>
>>>> *gluster rebalance status :*
>>>>
>>>> Node Rebalanced-files          size       scanned      failures
>>>> skipped               status  run time in h:m:s
>>>> ---------      -----------   -----------   -----------   -----------
>>>> -----------         ------------     --------------
>>>> localhost              176         3.5GB         12790             0
>>>>        8552          in progress       21:36:01
>>>> 10.132.0.72             8232       394.8GB         19995            21
>>>>            26               failed       14:50:30
>>>> 10.132.0.44            12625         1.0TB         50023             1
>>>>         10202          in progress       21:36:00
>>>> 10.132.0.3            21982       956.8GB         79145             1
>>>>       34571          in progress       21:36:00
>>>> 10.132.0.9             7975       355.8GB         20157             6
>>>>        1522               failed       14:51:45
>>>> 10.132.0.73             6293       394.5GB         26414           151
>>>>          8085               failed       14:51:45
>>>> 10.132.0.70             6480       477.1GB         21058            27
>>>>          1787               failed       14:50:32
>>>> Estimated time left for rebalance to complete :      130:56:28
>>>>
>>>>
>>>> *Logs from one of the clients below:*
>>>>
>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>>>> layout - 920400036 - 1227153471 - 4159036738
>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>>>> layout - 613646600 - 920400035 - 4159036738
>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>> data-client-7 (hashed subvol is data-client-17)
>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>> on data-client-9 (hashed subvol is data-client-19)
>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>> data-client-17 (hashed subvol is data-client-9)
>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>> on data-client-9 (hashed subvol is data-client-18)
>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>> data-client-6 (hashed subvol is data-client-15)
>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>> on data-client-11 (hashed subvol is data-client-3)
>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>> data-client-15 (hashed subvol is data-client-7)
>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>> on data-client-7 (hashed subvol is data-client-16)
>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>> data-client-2 (hashed subvol is data-client-11)
>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>> on data-client-15 (hashed subvol is data-client-7)
>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>> data-client-14 (hashed subvol is data-client-7)
>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>> on data-client-10 (hashed subvol is data-client-2)
>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>> on data-client-15 (hashed subvol is data-client-8)
>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>>>> (hashed subvol is data-client-6)
>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>> data-client-17 (hashed subvol is data-client-9)
>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>>>> subvol is data-client-6)
>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>> data-client-2 (hashed subvol is data-client-11)
>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>>>> subvol is data-client-15)
>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>> data-client-17 (hashed subvol is data-client-10)
>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>>>> subvol is data-client-15)
>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>> on data-client-9 (hashed subvol is data-client-19)
>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>>>> subvol is data-client-10)
>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>> on data-client-11 (hashed subvol is data-client-3)
>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>>>> (hashed subvol is data-client-8)
>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>> on data-client-9 (hashed subvol is data-client-18)
>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>>>> (hashed subvol is data-client-2)
>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>> on data-client-10 (hashed subvol is data-client-2)
>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>>>> subvol is data-client-18)
>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>> on data-client-2 (hashed subvol is data-client-11)
>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>> on data-client-7 (hashed subvol is data-client-16)
>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>>>> subvol is data-client-15)
>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>> data-client-17 (hashed subvol is data-client-10)
>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>>>> subvol is data-client-18)
>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>> deletion of stale linkfile
>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>>>> (hashed subvol is data-client-19)
>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>>>> 613576736 - 920330171 - 4159036738
>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>>>> 920330172 - 1227083607 - 4159036738
>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>>>> 3374637116 - 3681390551 - 4159036738
>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>> fff324f2-f855-4881-b77c-81e856522373
>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>>>> 3681390552 - 3988143987 - 4159036738
>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>> fff324f2-f855-4881-b77c-81e856522373
>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>> f8447150-4801-4188-add9-ea295bb88729
>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>> f8447150-4801-4188-add9-ea295bb88729
>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>>>> - 306753435 - 4159036738
>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>> f8447150-4801-4188-add9-ea295bb88729
>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>> f8447150-4801-4188-add9-ea295bb88729
>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>>>> 3681460416 - 3988213851 - 4159036738
>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>>>> 3988213852 - 4294967295 - 4159036738
>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>> (hash=data-client-6/cache=data-client-18) =>
>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>> (hash=data-client-6/cache=<nul>)
>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>> (hash=data-client-17/cache=data-client-18) =>
>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>> (hash=data-client-17/cache=<nul>)
>>>> ....
>>>>
>>>>
>>>> Please look into this at top-priority if possible.
>>>> Let me know if anything else is required.
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shreyansh Shah
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shreyansh Shah
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage Development
>>
>> Red Hat Israel <https://www.redhat.com/>
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>> M: *+972-52-4326355*
>> @RedHat <https://twitter.com/redhat>   Red Hat
>> <https://www.linkedin.com/company/red-hat>  Red Hat
>> <https://www.facebook.com/redhat.il/>
>> <https://red.ht/sig>
>>
>
>
> --
> Regards,
> Shreyansh Shah
>


-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel <https://www.redhat.com/>

34 Jerusalem rd. Ra'anana, 43501

bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
M: *+972-52-4326355*
@RedHat <https://twitter.com/redhat>   Red Hat
<https://www.linkedin.com/company/red-hat>  Red Hat
<https://www.facebook.com/redhat.il/>
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200706/cbc2fd14/attachment.html>

From shanondink at gmail.com  Mon Jul  6 20:32:28 2020
From: shanondink at gmail.com (Shanon Swafford)
Date: Mon, 6 Jul 2020 15:32:28 -0500
Subject: [Gluster-users] Restore a replica after failed hardware
Message-ID: <008b01d653d4$91285500$b378ff00$@gmail.com>

Hi guys,

 

I lost a brick in a 2x replicated system.  The volume is 17TB with 9TB used
( small files ).  3 drives failed in 2 hours in a raid-5 array.

 

Gluster version: 3.8.15

 

So "reset-brick" isn't available on this version.

 

I've googled all weekend and I'm overwhelmed so I'd like to verify before I
muck everything up.

 

Is this the correct procedure to restore the failed brick?

 

# Replace drive

# Use parted to create /dev/sdb1

# Make xfs filesystem on /dev/sdb1

# Mount /var/glusterfs/sdb1

# gluster volume replace-brick myvol d-es2-nfs-a:/var/glusterfs/sdb1/myvol
d-es2-nfs-a:/var/glusterfs/sdb1/myvol commit force 

 

I read about using different brick names but again, I'm overwhelmed with all
the info on google.

 

I also saw something as simple as remove failed and re-add as new but..

 

Now I just read about xfsdump | xfsrestore to preload, but how would that
work with healing?

 

Thanks a ton in advance.

 

Shanon

 

 

 

[root at es2-nfs-a ~]# parted /dev/sdb print

Error: /dev/sdb: unrecognised disk label

Model: DELL PERC H700 (scsi)

Disk /dev/sdb: 18.2TB

Sector size (logical/physical): 512B/512B

Partition Table: unknown

Disk Flags:

 

 

[root at es2-nfs-a ~]# grep sdb /etc/fstab

/dev/sdb1               /var/glusterfs/sdb1     xfs     inode64         0 0

 

 

[root at es2-nfs-a ~]# gluster volume info

 

Volume Name: myvol

Type: Replicate

Volume ID: 49fd2a63-f887-4478-9242-69030a7a565d

Status: Started

Snapshot Count: 0

Number of Bricks: 1 x 2 = 2

Transport-type: tcp

Bricks:

Brick1: d-es2-nfs-a:/var/glusterfs/sdb1/myvol

Brick2: d-es2-nfs-b:/var/glusterfs/sdb1/myvol

Options Reconfigured:

nfs.disable: on

performance.readdir-ahead: on

transport.address-family: inet

performance.cache-size: 1GB

 

 

[root at es2-nfs-a ~]# gluster volume status

Status of volume: myvol

Gluster process                             TCP Port  RDMA Port  Online  Pid

----------------------------------------------------------------------------
--

Brick d-es2-nfs-a:/var/glusterfs/sdb1/myvol

Brick d-es2-nfs-b:/var/glusterfs/sdb1/myvol

Self-heal Daemon on localhost               N/A       N/A        Y
2475

Self-heal Daemon on d-es2-nfs-b             N/A       N/A        Y
8663

 

Task Status of Volume myvol

----------------------------------------------------------------------------
--

There are no active volume tasks

 

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200706/a9ba1c68/attachment.html>

From hunter86_bg at yahoo.com  Tue Jul  7 04:02:29 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Tue, 07 Jul 2020 07:02:29 +0300
Subject: [Gluster-users] Restore a replica after failed hardware
In-Reply-To: <008b01d653d4$91285500$b378ff00$@gmail.com>
References: <008b01d653d4$91285500$b378ff00$@gmail.com>
Message-ID: <E1328BE8-7C22-492C-9A97-822C6180350A@yahoo.com>

It looks  OK.
Usually in the doc you got 'replace-brick source  destination start' which stops  the brick process  and I do it this way.

Also, you should consider:
- adding 'noatime' to the mount options
- Check what is your stripe width (stride multiplied by data disks)  and then create xfs  with the necessary options to align it properly.

I have never used xfsdump to recover a brick. Just ensure gluster brick process is not running on the node during the restore.

Best Regards,
Strahil Nikolov


?? 6 ??? 2020 ?. 23:32:28 GMT+03:00, Shanon Swafford <shanondink at gmail.com> ??????:
>Hi guys,
>
> 
>
>I lost a brick in a 2x replicated system.  The volume is 17TB with 9TB
>used
>( small files ).  3 drives failed in 2 hours in a raid-5 array.
>
> 
>
>Gluster version: 3.8.15
>
> 
>
>So "reset-brick" isn't available on this version.
>
> 
>
>I've googled all weekend and I'm overwhelmed so I'd like to verify
>before I
>muck everything up.
>
> 
>
>Is this the correct procedure to restore the failed brick?
>
> 
>
># Replace drive
>
># Use parted to create /dev/sdb1
>
># Make xfs filesystem on /dev/sdb1
>
># Mount /var/glusterfs/sdb1
>
># gluster volume replace-brick myvol
>d-es2-nfs-a:/var/glusterfs/sdb1/myvol
>d-es2-nfs-a:/var/glusterfs/sdb1/myvol commit force 
>
> 
>
>I read about using different brick names but again, I'm overwhelmed
>with all
>the info on google.
>
> 
>
>I also saw something as simple as remove failed and re-add as new but..
>
> 
>
>Now I just read about xfsdump | xfsrestore to preload, but how would
>that
>work with healing?
>
> 
>
>Thanks a ton in advance.
>
> 
>
>Shanon
>
> 
>
> 
>
> 
>
>[root at es2-nfs-a ~]# parted /dev/sdb print
>
>Error: /dev/sdb: unrecognised disk label
>
>Model: DELL PERC H700 (scsi)
>
>Disk /dev/sdb: 18.2TB
>
>Sector size (logical/physical): 512B/512B
>
>Partition Table: unknown
>
>Disk Flags:
>
> 
>
> 
>
>[root at es2-nfs-a ~]# grep sdb /etc/fstab
>
>/dev/sdb1               /var/glusterfs/sdb1     xfs     inode64        
>0 0
>
> 
>
> 
>
>[root at es2-nfs-a ~]# gluster volume info
>
> 
>
>Volume Name: myvol
>
>Type: Replicate
>
>Volume ID: 49fd2a63-f887-4478-9242-69030a7a565d
>
>Status: Started
>
>Snapshot Count: 0
>
>Number of Bricks: 1 x 2 = 2
>
>Transport-type: tcp
>
>Bricks:
>
>Brick1: d-es2-nfs-a:/var/glusterfs/sdb1/myvol
>
>Brick2: d-es2-nfs-b:/var/glusterfs/sdb1/myvol
>
>Options Reconfigured:
>
>nfs.disable: on
>
>performance.readdir-ahead: on
>
>transport.address-family: inet
>
>performance.cache-size: 1GB
>
> 
>
> 
>
>[root at es2-nfs-a ~]# gluster volume status
>
>Status of volume: myvol
>
>Gluster process                             TCP Port  RDMA Port  Online
> Pid
>
>----------------------------------------------------------------------------
>--
>
>Brick d-es2-nfs-a:/var/glusterfs/sdb1/myvol
>
>Brick d-es2-nfs-b:/var/glusterfs/sdb1/myvol
>
>Self-heal Daemon on localhost               N/A       N/A        Y
>2475
>
>Self-heal Daemon on d-es2-nfs-b             N/A       N/A        Y
>8663
>
> 
>
>Task Status of Volume myvol
>
>----------------------------------------------------------------------------
>--
>
>There are no active volume tasks
>
> 
>
> 

From bsasonro at redhat.com  Tue Jul  7 08:42:36 2020
From: bsasonro at redhat.com (Barak Sason Rofman)
Date: Tue, 7 Jul 2020 11:42:36 +0300
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+7A4o1mXeHsje-jCkpsPsS0AJcTn5V5dDRsw-7DLw358Q@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
	<CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>
	<CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
	<CAFx2kQ3H9A4fbKbLhd1iCcJ-k8M4QGj+tyGumPC1sx-VBkzADA@mail.gmail.com>
	<CAH1hu+52marGwdRmpxBKJweLW6e3J-uLQSVKRyKRP_3PZ=0FoA@mail.gmail.com>
	<CAFx2kQ18jJ4Zb6iy0TB=Bmk6BQ5azhywcM3g4c1dX4SttddgZA@mail.gmail.com>
	<CAH1hu+7A4o1mXeHsje-jCkpsPsS0AJcTn5V5dDRsw-7DLw358Q@mail.gmail.com>
Message-ID: <CAFx2kQ0RguVjus1JggwyZNKTJLv-zWFc7zHTM2fkyVSxoU_=0Q@mail.gmail.com>

Thanks Shreyansh,

I'll look into it, however I'll likely need some help from more senior team
members to perform RCA.
I'll update once I have new insights.

My regards,

On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <
shreyansh.shah at alpha-grep.com> wrote:

> Hi Barak,
> Thanks for looking into this and helping me out,
> The fix-layout was successful, and I ran a rebalance after completion of
> fix-layout.
> The rebalance status though did show failure for 3 nodes.
>
> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com>
> wrote:
>
>> Greetings again Shreyansh,
>>
>> I'm indeed seeing a lot of errors in the log file - still unsure about
>> the RC.
>> You mentioned that prior to running rebalance you ran fix-layout, was the
>> fix-layout successful?
>> Another question - did you wait until fix-layout was completed before
>> running rebalance?
>>
>> My thanks,
>>
>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <
>> shreyansh.shah at alpha-grep.com> wrote:
>>
>>> Hi,
>>> Attaching rebalance logs
>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster
>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout"
>>>
>>>
>>> PFA
>>>
>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <bsasonro at redhat.com>
>>> wrote:
>>>
>>>> I think it would be best.
>>>> As I can't say at this point where the problem is originating from,
>>>> brick logs might also be necessary (I assume I would have a better picture
>>>> once I have the rebalance logs).
>>>>
>>>> Cheers,
>>>>
>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <
>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>
>>>>> Hi Barak,
>>>>> Can provide the rebalance logs. Do you require all the brick logs (14
>>>>> in total)?
>>>>>
>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <
>>>>> bsasonro at redhat.com> wrote:
>>>>>
>>>>>> Greetings Shreyansh,
>>>>>>
>>>>>> Off-hand I can't come up with a reason for these failures.
>>>>>> In order to start looking into this, access to the full rebalance
>>>>>> logs is required (possibly brick logs as well).
>>>>>> Can you provide those?
>>>>>>
>>>>>> My regards,
>>>>>>
>>>>>>
>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> Did anyone get a chance to look into this?
>>>>>>>
>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>
>>>>>>>> Hi All,
>>>>>>>>
>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>"
>>>>>>>> errors.*
>>>>>>>>
>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB
>>>>>>>> each) on each node, 7 nodes in total. We added new bricks yesterday to the
>>>>>>>> existing setup.
>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance (which
>>>>>>>> is currently still in progress). The status shows "failed" on certain
>>>>>>>> bricks but "in progress" for others. Adding output for gluster rebalance
>>>>>>>> status below.
>>>>>>>>
>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts for
>>>>>>>> <directory>,gfid = <id>"
>>>>>>>> The performance too seems to have degraded due to this, even basic
>>>>>>>> commands like `cd` and `ls` are taking more than a minute compared to
>>>>>>>> sub-second number before brick addition.
>>>>>>>> Apart from that we also experienced many binaries and files giving
>>>>>>>> error stale file handle error even though the files were present.
>>>>>>>>
>>>>>>>>
>>>>>>>> *gluster rebalance status :*
>>>>>>>>
>>>>>>>> Node Rebalanced-files          size       scanned      failures
>>>>>>>>   skipped               status  run time in h:m:s
>>>>>>>> ---------      -----------   -----------   -----------
>>>>>>>> -----------   -----------         ------------     --------------
>>>>>>>> localhost              176         3.5GB         12790
>>>>>>>> 0          8552          in progress       21:36:01
>>>>>>>> 10.132.0.72             8232       394.8GB         19995
>>>>>>>>  21            26               failed       14:50:30
>>>>>>>> 10.132.0.44            12625         1.0TB         50023
>>>>>>>>   1         10202          in progress       21:36:00
>>>>>>>> 10.132.0.3            21982       956.8GB         79145
>>>>>>>> 1         34571          in progress       21:36:00
>>>>>>>> 10.132.0.9             7975       355.8GB         20157
>>>>>>>> 6          1522               failed       14:51:45
>>>>>>>> 10.132.0.73             6293       394.5GB         26414
>>>>>>>> 151          8085               failed       14:51:45
>>>>>>>> 10.132.0.70             6480       477.1GB         21058
>>>>>>>>  27          1787               failed       14:50:32
>>>>>>>> Estimated time left for rebalance to complete :      130:56:28
>>>>>>>>
>>>>>>>>
>>>>>>>> *Logs from one of the clients below:*
>>>>>>>>
>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>>>>>>>> layout - 920400036 - 1227153471 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>>>>>>>> layout - 613646600 - 920400035 - 4159036738
>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>>>>>> data-client-7 (hashed subvol is data-client-17)
>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>>>>>> data-client-6 (hashed subvol is data-client-15)
>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>>>>>> data-client-15 (hashed subvol is data-client-7)
>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>> on data-client-15 (hashed subvol is data-client-7)
>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>>>>>> data-client-14 (hashed subvol is data-client-7)
>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>> on data-client-15 (hashed subvol is data-client-8)
>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>>>>>>>> (hashed subvol is data-client-6)
>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>>>>>>>> subvol is data-client-6)
>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>>>>>>>> subvol is data-client-15)
>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>>>>>>>> subvol is data-client-15)
>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>>>>>>>> subvol is data-client-10)
>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>>>>>>>> (hashed subvol is data-client-8)
>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>>>>>>>> (hashed subvol is data-client-2)
>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>>>>>>>> subvol is data-client-18)
>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>> on data-client-2 (hashed subvol is data-client-11)
>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>>>>>>>> subvol is data-client-15)
>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>>>>>>>> subvol is data-client-18)
>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>> deletion of stale linkfile
>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>>>>>>>> (hashed subvol is data-client-19)
>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>>>>>>>> 613576736 - 920330171 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>>>>>>>> 920330172 - 1227083607 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>>>>>>>> 3374637116 - 3681390551 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>>>>>>>> 3681390552 - 3988143987 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>>>>>>>> - 306753435 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>>>>>>>> 3681460416 - 3988213851 - 4159036738
>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>>>>>>>> 3988213852 - 4294967295 - 4159036738
>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>>>>>> (hash=data-client-6/cache=data-client-18) =>
>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>>>>>> (hash=data-client-6/cache=<nul>)
>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>>>>>> (hash=data-client-17/cache=data-client-18) =>
>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>>>>>> (hash=data-client-17/cache=<nul>)
>>>>>>>> ....
>>>>>>>>
>>>>>>>>
>>>>>>>> Please look into this at top-priority if possible.
>>>>>>>> Let me know if anything else is required.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shreyansh Shah
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shreyansh Shah
>>>>>>> ________
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Community Meeting Calendar:
>>>>>>>
>>>>>>> Schedule -
>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Barak Sason Rofman*
>>>>>>
>>>>>> Gluster Storage Development
>>>>>>
>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>
>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>
>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>> M: *+972-52-4326355*
>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>> <https://red.ht/sig>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shreyansh Shah
>>>>>
>>>>
>>>>
>>>> --
>>>> *Barak Sason Rofman*
>>>>
>>>> Gluster Storage Development
>>>>
>>>> Red Hat Israel <https://www.redhat.com/>
>>>>
>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>
>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>> M: *+972-52-4326355*
>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>> <https://www.facebook.com/redhat.il/>
>>>> <https://red.ht/sig>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shreyansh Shah
>>>
>>
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage Development
>>
>> Red Hat Israel <https://www.redhat.com/>
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>> M: *+972-52-4326355*
>> @RedHat <https://twitter.com/redhat>   Red Hat
>> <https://www.linkedin.com/company/red-hat>  Red Hat
>> <https://www.facebook.com/redhat.il/>
>> <https://red.ht/sig>
>>
>
>
> --
> Regards,
> Shreyansh Shah
>


-- 
*Barak Sason Rofman*

Gluster Storage Development

Red Hat Israel <https://www.redhat.com/>

34 Jerusalem rd. Ra'anana, 43501

bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
M: *+972-52-4326355*
@RedHat <https://twitter.com/redhat>   Red Hat
<https://www.linkedin.com/company/red-hat>  Red Hat
<https://www.facebook.com/redhat.il/>
<https://red.ht/sig>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/4253c597/attachment.html>

From shreyansh.shah at alpha-grep.com  Tue Jul  7 08:46:52 2020
From: shreyansh.shah at alpha-grep.com (Shreyansh Shah)
Date: Tue, 7 Jul 2020 14:16:52 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAFx2kQ0RguVjus1JggwyZNKTJLv-zWFc7zHTM2fkyVSxoU_=0Q@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
	<CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>
	<CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
	<CAFx2kQ3H9A4fbKbLhd1iCcJ-k8M4QGj+tyGumPC1sx-VBkzADA@mail.gmail.com>
	<CAH1hu+52marGwdRmpxBKJweLW6e3J-uLQSVKRyKRP_3PZ=0FoA@mail.gmail.com>
	<CAFx2kQ18jJ4Zb6iy0TB=Bmk6BQ5azhywcM3g4c1dX4SttddgZA@mail.gmail.com>
	<CAH1hu+7A4o1mXeHsje-jCkpsPsS0AJcTn5V5dDRsw-7DLw358Q@mail.gmail.com>
	<CAFx2kQ0RguVjus1JggwyZNKTJLv-zWFc7zHTM2fkyVSxoU_=0Q@mail.gmail.com>
Message-ID: <CAH1hu+4D4SH1LT0=juPw4QU3zy33HVuOytMKn1mrjx0c6Mdtsg@mail.gmail.com>

Sounds good, thank you.

On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com>
wrote:

> Thanks Shreyansh,
>
> I'll look into it, however I'll likely need some help from more senior
> team members to perform RCA.
> I'll update once I have new insights.
>
> My regards,
>
> On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <
> shreyansh.shah at alpha-grep.com> wrote:
>
>> Hi Barak,
>> Thanks for looking into this and helping me out,
>> The fix-layout was successful, and I ran a rebalance after completion of
>> fix-layout.
>> The rebalance status though did show failure for 3 nodes.
>>
>> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com>
>> wrote:
>>
>>> Greetings again Shreyansh,
>>>
>>> I'm indeed seeing a lot of errors in the log file - still unsure about
>>> the RC.
>>> You mentioned that prior to running rebalance you ran fix-layout, was
>>> the fix-layout successful?
>>> Another question - did you wait until fix-layout was completed before
>>> running rebalance?
>>>
>>> My thanks,
>>>
>>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <
>>> shreyansh.shah at alpha-grep.com> wrote:
>>>
>>>> Hi,
>>>> Attaching rebalance logs
>>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster
>>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout"
>>>>
>>>>
>>>> PFA
>>>>
>>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <bsasonro at redhat.com>
>>>> wrote:
>>>>
>>>>> I think it would be best.
>>>>> As I can't say at this point where the problem is originating from,
>>>>> brick logs might also be necessary (I assume I would have a better picture
>>>>> once I have the rebalance logs).
>>>>>
>>>>> Cheers,
>>>>>
>>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <
>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>
>>>>>> Hi Barak,
>>>>>> Can provide the rebalance logs. Do you require all the brick logs (14
>>>>>> in total)?
>>>>>>
>>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <
>>>>>> bsasonro at redhat.com> wrote:
>>>>>>
>>>>>>> Greetings Shreyansh,
>>>>>>>
>>>>>>> Off-hand I can't come up with a reason for these failures.
>>>>>>> In order to start looking into this, access to the full rebalance
>>>>>>> logs is required (possibly brick logs as well).
>>>>>>> Can you provide those?
>>>>>>>
>>>>>>> My regards,
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>> Did anyone get a chance to look into this?
>>>>>>>>
>>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>
>>>>>>>>> Hi All,
>>>>>>>>>
>>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>"
>>>>>>>>> errors.*
>>>>>>>>>
>>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks
>>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to
>>>>>>>>> the existing setup.
>>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance
>>>>>>>>> (which is currently still in progress). The status shows "failed" on
>>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster
>>>>>>>>> rebalance status below.
>>>>>>>>>
>>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts
>>>>>>>>> for <directory>,gfid = <id>"
>>>>>>>>> The performance too seems to have degraded due to this, even basic
>>>>>>>>> commands like `cd` and `ls` are taking more than a minute compared to
>>>>>>>>> sub-second number before brick addition.
>>>>>>>>> Apart from that we also experienced many binaries and files giving
>>>>>>>>> error stale file handle error even though the files were present.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *gluster rebalance status :*
>>>>>>>>>
>>>>>>>>> Node Rebalanced-files          size       scanned      failures
>>>>>>>>>     skipped               status  run time in h:m:s
>>>>>>>>> ---------      -----------   -----------   -----------
>>>>>>>>> -----------   -----------         ------------     --------------
>>>>>>>>> localhost              176         3.5GB         12790
>>>>>>>>> 0          8552          in progress       21:36:01
>>>>>>>>> 10.132.0.72             8232       394.8GB         19995
>>>>>>>>>  21            26               failed       14:50:30
>>>>>>>>> 10.132.0.44            12625         1.0TB         50023
>>>>>>>>>   1         10202          in progress       21:36:00
>>>>>>>>> 10.132.0.3            21982       956.8GB         79145
>>>>>>>>>   1         34571          in progress       21:36:00
>>>>>>>>> 10.132.0.9             7975       355.8GB         20157
>>>>>>>>>   6          1522               failed       14:51:45
>>>>>>>>> 10.132.0.73             6293       394.5GB         26414
>>>>>>>>> 151          8085               failed       14:51:45
>>>>>>>>> 10.132.0.70             6480       477.1GB         21058
>>>>>>>>>  27          1787               failed       14:50:32
>>>>>>>>> Estimated time left for rebalance to complete :      130:56:28
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Logs from one of the clients below:*
>>>>>>>>>
>>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>>>>>>>>> layout - 920400036 - 1227153471 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>>>>>>>>> layout - 613646600 - 920400035 - 4159036738
>>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>>>>>>> data-client-7 (hashed subvol is data-client-17)
>>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>>>>>>> data-client-6 (hashed subvol is data-client-15)
>>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>>>>>>> data-client-15 (hashed subvol is data-client-7)
>>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>> on data-client-15 (hashed subvol is data-client-7)
>>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>>>>>>> data-client-14 (hashed subvol is data-client-7)
>>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>> on data-client-15 (hashed subvol is data-client-8)
>>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>>>>>>>>> (hashed subvol is data-client-6)
>>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>>>>>>>>> subvol is data-client-6)
>>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>>>>>>>>> subvol is data-client-15)
>>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>>>>>>>>> subvol is data-client-15)
>>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>>>>>>>>> subvol is data-client-10)
>>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>>>>>>>>> (hashed subvol is data-client-8)
>>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>>>>>>>>> (hashed subvol is data-client-2)
>>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>>>>>>>>> subvol is data-client-18)
>>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>> on data-client-2 (hashed subvol is data-client-11)
>>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>>>>>>>>> subvol is data-client-15)
>>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>>>>>>>>> subvol is data-client-18)
>>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>> deletion of stale linkfile
>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>>>>>>>>> (hashed subvol is data-client-19)
>>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>>>>>>>>> 613576736 - 920330171 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>>>>>>>>> 920330172 - 1227083607 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>>>>>>>>> 3374637116 - 3681390551 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>>>>>>>>> 3681390552 - 3988143987 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>>>>>>>>> - 306753435 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>>>>>>>>> 3681460416 - 3988213851 - 4159036738
>>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>>>>>>>>> 3988213852 - 4294967295 - 4159036738
>>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>>>>>>> (hash=data-client-6/cache=data-client-18) =>
>>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>>>>>>> (hash=data-client-6/cache=<nul>)
>>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>>>>>>> (hash=data-client-17/cache=data-client-18) =>
>>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>>>>>>> (hash=data-client-17/cache=<nul>)
>>>>>>>>> ....
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Please look into this at top-priority if possible.
>>>>>>>>> Let me know if anything else is required.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Shreyansh Shah
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shreyansh Shah
>>>>>>>> ________
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Community Meeting Calendar:
>>>>>>>>
>>>>>>>> Schedule -
>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>>
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Barak Sason Rofman*
>>>>>>>
>>>>>>> Gluster Storage Development
>>>>>>>
>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>
>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>
>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>> M: *+972-52-4326355*
>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>> <https://red.ht/sig>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shreyansh Shah
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Barak Sason Rofman*
>>>>>
>>>>> Gluster Storage Development
>>>>>
>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>
>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>
>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>> M: *+972-52-4326355*
>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>> <https://www.facebook.com/redhat.il/>
>>>>> <https://red.ht/sig>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shreyansh Shah
>>>>
>>>
>>>
>>> --
>>> *Barak Sason Rofman*
>>>
>>> Gluster Storage Development
>>>
>>> Red Hat Israel <https://www.redhat.com/>
>>>
>>> 34 Jerusalem rd. Ra'anana, 43501
>>>
>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>> M: *+972-52-4326355*
>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>> <https://www.facebook.com/redhat.il/>
>>> <https://red.ht/sig>
>>>
>>
>>
>> --
>> Regards,
>> Shreyansh Shah
>>
>
>
> --
> *Barak Sason Rofman*
>
> Gluster Storage Development
>
> Red Hat Israel <https://www.redhat.com/>
>
> 34 Jerusalem rd. Ra'anana, 43501
>
> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
> M: *+972-52-4326355*
> @RedHat <https://twitter.com/redhat>   Red Hat
> <https://www.linkedin.com/company/red-hat>  Red Hat
> <https://www.facebook.com/redhat.il/>
> <https://red.ht/sig>
>


-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/6489fa89/attachment.html>

From evilmf at gmail.com  Tue Jul  7 21:46:21 2020
From: evilmf at gmail.com (Marco Fais)
Date: Tue, 7 Jul 2020 22:46:21 +0100
Subject: [Gluster-users] Problems with qemu and disperse volumes (live
	merge)
In-Reply-To: <CCF9A9E3-151F-4D9B-A510-7E2675BD7BF4@yahoo.com>
References: <CAC7CbKH2pgyAKLNfdPvSZSvTL2Dc5USngubNr0BMD573c_VgBQ@mail.gmail.com>
	<F1022CAA-A397-483D-AA12-98F6A20FD591@yahoo.com>
	<CAC7CbKHG4U5sCyru8jN7UgYzb4BbERuJnhsj=mOe15EH78kiJg@mail.gmail.com>
	<93D3EE3B-B5B3-4689-BF66-C1442A03971E@yahoo.com>
	<CAC7CbKHc24RzVgQhRJiKpjjxCLGDjf1rhK-FEHUu4JbYyHjSJw@mail.gmail.com>
	<CCF9A9E3-151F-4D9B-A510-7E2675BD7BF4@yahoo.com>
Message-ID: <CAC7CbKFQP=5WqxE0XEKrE8u2B8oSyDnrad62mpkp+OFMLhh_DQ@mail.gmail.com>

Hi Strahil

first of all thanks a million for your help -- really appreciate it.
Thanks also for the pointers on the debug. I have tried it, and while I
can't interpret the results I think I might have found something.

There is a lot of information so hopefully this is relevant. During the
snapshot creation and deletion, I can see the following errors in the
client log:

[2020-07-07 21:23:06.837381] W [MSGID: 122019]
[ec-helpers.c:401:ec_loc_gfid_check] 0-SSD_Storage-disperse-0: Mismatching
GFID's in loc
[2020-07-07 21:23:06.837387] D [MSGID: 0]
[defaults.c:1328:default_mknod_cbk] 0-stack-trace: stack-address:
0x7f0dc0001a78, SSD_Storage-disperse-0 returned -1 error: Input/output
error [Input/output error]
[2020-07-07 21:23:06.837392] W [MSGID: 109002]
[dht-rename.c:1019:dht_rename_links_create_cbk] 0-SSD_Storage-dht:
link/file
/8d49207e-f6b9-41d1-8d35-f6e0fb121980/images/4802e66e-a7e3-42df-a570-7155135566ad/b51133ee-54e0-4001-ab4b-9f0dc1e5c6fc.meta
on SSD_Storage-disperse-0 failed [Input/output error]
[2020-07-07 21:23:06.837850] D [MSGID: 0] [stack.h:502:copy_frame] 0-stack:
groups is null (ngrps: 0) [Invalid argument]
[2020-07-07 21:23:06.839252] D [dict.c:1168:data_to_uint32]
(-->/lib64/libglusterfs.so.0(dict_foreach_match+0x77) [0x7f0ddb1855e7]
-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x384cf)
[0x7f0dd23c54cf] -->/lib64/libglusterfs.so.0(data_to_uint32+0x8e)
[0x7f0ddb184f2e] ) 0-dict: key null, unsigned integer type asked, has
integer type [Invalid argument]
[2020-07-07 21:23:06.839272] D [MSGID: 0]
[dht-common.c:6674:dht_readdirp_cbk] 0-SSD_Storage-dht: Processing entries
from SSD_Storage-disperse-0
[2020-07-07 21:23:06.839281] D [MSGID: 0]
[dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht:
SSD_Storage-disperse-0: entry = ., type = 4
[2020-07-07 21:23:06.839291] D [MSGID: 0]
[dht-common.c:6813:dht_readdirp_cbk] 0-SSD_Storage-dht:
SSD_Storage-disperse-0: Adding entry = .
[2020-07-07 21:23:06.839297] D [MSGID: 0]
[dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht:
SSD_Storage-disperse-0: entry = .., type = 4
[2020-07-07 21:23:06.839324] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0034598, SSD_Storage-client-6 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839327] D [dict.c:1800:dict_get_int32]
(-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x227d6)
[0x7f0dd23af7d6]
-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x17661)
[0x7f0dd23a4661] -->/lib64/libglusterfs.so.0(dict_get_int32+0x107)
[0x7f0ddb186437] ) 0-dict: key glusterfs.inodelk-count, integer type asked,
has unsigned integer type [Invalid argument]
[2020-07-07 21:23:06.839361] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0034598, SSD_Storage-client-11 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839395] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc00395a8, SSD_Storage-client-15 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839419] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0034598, SSD_Storage-client-9 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839473] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc009c108, SSD_Storage-client-18 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839471] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0034598, SSD_Storage-client-10 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839491] D [dict.c:1800:dict_get_int32]
(-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x256ad)
[0x7f0dd23b26ad]
-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x17661)
[0x7f0dd23a4661] -->/lib64/libglusterfs.so.0(dict_get_int32+0x107)
[0x7f0ddb186437] ) 0-dict: key glusterfs.inodelk-count, integer type asked,
has unsigned integer type [Invalid argument]
[2020-07-07 21:23:06.839512] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0034598, SSD_Storage-client-7 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839526] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc009c108, SSD_Storage-client-23 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839543] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc009c108, SSD_Storage-client-22 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839543] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc00395a8, SSD_Storage-client-16 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839556] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc009c108, SSD_Storage-client-21 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839596] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc00395a8, SSD_Storage-client-12 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839617] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc00395a8, SSD_Storage-client-14 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839631] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc00395a8, SSD_Storage-client-13 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839636] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc00395a8, SSD_Storage-client-17 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839643] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0034598, SSD_Storage-client-8 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839656] D [MSGID: 0]
[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
0x7f0dc007c428, SSD_Storage-disperse-2 returned -1 error: Stale file handle
[Stale file handle]
[2020-07-07 21:23:06.839665] D [MSGID: 0]
[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on
SSD_Storage-disperse-2 returned error [Stale file handle]
[2020-07-07 21:23:06.839666] D [MSGID: 0]
[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
0x7f0dc007c428, SSD_Storage-disperse-1 returned -1 error: Stale file handle
[Stale file handle]
[2020-07-07 21:23:06.839683] D [MSGID: 0]
[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on
SSD_Storage-disperse-1 returned error [Stale file handle]
[2020-07-07 21:23:06.839686] D [dict.c:1168:data_to_uint32]
(-->/lib64/libglusterfs.so.0(dict_foreach_match+0x77) [0x7f0ddb1855e7]
-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x384cf)
[0x7f0dd23c54cf] -->/lib64/libglusterfs.so.0(data_to_uint32+0x8e)
[0x7f0ddb184f2e] ) 0-dict: key null, unsigned integer type asked, has
integer type [Invalid argument]
[2020-07-07 21:23:06.839698] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc009c108, SSD_Storage-client-19 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839703] D [MSGID: 0]
[dht-common.c:6674:dht_readdirp_cbk] 0-SSD_Storage-dht: Processing entries
from SSD_Storage-disperse-0
[2020-07-07 21:23:06.839714] D [MSGID: 0]
[dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht:
SSD_Storage-disperse-0: entry = .., type = 4
[2020-07-07 21:23:06.839716] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0024b48, SSD_Storage-client-30 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839724] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0024b48, SSD_Storage-client-34 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839720] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0024b48, SSD_Storage-client-35 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839755] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0024b48, SSD_Storage-client-31 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839759] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc009c108, SSD_Storage-client-20 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839774] D [MSGID: 0]
[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
0x7f0dc007c428, SSD_Storage-disperse-3 returned -1 error: Stale file handle
[Stale file handle]
[2020-07-07 21:23:06.839775] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0024b48, SSD_Storage-client-32 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839783] D [MSGID: 0]
[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on
SSD_Storage-disperse-3 returned error [Stale file handle]
[2020-07-07 21:23:06.839798] D [MSGID: 0]
[dht-common.c:601:dht_discover_complete] 0-SSD_Storage-dht: key =
trusted.glusterfs.quota.read-only not present in dict
[2020-07-07 21:23:06.839807] D [MSGID: 0]
[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
stack-address: 0x7f0dc0024b48, SSD_Storage-client-33 returned -1 error:
Stale file handle [Stale file handle]
[2020-07-07 21:23:06.839807] D [MSGID: 0]
[dht-layout.c:789:dht_layout_preset] 0-SSD_Storage-dht: file =
00000000-0000-0000-0000-000000000000, subvol = SSD_Storage-disperse-4
[2020-07-07 21:23:06.839825] D [MSGID: 0]
[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
0x7f0dc007c428, SSD_Storage-disperse-5 returned -1 error: Stale file handle
[Stale file handle]
[2020-07-07 21:23:06.839835] D [MSGID: 0]
[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null) on
SSD_Storage-disperse-5 returned error [Stale file handle]

The above is logged just shortly before the qemu-kvm process crashes with
the usual error:

Unexpected error in raw_check_lock_bytes() at block/file-posix.c:811:
2020-07-07T21:23:06.847336Z qemu-kvm: Failed to get shared "write" lock

I have looked also on the bricks logs, but there is too much information
there and will need to know what to look for.

Not sure if there is any benefit in looking into this any further?

Thanks,
Marco

On Thu, 2 Jul 2020 at 15:45, Strahil Nikolov <hunter86_bg at yahoo.com> wrote:

>
>
> ?? 2 ??? 2020 ?. 16:33:51 GMT+03:00, Marco Fais <evilmf at gmail.com> ??????:
> >Hi Strahil,
> >
> >WARNING: As you enabled sharding - NEVER DISABLE SHARDING, EVER !
> >>
> >
> >Thanks -- good to be reminded :)
> >
> >
> >> >When you say they will not be optimal are you referring mainly to
> >> >performance considerations? We did plenty of testing, and in terms
> >of
> >> >performance didn't have issues even with I/O intensive workloads
> >(using
> >> >SSDs, I had issues with spinning disks).
> >>
> >> Yes, the client side has  to connect to 6 bricks (4+2) at a time  and
> >> calculate the data in order to obtain the necessary information.Same
> >is
> >> valid for writing.
> >> If you need to conserve space, you can test VDO without compression
> >(of
> >> even with it).
> >>
> >
> >Understood -- will explore VDO. Storage usage efficiency is less
> >important
> >than fault tolerance or performance for us -- disperse volumes seemed
> >to
> >tick all the boxes so we looked at them primarily.
> >But clearly I had missed that they are not used as mainstream VM
> >storage
> >for oVirt (I did know they weren't supported, but as explained thought
> >was
> >more on the management side).
> >
> >
> >>
> >> Also  with replica  volumes,  you can use 'choose-local'  /in case
> >you
> >> have faster than the network storage (like  NVMe)/ and increase the
> >read
> >> speed. Of course  this feature is useful for Hyperconverged setup
> >(gluster
> >> + ovirt on the same node).
> >>
> >
> >Will explore this option as well, thanks for the suggestion.
> >
> >
> >> If you were using ovirt 4.3 ,  I  would  recommend you to focus  on
> >> gluster. Yet,  you  use  oVirt 4.4 which is quite  newer and it needs
> > some
> >> polishing.
> >>
> >
> >Ovirt 4.3.9 (using the older Centos 7 qemu/libvirt) unfortunately had
> >similar issues with the disperse volumes. Not sure if exactly the same,
> >as
> >never looked deeper into it, but the results were similar.
> >Ovirt 4.4.0 has some issues with snapshot deletion that are independent
> >from Gluster (I have raised the issue here,
> >https://bugzilla.redhat.com/show_bug.cgi?id=1840414, should be sorted
> >with
> >4.4.2 I guess), so at the moment it only works with the "testing" AV
> >repo.
>
>
>
> In such case I can recommend you to:
> 1. Ensure you have enough space on all bricks for the logs
> (/var/log/gluster). Several gigs should be OK
> 2. Enable all logs to 'TRACE' . Red Hat's documentation on the topic is
> quite good:
>
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level
> 3. Reproduce the issue on a fresh VM (never done snapshot deletion)
> 4. Disable (switch to info)  all logs as per the link in point 2
>
> The logs will be spread among all nodes. If you have remote logging
> available, you can also use it for analysis of the logs.
>
> Most probably the brick logs can provide useful information.
>
>
> >
> >> Check ovirt  engine  logs (on the HostedEngine VM or your standalone
> >> engine) ,  vdsm logs  on the host that was running the VM  and next -
> >check
> >> the brick  logs.
> >>
> >
> >Will do.
> >
> >Thanks,
> >Marco
>
>
> About VDO - it might require some tuning and even afterwards it won't be
> very performant, so it depends on your needs.
>
> Best Regards,
> Strahil Nikolov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/81f95077/attachment.html>

From archon810 at gmail.com  Wed Jul  8 06:02:10 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Tue, 7 Jul 2020 23:02:10 -0700
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+4D4SH1LT0=juPw4QU3zy33HVuOytMKn1mrjx0c6Mdtsg@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
	<CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>
	<CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
	<CAFx2kQ3H9A4fbKbLhd1iCcJ-k8M4QGj+tyGumPC1sx-VBkzADA@mail.gmail.com>
	<CAH1hu+52marGwdRmpxBKJweLW6e3J-uLQSVKRyKRP_3PZ=0FoA@mail.gmail.com>
	<CAFx2kQ18jJ4Zb6iy0TB=Bmk6BQ5azhywcM3g4c1dX4SttddgZA@mail.gmail.com>
	<CAH1hu+7A4o1mXeHsje-jCkpsPsS0AJcTn5V5dDRsw-7DLw358Q@mail.gmail.com>
	<CAFx2kQ0RguVjus1JggwyZNKTJLv-zWFc7zHTM2fkyVSxoU_=0Q@mail.gmail.com>
	<CAH1hu+4D4SH1LT0=juPw4QU3zy33HVuOytMKn1mrjx0c6Mdtsg@mail.gmail.com>
Message-ID: <CAD+dzQf2cWoejX3+eTHM043pYJnm2ZTf9PDH2URcFETN+1ZnUw@mail.gmail.com>

I think it'd be extremely helpful if gluster had a feature to grab all the
necessary logs/debug info (maybe a few variations depending on the bug) so
that all the user would have to do is enter a simple command and have
gluster generate the whole bug report, ready to be sent to to the gluster
team.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Tue, Jul 7, 2020 at 1:47 AM Shreyansh Shah <shreyansh.shah at alpha-grep.com>
wrote:

> Sounds good, thank you.
>
> On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com>
> wrote:
>
>> Thanks Shreyansh,
>>
>> I'll look into it, however I'll likely need some help from more senior
>> team members to perform RCA.
>> I'll update once I have new insights.
>>
>> My regards,
>>
>> On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <
>> shreyansh.shah at alpha-grep.com> wrote:
>>
>>> Hi Barak,
>>> Thanks for looking into this and helping me out,
>>> The fix-layout was successful, and I ran a rebalance after completion of
>>> fix-layout.
>>> The rebalance status though did show failure for 3 nodes.
>>>
>>> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com>
>>> wrote:
>>>
>>>> Greetings again Shreyansh,
>>>>
>>>> I'm indeed seeing a lot of errors in the log file - still unsure about
>>>> the RC.
>>>> You mentioned that prior to running rebalance you ran fix-layout, was
>>>> the fix-layout successful?
>>>> Another question - did you wait until fix-layout was completed before
>>>> running rebalance?
>>>>
>>>> My thanks,
>>>>
>>>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <
>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>
>>>>> Hi,
>>>>> Attaching rebalance logs
>>>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster
>>>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout"
>>>>>
>>>>>
>>>>> PFA
>>>>>
>>>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <
>>>>> bsasonro at redhat.com> wrote:
>>>>>
>>>>>> I think it would be best.
>>>>>> As I can't say at this point where the problem is originating from,
>>>>>> brick logs might also be necessary (I assume I would have a better picture
>>>>>> once I have the rebalance logs).
>>>>>>
>>>>>> Cheers,
>>>>>>
>>>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <
>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>
>>>>>>> Hi Barak,
>>>>>>> Can provide the rebalance logs. Do you require all the brick logs
>>>>>>> (14 in total)?
>>>>>>>
>>>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <
>>>>>>> bsasonro at redhat.com> wrote:
>>>>>>>
>>>>>>>> Greetings Shreyansh,
>>>>>>>>
>>>>>>>> Off-hand I can't come up with a reason for these failures.
>>>>>>>> In order to start looking into this, access to the full rebalance
>>>>>>>> logs is required (possibly brick logs as well).
>>>>>>>> Can you provide those?
>>>>>>>>
>>>>>>>> My regards,
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>> Did anyone get a chance to look into this?
>>>>>>>>>
>>>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi All,
>>>>>>>>>>
>>>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>"
>>>>>>>>>> errors.*
>>>>>>>>>>
>>>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks
>>>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to
>>>>>>>>>> the existing setup.
>>>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance
>>>>>>>>>> (which is currently still in progress). The status shows "failed" on
>>>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster
>>>>>>>>>> rebalance status below.
>>>>>>>>>>
>>>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts
>>>>>>>>>> for <directory>,gfid = <id>"
>>>>>>>>>> The performance too seems to have degraded due to this, even
>>>>>>>>>> basic commands like `cd` and `ls` are taking more than a minute compared to
>>>>>>>>>> sub-second number before brick addition.
>>>>>>>>>> Apart from that we also experienced many binaries and files
>>>>>>>>>> giving error stale file handle error even though the files were present.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *gluster rebalance status :*
>>>>>>>>>>
>>>>>>>>>> Node Rebalanced-files          size       scanned      failures
>>>>>>>>>>     skipped               status  run time in h:m:s
>>>>>>>>>> ---------      -----------   -----------   -----------
>>>>>>>>>> -----------   -----------         ------------     --------------
>>>>>>>>>> localhost              176         3.5GB         12790
>>>>>>>>>>   0          8552          in progress       21:36:01
>>>>>>>>>> 10.132.0.72             8232       394.8GB         19995
>>>>>>>>>>    21            26               failed       14:50:30
>>>>>>>>>> 10.132.0.44            12625         1.0TB         50023
>>>>>>>>>>     1         10202          in progress       21:36:00
>>>>>>>>>> 10.132.0.3            21982       956.8GB         79145
>>>>>>>>>>   1         34571          in progress       21:36:00
>>>>>>>>>> 10.132.0.9             7975       355.8GB         20157
>>>>>>>>>>   6          1522               failed       14:51:45
>>>>>>>>>> 10.132.0.73             6293       394.5GB         26414
>>>>>>>>>>   151          8085               failed       14:51:45
>>>>>>>>>> 10.132.0.70             6480       477.1GB         21058
>>>>>>>>>>    27          1787               failed       14:50:32
>>>>>>>>>> Estimated time left for rebalance to complete :      130:56:28
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Logs from one of the clients below:*
>>>>>>>>>>
>>>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>>>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>>>>>>>>>> layout - 920400036 - 1227153471 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>>>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>>>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>>>>>>>>>> layout - 613646600 - 920400035 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>>>>>>>> data-client-7 (hashed subvol is data-client-17)
>>>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>>>>>>>> data-client-6 (hashed subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>>>>>>>> data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>> on data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>>>>>>>> data-client-14 (hashed subvol is data-client-7)
>>>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>> on data-client-15 (hashed subvol is data-client-8)
>>>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>>>>>>>>>> (hashed subvol is data-client-6)
>>>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>>>>>>>>>> subvol is data-client-6)
>>>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>>>>>>>>>> subvol is data-client-10)
>>>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>>>>>>>>>> (hashed subvol is data-client-8)
>>>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>>>>>>>>>> (hashed subvol is data-client-2)
>>>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>> on data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>>>>>>>>>> (hashed subvol is data-client-19)
>>>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>>>>>>>>>> 613576736 - 920330171 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>>>>>>>>>> 920330172 - 1227083607 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>>>>>>>>>> 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>>>>>>>>>> 3681390552 - 3988143987 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>>>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>>>>>>>>>> - 306753435 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>>>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>>>>>>>>>> 3681460416 - 3988213851 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>>>>>>>>>> 3988213852 - 4294967295 - 4159036738
>>>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>>>>>>>> (hash=data-client-6/cache=data-client-18) =>
>>>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>>>>>>>> (hash=data-client-6/cache=<nul>)
>>>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>>>>>>>> (hash=data-client-17/cache=data-client-18) =>
>>>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>>>>>>>> (hash=data-client-17/cache=<nul>)
>>>>>>>>>> ....
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Please look into this at top-priority if possible.
>>>>>>>>>> Let me know if anything else is required.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Shreyansh Shah
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Shreyansh Shah
>>>>>>>>> ________
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Community Meeting Calendar:
>>>>>>>>>
>>>>>>>>> Schedule -
>>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>>>
>>>>>>>>> Gluster-users mailing list
>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Barak Sason Rofman*
>>>>>>>>
>>>>>>>> Gluster Storage Development
>>>>>>>>
>>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>>
>>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>>
>>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>>> M: *+972-52-4326355*
>>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>>> <https://red.ht/sig>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Shreyansh Shah
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Barak Sason Rofman*
>>>>>>
>>>>>> Gluster Storage Development
>>>>>>
>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>
>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>
>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>> M: *+972-52-4326355*
>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>> <https://red.ht/sig>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Shreyansh Shah
>>>>>
>>>>
>>>>
>>>> --
>>>> *Barak Sason Rofman*
>>>>
>>>> Gluster Storage Development
>>>>
>>>> Red Hat Israel <https://www.redhat.com/>
>>>>
>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>
>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>> M: *+972-52-4326355*
>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>> <https://www.facebook.com/redhat.il/>
>>>> <https://red.ht/sig>
>>>>
>>>
>>>
>>> --
>>> Regards,
>>> Shreyansh Shah
>>>
>>
>>
>> --
>> *Barak Sason Rofman*
>>
>> Gluster Storage Development
>>
>> Red Hat Israel <https://www.redhat.com/>
>>
>> 34 Jerusalem rd. Ra'anana, 43501
>>
>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>> M: *+972-52-4326355*
>> @RedHat <https://twitter.com/redhat>   Red Hat
>> <https://www.linkedin.com/company/red-hat>  Red Hat
>> <https://www.facebook.com/redhat.il/>
>> <https://red.ht/sig>
>>
>
>
> --
> Regards,
> Shreyansh Shah
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200707/47b1369d/attachment.html>

From hunter86_bg at yahoo.com  Wed Jul  8 11:27:49 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Wed, 08 Jul 2020 14:27:49 +0300
Subject: [Gluster-users] Problems with qemu and disperse volumes (live
	merge)
In-Reply-To: <CAC7CbKFQP=5WqxE0XEKrE8u2B8oSyDnrad62mpkp+OFMLhh_DQ@mail.gmail.com>
References: <CAC7CbKH2pgyAKLNfdPvSZSvTL2Dc5USngubNr0BMD573c_VgBQ@mail.gmail.com>
	<F1022CAA-A397-483D-AA12-98F6A20FD591@yahoo.com>
	<CAC7CbKHG4U5sCyru8jN7UgYzb4BbERuJnhsj=mOe15EH78kiJg@mail.gmail.com>
	<93D3EE3B-B5B3-4689-BF66-C1442A03971E@yahoo.com>
	<CAC7CbKHc24RzVgQhRJiKpjjxCLGDjf1rhK-FEHUu4JbYyHjSJw@mail.gmail.com>
	<CCF9A9E3-151F-4D9B-A510-7E2675BD7BF4@yahoo.com>
	<CAC7CbKFQP=5WqxE0XEKrE8u2B8oSyDnrad62mpkp+OFMLhh_DQ@mail.gmail.com>
Message-ID: <7C1FC636-CD69-46D1-8916-B583164B4AD5@yahoo.com>



See  my comments inline.


?? 8 ??? 2020 ?. 0:46:21 GMT+03:00, Marco Fais <evilmf at gmail.com> ??????:
>Hi Strahil
>
>first of all thanks a million for your help -- really appreciate it.
>Thanks also for the pointers on the debug. I have tried it, and while I
>can't interpret the results I think I might have found something.
>
>There is a lot of information so hopefully this is relevant. During the
>snapshot creation and deletion, I can see the following errors in the
>client log:
>
>[2020-07-07 21:23:06.837381] W [MSGID: 122019]
>[ec-helpers.c:401:ec_loc_gfid_check] 0-SSD_Storage-disperse-0:
>Mismatching
>GFID's in loc
>[2020-07-07 21:23:06.837387] D [MSGID: 0]
>[defaults.c:1328:default_mknod_cbk] 0-stack-trace: stack-address:
>0x7f0dc0001a78, SSD_Storage-disperse-0 returned -1 error: Input/output
>error [Input/output error]


You have to check brick logs for  the first brick in the  volume  list.


>[2020-07-07 21:23:06.837392] W [MSGID: 109002]
>[dht-rename.c:1019:dht_rename_links_create_cbk] 0-SSD_Storage-dht:
>link/file
>/8d49207e-f6b9-41d1-8d35-f6e0fb121980/images/4802e66e-a7e3-42df-a570-7155135566ad/b51133ee-54e0-4001-ab4b-9f0dc1e5c6fc.meta

Check the meta file. There was a problem with Gluster where it healed it before the other replica  has  come up (in your case is  a little bit different.Usually only the timestamp inside the file is changed,  so you can force gluster to update it  by changing the timestamp inside.


>on SSD_Storage-disperse-0 failed [Input/output error]

Already mentioned it.

>[2020-07-07 21:23:06.837850] D [MSGID: 0] [stack.h:502:copy_frame]
>0-stack:
>groups is null (ngrps: 0) [Invalid argument]
>[2020-07-07 21:23:06.839252] D [dict.c:1168:data_to_uint32]
>(-->/lib64/libglusterfs.so.0(dict_foreach_match+0x77) [0x7f0ddb1855e7]
>-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x384cf)
>[0x7f0dd23c54cf] -->/lib64/libglusterfs.so.0(data_to_uint32+0x8e)
>[0x7f0ddb184f2e] ) 0-dict: key null, unsigned integer type asked, has
>integer type [Invalid argument]
>[2020-07-07 21:23:06.839272] D [MSGID: 0]
>[dht-common.c:6674:dht_readdirp_cbk] 0-SSD_Storage-dht: Processing
>entries
>from SSD_Storage-disperse-0
>[2020-07-07 21:23:06.839281] D [MSGID: 0]
>[dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht:
>SSD_Storage-disperse-0: entry = ., type = 4
>[2020-07-07 21:23:06.839291] D [MSGID: 0]
>[dht-common.c:6813:dht_readdirp_cbk] 0-SSD_Storage-dht:
>SSD_Storage-disperse-0: Adding entry = .
>[2020-07-07 21:23:06.839297] D [MSGID: 0]
>[dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht:
>SSD_Storage-disperse-0: entry = .., type = 4
>[2020-07-07 21:23:06.839324] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0034598, SSD_Storage-client-6 returned -1 error:
>Stale file handle [Stale file handle]


I see multiple of these, but as the message is not 'W' or 'E' ,  I assume it could happen and it's normal.

>[2020-07-07 21:23:06.839327] D [dict.c:1800:dict_get_int32]
>(-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x227d6)
>[0x7f0dd23af7d6]
>-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x17661)
>[0x7f0dd23a4661] -->/lib64/libglusterfs.so.0(dict_get_int32+0x107)
>[0x7f0ddb186437] ) 0-dict: key glusterfs.inodelk-count, integer type
>asked,
>has unsigned integer type [Invalid argument]
>[2020-07-07 21:23:06.839361] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0034598, SSD_Storage-client-11 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839395] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc00395a8, SSD_Storage-client-15 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839419] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0034598, SSD_Storage-client-9 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839473] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc009c108, SSD_Storage-client-18 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839471] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0034598, SSD_Storage-client-10 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839491] D [dict.c:1800:dict_get_int32]
>(-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x256ad)
>[0x7f0dd23b26ad]
>-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x17661)
>[0x7f0dd23a4661] -->/lib64/libglusterfs.so.0(dict_get_int32+0x107)
>[0x7f0ddb186437] ) 0-dict: key glusterfs.inodelk-count, integer type
>asked,
>has unsigned integer type [Invalid argument]
>[2020-07-07 21:23:06.839512] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0034598, SSD_Storage-client-7 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839526] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc009c108, SSD_Storage-client-23 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839543] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc009c108, SSD_Storage-client-22 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839543] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc00395a8, SSD_Storage-client-16 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839556] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc009c108, SSD_Storage-client-21 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839596] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc00395a8, SSD_Storage-client-12 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839617] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc00395a8, SSD_Storage-client-14 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839631] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc00395a8, SSD_Storage-client-13 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839636] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc00395a8, SSD_Storage-client-17 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839643] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0034598, SSD_Storage-client-8 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839656] D [MSGID: 0]
>[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
>0x7f0dc007c428, SSD_Storage-disperse-2 returned -1 error: Stale file
>handle
>[Stale file handle]
>[2020-07-07 21:23:06.839665] D [MSGID: 0]
>[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null)
>on
>SSD_Storage-disperse-2 returned error [Stale file handle]
>[2020-07-07 21:23:06.839666] D [MSGID: 0]
>[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
>0x7f0dc007c428, SSD_Storage-disperse-1 returned -1 error: Stale file
>handle
>[Stale file handle]
>[2020-07-07 21:23:06.839683] D [MSGID: 0]
>[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null)
>on
>SSD_Storage-disperse-1 returned error [Stale file handle]
>[2020-07-07 21:23:06.839686] D [dict.c:1168:data_to_uint32]
>(-->/lib64/libglusterfs.so.0(dict_foreach_match+0x77) [0x7f0ddb1855e7]
>-->/usr/lib64/glusterfs/7.5/xlator/cluster/disperse.so(+0x384cf)
>[0x7f0dd23c54cf] -->/lib64/libglusterfs.so.0(data_to_uint32+0x8e)
>[0x7f0ddb184f2e] ) 0-dict: key null, unsigned integer type asked, has
>integer type [Invalid argument]
>[2020-07-07 21:23:06.839698] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc009c108, SSD_Storage-client-19 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839703] D [MSGID: 0]
>[dht-common.c:6674:dht_readdirp_cbk] 0-SSD_Storage-dht: Processing
>entries
>from SSD_Storage-disperse-0
>[2020-07-07 21:23:06.839714] D [MSGID: 0]
>[dht-common.c:6681:dht_readdirp_cbk] 0-SSD_Storage-dht:
>SSD_Storage-disperse-0: entry = .., type = 4
>[2020-07-07 21:23:06.839716] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0024b48, SSD_Storage-client-30 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839724] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0024b48, SSD_Storage-client-34 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839720] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0024b48, SSD_Storage-client-35 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839755] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0024b48, SSD_Storage-client-31 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839759] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc009c108, SSD_Storage-client-20 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839774] D [MSGID: 0]
>[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
>0x7f0dc007c428, SSD_Storage-disperse-3 returned -1 error: Stale file
>handle
>[Stale file handle]
>[2020-07-07 21:23:06.839775] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0024b48, SSD_Storage-client-32 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839783] D [MSGID: 0]
>[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null)
>on
>SSD_Storage-disperse-3 returned error [Stale file handle]
>[2020-07-07 21:23:06.839798] D [MSGID: 0]
>[dht-common.c:601:dht_discover_complete] 0-SSD_Storage-dht: key =
>trusted.glusterfs.quota.read-only not present in dict
>[2020-07-07 21:23:06.839807] D [MSGID: 0]
>[client-rpc-fops_v2.c:2641:client4_0_lookup_cbk] 0-stack-trace:
>stack-address: 0x7f0dc0024b48, SSD_Storage-client-33 returned -1 error:
>Stale file handle [Stale file handle]
>[2020-07-07 21:23:06.839807] D [MSGID: 0]
>[dht-layout.c:789:dht_layout_preset] 0-SSD_Storage-dht: file =
>00000000-0000-0000-0000-000000000000, subvol = SSD_Storage-disperse-4
>[2020-07-07 21:23:06.839825] D [MSGID: 0]
>[defaults.c:1548:default_lookup_cbk] 0-stack-trace: stack-address:
>0x7f0dc007c428, SSD_Storage-disperse-5 returned -1 error: Stale file
>handle
>[Stale file handle]
>[2020-07-07 21:23:06.839835] D [MSGID: 0]
>[dht-common.c:998:dht_discover_cbk] 0-SSD_Storage-dht: lookup of (null)
>on
>SSD_Storage-disperse-5 returned error [Stale file handle]
>
>The above is logged just shortly before the qemu-kvm process crashes
>with
>the usual error:
>
>Unexpected error in raw_check_lock_bytes() at block/file-posix.c:811:
>2020-07-07T21:23:06.847336Z qemu-kvm: Failed to get shared "write" lock


That's strange. Can you check the sanlock logs  for anything reported  there  ?


>I have looked also on the bricks logs, but there is too much
>information
>there and will need to know what to look for.
>
>Not sure if there is any benefit in looking into this any further?
>
>Thanks,
>Marco
>
>On Thu, 2 Jul 2020 at 15:45, Strahil Nikolov <hunter86_bg at yahoo.com>
>wrote:
>
>>
>>
>> ?? 2 ??? 2020 ?. 16:33:51 GMT+03:00, Marco Fais <evilmf at gmail.com>
>??????:
>> >Hi Strahil,
>> >
>> >WARNING: As you enabled sharding - NEVER DISABLE SHARDING, EVER !
>> >>
>> >
>> >Thanks -- good to be reminded :)
>> >
>> >
>> >> >When you say they will not be optimal are you referring mainly to
>> >> >performance considerations? We did plenty of testing, and in
>terms
>> >of
>> >> >performance didn't have issues even with I/O intensive workloads
>> >(using
>> >> >SSDs, I had issues with spinning disks).
>> >>
>> >> Yes, the client side has  to connect to 6 bricks (4+2) at a time 
>and
>> >> calculate the data in order to obtain the necessary
>information.Same
>> >is
>> >> valid for writing.
>> >> If you need to conserve space, you can test VDO without
>compression
>> >(of
>> >> even with it).
>> >>
>> >
>> >Understood -- will explore VDO. Storage usage efficiency is less
>> >important
>> >than fault tolerance or performance for us -- disperse volumes
>seemed
>> >to
>> >tick all the boxes so we looked at them primarily.
>> >But clearly I had missed that they are not used as mainstream VM
>> >storage
>> >for oVirt (I did know they weren't supported, but as explained
>thought
>> >was
>> >more on the management side).
>> >
>> >
>> >>
>> >> Also  with replica  volumes,  you can use 'choose-local'  /in case
>> >you
>> >> have faster than the network storage (like  NVMe)/ and increase
>the
>> >read
>> >> speed. Of course  this feature is useful for Hyperconverged setup
>> >(gluster
>> >> + ovirt on the same node).
>> >>
>> >
>> >Will explore this option as well, thanks for the suggestion.
>> >
>> >
>> >> If you were using ovirt 4.3 ,  I  would  recommend you to focus 
>on
>> >> gluster. Yet,  you  use  oVirt 4.4 which is quite  newer and it
>needs
>> > some
>> >> polishing.
>> >>
>> >
>> >Ovirt 4.3.9 (using the older Centos 7 qemu/libvirt) unfortunately
>had
>> >similar issues with the disperse volumes. Not sure if exactly the
>same,
>> >as
>> >never looked deeper into it, but the results were similar.
>> >Ovirt 4.4.0 has some issues with snapshot deletion that are
>independent
>> >from Gluster (I have raised the issue here,
>> >https://bugzilla.redhat.com/show_bug.cgi?id=1840414, should be
>sorted
>> >with
>> >4.4.2 I guess), so at the moment it only works with the "testing" AV
>> >repo.
>>
>>
>>
>> In such case I can recommend you to:
>> 1. Ensure you have enough space on all bricks for the logs
>> (/var/log/gluster). Several gigs should be OK
>> 2. Enable all logs to 'TRACE' . Red Hat's documentation on the topic
>is
>> quite good:
>>
>>
>https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3/html/administration_guide/configuring_the_log_level
>> 3. Reproduce the issue on a fresh VM (never done snapshot deletion)
>> 4. Disable (switch to info)  all logs as per the link in point 2
>>
>> The logs will be spread among all nodes. If you have remote logging
>> available, you can also use it for analysis of the logs.
>>
>> Most probably the brick logs can provide useful information.
>>
>>
>> >
>> >> Check ovirt  engine  logs (on the HostedEngine VM or your
>standalone
>> >> engine) ,  vdsm logs  on the host that was running the VM  and
>next -
>> >check
>> >> the brick  logs.
>> >>
>> >
>> >Will do.
>> >
>> >Thanks,
>> >Marco
>>
>>
>> About VDO - it might require some tuning and even afterwards it won't
>be
>> very performant, so it depends on your needs.
>>
>> Best Regards,
>> Strahil Nikolov
>>

From hunter86_bg at yahoo.com  Wed Jul  8 11:35:35 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Wed, 08 Jul 2020 14:35:35 +0300
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
	after new brick addition and rebalance
In-Reply-To: <CAD+dzQf2cWoejX3+eTHM043pYJnm2ZTf9PDH2URcFETN+1ZnUw@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
	<CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>
	<CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
	<CAFx2kQ3H9A4fbKbLhd1iCcJ-k8M4QGj+tyGumPC1sx-VBkzADA@mail.gmail.com>
	<CAH1hu+52marGwdRmpxBKJweLW6e3J-uLQSVKRyKRP_3PZ=0FoA@mail.gmail.com>
	<CAFx2kQ18jJ4Zb6iy0TB=Bmk6BQ5azhywcM3g4c1dX4SttddgZA@mail.gmail.com>
	<CAH1hu+7A4o1mXeHsje-jCkpsPsS0AJcTn5V5dDRsw-7DLw358Q@mail.gmail.com>
	<CAFx2kQ0RguVjus1JggwyZNKTJLv-zWFc7zHTM2fkyVSxoU_=0Q@mail.gmail.com>
	<CAH1hu+4D4SH1LT0=juPw4QU3zy33HVuOytMKn1mrjx0c6Mdtsg@mail.gmail.com>
	<CAD+dzQf2cWoejX3+eTHM043pYJnm2ZTf9PDH2URcFETN+1ZnUw@mail.gmail.com>
Message-ID: <0EB17979-D921-492C-B3AE-852F2B4A2F8D@yahoo.com>

At least for EL 7 ,there are 2  modules for sosreport:
gluster & gluster_block

Best Regards,
Strahil Nikolov

?? 8 ??? 2020 ?. 9:02:10 GMT+03:00, Artem Russakovskii <archon810 at gmail.com> ??????:
>I think it'd be extremely helpful if gluster had a feature to grab all
>the
>necessary logs/debug info (maybe a few variations depending on the bug)
>so
>that all the user would have to do is enter a simple command and have
>gluster generate the whole bug report, ready to be sent to to the
>gluster
>team.
>
>Sincerely,
>Artem
>
>--
>Founder, Android Police <http://www.androidpolice.com>, APK Mirror
><http://www.apkmirror.com/>, Illogical Robot LLC
>beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
>On Tue, Jul 7, 2020 at 1:47 AM Shreyansh Shah
><shreyansh.shah at alpha-grep.com>
>wrote:
>
>> Sounds good, thank you.
>>
>> On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman
><bsasonro at redhat.com>
>> wrote:
>>
>>> Thanks Shreyansh,
>>>
>>> I'll look into it, however I'll likely need some help from more
>senior
>>> team members to perform RCA.
>>> I'll update once I have new insights.
>>>
>>> My regards,
>>>
>>> On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <
>>> shreyansh.shah at alpha-grep.com> wrote:
>>>
>>>> Hi Barak,
>>>> Thanks for looking into this and helping me out,
>>>> The fix-layout was successful, and I ran a rebalance after
>completion of
>>>> fix-layout.
>>>> The rebalance status though did show failure for 3 nodes.
>>>>
>>>> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman
><bsasonro at redhat.com>
>>>> wrote:
>>>>
>>>>> Greetings again Shreyansh,
>>>>>
>>>>> I'm indeed seeing a lot of errors in the log file - still unsure
>about
>>>>> the RC.
>>>>> You mentioned that prior to running rebalance you ran fix-layout,
>was
>>>>> the fix-layout successful?
>>>>> Another question - did you wait until fix-layout was completed
>before
>>>>> running rebalance?
>>>>>
>>>>> My thanks,
>>>>>
>>>>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <
>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Attaching rebalance logs
>>>>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster
>>>>>> rebalance" on 20200701 and today we again ran "gluster rebalance
>fix-layout"
>>>>>>
>>>>>>
>>>>>> PFA
>>>>>>
>>>>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <
>>>>>> bsasonro at redhat.com> wrote:
>>>>>>
>>>>>>> I think it would be best.
>>>>>>> As I can't say at this point where the problem is originating
>from,
>>>>>>> brick logs might also be necessary (I assume I would have a
>better picture
>>>>>>> once I have the rebalance logs).
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <
>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>
>>>>>>>> Hi Barak,
>>>>>>>> Can provide the rebalance logs. Do you require all the brick
>logs
>>>>>>>> (14 in total)?
>>>>>>>>
>>>>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <
>>>>>>>> bsasonro at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> Greetings Shreyansh,
>>>>>>>>>
>>>>>>>>> Off-hand I can't come up with a reason for these failures.
>>>>>>>>> In order to start looking into this, access to the full
>rebalance
>>>>>>>>> logs is required (possibly brick logs as well).
>>>>>>>>> Can you provide those?
>>>>>>>>>
>>>>>>>>> My regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Did anyone get a chance to look into this?
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid =
><id>"
>>>>>>>>>>> errors.*
>>>>>>>>>>>
>>>>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2
>bricks
>>>>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new
>bricks yesterday to
>>>>>>>>>>> the existing setup.
>>>>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance
>>>>>>>>>>> (which is currently still in progress). The status shows
>"failed" on
>>>>>>>>>>> certain bricks but "in progress" for others. Adding output
>for gluster
>>>>>>>>>>> rebalance status below.
>>>>>>>>>>>
>>>>>>>>>>> The glusterfs client logs are flooded with "Mismatching
>layouts
>>>>>>>>>>> for <directory>,gfid = <id>"
>>>>>>>>>>> The performance too seems to have degraded due to this, even
>>>>>>>>>>> basic commands like `cd` and `ls` are taking more than a
>minute compared to
>>>>>>>>>>> sub-second number before brick addition.
>>>>>>>>>>> Apart from that we also experienced many binaries and files
>>>>>>>>>>> giving error stale file handle error even though the files
>were present.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *gluster rebalance status :*
>>>>>>>>>>>
>>>>>>>>>>> Node Rebalanced-files          size       scanned     
>failures
>>>>>>>>>>>     skipped               status  run time in h:m:s
>>>>>>>>>>> ---------      -----------   -----------   -----------
>>>>>>>>>>> -----------   -----------         ------------    
>--------------
>>>>>>>>>>> localhost              176         3.5GB         12790
>>>>>>>>>>>   0          8552          in progress       21:36:01
>>>>>>>>>>> 10.132.0.72             8232       394.8GB         19995
>>>>>>>>>>>    21            26               failed       14:50:30
>>>>>>>>>>> 10.132.0.44            12625         1.0TB         50023
>>>>>>>>>>>     1         10202          in progress       21:36:00
>>>>>>>>>>> 10.132.0.3            21982       956.8GB         79145
>>>>>>>>>>>   1         34571          in progress       21:36:00
>>>>>>>>>>> 10.132.0.9             7975       355.8GB         20157
>>>>>>>>>>>   6          1522               failed       14:51:45
>>>>>>>>>>> 10.132.0.73             6293       394.5GB         26414
>>>>>>>>>>>   151          8085               failed       14:51:45
>>>>>>>>>>> 10.132.0.70             6480       477.1GB         21058
>>>>>>>>>>>    27          1787               failed       14:50:32
>>>>>>>>>>> Estimated time left for rebalance to complete :     
>130:56:28
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Logs from one of the clients below:*
>>>>>>>>>>>
>>>>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 -
>3995747641; disk
>>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI, gfid =
>b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 -
>3995747641; disk
>>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI, gfid =
>b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 -
>3995747641; disk
>>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI, gfid =
>b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 -
>3997647794; disk
>>>>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 -
>3997647794; disk
>>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 -
>3997647794; disk
>>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 -
>3997647794; disk
>>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid =
>42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 -
>3997647794; disk
>>>>>>>>>>> layout - 920400036 - 1227153471 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid =
>1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 -
>3997647794; disk
>>>>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid =
>1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 -
>3997647794; disk
>>>>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid =
>1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 -
>3997647794; disk
>>>>>>>>>>> layout - 613646600 - 920400035 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid =
>1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>>>>>>>>> data-client-7 (hashed subvol is data-client-17)
>>>>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>>>>>>>>> data-client-6 (hashed subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>>>>>>>>> data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>>> on data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>>>>>>>>> data-client-14 (hashed subvol is data-client-7)
>>>>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>>> on data-client-15 (hashed subvol is data-client-8)
>>>>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on
>data-client-11
>>>>>>>>>>> (hashed subvol is data-client-6)
>>>>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on
>data-client-14 (hashed
>>>>>>>>>>> subvol is data-client-6)
>>>>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on
>data-client-6 (hashed
>>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on
>data-client-6 (hashed
>>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on
>data-client-2 (hashed
>>>>>>>>>>> subvol is data-client-10)
>>>>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on
>data-client-15
>>>>>>>>>>> (hashed subvol is data-client-8)
>>>>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on
>data-client-10
>>>>>>>>>>> (hashed subvol is data-client-2)
>>>>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on
>data-client-8 (hashed
>>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>>> on data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on
>data-client-6 (hashed
>>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>/processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on
>data-client-9 (hashed
>>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht:
>attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on
>data-client-9
>>>>>>>>>>> (hashed subvol is data-client-19)
>>>>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht:
>lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1;
>disk layout -
>>>>>>>>>>> 613576736 - 920330171 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes, gfid =
>21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1;
>disk layout -
>>>>>>>>>>> 920330172 - 1227083607 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes, gfid =
>21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1;
>disk layout -
>>>>>>>>>>> 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1;
>disk layout -
>>>>>>>>>>> 3681390552 - 3988143987 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 -
>3995747641; disk
>>>>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 -
>3995747641; disk
>>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641;
>disk layout - 0
>>>>>>>>>>> - 306753435 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 -
>3995747641; disk
>>>>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1;
>disk layout -
>>>>>>>>>>> 3681460416 - 3988213851 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht:
>subvol:
>>>>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1;
>disk layout -
>>>>>>>>>>> 3988213852 - 4294967295 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht:
>Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>>>>>>>>> (hash=data-client-6/cache=data-client-18) =>
>>>>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>>>>>>>>> (hash=data-client-6/cache=<nul>)
>>>>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>>>>>>>>> (hash=data-client-17/cache=data-client-18) =>
>>>>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>>>>>>>>> (hash=data-client-17/cache=<nul>)
>>>>>>>>>>> ....
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please look into this at top-priority if possible.
>>>>>>>>>>> Let me know if anything else is required.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Shreyansh Shah
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Shreyansh Shah
>>>>>>>>>> ________
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Community Meeting Calendar:
>>>>>>>>>>
>>>>>>>>>> Schedule -
>>>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>>>>
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Barak Sason Rofman*
>>>>>>>>>
>>>>>>>>> Gluster Storage Development
>>>>>>>>>
>>>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>>>
>>>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>>>
>>>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>>>> M: *+972-52-4326355*
>>>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>>>> <https://red.ht/sig>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shreyansh Shah
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Barak Sason Rofman*
>>>>>>>
>>>>>>> Gluster Storage Development
>>>>>>>
>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>
>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>
>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>> M: *+972-52-4326355*
>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>> <https://red.ht/sig>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shreyansh Shah
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Barak Sason Rofman*
>>>>>
>>>>> Gluster Storage Development
>>>>>
>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>
>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>
>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>> M: *+972-52-4326355*
>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>> <https://www.facebook.com/redhat.il/>
>>>>> <https://red.ht/sig>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shreyansh Shah
>>>>
>>>
>>>
>>> --
>>> *Barak Sason Rofman*
>>>
>>> Gluster Storage Development
>>>
>>> Red Hat Israel <https://www.redhat.com/>
>>>
>>> 34 Jerusalem rd. Ra'anana, 43501
>>>
>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>> M: *+972-52-4326355*
>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>> <https://www.facebook.com/redhat.il/>
>>> <https://red.ht/sig>
>>>
>>
>>
>> --
>> Regards,
>> Shreyansh Shah
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>

From cjeppesen at datto.com  Wed Jul  8 11:33:29 2020
From: cjeppesen at datto.com (Claus Jeppesen)
Date: Wed, 8 Jul 2020 13:33:29 +0200
Subject: [Gluster-users] Sharding on 7.x - file sizes are wrong after a
	large copy.
Message-ID: <CA+RqqpLwpyjUk7rgnnNjx9dw7Fi0N+yN7VJgu_4XX-SGnEanAQ@mail.gmail.com>

In April of this year I reported the problem using sharding on gluster 7.4:

====
We're using GlusterFS in a replicated brick setup with 2 bricks with
sharding turned on (shardsize 128MB).

There is something funny going on as we can see that if we copy large VM
files to the volume we can end up with files that are a bit larger than the
source files DEPENDING on the speed with which we copied the files - e.g.:

   dd if=SOURCE bs=1M | pv -L NNm | ssh gluster_server "dd
of=/gluster/VOL_NAME/TARGET
bs=1M"

It seems that if NN is <= 25 (i.e. 25 MB/s) the size of SOURCE and TARGET
will be the same.

If we crank NN to, say, 50 we sometimes risk that a 25G file ends up having
a slightly larger size, e.g. 26844413952 or 26844233728 - larger than the
expected 26843545600.
Unfortunately this is not an illusion ! If we dd the files out of Gluster
we will receive the amount of data that 'ls' showed us.

In the brick directory (incl .shard directory) we have the expected amount
of shards for a 25G files (200) with size precisely equal to 128MB - but
there is an additional 0 size shard file created.

Has anyone else seen a phenomenon like this ?
====

After upgrade to 7.6 we're still seeing this problem - now,  the extra
bytes that are appearing can be removed using truncate in the mounted
gluster volume, and md5sum can confirm that after truncate the content is
identical to the source - however, it may point to an underlying issue.

I hope someone can reproduce this behaviour,

Thanx,

Claus.

-- 
*Claus Jeppesen*
Manager, Network Services
Datto, Inc.
p +45 6170 5901 | Copenhagen Office
www.datto.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200708/11410d1b/attachment.html>

From duc.to.ho at gmail.com  Wed Jul  8 19:25:57 2020
From: duc.to.ho at gmail.com (Duc To Ho)
Date: Thu, 9 Jul 2020 02:25:57 +0700
Subject: [Gluster-users] Setup GlusterFS geo-replication between Ubuntu
	server and CentOS server
Message-ID: <CACOwRaEy_=Bbo+BFzVCeG_+8tQ+a6nfVu8bvTwDPYRQ7L6i8NA@mail.gmail.com>

Hello all,

Recently I need to setup GlusterFS geo-replication between Ubuntu server
and CentOs server. I setup following the guideline and everything was fine,
checking with "gluster volume geo-replication gvol0 dev-3-15:gvol0 status"
command returns me the info with Active status. But then when I started
copying data to the master node, I got Exception error in log and status
has become Faulty.

I tried to set up the same in both CentOs-CentOs and Ubuntu-Ubuntu and
there is no such issue.

I wonder if anyone has set up the same case CentOS-Ubuntu and feedback if
it is OK or had a similar issue and hopefully, the solution for it?

Thank you and I look forward to hearing from you,
Duc

<nnlong at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200709/73077d7f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image_2020_07_06T03_10_34_652Z.png
Type: image/png
Size: 207811 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200709/73077d7f/attachment.png>

From sacharya at redhat.com  Thu Jul  9 11:29:58 2020
From: sacharya at redhat.com (Shwetha Acharya)
Date: Thu, 9 Jul 2020 16:59:58 +0530
Subject: [Gluster-users] Reliable Geo-Replication
In-Reply-To: <b8e32f0c-635c-f39f-e687-1019ef7b6e85@gmx.de>
References: <75c1490e-2a69-7bc2-9ddd-a26fa5a225f5@gmx.de>
	<CAAwt65CmBh8Anfvb9ijnicz82ONFr10PHrTd7_TM_50YO5jUDA@mail.gmail.com>
	<93e7700c-c4df-dd3c-ba15-f7a815ae7e6a@gmx.de>
	<CAERh03pC1OkHBOO_Q-YYQnkRsReAUN9ynHssRL+fveHa1QOsOg@mail.gmail.com>
	<841137d6-768f-7560-24b3-c80fc8ceffa9@gmx.de>
	<CAERh03qjLu0s2iv2CPEbj0j5p8WZn=J=jaBTn-ZoX3DrHAuQog@mail.gmail.com>
	<ae9cf3f3-96c2-216f-a8d3-b540100999f6@gmx.de>
	<CAERh03rvwCURTbO+CJ1tUY_hwRSdmQqMza-MTNQShWfeLvCdFg@mail.gmail.com>
	<3b29ee38-991d-6253-f3da-504f4414c723@gmx.de>
	<CAERh03o9h10B7cvR5mUrrCbNvB27TRiH12i3Ki0Hdbq_XYbGHw@mail.gmail.com>
	<93624ed1-4c0a-100f-344c-1cb99b30f94b@mpa-ifw.tu-darmstadt.de>
	<b8e32f0c-635c-f39f-e687-1019ef7b6e85@gmx.de>
Message-ID: <CAERh03rV994CW3MGhUPB4z0sk9SbcECQtEg3Yo5H1ejvQO7CQg@mail.gmail.com>

Hi Felix,

Find my reply inline.

Regards,
Shwetha

On Thu, Jun 25, 2020 at 12:25 PM Felix K?lzow <felix.koelzow at gmx.de> wrote:

> Dear Gluster-users,
>
> I deleted a further the geo-replication session with [reset-sync-time]
> option. Afterwards,
> I recreated the session, and as expected, the session starts in the
> hybrid crawl.
> I can see some sync jobs are running in the gsyncd.log file and after a
> couple of hours,
> there are no such entries anymore.
>
> I switched into the log_level DEBUG mode to see what's going on:
> gluster volume masterVOlume geoRepHost:slaveVol config log_level DEBUG
>
> It seems to me that the xsync mode is in loop since the same files
> appear over and over again in the log-file.

Can you elaborate this? Where are the same files appearing? Are they
getting synced

Now we have two volume in this "loop"-state and the third volume also
> still has a broken geo-replication.
>

Is the worker status changing from initializing to faulty or initializing
to active/passive? Is any worker active?

> So any help is appreciated how to fix this or which information is
> required to find the root cause?
>
> As mentioned before, all these gathered information could be used to
> improve the geo-replication trouble-shooting documentation.
>
> Thanks in advance.
>
> Regards,
> Felix
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200709/cb659b8c/attachment.html>

From rkothiya at redhat.com  Thu Jul  9 12:46:32 2020
From: rkothiya at redhat.com (Rinku Kothiya)
Date: Thu, 9 Jul 2020 18:16:32 +0530
Subject: [Gluster-users] Announcing Gluster release 8
Message-ID: <CAMX-mG07dUydpK5gD4PMwDXHD3ymD=OUc_svT+mLHXVfEkpWcA@mail.gmail.com>

Hi,

The Gluster community is pleased to announce the release of 8.0, our latest
release.

This is a major release that includes a range of code improvements and
stability fixes along with a few features as noted below. A selection of
the key features and bugs addressed are documented in this [1] page.

*Announcements:*

1. Releases that receive maintenance updates post release 8 are 7 and 8 [2]

2. Release 8 will receive maintenance updates around the 10th of every
month for the first 3 months post release (i.e Aug'20, Sep'20, Oct'20).
    Post the initial 3 months, it will receive maintenance updates every 2
months till EOL.

3. For upgrading to release 8 refer to the release 8 upgrade guide [3].

   Make sure you are not using any of the following deprecated features :

- Block device (bd) xlator
- Decompounder feature
- Crypt xlator
- Symlink-cache xlator
- Stripe feature
- Tiering support (tier xlator and changetimerecorder)
- Glupy


*Highlights of this release are:*
*Highlights:*

- Several stability fixes addressing
* coverity, clang-scan, address sanitizer and valgrind reported issues
* removal of unused and hence, deprecated code and features

- Performance Improvements

*Features:*

- Implemented seek file operation for open-behind
- Now storage.reserve option will take size of disk as input instead of
percentage
- Added Functionality to enable log rotation for user serviceable
snapshot?s logs
- Mandatory locks enhancements in replicate subvolumes
- To validate other memory allocation implementations instead of libc?s
malloc added an option to build with tcmalloc library
- Integrated Thin-arbiter with GD1
- Client Handling of Elastic Clusters


*Major issues:*

- None

Bugs addressed are provided towards the end, in the release notes [1]


Thank you,
Gluster community

References:
[1] Release notes:
     https://docs.gluster.org/en/latest/release-notes/8.0/

[2] Release schedule:
     https://www.gluster.org/release-schedule/

[3] Upgrade guide to release-8:
     https://docs.gluster.org/en/latest/Upgrade-Guide/upgrade_to_8/

[4] Packages:

     Packages that will be available :

https://github.com/gluster/glusterdocs/blob/master/docs/Install-Guide/Community_Packages.md

     Packages at :
     https://download.gluster.org/pub/gluster/glusterfs/8/8.0/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200709/07dd6b1b/attachment.html>

From sacharya at redhat.com  Thu Jul  9 13:00:23 2020
From: sacharya at redhat.com (Shwetha Acharya)
Date: Thu, 9 Jul 2020 18:30:23 +0530
Subject: [Gluster-users] Setup GlusterFS geo-replication between Ubuntu
 server and CentOS server
In-Reply-To: <CACOwRaEy_=Bbo+BFzVCeG_+8tQ+a6nfVu8bvTwDPYRQ7L6i8NA@mail.gmail.com>
References: <CACOwRaEy_=Bbo+BFzVCeG_+8tQ+a6nfVu8bvTwDPYRQ7L6i8NA@mail.gmail.com>
Message-ID: <CAERh03rA9AEooQU4S1JHy7PFuSg_Tb3ieo6xwKueDJFXxLukbQ@mail.gmail.com>

Hi Duc,

Can we confirm if the gluster version  and python version are same across
the machines you are using?

Regards,
Shwetha

On Thu, Jul 9, 2020 at 12:56 AM Duc To Ho <duc.to.ho at gmail.com> wrote:

> Hello all,
>
> Recently I need to setup GlusterFS geo-replication between Ubuntu server
> and CentOs server. I setup following the guideline and everything was fine,
> checking with "gluster volume geo-replication gvol0 dev-3-15:gvol0 status"
> command returns me the info with Active status. But then when I started
> copying data to the master node, I got Exception error in log and status
> has become Faulty.
>
> I tried to set up the same in both CentOs-CentOs and Ubuntu-Ubuntu and
> there is no such issue.
>
> I wonder if anyone has set up the same case CentOS-Ubuntu and feedback if
> it is OK or had a similar issue and hopefully, the solution for it?
>
> Thank you and I look forward to hearing from you,
> Duc
>
> <nnlong at gmail.com>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200709/b2e05e42/attachment.html>

From sasundar at redhat.com  Sun Jul 12 02:44:03 2020
From: sasundar at redhat.com (Satheesaran Sundaramoorthi)
Date: Sun, 12 Jul 2020 08:14:03 +0530
Subject: [Gluster-users] gluster geo-replications fails to sync with IPV6
	hostnames
Message-ID: <CAK0rum+w005y=+ct0vmqU66cjj57VBFb9Qdn8QZHSKJtL2zcNg@mail.gmail.com>

Hello All,

While testing with glusterfs -6.0-37, I found that the geo-rep session is
getting created with IPv6 hostnames,
but the sync is not happening. I see traceback with gsyncd.log.

<snip>
[2020-07-11 09:29:20.735708] I [gsyncd(config-get):318:main] <top>:
Using session config
file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
[2020-07-11 09:29:21.346986] I [gsyncd(config-get):318:main] <top>:
Using session config
file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
[2020-07-11 09:29:21.515802] I [gsyncd(monitor):318:main] <top>: Using
session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
[2020-07-11 09:29:21.953314] E
[syncdutils(monitor):339:log_raise_exception] <top>: FAIL:
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 332, in main
    func(args)
  File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 60,
in subcmd_monitor
    return monitor.monitor(local, remote)
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line
431, in monitor
    return Monitor().multiplex(*distribute(local, remote))
  File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line
392, in distribute
    sbricks = svol.bricks
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 515, in ff
    rv = f(self, *a, **kw)
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line
916, in bricks
    return [bparse(b) for b in self.get('brick')]
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line
916, in <listcomp>
    return [bparse(b) for b in self.get('brick')]
  File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line
914, in bparse
    host, dirp = b.find("name").text.split(':', 2)
ValueError: too many values to unpack (expected 2)
</snip>

I have raised the github issue[1] for the same. Any help wrt to this
issue will help a lot.

[1] - https://github.com/gluster/glusterfs/issues/1366

Thanks,

Satheesaran S
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200712/6e5fb574/attachment.html>

From aravinda at kadalu.io  Sun Jul 12 07:28:19 2020
From: aravinda at kadalu.io (Aravinda VK)
Date: Sun, 12 Jul 2020 12:58:19 +0530
Subject: [Gluster-users] gluster geo-replications fails to sync with
 IPV6 hostnames
In-Reply-To: <CAK0rum+w005y=+ct0vmqU66cjj57VBFb9Qdn8QZHSKJtL2zcNg@mail.gmail.com>
References: <CAK0rum+w005y=+ct0vmqU66cjj57VBFb9Qdn8QZHSKJtL2zcNg@mail.gmail.com>
Message-ID: <D9C89014-17F6-4044-A164-483992EED34A@kadalu.io>

Hi Satheesaran,

Posted a patch to fix the parsing issue. Please check.

https://review.gluster.org/c/glusterfs/+/24706 <https://review.gluster.org/c/glusterfs/+/24706>

--
Aravinda Vishwanathapura
https://kadalu.io <https://kadalu.io/>

> On 12-Jul-2020, at 8:14 AM, Satheesaran Sundaramoorthi <sasundar at redhat.com> wrote:
> 
> Hello All,
> 
> While testing with glusterfs -6.0-37, I found that the geo-rep session is getting created with IPv6 hostnames,
> but the sync is not happening. I see traceback with gsyncd.log.
> <snip>
> [2020-07-11 09:29:20.735708] I [gsyncd(config-get):318:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
> [2020-07-11 09:29:21.346986] I [gsyncd(config-get):318:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
> [2020-07-11 09:29:21.515802] I [gsyncd(monitor):318:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
> [2020-07-11 09:29:21.953314] E [syncdutils(monitor):339:log_raise_exception] <top>: FAIL: 
> Traceback (most recent call last):
>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 332, in main
>     func(args)
>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 60, in subcmd_monitor
>     return monitor.monitor(local, remote)
>   File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 431, in monitor
>     return Monitor().multiplex(*distribute(local, remote))
>   File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 392, in distribute
>     sbricks = svol.bricks
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 515, in ff
>     rv = f(self, *a, **kw)
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 916, in bricks
>     return [bparse(b) for b in self.get('brick')]
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 916, in <listcomp>
>     return [bparse(b) for b in self.get('brick')]
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 914, in bparse
>     host, dirp = b.find("name").text.split(':', 2)
> ValueError: too many values to unpack (expected 2)
> </snip>
> 
> I have raised the github issue[1] for the same. Any help wrt to this issue will help a lot.
> 
> [1] - https://github.com/gluster/glusterfs/issues/1366 <https://github.com/gluster/glusterfs/issues/1366>
> Thanks,
> Satheesaran S
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users





-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200712/dd9f696a/attachment.html>

From sasundar at redhat.com  Mon Jul 13 03:41:10 2020
From: sasundar at redhat.com (Satheesaran Sundaramoorthi)
Date: Mon, 13 Jul 2020 09:11:10 +0530
Subject: [Gluster-users] gluster geo-replications fails to sync with
	IPV6 hostnames
In-Reply-To: <D9C89014-17F6-4044-A164-483992EED34A@kadalu.io>
References: <CAK0rum+w005y=+ct0vmqU66cjj57VBFb9Qdn8QZHSKJtL2zcNg@mail.gmail.com>
	<D9C89014-17F6-4044-A164-483992EED34A@kadalu.io>
Message-ID: <CAK0rumLwn=Tc1K_z=qBoKyWdQymhmxx5igQ2wGtQNzmNJ21M7g@mail.gmail.com>

Hello Aravinda,

Thanks for that patch. I will check that out.

-- Satheesaran S

On Sun, Jul 12, 2020 at 12:58 PM Aravinda VK <aravinda at kadalu.io> wrote:

> Hi Satheesaran,
>
> Posted a patch to fix the parsing issue. Please check.
>
> https://review.gluster.org/c/glusterfs/+/24706
>
> --
> Aravinda Vishwanathapura
> https://kadalu.io
>
> On 12-Jul-2020, at 8:14 AM, Satheesaran Sundaramoorthi <
> sasundar at redhat.com> wrote:
>
> Hello All,
>
> While testing with glusterfs -6.0-37, I found that the geo-rep session is
> getting created with IPv6 hostnames,
> but the sync is not happening. I see traceback with gsyncd.log.
>
> <snip>
> [2020-07-11 09:29:20.735708] I [gsyncd(config-get):318:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
> [2020-07-11 09:29:21.346986] I [gsyncd(config-get):318:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
> [2020-07-11 09:29:21.515802] I [gsyncd(monitor):318:main] <top>: Using session config file	path=/var/lib/glusterd/geo-replication/testvol_slave.lab.eng.blr.redhat.com_svol/gsyncd.conf
> [2020-07-11 09:29:21.953314] E [syncdutils(monitor):339:log_raise_exception] <top>: FAIL:
> Traceback (most recent call last):
>   File "/usr/libexec/glusterfs/python/syncdaemon/gsyncd.py", line 332, in main
>     func(args)
>   File "/usr/libexec/glusterfs/python/syncdaemon/subcmds.py", line 60, in subcmd_monitor
>     return monitor.monitor(local, remote)
>   File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 431, in monitor
>     return Monitor().multiplex(*distribute(local, remote))
>   File "/usr/libexec/glusterfs/python/syncdaemon/monitor.py", line 392, in distribute
>     sbricks = svol.bricks
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 515, in ff
>     rv = f(self, *a, **kw)
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 916, in bricks
>     return [bparse(b) for b in self.get('brick')]
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 916, in <listcomp>
>     return [bparse(b) for b in self.get('brick')]
>   File "/usr/libexec/glusterfs/python/syncdaemon/syncdutils.py", line 914, in bparse
>     host, dirp = b.find("name").text.split(':', 2)
> ValueError: too many values to unpack (expected 2)
> </snip>
>
> I have raised the github issue[1] for the same. Any help wrt to this issue will help a lot.
>
> [1] - https://github.com/gluster/glusterfs/issues/1366
>
> Thanks,
>
> Satheesaran S
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/5bb77fb9/attachment.html>

From jahernan at redhat.com  Mon Jul 13 07:17:20 2020
From: jahernan at redhat.com (Xavi Hernandez)
Date: Mon, 13 Jul 2020 09:17:20 +0200
Subject: [Gluster-users] gluster cmd output - how to format
In-Reply-To: <87c430eb-56b6-57d2-ef2a-f974b69cc5fe@yahoo.co.uk>
References: <87c430eb-56b6-57d2-ef2a-f974b69cc5fe.ref@yahoo.co.uk>
	<87c430eb-56b6-57d2-ef2a-f974b69cc5fe@yahoo.co.uk>
Message-ID: <CALtvDK7=VvDDEQmYPjwXfsjjT-yafjZA34_8-5-P2Wroy4QkEw@mail.gmail.com>

Hi.

If you need to parse input from a script, maybe it would be better to use
xml output:

# gluster --xml volume status


This should be easily parsable by any standard xml library or tool.

Xavi

On Thu, Jul 2, 2020 at 3:32 PM lejeczek <peljasz at yahoo.co.uk> wrote:

> hi guys
>
> Would you know if it's possible to format gluster cmd output?
> What frustrates me personally is "forced" line wrapping, as
> example of:
>
> $ gluster volume status
>
> many thanks, L.
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/31ac8928/attachment.html>

From shreyansh.shah at alpha-grep.com  Mon Jul 13 07:44:50 2020
From: shreyansh.shah at alpha-grep.com (Shreyansh Shah)
Date: Mon, 13 Jul 2020 13:14:50 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAD+dzQf2cWoejX3+eTHM043pYJnm2ZTf9PDH2URcFETN+1ZnUw@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<CAH1hu+4w7MVjcD_xKtCUGOpVy36DCteYv21E0gdT85ZQi_H-mw@mail.gmail.com>
	<CAFx2kQ03h=NTRXrkG2=sP7z4PXdAThbXqw9r2NCbZ2SebYv_jQ@mail.gmail.com>
	<CAH1hu+6H7A5SfeYL-Uc3OAfci4YhZL9uoOUOdm5mi_M1STMOwQ@mail.gmail.com>
	<CAFx2kQ3H9A4fbKbLhd1iCcJ-k8M4QGj+tyGumPC1sx-VBkzADA@mail.gmail.com>
	<CAH1hu+52marGwdRmpxBKJweLW6e3J-uLQSVKRyKRP_3PZ=0FoA@mail.gmail.com>
	<CAFx2kQ18jJ4Zb6iy0TB=Bmk6BQ5azhywcM3g4c1dX4SttddgZA@mail.gmail.com>
	<CAH1hu+7A4o1mXeHsje-jCkpsPsS0AJcTn5V5dDRsw-7DLw358Q@mail.gmail.com>
	<CAFx2kQ0RguVjus1JggwyZNKTJLv-zWFc7zHTM2fkyVSxoU_=0Q@mail.gmail.com>
	<CAH1hu+4D4SH1LT0=juPw4QU3zy33HVuOytMKn1mrjx0c6Mdtsg@mail.gmail.com>
	<CAD+dzQf2cWoejX3+eTHM043pYJnm2ZTf9PDH2URcFETN+1ZnUw@mail.gmail.com>
Message-ID: <CAH1hu+7UegzS2XDGKDaQ6vb_mR0=u1HXQsexBjnjyqRFKxBqvA@mail.gmail.com>

Hi Barak,
Hope you had a great weekend.
Did you get a chance to look into the issue and figure out the problem?

On Wed, Jul 8, 2020 at 11:32 AM Artem Russakovskii <archon810 at gmail.com>
wrote:

> I think it'd be extremely helpful if gluster had a feature to grab all the
> necessary logs/debug info (maybe a few variations depending on the bug) so
> that all the user would have to do is enter a simple command and have
> gluster generate the whole bug report, ready to be sent to to the gluster
> team.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Tue, Jul 7, 2020 at 1:47 AM Shreyansh Shah <
> shreyansh.shah at alpha-grep.com> wrote:
>
>> Sounds good, thank you.
>>
>> On Tue, Jul 7, 2020 at 2:12 PM Barak Sason Rofman <bsasonro at redhat.com>
>> wrote:
>>
>>> Thanks Shreyansh,
>>>
>>> I'll look into it, however I'll likely need some help from more senior
>>> team members to perform RCA.
>>> I'll update once I have new insights.
>>>
>>> My regards,
>>>
>>> On Tue, Jul 7, 2020 at 11:40 AM Shreyansh Shah <
>>> shreyansh.shah at alpha-grep.com> wrote:
>>>
>>>> Hi Barak,
>>>> Thanks for looking into this and helping me out,
>>>> The fix-layout was successful, and I ran a rebalance after
>>>> completion of fix-layout.
>>>> The rebalance status though did show failure for 3 nodes.
>>>>
>>>> On Tue, Jul 7, 2020 at 2:07 PM Barak Sason Rofman <bsasonro at redhat.com>
>>>> wrote:
>>>>
>>>>> Greetings again Shreyansh,
>>>>>
>>>>> I'm indeed seeing a lot of errors in the log file - still unsure about
>>>>> the RC.
>>>>> You mentioned that prior to running rebalance you ran fix-layout, was
>>>>> the fix-layout successful?
>>>>> Another question - did you wait until fix-layout was completed before
>>>>> running rebalance?
>>>>>
>>>>> My thanks,
>>>>>
>>>>> On Mon, Jul 6, 2020 at 9:33 PM Shreyansh Shah <
>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Attaching rebalance logs
>>>>>> FYI, we ran "gluster rebalance fix-layout" followed by "gluster
>>>>>> rebalance" on 20200701 and today we again ran "gluster rebalance fix-layout"
>>>>>>
>>>>>>
>>>>>> PFA
>>>>>>
>>>>>> On Mon, Jul 6, 2020 at 11:08 PM Barak Sason Rofman <
>>>>>> bsasonro at redhat.com> wrote:
>>>>>>
>>>>>>> I think it would be best.
>>>>>>> As I can't say at this point where the problem is originating from,
>>>>>>> brick logs might also be necessary (I assume I would have a better picture
>>>>>>> once I have the rebalance logs).
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> On Mon, Jul 6, 2020 at 8:16 PM Shreyansh Shah <
>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>
>>>>>>>> Hi Barak,
>>>>>>>> Can provide the rebalance logs. Do you require all the brick logs
>>>>>>>> (14 in total)?
>>>>>>>>
>>>>>>>> On Mon, Jul 6, 2020 at 10:43 PM Barak Sason Rofman <
>>>>>>>> bsasonro at redhat.com> wrote:
>>>>>>>>
>>>>>>>>> Greetings Shreyansh,
>>>>>>>>>
>>>>>>>>> Off-hand I can't come up with a reason for these failures.
>>>>>>>>> In order to start looking into this, access to the full rebalance
>>>>>>>>> logs is required (possibly brick logs as well).
>>>>>>>>> Can you provide those?
>>>>>>>>>
>>>>>>>>> My regards,
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jul 6, 2020 at 11:41 AM Shreyansh Shah <
>>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>> Did anyone get a chance to look into this?
>>>>>>>>>>
>>>>>>>>>> On Thu, Jul 2, 2020 at 8:09 PM Shreyansh Shah <
>>>>>>>>>> shreyansh.shah at alpha-grep.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi All,
>>>>>>>>>>>
>>>>>>>>>>> *We are facing "Mismatching layouts for <directory>,gfid = <id>"
>>>>>>>>>>> errors.*
>>>>>>>>>>>
>>>>>>>>>>> We have a distributed glusterfs 5.10, no replication, 2 bricks
>>>>>>>>>>> (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to
>>>>>>>>>>> the existing setup.
>>>>>>>>>>> Post that we did a rebalance fix-layout and then a rebalance
>>>>>>>>>>> (which is currently still in progress). The status shows "failed" on
>>>>>>>>>>> certain bricks but "in progress" for others. Adding output for gluster
>>>>>>>>>>> rebalance status below.
>>>>>>>>>>>
>>>>>>>>>>> The glusterfs client logs are flooded with "Mismatching layouts
>>>>>>>>>>> for <directory>,gfid = <id>"
>>>>>>>>>>> The performance too seems to have degraded due to this, even
>>>>>>>>>>> basic commands like `cd` and `ls` are taking more than a minute compared to
>>>>>>>>>>> sub-second number before brick addition.
>>>>>>>>>>> Apart from that we also experienced many binaries and files
>>>>>>>>>>> giving error stale file handle error even though the files were present.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *gluster rebalance status :*
>>>>>>>>>>>
>>>>>>>>>>> Node Rebalanced-files          size       scanned      failures
>>>>>>>>>>>       skipped               status  run time in h:m:s
>>>>>>>>>>> ---------      -----------   -----------   -----------
>>>>>>>>>>> -----------   -----------         ------------     --------------
>>>>>>>>>>> localhost              176         3.5GB         12790
>>>>>>>>>>>   0          8552          in progress       21:36:01
>>>>>>>>>>> 10.132.0.72             8232       394.8GB         19995
>>>>>>>>>>>    21            26               failed       14:50:30
>>>>>>>>>>> 10.132.0.44            12625         1.0TB         50023
>>>>>>>>>>>     1         10202          in progress       21:36:00
>>>>>>>>>>> 10.132.0.3            21982       956.8GB         79145
>>>>>>>>>>>     1         34571          in progress       21:36:00
>>>>>>>>>>> 10.132.0.9             7975       355.8GB         20157
>>>>>>>>>>>     6          1522               failed       14:51:45
>>>>>>>>>>> 10.132.0.73             6293       394.5GB         26414
>>>>>>>>>>>   151          8085               failed       14:51:45
>>>>>>>>>>> 10.132.0.70             6480       477.1GB         21058
>>>>>>>>>>>    27          1787               failed       14:50:32
>>>>>>>>>>> Estimated time left for rebalance to complete :      130:56:28
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Logs from one of the clients below:*
>>>>>>>>>>>
>>>>>>>>>>> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
>>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>>> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
>>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>>> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>>>>>>>>>>> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
>>>>>>>>>>> layout - 2454306944 - 2761060379 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
>>>>>>>>>>> layout - 2761060380 - 3067813815 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
>>>>>>>>>>> layout - 3681390552 - 3988143987 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
>>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>>>>>>>>>>> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
>>>>>>>>>>> layout - 920400036 - 1227153471 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
>>>>>>>>>>> layout - 1840730208 - 2147483643 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
>>>>>>>>>>> layout - 1533976772 - 1840730207 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
>>>>>>>>>>> layout - 613646600 - 920400035 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>>>>>>>>>>> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
>>>>>>>>>>> data-client-7 (hashed subvol is data-client-17)
>>>>>>>>>>> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>>> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>>> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_CDS_2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
>>>>>>>>>>> data-client-6 (hashed subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>>> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
>>>>>>>>>>> data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>>> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>>> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_3_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
>>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>>> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>>> on data-client-15 (hashed subvol is data-client-7)
>>>>>>>>>>> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_4_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
>>>>>>>>>>> data-client-14 (hashed subvol is data-client-7)
>>>>>>>>>>> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>>> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/BSE_EQ_6_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>>> on data-client-15 (hashed subvol is data-client-8)
>>>>>>>>>>> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
>>>>>>>>>>> (hashed subvol is data-client-6)
>>>>>>>>>>> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-9)
>>>>>>>>>>> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
>>>>>>>>>>> subvol is data-client-6)
>>>>>>>>>>> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
>>>>>>>>>>> data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>>> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
>>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>>> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
>>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSECM3_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-19)
>>>>>>>>>>> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
>>>>>>>>>>> subvol is data-client-10)
>>>>>>>>>>> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>>> on data-client-11 (hashed subvol is data-client-3)
>>>>>>>>>>> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
>>>>>>>>>>> (hashed subvol is data-client-8)
>>>>>>>>>>> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>>> on data-client-9 (hashed subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
>>>>>>>>>>> (hashed subvol is data-client-2)
>>>>>>>>>>> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>>> on data-client-10 (hashed subvol is data-client-2)
>>>>>>>>>>> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
>>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>>> on data-client-2 (hashed subvol is data-client-11)
>>>>>>>>>>> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>>> on data-client-7 (hashed subvol is data-client-16)
>>>>>>>>>>> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>>>>>>>>>>> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
>>>>>>>>>>> subvol is data-client-15)
>>>>>>>>>>> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
>>>>>>>>>>> data-client-17 (hashed subvol is data-client-10)
>>>>>>>>>>> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>>>>>>>>>>> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
>>>>>>>>>>> subvol is data-client-18)
>>>>>>>>>>> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
>>>>>>>>>>> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
>>>>>>>>>>> deletion of stale linkfile
>>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
>>>>>>>>>>> (hashed subvol is data-client-19)
>>>>>>>>>>> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
>>>>>>>>>>> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
>>>>>>>>>>> returned with op_ret -> 0 and op-errno -> 0 for
>>>>>>>>>>> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>>>>>>>>>>> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
>>>>>>>>>>> 613576736 - 920330171 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>>> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
>>>>>>>>>>> 920330172 - 1227083607 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>>>>>>>>>>> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
>>>>>>>>>>> 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>>> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
>>>>>>>>>>> 3681390552 - 3988143987 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX, gfid =
>>>>>>>>>>> fff324f2-f855-4881-b77c-81e856522373
>>>>>>>>>>> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
>>>>>>>>>>> layout - 3067883680 - 3374637115 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
>>>>>>>>>>> layout - 3374637116 - 3681390551 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
>>>>>>>>>>> - 306753435 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
>>>>>>>>>>> layout - 3988213852 - 4294967295 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
>>>>>>>>>>> f8447150-4801-4188-add9-ea295bb88729
>>>>>>>>>>> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
>>>>>>>>>>> 3681460416 - 3988213851 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>>> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
>>>>>>>>>>> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
>>>>>>>>>>> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
>>>>>>>>>>> 3988213852 - 4294967295 - 4159036738
>>>>>>>>>>> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
>>>>>>>>>>> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
>>>>>>>>>>> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
>>>>>>>>>>> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>>>>>>>>>>> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
>>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>>> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
>>>>>>>>>>> (2cb54500-814d-4e85-83e7-e33d9440b18d)
>>>>>>>>>>> (hash=data-client-6/cache=data-client-18) =>
>>>>>>>>>>> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
>>>>>>>>>>> (hash=data-client-6/cache=<nul>)
>>>>>>>>>>> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
>>>>>>>>>>> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
>>>>>>>>>>> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
>>>>>>>>>>> (99938ee6-6986-4123-9d72-ec09e2310b4f)
>>>>>>>>>>> (hash=data-client-17/cache=data-client-18) =>
>>>>>>>>>>> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
>>>>>>>>>>> (hash=data-client-17/cache=<nul>)
>>>>>>>>>>> ....
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Please look into this at top-priority if possible.
>>>>>>>>>>> Let me know if anything else is required.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Shreyansh Shah
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>> Shreyansh Shah
>>>>>>>>>> ________
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Community Meeting Calendar:
>>>>>>>>>>
>>>>>>>>>> Schedule -
>>>>>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>>>>>
>>>>>>>>>> Gluster-users mailing list
>>>>>>>>>> Gluster-users at gluster.org
>>>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> *Barak Sason Rofman*
>>>>>>>>>
>>>>>>>>> Gluster Storage Development
>>>>>>>>>
>>>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>>>
>>>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>>>
>>>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>>>> M: *+972-52-4326355*
>>>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>>>> <https://red.ht/sig>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>> Shreyansh Shah
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> *Barak Sason Rofman*
>>>>>>>
>>>>>>> Gluster Storage Development
>>>>>>>
>>>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>>>
>>>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>>>
>>>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>>>> M: *+972-52-4326355*
>>>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>>>> <https://www.facebook.com/redhat.il/>
>>>>>>> <https://red.ht/sig>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>> Shreyansh Shah
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *Barak Sason Rofman*
>>>>>
>>>>> Gluster Storage Development
>>>>>
>>>>> Red Hat Israel <https://www.redhat.com/>
>>>>>
>>>>> 34 Jerusalem rd. Ra'anana, 43501
>>>>>
>>>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>>>> M: *+972-52-4326355*
>>>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>>>> <https://www.facebook.com/redhat.il/>
>>>>> <https://red.ht/sig>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Shreyansh Shah
>>>>
>>>
>>>
>>> --
>>> *Barak Sason Rofman*
>>>
>>> Gluster Storage Development
>>>
>>> Red Hat Israel <https://www.redhat.com/>
>>>
>>> 34 Jerusalem rd. Ra'anana, 43501
>>>
>>> bsasonro at redhat.com <adi at redhat.com>    T: *+972-9-7692304*
>>> M: *+972-52-4326355*
>>> @RedHat <https://twitter.com/redhat>   Red Hat
>>> <https://www.linkedin.com/company/red-hat>  Red Hat
>>> <https://www.facebook.com/redhat.il/>
>>> <https://red.ht/sig>
>>>
>>
>>
>> --
>> Regards,
>> Shreyansh Shah
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>

-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/5d3fd549/attachment.html>

From spalai at redhat.com  Mon Jul 13 08:05:26 2020
From: spalai at redhat.com (Susant Palai)
Date: Mon, 13 Jul 2020 13:35:26 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
Message-ID: <2A780DA5-52D9-4113-9456-F08C42B31B78@redhat.com>

The log messages are fine. Since you added a new brick, the client is responding to that by syncing its in-memory layout with latest server layout. The performance drop could be because of locks taken during this layout sync.


> On 02-Jul-2020, at 20:09, Shreyansh Shah <shreyansh.shah at alpha-grep.com> wrote:
> 
> Hi All,
> 
> We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.
> 
> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to the existing setup.
> Post that we did a rebalance fix-layout and then a rebalance (which is currently still in progress). The status shows "failed" on certain bricks but "in progress" for others. Adding output for gluster rebalance status below.
> 
> The glusterfs client logs are flooded with "Mismatching layouts for <directory>,gfid = <id>"
> The performance too seems to have degraded due to this, even basic commands like `cd` and `ls` are taking more than a minute compared to sub-second number before brick addition.
> Apart from that we also experienced many binaries and files giving error stale file handle error even though the files were present.
> 
> 
> gluster rebalance status :
> 
> Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
> ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
> localhost              176         3.5GB         12790             0          8552          in progress       21:36:01
> 10.132.0.72             8232       394.8GB         19995            21            26               failed       14:50:30
> 10.132.0.44            12625         1.0TB         50023             1         10202          in progress       21:36:00
> 10.132.0.3            21982       956.8GB         79145             1         34571          in progress       21:36:00
> 10.132.0.9             7975       355.8GB         20157             6          1522               failed       14:51:45
> 10.132.0.73             6293       394.5GB         26414           151          8085               failed       14:51:45
> 10.132.0.70             6480       477.1GB         21058            27          1787               failed       14:50:32
> Estimated time left for rebalance to complete :      130:56:28
> 
> 
> Logs from one of the clients below:
> 
> [2020-07-02 12:30:14.971916] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:14.971935] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032013] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.032059] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032107] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.032153] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.093329] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk layout - 2454306944 - 2761060379 - 4159036738
> [2020-07-02 12:30:15.093373] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.093460] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:15.093515] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151063] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.151108] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151149] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.151162] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.424321] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk layout - 920400036 - 1227153471 - 4159036738
> [2020-07-02 12:30:15.424380] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424456] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk layout - 1840730208 - 2147483643 - 4159036738
> [2020-07-02 12:30:15.424484] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424525] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk layout - 1533976772 - 1840730207 - 4159036738
> [2020-07-02 12:30:15.424542] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424596] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk layout - 613646600 - 920400035 - 4159036738
> [2020-07-02 12:30:15.424607] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:16.004482] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on data-client-7 (hashed subvol is data-client-17)
> [2020-07-02 12:30:16.005523] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_DATA.dat
> [2020-07-02 12:30:16.531047] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:16.532086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_METADATA.dat
> [2020-07-02 12:30:18.733229] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:18.734421] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_DATA.dat
> [2020-07-02 12:30:19.171930] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:19.172901] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_METADATA.dat
> [2020-07-02 12:30:21.028495] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:21.029836] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_DATA.dat
> [2020-07-02 12:30:21.127648] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:21.128713] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_METADATA.dat
> [2020-07-02 12:30:21.201126] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.201928] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_DATA.dat
> [2020-07-02 12:30:21.566158] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:21.567123] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_METADATA.dat
> [2020-07-02 12:30:21.649357] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:21.661381] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_DATA.dat
> [2020-07-02 12:30:21.748937] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat on data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.749481] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_METADATA.dat
> [2020-07-02 12:30:21.898593] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on data-client-14 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.899442] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_DATA.dat
> [2020-07-02 12:30:22.039337] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:22.040086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_METADATA.dat
> [2020-07-02 12:30:22.501877] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat on data-client-15 (hashed subvol is data-client-8)
> [2020-07-02 12:30:22.502712] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_DATA.dat
> [2020-07-02 12:30:22.782577] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 (hashed subvol is data-client-6)
> [2020-07-02 12:30:22.783777] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_METADATA.dat
> [2020-07-02 12:30:23.146847] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:23.148009] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_DATA.dat
> [2020-07-02 12:30:23.229290] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed subvol is data-client-6)
> [2020-07-02 12:30:23.230151] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_METADATA.dat
> [2020-07-02 12:30:23.889520] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:23.896618] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_DATA.dat
> [2020-07-02 12:30:24.093017] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:24.094117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_METADATA.dat
> [2020-07-02 12:30:24.345257] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:24.346234] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_DATA.dat
> [2020-07-02 12:30:24.425835] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:24.426880] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_METADATA.dat
> [2020-07-02 12:30:25.158718] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:25.159619] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> [2020-07-02 12:30:25.531479] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed subvol is data-client-10)
> [2020-07-02 12:30:25.540569] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
> [2020-07-02 12:30:25.771692] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:25.772610] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> [2020-07-02 12:30:25.866118] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 (hashed subvol is data-client-8)
> [2020-07-02 12:30:25.866917] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
> [2020-07-02 12:30:26.424386] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:26.425309] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> [2020-07-02 12:30:26.818852] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:26.819890] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
> [2020-07-02 12:30:27.352405] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:27.352914] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> [2020-07-02 12:30:27.521286] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed subvol is data-client-18)
> [2020-07-02 12:30:27.522325] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
> [2020-07-02 12:30:28.566634] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat on data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:28.579295] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> [2020-07-02 12:30:28.958028] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:28.959102] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> [2020-07-02 12:30:29.012429] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:29.013416] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
> [2020-07-02 12:30:29.396716] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:29.397740] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSDATA.dat
> [2020-07-02 12:30:29.556312] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:29.557197] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
> [2020-07-02 12:30:30.605354] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:30.606117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
> [2020-07-02 12:30:31.559206] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - 613576736 - 920330171 - 4159036738
> [2020-07-02 12:30:31.559255] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.569025] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - 920330172 - 1227083607 - 4159036738
> [2020-07-02 12:30:31.569067] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.701849] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.701895] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.738464] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:31.738507] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.857102] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk layout - 3067883680 - 3374637115 - 4159036738
> [2020-07-02 12:30:31.857147] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.857180] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.857197] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917705] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 - 306753435 - 4159036738
> [2020-07-02 12:30:31.917781] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917855] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk layout - 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:31.917874] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:32.390945] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - 3681460416 - 3988213851 - 4159036738
> [2020-07-02 12:30:32.390998] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:30:32.391056] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:32.391075] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:33:50.915279] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 (2cb54500-814d-4e85-83e7-e33d9440b18d) (hash=data-client-6/cache=data-client-18) => /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) (hash=data-client-6/cache=<nul>)
> [2020-07-02 12:34:09.799586] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k (99938ee6-6986-4123-9d72-ec09e2310b4f) (hash=data-client-17/cache=data-client-18) => /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) (hash=data-client-17/cache=<nul>)
> ....
> 
> 
> Please look into this at top-priority if possible.
> Let me know if anything else is required.
> 
> 
> -- 
> Regards,
> Shreyansh Shah
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/e1bea7ca/attachment.html>

From shreyansh.shah at alpha-grep.com  Mon Jul 13 10:40:21 2020
From: shreyansh.shah at alpha-grep.com (Shreyansh Shah)
Date: Mon, 13 Jul 2020 16:10:21 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <2A780DA5-52D9-4113-9456-F08C42B31B78@redhat.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<2A780DA5-52D9-4113-9456-F08C42B31B78@redhat.com>
Message-ID: <CAH1hu+7KZ_Z0N7AeFwsYSgdGEFSab=tEC4RxzaLF+Pjau+FrWQ@mail.gmail.com>

Hi Susant,
Thank you for your response. We are experience performance impact till
date, is this expected?
Is there any way though which we can sync the complete new layout on the
clients?

On Mon, Jul 13, 2020 at 1:35 PM Susant Palai <spalai at redhat.com> wrote:

> The log messages are fine. Since you added a new brick, the client is
> responding to that by syncing its in-memory layout with latest server
> layout. The performance drop could be because of locks taken during this
> layout sync.
>
>
> On 02-Jul-2020, at 20:09, Shreyansh Shah <shreyansh.shah at alpha-grep.com>
> wrote:
>
> Hi All,
>
> *We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.*
>
> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each)
> on each node, 7 nodes in total. We added new bricks yesterday to the
> existing setup.
> Post that we did a rebalance fix-layout and then a rebalance (which is
> currently still in progress). The status shows "failed" on certain bricks
> but "in progress" for others. Adding output for gluster rebalance status
> below.
>
> The glusterfs client logs are flooded with "Mismatching layouts for
> <directory>,gfid = <id>"
> The performance too seems to have degraded due to this, even basic
> commands like `cd` and `ls` are taking more than a minute compared to
> sub-second number before brick addition.
> Apart from that we also experienced many binaries and files giving error
> stale file handle error even though the files were present.
>
>
> *gluster rebalance status :*
>
> Node Rebalanced-files          size       scanned      failures
> skipped               status  run time in h:m:s
> ---------      -----------   -----------   -----------   -----------
> -----------         ------------     --------------
> localhost              176         3.5GB         12790             0
>    8552          in progress       21:36:01
> 10.132.0.72             8232       394.8GB         19995            21
>        26               failed       14:50:30
> 10.132.0.44            12625         1.0TB         50023             1
>     10202          in progress       21:36:00
> 10.132.0.3            21982       956.8GB         79145             1
>     34571          in progress       21:36:00
> 10.132.0.9             7975       355.8GB         20157             6
>      1522               failed       14:51:45
> 10.132.0.73             6293       394.5GB         26414           151
>      8085               failed       14:51:45
> 10.132.0.70             6480       477.1GB         21058            27
>      1787               failed       14:50:32
> Estimated time left for rebalance to complete :      130:56:28
>
>
> *Logs from one of the clients below:*
>
> [2020-07-02 12:30:14.971916] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk
> layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:14.971935] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032013] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk
> layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.032059] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.032107] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.032153] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
> [2020-07-02 12:30:15.093329] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk
> layout - 2454306944 - 2761060379 - 4159036738
> [2020-07-02 12:30:15.093373] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.093460] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk
> layout - 2761060380 - 3067813815 - 4159036738
> [2020-07-02 12:30:15.093515] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151063] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk
> layout - 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:15.151108] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.151149] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:15.151162] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
> [2020-07-02 12:30:15.424321] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk
> layout - 920400036 - 1227153471 - 4159036738
> [2020-07-02 12:30:15.424380] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424456] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk
> layout - 1840730208 - 2147483643 - 4159036738
> [2020-07-02 12:30:15.424484] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424525] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk
> layout - 1533976772 - 1840730207 - 4159036738
> [2020-07-02 12:30:15.424542] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:15.424596] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk
> layout - 613646600 - 920400035 - 4159036738
> [2020-07-02 12:30:15.424607] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
> [2020-07-02 12:30:16.004482] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on
> data-client-7 (hashed subvol is data-client-17)
> [2020-07-02 12:30:16.005523] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_1_DATA.dat
> [2020-07-02 12:30:16.531047] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat
> on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:16.532086] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_1_METADATA.dat
> [2020-07-02 12:30:18.733229] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on
> data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:18.734421] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_2_DATA.dat
> [2020-07-02 12:30:19.171930] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat
> on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:19.172901] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_CDS_2_METADATA.dat
> [2020-07-02 12:30:21.028495] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on
> data-client-6 (hashed subvol is data-client-15)
> [2020-07-02 12:30:21.029836] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_2_DATA.dat
> [2020-07-02 12:30:21.127648] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat
> on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:21.128713] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_2_METADATA.dat
> [2020-07-02 12:30:21.201126] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on
> data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.201928] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_3_DATA.dat
> [2020-07-02 12:30:21.566158] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat
> on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:21.567123] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_3_METADATA.dat
> [2020-07-02 12:30:21.649357] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on
> data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:21.661381] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_4_DATA.dat
> [2020-07-02 12:30:21.748937] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat
> on data-client-15 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.749481] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_4_METADATA.dat
> [2020-07-02 12:30:21.898593] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on
> data-client-14 (hashed subvol is data-client-7)
> [2020-07-02 12:30:21.899442] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_6_DATA.dat
> [2020-07-02 12:30:22.039337] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat
> on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:22.040086] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/BSE_EQ_6_METADATA.dat
> [2020-07-02 12:30:22.501877] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat
> on data-client-15 (hashed subvol is data-client-8)
> [2020-07-02 12:30:22.502712] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECDS1_DATA.dat
> [2020-07-02 12:30:22.782577] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11
> (hashed subvol is data-client-6)
> [2020-07-02 12:30:22.783777] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECDS1_METADATA.dat
> [2020-07-02 12:30:23.146847] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on
> data-client-17 (hashed subvol is data-client-9)
> [2020-07-02 12:30:23.148009] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM1_DATA.dat
> [2020-07-02 12:30:23.229290] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed
> subvol is data-client-6)
> [2020-07-02 12:30:23.230151] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM1_METADATA.dat
> [2020-07-02 12:30:23.889520] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on
> data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:23.896618] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM2_DATA.dat
> [2020-07-02 12:30:24.093017] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:24.094117] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM2_METADATA.dat
> [2020-07-02 12:30:24.345257] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on
> data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:24.346234] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM3_DATA.dat
> [2020-07-02 12:30:24.425835] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:24.426880] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSECM3_METADATA.dat
> [2020-07-02 12:30:25.158718] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> on data-client-9 (hashed subvol is data-client-19)
> [2020-07-02 12:30:25.159619] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO1_DATA.dat
> [2020-07-02 12:30:25.531479] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed
> subvol is data-client-10)
> [2020-07-02 12:30:25.540569] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
> [2020-07-02 12:30:25.771692] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> on data-client-11 (hashed subvol is data-client-3)
> [2020-07-02 12:30:25.772610] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO2_DATA.dat
> [2020-07-02 12:30:25.866118] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15
> (hashed subvol is data-client-8)
> [2020-07-02 12:30:25.866917] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
> [2020-07-02 12:30:26.424386] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> on data-client-9 (hashed subvol is data-client-18)
> [2020-07-02 12:30:26.425309] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO3_DATA.dat
> [2020-07-02 12:30:26.818852] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10
> (hashed subvol is data-client-2)
> [2020-07-02 12:30:26.819890] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
> [2020-07-02 12:30:27.352405] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> on data-client-10 (hashed subvol is data-client-2)
> [2020-07-02 12:30:27.352914] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO4_DATA.dat
> [2020-07-02 12:30:27.521286] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed
> subvol is data-client-18)
> [2020-07-02 12:30:27.522325] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
> [2020-07-02 12:30:28.566634] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> on data-client-2 (hashed subvol is data-client-11)
> [2020-07-02 12:30:28.579295] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO5_DATA.dat
> [2020-07-02 12:30:28.958028] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> on data-client-7 (hashed subvol is data-client-16)
> [2020-07-02 12:30:28.959102] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO6_DATA.dat
> [2020-07-02 12:30:29.012429] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed
> subvol is data-client-15)
> [2020-07-02 12:30:29.013416] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
> [2020-07-02 12:30:29.396716] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on
> data-client-17 (hashed subvol is data-client-10)
> [2020-07-02 12:30:29.397740] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSEFO_BSE_TSDATA.dat
> [2020-07-02 12:30:29.556312] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed
> subvol is data-client-18)
> [2020-07-02 12:30:29.557197] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
> [2020-07-02 12:30:30.605354] I [MSGID: 109045]
> [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting
> deletion of stale linkfile
> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9
> (hashed subvol is data-client-19)
> [2020-07-02 12:30:30.606117] I [MSGID: 109069]
> [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink
> returned with op_ret -> 0 and op-errno -> 0 for
> /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
> [2020-07-02 12:30:31.559206] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout -
> 613576736 - 920330171 - 4159036738
> [2020-07-02 12:30:31.559255] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.569025] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout -
> 920330172 - 1227083607 - 4159036738
> [2020-07-02 12:30:31.569067] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
> [2020-07-02 12:30:31.701849] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout -
> 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.701895] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX, gfid =
> fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.738464] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout -
> 3681390552 - 3988143987 - 4159036738
> [2020-07-02 12:30:31.738507] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX, gfid =
> fff324f2-f855-4881-b77c-81e856522373
> [2020-07-02 12:30:31.857102] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk
> layout - 3067883680 - 3374637115 - 4159036738
> [2020-07-02 12:30:31.857147] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.857180] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk
> layout - 3374637116 - 3681390551 - 4159036738
> [2020-07-02 12:30:31.857197] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917705] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0
> - 306753435 - 4159036738
> [2020-07-02 12:30:31.917781] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:31.917855] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk
> layout - 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:31.917874] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid =
> f8447150-4801-4188-add9-ea295bb88729
> [2020-07-02 12:30:32.390945] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout -
> 3681460416 - 3988213851 - 4159036738
> [2020-07-02 12:30:32.390998] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:30:32.391056] I [MSGID: 109064]
> [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol:
> data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout -
> 3988213852 - 4294967295 - 4159036738
> [2020-07-02 12:30:32.391075] I [MSGID: 109018]
> [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for
> /processed_data/Indexes/NSEINDEX/NIFTY, gfid =
> b2d4deb7-c58c-4046-b6f2-7c7f44d71311
> [2020-07-02 12:33:50.915279] I [MSGID: 109066]
> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
> /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2
> (2cb54500-814d-4e85-83e7-e33d9440b18d)
> (hash=data-client-6/cache=data-client-18) =>
> /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null))
> (hash=data-client-6/cache=<nul>)
> [2020-07-02 12:34:09.799586] I [MSGID: 109066]
> [dht-rename.c:1922:dht_rename] 4-data-dht: renaming
> /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k
> (99938ee6-6986-4123-9d72-ec09e2310b4f)
> (hash=data-client-17/cache=data-client-18) =>
> /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null))
> (hash=data-client-17/cache=<nul>)
> ....
>
>
> Please look into this at top-priority if possible.
> Let me know if anything else is required.
>
>
> --
> Regards,
> Shreyansh Shah
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>

-- 
Regards,
Shreyansh Shah
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/2c0b89ea/attachment.html>

From spalai at redhat.com  Mon Jul 13 10:49:58 2020
From: spalai at redhat.com (Susant Palai)
Date: Mon, 13 Jul 2020 16:19:58 +0530
Subject: [Gluster-users] "Mismatching layouts" in glusterfs client logs
 after new brick addition and rebalance
In-Reply-To: <CAH1hu+7KZ_Z0N7AeFwsYSgdGEFSab=tEC4RxzaLF+Pjau+FrWQ@mail.gmail.com>
References: <CAH1hu+4Cx0=7wjLysWxiT757dc+XYchTXPRNzitPD5YZtutPjw@mail.gmail.com>
	<2A780DA5-52D9-4113-9456-F08C42B31B78@redhat.com>
	<CAH1hu+7KZ_Z0N7AeFwsYSgdGEFSab=tEC4RxzaLF+Pjau+FrWQ@mail.gmail.com>
Message-ID: <9D7DBD96-1623-497E-9079-207C433AE4FC@redhat.com>

I missed the latency number in the report. You said it takes around a minute to do ?CD?. Is that still the case?

If yes, could you get the follow these steps to get statedump.
a - kill -USR2 <pid> (enable latency capture)
b - Initiate the operation (Do once for "CD dir")
c - take statedump (kill -USR1 <pid> or gluster commands)
d - kill -USR2 <pid> (disablee latency capture)



> On 13-Jul-2020, at 16:10, Shreyansh Shah <shreyansh.shah at alpha-grep.com> wrote:
> 
> Hi Susant,
> Thank you for your response. We are experience performance impact till date, is this expected?
> Is there any way though which we can sync the complete new layout on the clients?

The client gets its layout updated with a lookup. Generally when you do a ls on directory, it will sync the layout. (You can trigger ls -lR once rebalance is complete. But you need not necessarily do this. The lookup gets triggered from kernel as part of a fop and the directory will be updated with the layout)

> On Mon, Jul 13, 2020 at 1:35 PM Susant Palai <spalai at redhat.com <mailto:spalai at redhat.com>> wrote:
> The log messages are fine. Since you added a new brick, the client is responding to that by syncing its in-memory layout with latest server layout. The performance drop could be because of locks taken during this layout sync.
> 
> 
>> On 02-Jul-2020, at 20:09, Shreyansh Shah <shreyansh.shah at alpha-grep.com <mailto:shreyansh.shah at alpha-grep.com>> wrote:
>> 
>> Hi All,
>> 
>> We are facing "Mismatching layouts for <directory>,gfid = <id>" errors.
>> 
>> We have a distributed glusterfs 5.10, no replication, 2 bricks (4TB each) on each node, 7 nodes in total. We added new bricks yesterday to the existing setup.
>> Post that we did a rebalance fix-layout and then a rebalance (which is currently still in progress). The status shows "failed" on certain bricks but "in progress" for others. Adding output for gluster rebalance status below.
>> 
>> The glusterfs client logs are flooded with "Mismatching layouts for <directory>,gfid = <id>"
>> The performance too seems to have degraded due to this, even basic commands like `cd` and `ls` are taking more than a minute compared to sub-second number before brick addition.
>> Apart from that we also experienced many binaries and files giving error stale file handle error even though the files were present.
>> 
>> 
>> gluster rebalance status :
>> 
>> Node Rebalanced-files          size       scanned      failures       skipped               status  run time in h:m:s
>> ---------      -----------   -----------   -----------   -----------   -----------         ------------     --------------
>> localhost              176         3.5GB         12790             0          8552          in progress       21:36:01
>> 10.132.0.72             8232       394.8GB         19995            21            26               failed       14:50:30
>> 10.132.0.44            12625         1.0TB         50023             1         10202          in progress       21:36:00
>> 10.132.0.3            21982       956.8GB         79145             1         34571          in progress       21:36:00
>> 10.132.0.9             7975       355.8GB         20157             6          1522               failed       14:51:45
>> 10.132.0.73             6293       394.5GB         26414           151          8085               failed       14:51:45
>> 10.132.0.70             6480       477.1GB         21058            27          1787               failed       14:50:32
>> Estimated time left for rebalance to complete :      130:56:28
>> 
>> 
>> Logs from one of the clients below:
>> 
>> [2020-07-02 12:30:14.971916] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3995747641; disk layout - 2761060380 - 3067813815 - 4159036738
>> [2020-07-02 12:30:14.971935] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>> [2020-07-02 12:30:15.032013] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3995747641; disk layout - 3681390552 - 3988143987 - 4159036738
>> [2020-07-02 12:30:15.032059] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>> [2020-07-02 12:30:15.032107] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:15.032153] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI, gfid = b40e4c58-67b3-4d9e-b708-1ebd23f50dcc
>> [2020-07-02 12:30:15.093329] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 2454306944 - 2761060379 - 3997647794; disk layout - 2454306944 - 2761060379 - 4159036738
>> [2020-07-02 12:30:15.093373] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.093460] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 2761060380 - 3067813815 - 3997647794; disk layout - 2761060380 - 3067813815 - 4159036738
>> [2020-07-02 12:30:15.093515] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.151063] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 3997647794; disk layout - 3681390552 - 3988143987 - 4159036738
>> [2020-07-02 12:30:15.151108] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.151149] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 3997647794; disk layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:15.151162] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/BSE_EOBI/20200630, gfid = 42a506b3-7aff-4935-8ef7-ecb8877c8222
>> [2020-07-02 12:30:15.424321] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-11; inode layout - 920400036 - 1227153471 - 3997647794; disk layout - 920400036 - 1227153471 - 4159036738
>> [2020-07-02 12:30:15.424380] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:15.424456] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 1840730208 - 2147483643 - 3997647794; disk layout - 1840730208 - 2147483643 - 4159036738
>> [2020-07-02 12:30:15.424484] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:15.424525] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 1533976772 - 1840730207 - 3997647794; disk layout - 1533976772 - 1840730207 - 4159036738
>> [2020-07-02 12:30:15.424542] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:15.424596] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-10; inode layout - 613646600 - 920400035 - 3997647794; disk layout - 613646600 - 920400035 - 4159036738
>> [2020-07-02 12:30:15.424607] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /raw_data/NSE/20200630, gfid = 1a1c92db-503a-4126-911c-06d3a8ad9ea1
>> [2020-07-02 12:30:16.004482] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_DATA.dat on data-client-7 (hashed subvol is data-client-17)
>> [2020-07-02 12:30:16.005523] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_DATA.dat
>> [2020-07-02 12:30:16.531047] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_1_METADATA.dat on data-client-9 (hashed subvol is data-client-19)
>> [2020-07-02 12:30:16.532086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_1_METADATA.dat
>> [2020-07-02 12:30:18.733229] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_DATA.dat on data-client-17 (hashed subvol is data-client-9)
>> [2020-07-02 12:30:18.734421] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_DATA.dat
>> [2020-07-02 12:30:19.171930] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_CDS_2_METADATA.dat on data-client-9 (hashed subvol is data-client-18)
>> [2020-07-02 12:30:19.172901] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_CDS_2_METADATA.dat
>> [2020-07-02 12:30:21.028495] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_DATA.dat on data-client-6 (hashed subvol is data-client-15)
>> [2020-07-02 12:30:21.029836] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_DATA.dat
>> [2020-07-02 12:30:21.127648] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_2_METADATA.dat on data-client-11 (hashed subvol is data-client-3)
>> [2020-07-02 12:30:21.128713] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_2_METADATA.dat
>> [2020-07-02 12:30:21.201126] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_DATA.dat on data-client-15 (hashed subvol is data-client-7)
>> [2020-07-02 12:30:21.201928] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_DATA.dat
>> [2020-07-02 12:30:21.566158] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_3_METADATA.dat on data-client-7 (hashed subvol is data-client-16)
>> [2020-07-02 12:30:21.567123] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_3_METADATA.dat
>> [2020-07-02 12:30:21.649357] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_DATA.dat on data-client-2 (hashed subvol is data-client-11)
>> [2020-07-02 12:30:21.661381] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_DATA.dat
>> [2020-07-02 12:30:21.748937] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_4_METADATA.dat on data-client-15 (hashed subvol is data-client-7)
>> [2020-07-02 12:30:21.749481] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_4_METADATA.dat
>> [2020-07-02 12:30:21.898593] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_DATA.dat on data-client-14 (hashed subvol is data-client-7)
>> [2020-07-02 12:30:21.899442] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_DATA.dat
>> [2020-07-02 12:30:22.039337] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/BSE_EQ_6_METADATA.dat on data-client-10 (hashed subvol is data-client-2)
>> [2020-07-02 12:30:22.040086] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/BSE_EQ_6_METADATA.dat
>> [2020-07-02 12:30:22.501877] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_DATA.dat on data-client-15 (hashed subvol is data-client-8)
>> [2020-07-02 12:30:22.502712] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_DATA.dat
>> [2020-07-02 12:30:22.782577] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECDS1_METADATA.dat on data-client-11 (hashed subvol is data-client-6)
>> [2020-07-02 12:30:22.783777] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECDS1_METADATA.dat
>> [2020-07-02 12:30:23.146847] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_DATA.dat on data-client-17 (hashed subvol is data-client-9)
>> [2020-07-02 12:30:23.148009] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_DATA.dat
>> [2020-07-02 12:30:23.229290] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM1_METADATA.dat on data-client-14 (hashed subvol is data-client-6)
>> [2020-07-02 12:30:23.230151] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM1_METADATA.dat
>> [2020-07-02 12:30:23.889520] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_DATA.dat on data-client-2 (hashed subvol is data-client-11)
>> [2020-07-02 12:30:23.896618] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_DATA.dat
>> [2020-07-02 12:30:24.093017] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM2_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
>> [2020-07-02 12:30:24.094117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM2_METADATA.dat
>> [2020-07-02 12:30:24.345257] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_DATA.dat on data-client-17 (hashed subvol is data-client-10)
>> [2020-07-02 12:30:24.346234] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_DATA.dat
>> [2020-07-02 12:30:24.425835] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSECM3_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
>> [2020-07-02 12:30:24.426880] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSECM3_METADATA.dat
>> [2020-07-02 12:30:25.158718] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_DATA.dat on data-client-9 (hashed subvol is data-client-19)
>> [2020-07-02 12:30:25.159619] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_DATA.dat
>> [2020-07-02 12:30:25.531479] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO1_METADATA.dat on data-client-2 (hashed subvol is data-client-10)
>> [2020-07-02 12:30:25.540569] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO1_METADATA.dat
>> [2020-07-02 12:30:25.771692] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_DATA.dat on data-client-11 (hashed subvol is data-client-3)
>> [2020-07-02 12:30:25.772610] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_DATA.dat
>> [2020-07-02 12:30:25.866118] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO2_METADATA.dat on data-client-15 (hashed subvol is data-client-8)
>> [2020-07-02 12:30:25.866917] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO2_METADATA.dat
>> [2020-07-02 12:30:26.424386] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_DATA.dat on data-client-9 (hashed subvol is data-client-18)
>> [2020-07-02 12:30:26.425309] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_DATA.dat
>> [2020-07-02 12:30:26.818852] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO3_METADATA.dat on data-client-10 (hashed subvol is data-client-2)
>> [2020-07-02 12:30:26.819890] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO3_METADATA.dat
>> [2020-07-02 12:30:27.352405] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_DATA.dat on data-client-10 (hashed subvol is data-client-2)
>> [2020-07-02 12:30:27.352914] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_DATA.dat
>> [2020-07-02 12:30:27.521286] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO4_METADATA.dat on data-client-8 (hashed subvol is data-client-18)
>> [2020-07-02 12:30:27.522325] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO4_METADATA.dat
>> [2020-07-02 12:30:28.566634] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO5_DATA.dat on data-client-2 (hashed subvol is data-client-11)
>> [2020-07-02 12:30:28.579295] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO5_DATA.dat
>> [2020-07-02 12:30:28.958028] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_DATA.dat on data-client-7 (hashed subvol is data-client-16)
>> [2020-07-02 12:30:28.959102] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_DATA.dat
>> [2020-07-02 12:30:29.012429] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/MCASTNSEFNO6_METADATA.dat on data-client-6 (hashed subvol is data-client-15)
>> [2020-07-02 12:30:29.013416] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/MCASTNSEFNO6_METADATA.dat
>> [2020-07-02 12:30:29.396716] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSDATA.dat on data-client-17 (hashed subvol is data-client-10)
>> [2020-07-02 12:30:29.397740] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSDATA.dat
>> [2020-07-02 12:30:29.556312] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat on data-client-9 (hashed subvol is data-client-18)
>> [2020-07-02 12:30:29.557197] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSEFO_BSE_TSMETADATA.dat
>> [2020-07-02 12:30:30.605354] I [MSGID: 109045] [dht-common.c:2701:dht_lookup_everywhere_cbk] 4-data-dht: attempting deletion of stale linkfile /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat on data-client-9 (hashed subvol is data-client-19)
>> [2020-07-02 12:30:30.606117] I [MSGID: 109069] [dht-common.c:1946:dht_lookup_unlink_cbk] 4-data-dht: lookup_unlink returned with op_ret -> 0 and op-errno -> 0 for /processed_data/20200630/NSETOBSEPUBLISHER_METADATA.dat
>> [2020-07-02 12:30:31.559206] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 613576736 - 920330171 - 1; disk layout - 613576736 - 920330171 - 4159036738
>> [2020-07-02 12:30:31.559255] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>> [2020-07-02 12:30:31.569025] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 920330172 - 1227083607 - 1; disk layout - 920330172 - 1227083607 - 4159036738
>> [2020-07-02 12:30:31.569067] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes, gfid = 21f02cb8-f5d4-4a11-a5ce-a557f5e42e99
>> [2020-07-02 12:30:31.701849] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3374637116 - 3681390551 - 1; disk layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:31.701895] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
>> [2020-07-02 12:30:31.738464] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3681390552 - 3988143987 - 1; disk layout - 3681390552 - 3988143987 - 4159036738
>> [2020-07-02 12:30:31.738507] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX, gfid = fff324f2-f855-4881-b77c-81e856522373
>> [2020-07-02 12:30:31.857102] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-15; inode layout - 3067883680 - 3374637115 - 3995747641; disk layout - 3067883680 - 3374637115 - 4159036738
>> [2020-07-02 12:30:31.857147] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:31.857180] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-16; inode layout - 3374637116 - 3681390551 - 3995747641; disk layout - 3374637116 - 3681390551 - 4159036738
>> [2020-07-02 12:30:31.857197] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:31.917705] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 0 - 306753435 - 3995747641; disk layout - 0 - 306753435 - 4159036738
>> [2020-07-02 12:30:31.917781] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:31.917855] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3988213852 - 4294967295 - 3995747641; disk layout - 3988213852 - 4294967295 - 4159036738
>> [2020-07-02 12:30:31.917874] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/BANKNIFTY, gfid = f8447150-4801-4188-add9-ea295bb88729
>> [2020-07-02 12:30:32.390945] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-18; inode layout - 3681460416 - 3988213851 - 1; disk layout - 3681460416 - 3988213851 - 4159036738
>> [2020-07-02 12:30:32.390998] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>> [2020-07-02 12:30:32.391056] I [MSGID: 109064] [dht-layout.c:771:dht_layout_dir_mismatch] 4-data-dht: subvol: data-client-19; inode layout - 3988213852 - 4294967295 - 1; disk layout - 3988213852 - 4294967295 - 4159036738
>> [2020-07-02 12:30:32.391075] I [MSGID: 109018] [dht-common.c:1686:dht_revalidate_cbk] 4-data-dht: Mismatching layouts for /processed_data/Indexes/NSEINDEX/NIFTY, gfid = b2d4deb7-c58c-4046-b6f2-7c7f44d71311
>> [2020-07-02 12:33:50.915279] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INCREMENTAL.dat.gz.IwE7T2 (2cb54500-814d-4e85-83e7-e33d9440b18d) (hash=data-client-6/cache=data-client-18) => /raw_data/Brazil/20200414/260_INCREMENTAL.dat.gz ((null)) (hash=data-client-6/cache=<nul>)
>> [2020-07-02 12:34:09.799586] I [MSGID: 109066] [dht-rename.c:1922:dht_rename] 4-data-dht: renaming /raw_data/Brazil/20200414/.260_INSTRUMENTS.dat.gz.1jUL1k (99938ee6-6986-4123-9d72-ec09e2310b4f) (hash=data-client-17/cache=data-client-18) => /raw_data/Brazil/20200414/260_INSTRUMENTS.dat.gz ((null)) (hash=data-client-17/cache=<nul>)
>> ....
>> 
>> 
>> Please look into this at top-priority if possible.
>> Let me know if anything else is required.
>> 
>> 
>> -- 
>> Regards,
>> Shreyansh Shah
>> ________
>> 
>> 
>> 
>> Community Meeting Calendar:
>> 
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968 <https://bluejeans.com/441850968>
>> 
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> https://lists.gluster.org/mailman/listinfo/gluster-users <https://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> 
> 
> -- 
> Regards,
> Shreyansh Shah

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/cc1ec898/attachment.html>

From felix.koelzow at gmx.de  Mon Jul 13 14:08:42 2020
From: felix.koelzow at gmx.de (=?UTF-8?Q?Felix_K=c3=b6lzow?=)
Date: Mon, 13 Jul 2020 16:08:42 +0200
Subject: [Gluster-users] Reliable Geo-Replication
In-Reply-To: <CAERh03rV994CW3MGhUPB4z0sk9SbcECQtEg3Yo5H1ejvQO7CQg@mail.gmail.com>
References: <75c1490e-2a69-7bc2-9ddd-a26fa5a225f5@gmx.de>
	<CAAwt65CmBh8Anfvb9ijnicz82ONFr10PHrTd7_TM_50YO5jUDA@mail.gmail.com>
	<93e7700c-c4df-dd3c-ba15-f7a815ae7e6a@gmx.de>
	<CAERh03pC1OkHBOO_Q-YYQnkRsReAUN9ynHssRL+fveHa1QOsOg@mail.gmail.com>
	<841137d6-768f-7560-24b3-c80fc8ceffa9@gmx.de>
	<CAERh03qjLu0s2iv2CPEbj0j5p8WZn=J=jaBTn-ZoX3DrHAuQog@mail.gmail.com>
	<ae9cf3f3-96c2-216f-a8d3-b540100999f6@gmx.de>
	<CAERh03rvwCURTbO+CJ1tUY_hwRSdmQqMza-MTNQShWfeLvCdFg@mail.gmail.com>
	<3b29ee38-991d-6253-f3da-504f4414c723@gmx.de>
	<CAERh03o9h10B7cvR5mUrrCbNvB27TRiH12i3Ki0Hdbq_XYbGHw@mail.gmail.com>
	<93624ed1-4c0a-100f-344c-1cb99b30f94b@mpa-ifw.tu-darmstadt.de>
	<b8e32f0c-635c-f39f-e687-1019ef7b6e85@gmx.de>
	<CAERh03rV994CW3MGhUPB4z0sk9SbcECQtEg3Yo5H1ejvQO7CQg@mail.gmail.com>
Message-ID: <6fe5442b-6b6e-cce5-ffd8-d73732a1c28d@gmx.de>

Dear Shwetha,

> Can you elaborate this? Where are the same files appearing? Are they
> getting synced
I changed to geo-repl config log_level to DEBUG and observed the log-file.

Already the same files are going to be synced again and again. Furthermore,

some files were marked as synced candidate but nothing happens.

There is no network traffic, the disk are almost idle and just nothing
happens.

No progress can be observed, even after 3 weeks of this state.


Currently, it changes from from faulty to initializing to fauly and so
forth. No worker is currently

active or passive.


Regards,

Felix

On 09/07/2020 13:29, Shwetha Acharya wrote:
> Hi?Felix,
>
> Find my reply inline.
>
> Regards,
> Shwetha
>
> On Thu, Jun 25, 2020 at 12:25 PM Felix K?lzow <felix.koelzow at gmx.de
> <mailto:felix.koelzow at gmx.de>> wrote:
>
>     Dear Gluster-users,
>
>     I deleted a further the geo-replication session with [reset-sync-time]
>     option. Afterwards,
>     I recreated the session, and as expected, the session starts in the
>     hybrid crawl.
>     I can see some sync jobs are running in the gsyncd.log file and
>     after a
>     couple of hours,
>     there are no such entries anymore.
>
>     I switched into the log_level DEBUG mode to see what's going on:
>     gluster volume masterVOlume geoRepHost:slaveVol config log_level DEBUG
>
>     It seems to me that the xsync mode is in loop since the same files
>     appear over and over again in the log-file.
>
> Can you elaborate this? Where are the same files appearing? Are they
> getting synced
>
>     Now we have two volume in this "loop"-state and the third volume also
>     still has a broken geo-replication.
>
>
> Is the worker status changing from initializing to faulty or
> initializing to active/passive? Is any worker active?
>
>     So any help is appreciated how to fix this or which information is
>     required to find the root cause?
>
>     As mentioned before, all these gathered information could be used to
>     improve the geo-replication trouble-shooting documentation.
>
>     Thanks in advance.
>
>     Regards,
>     Felix
>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
>
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/cc72f5b8/attachment.html>

From sacharya at redhat.com  Mon Jul 13 14:44:02 2020
From: sacharya at redhat.com (Shwetha Acharya)
Date: Mon, 13 Jul 2020 20:14:02 +0530
Subject: [Gluster-users] Reliable Geo-Replication
In-Reply-To: <6fe5442b-6b6e-cce5-ffd8-d73732a1c28d@gmx.de>
References: <75c1490e-2a69-7bc2-9ddd-a26fa5a225f5@gmx.de>
	<CAAwt65CmBh8Anfvb9ijnicz82ONFr10PHrTd7_TM_50YO5jUDA@mail.gmail.com>
	<93e7700c-c4df-dd3c-ba15-f7a815ae7e6a@gmx.de>
	<CAERh03pC1OkHBOO_Q-YYQnkRsReAUN9ynHssRL+fveHa1QOsOg@mail.gmail.com>
	<841137d6-768f-7560-24b3-c80fc8ceffa9@gmx.de>
	<CAERh03qjLu0s2iv2CPEbj0j5p8WZn=J=jaBTn-ZoX3DrHAuQog@mail.gmail.com>
	<ae9cf3f3-96c2-216f-a8d3-b540100999f6@gmx.de>
	<CAERh03rvwCURTbO+CJ1tUY_hwRSdmQqMza-MTNQShWfeLvCdFg@mail.gmail.com>
	<3b29ee38-991d-6253-f3da-504f4414c723@gmx.de>
	<CAERh03o9h10B7cvR5mUrrCbNvB27TRiH12i3Ki0Hdbq_XYbGHw@mail.gmail.com>
	<93624ed1-4c0a-100f-344c-1cb99b30f94b@mpa-ifw.tu-darmstadt.de>
	<b8e32f0c-635c-f39f-e687-1019ef7b6e85@gmx.de>
	<CAERh03rV994CW3MGhUPB4z0sk9SbcECQtEg3Yo5H1ejvQO7CQg@mail.gmail.com>
	<6fe5442b-6b6e-cce5-ffd8-d73732a1c28d@gmx.de>
Message-ID: <CAERh03r7gEBCE+sfCQF4Wth=F2jEznDDM_=J60UGpypRyXpSpQ@mail.gmail.com>

Hi Felix,

If the geo-rep status is always initializing to faulty, we will not see any
syncing of files.
We would need full logs to analyse the issue.

Regards,
Shwetha

On Mon, Jul 13, 2020 at 7:39 PM Felix K?lzow <felix.koelzow at gmx.de> wrote:

> Dear Shwetha,
>
> Can you elaborate this? Where are the same files appearing? Are they
> getting synced
>
> I changed to geo-repl config log_level to DEBUG and observed the log-file.
>
> Already the same files are going to be synced again and again. Furthermore,
>
> some files were marked as synced candidate but nothing happens.
>
> There is no network traffic, the disk are almost idle and just nothing
> happens.
>
> No progress can be observed, even after 3 weeks of this state.
>
>
> Currently, it changes from from faulty to initializing to fauly and so
> forth. No worker is currently
>
> active or passive.
>
>
> Regards,
>
> Felix
> On 09/07/2020 13:29, Shwetha Acharya wrote:
>
> Hi Felix,
>
> Find my reply inline.
>
> Regards,
> Shwetha
>
> On Thu, Jun 25, 2020 at 12:25 PM Felix K?lzow <felix.koelzow at gmx.de>
> wrote:
>
>> Dear Gluster-users,
>>
>> I deleted a further the geo-replication session with [reset-sync-time]
>> option. Afterwards,
>> I recreated the session, and as expected, the session starts in the
>> hybrid crawl.
>> I can see some sync jobs are running in the gsyncd.log file and after a
>> couple of hours,
>> there are no such entries anymore.
>>
>> I switched into the log_level DEBUG mode to see what's going on:
>> gluster volume masterVOlume geoRepHost:slaveVol config log_level DEBUG
>>
>> It seems to me that the xsync mode is in loop since the same files
>> appear over and over again in the log-file.
>
> Can you elaborate this? Where are the same files appearing? Are they
> getting synced
>
> Now we have two volume in this "loop"-state and the third volume also
>> still has a broken geo-replication.
>>
>
> Is the worker status changing from initializing to faulty or initializing
> to active/passive? Is any worker active?
>
>> So any help is appreciated how to fix this or which information is
>> required to find the root cause?
>>
>> As mentioned before, all these gathered information could be used to
>> improve the geo-replication trouble-shooting documentation.
>>
>> Thanks in advance.
>>
>> Regards,
>> Felix
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200713/ff241dde/attachment.html>

From kompastver at gmail.com  Thu Jul 16 08:19:45 2020
From: kompastver at gmail.com (Pavel Znamensky)
Date: Thu, 16 Jul 2020 11:19:45 +0300
Subject: [Gluster-users] One of cluster work super slow (v6.8)
In-Reply-To: <280988A1-D5DA-4960-A3A8-7A7A8103F1A1@yahoo.com>
References: <CALV6Y7prQ_vdt3MdmKWmkorS0znBogyQn0cU1M-+nUhUYY_NkQ@mail.gmail.com>
	<280988A1-D5DA-4960-A3A8-7A7A8103F1A1@yahoo.com>
Message-ID: <CALV6Y7qWV3t8f6PrOQiegMeaN9oWD998-S89cyF7=imXdQu=3Q@mail.gmail.com>

Sorry for the delay. Somehow Gmail decided to put almost all email from
this list to spam.
Anyway, yes, I checked the processes. Gluster processes are in 'R' state,
the others in 'S' state.
You can find 'top -H' output in the first message.
We're running glusterfs 6.8 on CentOS 7.8. Linux kernel 4.19.

Thanks.


??, 23 ???. 2020 ?. ? 21:49, Strahil Nikolov <hunter86_bg at yahoo.com>:

> What  is the OS and it's version ?
> I have seen similar behaviour (different workload)  on RHEL 7.6 (and
> below).
>
> Have you checked  what processes are in 'R' or 'D' state on  st2a  ?
>
> Best Regards,
> Strahil Nikolov
>
> ?? 23 ??? 2020 ?. 19:31:12 GMT+03:00, Pavel Znamensky <
> kompastver at gmail.com> ??????:
> >Hi all,
> >There's something strange with one of our clusters and glusterfs
> >version
> >6.8: it's quite slow and one node is overloaded.
> >This is distributed cluster with four servers with the same
> >specs/OS/versions:
> >
> >Volume Name: st2
> >Type: Distributed-Replicate
> >Volume ID: 4755753b-37c4-403b-b1c8-93099bfc4c45
> >Status: Started
> >Snapshot Count: 0
> >Number of Bricks: 2 x 2 = 4
> >Transport-type: tcp
> >Bricks:
> >Brick1: st2a:/vol3/st2
> >Brick2: st2b:/vol3/st2
> >Brick3: st2c:/vol3/st2
> >Brick4: st2d:/vol3/st2
> >Options Reconfigured:
> >cluster.rebal-throttle: aggressive
> >nfs.disable: on
> >performance.readdir-ahead: off
> >transport.address-family: inet6
> >performance.quick-read: off
> >performance.cache-size: 1GB
> >performance.io-cache: on
> >performance.io-thread-count: 16
> >cluster.data-self-heal-algorithm: full
> >network.ping-timeout: 20
> >server.event-threads: 2
> >client.event-threads: 2
> >cluster.readdir-optimize: on
> >performance.read-ahead: off
> >performance.parallel-readdir: on
> >cluster.self-heal-daemon: enable
> >storage.health-check-timeout: 20
> >
> >op.version for this cluster remains 50400
> >
> >st2a is a replica for the st2b and st2c is a replica for st2d.
> >All our 50 clients mount this volume using FUSE and in contrast with
> >other
> >our cluster this one works terrible slow.
> >Interesting thing here is that there are very low HDDs and network
> >utilization from one hand and quite overloaded server from another
> >hand.
> >Also, there are no files which should be healed according to `gluster
> >volume heal st2 info`.
> >Load average across servers:
> >st2a:
> >load average: 28,73, 26,39, 27,44
> >st2b:
> >load average: 0,24, 0,46, 0,76
> >st2c:
> >load average: 0,13, 0,20, 0,27
> >st2d:
> >load average:2,93, 2,11, 1,50
> >
> >If we stop glusterfs on st2a server the cluster will work as fast as we
> >expected.
> >Previously the cluster worked on a version 5.x and there were no such
> >problems.
> >
> >Interestingly, that almost all CPU usage on st2a generates by a
> >"system"
> >load.
> >The most CPU intensive process is glusterfsd.
> >`top -H` for glusterfsd process shows this:
> >
> >PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+
> >COMMAND
> >
> >13894 root      20   0 2172892  96488   9056 R 74,0  0,1 122:09.14
> >glfs_iotwr00a
> >13888 root      20   0 2172892  96488   9056 R 73,7  0,1 121:38.26
> >glfs_iotwr004
> >13891 root      20   0 2172892  96488   9056 R 73,7  0,1 121:53.83
> >glfs_iotwr007
> >13920 root      20   0 2172892  96488   9056 R 73,0  0,1 122:11.27
> >glfs_iotwr00f
> >13897 root      20   0 2172892  96488   9056 R 68,3  0,1 121:09.82
> >glfs_iotwr00d
> >13896 root      20   0 2172892  96488   9056 R 68,0  0,1 122:03.99
> >glfs_iotwr00c
> >13868 root      20   0 2172892  96488   9056 R 67,7  0,1 122:42.55
> >glfs_iotwr000
> >13889 root      20   0 2172892  96488   9056 R 67,3  0,1 122:17.02
> >glfs_iotwr005
> >13887 root      20   0 2172892  96488   9056 R 67,0  0,1 122:29.88
> >glfs_iotwr003
> >13885 root      20   0 2172892  96488   9056 R 65,0  0,1 122:04.85
> >glfs_iotwr001
> >13892 root      20   0 2172892  96488   9056 R 55,0  0,1 121:15.23
> >glfs_iotwr008
> >13890 root      20   0 2172892  96488   9056 R 54,7  0,1 121:27.88
> >glfs_iotwr006
> >13895 root      20   0 2172892  96488   9056 R 54,0  0,1 121:28.35
> >glfs_iotwr00b
> >13893 root      20   0 2172892  96488   9056 R 53,0  0,1 122:23.12
> >glfs_iotwr009
> >13898 root      20   0 2172892  96488   9056 R 52,0  0,1 122:30.67
> >glfs_iotwr00e
> >13886 root      20   0 2172892  96488   9056 R 41,3  0,1 121:26.97
> >glfs_iotwr002
> >13878 root      20   0 2172892  96488   9056 S  1,0  0,1   1:20.34
> >glfs_rpcrqhnd
> >13840 root      20   0 2172892  96488   9056 S  0,7  0,1   0:51.54
> >glfs_epoll000
> >13841 root      20   0 2172892  96488   9056 S  0,7  0,1   0:51.14
> >glfs_epoll001
> >13877 root      20   0 2172892  96488   9056 S  0,3  0,1   1:20.02
> >glfs_rpcrqhnd
> >13833 root      20   0 2172892  96488   9056 S  0,0  0,1   0:00.00
> >glusterfsd
> >13834 root      20   0 2172892  96488   9056 S  0,0  0,1   0:00.14
> >glfs_timer
> >13835 root      20   0 2172892  96488   9056 S  0,0  0,1   0:00.00
> >glfs_sigwait
> >13836 root      20   0 2172892  96488   9056 S  0,0  0,1   0:00.16
> >glfs_memsweep
> >13837 root      20   0 2172892  96488   9056 S  0,0  0,1   0:00.05
> >glfs_sproc0
> >
> >Also I didn't find relevant messages in log files.
> >Honestly, don't know what to do. Does someone know how to debug or fix
> >this
> >behaviour?
> >
> >Best regards,
> >Pavel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200716/1be0173a/attachment.html>

From nladha at redhat.com  Fri Jul 17 10:59:50 2020
From: nladha at redhat.com (Nikhil Ladha)
Date: Fri, 17 Jul 2020 16:29:50 +0530
Subject: [Gluster-users] Issues with replicated gluster volume
Message-ID: <CAENLX8=Q9EGGUXOvCOzhM+pbgRQOCW+PYZuo4xwJNrqSmRh37g@mail.gmail.com>

Hi Ahemad,

A few days back, you had some issue with the replicate volumes and you had
a log like this

``[2020-06-16 07:19:27.418884] E [MSGID: 101097]
[xlator.c:218:xlator_volopt_dynload] 0-xlator: dlsym(xlator_api) missing:
/usr/lib64/glusterfs/7.5/rpc-transport/socket.so: undefined symbol:
xlator_api``

present in the glusterd.log file. So, I just want to clarify which steps
you followed that led to such a log, as it was not reproducible by me. On
running a few common gluster commands like create, start, stop, status. If
there is any other configuration that you made in the volume or performed
any other operation, then please let me know.

Regards
Nikhil Ladha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200717/2a563105/attachment.html>

From gilberto.nunes32 at gmail.com  Fri Jul 17 13:49:27 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Fri, 17 Jul 2020 10:49:27 -0300
Subject: [Gluster-users] Recovery when 2 of 3 servers goes down!
Message-ID: <CAOKSTBtvGoGZ3GogjLfS8h8_k_Oa7UcXsf=6=hh47oE3U9_A4g@mail.gmail.com>

How there

I have 3 servers with gluster 7 installed and setting up with replica 3 and
arbiter 1.
Here's the commands I used:
- First create a simple volume with one server:
gluster volume create VMS proxmox01:/DATA/vms
- Then add the second one
gluster peer probe proxmox02
gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
- And finally and the third:
gluster peer probe proxmox03
gluster volume add-brick VMS replica 3 arbiter 1 proxmox03:/DATA/vms

But then I decide to test the environment and bring proxmox02 and proxmox03
down and get Transport endpoint is not connected after a few seconds.
Is there a way to keep one server up if 2 goes down?
gluster vol info

Volume Name: VMS
Type: Replicate
Volume ID: 64735da4-8671-4c5e-b832-d15f5c03e9f0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: proxmox01:/DATA/vms
Brick2: proxmox02:/DATA/vms
Brick3: proxmox03:/DATA/vms (arbiter)
Options Reconfigured:
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
performance.client-io-threads: off
cluster.self-heal-daemon: enable
cluster.quorum-reads: false
cluster.quorum-count: 1

gluster vol status
Status of volume: VMS
Gluster process                             TCP Port  RDMA Port  Online
 Pid
------------------------------------------------------------------------------

Brick proxmox01:/DATA/vms                   49152     0          Y
      1526
Self-heal Daemon on localhost               N/A       N/A        Y
      1537

Task Status of Volume VMS
------------------------------------------------------------------------------

There are no active volume tasks


Thanks a lot


---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200717/ec7c835a/attachment.html>

From archon810 at gmail.com  Fri Jul 17 19:56:10 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Fri, 17 Jul 2020 12:56:10 -0700
Subject: [Gluster-users] Recovery when 2 of 3 servers goes down!
In-Reply-To: <CAOKSTBtvGoGZ3GogjLfS8h8_k_Oa7UcXsf=6=hh47oE3U9_A4g@mail.gmail.com>
References: <CAOKSTBtvGoGZ3GogjLfS8h8_k_Oa7UcXsf=6=hh47oE3U9_A4g@mail.gmail.com>
Message-ID: <CAD+dzQejAcA84kz1tg2R1rcz4YC+h2pahi_FDV75Ui19x6n=xQ@mail.gmail.com>

I had the same requirements (except with 4 servers and no arbiter), and
this was the solution:

gluster v set VMS cluster.quorum-count 1

gluster v set VMS cluster.quorum-type fixed

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Fri, Jul 17, 2020 at 6:50 AM Gilberto Nunes <gilberto.nunes32 at gmail.com>
wrote:

> How there
>
> I have 3 servers with gluster 7 installed and setting up with replica 3
> and arbiter 1.
> Here's the commands I used:
> - First create a simple volume with one server:
> gluster volume create VMS proxmox01:/DATA/vms
> - Then add the second one
> gluster peer probe proxmox02
> gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
> - And finally and the third:
> gluster peer probe proxmox03
> gluster volume add-brick VMS replica 3 arbiter 1 proxmox03:/DATA/vms
>
> But then I decide to test the environment and bring proxmox02 and
> proxmox03 down and get Transport endpoint is not connected after a few
> seconds.
> Is there a way to keep one server up if 2 goes down?
> gluster vol info
>
> Volume Name: VMS
> Type: Replicate
> Volume ID: 64735da4-8671-4c5e-b832-d15f5c03e9f0
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 1 x (2 + 1) = 3
> Transport-type: tcp
> Bricks:
> Brick1: proxmox01:/DATA/vms
> Brick2: proxmox02:/DATA/vms
> Brick3: proxmox03:/DATA/vms (arbiter)
> Options Reconfigured:
> nfs.disable: on
> storage.fips-mode-rchecksum: on
> transport.address-family: inet
> performance.client-io-threads: off
> cluster.self-heal-daemon: enable
> cluster.quorum-reads: false
> cluster.quorum-count: 1
>
> gluster vol status
> Status of volume: VMS
> Gluster process                             TCP Port  RDMA Port  Online
>  Pid
> ------------------------------------------------------------------------------
>
> Brick proxmox01:/DATA/vms                   49152     0          Y
>       1526
> Self-heal Daemon on localhost               N/A       N/A        Y
>       1537
>
> Task Status of Volume VMS
> ------------------------------------------------------------------------------
>
> There are no active volume tasks
>
>
> Thanks a lot
>
>
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200717/01c9dd78/attachment.html>

From gilberto.nunes32 at gmail.com  Fri Jul 17 21:38:23 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Fri, 17 Jul 2020 18:38:23 -0300
Subject: [Gluster-users] Recovery when 2 of 3 servers goes down!
In-Reply-To: <CAD+dzQejAcA84kz1tg2R1rcz4YC+h2pahi_FDV75Ui19x6n=xQ@mail.gmail.com>
References: <CAOKSTBtvGoGZ3GogjLfS8h8_k_Oa7UcXsf=6=hh47oE3U9_A4g@mail.gmail.com>
	<CAD+dzQejAcA84kz1tg2R1rcz4YC+h2pahi_FDV75Ui19x6n=xQ@mail.gmail.com>
Message-ID: <CAOKSTBsEGM+_04FH1oyV87vF4DcK5esBANBFcE9pt16onnkQoQ@mail.gmail.com>

Yes Artem! That's it!
I used the following commands and everything works as expected with 3 nodes:

gluster volume create VMS proxmox01:/DATA/vms

gluster vol start VMS
gluster vol status VMS

gluster peer probe proxmox02
gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms

gluster vol status VMS
gluster vol info VMS

gluster peer probe proxmox03
gluster volume add-brick VMS replica 3 proxmox03:/DATA/vms

gluster vol set VMS cluster.heal-timeout 60
gluster volume heal VMS enable
gluster vol set VMS cluster.quorum-reads false
gluster vol set VMS cluster.quorum-count 1


Thanks for you replay

Cheers


---
Gilberto Nunes Ferreira



Em sex., 17 de jul. de 2020 ?s 16:56, Artem Russakovskii <
archon810 at gmail.com> escreveu:

> I had the same requirements (except with 4 servers and no arbiter), and
> this was the solution:
>
> gluster v set VMS cluster.quorum-count 1
>
> gluster v set VMS cluster.quorum-type fixed
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Fri, Jul 17, 2020 at 6:50 AM Gilberto Nunes <gilberto.nunes32 at gmail.com>
> wrote:
>
>> How there
>>
>> I have 3 servers with gluster 7 installed and setting up with replica 3
>> and arbiter 1.
>> Here's the commands I used:
>> - First create a simple volume with one server:
>> gluster volume create VMS proxmox01:/DATA/vms
>> - Then add the second one
>> gluster peer probe proxmox02
>> gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
>> - And finally and the third:
>> gluster peer probe proxmox03
>> gluster volume add-brick VMS replica 3 arbiter 1 proxmox03:/DATA/vms
>>
>> But then I decide to test the environment and bring proxmox02 and
>> proxmox03 down and get Transport endpoint is not connected after a few
>> seconds.
>> Is there a way to keep one server up if 2 goes down?
>> gluster vol info
>>
>> Volume Name: VMS
>> Type: Replicate
>> Volume ID: 64735da4-8671-4c5e-b832-d15f5c03e9f0
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 1 x (2 + 1) = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: proxmox01:/DATA/vms
>> Brick2: proxmox02:/DATA/vms
>> Brick3: proxmox03:/DATA/vms (arbiter)
>> Options Reconfigured:
>> nfs.disable: on
>> storage.fips-mode-rchecksum: on
>> transport.address-family: inet
>> performance.client-io-threads: off
>> cluster.self-heal-daemon: enable
>> cluster.quorum-reads: false
>> cluster.quorum-count: 1
>>
>> gluster vol status
>> Status of volume: VMS
>> Gluster process                             TCP Port  RDMA Port  Online
>>  Pid
>> ------------------------------------------------------------------------------
>>
>> Brick proxmox01:/DATA/vms                   49152     0          Y
>>       1526
>> Self-heal Daemon on localhost               N/A       N/A        Y
>>       1537
>>
>> Task Status of Volume VMS
>> ------------------------------------------------------------------------------
>>
>> There are no active volume tasks
>>
>>
>> Thanks a lot
>>
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200717/2aef9624/attachment.html>

From archon810 at gmail.com  Fri Jul 17 23:16:39 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Fri, 17 Jul 2020 16:16:39 -0700
Subject: [Gluster-users] Recovery when 2 of 3 servers goes down!
In-Reply-To: <CAOKSTBsEGM+_04FH1oyV87vF4DcK5esBANBFcE9pt16onnkQoQ@mail.gmail.com>
References: <CAOKSTBtvGoGZ3GogjLfS8h8_k_Oa7UcXsf=6=hh47oE3U9_A4g@mail.gmail.com>
	<CAD+dzQejAcA84kz1tg2R1rcz4YC+h2pahi_FDV75Ui19x6n=xQ@mail.gmail.com>
	<CAOKSTBsEGM+_04FH1oyV87vF4DcK5esBANBFcE9pt16onnkQoQ@mail.gmail.com>
Message-ID: <CAD+dzQdOwqBeEyuD69QRKsSfdgx8C-WMRd7LLeTir7heLkgpFw@mail.gmail.com>

No problem.

Oh I also had to set

> gluster v set VMS network.ping-timeout 5

because in case a server went down and started timing out (full shutdown),
the default value was so high (I think 60s) that it made all nodes straight
up freeze for this long before serving the files.

Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Fri, Jul 17, 2020 at 2:39 PM Gilberto Nunes <gilberto.nunes32 at gmail.com>
wrote:

> Yes Artem! That's it!
> I used the following commands and everything works as expected with 3
> nodes:
>
> gluster volume create VMS proxmox01:/DATA/vms
>
> gluster vol start VMS
> gluster vol status VMS
>
> gluster peer probe proxmox02
> gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
>
> gluster vol status VMS
> gluster vol info VMS
>
> gluster peer probe proxmox03
> gluster volume add-brick VMS replica 3 proxmox03:/DATA/vms
>
> gluster vol set VMS cluster.heal-timeout 60
> gluster volume heal VMS enable
> gluster vol set VMS cluster.quorum-reads false
> gluster vol set VMS cluster.quorum-count 1
>
>
> Thanks for you replay
>
> Cheers
>
>
> ---
> Gilberto Nunes Ferreira
>
>
>
> Em sex., 17 de jul. de 2020 ?s 16:56, Artem Russakovskii <
> archon810 at gmail.com> escreveu:
>
>> I had the same requirements (except with 4 servers and no arbiter), and
>> this was the solution:
>>
>> gluster v set VMS cluster.quorum-count 1
>>
>> gluster v set VMS cluster.quorum-type fixed
>>
>> Sincerely,
>> Artem
>>
>> --
>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>> <http://www.apkmirror.com/>, Illogical Robot LLC
>> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>>
>>
>> On Fri, Jul 17, 2020 at 6:50 AM Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> wrote:
>>
>>> How there
>>>
>>> I have 3 servers with gluster 7 installed and setting up with replica 3
>>> and arbiter 1.
>>> Here's the commands I used:
>>> - First create a simple volume with one server:
>>> gluster volume create VMS proxmox01:/DATA/vms
>>> - Then add the second one
>>> gluster peer probe proxmox02
>>> gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
>>> - And finally and the third:
>>> gluster peer probe proxmox03
>>> gluster volume add-brick VMS replica 3 arbiter 1 proxmox03:/DATA/vms
>>>
>>> But then I decide to test the environment and bring proxmox02 and
>>> proxmox03 down and get Transport endpoint is not connected after a few
>>> seconds.
>>> Is there a way to keep one server up if 2 goes down?
>>> gluster vol info
>>>
>>> Volume Name: VMS
>>> Type: Replicate
>>> Volume ID: 64735da4-8671-4c5e-b832-d15f5c03e9f0
>>> Status: Started
>>> Snapshot Count: 0
>>> Number of Bricks: 1 x (2 + 1) = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: proxmox01:/DATA/vms
>>> Brick2: proxmox02:/DATA/vms
>>> Brick3: proxmox03:/DATA/vms (arbiter)
>>> Options Reconfigured:
>>> nfs.disable: on
>>> storage.fips-mode-rchecksum: on
>>> transport.address-family: inet
>>> performance.client-io-threads: off
>>> cluster.self-heal-daemon: enable
>>> cluster.quorum-reads: false
>>> cluster.quorum-count: 1
>>>
>>> gluster vol status
>>> Status of volume: VMS
>>> Gluster process                             TCP Port  RDMA Port  Online
>>>  Pid
>>> ------------------------------------------------------------------------------
>>>
>>> Brick proxmox01:/DATA/vms                   49152     0          Y
>>>       1526
>>> Self-heal Daemon on localhost               N/A       N/A        Y
>>>       1537
>>>
>>> Task Status of Volume VMS
>>> ------------------------------------------------------------------------------
>>>
>>> There are no active volume tasks
>>>
>>>
>>> Thanks a lot
>>>
>>>
>>> ---
>>> Gilberto Nunes Ferreira
>>>
>>> (47) 3025-5907
>>> (47) 99676-7530 - Whatsapp / Telegram
>>>
>>> Skype: gilberto.nunes36
>>>
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200717/088f94d8/attachment.html>

From gilberto.nunes32 at gmail.com  Sat Jul 18 02:42:12 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Fri, 17 Jul 2020 23:42:12 -0300
Subject: [Gluster-users] Recovery when 2 of 3 servers goes down!
In-Reply-To: <CAD+dzQdOwqBeEyuD69QRKsSfdgx8C-WMRd7LLeTir7heLkgpFw@mail.gmail.com>
References: <CAOKSTBtvGoGZ3GogjLfS8h8_k_Oa7UcXsf=6=hh47oE3U9_A4g@mail.gmail.com>
	<CAD+dzQejAcA84kz1tg2R1rcz4YC+h2pahi_FDV75Ui19x6n=xQ@mail.gmail.com>
	<CAOKSTBsEGM+_04FH1oyV87vF4DcK5esBANBFcE9pt16onnkQoQ@mail.gmail.com>
	<CAD+dzQdOwqBeEyuD69QRKsSfdgx8C-WMRd7LLeTir7heLkgpFw@mail.gmail.com>
Message-ID: <CAOKSTBsg_A8LAmXuVdE8R9ARY249Fa2-yZkFEkpmm4xNGam4Jw@mail.gmail.com>

yeah... 600 secs in matter of fact! Don't know why this value is so high.
---
Gilberto Nunes Ferreira





Em sex., 17 de jul. de 2020 ?s 20:17, Artem Russakovskii <
archon810 at gmail.com> escreveu:

> No problem.
>
> Oh I also had to set
>
>> gluster v set VMS network.ping-timeout 5
>
> because in case a server went down and started timing out (full shutdown),
> the default value was so high (I think 60s) that it made all nodes straight
> up freeze for this long before serving the files.
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Fri, Jul 17, 2020 at 2:39 PM Gilberto Nunes <gilberto.nunes32 at gmail.com>
> wrote:
>
>> Yes Artem! That's it!
>> I used the following commands and everything works as expected with 3
>> nodes:
>>
>> gluster volume create VMS proxmox01:/DATA/vms
>>
>> gluster vol start VMS
>> gluster vol status VMS
>>
>> gluster peer probe proxmox02
>> gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
>>
>> gluster vol status VMS
>> gluster vol info VMS
>>
>> gluster peer probe proxmox03
>> gluster volume add-brick VMS replica 3 proxmox03:/DATA/vms
>>
>> gluster vol set VMS cluster.heal-timeout 60
>> gluster volume heal VMS enable
>> gluster vol set VMS cluster.quorum-reads false
>> gluster vol set VMS cluster.quorum-count 1
>>
>>
>> Thanks for you replay
>>
>> Cheers
>>
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>>
>>
>> Em sex., 17 de jul. de 2020 ?s 16:56, Artem Russakovskii <
>> archon810 at gmail.com> escreveu:
>>
>>> I had the same requirements (except with 4 servers and no arbiter), and
>>> this was the solution:
>>>
>>> gluster v set VMS cluster.quorum-count 1
>>>
>>> gluster v set VMS cluster.quorum-type fixed
>>>
>>> Sincerely,
>>> Artem
>>>
>>> --
>>> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
>>> <http://www.apkmirror.com/>, Illogical Robot LLC
>>> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>>>
>>>
>>> On Fri, Jul 17, 2020 at 6:50 AM Gilberto Nunes <
>>> gilberto.nunes32 at gmail.com> wrote:
>>>
>>>> How there
>>>>
>>>> I have 3 servers with gluster 7 installed and setting up with replica 3
>>>> and arbiter 1.
>>>> Here's the commands I used:
>>>> - First create a simple volume with one server:
>>>> gluster volume create VMS proxmox01:/DATA/vms
>>>> - Then add the second one
>>>> gluster peer probe proxmox02
>>>> gluster volume add-brick VMS replica 2 proxmox02:/DATA/vms
>>>> - And finally and the third:
>>>> gluster peer probe proxmox03
>>>> gluster volume add-brick VMS replica 3 arbiter 1 proxmox03:/DATA/vms
>>>>
>>>> But then I decide to test the environment and bring proxmox02 and
>>>> proxmox03 down and get Transport endpoint is not connected after a few
>>>> seconds.
>>>> Is there a way to keep one server up if 2 goes down?
>>>> gluster vol info
>>>>
>>>> Volume Name: VMS
>>>> Type: Replicate
>>>> Volume ID: 64735da4-8671-4c5e-b832-d15f5c03e9f0
>>>> Status: Started
>>>> Snapshot Count: 0
>>>> Number of Bricks: 1 x (2 + 1) = 3
>>>> Transport-type: tcp
>>>> Bricks:
>>>> Brick1: proxmox01:/DATA/vms
>>>> Brick2: proxmox02:/DATA/vms
>>>> Brick3: proxmox03:/DATA/vms (arbiter)
>>>> Options Reconfigured:
>>>> nfs.disable: on
>>>> storage.fips-mode-rchecksum: on
>>>> transport.address-family: inet
>>>> performance.client-io-threads: off
>>>> cluster.self-heal-daemon: enable
>>>> cluster.quorum-reads: false
>>>> cluster.quorum-count: 1
>>>>
>>>> gluster vol status
>>>> Status of volume: VMS
>>>> Gluster process                             TCP Port  RDMA Port  Online
>>>>  Pid
>>>> ------------------------------------------------------------------------------
>>>>
>>>> Brick proxmox01:/DATA/vms                   49152     0          Y
>>>>       1526
>>>> Self-heal Daemon on localhost               N/A       N/A        Y
>>>>       1537
>>>>
>>>> Task Status of Volume VMS
>>>> ------------------------------------------------------------------------------
>>>>
>>>> There are no active volume tasks
>>>>
>>>>
>>>> Thanks a lot
>>>>
>>>>
>>>> ---
>>>> Gilberto Nunes Ferreira
>>>>
>>>> (47) 3025-5907
>>>> (47) 99676-7530 - Whatsapp / Telegram
>>>>
>>>> Skype: gilberto.nunes36
>>>>
>>>>
>>>>
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200717/d905d71b/attachment.html>

From s.danzi at hawai.it  Tue Jul 21 12:45:14 2020
From: s.danzi at hawai.it (Stefano Danzi)
Date: Tue, 21 Jul 2020 14:45:14 +0200
Subject: [Gluster-users] Replica scenario in low speed link
Message-ID: <32558859-9d58-294a-a987-c619bb83dfb6@hawai.it>

Hi!

I have a strange request. I don't know if some gluster settings could 
help me.

There are two buildings linked using a wifi bridge.
The main building host the data centre.? In the other building there are 
an office
that need to use a file server with high speed, so the wifi link is not 
optimal.

My fantasy suggest me to have a server in this office running samba and 
gluster.
Gluster have to read/write immediately from the local storage and 
leisurely replicate on remote storage (maybe a replica 3,
or replica 2 + arbiter with locally one replica and the arbiter).
In case of local storage/machine fault the system have to read/write 
using the remote brick over "slow" link
and restart to use local storage when it return to be available and synced.

Samba could be running only on this office or have two instances, in HA 
cluster, one in this office and
other in the main building.

Anyone have any suggestions or have they had a similar need?

Thanks, bye.


From qw at g.clemson.edu  Tue Jul 21 18:30:03 2020
From: qw at g.clemson.edu (Qing Wang)
Date: Tue, 21 Jul 2020 14:30:03 -0400
Subject: [Gluster-users] Gluster linear scale-out performance
Message-ID: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>

Hi,

I am trying to test Gluster linear scale-out performance by adding more
storage server/bricks, and measure the storage I/O performance. To vary the
storage server number, I create several "stripe" volumes that contain 2
brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g
bs=1M count=26000" to create 26G data (or larger size), and those data will
be distributed to the corresponding gluster servers (each has gluster brick
on it) and "dd" returns the final I/O throughput. The Internet is 40G
infiniband, although I didn't do any specific configurations to use
advanced features.

What confuses me is that the storage I/O seems not to relate to the number
of storage nodes, but Gluster documents said it should be linear scaling.
For example, when "write-behind" is on, and when Infiniband "jumbo frame"
(connected mode) is on, I can get ~800 MB/sec reported by "dd", no matter I
have 2 brick servers or 8 brick servers -- for 2 server case, each server
can have ~400 MB/sec; for 4 server case, each server can have ~200MB/sec.
That said, each server I/O does aggregate to the final storage I/O (800
MB/sec), but this is not "linear scale-out".

Can somebody help me to understand why this is the case? I certainly can
have some misunderstanding/misconfiguration here. Please correct me if I
do, thanks!

Best,
Qing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/22f376c9/attachment.html>

From ykaul at redhat.com  Tue Jul 21 18:37:44 2020
From: ykaul at redhat.com (Yaniv Kaul)
Date: Tue, 21 Jul 2020 21:37:44 +0300
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
Message-ID: <CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>

On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:

> Hi,
>
> I am trying to test Gluster linear scale-out performance by adding more
> storage server/bricks, and measure the storage I/O performance. To vary the
> storage server number, I create several "stripe" volumes that contain 2
> brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
> client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g
> bs=1M count=26000" to create 26G data (or larger size), and those data will
> be distributed to the corresponding gluster servers (each has gluster brick
> on it) and "dd" returns the final I/O throughput. The Internet is 40G
> infiniband, although I didn't do any specific configurations to use
> advanced features.
>

Your dd command is inaccurate, as it'll hit the client cache. It is also
single threaded. I suggest switching to fio.
Y.


> What confuses me is that the storage I/O seems not to relate to the number
> of storage nodes, but Gluster documents said it should be linear scaling.
> For example, when "write-behind" is on, and when Infiniband "jumbo frame"
> (connected mode) is on, I can get ~800 MB/sec reported by "dd", no matter I
> have 2 brick servers or 8 brick servers -- for 2 server case, each server
> can have ~400 MB/sec; for 4 server case, each server can have ~200MB/sec.
> That said, each server I/O does aggregate to the final storage I/O (800
> MB/sec), but this is not "linear scale-out".
>
> Can somebody help me to understand why this is the case? I certainly can
> have some misunderstanding/misconfiguration here. Please correct me if I
> do, thanks!
>
> Best,
> Qing
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/1524ac67/attachment.html>

From qw at g.clemson.edu  Tue Jul 21 18:43:07 2020
From: qw at g.clemson.edu (Qing Wang)
Date: Tue, 21 Jul 2020 14:43:07 -0400
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
Message-ID: <CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>

Hi Yaniv,

Thanks for the quick response. I forget to mention I am testing the writing
performance, not reading. In this case, would the client cache hit rate
still be a big issue?

I'll use fio to run my test once again, thanks for the suggestion.

Thanks,
Qing

On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <ykaul at redhat.com> wrote:

>
>
> On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
>
>> Hi,
>>
>> I am trying to test Gluster linear scale-out performance by adding more
>> storage server/bricks, and measure the storage I/O performance. To vary the
>> storage server number, I create several "stripe" volumes that contain 2
>> brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
>> client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g
>> bs=1M count=26000" to create 26G data (or larger size), and those data will
>> be distributed to the corresponding gluster servers (each has gluster brick
>> on it) and "dd" returns the final I/O throughput. The Internet is 40G
>> infiniband, although I didn't do any specific configurations to use
>> advanced features.
>>
>
> Your dd command is inaccurate, as it'll hit the client cache. It is also
> single threaded. I suggest switching to fio.
> Y.
>
>
>> What confuses me is that the storage I/O seems not to relate to the
>> number of storage nodes, but Gluster documents said it should be linear
>> scaling. For example, when "write-behind" is on, and when Infiniband "jumbo
>> frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no
>> matter I have 2 brick servers or 8 brick servers -- for 2 server case, each
>> server can have ~400 MB/sec; for 4 server case, each server can have
>> ~200MB/sec. That said, each server I/O does aggregate to the final storage
>> I/O (800 MB/sec), but this is not "linear scale-out".
>>
>> Can somebody help me to understand why this is the case? I certainly can
>> have some misunderstanding/misconfiguration here. Please correct me if I
>> do, thanks!
>>
>> Best,
>> Qing
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/a535ccd0/attachment.html>

From ykaul at redhat.com  Tue Jul 21 18:53:29 2020
From: ykaul at redhat.com (Yaniv Kaul)
Date: Tue, 21 Jul 2020 21:53:29 +0300
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
	<CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
Message-ID: <CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>

On Tue, 21 Jul 2020, 21:43 Qing Wang <qw at g.clemson.edu> wrote:

> Hi Yaniv,
>
> Thanks for the quick response. I forget to mention I am testing the
> writing performance, not reading. In this case, would the client cache hit
> rate still be a big issue?
>

It's not hitting the storage directly. Since it's also single threaded, it
may also not saturate it. I highly recommend testing properly.
Y.


> I'll use fio to run my test once again, thanks for the suggestion.
>
> Thanks,
> Qing
>
> On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>
>>
>>
>> On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
>>
>>> Hi,
>>>
>>> I am trying to test Gluster linear scale-out performance by adding more
>>> storage server/bricks, and measure the storage I/O performance. To vary the
>>> storage server number, I create several "stripe" volumes that contain 2
>>> brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
>>> client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g
>>> bs=1M count=26000" to create 26G data (or larger size), and those data will
>>> be distributed to the corresponding gluster servers (each has gluster brick
>>> on it) and "dd" returns the final I/O throughput. The Internet is 40G
>>> infiniband, although I didn't do any specific configurations to use
>>> advanced features.
>>>
>>
>> Your dd command is inaccurate, as it'll hit the client cache. It is also
>> single threaded. I suggest switching to fio.
>> Y.
>>
>>
>>> What confuses me is that the storage I/O seems not to relate to the
>>> number of storage nodes, but Gluster documents said it should be linear
>>> scaling. For example, when "write-behind" is on, and when Infiniband "jumbo
>>> frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no
>>> matter I have 2 brick servers or 8 brick servers -- for 2 server case, each
>>> server can have ~400 MB/sec; for 4 server case, each server can have
>>> ~200MB/sec. That said, each server I/O does aggregate to the final storage
>>> I/O (800 MB/sec), but this is not "linear scale-out".
>>>
>>> Can somebody help me to understand why this is the case? I certainly can
>>> have some misunderstanding/misconfiguration here. Please correct me if I
>>> do, thanks!
>>>
>>> Best,
>>> Qing
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/f88024fd/attachment.html>

From g.danti at assyoma.it  Tue Jul 21 20:22:27 2020
From: g.danti at assyoma.it (Gionatan Danti)
Date: Tue, 21 Jul 2020 22:22:27 +0200
Subject: [Gluster-users] Replica scenario in low speed link
In-Reply-To: <32558859-9d58-294a-a987-c619bb83dfb6@hawai.it>
References: <32558859-9d58-294a-a987-c619bb83dfb6@hawai.it>
Message-ID: <331b41002bf5e9d9e6631be7259042c2@assyoma.it>

Il 2020-07-21 14:45 Stefano Danzi ha scritto:
> Hi!
> 
> I have a strange request. I don't know if some gluster settings could 
> help me.
> 
> There are two buildings linked using a wifi bridge.
> The main building host the data centre.? In the other building there
> are an office
> that need to use a file server with high speed, so the wifi link is not 
> optimal.
> 
> My fantasy suggest me to have a server in this office running samba and 
> gluster.
> Gluster have to read/write immediately from the local storage and
> leisurely replicate on remote storage (maybe a replica 3,
> or replica 2 + arbiter with locally one replica and the arbiter).
> In case of local storage/machine fault the system have to read/write
> using the remote brick over "slow" link
> and restart to use local storage when it return to be available and 
> synced.
> 
> Samba could be running only on this office or have two instances, in
> HA cluster, one in this office and
> other in the main building.
> 
> Anyone have any suggestions or have they had a similar need?
> 
> Thanks, bye.

Hi, from what I remember, Gluster (and especially the AFR module) is not 
well suited for high latency, low bandwidth scenario: the risk of split 
brain and link congestion is simply too high.

Full disclaimer: I tried a similar setup in the Gluster 3.3.x days, and 
it did not work well. But this was many years ago, maybe other can prove 
me wrong or give some other advice.

Regards.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8

From qw at g.clemson.edu  Tue Jul 21 23:29:43 2020
From: qw at g.clemson.edu (Qing Wang)
Date: Tue, 21 Jul 2020 19:29:43 -0400
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
	<CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
	<CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>
Message-ID: <CADkK8gvKdO6EapmAA12VzCGj8m=BAvnEbW32QEaB-tTnrf6gQQ@mail.gmail.com>

fio gives me the correct linear scale-out results, and you're right, the
storage cache is the root cause that makes the dd measurement results not
accurate at all.

Thanks,
Qing


On Tue, Jul 21, 2020 at 2:53 PM Yaniv Kaul <ykaul at redhat.com> wrote:

>
>
> On Tue, 21 Jul 2020, 21:43 Qing Wang <qw at g.clemson.edu> wrote:
>
>> Hi Yaniv,
>>
>> Thanks for the quick response. I forget to mention I am testing the
>> writing performance, not reading. In this case, would the client cache hit
>> rate still be a big issue?
>>
>
> It's not hitting the storage directly. Since it's also single threaded, it
> may also not saturate it. I highly recommend testing properly.
> Y.
>
>
>> I'll use fio to run my test once again, thanks for the suggestion.
>>
>> Thanks,
>> Qing
>>
>> On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>>
>>>
>>>
>>> On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to test Gluster linear scale-out performance by adding more
>>>> storage server/bricks, and measure the storage I/O performance. To vary the
>>>> storage server number, I create several "stripe" volumes that contain 2
>>>> brick servers, 3 brick servers, 4 brick servers, and so on. On gluster
>>>> client side, I used "dd if=/dev/zero of=/mnt/glusterfs/dns_test_data_26g
>>>> bs=1M count=26000" to create 26G data (or larger size), and those data will
>>>> be distributed to the corresponding gluster servers (each has gluster brick
>>>> on it) and "dd" returns the final I/O throughput. The Internet is 40G
>>>> infiniband, although I didn't do any specific configurations to use
>>>> advanced features.
>>>>
>>>
>>> Your dd command is inaccurate, as it'll hit the client cache. It is also
>>> single threaded. I suggest switching to fio.
>>> Y.
>>>
>>>
>>>> What confuses me is that the storage I/O seems not to relate to the
>>>> number of storage nodes, but Gluster documents said it should be linear
>>>> scaling. For example, when "write-behind" is on, and when Infiniband "jumbo
>>>> frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no
>>>> matter I have 2 brick servers or 8 brick servers -- for 2 server case, each
>>>> server can have ~400 MB/sec; for 4 server case, each server can have
>>>> ~200MB/sec. That said, each server I/O does aggregate to the final storage
>>>> I/O (800 MB/sec), but this is not "linear scale-out".
>>>>
>>>> Can somebody help me to understand why this is the case? I certainly
>>>> can have some misunderstanding/misconfiguration here. Please correct me if
>>>> I do, thanks!
>>>>
>>>> Best,
>>>> Qing
>>>> ________
>>>>
>>>>
>>>>
>>>> Community Meeting Calendar:
>>>>
>>>> Schedule -
>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>> Bridge: https://bluejeans.com/441850968
>>>>
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200721/92099dfe/attachment.html>

From rkothiya at redhat.com  Wed Jul 22 16:27:01 2020
From: rkothiya at redhat.com (Rinku Kothiya)
Date: Wed, 22 Jul 2020 21:57:01 +0530
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
Message-ID: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>

Hi,

The Gluster community is pleased to announce the release of Gluster7.7
(packages available at [1]).
Release notes for the release can be found at [2].

Major changes, features and limitations addressed in this release:
None

Please Note: Some of the packages are unavailable and we are working on it.
We will release them soon.

Thanks,
Gluster community

References:

[1] Packages for 7.7:
https://download.gluster.org/pub/gluster/glusterfs/7/7.7/

[2] Release notes for 7.7:
https://docs.gluster.org/en/latest/release-notes/7.7/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200722/3f206862/attachment.html>

From qw at g.clemson.edu  Thu Jul 23 07:08:05 2020
From: qw at g.clemson.edu (Qing Wang)
Date: Thu, 23 Jul 2020 03:08:05 -0400
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CADkK8gvKdO6EapmAA12VzCGj8m=BAvnEbW32QEaB-tTnrf6gQQ@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
	<CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
	<CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>
	<CADkK8gvKdO6EapmAA12VzCGj8m=BAvnEbW32QEaB-tTnrf6gQQ@mail.gmail.com>
Message-ID: <CADkK8gs1pDrAmMA_ONnc06kGAX8uxr5A7CarGtkYMFfTg=1sTA@mail.gmail.com>

Hi,

I have one more question about the Gluster linear scale-out performance
regarding the "write-behind off" case specifically -- when "write-behind"
is off, and still the stripe volumes and other settings as early thread
posted, the storage I/O seems not to relate to the number of storage nodes.
In my experiment, no matter I have 2 brick server nodes or 8 brick server
nodes, the aggregated gluster I/O performance is ~100MB/sec. And fio
benchmark measurement gives the same result. If "write behind" is on, then
the storage performance is linear scale-out along with the # of brick
server nodes increasing.

No matter the write behind option is on/off, I thought the gluster I/O
performance should be pulled and aggregated together as a whole. If that is
the case, why do I get a consistent gluster performance (~100MB/sec) when
"write behind" is off? Please advise me if I misunderstood something.

Thanks,
Qing




On Tue, Jul 21, 2020 at 7:29 PM Qing Wang <qw at g.clemson.edu> wrote:

> fio gives me the correct linear scale-out results, and you're right, the
> storage cache is the root cause that makes the dd measurement results not
> accurate at all.
>
> Thanks,
> Qing
>
>
> On Tue, Jul 21, 2020 at 2:53 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>
>>
>>
>> On Tue, 21 Jul 2020, 21:43 Qing Wang <qw at g.clemson.edu> wrote:
>>
>>> Hi Yaniv,
>>>
>>> Thanks for the quick response. I forget to mention I am testing the
>>> writing performance, not reading. In this case, would the client cache hit
>>> rate still be a big issue?
>>>
>>
>> It's not hitting the storage directly. Since it's also single threaded,
>> it may also not saturate it. I highly recommend testing properly.
>> Y.
>>
>>
>>> I'll use fio to run my test once again, thanks for the suggestion.
>>>
>>> Thanks,
>>> Qing
>>>
>>> On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>>>
>>>>
>>>>
>>>> On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to test Gluster linear scale-out performance by adding
>>>>> more storage server/bricks, and measure the storage I/O performance. To
>>>>> vary the storage server number, I create several "stripe" volumes that
>>>>> contain 2 brick servers, 3 brick servers, 4 brick servers, and so on. On
>>>>> gluster client side, I used "dd if=/dev/zero
>>>>> of=/mnt/glusterfs/dns_test_data_26g bs=1M count=26000" to create 26G data
>>>>> (or larger size), and those data will be distributed to the corresponding
>>>>> gluster servers (each has gluster brick on it) and "dd" returns the final
>>>>> I/O throughput. The Internet is 40G infiniband, although I didn't do any
>>>>> specific configurations to use advanced features.
>>>>>
>>>>
>>>> Your dd command is inaccurate, as it'll hit the client cache. It is
>>>> also single threaded. I suggest switching to fio.
>>>> Y.
>>>>
>>>>
>>>>> What confuses me is that the storage I/O seems not to relate to the
>>>>> number of storage nodes, but Gluster documents said it should be linear
>>>>> scaling. For example, when "write-behind" is on, and when Infiniband "jumbo
>>>>> frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no
>>>>> matter I have 2 brick servers or 8 brick servers -- for 2 server case, each
>>>>> server can have ~400 MB/sec; for 4 server case, each server can have
>>>>> ~200MB/sec. That said, each server I/O does aggregate to the final storage
>>>>> I/O (800 MB/sec), but this is not "linear scale-out".
>>>>>
>>>>> Can somebody help me to understand why this is the case? I certainly
>>>>> can have some misunderstanding/misconfiguration here. Please correct me if
>>>>> I do, thanks!
>>>>>
>>>>> Best,
>>>>> Qing
>>>>> ________
>>>>>
>>>>>
>>>>>
>>>>> Community Meeting Calendar:
>>>>>
>>>>> Schedule -
>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>> Bridge: https://bluejeans.com/441850968
>>>>>
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200723/ee42cb09/attachment.html>

From nico at furyweb.fr  Fri Jul 24 06:49:53 2020
From: nico at furyweb.fr (nico at furyweb.fr)
Date: Fri, 24 Jul 2020 08:49:53 +0200 (CEST)
Subject: [Gluster-users] fuse client 7.6 crashing regularly
Message-ID: <1966362365.24562.1595573393679.JavaMail.zimbra@furyweb.fr>

We're using gluster in a production environement, 3 nodes (2 data + 1 arbiter). 
One of our VM gluster fuse client is regularly crashing on a particular volume, we recently upgraded all nodes and client to 7.6 but client is still crashing. 

All cluster nodes & client are Debian stretch (9.12), gluster was installed from our local gluster apt repository mirror and op-version is set to 70200. 

Volume contains a lot of files & directories but performance doesn't really matters, it seems to crash during this command : 
find logscli -mtime +1 -type f | tar c -T - -f - --remove-files | tar xpf - -C /drbd 

Volume was remonted this morning with DEBUG log level, waiting for next crash. 

Volume attributes are 
Option Value 
------ ----- 
cluster.lookup-unhashed on 
cluster.lookup-optimize on 
cluster.min-free-disk 10% 
cluster.min-free-inodes 5% 
cluster.rebalance-stats off 
cluster.subvols-per-directory (null) 
cluster.readdir-optimize off 
cluster.rsync-hash-regex (null) 
cluster.extra-hash-regex (null) 
cluster.dht-xattr-name trusted.glusterfs.dht 
cluster.randomize-hash-range-by-gfid off 
cluster.rebal-throttle normal 
cluster.lock-migration off 
cluster.force-migration off 
cluster.local-volume-name (null) 
cluster.weighted-rebalance on 
cluster.switch-pattern (null) 
cluster.entry-change-log on 
cluster.read-subvolume (null) 
cluster.read-subvolume-index -1 
cluster.read-hash-mode 1 
cluster.background-self-heal-count 8 
cluster.metadata-self-heal off 
cluster.data-self-heal off 
cluster.entry-self-heal off 
cluster.self-heal-daemon enable 
cluster.heal-timeout 60 
cluster.self-heal-window-size 1 
cluster.data-change-log on 
cluster.metadata-change-log on 
cluster.data-self-heal-algorithm full 
cluster.eager-lock on 
disperse.eager-lock on 
disperse.other-eager-lock on 
disperse.eager-lock-timeout 1 
disperse.other-eager-lock-timeout 1 
cluster.quorum-type fixed 
cluster.quorum-count 1 
cluster.choose-local true 
cluster.self-heal-readdir-size 1KB 
cluster.post-op-delay-secs 1 
cluster.ensure-durability on 
cluster.consistent-metadata no 
cluster.heal-wait-queue-length 128 
cluster.favorite-child-policy none 
cluster.full-lock yes 
cluster.optimistic-change-log on 
diagnostics.latency-measurement off 
diagnostics.dump-fd-stats off 
diagnostics.count-fop-hits off 
diagnostics.brick-log-level INFO 
diagnostics.client-log-level ERROR 
diagnostics.brick-sys-log-level CRITICAL 
diagnostics.client-sys-log-level CRITICAL 
diagnostics.brick-logger (null) 
diagnostics.client-logger (null) 
diagnostics.brick-log-format (null) 
diagnostics.client-log-format (null) 
diagnostics.brick-log-buf-size 5 
diagnostics.client-log-buf-size 5 
diagnostics.brick-log-flush-timeout 120 
diagnostics.client-log-flush-timeout 120 
diagnostics.stats-dump-interval 0 
diagnostics.fop-sample-interval 0 
diagnostics.stats-dump-format json 
diagnostics.fop-sample-buf-size 65535 
diagnostics.stats-dnscache-ttl-sec 86400 
performance.cache-max-file-size 0 
performance.cache-min-file-size 0 
performance.cache-refresh-timeout 1 
performance.cache-priority 
performance.cache-size 32MB 
performance.io-thread-count 16 
performance.high-prio-threads 16 
performance.normal-prio-threads 16 
performance.low-prio-threads 16 
performance.least-prio-threads 1 
performance.enable-least-priority on 
performance.iot-watchdog-secs (null) 
performance.iot-cleanup-disconnected-reqsoff 
performance.iot-pass-through false 
performance.io-cache-pass-through false 
performance.cache-size 128MB 
performance.qr-cache-timeout 1 
performance.cache-invalidation false 
performance.ctime-invalidation false 
performance.flush-behind on 
performance.nfs.flush-behind on 
performance.write-behind-window-size 1MB 
performance.resync-failed-syncs-after-fsyncoff 
performance.nfs.write-behind-window-size1MB 
performance.strict-o-direct off 
performance.nfs.strict-o-direct off 
performance.strict-write-ordering off 
performance.nfs.strict-write-ordering off 
performance.write-behind-trickling-writeson 
performance.aggregate-size 128KB 
performance.nfs.write-behind-trickling-writeson 
performance.lazy-open yes 
performance.read-after-open yes 
performance.open-behind-pass-through false 
performance.read-ahead-page-count 4 
performance.read-ahead-pass-through false 
performance.readdir-ahead-pass-through false 
performance.md-cache-pass-through false 
performance.md-cache-timeout 1 
performance.cache-swift-metadata true 
performance.cache-samba-metadata false 
performance.cache-capability-xattrs true 
performance.cache-ima-xattrs true 
performance.md-cache-statfs off 
performance.xattr-cache-list 
performance.nl-cache-pass-through false 
network.frame-timeout 1800 
network.ping-timeout 5 
network.tcp-window-size (null) 
client.ssl on 
network.remote-dio disable 
client.event-threads 2 
client.tcp-user-timeout 0 
client.keepalive-time 20 
client.keepalive-interval 2 
client.keepalive-count 9 
network.tcp-window-size (null) 
network.inode-lru-limit 16384 
auth.allow * 
auth.reject (null) 
transport.keepalive 1 
server.allow-insecure on 
server.root-squash off 
server.all-squash off 
server.anonuid 65534 
server.anongid 65534 
server.statedump-path /var/run/gluster 
server.outstanding-rpc-limit 64 
server.ssl on 
auth.ssl-allow * 
server.manage-gids off 
server.dynamic-auth on 
client.send-gids on 
server.gid-timeout 300 
server.own-thread (null) 
server.event-threads 2 
server.tcp-user-timeout 42 
server.keepalive-time 20 
server.keepalive-interval 2 
server.keepalive-count 9 
transport.listen-backlog 1024 
ssl.cipher-list HIGH:!SSLv2 
transport.address-family inet 
performance.write-behind on 
performance.read-ahead on 
performance.readdir-ahead on 
performance.io-cache on 
performance.open-behind on 
performance.quick-read on 
performance.nl-cache off 
performance.stat-prefetch on 
performance.client-io-threads off 
performance.nfs.write-behind on 
performance.nfs.read-ahead off 
performance.nfs.io-cache off 
performance.nfs.quick-read off 
performance.nfs.stat-prefetch off 
performance.nfs.io-threads off 
performance.force-readdirp true 
performance.cache-invalidation false 
performance.global-cache-invalidation true 
features.uss off 
features.snapshot-directory .snaps 
features.show-snapshot-directory off 
features.tag-namespaces off 
network.compression off 
network.compression.window-size -15 
network.compression.mem-level 8 
network.compression.min-size 0 
network.compression.compression-level -1 
network.compression.debug false 
features.default-soft-limit 80% 
features.soft-timeout 60 
features.hard-timeout 5 
features.alert-time 86400 
features.quota-deem-statfs off 
geo-replication.indexing off 
geo-replication.indexing off 
geo-replication.ignore-pid-check off 
geo-replication.ignore-pid-check off 
features.quota off 
features.inode-quota off 
features.bitrot disable 
debug.trace off 
debug.log-history no 
debug.log-file no 
debug.exclude-ops (null) 
debug.include-ops (null) 
debug.error-gen off 
debug.error-failure (null) 
debug.error-number (null) 
debug.random-failure off 
debug.error-fops (null) 
nfs.disable on 
features.read-only off 
features.worm off 
features.worm-file-level off 
features.worm-files-deletable on 
features.default-retention-period 120 
features.retention-mode relax 
features.auto-commit-period 180 
storage.linux-aio off 
storage.batch-fsync-mode reverse-fsync 
storage.batch-fsync-delay-usec 0 
storage.owner-uid -1 
storage.owner-gid -1 
storage.node-uuid-pathinfo off 
storage.health-check-interval 30 
storage.build-pgfid off 
storage.gfid2path on 
storage.gfid2path-separator : 
storage.reserve 1 
storage.reserve-size 0 
storage.health-check-timeout 10 
storage.fips-mode-rchecksum off 
storage.force-create-mode 0000 
storage.force-directory-mode 0000 
storage.create-mask 0777 
storage.create-directory-mask 0777 
storage.max-hardlinks 100 
features.ctime off 
config.gfproxyd off 
cluster.server-quorum-type off 
cluster.server-quorum-ratio 51 
changelog.changelog off 
changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs 
changelog.encoding ascii 
changelog.rollover-time 15 
changelog.fsync-interval 5 
changelog.changelog-barrier-timeout 120 
changelog.capture-del-path off 
features.barrier disable 
features.barrier-timeout 120 
features.trash off 
features.trash-dir .trashcan 
features.trash-eliminate-path (null) 
features.trash-max-filesize 5MB 
features.trash-internal-op off 
cluster.enable-shared-storage disable 
locks.trace off 
locks.mandatory-locking off 
cluster.disperse-self-heal-daemon enable 
cluster.quorum-reads false 
client.bind-insecure (null) 
features.timeout 45 
features.failover-hosts (null) 
features.shard off 
features.shard-block-size 64MB 
features.shard-lru-limit 16384 
features.shard-deletion-rate 100 
features.scrub-throttle lazy 
features.scrub-freq biweekly 
features.scrub false 
features.expiry-time 120 
features.cache-invalidation off 
features.cache-invalidation-timeout 60 
features.leases off 
features.lease-lock-recall-timeout 60 
disperse.background-heals 8 
disperse.heal-wait-qlength 128 
cluster.heal-timeout 60 
dht.force-readdirp on 
disperse.read-policy gfid-hash 
cluster.shd-max-threads 1 
cluster.shd-wait-qlength 1024 
cluster.locking-scheme full 
cluster.granular-entry-heal no 
features.locks-revocation-secs 0 
features.locks-revocation-clear-all false 
features.locks-revocation-max-blocked 0 
features.locks-monkey-unlocking false 
features.locks-notify-contention no 
features.locks-notify-contention-delay 5 
disperse.shd-max-threads 1 
disperse.shd-wait-qlength 1024 
disperse.cpu-extensions auto 
disperse.self-heal-window-size 1 
cluster.use-compound-fops off 
performance.parallel-readdir off 
performance.rda-request-size 131072 
performance.rda-low-wmark 4096 
performance.rda-high-wmark 128KB 
performance.rda-cache-limit 10MB 
performance.nl-cache-positive-entry false 
performance.nl-cache-limit 10MB 
performance.nl-cache-timeout 60 
cluster.brick-multiplex disable 
glusterd.vol_count_per_thread 100 
cluster.max-bricks-per-process 250 
disperse.optimistic-change-log on 
disperse.stripe-cache 4 
cluster.halo-enabled False 
cluster.halo-shd-max-latency 99999 
cluster.halo-nfsd-max-latency 5 
cluster.halo-max-latency 5 
cluster.halo-max-replicas 99999 
cluster.halo-min-replicas 2 
features.selinux on 
cluster.daemon-log-level INFO 
debug.delay-gen off 
delay-gen.delay-percentage 10% 
delay-gen.delay-duration 100000 
delay-gen.enable 
disperse.parallel-writes on 
features.sdfs off 
features.cloudsync off 
features.ctime off 
ctime.noatime on 
features.cloudsync-storetype (null) 
features.enforce-mandatory-lock off 
config.global-threading off 
config.client-threads 16 
config.brick-threads 16 
features.cloudsync-remote-read off 
features.cloudsync-store-id (null) 
features.cloudsync-product-id (null) 


Crash log found in /var/log/glusterfs/partage-logscli.log 
2020-07-23 02:34:36 
configuration details: 
argp 1 
backtrace 1 
dlfcn 1 
libpthread 1 
llistxattr 1 
setfsid 1 
spinlock 1 
epoll.h 1 
xattr.h 1 
st_atim.tv_nsec 1 
package-string: glusterfs 7.6 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x25e50)[0x7fbe02138e50] 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7fbe021434b7] 
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7fbe00b87060] 
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x0)[0x7fbe01396b40] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x32f5)[0x7fbdfa89b2f5] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3c52)[0x7fbdfa89bc52] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3dac)[0x7fbdfa89bdac] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3f73)[0x7fbdfa89bf73] 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_unlink+0xbc)[0x7fbe021c401c] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/md-cache.so(+0x4495)[0x7fbdfa46c495] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/debug/io-stats.so(+0x5f44)[0x7fbdfa23af44] 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_unlink+0xbc)[0x7fbe021c401c] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x1154f)[0x7fbdff7dc54f] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x7775)[0x7fbdff7d2775] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x74c8)[0x7fbdff7d24c8] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x77be)[0x7fbdff7d27be] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x6ac3)[0x7fbdff7d1ac3] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x7188)[0x7fbdff7d2188] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x74e8)[0x7fbdff7d24e8] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x779e)[0x7fbdff7d279e] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x77e0)[0x7fbdff7d27e0] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x83f9)[0x7fbdff7d33f9] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x21d3c)[0x7fbdff7ecd3c] 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fbe013944a4] 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fbe00c3cd0f] 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200724/a4c7008e/attachment.html>

From jahernan at redhat.com  Fri Jul 24 07:51:22 2020
From: jahernan at redhat.com (Xavi Hernandez)
Date: Fri, 24 Jul 2020 09:51:22 +0200
Subject: [Gluster-users] fuse client 7.6 crashing regularly
In-Reply-To: <1966362365.24562.1595573393679.JavaMail.zimbra@furyweb.fr>
References: <1966362365.24562.1595573393679.JavaMail.zimbra@furyweb.fr>
Message-ID: <CALtvDK5GfRbANG_+J8teUo4=dqg19Jg2X5mZsSMNirZW+AFfRw@mail.gmail.com>

Hi,

it seems to be crashing inside open-behind. There's a known bug in that
xlator that caused a crash. It has been fixed in 7.7, recently released.
Can you try to upgrade ?

Xavi

On Fri, Jul 24, 2020 at 8:50 AM <nico at furyweb.fr> wrote:

> We're using gluster in a production environement, 3 nodes (2 data + 1
> arbiter).
> One of our VM gluster fuse client is regularly crashing on a particular
> volume, we recently upgraded all nodes and client to 7.6 but client is
> still crashing.
>
> All cluster nodes & client are Debian stretch (9.12), gluster was
> installed from our local gluster apt repository mirror and op-version is
> set to 70200.
>
> Volume contains a lot of files & directories but performance doesn't
> really matters, it seems to crash during this command :
> find logscli -mtime +1 -type f | tar c -T - -f - --remove-files | tar xpf
> - -C /drbd
>
> Volume was remonted this morning with DEBUG log level, waiting for next
> crash.
>
> Volume attributes are
> Option                                  Value
>
> ------                                  -----
>
> cluster.lookup-unhashed                 on
>
> cluster.lookup-optimize                 on
>
> cluster.min-free-disk                   10%
>
> cluster.min-free-inodes                 5%
>
> cluster.rebalance-stats                 off
>
> cluster.subvols-per-directory           (null)
>
> cluster.readdir-optimize                off
>
> cluster.rsync-hash-regex                (null)
>
> cluster.extra-hash-regex                (null)
>
> cluster.dht-xattr-name                  trusted.glusterfs.dht
>
> cluster.randomize-hash-range-by-gfid    off
>
> cluster.rebal-throttle                  normal
>
> cluster.lock-migration                  off
>
> cluster.force-migration                 off
>
> cluster.local-volume-name               (null)
>
> cluster.weighted-rebalance              on
>
> cluster.switch-pattern                  (null)
>
> cluster.entry-change-log                on
>
> cluster.read-subvolume                  (null)
>
> cluster.read-subvolume-index            -1
>
> cluster.read-hash-mode                  1
>
> cluster.background-self-heal-count      8
>
> cluster.metadata-self-heal              off
>
> cluster.data-self-heal                  off
>
> cluster.entry-self-heal                 off
>
> cluster.self-heal-daemon                enable
>
> cluster.heal-timeout                    60
>
> cluster.self-heal-window-size           1
>
> cluster.data-change-log                 on
>
> cluster.metadata-change-log             on
>
> cluster.data-self-heal-algorithm        full
>
> cluster.eager-lock                      on
>
> disperse.eager-lock                     on
>
> disperse.other-eager-lock               on
>
> disperse.eager-lock-timeout             1
>
> disperse.other-eager-lock-timeout       1
>
> cluster.quorum-type                     fixed
>
> cluster.quorum-count                    1
>
> cluster.choose-local                    true
>
> cluster.self-heal-readdir-size          1KB
>
> cluster.post-op-delay-secs              1
>
> cluster.ensure-durability               on
>
> cluster.consistent-metadata             no
>
> cluster.heal-wait-queue-length          128
>
> cluster.favorite-child-policy           none
>
> cluster.full-lock                       yes
>
> cluster.optimistic-change-log           on
>
> diagnostics.latency-measurement         off
>
> diagnostics.dump-fd-stats               off
>
> diagnostics.count-fop-hits              off
>
> diagnostics.brick-log-level             INFO
>
> diagnostics.client-log-level            ERROR
>
> diagnostics.brick-sys-log-level         CRITICAL
>
> diagnostics.client-sys-log-level        CRITICAL
>
> diagnostics.brick-logger                (null)
>
> diagnostics.client-logger               (null)
>
> diagnostics.brick-log-format            (null)
>
> diagnostics.client-log-format           (null)
>
> diagnostics.brick-log-buf-size          5
>
> diagnostics.client-log-buf-size         5
>
> diagnostics.brick-log-flush-timeout     120
>
> diagnostics.client-log-flush-timeout    120
>
> diagnostics.stats-dump-interval         0
>
> diagnostics.fop-sample-interval         0
>
> diagnostics.stats-dump-format           json
>
> diagnostics.fop-sample-buf-size         65535
>
> diagnostics.stats-dnscache-ttl-sec      86400
>
> performance.cache-max-file-size         0
>
> performance.cache-min-file-size         0
>
> performance.cache-refresh-timeout       1
>
> performance.cache-priority
>
> performance.cache-size                  32MB
>
> performance.io-thread-count             16
>
> performance.high-prio-threads           16
>
> performance.normal-prio-threads         16
>
> performance.low-prio-threads            16
>
> performance.least-prio-threads          1
>
> performance.enable-least-priority       on
>
> performance.iot-watchdog-secs           (null)
>
> performance.iot-cleanup-disconnected-reqsoff
>
> performance.iot-pass-through            false
>
> performance.io-cache-pass-through       false
>
> performance.cache-size                  128MB
>
> performance.qr-cache-timeout            1
>
> performance.cache-invalidation          false
>
> performance.ctime-invalidation          false
>
> performance.flush-behind                on
>
> performance.nfs.flush-behind            on
>
> performance.write-behind-window-size    1MB
>
> performance.resync-failed-syncs-after-fsyncoff
>
> performance.nfs.write-behind-window-size1MB
>
> performance.strict-o-direct             off
>
> performance.nfs.strict-o-direct         off
>
> performance.strict-write-ordering       off
>
> performance.nfs.strict-write-ordering   off
>
> performance.write-behind-trickling-writeson
>
> performance.aggregate-size              128KB
>
> performance.nfs.write-behind-trickling-writeson
>
> performance.lazy-open                   yes
>
> performance.read-after-open             yes
>
> performance.open-behind-pass-through    false
>
> performance.read-ahead-page-count       4
>
> performance.read-ahead-pass-through     false
>
> performance.readdir-ahead-pass-through  false
>
> performance.md-cache-pass-through       false
>
> performance.md-cache-timeout            1
>
> performance.cache-swift-metadata        true
>
> performance.cache-samba-metadata        false
>
> performance.cache-capability-xattrs     true
>
> performance.cache-ima-xattrs            true
>
> performance.md-cache-statfs             off
>
> performance.xattr-cache-list
>
> performance.nl-cache-pass-through       false
>
> network.frame-timeout                   1800
>
> network.ping-timeout                    5
>
> network.tcp-window-size                 (null)
>
> client.ssl                              on
>
> network.remote-dio                      disable
>
> client.event-threads                    2
>
> client.tcp-user-timeout                 0
>
> client.keepalive-time                   20
>
> client.keepalive-interval               2
>
> client.keepalive-count                  9
>
> network.tcp-window-size                 (null)
>
> network.inode-lru-limit                 16384
>
> auth.allow                              *
>
> auth.reject                             (null)
>
> transport.keepalive                     1
>
> server.allow-insecure                   on
>
> server.root-squash                      off
>
> server.all-squash                       off
>
> server.anonuid                          65534
>
> server.anongid                          65534
>
> server.statedump-path                   /var/run/gluster
>
> server.outstanding-rpc-limit            64
>
> server.ssl                              on
>
> auth.ssl-allow                          *
>
> server.manage-gids                      off
>
> server.dynamic-auth                     on
>
> client.send-gids                        on
>
> server.gid-timeout                      300
>
> server.own-thread                       (null)
>
> server.event-threads                    2
>
> server.tcp-user-timeout                 42
>
> server.keepalive-time                   20
>
> server.keepalive-interval               2
>
> server.keepalive-count                  9
>
> transport.listen-backlog                1024
>
> ssl.cipher-list                         HIGH:!SSLv2
>
> transport.address-family                inet
>
> performance.write-behind                on
>
> performance.read-ahead                  on
>
> performance.readdir-ahead               on
>
> performance.io-cache                    on
>
> performance.open-behind                 on
>
> performance.quick-read                  on
>
> performance.nl-cache                    off
>
> performance.stat-prefetch               on
>
> performance.client-io-threads           off
>
> performance.nfs.write-behind            on
>
> performance.nfs.read-ahead              off
>
> performance.nfs.io-cache                off
>
> performance.nfs.quick-read              off
>
> performance.nfs.stat-prefetch           off
>
> performance.nfs.io-threads              off
>
> performance.force-readdirp              true
>
> performance.cache-invalidation          false
>
> performance.global-cache-invalidation   true
>
> features.uss                            off
>
> features.snapshot-directory             .snaps
>
> features.show-snapshot-directory        off
>
> features.tag-namespaces                 off
>
> network.compression                     off
>
> network.compression.window-size         -15
>
> network.compression.mem-level           8
>
> network.compression.min-size            0
>
> network.compression.compression-level   -1
>
> network.compression.debug               false
>
> features.default-soft-limit             80%
>
> features.soft-timeout                   60
>
> features.hard-timeout                   5
>
> features.alert-time                     86400
>
> features.quota-deem-statfs              off
>
> geo-replication.indexing                off
>
> geo-replication.indexing                off
>
> geo-replication.ignore-pid-check        off
>
> geo-replication.ignore-pid-check        off
>
> features.quota                          off
>
> features.inode-quota                    off
>
> features.bitrot                         disable
>
> debug.trace                             off
>
> debug.log-history                       no
>
> debug.log-file                          no
>
> debug.exclude-ops                       (null)
>
> debug.include-ops                       (null)
>
> debug.error-gen                         off
>
> debug.error-failure                     (null)
>
> debug.error-number                      (null)
>
> debug.random-failure                    off
>
> debug.error-fops                        (null)
>
> nfs.disable                             on
>
> features.read-only                      off
>
> features.worm                           off
>
> features.worm-file-level                off
>
> features.worm-files-deletable           on
>
> features.default-retention-period       120
>
> features.retention-mode                 relax
>
> features.auto-commit-period             180
>
> storage.linux-aio                       off
>
> storage.batch-fsync-mode                reverse-fsync
>
> storage.batch-fsync-delay-usec          0
>
> storage.owner-uid                       -1
>
> storage.owner-gid                       -1
>
> storage.node-uuid-pathinfo              off
>
> storage.health-check-interval           30
>
> storage.build-pgfid                     off
>
> storage.gfid2path                       on
>
> storage.gfid2path-separator             :
>
> storage.reserve                         1
>
> storage.reserve-size                    0
>
> storage.health-check-timeout            10
>
> storage.fips-mode-rchecksum             off
>
> storage.force-create-mode               0000
>
> storage.force-directory-mode            0000
>
> storage.create-mask                     0777
>
> storage.create-directory-mask           0777
>
> storage.max-hardlinks                   100
>
> features.ctime                          off
>
> config.gfproxyd                         off
>
> cluster.server-quorum-type              off
>
> cluster.server-quorum-ratio             51
>
> changelog.changelog                     off
>
> changelog.changelog-dir                 {{ brick.path
> }}/.glusterfs/changelogs
> changelog.encoding                      ascii
>
> changelog.rollover-time                 15
>
> changelog.fsync-interval                5
>
> changelog.changelog-barrier-timeout     120
>
> changelog.capture-del-path              off
>
> features.barrier                        disable
>
> features.barrier-timeout                120
>
> features.trash                          off
>
> features.trash-dir                      .trashcan
>
> features.trash-eliminate-path           (null)
>
> features.trash-max-filesize             5MB
>
> features.trash-internal-op              off
>
> cluster.enable-shared-storage           disable
>
> locks.trace                             off
>
> locks.mandatory-locking                 off
>
> cluster.disperse-self-heal-daemon       enable
>
> cluster.quorum-reads                    false
>
> client.bind-insecure                    (null)
>
> features.timeout                        45
>
> features.failover-hosts                 (null)
>
> features.shard                          off
>
> features.shard-block-size               64MB
>
> features.shard-lru-limit                16384
>
> features.shard-deletion-rate            100
>
> features.scrub-throttle                 lazy
>
> features.scrub-freq                     biweekly
>
> features.scrub                          false
>
> features.expiry-time                    120
>
> features.cache-invalidation             off
>
> features.cache-invalidation-timeout     60
>
> features.leases                         off
>
> features.lease-lock-recall-timeout      60
>
> disperse.background-heals               8
>
> disperse.heal-wait-qlength              128
>
> cluster.heal-timeout                    60
>
> dht.force-readdirp                      on
>
> disperse.read-policy                    gfid-hash
>
> cluster.shd-max-threads                 1
>
> cluster.shd-wait-qlength                1024
>
> cluster.locking-scheme                  full
>
> cluster.granular-entry-heal             no
>
> features.locks-revocation-secs          0
>
> features.locks-revocation-clear-all     false
>
> features.locks-revocation-max-blocked   0
>
> features.locks-monkey-unlocking         false
>
> features.locks-notify-contention        no
>
> features.locks-notify-contention-delay  5
>
> disperse.shd-max-threads                1
>
> disperse.shd-wait-qlength               1024
>
> disperse.cpu-extensions                 auto
>
> disperse.self-heal-window-size          1
>
> cluster.use-compound-fops               off
>
> performance.parallel-readdir            off
>
> performance.rda-request-size            131072
>
> performance.rda-low-wmark               4096
>
> performance.rda-high-wmark              128KB
>
> performance.rda-cache-limit             10MB
>
> performance.nl-cache-positive-entry     false
>
> performance.nl-cache-limit              10MB
>
> performance.nl-cache-timeout            60
>
> cluster.brick-multiplex                 disable
>
> glusterd.vol_count_per_thread           100
>
> cluster.max-bricks-per-process          250
>
> disperse.optimistic-change-log          on
>
> disperse.stripe-cache                   4
>
> cluster.halo-enabled                    False
>
> cluster.halo-shd-max-latency            99999
>
> cluster.halo-nfsd-max-latency           5
>
> cluster.halo-max-latency                5
>
> cluster.halo-max-replicas               99999
>
> cluster.halo-min-replicas               2
>
> features.selinux                        on
>
> cluster.daemon-log-level                INFO
>
> debug.delay-gen                         off
>
> delay-gen.delay-percentage              10%
>
> delay-gen.delay-duration                100000
>
> delay-gen.enable
>
> disperse.parallel-writes                on
>
> features.sdfs                           off
>
> features.cloudsync                      off
>
> features.ctime                          off
>
> ctime.noatime                           on
>
> features.cloudsync-storetype            (null)
>
> features.enforce-mandatory-lock         off
>
> config.global-threading                 off
>
> config.client-threads                   16
>
> config.brick-threads                    16
>
> features.cloudsync-remote-read          off
>
> features.cloudsync-store-id             (null)
>
> features.cloudsync-product-id           (null)
>
>
> Crash log found in /var/log/glusterfs/partage-logscli.log
> 2020-07-23 02:34:36
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 7.6
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x25e50)[0x7fbe02138e50]
>
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7fbe021434b7]
> /lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7fbe00b87060]
>
> /lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x0)[0x7fbe01396b40]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x32f5)[0x7fbdfa89b2f5]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3c52)[0x7fbdfa89bc52]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3dac)[0x7fbdfa89bdac]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3f73)[0x7fbdfa89bf73]
>
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_unlink+0xbc)[0x7fbe021c401c]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/md-cache.so(+0x4495)[0x7fbdfa46c495]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/debug/io-stats.so(+0x5f44)[0x7fbdfa23af44]
>
> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_unlink+0xbc)[0x7fbe021c401c]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x1154f)[0x7fbdff7dc54f]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x7775)[0x7fbdff7d2775]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x74c8)[0x7fbdff7d24c8]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x77be)[0x7fbdff7d27be]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x6ac3)[0x7fbdff7d1ac3]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x7188)[0x7fbdff7d2188]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x74e8)[0x7fbdff7d24e8]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x779e)[0x7fbdff7d279e]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x77e0)[0x7fbdff7d27e0]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x83f9)[0x7fbdff7d33f9]
>
> /usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x21d3c)[0x7fbdff7ecd3c]
> /lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fbe013944a4]
> /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fbe00c3cd0f]
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200724/7a367b90/attachment.html>

From archon810 at gmail.com  Fri Jul 24 23:05:10 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Fri, 24 Jul 2020 16:05:10 -0700
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CADkK8gs1pDrAmMA_ONnc06kGAX8uxr5A7CarGtkYMFfTg=1sTA@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
	<CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
	<CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>
	<CADkK8gvKdO6EapmAA12VzCGj8m=BAvnEbW32QEaB-tTnrf6gQQ@mail.gmail.com>
	<CADkK8gs1pDrAmMA_ONnc06kGAX8uxr5A7CarGtkYMFfTg=1sTA@mail.gmail.com>
Message-ID: <CAD+dzQdYk2xvDd2XifwFroO7nEYMJzDs-3ZOx+_-=g8czJNV-Q@mail.gmail.com>

Speaking of fio, could the gluster team please help me understand something?

We've been having lots of performance issues related to gluster using
attached block storage on Linode. At some point, I figured out that Linode
has a cap of 500 IOPS on their block storage
<https://www.linode.com/community/questions/19437/does-a-dedicated-cpu-or-high-memory-plan-improve-disk-io-performance#answer-72142>
(with spikes to 1500 IOPS). The block storage we use is formatted xfs with
4KB bsize (block size).

I then ran a bunch of fio tests on the block storage itself (not the
gluster fuse mount), which performed horribly when the bs parameter was set
to 4k:
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test
--filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randwrite
--ramp_time=4
During these tests, fio ETA crawled to over an hour, at some point dropped
to 45min and I did see 500-1500 IOPS flash by briefly, then it went back
down to 0. I/O seems majorly choked for some reason, likely because gluster
is using some of it. Transfer speed with such 4k block size is 2 MB/s with
spikes to 6MB/s. This causes the load on the server to spike up to 100+ and
brings down all our servers.

Jobs: 1 (f=1): [w(1)][20.3%][r=0KiB/s,w=5908KiB/s][r=0,w=1477
IOPS][eta 43m:00s]    Jobs: 1 (f=1):
[w(1)][21.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 44m:54s]

xfs_info /mnt/citadel_block1
meta-data=/dev/sdc               isize=512    agcount=103, agsize=26214400 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=0, rmapbt=0
         =                       reflink=0
data     =                       bsize=4096   blocks=2684354560, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1log
   =internal log           bsize=4096   blocks=51200, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

When I increase the --bs param to fio from 4k to, say, 64k, transfer speed
goes up significantly and is more like 50MB/s, and at 256k, it's 200MB/s.

So what I'm trying to understand is:

   1. How does the xfs block size (4KB) relate to the block size in fio
   tests? If we're limited by IOPS, and xfs block size is 4KB, how can fio
   produce better results with varying --bs param?
   2. Would increasing the xfs data block size to something like 64-256KB
   help with our issue of choking IO and skyrocketing load?
   3. The worst hangs and load spikes happen when we reboot one of the
   gluster servers, but not when it's down - when it comes back online. Even
   with gluster not showing anything pending heal, my guess is it's still
   trying to do lots of IO between the 4 nodes for some reason, but I don't
   understand why.

I've been banging my head on the wall with this problem for months.
Appreciate any feedback here.

Thank you.

gluster volume info below

Volume Name: SNIP_data1
Type: Replicate
Volume ID: SNIP
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 4 = 4
Transport-type: tcp
Bricks:
Brick1: nexus2:/mnt/SNIP_block1/SNIP_data1
Brick2: forge:/mnt/SNIP_block1/SNIP_data1
Brick3: hive:/mnt/SNIP_block1/SNIP_data1
Brick4: citadel:/mnt/SNIP_block1/SNIP_data1
Options Reconfigured:
cluster.quorum-count: 1
cluster.quorum-type: fixed
network.ping-timeout: 5
network.remote-dio: enable
performance.rda-cache-limit: 256MB
performance.readdir-ahead: on
performance.parallel-readdir: on
network.inode-lru-limit: 500000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
cluster.readdir-optimize: on
performance.io-thread-count: 32
server.event-threads: 4
client.event-threads: 4
performance.read-ahead: off
cluster.lookup-optimize: on
performance.cache-size: 1GB
cluster.self-heal-daemon: enable
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: on
cluster.granular-entry-heal: enable
cluster.data-self-heal-algorithm: full


Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Thu, Jul 23, 2020 at 12:08 AM Qing Wang <qw at g.clemson.edu> wrote:

> Hi,
>
> I have one more question about the Gluster linear scale-out performance
> regarding the "write-behind off" case specifically -- when "write-behind"
> is off, and still the stripe volumes and other settings as early thread
> posted, the storage I/O seems not to relate to the number of storage
> nodes. In my experiment, no matter I have 2 brick server nodes or 8 brick
> server nodes, the aggregated gluster I/O performance is ~100MB/sec. And fio
> benchmark measurement gives the same result. If "write behind" is on, then
> the storage performance is linear scale-out along with the # of brick
> server nodes increasing.
>
> No matter the write behind option is on/off, I thought the gluster I/O
> performance should be pulled and aggregated together as a whole. If that is
> the case, why do I get a consistent gluster performance (~100MB/sec) when
> "write behind" is off? Please advise me if I misunderstood something.
>
> Thanks,
> Qing
>
>
>
>
> On Tue, Jul 21, 2020 at 7:29 PM Qing Wang <qw at g.clemson.edu> wrote:
>
>> fio gives me the correct linear scale-out results, and you're right, the
>> storage cache is the root cause that makes the dd measurement results not
>> accurate at all.
>>
>> Thanks,
>> Qing
>>
>>
>> On Tue, Jul 21, 2020 at 2:53 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>>
>>>
>>>
>>> On Tue, 21 Jul 2020, 21:43 Qing Wang <qw at g.clemson.edu> wrote:
>>>
>>>> Hi Yaniv,
>>>>
>>>> Thanks for the quick response. I forget to mention I am testing the
>>>> writing performance, not reading. In this case, would the client cache hit
>>>> rate still be a big issue?
>>>>
>>>
>>> It's not hitting the storage directly. Since it's also single threaded,
>>> it may also not saturate it. I highly recommend testing properly.
>>> Y.
>>>
>>>
>>>> I'll use fio to run my test once again, thanks for the suggestion.
>>>>
>>>> Thanks,
>>>> Qing
>>>>
>>>> On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul <ykaul at redhat.com> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, 21 Jul 2020, 21:30 Qing Wang <qw at g.clemson.edu> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I am trying to test Gluster linear scale-out performance by adding
>>>>>> more storage server/bricks, and measure the storage I/O performance. To
>>>>>> vary the storage server number, I create several "stripe" volumes that
>>>>>> contain 2 brick servers, 3 brick servers, 4 brick servers, and so on. On
>>>>>> gluster client side, I used "dd if=/dev/zero
>>>>>> of=/mnt/glusterfs/dns_test_data_26g bs=1M count=26000" to create 26G data
>>>>>> (or larger size), and those data will be distributed to the corresponding
>>>>>> gluster servers (each has gluster brick on it) and "dd" returns the final
>>>>>> I/O throughput. The Internet is 40G infiniband, although I didn't do any
>>>>>> specific configurations to use advanced features.
>>>>>>
>>>>>
>>>>> Your dd command is inaccurate, as it'll hit the client cache. It is
>>>>> also single threaded. I suggest switching to fio.
>>>>> Y.
>>>>>
>>>>>
>>>>>> What confuses me is that the storage I/O seems not to relate to the
>>>>>> number of storage nodes, but Gluster documents said it should be linear
>>>>>> scaling. For example, when "write-behind" is on, and when Infiniband "jumbo
>>>>>> frame" (connected mode) is on, I can get ~800 MB/sec reported by "dd", no
>>>>>> matter I have 2 brick servers or 8 brick servers -- for 2 server case, each
>>>>>> server can have ~400 MB/sec; for 4 server case, each server can have
>>>>>> ~200MB/sec. That said, each server I/O does aggregate to the final storage
>>>>>> I/O (800 MB/sec), but this is not "linear scale-out".
>>>>>>
>>>>>> Can somebody help me to understand why this is the case? I certainly
>>>>>> can have some misunderstanding/misconfiguration here. Please correct me if
>>>>>> I do, thanks!
>>>>>>
>>>>>> Best,
>>>>>> Qing
>>>>>> ________
>>>>>>
>>>>>>
>>>>>>
>>>>>> Community Meeting Calendar:
>>>>>>
>>>>>> Schedule -
>>>>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>>>>> Bridge: https://bluejeans.com/441850968
>>>>>>
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>
>>>>> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200724/a66652e0/attachment.html>

From nico at furyweb.fr  Sat Jul 25 04:24:14 2020
From: nico at furyweb.fr (nico at furyweb.fr)
Date: Sat, 25 Jul 2020 06:24:14 +0200 (CEST)
Subject: [Gluster-users] fuse client 7.6 crashing regularly
In-Reply-To: <CALtvDK5GfRbANG_+J8teUo4=dqg19Jg2X5mZsSMNirZW+AFfRw@mail.gmail.com>
References: <1966362365.24562.1595573393679.JavaMail.zimbra@furyweb.fr>
	<CALtvDK5GfRbANG_+J8teUo4=dqg19Jg2X5mZsSMNirZW+AFfRw@mail.gmail.com>
Message-ID: <1151040829.26497.1595651054271.JavaMail.zimbra@furyweb.fr>

Thank's Xavi. 

We'll try to upgrade, I'll keep you informed but it may take some time. 

KR, 
Nicolas. 


De: "Xavi Hernandez" <jahernan at redhat.com> 
?: nico at furyweb.fr 
Cc: "gluster-users" <gluster-users at gluster.org> 
Envoy?: Vendredi 24 Juillet 2020 09:51:22 
Objet: Re: [Gluster-users] fuse client 7.6 crashing regularly 

Hi, 
it seems to be crashing inside open-behind. There's a known bug in that xlator that caused a crash. It has been fixed in 7.7, recently released. Can you try to upgrade ? 

Xavi 

On Fri, Jul 24, 2020 at 8:50 AM < [ mailto:nico at furyweb.fr | nico at furyweb.fr ] > wrote: 



We're using gluster in a production environement, 3 nodes (2 data + 1 arbiter). 
One of our VM gluster fuse client is regularly crashing on a particular volume, we recently upgraded all nodes and client to 7.6 but client is still crashing. 

All cluster nodes & client are Debian stretch (9.12), gluster was installed from our local gluster apt repository mirror and op-version is set to 70200. 

Volume contains a lot of files & directories but performance doesn't really matters, it seems to crash during this command : 
find logscli -mtime +1 -type f | tar c -T - -f - --remove-files | tar xpf - -C /drbd 

Volume was remonted this morning with DEBUG log level, waiting for next crash. 

Volume attributes are 
Option Value 
------ ----- 
cluster.lookup-unhashed on 
cluster.lookup-optimize on 
cluster.min-free-disk 10% 
cluster.min-free-inodes 5% 
cluster.rebalance-stats off 
cluster.subvols-per-directory (null) 
cluster.readdir-optimize off 
cluster.rsync-hash-regex (null) 
cluster.extra-hash-regex (null) 
cluster.dht-xattr-name trusted.glusterfs.dht 
cluster.randomize-hash-range-by-gfid off 
cluster.rebal-throttle normal 
cluster.lock-migration off 
cluster.force-migration off 
cluster.local-volume-name (null) 
cluster.weighted-rebalance on 
cluster.switch-pattern (null) 
cluster.entry-change-log on 
cluster.read-subvolume (null) 
cluster.read-subvolume-index -1 
cluster.read-hash-mode 1 
cluster.background-self-heal-count 8 
cluster.metadata-self-heal off 
cluster.data-self-heal off 
cluster.entry-self-heal off 
cluster.self-heal-daemon enable 
cluster.heal-timeout 60 
cluster.self-heal-window-size 1 
cluster.data-change-log on 
cluster.metadata-change-log on 
cluster.data-self-heal-algorithm full 
cluster.eager-lock on 
disperse.eager-lock on 
disperse.other-eager-lock on 
disperse.eager-lock-timeout 1 
disperse.other-eager-lock-timeout 1 
cluster.quorum-type fixed 
cluster.quorum-count 1 
cluster.choose-local true 
cluster.self-heal-readdir-size 1KB 
cluster.post-op-delay-secs 1 
cluster.ensure-durability on 
cluster.consistent-metadata no 
cluster.heal-wait-queue-length 128 
cluster.favorite-child-policy none 
cluster.full-lock yes 
cluster.optimistic-change-log on 
diagnostics.latency-measurement off 
diagnostics.dump-fd-stats off 
diagnostics.count-fop-hits off 
diagnostics.brick-log-level INFO 
diagnostics.client-log-level ERROR 
diagnostics.brick-sys-log-level CRITICAL 
diagnostics.client-sys-log-level CRITICAL 
diagnostics.brick-logger (null) 
diagnostics.client-logger (null) 
diagnostics.brick-log-format (null) 
diagnostics.client-log-format (null) 
diagnostics.brick-log-buf-size 5 
diagnostics.client-log-buf-size 5 
diagnostics.brick-log-flush-timeout 120 
diagnostics.client-log-flush-timeout 120 
diagnostics.stats-dump-interval 0 
diagnostics.fop-sample-interval 0 
diagnostics.stats-dump-format json 
diagnostics.fop-sample-buf-size 65535 
diagnostics.stats-dnscache-ttl-sec 86400 
performance.cache-max-file-size 0 
performance.cache-min-file-size 0 
performance.cache-refresh-timeout 1 
performance.cache-priority 
performance.cache-size 32MB 
performance.io-thread-count 16 
performance.high-prio-threads 16 
performance.normal-prio-threads 16 
performance.low-prio-threads 16 
performance.least-prio-threads 1 
performance.enable-least-priority on 
performance.iot-watchdog-secs (null) 
performance.iot-cleanup-disconnected-reqsoff 
performance.iot-pass-through false 
performance.io-cache-pass-through false 
performance.cache-size 128MB 
performance.qr-cache-timeout 1 
performance.cache-invalidation false 
performance.ctime-invalidation false 
performance.flush-behind on 
performance.nfs.flush-behind on 
performance.write-behind-window-size 1MB 
performance.resync-failed-syncs-after-fsyncoff 
performance.nfs.write-behind-window-size1MB 
performance.strict-o-direct off 
performance.nfs.strict-o-direct off 
performance.strict-write-ordering off 
performance.nfs.strict-write-ordering off 
performance.write-behind-trickling-writeson 
performance.aggregate-size 128KB 
performance.nfs.write-behind-trickling-writeson 
performance.lazy-open yes 
performance.read-after-open yes 
performance.open-behind-pass-through false 
performance.read-ahead-page-count 4 
performance.read-ahead-pass-through false 
performance.readdir-ahead-pass-through false 
performance.md-cache-pass-through false 
performance.md-cache-timeout 1 
performance.cache-swift-metadata true 
performance.cache-samba-metadata false 
performance.cache-capability-xattrs true 
performance.cache-ima-xattrs true 
performance.md-cache-statfs off 
performance.xattr-cache-list 
performance.nl-cache-pass-through false 
network.frame-timeout 1800 
network.ping-timeout 5 
network.tcp-window-size (null) 
client.ssl on 
network.remote-dio disable 
client.event-threads 2 
client.tcp-user-timeout 0 
client.keepalive-time 20 
client.keepalive-interval 2 
client.keepalive-count 9 
network.tcp-window-size (null) 
network.inode-lru-limit 16384 
auth.allow * 
auth.reject (null) 
transport.keepalive 1 
server.allow-insecure on 
server.root-squash off 
server.all-squash off 
server.anonuid 65534 
server.anongid 65534 
server.statedump-path /var/run/gluster 
server.outstanding-rpc-limit 64 
server.ssl on 
auth.ssl-allow * 
server.manage-gids off 
server.dynamic-auth on 
client.send-gids on 
server.gid-timeout 300 
server.own-thread (null) 
server.event-threads 2 
server.tcp-user-timeout 42 
server.keepalive-time 20 
server.keepalive-interval 2 
server.keepalive-count 9 
transport.listen-backlog 1024 
ssl.cipher-list HIGH:!SSLv2 
transport.address-family inet 
performance.write-behind on 
performance.read-ahead on 
performance.readdir-ahead on 
performance.io-cache on 
performance.open-behind on 
performance.quick-read on 
performance.nl-cache off 
performance.stat-prefetch on 
performance.client-io-threads off 
performance.nfs.write-behind on 
performance.nfs.read-ahead off 
performance.nfs.io-cache off 
performance.nfs.quick-read off 
performance.nfs.stat-prefetch off 
performance.nfs.io-threads off 
performance.force-readdirp true 
performance.cache-invalidation false 
performance.global-cache-invalidation true 
features.uss off 
features.snapshot-directory .snaps 
features.show-snapshot-directory off 
features.tag-namespaces off 
network.compression off 
network.compression.window-size -15 
network.compression.mem-level 8 
network.compression.min-size 0 
network.compression.compression-level -1 
network.compression.debug false 
features.default-soft-limit 80% 
features.soft-timeout 60 
features.hard-timeout 5 
features.alert-time 86400 
features.quota-deem-statfs off 
geo-replication.indexing off 
geo-replication.indexing off 
geo-replication.ignore-pid-check off 
geo-replication.ignore-pid-check off 
features.quota off 
features.inode-quota off 
features.bitrot disable 
debug.trace off 
debug.log-history no 
debug.log-file no 
debug.exclude-ops (null) 
debug.include-ops (null) 
debug.error-gen off 
debug.error-failure (null) 
debug.error-number (null) 
debug.random-failure off 
debug.error-fops (null) 
nfs.disable on 
features.read-only off 
features.worm off 
features.worm-file-level off 
features.worm-files-deletable on 
features.default-retention-period 120 
features.retention-mode relax 
features.auto-commit-period 180 
storage.linux-aio off 
storage.batch-fsync-mode reverse-fsync 
storage.batch-fsync-delay-usec 0 
storage.owner-uid -1 
storage.owner-gid -1 
storage.node-uuid-pathinfo off 
storage.health-check-interval 30 
storage.build-pgfid off 
storage.gfid2path on 
storage.gfid2path-separator : 
storage.reserve 1 
storage.reserve-size 0 
storage.health-check-timeout 10 
storage.fips-mode-rchecksum off 
storage.force-create-mode 0000 
storage.force-directory-mode 0000 
storage.create-mask 0777 
storage.create-directory-mask 0777 
storage.max-hardlinks 100 
features.ctime off 
config.gfproxyd off 
cluster.server-quorum-type off 
cluster.server-quorum-ratio 51 
changelog.changelog off 
changelog.changelog-dir {{ brick.path }}/.glusterfs/changelogs 
changelog.encoding ascii 
changelog.rollover-time 15 
changelog.fsync-interval 5 
changelog.changelog-barrier-timeout 120 
changelog.capture-del-path off 
features.barrier disable 
features.barrier-timeout 120 
features.trash off 
features.trash-dir .trashcan 
features.trash-eliminate-path (null) 
features.trash-max-filesize 5MB 
features.trash-internal-op off 
cluster.enable-shared-storage disable 
locks.trace off 
locks.mandatory-locking off 
cluster.disperse-self-heal-daemon enable 
cluster.quorum-reads false 
client.bind-insecure (null) 
features.timeout 45 
features.failover-hosts (null) 
features.shard off 
features.shard-block-size 64MB 
features.shard-lru-limit 16384 
features.shard-deletion-rate 100 
features.scrub-throttle lazy 
features.scrub-freq biweekly 
features.scrub false 
features.expiry-time 120 
features.cache-invalidation off 
features.cache-invalidation-timeout 60 
features.leases off 
features.lease-lock-recall-timeout 60 
disperse.background-heals 8 
disperse.heal-wait-qlength 128 
cluster.heal-timeout 60 
dht.force-readdirp on 
disperse.read-policy gfid-hash 
cluster.shd-max-threads 1 
cluster.shd-wait-qlength 1024 
cluster.locking-scheme full 
cluster.granular-entry-heal no 
features.locks-revocation-secs 0 
features.locks-revocation-clear-all false 
features.locks-revocation-max-blocked 0 
features.locks-monkey-unlocking false 
features.locks-notify-contention no 
features.locks-notify-contention-delay 5 
disperse.shd-max-threads 1 
disperse.shd-wait-qlength 1024 
disperse.cpu-extensions auto 
disperse.self-heal-window-size 1 
cluster.use-compound-fops off 
performance.parallel-readdir off 
performance.rda-request-size 131072 
performance.rda-low-wmark 4096 
performance.rda-high-wmark 128KB 
performance.rda-cache-limit 10MB 
performance.nl-cache-positive-entry false 
performance.nl-cache-limit 10MB 
performance.nl-cache-timeout 60 
cluster.brick-multiplex disable 
glusterd.vol_count_per_thread 100 
cluster.max-bricks-per-process 250 
disperse.optimistic-change-log on 
disperse.stripe-cache 4 
cluster.halo-enabled False 
cluster.halo-shd-max-latency 99999 
cluster.halo-nfsd-max-latency 5 
cluster.halo-max-latency 5 
cluster.halo-max-replicas 99999 
cluster.halo-min-replicas 2 
features.selinux on 
cluster.daemon-log-level INFO 
debug.delay-gen off 
delay-gen.delay-percentage 10% 
delay-gen.delay-duration 100000 
delay-gen.enable 
disperse.parallel-writes on 
features.sdfs off 
features.cloudsync off 
features.ctime off 
ctime.noatime on 
features.cloudsync-storetype (null) 
features.enforce-mandatory-lock off 
config.global-threading off 
config.client-threads 16 
config.brick-threads 16 
features.cloudsync-remote-read off 
features.cloudsync-store-id (null) 
features.cloudsync-product-id (null) 


Crash log found in /var/log/glusterfs/partage-logscli.log 
2020-07-23 02:34:36 
configuration details: 
argp 1 
backtrace 1 
dlfcn 1 
libpthread 1 
llistxattr 1 
setfsid 1 
spinlock 1 
epoll.h 1 
xattr.h 1 
st_atim.tv_nsec 1 
package-string: glusterfs 7.6 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x25e50)[0x7fbe02138e50] 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x2f7)[0x7fbe021434b7] 
/lib/x86_64-linux-gnu/libc.so.6(+0x33060)[0x7fbe00b87060] 
/lib/x86_64-linux-gnu/libpthread.so.0(pthread_mutex_lock+0x0)[0x7fbe01396b40] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x32f5)[0x7fbdfa89b2f5] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3c52)[0x7fbdfa89bc52] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3dac)[0x7fbdfa89bdac] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/open-behind.so(+0x3f73)[0x7fbdfa89bf73] 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_unlink+0xbc)[0x7fbe021c401c] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/performance/md-cache.so(+0x4495)[0x7fbdfa46c495] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/debug/io-stats.so(+0x5f44)[0x7fbdfa23af44] 
/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(default_unlink+0xbc)[0x7fbe021c401c] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x1154f)[0x7fbdff7dc54f] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x7775)[0x7fbdff7d2775] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x74c8)[0x7fbdff7d24c8] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x77be)[0x7fbdff7d27be] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x6ac3)[0x7fbdff7d1ac3] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x7188)[0x7fbdff7d2188] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x74e8)[0x7fbdff7d24e8] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x779e)[0x7fbdff7d279e] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x77e0)[0x7fbdff7d27e0] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x83f9)[0x7fbdff7d33f9] 
/usr/lib/x86_64-linux-gnu/glusterfs/7.6/xlator/mount/fuse.so(+0x21d3c)[0x7fbdff7ecd3c] 
/lib/x86_64-linux-gnu/libpthread.so.0(+0x74a4)[0x7fbe013944a4] 
/lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fbe00c3cd0f] 
________ 



Community Meeting Calendar: 

Schedule - 
Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC 
Bridge: [ https://bluejeans.com/441850968 | https://bluejeans.com/441850968 ] 

Gluster-users mailing list 
[ mailto:Gluster-users at gluster.org | Gluster-users at gluster.org ] 
[ https://lists.gluster.org/mailman/listinfo/gluster-users | https://lists.gluster.org/mailman/listinfo/gluster-users ] 




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200725/88669837/attachment.html>

From amar at kadalu.io  Mon Jul 27 10:35:51 2020
From: amar at kadalu.io (amar at kadalu.io)
Date: Mon, 27 Jul 2020 10:35:51 +0000
Subject: [Gluster-users] Invitation: Gluter Community Meeting @ Tue Jul 28,
 2020 2:30pm - 3:30pm (IST) (gluster-users@gluster.org)
Message-ID: <00000000000001191005ab69e3fd@google.com>

You have been invited to the following event.

Title: Gluter Community Meeting
Hi,Please join us for the Gluster community meeting. The meeting details  
are  
below.Bridge:&nbsp;https://bluejeans.com/441850968Agenda:&nbsp;&nbsp;https://hackmd.io/DpzO_gRXTcSi5w_zXIB7DAPrevious  
Meeting notes:https://github.com/gluster/community/meetings
When: Tue Jul 28, 2020 2:30pm ? 3:30pm India Standard Time - Kolkata
Where: https://bluejeans.com/441850968

Joining info: Join with Google Meet
https://meet.google.com/yrj-wynz-eyz

Join by phone
+1 513-486-2458 (PIN: 274876546)

More phone numbers: https://tel.meet/yrj-wynz-eyz?pin=2420181636757&hs=0

Calendar: gluster-users at gluster.org
Who:
     * amar at kadalu.io - organizer
     * gluster-users at gluster.org
     * gluster-devel at gluster.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=N2lwc3Fwcm9qbTg5NTlmZTU0YmFzcHQ2ZGIgZ2x1c3Rlci11c2Vyc0BnbHVzdGVyLm9yZw&tok=MTQjYW1hckBrYWRhbHUuaW8xZTNmNjk3NDkzMTAwOWRhN2I0ZmE4NmExZWQwODVkODg1MmM0ZmRm&ctz=Asia%2FKolkata&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account  
gluster-users at gluster.org because you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200727/b63e9cbb/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 2225 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200727/b63e9cbb/attachment.ics>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 2270 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200727/b63e9cbb/attachment.bin>

From amar at kadalu.io  Mon Jul 27 10:36:09 2020
From: amar at kadalu.io (amar at kadalu.io)
Date: Mon, 27 Jul 2020 10:36:09 +0000
Subject: [Gluster-users] Updated invitation: Gluter Community Meeting @ Tue
 Jul 28, 2020 2:30pm - 3:30pm (IST) (gluster-users@gluster.org)
Message-ID: <00000000000010f76405ab69e4d4@google.com>

This event has been changed.

Title: Gluter Community Meeting
Hi,Please join us for the Gluster community meeting. The meeting details  
are  
below.Bridge:&nbsp;https://bluejeans.com/441850968Agenda:&nbsp;&nbsp;https://hackmd.io/DpzO_gRXTcSi5w_zXIB7DAPrevious  
Meeting notes:https://github.com/gluster/community/meetings
When: Tue Jul 28, 2020 2:30pm ? 3:30pm India Standard Time - Kolkata
Where: https://bluejeans.com/441850968
Calendar: gluster-users at gluster.org
Who:
     * amar at kadalu.io - organizer
     * gluster-users at gluster.org
     * gluster-devel at gluster.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=N2lwc3Fwcm9qbTg5NTlmZTU0YmFzcHQ2ZGIgZ2x1c3Rlci11c2Vyc0BnbHVzdGVyLm9yZw&tok=MTQjYW1hckBrYWRhbHUuaW8xZTNmNjk3NDkzMTAwOWRhN2I0ZmE4NmExZWQwODVkODg1MmM0ZmRm&ctz=Asia%2FKolkata&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account  
gluster-users at gluster.org because you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200727/f53eb599/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 2031 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200727/f53eb599/attachment.ics>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 2073 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200727/f53eb599/attachment.bin>

From gilberto.nunes32 at gmail.com  Tue Jul 28 19:43:39 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Tue, 28 Jul 2020 16:43:39 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
Message-ID: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>

Hi there....

'till now, I am using glusterfs over XFS and so far so good.
Using LVM too....
Unfortunately, there is no way with XFS to merge two or more HDD, in order
to use more than one HDD, like RAID1 or RAID5.
My primary goal is to use two server with GlusterFS on top of multiples
HDDs for qemu images.
I have think about BTRFS or mdadm.
Anybody has some experience on this?

Thanks a lot

---
Gilberto Nunes Ferreira
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200728/78de5773/attachment.html>

From gilberto.nunes32 at gmail.com  Tue Jul 28 20:16:47 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Tue, 28 Jul 2020 17:16:47 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <331cc752-1d70-bc0f-db79-12c15b041f39@netvel.net>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<331cc752-1d70-bc0f-db79-12c15b041f39@netvel.net>
Message-ID: <CAOKSTBu1cV1ff_4jvUkM=J_kWYjkuz0iObiSmV1S3hv3ac4EJw@mail.gmail.com>

Good to know that...
Thanks
---
Gilberto Nunes Ferreira




Em ter., 28 de jul. de 2020 ?s 17:08, Alvin Starr <alvin at netvel.net>
escreveu:

> Having just been burnt by BTRFS I would stick with XFS and LVM/others.
>
> LVM will do disk replication or raid1. I do not believe that raid3,4,5,6..
> is supported.
> mdadm does support all the various raid modes and I have used it quite
> reliably for years.
> You may want to look at the raid456 write-journal but that will require an
> SSD or NVME deivce to be used effectively.
>
>
> On 7/28/20 3:43 PM, Gilberto Nunes wrote:
>
> Hi there....
>
> 'till now, I am using glusterfs over XFS and so far so good.
> Using LVM too....
> Unfortunately, there is no way with XFS to merge two or more HDD, in order
> to use more than one HDD, like RAID1 or RAID5.
> My primary goal is to use two server with GlusterFS on top of multiples
> HDDs for qemu images.
> I have think about BTRFS or mdadm.
> Anybody has some experience on this?
>
> Thanks a lot
>
> ---
> Gilberto Nunes Ferreira
>
>
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Alvin Starr                   ||   land:  (647)478-6285
> Netvel Inc.                   ||   Cell:  (416)806-0133alvin at netvel.net              ||
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200728/6cf6f91a/attachment.html>

From gilberto.nunes32 at gmail.com  Tue Jul 28 20:27:50 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Tue, 28 Jul 2020 17:27:50 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBu1cV1ff_4jvUkM=J_kWYjkuz0iObiSmV1S3hv3ac4EJw@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<331cc752-1d70-bc0f-db79-12c15b041f39@netvel.net>
	<CAOKSTBu1cV1ff_4jvUkM=J_kWYjkuz0iObiSmV1S3hv3ac4EJw@mail.gmail.com>
Message-ID: <CAOKSTBtaR1uv8Pa2C8fcc94TVDOwp=-Pis0WYAQFqmQ83GAWOQ@mail.gmail.com>

But still, have some doubt how LVM will handle in case of disk failure!
Or even mdadm.
Both need intervention when one or more disks die!
What I need is something like ZFS but that uses less resources...

Thanks any way



---
Gilberto Nunes Ferreira




Em ter., 28 de jul. de 2020 ?s 17:16, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> Good to know that...
> Thanks
> ---
> Gilberto Nunes Ferreira
>
>
>
>
> Em ter., 28 de jul. de 2020 ?s 17:08, Alvin Starr <alvin at netvel.net>
> escreveu:
>
>> Having just been burnt by BTRFS I would stick with XFS and LVM/others.
>>
>> LVM will do disk replication or raid1. I do not believe that
>> raid3,4,5,6.. is supported.
>> mdadm does support all the various raid modes and I have used it quite
>> reliably for years.
>> You may want to look at the raid456 write-journal but that will require
>> an SSD or NVME deivce to be used effectively.
>>
>>
>> On 7/28/20 3:43 PM, Gilberto Nunes wrote:
>>
>> Hi there....
>>
>> 'till now, I am using glusterfs over XFS and so far so good.
>> Using LVM too....
>> Unfortunately, there is no way with XFS to merge two or more HDD, in
>> order to use more than one HDD, like RAID1 or RAID5.
>> My primary goal is to use two server with GlusterFS on top of multiples
>> HDDs for qemu images.
>> I have think about BTRFS or mdadm.
>> Anybody has some experience on this?
>>
>> Thanks a lot
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>>
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> --
>> Alvin Starr                   ||   land:  (647)478-6285
>> Netvel Inc.                   ||   Cell:  (416)806-0133alvin at netvel.net              ||
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200728/aca4bc2f/attachment.html>

From budic at onholyground.com  Tue Jul 28 21:10:44 2020
From: budic at onholyground.com (Darrell Budic)
Date: Tue, 28 Jul 2020 16:10:44 -0500
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBtaR1uv8Pa2C8fcc94TVDOwp=-Pis0WYAQFqmQ83GAWOQ@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<331cc752-1d70-bc0f-db79-12c15b041f39@netvel.net>
	<CAOKSTBu1cV1ff_4jvUkM=J_kWYjkuz0iObiSmV1S3hv3ac4EJw@mail.gmail.com>
	<CAOKSTBtaR1uv8Pa2C8fcc94TVDOwp=-Pis0WYAQFqmQ83GAWOQ@mail.gmail.com>
Message-ID: <EEEE2FD1-22F6-4393-B03D-195B6C28F58B@onholyground.com>

ZFS isn?t that resource intensive, although it does like RAM.

But why not just add additional bricks? Gluster is kind of built to use disks as bricks, and rely on gluster replication to provide redundancy in the data. Either replication or distribute-replication can provide protection against disk failures.

> On Jul 28, 2020, at 3:27 PM, Gilberto Nunes <gilberto.nunes32 at gmail.com> wrote:
> 
> But still, have some doubt how LVM will handle in case of disk failure!
> Or even mdadm.
> Both need intervention when one or more disks die!
> What I need is something like ZFS but that uses less resources...
> 
> Thanks any way
> 
> 
> 
> ---
> Gilberto Nunes Ferreira
> 
> 
> 
> 
> Em ter., 28 de jul. de 2020 ?s 17:16, Gilberto Nunes <gilberto.nunes32 at gmail.com <mailto:gilberto.nunes32 at gmail.com>> escreveu:
> Good to know that...
> Thanks
> ---
> Gilberto Nunes Ferreira
> 
> 
> 
> 
> Em ter., 28 de jul. de 2020 ?s 17:08, Alvin Starr <alvin at netvel.net <mailto:alvin at netvel.net>> escreveu:
> Having just been burnt by BTRFS I would stick with XFS and LVM/others.
> 
> LVM will do disk replication or raid1. I do not believe that raid3,4,5,6.. is supported.
> mdadm does support all the various raid modes and I have used it quite reliably for years.
> You may want to look at the raid456 write-journal but that will require an SSD or NVME deivce to be used effectively.
> 
> 
> On 7/28/20 3:43 PM, Gilberto Nunes wrote:
>> Hi there....
>> 
>> 'till now, I am using glusterfs over XFS and so far so good.
>> Using LVM too....
>> Unfortunately, there is no way with XFS to merge two or more HDD, in order to use more than one HDD, like RAID1 or RAID5.
>> My primary goal is to use two server with GlusterFS on top of multiples HDDs for qemu images.
>> I have think about BTRFS or mdadm.
>> Anybody has some experience on this?
>> 
>> Thanks a lot
>> 
>> ---
>> Gilberto Nunes Ferreira
>> 
>> 
>> 
>> 
>> 
>> ________
>> 
>> 
>> 
>> Community Meeting Calendar:
>> 
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968 <https://bluejeans.com/441850968>
>> 
>> Gluster-users mailing list
>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>> https://lists.gluster.org/mailman/listinfo/gluster-users <https://lists.gluster.org/mailman/listinfo/gluster-users>
> 
> -- 
> Alvin Starr                   ||   land:  (647)478-6285
> Netvel Inc.                   ||   Cell:  (416)806-0133
> alvin at netvel.net <mailto:alvin at netvel.net>              ||
> 
> ________
> 
> 
> 
> Community Meeting Calendar:
> 
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200728/30457ed5/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul 30 05:02:28 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 08:02:28 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <EEEE2FD1-22F6-4393-B03D-195B6C28F58B@onholyground.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<331cc752-1d70-bc0f-db79-12c15b041f39@netvel.net>
	<CAOKSTBu1cV1ff_4jvUkM=J_kWYjkuz0iObiSmV1S3hv3ac4EJw@mail.gmail.com>
	<CAOKSTBtaR1uv8Pa2C8fcc94TVDOwp=-Pis0WYAQFqmQ83GAWOQ@mail.gmail.com>
	<EEEE2FD1-22F6-4393-B03D-195B6C28F58B@onholyground.com>
Message-ID: <2B58B6B6-AD48-4427-9BA6-D71FA9573008@yahoo.com>

Keep in mind that if you use each disk as a separate brick, use  replica 3 volumes.

Also,  mdadm  allows you to specify a hot spare,  so in case of disk failure it will automatically kick in and replace the failed drive.

I'm not sure if LVM has that option,  but as it relies  on the same mechanism mdadm is  using -> it should be possible.

Best Regards,
Strahil Nikolov

?? 29 ??? 2020 ?. 0:10:44 GMT+03:00, Darrell Budic <budic at onholyground.com> ??????:
>ZFS isn?t that resource intensive, although it does like RAM.
>
>But why not just add additional bricks? Gluster is kind of built to use
>disks as bricks, and rely on gluster replication to provide redundancy
>in the data. Either replication or distribute-replication can provide
>protection against disk failures.
>
>> On Jul 28, 2020, at 3:27 PM, Gilberto Nunes
><gilberto.nunes32 at gmail.com> wrote:
>> 
>> But still, have some doubt how LVM will handle in case of disk
>failure!
>> Or even mdadm.
>> Both need intervention when one or more disks die!
>> What I need is something like ZFS but that uses less resources...
>> 
>> Thanks any way
>> 
>> 
>> 
>> ---
>> Gilberto Nunes Ferreira
>> 
>> 
>> 
>> 
>> Em ter., 28 de jul. de 2020 ?s 17:16, Gilberto Nunes
><gilberto.nunes32 at gmail.com <mailto:gilberto.nunes32 at gmail.com>>
>escreveu:
>> Good to know that...
>> Thanks
>> ---
>> Gilberto Nunes Ferreira
>> 
>> 
>> 
>> 
>> Em ter., 28 de jul. de 2020 ?s 17:08, Alvin Starr <alvin at netvel.net
><mailto:alvin at netvel.net>> escreveu:
>> Having just been burnt by BTRFS I would stick with XFS and
>LVM/others.
>> 
>> LVM will do disk replication or raid1. I do not believe that
>raid3,4,5,6.. is supported.
>> mdadm does support all the various raid modes and I have used it
>quite reliably for years.
>> You may want to look at the raid456 write-journal but that will
>require an SSD or NVME deivce to be used effectively.
>> 
>> 
>> On 7/28/20 3:43 PM, Gilberto Nunes wrote:
>>> Hi there....
>>> 
>>> 'till now, I am using glusterfs over XFS and so far so good.
>>> Using LVM too....
>>> Unfortunately, there is no way with XFS to merge two or more HDD, in
>order to use more than one HDD, like RAID1 or RAID5.
>>> My primary goal is to use two server with GlusterFS on top of
>multiples HDDs for qemu images.
>>> I have think about BTRFS or mdadm.
>>> Anybody has some experience on this?
>>> 
>>> Thanks a lot
>>> 
>>> ---
>>> Gilberto Nunes Ferreira
>>> 
>>> 
>>> 
>>> 
>>> 
>>> ________
>>> 
>>> 
>>> 
>>> Community Meeting Calendar:
>>> 
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
><https://bluejeans.com/441850968>
>>> 
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
><https://lists.gluster.org/mailman/listinfo/gluster-users>
>> 
>> -- 
>> Alvin Starr                   ||   land:  (647)478-6285
>> Netvel Inc.                   ||   Cell:  (416)806-0133
>> alvin at netvel.net <mailto:alvin at netvel.net>              ||
>> 
>> ________
>> 
>> 
>> 
>> Community Meeting Calendar:
>> 
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>> 
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users

From hunter86_bg at yahoo.com  Thu Jul 30 04:58:59 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 07:58:59 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
Message-ID: <A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>

LVM allows creating/converting striped/mirrored LVs without any dowtime and it's using the md module.

Best Regards,
Strahil Nikolov

?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>Hi there....
>
>'till now, I am using glusterfs over XFS and so far so good.
>Using LVM too....
>Unfortunately, there is no way with XFS to merge two or more HDD, in
>order
>to use more than one HDD, like RAID1 or RAID5.
>My primary goal is to use two server with GlusterFS on top of multiples
>HDDs for qemu images.
>I have think about BTRFS or mdadm.
>Anybody has some experience on this?
>
>Thanks a lot
>
>---
>Gilberto Nunes Ferreira

From gilberto.nunes32 at gmail.com  Thu Jul 30 12:39:18 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 09:39:18 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
Message-ID: <CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>

Doing some research and the chvg command, which is responsable for create
hotspare disks in LVM is available only in AIX!
I have a Debian Buster box and there is no such command chvg. Correct me if
I am wrong....
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> LVM allows creating/converting striped/mirrored LVs without any dowtime
> and it's using the md module.
>
> Best Regards,
> Strahil Nikolov
>
> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >Hi there....
> >
> >'till now, I am using glusterfs over XFS and so far so good.
> >Using LVM too....
> >Unfortunately, there is no way with XFS to merge two or more HDD, in
> >order
> >to use more than one HDD, like RAID1 or RAID5.
> >My primary goal is to use two server with GlusterFS on top of multiples
> >HDDs for qemu images.
> >I have think about BTRFS or mdadm.
> >Anybody has some experience on this?
> >
> >Thanks a lot
> >
> >---
> >Gilberto Nunes Ferreira
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/75f6a715/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul 30 12:53:04 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 15:53:04 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
Message-ID: <4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>

I guess  there  is no  automatic  hot-spare  replacement in LVM, but  mdadm has  that  functionality.

Best  Regards,
Strahil Nikolov


?? 30 ??? 2020 ?. 15:39:18 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>Doing some research and the chvg command, which is responsable for
>create
>hotspare disks in LVM is available only in AIX!
>I have a Debian Buster box and there is no such command chvg. Correct
>me if
>I am wrong....
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov
><hunter86_bg at yahoo.com>
>escreveu:
>
>> LVM allows creating/converting striped/mirrored LVs without any
>dowtime
>> and it's using the md module.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >Hi there....
>> >
>> >'till now, I am using glusterfs over XFS and so far so good.
>> >Using LVM too....
>> >Unfortunately, there is no way with XFS to merge two or more HDD, in
>> >order
>> >to use more than one HDD, like RAID1 or RAID5.
>> >My primary goal is to use two server with GlusterFS on top of
>multiples
>> >HDDs for qemu images.
>> >I have think about BTRFS or mdadm.
>> >Anybody has some experience on this?
>> >
>> >Thanks a lot
>> >
>> >---
>> >Gilberto Nunes Ferreira
>>

From gilberto.nunes32 at gmail.com  Thu Jul 30 12:59:57 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 09:59:57 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>
Message-ID: <CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>

Yes! But still with mdadm if you lose 1 disk and reboot the server, the
system crashes.
But with ZFS there's no crash.

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 09:55, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> I guess  there  is no  automatic  hot-spare  replacement in LVM, but
> mdadm has  that  functionality.
>
> Best  Regards,
> Strahil Nikolov
>
>
> ?? 30 ??? 2020 ?. 15:39:18 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >Doing some research and the chvg command, which is responsable for
> >create
> >hotspare disks in LVM is available only in AIX!
> >I have a Debian Buster box and there is no such command chvg. Correct
> >me if
> >I am wrong....
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov
> ><hunter86_bg at yahoo.com>
> >escreveu:
> >
> >> LVM allows creating/converting striped/mirrored LVs without any
> >dowtime
> >> and it's using the md module.
> >>
> >> Best Regards,
> >> Strahil Nikolov
> >>
> >> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
> >> gilberto.nunes32 at gmail.com> ??????:
> >> >Hi there....
> >> >
> >> >'till now, I am using glusterfs over XFS and so far so good.
> >> >Using LVM too....
> >> >Unfortunately, there is no way with XFS to merge two or more HDD, in
> >> >order
> >> >to use more than one HDD, like RAID1 or RAID5.
> >> >My primary goal is to use two server with GlusterFS on top of
> >multiples
> >> >HDDs for qemu images.
> >> >I have think about BTRFS or mdadm.
> >> >Anybody has some experience on this?
> >> >
> >> >Thanks a lot
> >> >
> >> >---
> >> >Gilberto Nunes Ferreira
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/8732dcdb/attachment.html>

From gilberto.nunes32 at gmail.com  Thu Jul 30 13:08:13 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 10:08:13 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>
	<CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>
Message-ID: <CAOKSTBsWsDu-C564eOUjNzA+in-Nj5A8u4M2x7ucMZO_LUcZbw@mail.gmail.com>

I meant, if you power off the server, pull off 1 disk, and then power on we
get system errors....

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 09:59, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> Yes! But still with mdadm if you lose 1 disk and reboot the server, the
> system crashes.
> But with ZFS there's no crash.
>
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qui., 30 de jul. de 2020 ?s 09:55, Strahil Nikolov <
> hunter86_bg at yahoo.com> escreveu:
>
>> I guess  there  is no  automatic  hot-spare  replacement in LVM, but
>> mdadm has  that  functionality.
>>
>> Best  Regards,
>> Strahil Nikolov
>>
>>
>> ?? 30 ??? 2020 ?. 15:39:18 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >Doing some research and the chvg command, which is responsable for
>> >create
>> >hotspare disks in LVM is available only in AIX!
>> >I have a Debian Buster box and there is no such command chvg. Correct
>> >me if
>> >I am wrong....
>> >---
>> >Gilberto Nunes Ferreira
>> >
>> >(47) 3025-5907
>> >(47) 99676-7530 - Whatsapp / Telegram
>> >
>> >Skype: gilberto.nunes36
>> >
>> >
>> >
>> >
>> >
>> >Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov
>> ><hunter86_bg at yahoo.com>
>> >escreveu:
>> >
>> >> LVM allows creating/converting striped/mirrored LVs without any
>> >dowtime
>> >> and it's using the md module.
>> >>
>> >> Best Regards,
>> >> Strahil Nikolov
>> >>
>> >> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>> >> gilberto.nunes32 at gmail.com> ??????:
>> >> >Hi there....
>> >> >
>> >> >'till now, I am using glusterfs over XFS and so far so good.
>> >> >Using LVM too....
>> >> >Unfortunately, there is no way with XFS to merge two or more HDD, in
>> >> >order
>> >> >to use more than one HDD, like RAID1 or RAID5.
>> >> >My primary goal is to use two server with GlusterFS on top of
>> >multiples
>> >> >HDDs for qemu images.
>> >> >I have think about BTRFS or mdadm.
>> >> >Anybody has some experience on this?
>> >> >
>> >> >Thanks a lot
>> >> >
>> >> >---
>> >> >Gilberto Nunes Ferreira
>> >>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/578d1281/attachment.html>

From alvin at netvel.net  Thu Jul 30 13:11:46 2020
From: alvin at netvel.net (Alvin Starr)
Date: Thu, 30 Jul 2020 09:11:46 -0400
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
Message-ID: <f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>

LVM supports mirroring or raid1.
So to create a raid1 you would do something like "lvcreate -n raidvol? 
-L 1T --mirrors 1? /dev/bigvg"
You can add a mirror to an existing logical volume using lvconvert with 
something like "lvconvert --mirrors +1 bigvg/unmirroredlv"

But other than simple mirroring you will need mdadm.



On 7/30/20 8:39 AM, Gilberto Nunes wrote:
> Doing some research and the chvg command, which is responsable?for 
> create hotspare disks in LVM is available only in AIX!
> I have a Debian Buster box and there is no such command chvg. Correct 
> me if I am wrong....
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov 
> <hunter86_bg at yahoo.com <mailto:hunter86_bg at yahoo.com>> escreveu:
>
>     LVM allows creating/converting striped/mirrored LVs without any
>     dowtime and it's using the md module.
>
>     Best Regards,
>     Strahil Nikolov
>
>     ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes
>     <gilberto.nunes32 at gmail.com <mailto:gilberto.nunes32 at gmail.com>>
>     ??????:
>     >Hi there....
>     >
>     >'till now, I am using glusterfs over XFS and so far so good.
>     >Using LVM too....
>     >Unfortunately, there is no way with XFS to merge two or more HDD, in
>     >order
>     >to use more than one HDD, like RAID1 or RAID5.
>     >My primary goal is to use two server with GlusterFS on top of
>     multiples
>     >HDDs for qemu images.
>     >I have think about BTRFS or mdadm.
>     >Anybody has some experience on this?
>     >
>     >Thanks a lot
>     >
>     >---
>     >Gilberto Nunes Ferreira
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

-- 
Alvin Starr                   ||   land:  (647)478-6285
Netvel Inc.                   ||   Cell:  (416)806-0133
alvin at netvel.net              ||

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/8139d7d3/attachment.html>

From gilberto.nunes32 at gmail.com  Thu Jul 30 13:23:19 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 10:23:19 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
Message-ID: <CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>

Well, actually I am not concerned ONLY about mirroring the data, but with
reliability on it.
Here during my lab tests I notice if I pull off a disk and then power on
the server, the LVM crashes...
On the other hand, with ZFS even in degraded state the system booted
normally.
The same happens with mdadm. That's the point.

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr <alvin at netvel.net>
escreveu:

> LVM supports mirroring or raid1.
> So to create a raid1 you would do something like "lvcreate -n raidvol  -L
> 1T --mirrors 1  /dev/bigvg"
> You can add a mirror to an existing logical volume using lvconvert  with
> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
>
> But other than simple mirroring you will need mdadm.
>
>
>
> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
>
> Doing some research and the chvg command, which is responsable for create
> hotspare disks in LVM is available only in AIX!
> I have a Debian Buster box and there is no such command chvg. Correct me
> if I am wrong....
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
> hunter86_bg at yahoo.com> escreveu:
>
>> LVM allows creating/converting striped/mirrored LVs without any dowtime
>> and it's using the md module.
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >Hi there....
>> >
>> >'till now, I am using glusterfs over XFS and so far so good.
>> >Using LVM too....
>> >Unfortunately, there is no way with XFS to merge two or more HDD, in
>> >order
>> >to use more than one HDD, like RAID1 or RAID5.
>> >My primary goal is to use two server with GlusterFS on top of multiples
>> >HDDs for qemu images.
>> >I have think about BTRFS or mdadm.
>> >Anybody has some experience on this?
>> >
>> >Thanks a lot
>> >
>> >---
>> >Gilberto Nunes Ferreira
>>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> --
> Alvin Starr                   ||   land:  (647)478-6285
> Netvel Inc.                   ||   Cell:  (416)806-0133alvin at netvel.net              ||
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/6d3feae1/attachment.html>

From gilberto.nunes32 at gmail.com  Thu Jul 30 13:25:47 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 10:25:47 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
Message-ID: <CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>

But I will give it a try in the lvm mirroring process.... Thanks
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> Well, actually I am not concerned ONLY about mirroring the data, but with
> reliability on it.
> Here during my lab tests I notice if I pull off a disk and then power on
> the server, the LVM crashes...
> On the other hand, with ZFS even in degraded state the system booted
> normally.
> The same happens with mdadm. That's the point.
>
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr <alvin at netvel.net>
> escreveu:
>
>> LVM supports mirroring or raid1.
>> So to create a raid1 you would do something like "lvcreate -n raidvol  -L
>> 1T --mirrors 1  /dev/bigvg"
>> You can add a mirror to an existing logical volume using lvconvert  with
>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
>>
>> But other than simple mirroring you will need mdadm.
>>
>>
>>
>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
>>
>> Doing some research and the chvg command, which is responsable for create
>> hotspare disks in LVM is available only in AIX!
>> I have a Debian Buster box and there is no such command chvg. Correct me
>> if I am wrong....
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
>> hunter86_bg at yahoo.com> escreveu:
>>
>>> LVM allows creating/converting striped/mirrored LVs without any dowtime
>>> and it's using the md module.
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>>> gilberto.nunes32 at gmail.com> ??????:
>>> >Hi there....
>>> >
>>> >'till now, I am using glusterfs over XFS and so far so good.
>>> >Using LVM too....
>>> >Unfortunately, there is no way with XFS to merge two or more HDD, in
>>> >order
>>> >to use more than one HDD, like RAID1 or RAID5.
>>> >My primary goal is to use two server with GlusterFS on top of multiples
>>> >HDDs for qemu images.
>>> >I have think about BTRFS or mdadm.
>>> >Anybody has some experience on this?
>>> >
>>> >Thanks a lot
>>> >
>>> >---
>>> >Gilberto Nunes Ferreira
>>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>> --
>> Alvin Starr                   ||   land:  (647)478-6285
>> Netvel Inc.                   ||   Cell:  (416)806-0133alvin at netvel.net              ||
>>
>>
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/f7c6472a/attachment.html>

From gilberto.nunes32 at gmail.com  Thu Jul 30 14:45:07 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 11:45:07 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
	<CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
Message-ID: <CAOKSTBsqW4PJSPE20ESpanBO71HBagtp-_dybFdXmZEma8zLZA@mail.gmail.com>

Well... LVM doesn't do the job that I need...
But I found this article https://www.gonscak.sk/?p=201 which makes the
way....

Thanks for all the help and comments about this.... Cheers.

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 10:25, Gilberto Nunes <
gilberto.nunes32 at gmail.com> escreveu:

> But I will give it a try in the lvm mirroring process.... Thanks
> ---
> Gilberto Nunes Ferreira
>
> (47) 3025-5907
> (47) 99676-7530 - Whatsapp / Telegram
>
> Skype: gilberto.nunes36
>
>
>
>
>
> Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> escreveu:
>
>> Well, actually I am not concerned ONLY about mirroring the data, but with
>> reliability on it.
>> Here during my lab tests I notice if I pull off a disk and then power on
>> the server, the LVM crashes...
>> On the other hand, with ZFS even in degraded state the system booted
>> normally.
>> The same happens with mdadm. That's the point.
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr <alvin at netvel.net>
>> escreveu:
>>
>>> LVM supports mirroring or raid1.
>>> So to create a raid1 you would do something like "lvcreate -n raidvol
>>> -L 1T --mirrors 1  /dev/bigvg"
>>> You can add a mirror to an existing logical volume using lvconvert  with
>>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
>>>
>>> But other than simple mirroring you will need mdadm.
>>>
>>>
>>>
>>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
>>>
>>> Doing some research and the chvg command, which is responsable for
>>> create hotspare disks in LVM is available only in AIX!
>>> I have a Debian Buster box and there is no such command chvg. Correct me
>>> if I am wrong....
>>> ---
>>> Gilberto Nunes Ferreira
>>>
>>> (47) 3025-5907
>>> (47) 99676-7530 - Whatsapp / Telegram
>>>
>>> Skype: gilberto.nunes36
>>>
>>>
>>>
>>>
>>>
>>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
>>> hunter86_bg at yahoo.com> escreveu:
>>>
>>>> LVM allows creating/converting striped/mirrored LVs without any dowtime
>>>> and it's using the md module.
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>>>> gilberto.nunes32 at gmail.com> ??????:
>>>> >Hi there....
>>>> >
>>>> >'till now, I am using glusterfs over XFS and so far so good.
>>>> >Using LVM too....
>>>> >Unfortunately, there is no way with XFS to merge two or more HDD, in
>>>> >order
>>>> >to use more than one HDD, like RAID1 or RAID5.
>>>> >My primary goal is to use two server with GlusterFS on top of multiples
>>>> >HDDs for qemu images.
>>>> >I have think about BTRFS or mdadm.
>>>> >Anybody has some experience on this?
>>>> >
>>>> >Thanks a lot
>>>> >
>>>> >---
>>>> >Gilberto Nunes Ferreira
>>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>> --
>>> Alvin Starr                   ||   land:  (647)478-6285
>>> Netvel Inc.                   ||   Cell:  (416)806-0133alvin at netvel.net              ||
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/24b744ee/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul 30 15:12:02 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 18:12:02 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>
	<CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>
Message-ID: <FC4E80D9-52E7-45E7-827C-71493341E261@yahoo.com>

If it crashes  - that's  a  bug  and worth checking. What OS do you use ?

Best Regards,
Strahil Nikolov

?? 30 ??? 2020 ?. 15:59:57 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>Yes! But still with mdadm if you lose 1 disk and reboot the server, the
>system crashes.
>But with ZFS there's no crash.
>
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qui., 30 de jul. de 2020 ?s 09:55, Strahil Nikolov
><hunter86_bg at yahoo.com>
>escreveu:
>
>> I guess  there  is no  automatic  hot-spare  replacement in LVM, but
>> mdadm has  that  functionality.
>>
>> Best  Regards,
>> Strahil Nikolov
>>
>>
>> ?? 30 ??? 2020 ?. 15:39:18 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >Doing some research and the chvg command, which is responsable for
>> >create
>> >hotspare disks in LVM is available only in AIX!
>> >I have a Debian Buster box and there is no such command chvg.
>Correct
>> >me if
>> >I am wrong....
>> >---
>> >Gilberto Nunes Ferreira
>> >
>> >(47) 3025-5907
>> >(47) 99676-7530 - Whatsapp / Telegram
>> >
>> >Skype: gilberto.nunes36
>> >
>> >
>> >
>> >
>> >
>> >Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov
>> ><hunter86_bg at yahoo.com>
>> >escreveu:
>> >
>> >> LVM allows creating/converting striped/mirrored LVs without any
>> >dowtime
>> >> and it's using the md module.
>> >>
>> >> Best Regards,
>> >> Strahil Nikolov
>> >>
>> >> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>> >> gilberto.nunes32 at gmail.com> ??????:
>> >> >Hi there....
>> >> >
>> >> >'till now, I am using glusterfs over XFS and so far so good.
>> >> >Using LVM too....
>> >> >Unfortunately, there is no way with XFS to merge two or more HDD,
>in
>> >> >order
>> >> >to use more than one HDD, like RAID1 or RAID5.
>> >> >My primary goal is to use two server with GlusterFS on top of
>> >multiples
>> >> >HDDs for qemu images.
>> >> >I have think about BTRFS or mdadm.
>> >> >Anybody has some experience on this?
>> >> >
>> >> >Thanks a lot
>> >> >
>> >> >---
>> >> >Gilberto Nunes Ferreira
>> >>
>>

From gilberto.nunes32 at gmail.com  Thu Jul 30 15:14:46 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 12:14:46 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <FC4E80D9-52E7-45E7-827C-71493341E261@yahoo.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>
	<CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>
	<FC4E80D9-52E7-45E7-827C-71493341E261@yahoo.com>
Message-ID: <CAOKSTBuHT7xkS1q3G-JdiprGV2Q3cEaQkV_UsHXcAbrGXj9aBw@mail.gmail.com>

Debian Buster (Proxmox VE which is Debian Buster with Ubuntu kernel)
---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 12:12, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> If it crashes  - that's  a  bug  and worth checking. What OS do you use ?
>
> Best Regards,
> Strahil Nikolov
>
> ?? 30 ??? 2020 ?. 15:59:57 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >Yes! But still with mdadm if you lose 1 disk and reboot the server, the
> >system crashes.
> >But with ZFS there's no crash.
> >
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em qui., 30 de jul. de 2020 ?s 09:55, Strahil Nikolov
> ><hunter86_bg at yahoo.com>
> >escreveu:
> >
> >> I guess  there  is no  automatic  hot-spare  replacement in LVM, but
> >> mdadm has  that  functionality.
> >>
> >> Best  Regards,
> >> Strahil Nikolov
> >>
> >>
> >> ?? 30 ??? 2020 ?. 15:39:18 GMT+03:00, Gilberto Nunes <
> >> gilberto.nunes32 at gmail.com> ??????:
> >> >Doing some research and the chvg command, which is responsable for
> >> >create
> >> >hotspare disks in LVM is available only in AIX!
> >> >I have a Debian Buster box and there is no such command chvg.
> >Correct
> >> >me if
> >> >I am wrong....
> >> >---
> >> >Gilberto Nunes Ferreira
> >> >
> >> >(47) 3025-5907
> >> >(47) 99676-7530 - Whatsapp / Telegram
> >> >
> >> >Skype: gilberto.nunes36
> >> >
> >> >
> >> >
> >> >
> >> >
> >> >Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov
> >> ><hunter86_bg at yahoo.com>
> >> >escreveu:
> >> >
> >> >> LVM allows creating/converting striped/mirrored LVs without any
> >> >dowtime
> >> >> and it's using the md module.
> >> >>
> >> >> Best Regards,
> >> >> Strahil Nikolov
> >> >>
> >> >> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
> >> >> gilberto.nunes32 at gmail.com> ??????:
> >> >> >Hi there....
> >> >> >
> >> >> >'till now, I am using glusterfs over XFS and so far so good.
> >> >> >Using LVM too....
> >> >> >Unfortunately, there is no way with XFS to merge two or more HDD,
> >in
> >> >> >order
> >> >> >to use more than one HDD, like RAID1 or RAID5.
> >> >> >My primary goal is to use two server with GlusterFS on top of
> >> >multiples
> >> >> >HDDs for qemu images.
> >> >> >I have think about BTRFS or mdadm.
> >> >> >Anybody has some experience on this?
> >> >> >
> >> >> >Thanks a lot
> >> >> >
> >> >> >---
> >> >> >Gilberto Nunes Ferreira
> >> >>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/ada5b03e/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul 30 15:13:55 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 18:13:55 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
	<CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
Message-ID: <C73AA549-D4E7-476F-A19F-369FBD2C02D3@yahoo.com>

The crash  with failed  mdadm disk is not normal. We need to check it out.

Are  you using  Legacy or  UEFI  ?

Best Regards,
Strahil Nikolov

?? 30 ??? 2020 ?. 16:25:47 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>But I will give it a try in the lvm mirroring process.... Thanks
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
>gilberto.nunes32 at gmail.com> escreveu:
>
>> Well, actually I am not concerned ONLY about mirroring the data, but
>with
>> reliability on it.
>> Here during my lab tests I notice if I pull off a disk and then power
>on
>> the server, the LVM crashes...
>> On the other hand, with ZFS even in degraded state the system booted
>> normally.
>> The same happens with mdadm. That's the point.
>>
>> ---
>> Gilberto Nunes Ferreira
>>
>> (47) 3025-5907
>> (47) 99676-7530 - Whatsapp / Telegram
>>
>> Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr <alvin at netvel.net>
>> escreveu:
>>
>>> LVM supports mirroring or raid1.
>>> So to create a raid1 you would do something like "lvcreate -n
>raidvol  -L
>>> 1T --mirrors 1  /dev/bigvg"
>>> You can add a mirror to an existing logical volume using lvconvert 
>with
>>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
>>>
>>> But other than simple mirroring you will need mdadm.
>>>
>>>
>>>
>>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
>>>
>>> Doing some research and the chvg command, which is responsable for
>create
>>> hotspare disks in LVM is available only in AIX!
>>> I have a Debian Buster box and there is no such command chvg.
>Correct me
>>> if I am wrong....
>>> ---
>>> Gilberto Nunes Ferreira
>>>
>>> (47) 3025-5907
>>> (47) 99676-7530 - Whatsapp / Telegram
>>>
>>> Skype: gilberto.nunes36
>>>
>>>
>>>
>>>
>>>
>>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
>>> hunter86_bg at yahoo.com> escreveu:
>>>
>>>> LVM allows creating/converting striped/mirrored LVs without any
>dowtime
>>>> and it's using the md module.
>>>>
>>>> Best Regards,
>>>> Strahil Nikolov
>>>>
>>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>>>> gilberto.nunes32 at gmail.com> ??????:
>>>> >Hi there....
>>>> >
>>>> >'till now, I am using glusterfs over XFS and so far so good.
>>>> >Using LVM too....
>>>> >Unfortunately, there is no way with XFS to merge two or more HDD,
>in
>>>> >order
>>>> >to use more than one HDD, like RAID1 or RAID5.
>>>> >My primary goal is to use two server with GlusterFS on top of
>multiples
>>>> >HDDs for qemu images.
>>>> >I have think about BTRFS or mdadm.
>>>> >Anybody has some experience on this?
>>>> >
>>>> >Thanks a lot
>>>> >
>>>> >---
>>>> >Gilberto Nunes Ferreira
>>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing
>listGluster-users at gluster.orghttps://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>> --
>>> Alvin Starr                   ||   land:  (647)478-6285
>>> Netvel Inc.                   ||   Cell: 
>(416)806-0133alvin at netvel.net              ||
>>>
>>>
>>> ________
>>>
>>>
>>>
>>> Community Meeting Calendar:
>>>
>>> Schedule -
>>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> Bridge: https://bluejeans.com/441850968
>>>
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>

From gilberto.nunes32 at gmail.com  Thu Jul 30 15:17:49 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 12:17:49 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <C73AA549-D4E7-476F-A19F-369FBD2C02D3@yahoo.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
	<CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
	<C73AA549-D4E7-476F-A19F-369FBD2C02D3@yahoo.com>
Message-ID: <CAOKSTBtxiERY+dV5YKfPRZSNvzm02nZ-FjB8C=k+C4Ho-u9iWg@mail.gmail.com>

I am using Legacy... But I am using Virtualbox in my labs... Perhaps this
is the problem...
I don't do that in real hardware. But with spare disk (2 + 1) in mdadm it's
fine.

---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 12:16, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> The crash  with failed  mdadm disk is not normal. We need to check it out.
>
> Are  you using  Legacy or  UEFI  ?
>
> Best Regards,
> Strahil Nikolov
>
> ?? 30 ??? 2020 ?. 16:25:47 GMT+03:00, Gilberto Nunes <
> gilberto.nunes32 at gmail.com> ??????:
> >But I will give it a try in the lvm mirroring process.... Thanks
> >---
> >Gilberto Nunes Ferreira
> >
> >(47) 3025-5907
> >(47) 99676-7530 - Whatsapp / Telegram
> >
> >Skype: gilberto.nunes36
> >
> >
> >
> >
> >
> >Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
> >gilberto.nunes32 at gmail.com> escreveu:
> >
> >> Well, actually I am not concerned ONLY about mirroring the data, but
> >with
> >> reliability on it.
> >> Here during my lab tests I notice if I pull off a disk and then power
> >on
> >> the server, the LVM crashes...
> >> On the other hand, with ZFS even in degraded state the system booted
> >> normally.
> >> The same happens with mdadm. That's the point.
> >>
> >> ---
> >> Gilberto Nunes Ferreira
> >>
> >> (47) 3025-5907
> >> (47) 99676-7530 - Whatsapp / Telegram
> >>
> >> Skype: gilberto.nunes36
> >>
> >>
> >>
> >>
> >>
> >> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr <alvin at netvel.net>
> >> escreveu:
> >>
> >>> LVM supports mirroring or raid1.
> >>> So to create a raid1 you would do something like "lvcreate -n
> >raidvol  -L
> >>> 1T --mirrors 1  /dev/bigvg"
> >>> You can add a mirror to an existing logical volume using lvconvert
> >with
> >>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
> >>>
> >>> But other than simple mirroring you will need mdadm.
> >>>
> >>>
> >>>
> >>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
> >>>
> >>> Doing some research and the chvg command, which is responsable for
> >create
> >>> hotspare disks in LVM is available only in AIX!
> >>> I have a Debian Buster box and there is no such command chvg.
> >Correct me
> >>> if I am wrong....
> >>> ---
> >>> Gilberto Nunes Ferreira
> >>>
> >>> (47) 3025-5907
> >>> (47) 99676-7530 - Whatsapp / Telegram
> >>>
> >>> Skype: gilberto.nunes36
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
> >>> hunter86_bg at yahoo.com> escreveu:
> >>>
> >>>> LVM allows creating/converting striped/mirrored LVs without any
> >dowtime
> >>>> and it's using the md module.
> >>>>
> >>>> Best Regards,
> >>>> Strahil Nikolov
> >>>>
> >>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
> >>>> gilberto.nunes32 at gmail.com> ??????:
> >>>> >Hi there....
> >>>> >
> >>>> >'till now, I am using glusterfs over XFS and so far so good.
> >>>> >Using LVM too....
> >>>> >Unfortunately, there is no way with XFS to merge two or more HDD,
> >in
> >>>> >order
> >>>> >to use more than one HDD, like RAID1 or RAID5.
> >>>> >My primary goal is to use two server with GlusterFS on top of
> >multiples
> >>>> >HDDs for qemu images.
> >>>> >I have think about BTRFS or mdadm.
> >>>> >Anybody has some experience on this?
> >>>> >
> >>>> >Thanks a lot
> >>>> >
> >>>> >---
> >>>> >Gilberto Nunes Ferreira
> >>>>
> >>>
> >>> ________
> >>>
> >>>
> >>>
> >>> Community Meeting Calendar:
> >>>
> >>> Schedule -
> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> Bridge: https://bluejeans.com/441850968
> >>>
> >>> Gluster-users mailing
> >listGluster-users at gluster.orghttps://
> lists.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>>
> >>> --
> >>> Alvin Starr                   ||   land:  (647)478-6285
> >>> Netvel Inc.                   ||   Cell:
> >(416)806-0133alvin at netvel.net              ||
> >>>
> >>>
> >>> ________
> >>>
> >>>
> >>>
> >>> Community Meeting Calendar:
> >>>
> >>> Schedule -
> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> Bridge: https://bluejeans.com/441850968
> >>>
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
> >>>
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/db70069d/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul 30 16:32:32 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 19:32:32 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBtxiERY+dV5YKfPRZSNvzm02nZ-FjB8C=k+C4Ho-u9iWg@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
	<CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
	<C73AA549-D4E7-476F-A19F-369FBD2C02D3@yahoo.com>
	<CAOKSTBtxiERY+dV5YKfPRZSNvzm02nZ-FjB8C=k+C4Ho-u9iWg@mail.gmail.com>
Message-ID: <61359FD5-58A6-48D2-BC9F-4682808A8964@yahoo.com>

When using legacy,  you need to  prepare the MBR on each disk, so the BIOS will be able to boot from it and load grub.

On UEFI, you will need 2 entries each pointing to the other disk. There  is a thread for each approach in the CentOS7 forums and the peocedure is almost the same on all linux .

Best Regards,
Strahil Nikolov

?? 30 ??? 2020 ?. 18:17:49 GMT+03:00, Gilberto Nunes <gilberto.nunes32 at gmail.com> ??????:
>I am using Legacy... But I am using Virtualbox in my labs... Perhaps
>this
>is the problem...
>I don't do that in real hardware. But with spare disk (2 + 1) in mdadm
>it's
>fine.
>
>---
>Gilberto Nunes Ferreira
>
>(47) 3025-5907
>(47) 99676-7530 - Whatsapp / Telegram
>
>Skype: gilberto.nunes36
>
>
>
>
>
>Em qui., 30 de jul. de 2020 ?s 12:16, Strahil Nikolov
><hunter86_bg at yahoo.com>
>escreveu:
>
>> The crash  with failed  mdadm disk is not normal. We need to check it
>out.
>>
>> Are  you using  Legacy or  UEFI  ?
>>
>> Best Regards,
>> Strahil Nikolov
>>
>> ?? 30 ??? 2020 ?. 16:25:47 GMT+03:00, Gilberto Nunes <
>> gilberto.nunes32 at gmail.com> ??????:
>> >But I will give it a try in the lvm mirroring process.... Thanks
>> >---
>> >Gilberto Nunes Ferreira
>> >
>> >(47) 3025-5907
>> >(47) 99676-7530 - Whatsapp / Telegram
>> >
>> >Skype: gilberto.nunes36
>> >
>> >
>> >
>> >
>> >
>> >Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
>> >gilberto.nunes32 at gmail.com> escreveu:
>> >
>> >> Well, actually I am not concerned ONLY about mirroring the data,
>but
>> >with
>> >> reliability on it.
>> >> Here during my lab tests I notice if I pull off a disk and then
>power
>> >on
>> >> the server, the LVM crashes...
>> >> On the other hand, with ZFS even in degraded state the system
>booted
>> >> normally.
>> >> The same happens with mdadm. That's the point.
>> >>
>> >> ---
>> >> Gilberto Nunes Ferreira
>> >>
>> >> (47) 3025-5907
>> >> (47) 99676-7530 - Whatsapp / Telegram
>> >>
>> >> Skype: gilberto.nunes36
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr
><alvin at netvel.net>
>> >> escreveu:
>> >>
>> >>> LVM supports mirroring or raid1.
>> >>> So to create a raid1 you would do something like "lvcreate -n
>> >raidvol  -L
>> >>> 1T --mirrors 1  /dev/bigvg"
>> >>> You can add a mirror to an existing logical volume using
>lvconvert
>> >with
>> >>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
>> >>>
>> >>> But other than simple mirroring you will need mdadm.
>> >>>
>> >>>
>> >>>
>> >>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
>> >>>
>> >>> Doing some research and the chvg command, which is responsable
>for
>> >create
>> >>> hotspare disks in LVM is available only in AIX!
>> >>> I have a Debian Buster box and there is no such command chvg.
>> >Correct me
>> >>> if I am wrong....
>> >>> ---
>> >>> Gilberto Nunes Ferreira
>> >>>
>> >>> (47) 3025-5907
>> >>> (47) 99676-7530 - Whatsapp / Telegram
>> >>>
>> >>> Skype: gilberto.nunes36
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
>> >>> hunter86_bg at yahoo.com> escreveu:
>> >>>
>> >>>> LVM allows creating/converting striped/mirrored LVs without any
>> >dowtime
>> >>>> and it's using the md module.
>> >>>>
>> >>>> Best Regards,
>> >>>> Strahil Nikolov
>> >>>>
>> >>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>> >>>> gilberto.nunes32 at gmail.com> ??????:
>> >>>> >Hi there....
>> >>>> >
>> >>>> >'till now, I am using glusterfs over XFS and so far so good.
>> >>>> >Using LVM too....
>> >>>> >Unfortunately, there is no way with XFS to merge two or more
>HDD,
>> >in
>> >>>> >order
>> >>>> >to use more than one HDD, like RAID1 or RAID5.
>> >>>> >My primary goal is to use two server with GlusterFS on top of
>> >multiples
>> >>>> >HDDs for qemu images.
>> >>>> >I have think about BTRFS or mdadm.
>> >>>> >Anybody has some experience on this?
>> >>>> >
>> >>>> >Thanks a lot
>> >>>> >
>> >>>> >---
>> >>>> >Gilberto Nunes Ferreira
>> >>>>
>> >>>
>> >>> ________
>> >>>
>> >>>
>> >>>
>> >>> Community Meeting Calendar:
>> >>>
>> >>> Schedule -
>> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >>> Bridge: https://bluejeans.com/441850968
>> >>>
>> >>> Gluster-users mailing
>> >listGluster-users at gluster.orghttps://
>> lists.gluster.org/mailman/listinfo/gluster-users
>> >>>
>> >>>
>> >>> --
>> >>> Alvin Starr                   ||   land:  (647)478-6285
>> >>> Netvel Inc.                   ||   Cell:
>> >(416)806-0133alvin at netvel.net              ||
>> >>>
>> >>>
>> >>> ________
>> >>>
>> >>>
>> >>>
>> >>> Community Meeting Calendar:
>> >>>
>> >>> Schedule -
>> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> >>> Bridge: https://bluejeans.com/441850968
>> >>>
>> >>> Gluster-users mailing list
>> >>> Gluster-users at gluster.org
>> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
>> >>>
>> >>
>>

From hunter86_bg at yahoo.com  Thu Jul 30 16:34:40 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 19:34:40 +0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <61359FD5-58A6-48D2-BC9F-4682808A8964@yahoo.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
	<CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
	<C73AA549-D4E7-476F-A19F-369FBD2C02D3@yahoo.com>
	<CAOKSTBtxiERY+dV5YKfPRZSNvzm02nZ-FjB8C=k+C4Ho-u9iWg@mail.gmail.com>
	<61359FD5-58A6-48D2-BC9F-4682808A8964@yahoo.com>
Message-ID: <9E427424-3EA8-4888-80E5-C9AC65EB7EC7@yahoo.com>

Anyway in raid1+spare  on a replica 2  volume you will be using 6 disks  in total.
It  will  be more optimal to get all those  disks  in 'replica 3' or 'replica 3  arbiter 1' (for the arbiter it would be optimal  to have a small ssd and the data disks for tge actual data).

Best Regards,
Strahil Nikolov

?? 30 ??? 2020 ?. 19:32:32 GMT+03:00, Strahil Nikolov <hunter86_bg at yahoo.com> ??????:
>When using legacy,  you need to  prepare the MBR on each disk, so the
>BIOS will be able to boot from it and load grub.
>
>On UEFI, you will need 2 entries each pointing to the other disk. There
>is a thread for each approach in the CentOS7 forums and the peocedure
>is almost the same on all linux .
>
>Best Regards,
>Strahil Nikolov
>
>?? 30 ??? 2020 ?. 18:17:49 GMT+03:00, Gilberto Nunes
><gilberto.nunes32 at gmail.com> ??????:
>>I am using Legacy... But I am using Virtualbox in my labs... Perhaps
>>this
>>is the problem...
>>I don't do that in real hardware. But with spare disk (2 + 1) in mdadm
>>it's
>>fine.
>>
>>---
>>Gilberto Nunes Ferreira
>>
>>(47) 3025-5907
>>(47) 99676-7530 - Whatsapp / Telegram
>>
>>Skype: gilberto.nunes36
>>
>>
>>
>>
>>
>>Em qui., 30 de jul. de 2020 ?s 12:16, Strahil Nikolov
>><hunter86_bg at yahoo.com>
>>escreveu:
>>
>>> The crash  with failed  mdadm disk is not normal. We need to check
>it
>>out.
>>>
>>> Are  you using  Legacy or  UEFI  ?
>>>
>>> Best Regards,
>>> Strahil Nikolov
>>>
>>> ?? 30 ??? 2020 ?. 16:25:47 GMT+03:00, Gilberto Nunes <
>>> gilberto.nunes32 at gmail.com> ??????:
>>> >But I will give it a try in the lvm mirroring process.... Thanks
>>> >---
>>> >Gilberto Nunes Ferreira
>>> >
>>> >(47) 3025-5907
>>> >(47) 99676-7530 - Whatsapp / Telegram
>>> >
>>> >Skype: gilberto.nunes36
>>> >
>>> >
>>> >
>>> >
>>> >
>>> >Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
>>> >gilberto.nunes32 at gmail.com> escreveu:
>>> >
>>> >> Well, actually I am not concerned ONLY about mirroring the data,
>>but
>>> >with
>>> >> reliability on it.
>>> >> Here during my lab tests I notice if I pull off a disk and then
>>power
>>> >on
>>> >> the server, the LVM crashes...
>>> >> On the other hand, with ZFS even in degraded state the system
>>booted
>>> >> normally.
>>> >> The same happens with mdadm. That's the point.
>>> >>
>>> >> ---
>>> >> Gilberto Nunes Ferreira
>>> >>
>>> >> (47) 3025-5907
>>> >> (47) 99676-7530 - Whatsapp / Telegram
>>> >>
>>> >> Skype: gilberto.nunes36
>>> >>
>>> >>
>>> >>
>>> >>
>>> >>
>>> >> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr
>><alvin at netvel.net>
>>> >> escreveu:
>>> >>
>>> >>> LVM supports mirroring or raid1.
>>> >>> So to create a raid1 you would do something like "lvcreate -n
>>> >raidvol  -L
>>> >>> 1T --mirrors 1  /dev/bigvg"
>>> >>> You can add a mirror to an existing logical volume using
>>lvconvert
>>> >with
>>> >>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
>>> >>>
>>> >>> But other than simple mirroring you will need mdadm.
>>> >>>
>>> >>>
>>> >>>
>>> >>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
>>> >>>
>>> >>> Doing some research and the chvg command, which is responsable
>>for
>>> >create
>>> >>> hotspare disks in LVM is available only in AIX!
>>> >>> I have a Debian Buster box and there is no such command chvg.
>>> >Correct me
>>> >>> if I am wrong....
>>> >>> ---
>>> >>> Gilberto Nunes Ferreira
>>> >>>
>>> >>> (47) 3025-5907
>>> >>> (47) 99676-7530 - Whatsapp / Telegram
>>> >>>
>>> >>> Skype: gilberto.nunes36
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
>>> >>> hunter86_bg at yahoo.com> escreveu:
>>> >>>
>>> >>>> LVM allows creating/converting striped/mirrored LVs without any
>>> >dowtime
>>> >>>> and it's using the md module.
>>> >>>>
>>> >>>> Best Regards,
>>> >>>> Strahil Nikolov
>>> >>>>
>>> >>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
>>> >>>> gilberto.nunes32 at gmail.com> ??????:
>>> >>>> >Hi there....
>>> >>>> >
>>> >>>> >'till now, I am using glusterfs over XFS and so far so good.
>>> >>>> >Using LVM too....
>>> >>>> >Unfortunately, there is no way with XFS to merge two or more
>>HDD,
>>> >in
>>> >>>> >order
>>> >>>> >to use more than one HDD, like RAID1 or RAID5.
>>> >>>> >My primary goal is to use two server with GlusterFS on top of
>>> >multiples
>>> >>>> >HDDs for qemu images.
>>> >>>> >I have think about BTRFS or mdadm.
>>> >>>> >Anybody has some experience on this?
>>> >>>> >
>>> >>>> >Thanks a lot
>>> >>>> >
>>> >>>> >---
>>> >>>> >Gilberto Nunes Ferreira
>>> >>>>
>>> >>>
>>> >>> ________
>>> >>>
>>> >>>
>>> >>>
>>> >>> Community Meeting Calendar:
>>> >>>
>>> >>> Schedule -
>>> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >>> Bridge: https://bluejeans.com/441850968
>>> >>>
>>> >>> Gluster-users mailing
>>> >listGluster-users at gluster.orghttps://
>>> lists.gluster.org/mailman/listinfo/gluster-users
>>> >>>
>>> >>>
>>> >>> --
>>> >>> Alvin Starr                   ||   land:  (647)478-6285
>>> >>> Netvel Inc.                   ||   Cell:
>>> >(416)806-0133alvin at netvel.net              ||
>>> >>>
>>> >>>
>>> >>> ________
>>> >>>
>>> >>>
>>> >>>
>>> >>> Community Meeting Calendar:
>>> >>>
>>> >>> Schedule -
>>> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>>> >>> Bridge: https://bluejeans.com/441850968
>>> >>>
>>> >>> Gluster-users mailing list
>>> >>> Gluster-users at gluster.org
>>> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>> >>>
>>> >>
>>>

From gilberto.nunes32 at gmail.com  Thu Jul 30 16:39:16 2020
From: gilberto.nunes32 at gmail.com (Gilberto Nunes)
Date: Thu, 30 Jul 2020 13:39:16 -0300
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <9E427424-3EA8-4888-80E5-C9AC65EB7EC7@yahoo.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<f36ef289-4554-b610-8c58-45cebed17af8@netvel.net>
	<CAOKSTBuUKNCJ=A3hr6gUF1S-jRY8ZKKmVWsmH8j97L4yyao-FQ@mail.gmail.com>
	<CAOKSTBukVfH1Yxqp_ZvmKAxmJiU6YU_ugyR2Va0v=NL=GngNRA@mail.gmail.com>
	<C73AA549-D4E7-476F-A19F-369FBD2C02D3@yahoo.com>
	<CAOKSTBtxiERY+dV5YKfPRZSNvzm02nZ-FjB8C=k+C4Ho-u9iWg@mail.gmail.com>
	<61359FD5-58A6-48D2-BC9F-4682808A8964@yahoo.com>
	<9E427424-3EA8-4888-80E5-C9AC65EB7EC7@yahoo.com>
Message-ID: <CAOKSTBv=ZqoOfi9DUop-TiFyTt=p3_AgPH8pSRjjrdn=vhAv_Q@mail.gmail.com>

I am dealing here with just 2 nodes and a bunch of disks on it... Just a
scenario for study but real in many cases we face low budgets...
Anyway thanks for the tips!



---
Gilberto Nunes Ferreira

(47) 3025-5907
(47) 99676-7530 - Whatsapp / Telegram

Skype: gilberto.nunes36





Em qui., 30 de jul. de 2020 ?s 13:34, Strahil Nikolov <hunter86_bg at yahoo.com>
escreveu:

> Anyway in raid1+spare  on a replica 2  volume you will be using 6 disks
> in total.
> It  will  be more optimal to get all those  disks  in 'replica 3' or
> 'replica 3  arbiter 1' (for the arbiter it would be optimal  to have a
> small ssd and the data disks for tge actual data).
>
> Best Regards,
> Strahil Nikolov
>
> ?? 30 ??? 2020 ?. 19:32:32 GMT+03:00, Strahil Nikolov <
> hunter86_bg at yahoo.com> ??????:
> >When using legacy,  you need to  prepare the MBR on each disk, so the
> >BIOS will be able to boot from it and load grub.
> >
> >On UEFI, you will need 2 entries each pointing to the other disk. There
> >is a thread for each approach in the CentOS7 forums and the peocedure
> >is almost the same on all linux .
> >
> >Best Regards,
> >Strahil Nikolov
> >
> >?? 30 ??? 2020 ?. 18:17:49 GMT+03:00, Gilberto Nunes
> ><gilberto.nunes32 at gmail.com> ??????:
> >>I am using Legacy... But I am using Virtualbox in my labs... Perhaps
> >>this
> >>is the problem...
> >>I don't do that in real hardware. But with spare disk (2 + 1) in mdadm
> >>it's
> >>fine.
> >>
> >>---
> >>Gilberto Nunes Ferreira
> >>
> >>(47) 3025-5907
> >>(47) 99676-7530 - Whatsapp / Telegram
> >>
> >>Skype: gilberto.nunes36
> >>
> >>
> >>
> >>
> >>
> >>Em qui., 30 de jul. de 2020 ?s 12:16, Strahil Nikolov
> >><hunter86_bg at yahoo.com>
> >>escreveu:
> >>
> >>> The crash  with failed  mdadm disk is not normal. We need to check
> >it
> >>out.
> >>>
> >>> Are  you using  Legacy or  UEFI  ?
> >>>
> >>> Best Regards,
> >>> Strahil Nikolov
> >>>
> >>> ?? 30 ??? 2020 ?. 16:25:47 GMT+03:00, Gilberto Nunes <
> >>> gilberto.nunes32 at gmail.com> ??????:
> >>> >But I will give it a try in the lvm mirroring process.... Thanks
> >>> >---
> >>> >Gilberto Nunes Ferreira
> >>> >
> >>> >(47) 3025-5907
> >>> >(47) 99676-7530 - Whatsapp / Telegram
> >>> >
> >>> >Skype: gilberto.nunes36
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >
> >>> >Em qui., 30 de jul. de 2020 ?s 10:23, Gilberto Nunes <
> >>> >gilberto.nunes32 at gmail.com> escreveu:
> >>> >
> >>> >> Well, actually I am not concerned ONLY about mirroring the data,
> >>but
> >>> >with
> >>> >> reliability on it.
> >>> >> Here during my lab tests I notice if I pull off a disk and then
> >>power
> >>> >on
> >>> >> the server, the LVM crashes...
> >>> >> On the other hand, with ZFS even in degraded state the system
> >>booted
> >>> >> normally.
> >>> >> The same happens with mdadm. That's the point.
> >>> >>
> >>> >> ---
> >>> >> Gilberto Nunes Ferreira
> >>> >>
> >>> >> (47) 3025-5907
> >>> >> (47) 99676-7530 - Whatsapp / Telegram
> >>> >>
> >>> >> Skype: gilberto.nunes36
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >>
> >>> >> Em qui., 30 de jul. de 2020 ?s 10:18, Alvin Starr
> >><alvin at netvel.net>
> >>> >> escreveu:
> >>> >>
> >>> >>> LVM supports mirroring or raid1.
> >>> >>> So to create a raid1 you would do something like "lvcreate -n
> >>> >raidvol  -L
> >>> >>> 1T --mirrors 1  /dev/bigvg"
> >>> >>> You can add a mirror to an existing logical volume using
> >>lvconvert
> >>> >with
> >>> >>> something like "lvconvert --mirrors +1 bigvg/unmirroredlv"
> >>> >>>
> >>> >>> But other than simple mirroring you will need mdadm.
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> On 7/30/20 8:39 AM, Gilberto Nunes wrote:
> >>> >>>
> >>> >>> Doing some research and the chvg command, which is responsable
> >>for
> >>> >create
> >>> >>> hotspare disks in LVM is available only in AIX!
> >>> >>> I have a Debian Buster box and there is no such command chvg.
> >>> >Correct me
> >>> >>> if I am wrong....
> >>> >>> ---
> >>> >>> Gilberto Nunes Ferreira
> >>> >>>
> >>> >>> (47) 3025-5907
> >>> >>> (47) 99676-7530 - Whatsapp / Telegram
> >>> >>>
> >>> >>> Skype: gilberto.nunes36
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> Em qui., 30 de jul. de 2020 ?s 02:05, Strahil Nikolov <
> >>> >>> hunter86_bg at yahoo.com> escreveu:
> >>> >>>
> >>> >>>> LVM allows creating/converting striped/mirrored LVs without any
> >>> >dowtime
> >>> >>>> and it's using the md module.
> >>> >>>>
> >>> >>>> Best Regards,
> >>> >>>> Strahil Nikolov
> >>> >>>>
> >>> >>>> ?? 28 ??? 2020 ?. 22:43:39 GMT+03:00, Gilberto Nunes <
> >>> >>>> gilberto.nunes32 at gmail.com> ??????:
> >>> >>>> >Hi there....
> >>> >>>> >
> >>> >>>> >'till now, I am using glusterfs over XFS and so far so good.
> >>> >>>> >Using LVM too....
> >>> >>>> >Unfortunately, there is no way with XFS to merge two or more
> >>HDD,
> >>> >in
> >>> >>>> >order
> >>> >>>> >to use more than one HDD, like RAID1 or RAID5.
> >>> >>>> >My primary goal is to use two server with GlusterFS on top of
> >>> >multiples
> >>> >>>> >HDDs for qemu images.
> >>> >>>> >I have think about BTRFS or mdadm.
> >>> >>>> >Anybody has some experience on this?
> >>> >>>> >
> >>> >>>> >Thanks a lot
> >>> >>>> >
> >>> >>>> >---
> >>> >>>> >Gilberto Nunes Ferreira
> >>> >>>>
> >>> >>>
> >>> >>> ________
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> Community Meeting Calendar:
> >>> >>>
> >>> >>> Schedule -
> >>> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> >>> Bridge: https://bluejeans.com/441850968
> >>> >>>
> >>> >>> Gluster-users mailing
> >>> >listGluster-users at gluster.orghttps://
> >>> lists.gluster.org/mailman/listinfo/gluster-users
> >>> >>>
> >>> >>>
> >>> >>> --
> >>> >>> Alvin Starr                   ||   land:  (647)478-6285
> >>> >>> Netvel Inc.                   ||   Cell:
> >>> >(416)806-0133alvin at netvel.net              ||
> >>> >>>
> >>> >>>
> >>> >>> ________
> >>> >>>
> >>> >>>
> >>> >>>
> >>> >>> Community Meeting Calendar:
> >>> >>>
> >>> >>> Schedule -
> >>> >>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> >>> >>> Bridge: https://bluejeans.com/441850968
> >>> >>>
> >>> >>> Gluster-users mailing list
> >>> >>> Gluster-users at gluster.org
> >>> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
> >>> >>>
> >>> >>
> >>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/49accd1a/attachment.html>

From phaley at mit.edu  Thu Jul 30 17:22:18 2020
From: phaley at mit.edu (Pat Haley)
Date: Thu, 30 Jul 2020 13:22:18 -0400
Subject: [Gluster-users] gluster peer disconnected
Message-ID: <cb1c27e2-3bc5-b5f3-2102-544a2abc9125@mit.edu>


Hi,

We have a cluster whose common storage is a gluster volume consisting of 
4 bricks residing on 2 servers (more details at bottom).? Yesterday we 
experienced a power outage.? To start the gluster volume after the power 
came back I had to

  * manually start a gluster daemon on one of the servers (mseas-data3)
  * start the gluster volume on the other server (mseas-data2)
      o I had just tried starting the gluster volume without manually
        starting the other daemon but that was unsuccessful.

After this my recollection is that the peers were talking to each other 
at that time.

Today I was looking around and noticed that the mseas-data3 server is in 
a disconnected state (even though the compute nodes of our cluster are 
seeing the full gluster volume)

-----------------------

[root at mseas-data2 ~]# gluster peer status
Number of Peers: 1

Hostname: mseas-data3
Uuid: b39d4deb-c291-437e-8013-09050c1fa9e3
State: Peer in Cluster (Disconnected)

-----------------------

Following the advice on 
https://lists.gluster.org/pipermail/gluster-users/2015-April/021597.html 
, I confirmed that the 2 servers can ping each other.? The gluster 
daemon on mseas-data2 is active but the daemon on mseas-data3 shows

--------------------------------

[root at mseas-data3 ~]# service glusterd status
glusterd dead but pid file exists

--------------------------------

Is it safe to just restart that daemon on mseas-data3?? Is there some 
other procedure I should do? I ask because we have a number of job 
running that appear to be successfully writing to the gluster volume and 
I'd prefer that they continue if possible.

Any advice would be appreciated.? Thanks

---------------------------------------------------

[root at mseas-data2 ~]# gluster volume info

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 4
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Brick3: mseas-data3:/export/sda/brick3
Brick4: mseas-data3:/export/sdc/brick4
Options Reconfigured:
diagnostics.client-log-level: ERROR
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
cluster.min-free-disk: 1%

-- 

-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley                          Email:  phaley at mit.edu
Center for Ocean Engineering       Phone:  (617) 253-6824
Dept. of Mechanical Engineering    Fax:    (617) 253-8125
MIT, Room 5-213                    http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA  02139-4301

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/714988c3/attachment.html>

From archon810 at gmail.com  Thu Jul 30 19:32:24 2020
From: archon810 at gmail.com (Artem Russakovskii)
Date: Thu, 30 Jul 2020 12:32:24 -0700
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
In-Reply-To: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
References: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
Message-ID: <CAD+dzQep-zAiEW3GpKgF138SJ7EavM0t5Ck=A3+TVKdazSxrJA@mail.gmail.com>

Hi,

https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.1-7/openSUSE_Leap_15.1/x86_64/
is still missing 7.7. Is there an ETA please?

Thanks.


Sincerely,
Artem

--
Founder, Android Police <http://www.androidpolice.com>, APK Mirror
<http://www.apkmirror.com/>, Illogical Robot LLC
beerpla.net | @ArtemR <http://twitter.com/ArtemR>


On Wed, Jul 22, 2020 at 9:27 AM Rinku Kothiya <rkothiya at redhat.com> wrote:

> Hi,
>
> The Gluster community is pleased to announce the release of Gluster7.7
> (packages available at [1]).
> Release notes for the release can be found at [2].
>
> Major changes, features and limitations addressed in this release:
> None
>
> Please Note: Some of the packages are unavailable and we are working on
> it. We will release them soon.
>
> Thanks,
> Gluster community
>
> References:
>
> [1] Packages for 7.7:
> https://download.gluster.org/pub/gluster/glusterfs/7/7.7/
>
> [2] Release notes for 7.7:
> https://docs.gluster.org/en/latest/release-notes/7.7/
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200730/6e6085c2/attachment.html>

From hunter86_bg at yahoo.com  Thu Jul 30 20:14:32 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 23:14:32 +0300
Subject: [Gluster-users] gluster peer disconnected
In-Reply-To: <cb1c27e2-3bc5-b5f3-2102-544a2abc9125@mit.edu>
References: <cb1c27e2-3bc5-b5f3-2102-544a2abc9125@mit.edu>
Message-ID: <81289C71-68D1-4253-982F-79A30FDFD8C4@yahoo.com>

Is 'gluster  pool list' consistent on all nodes?
Do you have all your bricks properly mounted on the affected node?

Bet Regards,
Strahil Nikolov

?? 30 ??? 2020 ?. 20:22:18 GMT+03:00, Pat Haley <phaley at mit.edu> ??????:
>
>Hi,
>
>We have a cluster whose common storage is a gluster volume consisting
>of 
>4 bricks residing on 2 servers (more details at bottom).? Yesterday we 
>experienced a power outage.? To start the gluster volume after the
>power 
>came back I had to
>
>  * manually start a gluster daemon on one of the servers (mseas-data3)
>  * start the gluster volume on the other server (mseas-data2)
>      o I had just tried starting the gluster volume without manually
>        starting the other daemon but that was unsuccessful.
>
>After this my recollection is that the peers were talking to each other
>
>at that time.
>
>Today I was looking around and noticed that the mseas-data3 server is
>in 
>a disconnected state (even though the compute nodes of our cluster are 
>seeing the full gluster volume)
>
>-----------------------
>
>[root at mseas-data2 ~]# gluster peer status
>Number of Peers: 1
>
>Hostname: mseas-data3
>Uuid: b39d4deb-c291-437e-8013-09050c1fa9e3
>State: Peer in Cluster (Disconnected)
>
>-----------------------
>
>Following the advice on 
>https://lists.gluster.org/pipermail/gluster-users/2015-April/021597.html
>
>, I confirmed that the 2 servers can ping each other.? The gluster 
>daemon on mseas-data2 is active but the daemon on mseas-data3 shows
>
>--------------------------------
>
>[root at mseas-data3 ~]# service glusterd status
>glusterd dead but pid file exists
>
>--------------------------------
>
>Is it safe to just restart that daemon on mseas-data3?? Is there some 
>other procedure I should do? I ask because we have a number of job 
>running that appear to be successfully writing to the gluster volume
>and 
>I'd prefer that they continue if possible.
>
>Any advice would be appreciated.? Thanks
>
>---------------------------------------------------
>
>[root at mseas-data2 ~]# gluster volume info
>
>Volume Name: data-volume
>Type: Distribute
>Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
>Status: Started
>Number of Bricks: 4
>Transport-type: tcp
>Bricks:
>Brick1: mseas-data2:/mnt/brick1
>Brick2: mseas-data2:/mnt/brick2
>Brick3: mseas-data3:/export/sda/brick3
>Brick4: mseas-data3:/export/sdc/brick4
>Options Reconfigured:
>diagnostics.client-log-level: ERROR
>network.inode-lru-limit: 50000
>performance.md-cache-timeout: 60
>performance.open-behind: off
>disperse.eager-lock: off
>auth.allow: *
>server.allow-insecure: on
>nfs.exports-auth-enable: on
>diagnostics.brick-sys-log-level: WARNING
>performance.readdir-ahead: on
>nfs.disable: on
>nfs.export-volumes: off
>cluster.min-free-disk: 1%

From hunter86_bg at yahoo.com  Thu Jul 30 20:15:50 2020
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Thu, 30 Jul 2020 23:15:50 +0300
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
In-Reply-To: <CAD+dzQep-zAiEW3GpKgF138SJ7EavM0t5Ck=A3+TVKdazSxrJA@mail.gmail.com>
References: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
	<CAD+dzQep-zAiEW3GpKgF138SJ7EavM0t5Ck=A3+TVKdazSxrJA@mail.gmail.com>
Message-ID: <FC05E9D3-3B36-44E4-ABD5-B9D1116507CC@yahoo.com>

In CentOS7 , the packages were available several hours before the announcement.

Best Regards,
Strahil Nikolov

?? 30 ??? 2020 ?. 22:32:24 GMT+03:00, Artem Russakovskii <archon810 at gmail.com> ??????:
>Hi,
>
>https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.1-7/openSUSE_Leap_15.1/x86_64/
>is still missing 7.7. Is there an ETA please?
>
>Thanks.
>
>
>Sincerely,
>Artem
>
>--
>Founder, Android Police <http://www.androidpolice.com>, APK Mirror
><http://www.apkmirror.com/>, Illogical Robot LLC
>beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
>On Wed, Jul 22, 2020 at 9:27 AM Rinku Kothiya <rkothiya at redhat.com>
>wrote:
>
>> Hi,
>>
>> The Gluster community is pleased to announce the release of
>Gluster7.7
>> (packages available at [1]).
>> Release notes for the release can be found at [2].
>>
>> Major changes, features and limitations addressed in this release:
>> None
>>
>> Please Note: Some of the packages are unavailable and we are working
>on
>> it. We will release them soon.
>>
>> Thanks,
>> Gluster community
>>
>> References:
>>
>> [1] Packages for 7.7:
>> https://download.gluster.org/pub/gluster/glusterfs/7/7.7/
>>
>> [2] Release notes for 7.7:
>> https://docs.gluster.org/en/latest/release-notes/7.7/
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>

From ravishankar at redhat.com  Fri Jul 31 03:20:32 2020
From: ravishankar at redhat.com (Ravishankar N)
Date: Fri, 31 Jul 2020 08:50:32 +0530
Subject: [Gluster-users] Gluster linear scale-out performance
In-Reply-To: <CAD+dzQdYk2xvDd2XifwFroO7nEYMJzDs-3ZOx+_-=g8czJNV-Q@mail.gmail.com>
References: <CADkK8gtUR7915ocArbsnqdK6w455BKxzncFXndATPt8T7aBeVA@mail.gmail.com>
	<CAJgorsa-aqJ8Yhv22SPKSHWM5gyhB5nCU76CsuvDpg2PhNHO0g@mail.gmail.com>
	<CADkK8gtbeR1i+m=PuFmCcxzFWL2YeDPMT9COBRyZcBj-k_2f=Q@mail.gmail.com>
	<CAJgorsZPA+e76NHhAttFc-0Vs6p9UQ7aCz1baCVh6A3PotjK3g@mail.gmail.com>
	<CADkK8gvKdO6EapmAA12VzCGj8m=BAvnEbW32QEaB-tTnrf6gQQ@mail.gmail.com>
	<CADkK8gs1pDrAmMA_ONnc06kGAX8uxr5A7CarGtkYMFfTg=1sTA@mail.gmail.com>
	<CAD+dzQdYk2xvDd2XifwFroO7nEYMJzDs-3ZOx+_-=g8czJNV-Q@mail.gmail.com>
Message-ID: <ab84b036-4f0b-4583-6d7a-54bc5772bfd5@redhat.com>


On 25/07/20 4:35 am, Artem Russakovskii wrote:
> Speaking of fio, could the gluster team please help me understand 
> something?
>
> We've been having lots of performance issues related to gluster using 
> attached block storage on Linode. At some point, I figured out that 
> Linode has a cap of 500 IOPS on their block storage 
> <https://www.linode.com/community/questions/19437/does-a-dedicated-cpu-or-high-memory-plan-improve-disk-io-performance#answer-72142> 
> (with spikes to 1500 IOPS). The block storage we use is formatted xfs 
> with 4KB bsize (block size).
>
> I then ran a bunch of fio tests on the block storage itself (not the 
> gluster fuse mount), which performed horribly when the bs parameter 
> was set to 4k:
> fio--randrepeat=1--ioengine=libaio--direct=1--gtod_reduce=1--name=test--filename=test--bs=4k--iodepth=64--size=4G--readwrite=randwrite--ramp_time=4
> During these tests, fio ETA crawled to over an hour, at some point 
> dropped to 45min and I did see 500-1500 IOPS flash by briefly, then it 
> went back down to 0. I/O seems majorly choked for some reason,?likely 
> because gluster is using some of it. Transfer speed with such 4k block 
> size is 2 MB/s with spikes to 6MB/s. This causes the load on the 
> server to spike up to 100+ and brings down all our servers.
> |Jobs: 1 (f=1): [w(1)][20.3%][r=0KiB/s,w=5908KiB/s][r=0,w=1477 
> IOPS][eta 43m:00s] Jobs: 1 (f=1): 
> [w(1)][21.5%][r=0KiB/s,w=0KiB/s][r=0,w=0 IOPS][eta 44m:54s] |
> |xfs_info /mnt/citadel_block1 meta-data=/dev/sdc isize=512 
> agcount=103, agsize=26214400 blks = sectsz=512 attr=2, projid32bit=1 = 
> crc=1 finobt=1, sparse=0, rmapbt=0 = reflink=0 data = bsize=4096 
> blocks=2684354560, imaxpct=25 = sunit=0 swidth=0 blks naming =version 
> 2 bsize=4096 ascii-ci=0, ftype=1 log =internal log bsize=4096 
> blocks=51200, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 
> realtime =none extsz=4096 blocks=0, rtextents=0|
> When I increase the --bs param to fio from 4k to, say, 64k, transfer 
> speed goes up significantly and is more like 50MB/s, and at 256k, it's 
> 200MB/s.
>
> So what I'm trying to understand is:
>
>  1. How does the xfs block size (4KB) relate to the block size in fio
>     tests? If we're limited by IOPS, and xfs block size is 4KB, how
>     can fio produce better results with varying --bs param?
>  2. Would increasing the xfs data block size to something like
>     64-256KB help with our issue of choking IO and skyrocketing load?
>
I have experienced similar behavior when running fio tests with bs=4k on 
a gluster volume backed by XFS with a high load (numjobs=32) . When I 
observed the strace of the brick processes (fsync -f -T -p $PID), I saw 
fysnc system calls taking around 2500 seconds which is insane. I'm not 
sure if this is specific to the way fio does its i/o pattern and the way 
XFS handles it. When I used 64k block sizes, the fio tests completed 
just fine.
>
>  1. The worst hangs and load spikes happen when we reboot one of the
>     gluster servers, but not when it's down - when it comes back
>     online. Even with gluster not showing anything pending heal, my
>     guess is it's still trying to do lots of IO between the 4 nodes
>     for some reason, but I don't understand why.
>
Do you kill all gluster processes (not just glusterd but even the brick 
processes) before issuing reboot? This is necessary to prevent I/O 
stalls. There is stop-all-gluster-processes.sh which should be available 
as a part of the gluster installation (maybe in 
/usr/share/glusterfs/scripts/) which you can use.? Can you check if this 
helps?

Regards,

Ravi

> I've been banging my head on the wall with this problem for months. 
> Appreciate any feedback here.
>
> Thank you.
>
> gluster volume info below
> |Volume Name: SNIP_data1 Type: Replicate Volume ID: SNIP Status: 
> Started Snapshot Count: 0 Number of Bricks: 1 x 4 = 4 Transport-type: 
> tcp Bricks: Brick1: nexus2:/mnt/SNIP_block1/SNIP_data1 Brick2: 
> forge:/mnt/SNIP_block1/SNIP_data1 Brick3: 
> hive:/mnt/SNIP_block1/SNIP_data1 Brick4: 
> citadel:/mnt/SNIP_block1/SNIP_data1 Options Reconfigured: 
> cluster.quorum-count: 1 cluster.quorum-type: fixed 
> network.ping-timeout: 5 network.remote-dio: enable 
> performance.rda-cache-limit: 256MB performance.readdir-ahead: on 
> performance.parallel-readdir: on network.inode-lru-limit: 500000 
> performance.md-cache-timeout: 600 performance.cache-invalidation: on 
> performance.stat-prefetch: on features.cache-invalidation-timeout: 600 
> features.cache-invalidation: on cluster.readdir-optimize: on 
> performance.io-thread-count: 32 server.event-threads: 4 
> client.event-threads: 4 performance.read-ahead: off 
> cluster.lookup-optimize: on performance.cache-size: 1GB 
> cluster.self-heal-daemon: enable transport.address-family: inet 
> nfs.disable: on performance.client-io-threads: on 
> cluster.granular-entry-heal: enable cluster.data-self-heal-algorithm: full|
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror 
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net <http://beerpla.net/> | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Thu, Jul 23, 2020 at 12:08 AM Qing Wang <qw at g.clemson.edu 
> <mailto:qw at g.clemson.edu>> wrote:
>
>     Hi,
>
>     I have one more question about the Gluster linear scale-out
>     performance regarding the "write-behind off" case specifically --
>     when "write-behind" is off, and still the stripe volumes and other
>     settings as early thread posted, the storage I/O seems not to
>     relate to the number of storage nodes. In my experiment, no matter
>     I have 2 brick server nodes or 8 brick server nodes, the
>     aggregated gluster I/O performance is ~100MB/sec. And fio
>     benchmark measurement gives the same result. If "write behind" is
>     on, then the storage performance is linear scale-out along with
>     the # of brick server nodes increasing.
>
>     No matter the write behind option is on/off, I thought the gluster
>     I/O performance should be pulled and aggregated together as a
>     whole. If that?is the case, why do I get a consistent?gluster
>     performance (~100MB/sec) when "write behind" is off? Please?advise
>     me if I misunderstood something.
>
>     Thanks,
>     Qing
>
>
>
>
>     On Tue, Jul 21, 2020 at 7:29 PM Qing Wang <qw at g.clemson.edu
>     <mailto:qw at g.clemson.edu>> wrote:
>
>         fio gives me the correct linear scale-out results, and you're
>         right, the storage cache is the root cause that?makes the dd
>         measurement results not accurate at all.
>
>         Thanks,
>         Qing
>
>
>         On Tue, Jul 21, 2020 at 2:53 PM Yaniv Kaul <ykaul at redhat.com
>         <mailto:ykaul at redhat.com>> wrote:
>
>
>
>             On Tue, 21 Jul 2020, 21:43 Qing Wang <qw at g.clemson.edu
>             <mailto:qw at g.clemson.edu>> wrote:
>
>                 Hi?Yaniv,
>
>                 Thanks for the quick response. I forget to mention I
>                 am testing the writing performance, not reading. In
>                 this case, would the client cache hit rate still be a
>                 big issue?
>
>
>             It's not hitting the storage directly. Since it's also
>             single threaded, it may also not saturate it. I highly
>             recommend testing properly.
>             Y.
>
>
>                 I'll use fio to run my test once again, thanks for the
>                 suggestion.
>
>                 Thanks,
>                 Qing
>
>                 On Tue, Jul 21, 2020 at 2:38 PM Yaniv Kaul
>                 <ykaul at redhat.com <mailto:ykaul at redhat.com>> wrote:
>
>
>
>                     On Tue, 21 Jul 2020, 21:30 Qing Wang
>                     <qw at g.clemson.edu <mailto:qw at g.clemson.edu>> wrote:
>
>                         Hi,
>
>                         I am trying to test Gluster linear scale-out
>                         performance by adding more storage
>                         server/bricks, and measure the storage I/O
>                         performance. To vary the storage server
>                         number, I create several "stripe" volumes that
>                         contain 2 brick servers, 3 brick servers, 4
>                         brick servers, and so on. On gluster client
>                         side, I used "dd if=/dev/zero
>                         of=/mnt/glusterfs/dns_test_data_26g bs=1M
>                         count=26000" to create 26G data (or larger
>                         size), and those data will be distributed to
>                         the corresponding gluster?servers (each has
>                         gluster brick on it) and "dd" returns the
>                         final I/O throughput. The Internet is 40G
>                         infiniband, although I didn't do any specific
>                         configurations to use advanced features.
>
>
>                     Your dd command is inaccurate, as it'll hit the
>                     client cache. It is also single threaded. I
>                     suggest switching to fio.
>                     Y.
>
>
>                         What confuses me is that the storage I/O seems
>                         not to relate to the number of storage
>                         nodes,?but Gluster documents said it should be
>                         linear scaling. For example, when
>                         "write-behind" is on, and when Infiniband
>                         "jumbo frame" (connected mode) is on, I can
>                         get ~800 MB/sec reported by "dd", no matter I
>                         have 2 brick servers or 8 brick servers -- for
>                         2 server case, each server can have ~400
>                         MB/sec; for 4 server case, each server can
>                         have ~200MB/sec. That said, each server I/O
>                         does aggregate to the final storage I/O (800
>                         MB/sec), but this is not "linear scale-out".
>
>                         Can somebody help me to understand why this is
>                         the case? I certainly can have some
>                         misunderstanding/misconfiguration here. Please
>                         correct me if I do, thanks!
>
>                         Best,
>                         Qing
>                         ________
>
>
>
>                         Community Meeting Calendar:
>
>                         Schedule -
>                         Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>                         Bridge: https://bluejeans.com/441850968
>
>                         Gluster-users mailing list
>                         Gluster-users at gluster.org
>                         <mailto:Gluster-users at gluster.org>
>                         https://lists.gluster.org/mailman/listinfo/gluster-users
>
>     ________
>
>
>
>     Community Meeting Calendar:
>
>     Schedule -
>     Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>     Bridge: https://bluejeans.com/441850968
>
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200731/582cd5b1/attachment.html>

From sacharya at redhat.com  Fri Jul 31 05:38:54 2020
From: sacharya at redhat.com (Shwetha Acharya)
Date: Fri, 31 Jul 2020 11:08:54 +0530
Subject: [Gluster-users] [Gluster-devel] Announcing Gluster release 7.7
In-Reply-To: <CAD+dzQep-zAiEW3GpKgF138SJ7EavM0t5Ck=A3+TVKdazSxrJA@mail.gmail.com>
References: <CAMX-mG2V9u79Pcg5tKv92dHSHikYuoL5PfUjfGHq825ZdvNz2Q@mail.gmail.com>
	<CAD+dzQep-zAiEW3GpKgF138SJ7EavM0t5Ck=A3+TVKdazSxrJA@mail.gmail.com>
Message-ID: <CAERh03rWRpp87LCJ9AEu+w3Qw6eTbFbWca69NhtPAMChSCvaVw@mail.gmail.com>

Hi Artem,

As per current Tentative plans for community packages
<https://github.com/gluster/glusterdocs/blob/master/docs/Install-Guide/Community_Packages.md>
we
are supporting Leap15.2 only.

Regards,
Shwetha

On Fri, Jul 31, 2020 at 1:03 AM Artem Russakovskii <archon810 at gmail.com>
wrote:

> Hi,
>
>
> https://download.opensuse.org/repositories/home:/glusterfs:/Leap15.1-7/openSUSE_Leap_15.1/x86_64/
> is still missing 7.7. Is there an ETA please?
>
> Thanks.
>
>
> Sincerely,
> Artem
>
> --
> Founder, Android Police <http://www.androidpolice.com>, APK Mirror
> <http://www.apkmirror.com/>, Illogical Robot LLC
> beerpla.net | @ArtemR <http://twitter.com/ArtemR>
>
>
> On Wed, Jul 22, 2020 at 9:27 AM Rinku Kothiya <rkothiya at redhat.com> wrote:
>
>> Hi,
>>
>> The Gluster community is pleased to announce the release of Gluster7.7
>> (packages available at [1]).
>> Release notes for the release can be found at [2].
>>
>> Major changes, features and limitations addressed in this release:
>> None
>>
>> Please Note: Some of the packages are unavailable and we are working on
>> it. We will release them soon.
>>
>> Thanks,
>> Gluster community
>>
>> References:
>>
>> [1] Packages for 7.7:
>> https://download.gluster.org/pub/gluster/glusterfs/7/7.7/
>>
>> [2] Release notes for 7.7:
>> https://docs.gluster.org/en/latest/release-notes/7.7/
>> ________
>>
>>
>>
>> Community Meeting Calendar:
>>
>> Schedule -
>> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
>> Bridge: https://bluejeans.com/441850968
>>
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
> ________
>
>
>
> Community Meeting Calendar:
>
> Schedule -
> Every 2nd and 4th Tuesday at 14:30 IST / 09:00 UTC
> Bridge: https://bluejeans.com/441850968
>
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20200731/831b9ab8/attachment.html>

From g.danti at assyoma.it  Fri Jul 31 10:18:24 2020
From: g.danti at assyoma.it (Gionatan Danti)
Date: Fri, 31 Jul 2020 12:18:24 +0200
Subject: [Gluster-users] GlusterFS over multiples HDD
In-Reply-To: <CAOKSTBsWsDu-C564eOUjNzA+in-Nj5A8u4M2x7ucMZO_LUcZbw@mail.gmail.com>
References: <CAOKSTBvLqaB0JAA4JPbVO+fUz-ZyVHcsEvFWB6ZiX4vdZggFRw@mail.gmail.com>
	<A113C544-C332-462F-91B9-E99D93F8C951@yahoo.com>
	<CAOKSTBtWrKHDrMaR8_NLkgFXgqkDo9xfMRWUV25WzFspeYeLnw@mail.gmail.com>
	<4B36B235-A79E-4C09-BE33-9F61BEAB8D14@yahoo.com>
	<CAOKSTBszmeMpnWc90rQEiy2awZgj9_M2x_Q4ZpsNK4h62ss2qQ@mail.gmail.com>
	<CAOKSTBsWsDu-C564eOUjNzA+in-Nj5A8u4M2x7ucMZO_LUcZbw@mail.gmail.com>
Message-ID: <67862aca73832294f60eb7f83ed3c5f1@assyoma.it>

Il 2020-07-30 15:08 Gilberto Nunes ha scritto:
> I meant, if you power off the server, pull off 1 disk, and then power
> on we get system errors....

Hi, you are probably hitting some variant of this bug, rather than see 
LVM crashing: https://bugzilla.redhat.com/show_bug.cgi?id=1701504

If not, please write about your issue on the linux lvm mailing list.
Thanks.

-- 
Danti Gionatan
Supporto Tecnico
Assyoma S.r.l. - www.assyoma.it
email: g.danti at assyoma.it - info at assyoma.it
GPG public key ID: FF5F32A8