From jenkins at build.gluster.org  Mon Apr  1 01:45:03 2019
From: jenkins at build.gluster.org (jenkins at build.gluster.org)
Date: Mon, 1 Apr 2019 01:45:03 +0000 (UTC)
Subject: [Gluster-devel] Weekly Untriaged Bugs
Message-ID: <1758450900.40.1554083103600.JavaMail.jenkins@jenkins-el7.rht.gluster.org>

[...truncated 6 lines...]
https://bugzilla.redhat.com/1688226 / core: Brick Still Died After Restart Glusterd & Glusterfsd Services
https://bugzilla.redhat.com/1691833 / core: Client sends 128KByte network packet for 0 length file copy
https://bugzilla.redhat.com/1685023 / core: FD processes for larger files are not closed soon after FOP finished
https://bugzilla.redhat.com/1686396 / core: ls and rm run on contents of same directory from a single mount point results in ENOENT errors
https://bugzilla.redhat.com/1689981 / geo-replication: OSError: [Errno 1] Operation not permitted - failing with socket files?
https://bugzilla.redhat.com/1694139 / glusterd: Error waiting for job 'heketi-storage-copy-job' to complete on one-node k3s deployment.
https://bugzilla.redhat.com/1694010 / glusterd: peer gets disconnected during a rolling upgrade.
https://bugzilla.redhat.com/1690254 / glusterd: Volume create fails with "Commit failed" message if volumes is created using 3 nodes with glusterd restarts on 4th node.
https://bugzilla.redhat.com/1690753 / glusterd: Volume stop when quorum not met is successful
https://bugzilla.redhat.com/1686353 / libgfapi: flooding of "dict is NULL" logging
https://bugzilla.redhat.com/1687063 / locks: glusterd :symbol lookup error: undefined symbol :use_spinlocks
https://bugzilla.redhat.com/1690454 / posix-acl: mount-shared-storage.sh does not implement mount options
https://bugzilla.redhat.com/1691617 / project-infrastructure: clang-scan tests are failing nightly.
https://bugzilla.redhat.com/1691357 / project-infrastructure: core archive link from regression jobs throw not found error
https://bugzilla.redhat.com/1692349 / project-infrastructure: gluster-csi-containers job is failing
https://bugzilla.redhat.com/1685813 / project-infrastructure: Not able to run centos-regression getting exception error
https://bugzilla.redhat.com/1693385 / project-infrastructure: request to change the version of fedora in fedora-smoke-job
https://bugzilla.redhat.com/1693295 / project-infrastructure: rpc.statd not started on builder204.aws.gluster.org
https://bugzilla.redhat.com/1691789 / project-infrastructure: rpc-statd service stops on AWS builders
https://bugzilla.redhat.com/1694291 / project-infrastructure: Smoke test build artifacts do not contain gluster logs
https://bugzilla.redhat.com/1686461 / quota: Quotad.log filled with 0-dict is not sent on wire [Invalid argument] messages
https://bugzilla.redhat.com/1693184 / replicate: A brick process(glusterfsd) died with 'memory violation'
[...truncated 2 lines...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 2887 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190401/c3a383da/attachment.obj>

From pkarampu at redhat.com  Mon Apr  1 04:32:19 2019
From: pkarampu at redhat.com (Pranith Kumar Karampuri)
Date: Mon, 1 Apr 2019 10:02:19 +0530
Subject: [Gluster-devel] Issue with posix locks
In-Reply-To: <7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
References: <CALtvDK4W3DLvP64OdJb8wGw11d1X8sH-4naCi31cs0Nf0GdxCg@mail.gmail.com>
	<7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
Message-ID: <CAOgeEnaGgc5M5pHbNwEoanPafx1C5kpdcJB3o0fsH-BRRi3H6g@mail.gmail.com>

On Sun, Mar 31, 2019 at 11:29 PM Soumya Koduri <skoduri at redhat.com> wrote:

>
>
> On 3/29/19 11:55 PM, Xavi Hernandez wrote:
> > Hi all,
> >
> > there is one potential problem with posix locks when used in a
> > replicated or dispersed volume.
> >
> > Some background:
> >
> > Posix locks allow any process to lock a region of a file multiple times,
> > but a single unlock on a given region will release all previous locks.
> > Locked regions can be different for each lock request and they can
> > overlap. The resulting lock will cover the union of all locked regions.
> > A single unlock (the region doesn't necessarily need to match any of the
> > ranges used for locking) will create a "hole" in the currently locked
> > region, independently of how many times a lock request covered that
> region.
> >
> > For this reason, the locks xlator simply combines the locked regions
> > that are requested, but it doesn't track each individual lock range.
> >
> > Under normal circumstances this works fine. But there are some cases
> > where this behavior is not sufficient. For example, suppose we have a
> > replica 3 volume with quorum = 2. Given the special nature of posix
> > locks, AFR sends the lock request sequentially to each one of the
> > bricks, to avoid that conflicting lock requests from other clients could
> > require to unlock an already locked region on the client that has not
> > got enough successful locks (i.e. quorum). An unlock here not only would
> > cancel the current lock request. It would also cancel any previously
> > acquired lock.
> >
>
> I may not have fully understood, please correct me. AFAIU, lk xlator
> merges locks only if both the lk-owner and the client opaque matches.
>
> In the case which you have mentioned above, considering clientA acquired
> locks on majority of quorum (say nodeA and nodeB) and clientB on nodeC
> alone- clientB now has to unlock/cancel the lock it acquired on nodeC.
>
> You are saying the it could pose a problem if there were already
> successful locks taken by clientB for the same region which would get
> unlocked by this particular unlock request..right?
>
> Assuming the previous locks acquired by clientB are shared (otherwise
> clientA wouldn't have got granted lock for the same region on nodeA &
> nodeB), they would still hold true on nodeA & nodeB  as the unlock
> request was sent to only nodeC. Since the majority of quorum nodes still
> hold the locks by clientB, this isn't serious issue IMO.
>
> I haven't looked into heal part but would like to understand if this is
> really an issue in normal scenarios as well.
>

This is how I understood the code. Consider the following case:
Nodes A, B, C have locks with start and end offsets: 5-15 from mount-1 and
lock-range 2-3 from mount-2.
If mount-1 requests nonblocking lock with lock-range 1-7 and in parallel
lets say mount-2 issued unlock of 2-3 as well.

nodeA got unlock from mount-2 with range 2-3 then lock from mount-1 with
range 1-7, so the lock is granted and merged to give 1-15
nodeB got lock from mount-1 with range 1-7 before unlock of 2-3 which leads
to EAGAIN which will trigger unlocks on granted lock in mount-1 which will
end up doing unlock of 1-7 on nodeA leading to lock-range 8-15 instead of
the original 5-15 on nodeA. Whereas nodeB and nodeC will have range 5-15.

Let me know if my understanding is wrong.


> Thanks,
> Soumya
>
> > However, when something goes wrong (a brick dies during a lock request,
> > or there's a network partition or some other weird situation), it could
> > happen that even using sequential locking, only one brick succeeds the
> > lock request. In this case, AFR should cancel the previous lock (and it
> > does), but this also cancels any previously acquired lock on that
> > region, which is not good.
> >
> > A similar thing can happen if we try to recover (heal) posix locks that
> > were active after a brick has been disconnected (for any reason) and
> > then reconnected.
> >
> > To fix all these situations we need to change the way posix locks are
> > managed by locks xlator. One possibility would be to embed the lock
> > request inside an inode transaction using inodelk. Since inodelks do not
> > suffer this problem, the follwing posix lock could be sent safely.
> > However this implies an additional network request, which could cause
> > some performance impact. Eager-locking could minimize the impact in some
> > cases. However this approach won't work for lock recovery after a
> > disconnect.
> >
> > Another possibility is to send a special partial posix lock request
> > which won't be immediately merged with already existing locks once
> > granted. An additional confirmation request of the partial posix lock
> > will be required to fully grant the current lock and merge it with the
> > existing ones. This requires a new network request, which will add
> > latency, and makes everything more complex since there would be more
> > combinations of states in which something could fail.
> >
> > So I think one possible solution would be the following:
> >
> > 1. Keep each posix lock as an independent object in locks xlator. This
> > will make it possible to "invalidate" any already granted lock without
> > affecting already established locks.
> >
> > 2. Additionally, we'll keep a sorted list of non-overlapping segments of
> > locked regions. And we'll count, for each region, how many locks are
> > referencing it. One lock can reference multiple segments, and each
> > segment can be referenced by multiple locks.
> >
> > 3. An additional lock request that overlaps with an existing segment,
> > can cause this segment to be split to satisfy the non-overlapping
> property.
> >
> > 4. When an unlock request is received, all segments intersecting with
> > the region are eliminated (it may require some segment splits on the
> > edges), and the unlocked region is subtracted from each lock associated
> > to the segment. If a lock gets an empty region, it's removed.
> >
> > 5. We'll create a special "remove lock" request that doesn't unlock a
> > region but removes an already granted lock. This will decrease the
> > number of references to each of the segments this lock was covering. If
> > some segment reaches 0, it's removed. Otherwise it remains there. This
> > special request will only be used internally to cancel already acquired
> > locks that cannot be fully granted due to quorum issues or any other
> > problem.
> >
> > In some weird cases, the list of segments can be huge (many locks
> > overlapping only on a single byte, so each segment represents only one
> > byte). We can try to find some smarter structure that minimizes this
> > problem or limit the number of segments (for example returning ENOLCK
> > when there are too many).
> >
> > What do you think ?
> >
> > Xavi
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190401/a2dfec44/attachment-0001.html>

From xhernandez at redhat.com  Mon Apr  1 07:54:22 2019
From: xhernandez at redhat.com (Xavi Hernandez)
Date: Mon, 1 Apr 2019 09:54:22 +0200
Subject: [Gluster-devel] Issue with posix locks
In-Reply-To: <7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
References: <CALtvDK4W3DLvP64OdJb8wGw11d1X8sH-4naCi31cs0Nf0GdxCg@mail.gmail.com>
	<7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
Message-ID: <CALtvDK5rO4y-OZhEpkfCMLzYyXKaXkLk1o3r+mGe2Hav-978Lw@mail.gmail.com>

On Sun, Mar 31, 2019 at 7:59 PM Soumya Koduri <skoduri at redhat.com> wrote:

>
>
> On 3/29/19 11:55 PM, Xavi Hernandez wrote:
> > Hi all,
> >
> > there is one potential problem with posix locks when used in a
> > replicated or dispersed volume.
> >
> > Some background:
> >
> > Posix locks allow any process to lock a region of a file multiple times,
> > but a single unlock on a given region will release all previous locks.
> > Locked regions can be different for each lock request and they can
> > overlap. The resulting lock will cover the union of all locked regions.
> > A single unlock (the region doesn't necessarily need to match any of the
> > ranges used for locking) will create a "hole" in the currently locked
> > region, independently of how many times a lock request covered that
> region.
> >
> > For this reason, the locks xlator simply combines the locked regions
> > that are requested, but it doesn't track each individual lock range.
> >
> > Under normal circumstances this works fine. But there are some cases
> > where this behavior is not sufficient. For example, suppose we have a
> > replica 3 volume with quorum = 2. Given the special nature of posix
> > locks, AFR sends the lock request sequentially to each one of the
> > bricks, to avoid that conflicting lock requests from other clients could
> > require to unlock an already locked region on the client that has not
> > got enough successful locks (i.e. quorum). An unlock here not only would
> > cancel the current lock request. It would also cancel any previously
> > acquired lock.
> >
>
> I may not have fully understood, please correct me. AFAIU, lk xlator
> merges locks only if both the lk-owner and the client opaque matches.
>
> In the case which you have mentioned above, considering clientA acquired
> locks on majority of quorum (say nodeA and nodeB) and clientB on nodeC
> alone- clientB now has to unlock/cancel the lock it acquired on nodeC.
>
> You are saying the it could pose a problem if there were already
> successful locks taken by clientB for the same region which would get
> unlocked by this particular unlock request..right?
>

Yes


>
> Assuming the previous locks acquired by clientB are shared (otherwise
> clientA wouldn't have got granted lock for the same region on nodeA &
> nodeB), they would still hold true on nodeA & nodeB  as the unlock
> request was sent to only nodeC. Since the majority of quorum nodes still
> hold the locks by clientB, this isn't serious issue IMO.
>

Partially true. But if one of nodeA or nodeB dies or gets disconnected,
there won't be any majority of bricks with correct locks, even though there
are still 2 alive bricks. At this point, another client could successfully
acquire a lock that, in theory, is already acquired by another client.


> I haven't looked into heal part but would like to understand if this is
> really an issue in normal scenarios as well.
>

If we consider that a brick disconnection is a normal scenario (which I
think it should be on a large scale distributed file system), then this
issue exists. But even without brick disconnections we can get incorrect
results, as Pranith has just explained.

Xavi


>
> Thanks,
> Soumya
>
> > However, when something goes wrong (a brick dies during a lock request,
> > or there's a network partition or some other weird situation), it could
> > happen that even using sequential locking, only one brick succeeds the
> > lock request. In this case, AFR should cancel the previous lock (and it
> > does), but this also cancels any previously acquired lock on that
> > region, which is not good.
> >
> > A similar thing can happen if we try to recover (heal) posix locks that
> > were active after a brick has been disconnected (for any reason) and
> > then reconnected.
> >
> > To fix all these situations we need to change the way posix locks are
> > managed by locks xlator. One possibility would be to embed the lock
> > request inside an inode transaction using inodelk. Since inodelks do not
> > suffer this problem, the follwing posix lock could be sent safely.
> > However this implies an additional network request, which could cause
> > some performance impact. Eager-locking could minimize the impact in some
> > cases. However this approach won't work for lock recovery after a
> > disconnect.
> >
> > Another possibility is to send a special partial posix lock request
> > which won't be immediately merged with already existing locks once
> > granted. An additional confirmation request of the partial posix lock
> > will be required to fully grant the current lock and merge it with the
> > existing ones. This requires a new network request, which will add
> > latency, and makes everything more complex since there would be more
> > combinations of states in which something could fail.
> >
> > So I think one possible solution would be the following:
> >
> > 1. Keep each posix lock as an independent object in locks xlator. This
> > will make it possible to "invalidate" any already granted lock without
> > affecting already established locks.
> >
> > 2. Additionally, we'll keep a sorted list of non-overlapping segments of
> > locked regions. And we'll count, for each region, how many locks are
> > referencing it. One lock can reference multiple segments, and each
> > segment can be referenced by multiple locks.
> >
> > 3. An additional lock request that overlaps with an existing segment,
> > can cause this segment to be split to satisfy the non-overlapping
> property.
> >
> > 4. When an unlock request is received, all segments intersecting with
> > the region are eliminated (it may require some segment splits on the
> > edges), and the unlocked region is subtracted from each lock associated
> > to the segment. If a lock gets an empty region, it's removed.
> >
> > 5. We'll create a special "remove lock" request that doesn't unlock a
> > region but removes an already granted lock. This will decrease the
> > number of references to each of the segments this lock was covering. If
> > some segment reaches 0, it's removed. Otherwise it remains there. This
> > special request will only be used internally to cancel already acquired
> > locks that cannot be fully granted due to quorum issues or any other
> > problem.
> >
> > In some weird cases, the list of segments can be huge (many locks
> > overlapping only on a single byte, so each segment represents only one
> > byte). We can try to find some smarter structure that minimizes this
> > problem or limit the number of segments (for example returning ENOLCK
> > when there are too many).
> >
> > What do you think ?
> >
> > Xavi
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190401/f73efbfb/attachment.html>

From skoduri at redhat.com  Mon Apr  1 08:15:21 2019
From: skoduri at redhat.com (Soumya Koduri)
Date: Mon, 1 Apr 2019 13:45:21 +0530
Subject: [Gluster-devel] Issue with posix locks
In-Reply-To: <CAOgeEnaGgc5M5pHbNwEoanPafx1C5kpdcJB3o0fsH-BRRi3H6g@mail.gmail.com>
References: <CALtvDK4W3DLvP64OdJb8wGw11d1X8sH-4naCi31cs0Nf0GdxCg@mail.gmail.com>
	<7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
	<CAOgeEnaGgc5M5pHbNwEoanPafx1C5kpdcJB3o0fsH-BRRi3H6g@mail.gmail.com>
Message-ID: <6c12597f-634d-8d7e-5c47-2158f6989299@redhat.com>



On 4/1/19 10:02 AM, Pranith Kumar Karampuri wrote:
> 
> 
> On Sun, Mar 31, 2019 at 11:29 PM Soumya Koduri <skoduri at redhat.com 
> <mailto:skoduri at redhat.com>> wrote:
> 
> 
> 
>     On 3/29/19 11:55 PM, Xavi Hernandez wrote:
>      > Hi all,
>      >
>      > there is one potential problem with posix locks when used in a
>      > replicated or dispersed volume.
>      >
>      > Some background:
>      >
>      > Posix locks allow any process to lock a region of a file multiple
>     times,
>      > but a single unlock on a given region will release all previous
>     locks.
>      > Locked regions can be different for each lock request and they can
>      > overlap. The resulting lock will cover the union of all locked
>     regions.
>      > A single unlock (the region doesn't necessarily need to match any
>     of the
>      > ranges used for locking) will create a "hole" in the currently
>     locked
>      > region, independently of how many times a lock request covered
>     that region.
>      >
>      > For this reason, the locks xlator simply combines the locked regions
>      > that are requested, but it doesn't track each individual lock range.
>      >
>      > Under normal circumstances this works fine. But there are some cases
>      > where this behavior is not sufficient. For example, suppose we
>     have a
>      > replica 3 volume with quorum = 2. Given the special nature of posix
>      > locks, AFR sends the lock request sequentially to each one of the
>      > bricks, to avoid that conflicting lock requests from other
>     clients could
>      > require to unlock an already locked region on the client that has
>     not
>      > got enough successful locks (i.e. quorum). An unlock here not
>     only would
>      > cancel the current lock request. It would also cancel any previously
>      > acquired lock.
>      >
> 
>     I may not have fully understood, please correct me. AFAIU, lk xlator
>     merges locks only if both the lk-owner and the client opaque matches.
> 
>     In the case which you have mentioned above, considering clientA
>     acquired
>     locks on majority of quorum (say nodeA and nodeB) and clientB on nodeC
>     alone- clientB now has to unlock/cancel the lock it acquired on nodeC.
> 
>     You are saying the it could pose a problem if there were already
>     successful locks taken by clientB for the same region which would get
>     unlocked by this particular unlock request..right?
> 
>     Assuming the previous locks acquired by clientB are shared (otherwise
>     clientA wouldn't have got granted lock for the same region on nodeA &
>     nodeB), they would still hold true on nodeA & nodeB? as the unlock
>     request was sent to only nodeC. Since the majority of quorum nodes
>     still
>     hold the locks by clientB, this isn't serious issue IMO.
> 
>     I haven't looked into heal part but would like to understand if this is
>     really an issue in normal scenarios as well.
> 
> 
> This is how I understood the code. Consider the following case:
> Nodes A, B, C have locks with start and end offsets: 5-15 from mount-1 
> and lock-range 2-3 from mount-2.
> If mount-1 requests nonblocking lock with lock-range 1-7 and in parallel 
> lets say mount-2 issued unlock of 2-3 as well.
> 
> nodeA got unlock from mount-2 with range 2-3 then lock from mount-1 with 
> range 1-7, so the lock is granted and merged to give 1-15
> nodeB got lock from mount-1 with range 1-7 before unlock of 2-3 which 
> leads to EAGAIN which will trigger unlocks on granted lock in mount-1 
> which will end up doing unlock of 1-7 on nodeA leading to lock-range 
> 8-15 instead of the original 5-15 on nodeA. Whereas nodeB and nodeC will 
> have range 5-15.
> 
> Let me know if my understanding is wrong.

Both of us mentioned the same points. So in the example you gave , 
mount-1 lost its previous lock on nodeA but majority of the quorum 
(nodeB and nodeC) still have the previous lock  (range: 5-15) intact. So 
this shouldn't ideally lead to any issues as other conflicting locks are 
blocked or failed by majority of the nodes (provided there are no brick 
dis/re-connects).

Wrt to brick disconnects/re-connects, if we can get in general lock 
healing (not getting into implementation details atm) support, that 
should take care of correcting lock range on nodeA as well right?

That said I am not suggesting that we should stick to existing behavior, 
just trying to get clarification to check if we can avoid any 
overhead/side-effects with maintaining multiple locks.

Thanks,
Soumya


> 
> 
>     Thanks,
>     Soumya
> 
>      > However, when something goes wrong (a brick dies during a lock
>     request,
>      > or there's a network partition or some other weird situation), it
>     could
>      > happen that even using sequential locking, only one brick
>     succeeds the
>      > lock request. In this case, AFR should cancel the previous lock
>     (and it
>      > does), but this also cancels any previously acquired lock on that
>      > region, which is not good.
>      >
>      > A similar thing can happen if we try to recover (heal) posix
>     locks that
>      > were active after a brick has been disconnected (for any reason) and
>      > then reconnected.
>      >
>      > To fix all these situations we need to change the way posix locks
>     are
>      > managed by locks xlator. One possibility would be to embed the lock
>      > request inside an inode transaction using inodelk. Since inodelks
>     do not
>      > suffer this problem, the follwing posix lock could be sent safely.
>      > However this implies an additional network request, which could
>     cause
>      > some performance impact. Eager-locking could minimize the impact
>     in some
>      > cases. However this approach won't work for lock recovery after a
>      > disconnect.
>      >
>      > Another possibility is to send a special partial posix lock request
>      > which won't be immediately merged with already existing locks once
>      > granted. An additional confirmation request of the partial posix
>     lock
>      > will be required to fully grant the current lock and merge it
>     with the
>      > existing ones. This requires a new network request, which will add
>      > latency, and makes everything more complex since there would be more
>      > combinations of states in which something could fail.
>      >
>      > So I think one possible solution would be the following:
>      >
>      > 1. Keep each posix lock as an independent object in locks xlator.
>     This
>      > will make it possible to "invalidate" any already granted lock
>     without
>      > affecting already established locks.
>      >
>      > 2. Additionally, we'll keep a sorted list of non-overlapping
>     segments of
>      > locked regions. And we'll count, for each region, how many locks are
>      > referencing it. One lock can reference multiple segments, and each
>      > segment can be referenced by multiple locks.
>      >
>      > 3. An additional lock request that overlaps with an existing
>     segment,
>      > can cause this segment to be split to satisfy the non-overlapping
>     property.
>      >
>      > 4. When an unlock request is received, all segments intersecting
>     with
>      > the region are eliminated (it may require some segment splits on the
>      > edges), and the unlocked region is subtracted from each lock
>     associated
>      > to the segment. If a lock gets an empty region, it's removed.
>      >
>      > 5. We'll create a special "remove lock" request that doesn't
>     unlock a
>      > region but removes an already granted lock. This will decrease the
>      > number of references to each of the segments this lock was
>     covering. If
>      > some segment reaches 0, it's removed. Otherwise it remains there.
>     This
>      > special request will only be used internally to cancel already
>     acquired
>      > locks that cannot be fully granted due to quorum issues or any other
>      > problem.
>      >
>      > In some weird cases, the list of segments can be huge (many locks
>      > overlapping only on a single byte, so each segment represents
>     only one
>      > byte). We can try to find some smarter structure that minimizes this
>      > problem or limit the number of segments (for example returning
>     ENOLCK
>      > when there are too many).
>      >
>      > What do you think ?
>      >
>      > Xavi
>      >
>      > _______________________________________________
>      > Gluster-devel mailing list
>      > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>      > https://lists.gluster.org/mailman/listinfo/gluster-devel
>      >
> 
> 
> 
> -- 
> Pranith

From xhernandez at redhat.com  Mon Apr  1 08:53:23 2019
From: xhernandez at redhat.com (Xavi Hernandez)
Date: Mon, 1 Apr 2019 10:53:23 +0200
Subject: [Gluster-devel] Issue with posix locks
In-Reply-To: <6c12597f-634d-8d7e-5c47-2158f6989299@redhat.com>
References: <CALtvDK4W3DLvP64OdJb8wGw11d1X8sH-4naCi31cs0Nf0GdxCg@mail.gmail.com>
	<7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
	<CAOgeEnaGgc5M5pHbNwEoanPafx1C5kpdcJB3o0fsH-BRRi3H6g@mail.gmail.com>
	<6c12597f-634d-8d7e-5c47-2158f6989299@redhat.com>
Message-ID: <CALtvDK6PFdYfTDa_f0wE+Lzt5A4sJHFHQe1RJ4Lb2Z3Dc3J6+A@mail.gmail.com>

On Mon, Apr 1, 2019 at 10:15 AM Soumya Koduri <skoduri at redhat.com> wrote:

>
>
> On 4/1/19 10:02 AM, Pranith Kumar Karampuri wrote:
> >
> >
> > On Sun, Mar 31, 2019 at 11:29 PM Soumya Koduri <skoduri at redhat.com
> > <mailto:skoduri at redhat.com>> wrote:
> >
> >
> >
> >     On 3/29/19 11:55 PM, Xavi Hernandez wrote:
> >      > Hi all,
> >      >
> >      > there is one potential problem with posix locks when used in a
> >      > replicated or dispersed volume.
> >      >
> >      > Some background:
> >      >
> >      > Posix locks allow any process to lock a region of a file multiple
> >     times,
> >      > but a single unlock on a given region will release all previous
> >     locks.
> >      > Locked regions can be different for each lock request and they can
> >      > overlap. The resulting lock will cover the union of all locked
> >     regions.
> >      > A single unlock (the region doesn't necessarily need to match any
> >     of the
> >      > ranges used for locking) will create a "hole" in the currently
> >     locked
> >      > region, independently of how many times a lock request covered
> >     that region.
> >      >
> >      > For this reason, the locks xlator simply combines the locked
> regions
> >      > that are requested, but it doesn't track each individual lock
> range.
> >      >
> >      > Under normal circumstances this works fine. But there are some
> cases
> >      > where this behavior is not sufficient. For example, suppose we
> >     have a
> >      > replica 3 volume with quorum = 2. Given the special nature of
> posix
> >      > locks, AFR sends the lock request sequentially to each one of the
> >      > bricks, to avoid that conflicting lock requests from other
> >     clients could
> >      > require to unlock an already locked region on the client that has
> >     not
> >      > got enough successful locks (i.e. quorum). An unlock here not
> >     only would
> >      > cancel the current lock request. It would also cancel any
> previously
> >      > acquired lock.
> >      >
> >
> >     I may not have fully understood, please correct me. AFAIU, lk xlator
> >     merges locks only if both the lk-owner and the client opaque matches.
> >
> >     In the case which you have mentioned above, considering clientA
> >     acquired
> >     locks on majority of quorum (say nodeA and nodeB) and clientB on
> nodeC
> >     alone- clientB now has to unlock/cancel the lock it acquired on
> nodeC.
> >
> >     You are saying the it could pose a problem if there were already
> >     successful locks taken by clientB for the same region which would get
> >     unlocked by this particular unlock request..right?
> >
> >     Assuming the previous locks acquired by clientB are shared (otherwise
> >     clientA wouldn't have got granted lock for the same region on nodeA &
> >     nodeB), they would still hold true on nodeA & nodeB  as the unlock
> >     request was sent to only nodeC. Since the majority of quorum nodes
> >     still
> >     hold the locks by clientB, this isn't serious issue IMO.
> >
> >     I haven't looked into heal part but would like to understand if this
> is
> >     really an issue in normal scenarios as well.
> >
> >
> > This is how I understood the code. Consider the following case:
> > Nodes A, B, C have locks with start and end offsets: 5-15 from mount-1
> > and lock-range 2-3 from mount-2.
> > If mount-1 requests nonblocking lock with lock-range 1-7 and in parallel
> > lets say mount-2 issued unlock of 2-3 as well.
> >
> > nodeA got unlock from mount-2 with range 2-3 then lock from mount-1 with
> > range 1-7, so the lock is granted and merged to give 1-15
> > nodeB got lock from mount-1 with range 1-7 before unlock of 2-3 which
> > leads to EAGAIN which will trigger unlocks on granted lock in mount-1
> > which will end up doing unlock of 1-7 on nodeA leading to lock-range
> > 8-15 instead of the original 5-15 on nodeA. Whereas nodeB and nodeC will
> > have range 5-15.
> >
> > Let me know if my understanding is wrong.
>
> Both of us mentioned the same points. So in the example you gave ,
> mount-1 lost its previous lock on nodeA but majority of the quorum
> (nodeB and nodeC) still have the previous lock  (range: 5-15) intact. So
> this shouldn't ideally lead to any issues as other conflicting locks are
> blocked or failed by majority of the nodes (provided there are no brick
> dis/re-connects).
>

But brick disconnects will happen (upgrades, disk failures, server
maintenance, ...). Anyway, even without brick disconnects, in the previous
example we have nodeA with range 8-15, and nodes B and C with range 5-15.
If another lock from mount-2 comes for range 5-7, it will succeed on nodeA,
but it will block on nodeB. At this point, mount-1 could attempt a lock on
same range. It will block on nodeA, so we have a deadlock.

In general, having discrepancies between bricks is not good because sooner
or later it will cause some bad inconsistency.


> Wrt to brick disconnects/re-connects, if we can get in general lock
> healing (not getting into implementation details atm) support, that
> should take care of correcting lock range on nodeA as well right?
>

The problem we have seen is that to be able to correctly heal currently
acquired locks on brick reconnect, there are cases where we need to release
a lock that has already been granted (because the current owner doesn't
have enough quorum and a just recovered connection tries to claim/heal it).
In this case we need to deal with locks that have already been merged, but
without interfering with other existing locks that already have quorum.


> That said I am not suggesting that we should stick to existing behavior,
> just trying to get clarification to check if we can avoid any
> overhead/side-effects with maintaining multiple locks.
>

Right now is the only way we have found to provide a correct solution both
for some cases of concurrent lock/unlock requests, and lock healing.

Regards,

Xavi


> Thanks,
> Soumya
>
>
> >
> >
> >     Thanks,
> >     Soumya
> >
> >      > However, when something goes wrong (a brick dies during a lock
> >     request,
> >      > or there's a network partition or some other weird situation), it
> >     could
> >      > happen that even using sequential locking, only one brick
> >     succeeds the
> >      > lock request. In this case, AFR should cancel the previous lock
> >     (and it
> >      > does), but this also cancels any previously acquired lock on that
> >      > region, which is not good.
> >      >
> >      > A similar thing can happen if we try to recover (heal) posix
> >     locks that
> >      > were active after a brick has been disconnected (for any reason)
> and
> >      > then reconnected.
> >      >
> >      > To fix all these situations we need to change the way posix locks
> >     are
> >      > managed by locks xlator. One possibility would be to embed the
> lock
> >      > request inside an inode transaction using inodelk. Since inodelks
> >     do not
> >      > suffer this problem, the follwing posix lock could be sent safely.
> >      > However this implies an additional network request, which could
> >     cause
> >      > some performance impact. Eager-locking could minimize the impact
> >     in some
> >      > cases. However this approach won't work for lock recovery after a
> >      > disconnect.
> >      >
> >      > Another possibility is to send a special partial posix lock
> request
> >      > which won't be immediately merged with already existing locks once
> >      > granted. An additional confirmation request of the partial posix
> >     lock
> >      > will be required to fully grant the current lock and merge it
> >     with the
> >      > existing ones. This requires a new network request, which will add
> >      > latency, and makes everything more complex since there would be
> more
> >      > combinations of states in which something could fail.
> >      >
> >      > So I think one possible solution would be the following:
> >      >
> >      > 1. Keep each posix lock as an independent object in locks xlator.
> >     This
> >      > will make it possible to "invalidate" any already granted lock
> >     without
> >      > affecting already established locks.
> >      >
> >      > 2. Additionally, we'll keep a sorted list of non-overlapping
> >     segments of
> >      > locked regions. And we'll count, for each region, how many locks
> are
> >      > referencing it. One lock can reference multiple segments, and each
> >      > segment can be referenced by multiple locks.
> >      >
> >      > 3. An additional lock request that overlaps with an existing
> >     segment,
> >      > can cause this segment to be split to satisfy the non-overlapping
> >     property.
> >      >
> >      > 4. When an unlock request is received, all segments intersecting
> >     with
> >      > the region are eliminated (it may require some segment splits on
> the
> >      > edges), and the unlocked region is subtracted from each lock
> >     associated
> >      > to the segment. If a lock gets an empty region, it's removed.
> >      >
> >      > 5. We'll create a special "remove lock" request that doesn't
> >     unlock a
> >      > region but removes an already granted lock. This will decrease the
> >      > number of references to each of the segments this lock was
> >     covering. If
> >      > some segment reaches 0, it's removed. Otherwise it remains there.
> >     This
> >      > special request will only be used internally to cancel already
> >     acquired
> >      > locks that cannot be fully granted due to quorum issues or any
> other
> >      > problem.
> >      >
> >      > In some weird cases, the list of segments can be huge (many locks
> >      > overlapping only on a single byte, so each segment represents
> >     only one
> >      > byte). We can try to find some smarter structure that minimizes
> this
> >      > problem or limit the number of segments (for example returning
> >     ENOLCK
> >      > when there are too many).
> >      >
> >      > What do you think ?
> >      >
> >      > Xavi
> >      >
> >      > _______________________________________________
> >      > Gluster-devel mailing list
> >      > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
> >      > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >      >
> >
> >
> >
> > --
> > Pranith
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190401/eb9ace4d/attachment-0001.html>

From skoduri at redhat.com  Mon Apr  1 10:10:21 2019
From: skoduri at redhat.com (Soumya Koduri)
Date: Mon, 1 Apr 2019 15:40:21 +0530
Subject: [Gluster-devel] Issue with posix locks
In-Reply-To: <CALtvDK6PFdYfTDa_f0wE+Lzt5A4sJHFHQe1RJ4Lb2Z3Dc3J6+A@mail.gmail.com>
References: <CALtvDK4W3DLvP64OdJb8wGw11d1X8sH-4naCi31cs0Nf0GdxCg@mail.gmail.com>
	<7a1adf3a-bdc9-a1ef-6f3c-e69d17c37adc@redhat.com>
	<CAOgeEnaGgc5M5pHbNwEoanPafx1C5kpdcJB3o0fsH-BRRi3H6g@mail.gmail.com>
	<6c12597f-634d-8d7e-5c47-2158f6989299@redhat.com>
	<CALtvDK6PFdYfTDa_f0wE+Lzt5A4sJHFHQe1RJ4Lb2Z3Dc3J6+A@mail.gmail.com>
Message-ID: <4d3d57ca-64a7-ed76-f98d-01d8dfb76c39@redhat.com>



On 4/1/19 2:23 PM, Xavi Hernandez wrote:
> On Mon, Apr 1, 2019 at 10:15 AM Soumya Koduri <skoduri at redhat.com 
> <mailto:skoduri at redhat.com>> wrote:
> 
> 
> 
>     On 4/1/19 10:02 AM, Pranith Kumar Karampuri wrote:
>      >
>      >
>      > On Sun, Mar 31, 2019 at 11:29 PM Soumya Koduri
>     <skoduri at redhat.com <mailto:skoduri at redhat.com>
>      > <mailto:skoduri at redhat.com <mailto:skoduri at redhat.com>>> wrote:
>      >
>      >
>      >
>      >? ? ?On 3/29/19 11:55 PM, Xavi Hernandez wrote:
>      >? ? ? > Hi all,
>      >? ? ? >
>      >? ? ? > there is one potential problem with posix locks when used in a
>      >? ? ? > replicated or dispersed volume.
>      >? ? ? >
>      >? ? ? > Some background:
>      >? ? ? >
>      >? ? ? > Posix locks allow any process to lock a region of a file
>     multiple
>      >? ? ?times,
>      >? ? ? > but a single unlock on a given region will release all
>     previous
>      >? ? ?locks.
>      >? ? ? > Locked regions can be different for each lock request and
>     they can
>      >? ? ? > overlap. The resulting lock will cover the union of all locked
>      >? ? ?regions.
>      >? ? ? > A single unlock (the region doesn't necessarily need to
>     match any
>      >? ? ?of the
>      >? ? ? > ranges used for locking) will create a "hole" in the currently
>      >? ? ?locked
>      >? ? ? > region, independently of how many times a lock request covered
>      >? ? ?that region.
>      >? ? ? >
>      >? ? ? > For this reason, the locks xlator simply combines the
>     locked regions
>      >? ? ? > that are requested, but it doesn't track each individual
>     lock range.
>      >? ? ? >
>      >? ? ? > Under normal circumstances this works fine. But there are
>     some cases
>      >? ? ? > where this behavior is not sufficient. For example, suppose we
>      >? ? ?have a
>      >? ? ? > replica 3 volume with quorum = 2. Given the special nature
>     of posix
>      >? ? ? > locks, AFR sends the lock request sequentially to each one
>     of the
>      >? ? ? > bricks, to avoid that conflicting lock requests from other
>      >? ? ?clients could
>      >? ? ? > require to unlock an already locked region on the client
>     that has
>      >? ? ?not
>      >? ? ? > got enough successful locks (i.e. quorum). An unlock here not
>      >? ? ?only would
>      >? ? ? > cancel the current lock request. It would also cancel any
>     previously
>      >? ? ? > acquired lock.
>      >? ? ? >
>      >
>      >? ? ?I may not have fully understood, please correct me. AFAIU, lk
>     xlator
>      >? ? ?merges locks only if both the lk-owner and the client opaque
>     matches.
>      >
>      >? ? ?In the case which you have mentioned above, considering clientA
>      >? ? ?acquired
>      >? ? ?locks on majority of quorum (say nodeA and nodeB) and clientB
>     on nodeC
>      >? ? ?alone- clientB now has to unlock/cancel the lock it acquired
>     on nodeC.
>      >
>      >? ? ?You are saying the it could pose a problem if there were already
>      >? ? ?successful locks taken by clientB for the same region which
>     would get
>      >? ? ?unlocked by this particular unlock request..right?
>      >
>      >? ? ?Assuming the previous locks acquired by clientB are shared
>     (otherwise
>      >? ? ?clientA wouldn't have got granted lock for the same region on
>     nodeA &
>      >? ? ?nodeB), they would still hold true on nodeA & nodeB? as the
>     unlock
>      >? ? ?request was sent to only nodeC. Since the majority of quorum
>     nodes
>      >? ? ?still
>      >? ? ?hold the locks by clientB, this isn't serious issue IMO.
>      >
>      >? ? ?I haven't looked into heal part but would like to understand
>     if this is
>      >? ? ?really an issue in normal scenarios as well.
>      >
>      >
>      > This is how I understood the code. Consider the following case:
>      > Nodes A, B, C have locks with start and end offsets: 5-15 from
>     mount-1
>      > and lock-range 2-3 from mount-2.
>      > If mount-1 requests nonblocking lock with lock-range 1-7 and in
>     parallel
>      > lets say mount-2 issued unlock of 2-3 as well.
>      >
>      > nodeA got unlock from mount-2 with range 2-3 then lock from
>     mount-1 with
>      > range 1-7, so the lock is granted and merged to give 1-15
>      > nodeB got lock from mount-1 with range 1-7 before unlock of 2-3
>     which
>      > leads to EAGAIN which will trigger unlocks on granted lock in
>     mount-1
>      > which will end up doing unlock of 1-7 on nodeA leading to lock-range
>      > 8-15 instead of the original 5-15 on nodeA. Whereas nodeB and
>     nodeC will
>      > have range 5-15.
>      >
>      > Let me know if my understanding is wrong.
> 
>     Both of us mentioned the same points. So in the example you gave ,
>     mount-1 lost its previous lock on nodeA but majority of the quorum
>     (nodeB and nodeC) still have the previous lock? (range: 5-15)
>     intact. So
>     this shouldn't ideally lead to any issues as other conflicting locks
>     are
>     blocked or failed by majority of the nodes (provided there are no brick
>     dis/re-connects).
> 
> 
> But brick disconnects will happen (upgrades, disk failures, server 
> maintenance, ...). Anyway, even without brick disconnects, in the 
> previous example we have nodeA with range 8-15, and nodes B and C with 
> range 5-15. If another lock from mount-2 comes for range 5-7, it will 
> succeed on nodeA, but it will block on nodeB. At this point, mount-1 
> could attempt a lock on same range. It will block on nodeA, so we have a 
> deadlock.
> 
> In general, having discrepancies between bricks is not good because 
> sooner or later it will cause some bad inconsistency.
> 
> 
>     Wrt to brick disconnects/re-connects, if we can get in general lock
>     healing (not getting into implementation details atm) support, that
>     should take care of correcting lock range on nodeA as well right?
> 
> 
> The problem we have seen is that to be able to correctly heal currently 
> acquired locks on brick reconnect, there are cases where we need to 
> release a lock that has already been granted (because the current owner 
> doesn't have enough quorum and a just recovered connection tries to 
> claim/heal it). In this case we need to deal with locks that have 
> already been merged, but without interfering with other existing locks 
> that already have quorum.
> 

Okay. Thanks for the detailed explanation. That clears my doubts.

-Soumya

From sabose at redhat.com  Tue Apr  2 09:45:57 2019
From: sabose at redhat.com (Sahina Bose)
Date: Tue, 2 Apr 2019 15:15:57 +0530
Subject: [Gluster-devel] [ovirt-users] oVirt Survey 2019 results
In-Reply-To: <CAPQRNT=dEsGiuBtMoELqoenWbp2pe3-Q3RmkppmDb8Aw8XiS1Q@mail.gmail.com>
References: <CAPQRNT=dEsGiuBtMoELqoenWbp2pe3-Q3RmkppmDb8Aw8XiS1Q@mail.gmail.com>
Message-ID: <CACjzOvfagTWMu532Qo3vN34eD6wE6cHjT1h03ksF_TmBbp+sfw@mail.gmail.com>

On Tue, Apr 2, 2019 at 12:07 PM Sandro Bonazzola <sbonazzo at redhat.com>
wrote:

> Thanks to the 143 participants to oVirt Survey 2019!
> The survey is now closed and results are publicly available at
> https://bit.ly/2JYlI7U
> We'll analyze collected data in order to improve oVirt thanks to your
> feedback.
>
> As a first step after reading the results I'd like to invite the 30
> persons who replied they're willing to contribute code to send an email to
> devel at ovirt.org introducing themselves: we'll be more than happy to
> welcome them and helping them getting started.
>
> I would also like to invite the 17 people who replied they'd like to help
> organizing oVirt events in their area to either get in touch with me or
> introduce themselves to users at ovirt.org so we can discuss about events
> organization.
>
> Last but not least I'd like to invite the 38 people willing to contribute
> documentation and the one willing to contribute localization to introduce
> themselves to devel at ovirt.org.
>

Thank you all for the feedback.
I was looking at the feedback specific to Gluster. While it's disheartening
to see "Gluster weakest link in oVirt", I can understand where the feedback
and frustration is coming from.

Over the past month and in this survey, the common themes that have come up
- Ensure smoother upgrades for the hyperconverged deployments with
GlusterFS.  The oVirt 4.3 release with upgrade to gluster 5.3 caused
disruption for many users and we want to ensure this does not happen again.
To this end, we are working on adding upgrade tests to OST based CI .
Contributions are welcome.

- improve performance on gluster storage domain. While we have seen
promising results with gluster 6 release this is an ongoing effort. Please
help this effort with inputs on the specific workloads and usecases that
you run, gathering data and running tests.

- deployment issues. We have worked to improve the deployment flow in 4.3
by adding pre-checks and changing to gluster-ansible role based deployment.
We would love to hear specific issues that you're facing around this -
please raise bugs if you haven't already (
https://bugzilla.redhat.com/enter_bug.cgi?product=cockpit-ovirt)



> Thanks!
> --
>
> SANDRO BONAZZOLA
>
> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>
> Red Hat EMEA <https://www.redhat.com/>
>
> sbonazzo at redhat.com
> <https://red.ht/sig>
> _______________________________________________
> Users mailing list -- users at ovirt.org
> To unsubscribe send an email to users-leave at ovirt.org
> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users at ovirt.org/message/4N5DYCXY2S6ZAUI7BWD4DEKZ6JL6MSGN/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190402/e22d9b62/attachment.html>

From atin.mukherjee83 at gmail.com  Tue Apr  2 09:51:35 2019
From: atin.mukherjee83 at gmail.com (Atin Mukherjee)
Date: Tue, 2 Apr 2019 15:21:35 +0530
Subject: [Gluster-devel] [ovirt-users] oVirt Survey 2019 results
In-Reply-To: <CACjzOvfagTWMu532Qo3vN34eD6wE6cHjT1h03ksF_TmBbp+sfw@mail.gmail.com>
References: <CAPQRNT=dEsGiuBtMoELqoenWbp2pe3-Q3RmkppmDb8Aw8XiS1Q@mail.gmail.com>
	<CACjzOvfagTWMu532Qo3vN34eD6wE6cHjT1h03ksF_TmBbp+sfw@mail.gmail.com>
Message-ID: <CAGkR8FPmuJrdeapVbx8u3KSQ8fGdhtCuFTkWYgcCR5JUTJgVPQ@mail.gmail.com>

Thanks Sahina for including Gluster community mailing lists.

As Sahina already mentioned we had a strong focus on upgrade testing path
before releasing glusterfs-6. We conducted test day and along with
functional pieces, tested upgrade paths like from 3.12, 4 & 5 to release-6,
we encountered problems but we fixed them before releasing glusterfs-6. So
overall this experience should definitely improve with glusterfs-6.

On Tue, 2 Apr 2019 at 15:16, Sahina Bose <sabose at redhat.com> wrote:

>
>
> On Tue, Apr 2, 2019 at 12:07 PM Sandro Bonazzola <sbonazzo at redhat.com>
> wrote:
>
>> Thanks to the 143 participants to oVirt Survey 2019!
>> The survey is now closed and results are publicly available at
>> https://bit.ly/2JYlI7U
>> We'll analyze collected data in order to improve oVirt thanks to your
>> feedback.
>>
>> As a first step after reading the results I'd like to invite the 30
>> persons who replied they're willing to contribute code to send an email to
>> devel at ovirt.org introducing themselves: we'll be more than happy to
>> welcome them and helping them getting started.
>>
>> I would also like to invite the 17 people who replied they'd like to help
>> organizing oVirt events in their area to either get in touch with me or
>> introduce themselves to users at ovirt.org so we can discuss about events
>> organization.
>>
>> Last but not least I'd like to invite the 38 people willing to contribute
>> documentation and the one willing to contribute localization to introduce
>> themselves to devel at ovirt.org.
>>
>
> Thank you all for the feedback.
> I was looking at the feedback specific to Gluster. While it's
> disheartening to see "Gluster weakest link in oVirt", I can understand
> where the feedback and frustration is coming from.
>
> Over the past month and in this survey, the common themes that have come up
> - Ensure smoother upgrades for the hyperconverged deployments with
> GlusterFS.  The oVirt 4.3 release with upgrade to gluster 5.3 caused
> disruption for many users and we want to ensure this does not happen again.
> To this end, we are working on adding upgrade tests to OST based CI .
> Contributions are welcome.
>
> - improve performance on gluster storage domain. While we have seen
> promising results with gluster 6 release this is an ongoing effort. Please
> help this effort with inputs on the specific workloads and usecases that
> you run, gathering data and running tests.
>
> - deployment issues. We have worked to improve the deployment flow in 4.3
> by adding pre-checks and changing to gluster-ansible role based deployment.
> We would love to hear specific issues that you're facing around this -
> please raise bugs if you haven't already (
> https://bugzilla.redhat.com/enter_bug.cgi?product=cockpit-ovirt)
>
>
>
>> Thanks!
>> --
>>
>> SANDRO BONAZZOLA
>>
>> MANAGER, SOFTWARE ENGINEERING, EMEA R&D RHV
>>
>> Red Hat EMEA <https://www.redhat.com/>
>>
>> sbonazzo at redhat.com
>> <https://red.ht/sig>
>> _______________________________________________
>> Users mailing list -- users at ovirt.org
>> To unsubscribe send an email to users-leave at ovirt.org
>> Privacy Statement: https://www.ovirt.org/site/privacy-policy/
>> oVirt Code of Conduct:
>> https://www.ovirt.org/community/about/community-guidelines/
>> List Archives:
>> https://lists.ovirt.org/archives/list/users at ovirt.org/message/4N5DYCXY2S6ZAUI7BWD4DEKZ6JL6MSGN/
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel

-- 
--Atin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190402/479f5011/attachment-0001.html>

From nux at li.nux.ro  Tue Apr  2 15:37:17 2019
From: nux at li.nux.ro (Nux!)
Date: Tue, 2 Apr 2019 16:37:17 +0100 (BST)
Subject: [Gluster-devel] [Gluster-users] Prioritise local bricks for IO?
In-Reply-To: <CAOUCJ=jWKdVihOiF05TWjh-HBPibEPuk4zHp2FPozuuy2czXiQ@mail.gmail.com>
References: <29221907.583.1553599314586.JavaMail.zimbra@li.nux.ro>
	<CACeDKteJ8h58KtOXFwZ=qBHddZ=T9p0G_H_ztxcJkR5tXrYxow@mail.gmail.com>
	<CDC554FD-394C-490D-9CB9-A1D9FCADF27E@li.nux.ro>
	<CALXm_dF-R_Ubo9AFZn5UwBBpNQsVmkM3EdKAB6ugS2PTnTR_=A@mail.gmail.com>
	<CAOUCJ=jWKdVihOiF05TWjh-HBPibEPuk4zHp2FPozuuy2czXiQ@mail.gmail.com>
Message-ID: <383369409.4472.1554219437440.JavaMail.zimbra@li.nux.ro>

Ok, cool, thanks. So.. no go.

Any other ideas on how to accomplish task then?

--
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro

----- Original Message -----
> From: "Nithya Balachandran" <nbalacha at redhat.com>
> To: "Poornima Gurusiddaiah" <pgurusid at redhat.com>
> Cc: "Nux!" <nux at li.nux.ro>, "gluster-users" <gluster-users at gluster.org>, "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Thursday, 28 March, 2019 09:38:16
> Subject: Re: [Gluster-users] Prioritise local bricks for IO?

> On Wed, 27 Mar 2019 at 20:27, Poornima Gurusiddaiah <pgurusid at redhat.com>
> wrote:
> 
>> This feature is not under active development as it was not used widely.
>> AFAIK its not supported feature.
>> +Nithya +Raghavendra for further clarifications.
>>
> 
> This is not actively supported  - there has been no work done on this
> feature for a long time.
> 
> Regards,
> Nithya
> 
>>
>> Regards,
>> Poornima
>>
>> On Wed, Mar 27, 2019 at 12:33 PM Lucian <nux at li.nux.ro> wrote:
>>
>>> Oh, that's just what the doctor ordered!
>>> Hope it works, thanks
>>>
>>> On 27 March 2019 03:15:57 GMT, Vlad Kopylov <vladkopy at gmail.com> wrote:
>>>>
>>>> I don't remember if it still in works
>>>> NUFA
>>>>
>>>> https://github.com/gluster/glusterfs-specs/blob/master/done/Features/nufa.md
>>>>
>>>> v
>>>>
>>>> On Tue, Mar 26, 2019 at 7:27 AM Nux! <nux at li.nux.ro> wrote:
>>>>
>>>>> Hello,
>>>>>
>>>>> I'm trying to set up a distributed backup storage (no replicas), but
>>>>> I'd like to prioritise the local bricks for any IO done on the volume.
>>>>> This will be a backup stor, so in other words, I'd like the files to be
>>>>> written locally if there is space, so as to save the NICs for other traffic.
>>>>>
>>>>> Anyone knows how this might be achievable, if at all?
>>>>>
>>>>> --
>>>>> Sent from the Delta quadrant using Borg technology!
>>>>>
>>>>> Nux!
>>>>> www.nux.ro
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>> --
>>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users

From ykaul at redhat.com  Tue Apr  2 18:16:02 2019
From: ykaul at redhat.com (Yaniv Kaul)
Date: Tue, 2 Apr 2019 21:16:02 +0300
Subject: [Gluster-devel] [Gluster-users] Prioritise local bricks for IO?
In-Reply-To: <383369409.4472.1554219437440.JavaMail.zimbra@li.nux.ro>
References: <29221907.583.1553599314586.JavaMail.zimbra@li.nux.ro>
	<CACeDKteJ8h58KtOXFwZ=qBHddZ=T9p0G_H_ztxcJkR5tXrYxow@mail.gmail.com>
	<CDC554FD-394C-490D-9CB9-A1D9FCADF27E@li.nux.ro>
	<CALXm_dF-R_Ubo9AFZn5UwBBpNQsVmkM3EdKAB6ugS2PTnTR_=A@mail.gmail.com>
	<CAOUCJ=jWKdVihOiF05TWjh-HBPibEPuk4zHp2FPozuuy2czXiQ@mail.gmail.com>
	<383369409.4472.1554219437440.JavaMail.zimbra@li.nux.ro>
Message-ID: <CAJgorsa=z6aJ+rOCBzvGGDa2d29UOZ5SUat-OAxW4DJbRBK7gA@mail.gmail.com>

On Tue, Apr 2, 2019 at 6:37 PM Nux! <nux at li.nux.ro> wrote:

> Ok, cool, thanks. So.. no go.
>
> Any other ideas on how to accomplish task then?
>

While not a solution, I believe
https://review.gluster.org/#/c/glusterfs/+/21333/ - read selection based on
latency, is an interesting path towards this.
(Of course, you'd need later also add write...)
Y.


> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Nithya Balachandran" <nbalacha at redhat.com>
> > To: "Poornima Gurusiddaiah" <pgurusid at redhat.com>
> > Cc: "Nux!" <nux at li.nux.ro>, "gluster-users" <gluster-users at gluster.org>,
> "Gluster Devel" <gluster-devel at gluster.org>
> > Sent: Thursday, 28 March, 2019 09:38:16
> > Subject: Re: [Gluster-users] Prioritise local bricks for IO?
>
> > On Wed, 27 Mar 2019 at 20:27, Poornima Gurusiddaiah <pgurusid at redhat.com
> >
> > wrote:
> >
> >> This feature is not under active development as it was not used widely.
> >> AFAIK its not supported feature.
> >> +Nithya +Raghavendra for further clarifications.
> >>
> >
> > This is not actively supported  - there has been no work done on this
> > feature for a long time.
> >
> > Regards,
> > Nithya
> >
> >>
> >> Regards,
> >> Poornima
> >>
> >> On Wed, Mar 27, 2019 at 12:33 PM Lucian <nux at li.nux.ro> wrote:
> >>
> >>> Oh, that's just what the doctor ordered!
> >>> Hope it works, thanks
> >>>
> >>> On 27 March 2019 03:15:57 GMT, Vlad Kopylov <vladkopy at gmail.com>
> wrote:
> >>>>
> >>>> I don't remember if it still in works
> >>>> NUFA
> >>>>
> >>>>
> https://github.com/gluster/glusterfs-specs/blob/master/done/Features/nufa.md
> >>>>
> >>>> v
> >>>>
> >>>> On Tue, Mar 26, 2019 at 7:27 AM Nux! <nux at li.nux.ro> wrote:
> >>>>
> >>>>> Hello,
> >>>>>
> >>>>> I'm trying to set up a distributed backup storage (no replicas), but
> >>>>> I'd like to prioritise the local bricks for any IO done on the
> volume.
> >>>>> This will be a backup stor, so in other words, I'd like the files to
> be
> >>>>> written locally if there is space, so as to save the NICs for other
> traffic.
> >>>>>
> >>>>> Anyone knows how this might be achievable, if at all?
> >>>>>
> >>>>> --
> >>>>> Sent from the Delta quadrant using Borg technology!
> >>>>>
> >>>>> Nux!
> >>>>> www.nux.ro
> >>>>> _______________________________________________
> >>>>> Gluster-users mailing list
> >>>>> Gluster-users at gluster.org
> >>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
> >>>>>
> >>>>
> >>> --
> >>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
> >>> _______________________________________________
> >>> Gluster-users mailing list
> >>> Gluster-users at gluster.org
> >>> https://lists.gluster.org/mailman/listinfo/gluster-users
> >>
> >> _______________________________________________
> >> Gluster-users mailing list
> >> Gluster-users at gluster.org
> > > https://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190402/e8799b91/attachment.html>

From amukherj at redhat.com  Wed Apr  3 03:22:06 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Wed, 3 Apr 2019 08:52:06 +0530
Subject: [Gluster-devel] Backporting important fixes in release branches
Message-ID: <CAGNCGH02Zm9_fckw+7rdyua0o7oAAzus=8QvMSSyDf-JQDmBPA@mail.gmail.com>

Off late my observation has been that we're missing to backport
critical/important fixes into the release branches and we do a course of
correction when users discover the problems which isn't a great experience.
I request all developers and maintainers to pay some attention on (a)
deciding on which patches from mainline should be backported to what
release branches & (b) do the same right away once the patches are merged
in mainline branch instead of waiting to do them later.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/01ab402f/attachment.html>

From amukherj at redhat.com  Wed Apr  3 05:16:51 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Wed, 3 Apr 2019 10:46:51 +0530
Subject: [Gluster-devel] is_nfs_export_available from nfs.rc failing too
	often?
Message-ID: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>

I'm observing the above test function failing too often because of which
arbiter-mount.t test fails in many regression jobs. Such frequency of
failures wasn't there earlier. Does anyone know what has changed recently
to cause these failures in regression? I also hear when such failure
happens a reboot is required, is that true and if so why?

One of the reference :
https://build.gluster.org/job/centos7-regression/5340/consoleFull
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/f5d9df4f/attachment.html>

From jthottan at redhat.com  Wed Apr  3 06:26:20 2019
From: jthottan at redhat.com (Jiffin Thottan)
Date: Wed, 3 Apr 2019 02:26:20 -0400 (EDT)
Subject: [Gluster-devel] is_nfs_export_available from nfs.rc failing
	too	often?
In-Reply-To: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
Message-ID: <2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>

Hi,

is_nfs_export_available is just a wrapper around "showmount" command AFAIR.
I saw following messages in console output.
 mount.nfs: rpc.statd is not running but is required for remote locking.
05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
05:06:55 mount.nfs: an incorrect mount option was specified

For me it looks rpcbind may not be running on the machine.
Usually rpcbind starts automatically on machines, don't know whether it can happen or not. 

Regards,
Jiffin


----- Original Message -----
From: "Atin Mukherjee" <amukherj at redhat.com>
To: "gluster-infra" <gluster-infra at gluster.org>, "Gluster Devel" <gluster-devel at gluster.org>
Sent: Wednesday, April 3, 2019 10:46:51 AM
Subject: [Gluster-devel] is_nfs_export_available from nfs.rc failing too	often?

I'm observing the above test function failing too often because of which arbiter-mount.t test fails in many regression jobs. Such frequency of failures wasn't there earlier. Does anyone know what has changed recently to cause these failures in regression? I also hear when such failure happens a reboot is required, is that true and if so why? 

One of the reference : https://build.gluster.org/job/centos7-regression/5340/consoleFull 


_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-devel

From amukherj at redhat.com  Wed Apr  3 11:00:42 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Wed, 3 Apr 2019 16:30:42 +0530
Subject: [Gluster-devel] is_nfs_export_available from nfs.rc failing too
	often?
In-Reply-To: <2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
Message-ID: <CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>

On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <jthottan at redhat.com> wrote:

> Hi,
>
> is_nfs_export_available is just a wrapper around "showmount" command AFAIR.
> I saw following messages in console output.
>  mount.nfs: rpc.statd is not running but is required for remote locking.
> 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or start
> statd.
> 05:06:55 mount.nfs: an incorrect mount option was specified
>
> For me it looks rpcbind may not be running on the machine.
> Usually rpcbind starts automatically on machines, don't know whether it
> can happen or not.
>

That's precisely what the question is. Why suddenly we're seeing this
happening too frequently. Today I saw atleast 4 to 5 such failures already.

Deepshika - Can you please help in inspecting this?


> Regards,
> Jiffin
>
>
> ----- Original Message -----
> From: "Atin Mukherjee" <amukherj at redhat.com>
> To: "gluster-infra" <gluster-infra at gluster.org>, "Gluster Devel" <
> gluster-devel at gluster.org>
> Sent: Wednesday, April 3, 2019 10:46:51 AM
> Subject: [Gluster-devel] is_nfs_export_available from nfs.rc failing too
>       often?
>
> I'm observing the above test function failing too often because of which
> arbiter-mount.t test fails in many regression jobs. Such frequency of
> failures wasn't there earlier. Does anyone know what has changed recently
> to cause these failures in regression? I also hear when such failure
> happens a reboot is required, is that true and if so why?
>
> One of the reference :
> https://build.gluster.org/job/centos7-regression/5340/consoleFull
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/0e3b0b5f/attachment-0001.html>

From mscherer at redhat.com  Wed Apr  3 11:52:50 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Wed, 03 Apr 2019 13:52:50 +0200
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
Message-ID: <46932285269538f29a3bdd0ccb177bfce091bf85.camel@redhat.com>

Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <jthottan at redhat.com>
> wrote:
> 
> > Hi,
> > 
> > is_nfs_export_available is just a wrapper around "showmount"
> > command AFAIR.
> > I saw following messages in console output.
> >  mount.nfs: rpc.statd is not running but is required for remote
> > locking.
> > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or
> > start
> > statd.
> > 05:06:55 mount.nfs: an incorrect mount option was specified
> > 
> > For me it looks rpcbind may not be running on the machine.
> > Usually rpcbind starts automatically on machines, don't know
> > whether it
> > can happen or not.
> > 
> 
> That's precisely what the question is. Why suddenly we're seeing this
> happening too frequently. Today I saw atleast 4 to 5 such failures
> already.
> 
> Deepshika - Can you please help in inspecting this?

So in the past, this kind of stuff did happen with ipv6, so this could
be a change on AWS and/or a upgrade. 

We are currently investigating a set of failure that happen after
reboot (resulting in partial network bring up, causing all kind of
weird issue), but it take some time to verify it, and since we lost 33%
of the team with Nigel departure, stuff do not move as fast as before.


-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/1e288688/attachment.sig>

From ykaul at redhat.com  Wed Apr  3 12:12:16 2019
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 3 Apr 2019 15:12:16 +0300
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <46932285269538f29a3bdd0ccb177bfce091bf85.camel@redhat.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
	<46932285269538f29a3bdd0ccb177bfce091bf85.camel@redhat.com>
Message-ID: <CAJgorsZJX6GG3bdsAs4ZSEBEeF5td+r2pwcBVPC=DuLHX3NJZA@mail.gmail.com>

On Wed, Apr 3, 2019 at 2:53 PM Michael Scherer <mscherer at redhat.com> wrote:

> Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <jthottan at redhat.com>
> > wrote:
> >
> > > Hi,
> > >
> > > is_nfs_export_available is just a wrapper around "showmount"
> > > command AFAIR.
> > > I saw following messages in console output.
> > >  mount.nfs: rpc.statd is not running but is required for remote
> > > locking.
> > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or
> > > start
> > > statd.
> > > 05:06:55 mount.nfs: an incorrect mount option was specified
> > >
> > > For me it looks rpcbind may not be running on the machine.
> > > Usually rpcbind starts automatically on machines, don't know
> > > whether it
> > > can happen or not.
> > >
> >
> > That's precisely what the question is. Why suddenly we're seeing this
> > happening too frequently. Today I saw atleast 4 to 5 such failures
> > already.
> >
> > Deepshika - Can you please help in inspecting this?
>
> So in the past, this kind of stuff did happen with ipv6, so this could
> be a change on AWS and/or a upgrade.
>

We need to enable IPv6, for two reasons:
1. IPv6 is common these days, even if we don't test with it, it should be
there.
2. We should test with IPv6...

I'm not sure, but I suspect we do disable IPv6 here and there. Example[1].
Y.

[1]
https://github.com/gluster/centosci/blob/master/jobs/scripts/glusto/setup-glusto.yml

>
> We are currently investigating a set of failure that happen after
> reboot (resulting in partial network bring up, causing all kind of
> weird issue), but it take some time to verify it, and since we lost 33%
> of the team with Nigel departure, stuff do not move as fast as before.
>
>
> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/21240905/attachment.html>

From mscherer at redhat.com  Wed Apr  3 12:19:15 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Wed, 03 Apr 2019 14:19:15 +0200
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <CAJgorsZJX6GG3bdsAs4ZSEBEeF5td+r2pwcBVPC=DuLHX3NJZA@mail.gmail.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
	<46932285269538f29a3bdd0ccb177bfce091bf85.camel@redhat.com>
	<CAJgorsZJX6GG3bdsAs4ZSEBEeF5td+r2pwcBVPC=DuLHX3NJZA@mail.gmail.com>
Message-ID: <1658d7c7b3170ad7abe6afbcdf769775e9274da3.camel@redhat.com>

Le mercredi 03 avril 2019 ? 15:12 +0300, Yaniv Kaul a ?crit :
> On Wed, Apr 3, 2019 at 2:53 PM Michael Scherer <mscherer at redhat.com>
> wrote:
> 
> > Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> > > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <
> > > jthottan at redhat.com>
> > > wrote:
> > > 
> > > > Hi,
> > > > 
> > > > is_nfs_export_available is just a wrapper around "showmount"
> > > > command AFAIR.
> > > > I saw following messages in console output.
> > > >  mount.nfs: rpc.statd is not running but is required for remote
> > > > locking.
> > > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local,
> > > > or
> > > > start
> > > > statd.
> > > > 05:06:55 mount.nfs: an incorrect mount option was specified
> > > > 
> > > > For me it looks rpcbind may not be running on the machine.
> > > > Usually rpcbind starts automatically on machines, don't know
> > > > whether it
> > > > can happen or not.
> > > > 
> > > 
> > > That's precisely what the question is. Why suddenly we're seeing
> > > this
> > > happening too frequently. Today I saw atleast 4 to 5 such
> > > failures
> > > already.
> > > 
> > > Deepshika - Can you please help in inspecting this?
> > 
> > So in the past, this kind of stuff did happen with ipv6, so this
> > could
> > be a change on AWS and/or a upgrade.
> > 
> 
> We need to enable IPv6, for two reasons:
> 1. IPv6 is common these days, even if we don't test with it, it
> should be
> there.
> 2. We should test with IPv6...
> 
> I'm not sure, but I suspect we do disable IPv6 here and there.
> Example[1].
> Y.
> 
> [1]
> 
https://github.com/gluster/centosci/blob/master/jobs/scripts/glusto/setup-glusto.yml

We do disable ipv6 for sure, Nigel spent 3 days just on that for the
AWS migration, and we do have a dedicated playbook applied on all
builders that try to disable everything in every possible way:


https://github.com/gluster/gluster.org_ansible_configuration/blob/master/roles/jenkins_builder/tasks/disable_ipv6_linux.yml

According to the comment, that's from 2016, and I am sure this go
further in the past since it wasn't just documented before.


-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/837b9996/attachment.sig>

From mscherer at redhat.com  Wed Apr  3 13:56:36 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Wed, 03 Apr 2019 15:56:36 +0200
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
Message-ID: <797512f6ff7f1b9fedbf8b7968dd86a6968d9105.camel@redhat.com>

Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <jthottan at redhat.com>
> wrote:
> 
> > Hi,
> > 
> > is_nfs_export_available is just a wrapper around "showmount"
> > command AFAIR.
> > I saw following messages in console output.
> >  mount.nfs: rpc.statd is not running but is required for remote
> > locking.
> > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or
> > start
> > statd.
> > 05:06:55 mount.nfs: an incorrect mount option was specified
> > 
> > For me it looks rpcbind may not be running on the machine.
> > Usually rpcbind starts automatically on machines, don't know
> > whether it
> > can happen or not.
> > 
> 
> That's precisely what the question is. Why suddenly we're seeing this
> happening too frequently. Today I saw atleast 4 to 5 such failures
> already.
> 
> Deepshika - Can you please help in inspecting this?

So we think (we are not sure) that the issue is a bit complex.

What we were investigating was nightly run fail on aws. When the build
crash, the builder is restarted, since that's the easiest way to clean
everything (since even with a perfect test suite that would clean
itself, we could always end in a corrupt state on the system, WRT
mount, fs, etc).

In turn, this seems to cause trouble on aws, since cloud-init or
something rename eth0 interface to ens5, without cleaning to the
network configuration. 

So the network init script fail (because the image say "start eth0" and
that's not present), but fail in a weird way. Network is initialised
and working (we can connect), but the dhclient process is not in the
right cgroup, and network.service is in failed state. Restarting
network didn't work. In turn, this mean that rpc-statd refuse to start
(due to systemd dependencies), which seems to impact various NFS tests.

We have also seen that on some builders, rpcbind pick some IP v6
autoconfiguration, but we can't reproduce that, and there is no ip v6
set up anywhere. I suspect the network.service failure is somehow
involved, but fail to see how. In turn, rpcbind.socket not starting
could cause NFS test troubles.

Our current stop gap fix was to fix all the builders one by one. Remove
the config, kill the rogue dhclient, restart network service. 

However, we can't be sure this is going to fix the problem long term
since this only manifest after a crash of the test suite, and it
doesn't happen so often. (plus, it was working before some day in the
past, when something did make this fail, and I do not know if that's a
system upgrade, or a test change, or both).

So we are still looking at it to have a complete understanding of the
issue, but so far, we hacked our way to make it work (or so do I
think).

Deepshika is working to fix it long term, by fixing the issue regarding
eth0/ens5 with a new base image.
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190403/da985e90/attachment.sig>

From amukherj at redhat.com  Thu Apr  4 06:17:59 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Thu, 4 Apr 2019 11:47:59 +0530
Subject: [Gluster-devel] shd multiplexing patch has introduced coverity
	defects
Message-ID: <CAGNCGH1JxOic1_aVFcPMtN9JTGgZBfL78NHoHnKsE=RaGSo=Jg@mail.gmail.com>

Based on yesterday's coverity scan report, 6 defects are introduced because
of the shd multiplexing patch. Could you address them, Rafi?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/c8c70ef3/attachment.html>

From amukherj at redhat.com  Thu Apr  4 10:43:43 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Thu, 4 Apr 2019 16:13:43 +0530
Subject: [Gluster-devel] rebal-all-nodes-migrate.t always fails now
Message-ID: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>

Based on what I have seen that any multi node test case will fail and the
above one is picked first from that group and If I am correct none of the
code fixes will go through the regression until this is fixed. I suspect it
to be an infra issue again. If we look at
https://review.gluster.org/#/c/glusterfs/+/22501/ &
https://build.gluster.org/job/centos7-regression/5382/ peer handshaking is
stuck as 127.1.1.1 is unable to receive a response back, did we end up
having firewall and other n/w settings screwed up? The test never fails
locally.

*15:51:21* Number of Peers: 2*15:51:21* *15:51:21* Hostname:
127.1.1.2*15:51:21* Uuid:
0e689ca8-d522-4b2f-b437-9dcde3579401*15:51:21* State: Accepted peer
request (Connected)*15:51:21* *15:51:21* Hostname: 127.1.1.3*15:51:21*
Uuid: a83a3bfa-729f-4a1c-8f9a-ae7d04ee4544*15:51:21* State: Accepted
peer request (Connected)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/4262448f/attachment-0001.html>

From mscherer at redhat.com  Thu Apr  4 11:53:21 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Thu, 04 Apr 2019 13:53:21 +0200
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
	always fails now
In-Reply-To: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
Message-ID: <94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>

Le jeudi 04 avril 2019 ? 16:13 +0530, Atin Mukherjee a ?crit :
> Based on what I have seen that any multi node test case will fail and
> the
> above one is picked first from that group and If I am correct none of
> the
> code fixes will go through the regression until this is fixed. I
> suspect it
> to be an infra issue again. If we look at
> https://review.gluster.org/#/c/glusterfs/+/22501/ &
> https://build.gluster.org/job/centos7-regression/5382/ peer
> handshaking is
> stuck as 127.1.1.1 is unable to receive a response back, did we end
> up
> having firewall and other n/w settings screwed up? The test never
> fails
> locally.

The firewall didn't change, and since the start has a line:
"-A INPUT -i lo -j ACCEPT", so all traffic on the localhost interface
work. (I am not even sure that netfilter do anything meaningful on the
loopback interface, but maybe I am wrong, and not keen on looking
kernel code for that).


Ping seems to work fine as well, so we can exclude a routing issue.

Maybe we should look at the socket, does it listen to a specific
address or not ?



> *15:51:21* Number of Peers: 2*15:51:21* *15:51:21* Hostname:
> 127.1.1.2*15:51:21* Uuid:
> 0e689ca8-d522-4b2f-b437-9dcde3579401*15:51:21* State: Accepted peer
> request (Connected)*15:51:21* *15:51:21* Hostname:
> 127.1.1.3*15:51:21*
> Uuid: a83a3bfa-729f-4a1c-8f9a-ae7d04ee4544*15:51:21* State: Accepted
> peer request (Connected)
> _______________________________________________
> Gluster-infra mailing list
> Gluster-infra at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-infra
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/0aae5e33/attachment.sig>

From mscherer at redhat.com  Thu Apr  4 13:19:25 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Thu, 04 Apr 2019 15:19:25 +0200
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
Message-ID: <eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>

Le jeudi 04 avril 2019 ? 13:53 +0200, Michael Scherer a ?crit :
> Le jeudi 04 avril 2019 ? 16:13 +0530, Atin Mukherjee a ?crit :
> > Based on what I have seen that any multi node test case will fail
> > and
> > the
> > above one is picked first from that group and If I am correct none
> > of
> > the
> > code fixes will go through the regression until this is fixed. I
> > suspect it
> > to be an infra issue again. If we look at
> > https://review.gluster.org/#/c/glusterfs/+/22501/ &
> > https://build.gluster.org/job/centos7-regression/5382/ peer
> > handshaking is
> > stuck as 127.1.1.1 is unable to receive a response back, did we end
> > up
> > having firewall and other n/w settings screwed up? The test never
> > fails
> > locally.
> 
> The firewall didn't change, and since the start has a line:
> "-A INPUT -i lo -j ACCEPT", so all traffic on the localhost interface
> work. (I am not even sure that netfilter do anything meaningful on
> the
> loopback interface, but maybe I am wrong, and not keen on looking
> kernel code for that).
> 
> 
> Ping seems to work fine as well, so we can exclude a routing issue.
> 
> Maybe we should look at the socket, does it listen to a specific
> address or not ?

So, I did look at the 20 first ailure, removed all not related to
rebal-all-nodes-migrate.t and seen all were run on builder203, who was
freshly reinstalled. As Deepshika noticed today, this one had a issue
with ipv6, the 2nd issue we were tracking.

Summary, rpcbind.socket systemd unit listen on ipv6 despites ipv6 being
disabled, and the fix is to reload systemd. We have so far no idea on
why it happen, but suspect this might be related to the network issue
we did identify, as that happen only after a reboot, that happen only
if a build is cancelled/crashed/aborted.

I apply the workaround on builder203, so if the culprit is that
specific issue, guess that's fixed. 

I started a test to see how it go:
https://build.gluster.org/job/centos7-regression/5383/

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/4bbc9ad9/attachment.sig>

From mscherer at redhat.com  Thu Apr  4 13:54:22 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Thu, 04 Apr 2019 15:54:22 +0200
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
Message-ID: <f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>

Le jeudi 04 avril 2019 ? 15:19 +0200, Michael Scherer a ?crit :
> Le jeudi 04 avril 2019 ? 13:53 +0200, Michael Scherer a ?crit :
> > Le jeudi 04 avril 2019 ? 16:13 +0530, Atin Mukherjee a ?crit :
> > > Based on what I have seen that any multi node test case will fail
> > > and
> > > the
> > > above one is picked first from that group and If I am correct
> > > none
> > > of
> > > the
> > > code fixes will go through the regression until this is fixed. I
> > > suspect it
> > > to be an infra issue again. If we look at
> > > https://review.gluster.org/#/c/glusterfs/+/22501/ &
> > > https://build.gluster.org/job/centos7-regression/5382/ peer
> > > handshaking is
> > > stuck as 127.1.1.1 is unable to receive a response back, did we
> > > end
> > > up
> > > having firewall and other n/w settings screwed up? The test never
> > > fails
> > > locally.
> > 
> > The firewall didn't change, and since the start has a line:
> > "-A INPUT -i lo -j ACCEPT", so all traffic on the localhost
> > interface
> > work. (I am not even sure that netfilter do anything meaningful on
> > the
> > loopback interface, but maybe I am wrong, and not keen on looking
> > kernel code for that).
> > 
> > 
> > Ping seems to work fine as well, so we can exclude a routing issue.
> > 
> > Maybe we should look at the socket, does it listen to a specific
> > address or not ?
> 
> So, I did look at the 20 first ailure, removed all not related to
> rebal-all-nodes-migrate.t and seen all were run on builder203, who
> was
> freshly reinstalled. As Deepshika noticed today, this one had a issue
> with ipv6, the 2nd issue we were tracking.
> 
> Summary, rpcbind.socket systemd unit listen on ipv6 despites ipv6
> being
> disabled, and the fix is to reload systemd. We have so far no idea on
> why it happen, but suspect this might be related to the network issue
> we did identify, as that happen only after a reboot, that happen only
> if a build is cancelled/crashed/aborted.
> 
> I apply the workaround on builder203, so if the culprit is that
> specific issue, guess that's fixed. 
> 
> I started a test to see how it go:
> https://build.gluster.org/job/centos7-regression/5383/

The test did just pass, so I would assume the problem was local to
builder203. Not sure why it was always selected, except because this
was the only one that failed, so was always up for getting new jobs. 

Maybe we should increase the number of builder so this doesn't happen,
as I guess the others builders were busy at that time ?

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/461040fc/attachment.sig>

From amukherj at redhat.com  Thu Apr  4 15:55:39 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Thu, 4 Apr 2019 21:25:39 +0530
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
Message-ID: <CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>

Thanks misc. I have always seen a pattern that on a reattempt (recheck
centos) the same builder is picked up many time even though it's promised
to pick up the builders in a round robin manner.

On Thu, Apr 4, 2019 at 7:24 PM Michael Scherer <mscherer at redhat.com> wrote:

> Le jeudi 04 avril 2019 ? 15:19 +0200, Michael Scherer a ?crit :
> > Le jeudi 04 avril 2019 ? 13:53 +0200, Michael Scherer a ?crit :
> > > Le jeudi 04 avril 2019 ? 16:13 +0530, Atin Mukherjee a ?crit :
> > > > Based on what I have seen that any multi node test case will fail
> > > > and
> > > > the
> > > > above one is picked first from that group and If I am correct
> > > > none
> > > > of
> > > > the
> > > > code fixes will go through the regression until this is fixed. I
> > > > suspect it
> > > > to be an infra issue again. If we look at
> > > > https://review.gluster.org/#/c/glusterfs/+/22501/ &
> > > > https://build.gluster.org/job/centos7-regression/5382/ peer
> > > > handshaking is
> > > > stuck as 127.1.1.1 is unable to receive a response back, did we
> > > > end
> > > > up
> > > > having firewall and other n/w settings screwed up? The test never
> > > > fails
> > > > locally.
> > >
> > > The firewall didn't change, and since the start has a line:
> > > "-A INPUT -i lo -j ACCEPT", so all traffic on the localhost
> > > interface
> > > work. (I am not even sure that netfilter do anything meaningful on
> > > the
> > > loopback interface, but maybe I am wrong, and not keen on looking
> > > kernel code for that).
> > >
> > >
> > > Ping seems to work fine as well, so we can exclude a routing issue.
> > >
> > > Maybe we should look at the socket, does it listen to a specific
> > > address or not ?
> >
> > So, I did look at the 20 first ailure, removed all not related to
> > rebal-all-nodes-migrate.t and seen all were run on builder203, who
> > was
> > freshly reinstalled. As Deepshika noticed today, this one had a issue
> > with ipv6, the 2nd issue we were tracking.
> >
> > Summary, rpcbind.socket systemd unit listen on ipv6 despites ipv6
> > being
> > disabled, and the fix is to reload systemd. We have so far no idea on
> > why it happen, but suspect this might be related to the network issue
> > we did identify, as that happen only after a reboot, that happen only
> > if a build is cancelled/crashed/aborted.
> >
> > I apply the workaround on builder203, so if the culprit is that
> > specific issue, guess that's fixed.
> >
> > I started a test to see how it go:
> > https://build.gluster.org/job/centos7-regression/5383/
>
> The test did just pass, so I would assume the problem was local to
> builder203. Not sure why it was always selected, except because this
> was the only one that failed, so was always up for getting new jobs.
>
> Maybe we should increase the number of builder so this doesn't happen,
> as I guess the others builders were busy at that time ?
>
> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/d98ad6e2/attachment.html>

From ykaul at redhat.com  Thu Apr  4 16:10:34 2019
From: ykaul at redhat.com (Yaniv Kaul)
Date: Thu, 4 Apr 2019 19:10:34 +0300
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
Message-ID: <CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>

I'm not convinced this is solved. Just had what I believe is a similar
failure:

*00:12:02.532* A dependency job for rpc-statd.service failed. See
'journalctl -xe' for details.*00:12:02.532* mount.nfs: rpc.statd is
not running but is required for remote locking.*00:12:02.532*
mount.nfs: Either use '-o nolock' to keep locks local, or start
statd.*00:12:02.532* mount.nfs: an incorrect mount option was
specified

(of course, it can always be my patch!)

https://build.gluster.org/job/centos7-regression/5384/console


On Thu, Apr 4, 2019 at 6:56 PM Atin Mukherjee <amukherj at redhat.com> wrote:

> Thanks misc. I have always seen a pattern that on a reattempt (recheck
> centos) the same builder is picked up many time even though it's promised
> to pick up the builders in a round robin manner.
>
> On Thu, Apr 4, 2019 at 7:24 PM Michael Scherer <mscherer at redhat.com>
> wrote:
>
>> Le jeudi 04 avril 2019 ? 15:19 +0200, Michael Scherer a ?crit :
>> > Le jeudi 04 avril 2019 ? 13:53 +0200, Michael Scherer a ?crit :
>> > > Le jeudi 04 avril 2019 ? 16:13 +0530, Atin Mukherjee a ?crit :
>> > > > Based on what I have seen that any multi node test case will fail
>> > > > and
>> > > > the
>> > > > above one is picked first from that group and If I am correct
>> > > > none
>> > > > of
>> > > > the
>> > > > code fixes will go through the regression until this is fixed. I
>> > > > suspect it
>> > > > to be an infra issue again. If we look at
>> > > > https://review.gluster.org/#/c/glusterfs/+/22501/ &
>> > > > https://build.gluster.org/job/centos7-regression/5382/ peer
>> > > > handshaking is
>> > > > stuck as 127.1.1.1 is unable to receive a response back, did we
>> > > > end
>> > > > up
>> > > > having firewall and other n/w settings screwed up? The test never
>> > > > fails
>> > > > locally.
>> > >
>> > > The firewall didn't change, and since the start has a line:
>> > > "-A INPUT -i lo -j ACCEPT", so all traffic on the localhost
>> > > interface
>> > > work. (I am not even sure that netfilter do anything meaningful on
>> > > the
>> > > loopback interface, but maybe I am wrong, and not keen on looking
>> > > kernel code for that).
>> > >
>> > >
>> > > Ping seems to work fine as well, so we can exclude a routing issue.
>> > >
>> > > Maybe we should look at the socket, does it listen to a specific
>> > > address or not ?
>> >
>> > So, I did look at the 20 first ailure, removed all not related to
>> > rebal-all-nodes-migrate.t and seen all were run on builder203, who
>> > was
>> > freshly reinstalled. As Deepshika noticed today, this one had a issue
>> > with ipv6, the 2nd issue we were tracking.
>> >
>> > Summary, rpcbind.socket systemd unit listen on ipv6 despites ipv6
>> > being
>> > disabled, and the fix is to reload systemd. We have so far no idea on
>> > why it happen, but suspect this might be related to the network issue
>> > we did identify, as that happen only after a reboot, that happen only
>> > if a build is cancelled/crashed/aborted.
>> >
>> > I apply the workaround on builder203, so if the culprit is that
>> > specific issue, guess that's fixed.
>> >
>> > I started a test to see how it go:
>> > https://build.gluster.org/job/centos7-regression/5383/
>>
>> The test did just pass, so I would assume the problem was local to
>> builder203. Not sure why it was always selected, except because this
>> was the only one that failed, so was always up for getting new jobs.
>>
>> Maybe we should increase the number of builder so this doesn't happen,
>> as I guess the others builders were busy at that time ?
>>
>> --
>> Michael Scherer
>> Sysadmin, Community Infrastructure and Platform, OSAS
>>
>>
>> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/11c624ee/attachment-0001.html>

From mscherer at redhat.com  Thu Apr  4 16:24:56 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Thu, 04 Apr 2019 18:24:56 +0200
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
	<CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
Message-ID: <a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>

Le jeudi 04 avril 2019 ? 19:10 +0300, Yaniv Kaul a ?crit :
> I'm not convinced this is solved. Just had what I believe is a
> similar
> failure:
> 
> *00:12:02.532* A dependency job for rpc-statd.service failed. See
> 'journalctl -xe' for details.*00:12:02.532* mount.nfs: rpc.statd is
> not running but is required for remote locking.*00:12:02.532*
> mount.nfs: Either use '-o nolock' to keep locks local, or start
> statd.*00:12:02.532* mount.nfs: an incorrect mount option was
> specified
> 
> (of course, it can always be my patch!)
> 
> https://build.gluster.org/job/centos7-regression/5384/console

same issue, different builder (206). I will check them all, as the
issue is more widespread than I expected (or it did popup since last
time I checked).


> 
> On Thu, Apr 4, 2019 at 6:56 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
> 
> > Thanks misc. I have always seen a pattern that on a reattempt
> > (recheck
> > centos) the same builder is picked up many time even though it's
> > promised
> > to pick up the builders in a round robin manner.
> > 
> > On Thu, Apr 4, 2019 at 7:24 PM Michael Scherer <mscherer at redhat.com
> > >
> > wrote:
> > 
> > > Le jeudi 04 avril 2019 ? 15:19 +0200, Michael Scherer a ?crit :
> > > > Le jeudi 04 avril 2019 ? 13:53 +0200, Michael Scherer a ?crit :
> > > > > Le jeudi 04 avril 2019 ? 16:13 +0530, Atin Mukherjee a ?crit
> > > > > :
> > > > > > Based on what I have seen that any multi node test case
> > > > > > will fail
> > > > > > and
> > > > > > the
> > > > > > above one is picked first from that group and If I am
> > > > > > correct
> > > > > > none
> > > > > > of
> > > > > > the
> > > > > > code fixes will go through the regression until this is
> > > > > > fixed. I
> > > > > > suspect it
> > > > > > to be an infra issue again. If we look at
> > > > > > https://review.gluster.org/#/c/glusterfs/+/22501/ &
> > > > > > https://build.gluster.org/job/centos7-regression/5382/ peer
> > > > > > handshaking is
> > > > > > stuck as 127.1.1.1 is unable to receive a response back,
> > > > > > did we
> > > > > > end
> > > > > > up
> > > > > > having firewall and other n/w settings screwed up? The test
> > > > > > never
> > > > > > fails
> > > > > > locally.
> > > > > 
> > > > > The firewall didn't change, and since the start has a line:
> > > > > "-A INPUT -i lo -j ACCEPT", so all traffic on the localhost
> > > > > interface
> > > > > work. (I am not even sure that netfilter do anything
> > > > > meaningful on
> > > > > the
> > > > > loopback interface, but maybe I am wrong, and not keen on
> > > > > looking
> > > > > kernel code for that).
> > > > > 
> > > > > 
> > > > > Ping seems to work fine as well, so we can exclude a routing
> > > > > issue.
> > > > > 
> > > > > Maybe we should look at the socket, does it listen to a
> > > > > specific
> > > > > address or not ?
> > > > 
> > > > So, I did look at the 20 first ailure, removed all not related
> > > > to
> > > > rebal-all-nodes-migrate.t and seen all were run on builder203,
> > > > who
> > > > was
> > > > freshly reinstalled. As Deepshika noticed today, this one had a
> > > > issue
> > > > with ipv6, the 2nd issue we were tracking.
> > > > 
> > > > Summary, rpcbind.socket systemd unit listen on ipv6 despites
> > > > ipv6
> > > > being
> > > > disabled, and the fix is to reload systemd. We have so far no
> > > > idea on
> > > > why it happen, but suspect this might be related to the network
> > > > issue
> > > > we did identify, as that happen only after a reboot, that
> > > > happen only
> > > > if a build is cancelled/crashed/aborted.
> > > > 
> > > > I apply the workaround on builder203, so if the culprit is that
> > > > specific issue, guess that's fixed.
> > > > 
> > > > I started a test to see how it go:
> > > > https://build.gluster.org/job/centos7-regression/5383/
> > > 
> > > The test did just pass, so I would assume the problem was local
> > > to
> > > builder203. Not sure why it was always selected, except because
> > > this
> > > was the only one that failed, so was always up for getting new
> > > jobs.
> > > 
> > > Maybe we should increase the number of builder so this doesn't
> > > happen,
> > > as I guess the others builders were busy at that time ?
> > > 
> > > --
> > > Michael Scherer
> > > Sysadmin, Community Infrastructure and Platform, OSAS
> > > 
> > > 
> > > _______________________________________________
> > 
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190404/a71c5e63/attachment.sig>

From rkavunga at redhat.com  Fri Apr  5 05:05:44 2019
From: rkavunga at redhat.com (Rafi Kavungal Chundattu Parambil)
Date: Fri, 5 Apr 2019 01:05:44 -0400 (EDT)
Subject: [Gluster-devel] shd multiplexing patch has introduced coverity
	defects
In-Reply-To: <CAGNCGH1JxOic1_aVFcPMtN9JTGgZBfL78NHoHnKsE=RaGSo=Jg@mail.gmail.com>
References: <CAGNCGH1JxOic1_aVFcPMtN9JTGgZBfL78NHoHnKsE=RaGSo=Jg@mail.gmail.com>
Message-ID: <334597073.13088844.1554440744816.JavaMail.zimbra@redhat.com>

Yes. I will work on this.

Rafi KC

----- Original Message -----
From: "Atin Mukherjee" <amukherj at redhat.com>
To: "Rafi Kavungal Chundattu Parambil" <rkavunga at redhat.com>
Cc: "Gluster Devel" <gluster-devel at gluster.org>
Sent: Thursday, April 4, 2019 11:47:59 AM
Subject: shd multiplexing patch has introduced coverity defects

Based on yesterday's coverity scan report, 6 defects are introduced because
of the shd multiplexing patch. Could you address them, Rafi?

From mscherer at redhat.com  Fri Apr  5 06:45:19 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Fri, 05 Apr 2019 08:45:19 +0200
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
	<CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
	<a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>
Message-ID: <0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>

Le jeudi 04 avril 2019 ? 18:24 +0200, Michael Scherer a ?crit :
> Le jeudi 04 avril 2019 ? 19:10 +0300, Yaniv Kaul a ?crit :
> > I'm not convinced this is solved. Just had what I believe is a
> > similar
> > failure:
> > 
> > *00:12:02.532* A dependency job for rpc-statd.service failed. See
> > 'journalctl -xe' for details.*00:12:02.532* mount.nfs: rpc.statd is
> > not running but is required for remote locking.*00:12:02.532*
> > mount.nfs: Either use '-o nolock' to keep locks local, or start
> > statd.*00:12:02.532* mount.nfs: an incorrect mount option was
> > specified
> > 
> > (of course, it can always be my patch!)
> > 
> > https://build.gluster.org/job/centos7-regression/5384/console
> 
> same issue, different builder (206). I will check them all, as the
> issue is more widespread than I expected (or it did popup since last
> time I checked).

Deepshika did notice that the issue came back on one server
(builder202) after a reboot, so the rpcbind issue is not related to the
network initscript one, so the RCA continue.

We are looking for another workaround involving fiddling with the
socket (until we find why it do use ipv6 at boot, but not after, when
ipv6 is disabled).

Maybe we could run the test suite on a node without all the ipv6
disabling to see if that cause a issue ?

-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190405/0c8f9f17/attachment.sig>

From dkhandel at redhat.com  Fri Apr  5 06:55:02 2019
From: dkhandel at redhat.com (Deepshikha Khandelwal)
Date: Fri, 5 Apr 2019 12:25:02 +0530
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
	<CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
	<a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>
	<0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>
Message-ID: <CABGtVXGcOL0=D-kfkUJ5ZD_odf-LNw-vQtnW=RAo6Tz_-Cp7Ug@mail.gmail.com>

On Fri, Apr 5, 2019 at 12:16 PM Michael Scherer <mscherer at redhat.com> wrote:

> Le jeudi 04 avril 2019 ? 18:24 +0200, Michael Scherer a ?crit :
> > Le jeudi 04 avril 2019 ? 19:10 +0300, Yaniv Kaul a ?crit :
> > > I'm not convinced this is solved. Just had what I believe is a
> > > similar
> > > failure:
> > >
> > > *00:12:02.532* A dependency job for rpc-statd.service failed. See
> > > 'journalctl -xe' for details.*00:12:02.532* mount.nfs: rpc.statd is
> > > not running but is required for remote locking.*00:12:02.532*
> > > mount.nfs: Either use '-o nolock' to keep locks local, or start
> > > statd.*00:12:02.532* mount.nfs: an incorrect mount option was
> > > specified
> > >
> > > (of course, it can always be my patch!)
> > >
> > > https://build.gluster.org/job/centos7-regression/5384/console
> >
> > same issue, different builder (206). I will check them all, as the
> > issue is more widespread than I expected (or it did popup since last
> > time I checked).
>
> Deepshika did notice that the issue came back on one server
> (builder202) after a reboot, so the rpcbind issue is not related to the
> network initscript one, so the RCA continue.
>
> We are looking for another workaround involving fiddling with the
> socket (until we find why it do use ipv6 at boot, but not after, when
> ipv6 is disabled).
>
> Maybe we could run the test suite on a node without all the ipv6
> disabling to see if that cause a issue ?
>
Do our test regression suit started supporting ipv6 now? Else this
investigation would lead to further issues.

> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>
> _______________________________________________
> Gluster-infra mailing list
> Gluster-infra at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190405/580ed9ed/attachment.html>

From ykaul at redhat.com  Fri Apr  5 07:09:44 2019
From: ykaul at redhat.com (Yaniv Kaul)
Date: Fri, 5 Apr 2019 10:09:44 +0300
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <CABGtVXGcOL0=D-kfkUJ5ZD_odf-LNw-vQtnW=RAo6Tz_-Cp7Ug@mail.gmail.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
	<CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
	<a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>
	<0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>
	<CABGtVXGcOL0=D-kfkUJ5ZD_odf-LNw-vQtnW=RAo6Tz_-Cp7Ug@mail.gmail.com>
Message-ID: <CAJgorsbF80x-4jn5NwY7tAG6N5MbGWHDfdVDmhm2+4ggKSCAMw@mail.gmail.com>

On Fri, Apr 5, 2019 at 9:55 AM Deepshikha Khandelwal <dkhandel at redhat.com>
wrote:

>
>
> On Fri, Apr 5, 2019 at 12:16 PM Michael Scherer <mscherer at redhat.com>
> wrote:
>
>> Le jeudi 04 avril 2019 ? 18:24 +0200, Michael Scherer a ?crit :
>> > Le jeudi 04 avril 2019 ? 19:10 +0300, Yaniv Kaul a ?crit :
>> > > I'm not convinced this is solved. Just had what I believe is a
>> > > similar
>> > > failure:
>> > >
>> > > *00:12:02.532* A dependency job for rpc-statd.service failed. See
>> > > 'journalctl -xe' for details.*00:12:02.532* mount.nfs: rpc.statd is
>> > > not running but is required for remote locking.*00:12:02.532*
>> > > mount.nfs: Either use '-o nolock' to keep locks local, or start
>> > > statd.*00:12:02.532* mount.nfs: an incorrect mount option was
>> > > specified
>> > >
>> > > (of course, it can always be my patch!)
>> > >
>> > > https://build.gluster.org/job/centos7-regression/5384/console
>> >
>> > same issue, different builder (206). I will check them all, as the
>> > issue is more widespread than I expected (or it did popup since last
>> > time I checked).
>>
>> Deepshika did notice that the issue came back on one server
>> (builder202) after a reboot, so the rpcbind issue is not related to the
>> network initscript one, so the RCA continue.
>>
>> We are looking for another workaround involving fiddling with the
>> socket (until we find why it do use ipv6 at boot, but not after, when
>> ipv6 is disabled).
>>
>> Maybe we could run the test suite on a node without all the ipv6
>> disabling to see if that cause a issue ?
>>
> Do our test regression suit started supporting ipv6 now? Else this
> investigation would lead to further issues.
>

I suspect not yet. But we certainly would like to, at some point, to ensure
we run with IPv6 as well!
Y.

> --
>> Michael Scherer
>> Sysadmin, Community Infrastructure and Platform, OSAS
>>
>>
>> _______________________________________________
>> Gluster-infra mailing list
>> Gluster-infra at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-infra
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190405/9ad9033a/attachment-0001.html>

From rkavunga at redhat.com  Fri Apr  5 08:02:12 2019
From: rkavunga at redhat.com (RAFI KC)
Date: Fri, 5 Apr 2019 13:32:12 +0530
Subject: [Gluster-devel] shd multiplexing patch has introduced coverity
 defects
In-Reply-To: <334597073.13088844.1554440744816.JavaMail.zimbra@redhat.com>
References: <CAGNCGH1JxOic1_aVFcPMtN9JTGgZBfL78NHoHnKsE=RaGSo=Jg@mail.gmail.com>
	<334597073.13088844.1554440744816.JavaMail.zimbra@redhat.com>
Message-ID: <9e5f6cbf-7b6d-62d8-f386-899ab047be44@redhat.com>

This patch will try to address the issues reported 
https://review.gluster.org/#/c/glusterfs/+/22514/


Regards

Rafi KC

On 4/5/19 10:35 AM, Rafi Kavungal Chundattu Parambil wrote:
> Yes. I will work on this.
>
> Rafi KC
>
> ----- Original Message -----
> From: "Atin Mukherjee" <amukherj at redhat.com>
> To: "Rafi Kavungal Chundattu Parambil" <rkavunga at redhat.com>
> Cc: "Gluster Devel" <gluster-devel at gluster.org>
> Sent: Thursday, April 4, 2019 11:47:59 AM
> Subject: shd multiplexing patch has introduced coverity defects
>
> Based on yesterday's coverity scan report, 6 defects are introduced because
> of the shd multiplexing patch. Could you address them, Rafi?
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel

From nbalacha at redhat.com  Fri Apr  5 11:25:58 2019
From: nbalacha at redhat.com (Nithya Balachandran)
Date: Fri, 5 Apr 2019 16:55:58 +0530
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
	<CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
	<a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>
	<0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>
Message-ID: <CAOUCJ=jdXW5zf0K+EgRrhWRJN5yO7WrCZUF6YeCqidPxuv8+9Q@mail.gmail.com>

On Fri, 5 Apr 2019 at 12:16, Michael Scherer <mscherer at redhat.com> wrote:

> Le jeudi 04 avril 2019 ? 18:24 +0200, Michael Scherer a ?crit :
> > Le jeudi 04 avril 2019 ? 19:10 +0300, Yaniv Kaul a ?crit :
> > > I'm not convinced this is solved. Just had what I believe is a
> > > similar
> > > failure:
> > >
> > > *00:12:02.532* A dependency job for rpc-statd.service failed. See
> > > 'journalctl -xe' for details.*00:12:02.532* mount.nfs: rpc.statd is
> > > not running but is required for remote locking.*00:12:02.532*
> > > mount.nfs: Either use '-o nolock' to keep locks local, or start
> > > statd.*00:12:02.532* mount.nfs: an incorrect mount option was
> > > specified
> > >
> > > (of course, it can always be my patch!)
> > >
> > > https://build.gluster.org/job/centos7-regression/5384/console
> >
> > same issue, different builder (206). I will check them all, as the
> > issue is more widespread than I expected (or it did popup since last
> > time I checked).
>
> Deepshika did notice that the issue came back on one server
> (builder202) after a reboot, so the rpcbind issue is not related to the
> network initscript one, so the RCA continue.
>
> We are looking for another workaround involving fiddling with the
> socket (until we find why it do use ipv6 at boot, but not after, when
> ipv6 is disabled).
>

Could this be relevant?
https://access.redhat.com/solutions/2798411


>
> Maybe we could run the test suite on a node without all the ipv6
> disabling to see if that cause a issue ?
>
> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190405/1733be82/attachment.html>

From srangana at redhat.com  Fri Apr  5 13:36:49 2019
From: srangana at redhat.com (Shyam Ranganathan)
Date: Fri, 5 Apr 2019 09:36:49 -0400
Subject: [Gluster-devel] Announcing Gluster release 4.1.8
Message-ID: <a1fa8dc0-da19-2745-4ec5-ba57c947bd8b@redhat.com>

The Gluster community is pleased to announce the release of Gluster
4.1.8 (packages available at [1]).

Release notes for the release can be found at [2].

Major changes, features and limitations addressed in this release:

None

Thanks,
Gluster community

[1] Packages for 4.1.8:
https://download.gluster.org/pub/gluster/glusterfs/4.1/4.1.8/

[2] Release notes for 4.1.8:
https://docs.gluster.org/en/latest/release-notes/4.1.8/




From srangana at redhat.com  Fri Apr  5 14:33:05 2019
From: srangana at redhat.com (Shyam Ranganathan)
Date: Fri, 5 Apr 2019 10:33:05 -0400
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
Message-ID: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>

Hi,

Expected tagging date for release-6.1 is on April, 10th, 2019.

Please ensure required patches are backported and also are passing
regressions and are appropriately reviewed for easy merging and tagging
on the date.

Thanks,
Shyam

From mscherer at redhat.com  Fri Apr  5 14:40:08 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Fri, 05 Apr 2019 16:40:08 +0200
Subject: [Gluster-devel] [Gluster-infra] rebal-all-nodes-migrate.t
 always fails now
In-Reply-To: <CAOUCJ=jdXW5zf0K+EgRrhWRJN5yO7WrCZUF6YeCqidPxuv8+9Q@mail.gmail.com>
References: <CAGNCGH1ZUrGp_5M2q8n6rb-JmK+dVBZ4HM7AAvi-SUKuuse=QQ@mail.gmail.com>
	<94bd8147c5035da76c3ac3ae90a8a02ed000106a.camel@redhat.com>
	<eafdd551e08844aaad342ace8c03f207d0aa412b.camel@redhat.com>
	<f034b074885cc68b6e47a5644354d7aa961cb457.camel@redhat.com>
	<CAGNCGH2TWkqMrBXvM2+ePnhmzENBkaczemyjb99mxHbWfaFbkw@mail.gmail.com>
	<CAJgorsY-djXmSKDgsD4DGrD1SEaZ43k9-fxNpTCNceLPPPxLuQ@mail.gmail.com>
	<a6840369e5371e74523a5f23a470e6ad7bc81349.camel@redhat.com>
	<0ca34e42063ad77f323155c85a7bb3ba7a79931b.camel@redhat.com>
	<CAOUCJ=jdXW5zf0K+EgRrhWRJN5yO7WrCZUF6YeCqidPxuv8+9Q@mail.gmail.com>
Message-ID: <090785225412c2b5b269454f8812d0a165aea62d.camel@redhat.com>

Le vendredi 05 avril 2019 ? 16:55 +0530, Nithya Balachandran a ?crit :
> On Fri, 5 Apr 2019 at 12:16, Michael Scherer <mscherer at redhat.com>
> wrote:
> 
> > Le jeudi 04 avril 2019 ? 18:24 +0200, Michael Scherer a ?crit :
> > > Le jeudi 04 avril 2019 ? 19:10 +0300, Yaniv Kaul a ?crit :
> > > > I'm not convinced this is solved. Just had what I believe is a
> > > > similar
> > > > failure:
> > > > 
> > > > *00:12:02.532* A dependency job for rpc-statd.service failed.
> > > > See
> > > > 'journalctl -xe' for details.*00:12:02.532* mount.nfs:
> > > > rpc.statd is
> > > > not running but is required for remote locking.*00:12:02.532*
> > > > mount.nfs: Either use '-o nolock' to keep locks local, or start
> > > > statd.*00:12:02.532* mount.nfs: an incorrect mount option was
> > > > specified
> > > > 
> > > > (of course, it can always be my patch!)
> > > > 
> > > > https://build.gluster.org/job/centos7-regression/5384/console
> > > 
> > > same issue, different builder (206). I will check them all, as
> > > the
> > > issue is more widespread than I expected (or it did popup since
> > > last
> > > time I checked).
> > 
> > Deepshika did notice that the issue came back on one server
> > (builder202) after a reboot, so the rpcbind issue is not related to
> > the
> > network initscript one, so the RCA continue.
> > 
> > We are looking for another workaround involving fiddling with the
> > socket (until we find why it do use ipv6 at boot, but not after,
> > when
> > ipv6 is disabled).
> > 
> 
> Could this be relevant?
> https://access.redhat.com/solutions/2798411

Good catch.

So, we already do that, Nigel took care of that (after 2 days of
research). But I didn't knew the exact symptoms, and decided to double
check just in case.

And... there is no sysctl.conf in the initrd. Running dracut -v -f do
not change anything.

Running "dracut -v -f -H" take care of that (and this fix the problem),
but:
- our ansible script already run that
- -H is hostonly, which is already the default on EL7 according to the
doc.  

However, if dracut-config-generic is installed, it doesn't build a
hostonly initrd, and so do not include the sysctl.conf file (who break
rpcbnd, who break the test suite).

And for some reason, it is installed the image in ec2 (likely default),
but not by default on the builders.

So what happen is that after a kernel upgrade, dracut rebuild a generic
initrd instead of a hostonly one, who break things. And kernel was
likely upgraded recently (and upgrade happen nightly (for some value of
"night"), so we didn't see that earlier, nor with a fresh system.


So now, we have several solution:
- be explicit on using hostonly in dracut, so this doesn't happen again
(or not for this reason)

- disable ipv6 in rpcbind in a cleaner way (to be tested)

- get the test suite work with ip v6

In the long term, I also want to monitor the processes, but for that, I
need a VPN between the nagios server and ec2, and that project got
blocked by several issues (like EC2 not support ecdsa keys, and we use
that for ansible, so we have to come back to RSA for full automated
deployment, and openvon requires to use certificates, so I need a newer
python openssl for doing what I want, and RHEL 7 is too old, etc, etc).

As the weekend approach for me, I just rebuilt the initrd for the time
being. I guess forcing hostonly is the safest fix for now, but this
will be for monday.
-- 
Michael Scherer
Sysadmin, Community Infrastructure and Platform, OSAS


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190405/acdfd6a5/attachment.sig>

From phlogistonjohn at asynchrono.us  Fri Apr  5 19:30:29 2019
From: phlogistonjohn at asynchrono.us (John Mulligan)
Date: Fri, 05 Apr 2019 15:30:29 -0400
Subject: [Gluster-devel] Heketi v9.0.0 available for download
Message-ID: <2297373.DkY5oybNEL@abydos>

Heketi v9.0.0 is now available [1].

This is the new stable version of Heketi.

Major additions in this release:
* Limit volumes per Gluster cluster
* Prevent server from starting if db has unknown dbattributes
* Support a default admin mode option
* Add an option to enable strict zone checking on volume creation
* Add automatic pending operation clean-up functionality
* Configurable device formatting parameters
* Add consistency check feature and state examiner debugging tools
* The faulty and non-functional "db delete-pending-entries" command has been 
removed

This release contains numerous stability and bug fixes. A more detailed 
changelog is available at the release page [1].


-- John M. on behalf of the Heketi team


[1] https://github.com/heketi/heketi/releases/tag/v9.0.0



From ravishankar at redhat.com  Sat Apr  6 06:29:25 2019
From: ravishankar at redhat.com (Ravishankar N)
Date: Sat, 6 Apr 2019 11:59:25 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
Message-ID: <e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>

Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in 
case anyone wants to add blocker bugs.


On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
> Hi,
>
> Expected tagging date for release-6.1 is on April, 10th, 2019.
>
> Please ensure required patches are backported and also are passing
> regressions and are appropriately reviewed for easy merging and tagging
> on the date.
>
> Thanks,
> Shyam
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel

From amukherj at redhat.com  Sat Apr  6 08:38:44 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Sat, 6 Apr 2019 14:08:44 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
Message-ID: <CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>

Hi Mohit,

https://review.gluster.org/22495 should get into 6.1 as it?s a regression.
Can you please attach the respective bug to the tracker Ravi pointed out?


On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com> wrote:

> Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in
> case anyone wants to add blocker bugs.
>
>
> On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
> > Hi,
> >
> > Expected tagging date for release-6.1 is on April, 10th, 2019.
> >
> > Please ensure required patches are backported and also are passing
> > regressions and are appropriately reviewed for easy merging and tagging
> > on the date.
> >
> > Thanks,
> > Shyam
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
> --
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190406/5609a830/attachment.html>

From jenkins at build.gluster.org  Mon Apr  8 01:45:03 2019
From: jenkins at build.gluster.org (jenkins at build.gluster.org)
Date: Mon, 8 Apr 2019 01:45:03 +0000 (UTC)
Subject: [Gluster-devel] Weekly Untriaged Bugs
Message-ID: <825338949.63.1554687903451.JavaMail.jenkins@jenkins-el7.rht.gluster.org>

[...truncated 6 lines...]
https://bugzilla.redhat.com/1688226 / core: Brick Still Died After Restart Glusterd & Glusterfsd Services
https://bugzilla.redhat.com/1695416 / core: client log flooding with intentional socket shutdown message when a brick is down
https://bugzilla.redhat.com/1691833 / core: Client sends 128KByte network packet for 0 length file copy
https://bugzilla.redhat.com/1695480 / core: Global Thread Pool
https://bugzilla.redhat.com/1694943 / core: parallel-readdir slows down directory listing
https://bugzilla.redhat.com/1696721 / geo-replication: geo-replication failing after upgrade from 5.5 to 6.0
https://bugzilla.redhat.com/1694637 / geo-replication: Geo-rep: Rename to an existing file name destroys its content on slave
https://bugzilla.redhat.com/1689981 / geo-replication: OSError: [Errno 1] Operation not permitted - failing with socket files?
https://bugzilla.redhat.com/1694139 / glusterd: Error waiting for job 'heketi-storage-copy-job' to complete on one-node k3s deployment.
https://bugzilla.redhat.com/1695099 / glusterd: The number of glusterfs processes keeps increasing, using all available resources
https://bugzilla.redhat.com/1690454 / posix-acl: mount-shared-storage.sh does not implement mount options
https://bugzilla.redhat.com/1696518 / project-infrastructure: builder203 does not have a valid hostname set
https://bugzilla.redhat.com/1691617 / project-infrastructure: clang-scan tests are failing nightly.
https://bugzilla.redhat.com/1691357 / project-infrastructure: core archive link from regression jobs throw not found error
https://bugzilla.redhat.com/1692349 / project-infrastructure: gluster-csi-containers job is failing
https://bugzilla.redhat.com/1693385 / project-infrastructure: request to change the version of fedora in fedora-smoke-job
https://bugzilla.redhat.com/1693295 / project-infrastructure: rpc.statd not started on builder204.aws.gluster.org
https://bugzilla.redhat.com/1691789 / project-infrastructure: rpc-statd service stops on AWS builders
https://bugzilla.redhat.com/1695484 / project-infrastructure: smoke fails with "Build root is locked by another process"
https://bugzilla.redhat.com/1693184 / replicate: A brick process(glusterfsd) died with 'memory violation'
https://bugzilla.redhat.com/1696075 / replicate: Client lookup is unable to heal missing directory GFID entry
https://bugzilla.redhat.com/1696633 / tests: GlusterFs v4.1.5 Tests from /tests/bugs/ module failing on Intel
https://bugzilla.redhat.com/1694976 / unclassified: On Fedora 29 GlusterFS 4.1 repo has bad/missing rpm signs
[...truncated 2 lines...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 2926 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190408/49018fe2/attachment.obj>

From cynthia.zhou at nokia-sbell.com  Mon Apr  8 02:12:25 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 8 Apr 2019 02:12:25 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15
Message-ID: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>

Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190408/a79e3b15/attachment-0001.html>

From 1173701037 at qq.com  Mon Apr  8 06:41:24 2019
From: 1173701037 at qq.com (=?utf-8?B?UFND?=)
Date: Mon, 8 Apr 2019 14:41:24 +0800
Subject: [Gluster-devel] Hello,
	I have a question about the erasure code translator,
	hope someone give me some advice, thank you!
Message-ID: <tencent_4E1C86F16E103EADDE297F98C962843C7C09@qq.com>

Hi, I am a storage software coder who is interested in Gluster. I am trying to improve the read/write performance of it.  
 

 
 
I noticed that gluster is using Vandermonde matrix in erasure code encoding and decoding process. However, it is quite complicate to generate inverse matrix of a Vandermonde matrix, which is necessary for decode. The cost is O(n?).  
 

 
 
Use a Cauchy matrix, can greatly cut down the cost of the process to find an inverse matrix. Which is O(n?).
 

 
 
I use intel storage accelerate library to replace the original ec encode/decode part of gluster. And it reduce the encode and decode time to about 50% of the original one.
 

 
 
However, when I test the whole system. The read/write performance is almost the same as the original gluster.  
 

 
 
I test it on three machines as servers. Each one had two bricks, both of them are SSD. So the total  amount of bricks is 6. Use two of them as coding bricks. That is a 4+2 disperse volume configure.
 

 
 
The capability of network card is 10000Mbps. Theoretically it can support read and write with the speed faster than 1000MB/s.
 

 
 
The actually performance of read is about 492MB/s.
 
The actually performance of write is about 336MB/s.
 

 
 
While the original one read at 461MB/s, write at 322MB/s
 

 
 
Is there someone who can give me some advice about how to improve its performance? Which part is the critical defect on its performance if it?s not the ec translator?  
 

 
 
I did a time count on translators. It show me EC translator just take 7% in the whole read\write process. Even though I knew that some translators are run asynchronous, so the real percentage can be some how lager than that.  
 

 
 
Sincerely thank you for your patient to read my question!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190408/9a999642/attachment.html>

From jahernan at redhat.com  Mon Apr  8 08:02:00 2019
From: jahernan at redhat.com (Xavi Hernandez)
Date: Mon, 8 Apr 2019 10:02:00 +0200
Subject: [Gluster-devel] Hello,
 I have a question about the erasure code translator,
 hope someone give me some advice, thank you!
In-Reply-To: <tencent_4E1C86F16E103EADDE297F98C962843C7C09@qq.com>
References: <tencent_4E1C86F16E103EADDE297F98C962843C7C09@qq.com>
Message-ID: <CALtvDK4PEiAwcfk=XuC9L6a-n4a9r=F2s3zL=h1VOu9bx+A0dw@mail.gmail.com>

Hi,

On Mon, Apr 8, 2019 at 8:50 AM PSC <1173701037 at qq.com> wrote:

> Hi, I am a storage software coder who is interested in Gluster. I am
> trying to improve the read/write performance of it.
>
>
> I noticed that gluster is using Vandermonde matrix in erasure code
> encoding and decoding process. However, it is quite complicate to generate
> inverse matrix of a Vandermonde matrix, which is necessary for decode. The
> cost is O(n?).
>

That's not true, actually. A Vandermonde matrix can be inverted in O(n^2),
as the code currently does (look at ec_method_matrix_inverse() in
ec-method.c). Additionally, current code does caching of inverted matrices,
so in normal circumstances there shouldn't be many inverse computations.
Only when something changes (a brick dies or comes online), a new inverted
matrix could be needed.


>
> Use a Cauchy matrix, can greatly cut down the cost of the process to find
> an inverse matrix. Which is O(n?).
>
>
> I use intel storage accelerate library to replace the original ec
> encode/decode part of gluster. And it reduce the encode and decode time to
> about 50% of the original one.
>

How do you test that ? I also did some tests long ago and I didn't observe
that difference.

Doing a raw test of encoding/decoding performance of the current code using
Intel AVX2 extensions, it's able to process 7.6 GiB/s on a single core of
an Intel Xeon Silver 4114 when L1 cache is used. Without relying on
internal cache, it performs at 3.9 GiB/s. Does ISA-L provide better
performance for a matrix of the same size (4+2 non-systematic matrix) ?

>
> However, when I test the whole system. The read/write performance is
> almost the same as the original gluster.
>

Yes, there are many more things involved in the read and write operations
in gluster. For the particular case of EC, having to deal with many bricks
simultaneously (6 in this case) means that it's very sensitive to network
latency and communications delays, and this is probably one of the biggest
contributors. There some other small latencies added by other xlators.

>
> I test it on three machines as servers. Each one had two bricks, both of
> them are SSD. So the total amount of bricks is 6. Use two of them as coding
> bricks. That is a 4+2 disperse volume configure.
>
>
> The capability of network card is 10000Mbps. Theoretically it can support
> read and write with the speed faster than 1000MB/s.
>
>
> The actually performance of read is about 492MB/s.
>
> The actually performance of write is about 336MB/s.
>
>
> While the original one read at 461MB/s, write at 322MB/s
>
>
> Is there someone who can give me some advice about how to improve its
> performance? Which part is the critical defect on its performance if it?s
> not the ec translator?
>
>
> I did a time count on translators. It show me EC translator just take 7%
> in the whole read\write process. Even though I knew that some translators
> are run asynchronous, so the real percentage can be some how lager than
> that.
>
>
> Sincerely thank you for your patient to read my question!
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190408/ed753049/attachment.html>

From srakonde at redhat.com  Mon Apr  8 08:57:35 2019
From: srakonde at redhat.com (Sanju Rakonde)
Date: Mon, 8 Apr 2019 14:27:35 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
Message-ID: <CABj3WN2Jv0EPgUq3AcXHC02JtCGTwVysMb-cXk5y8ZrMRBQz0w@mail.gmail.com>

Can you please capture output of "pstack $(pidof glusterd)" and send it to
us? We need to capture this information when glusterd is struck.

On Mon, Apr 8, 2019 at 8:05 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177",
> __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced
> = -1, used = -1}
>
> (gdb) quit
>
> A debugging session is active.
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190408/bf16b666/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr  8 09:01:19 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 8 Apr 2019 09:01:19 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CABj3WN2Jv0EPgUq3AcXHC02JtCGTwVysMb-cXk5y8ZrMRBQz0w@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CABj3WN2Jv0EPgUq3AcXHC02JtCGTwVysMb-cXk5y8ZrMRBQz0w@mail.gmail.com>
Message-ID: <dbb1694cba2f4925a35b0ff6ba4f2c7b@nokia-sbell.com>

Hi,
The env is not there anymore, but I have collected the thread stack trace of glusterd with command ?thread apply all bt?
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
Missing separate debuginfos, use: dnf debuginfo-install rcp-pack-glusterfs-1.12.0_1_gc999db1-RCP2.wf29.x86_64
(gdb) thread apply all bt

Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f9ee49b3700 (LWP 1931)):
#0  0x00007f9ee9fd45bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9ee5e651b9 in hooks_worker (args=0x1813000) at glusterd-hooks.c:529
#2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f9ee692e700 (LWP 1762)):
#0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808e00) at syncop.c:603
#2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808e00) at syncop.c:695
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f9ee712f700 (LWP 1761)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808a40) at syncop.c:603
#2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808a40) at syncop.c:695
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f9ee7930700 (LWP 1760)):
#0  0x00007f9ee98725d0 in nanosleep () from /lib64/libc.so.6
#1  0x00007f9ee98724aa in sleep () from /lib64/libc.so.6
#2  0x00007f9eeb247fdf in pool_sweeper (arg=0x0) at mem-pool.c:481
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f9ee8131700 (LWP 1759)):
#0  0x00007f9ee97e3d7c in sigtimedwait () from /lib64/libc.so.6
#1  0x00007f9ee9fd8bac in sigwait () from /lib64/libpthread.so.0
#2  0x0000000000409ed7 in glusterfs_sigwaiter ()
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9ee8932700 (LWP 1758)):
#0  0x00007f9ee9fd83b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f9eeb224545 in gf_timer_proc (data=0x1808580) at timer.c:164
#2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f9eeb707780 (LWP 1757)):
#0  0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x00007f9eeb282b09 in event_dispatch_epoll (event_pool=0x17feb00) at event-epoll.c:746
#2  0x00007f9eeb246786 in event_dispatch (event_pool=0x17feb00) at event.c:124
#3  0x000000000040ab95 in main ()
(gdb)
(gdb)
(gdb) q!
A syntax error in expression, near `'.
(gdb) quit

From: Sanju Rakonde <srakonde at redhat.com>
Sent: Monday, April 08, 2019 4:58 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15

Can you please capture output of "pstack $(pidof glusterd)" and send it to us? We need to capture this information when glusterd is struck.

On Mon, Apr 8, 2019 at 8:05 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
(gdb) quit
A debugging session is active.
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190408/4dc343a3/attachment-0001.html>

From srakonde at redhat.com  Tue Apr  9 07:08:16 2019
From: srakonde at redhat.com (Sanju Rakonde)
Date: Tue, 9 Apr 2019 12:38:16 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <dbb1694cba2f4925a35b0ff6ba4f2c7b@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CABj3WN2Jv0EPgUq3AcXHC02JtCGTwVysMb-cXk5y8ZrMRBQz0w@mail.gmail.com>
	<dbb1694cba2f4925a35b0ff6ba4f2c7b@nokia-sbell.com>
Message-ID: <CABj3WN10kB1ceXzH71N4gUcVaHGZq5yHRrJTqGMS7whr2efhGg@mail.gmail.com>

Hello,

I'm unable to figure out the issue by just looking at the backtrace. You
might be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1650115. If
you come across the same problem in future, please capture pstack output
and share it with us.

On Mon, Apr 8, 2019 at 2:31 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi,
>
> The env is not there anymore, but I have collected the thread stack trace
> of glusterd with command ?thread apply all bt?
>
> Using host libthread_db library "/lib64/libthread_db.so.1".
>
> 0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
>
> Missing separate debuginfos, use: dnf debuginfo-install
> rcp-pack-glusterfs-1.12.0_1_gc999db1-RCP2.wf29.x86_64
>
> (gdb) thread apply all bt
>
>
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 7 (Thread 0x7f9ee49b3700 (LWP 1931)):
>
> #0  0x00007f9ee9fd45bc in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #1  0x00007f9ee5e651b9 in hooks_worker (args=0x1813000) at
> glusterd-hooks.c:529
>
> #2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 6 (Thread 0x7f9ee692e700 (LWP 1762)):
>
> #0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808e00) at syncop.c:603
>
> #2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808e00) at
> syncop.c:695
>
> #3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 5 (Thread 0x7f9ee712f700 (LWP 1761)):
>
> ---Type <return> to continue, or q <return> to quit---
>
> #0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808a40) at syncop.c:603
>
> #2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808a40) at
> syncop.c:695
>
> #3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 4 (Thread 0x7f9ee7930700 (LWP 1760)):
>
> #0  0x00007f9ee98725d0 in nanosleep () from /lib64/libc.so.6
>
> #1  0x00007f9ee98724aa in sleep () from /lib64/libc.so.6
>
> #2  0x00007f9eeb247fdf in pool_sweeper (arg=0x0) at mem-pool.c:481
>
> #3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 3 (Thread 0x7f9ee8131700 (LWP 1759)):
>
> #0  0x00007f9ee97e3d7c in sigtimedwait () from /lib64/libc.so.6
>
> #1  0x00007f9ee9fd8bac in sigwait () from /lib64/libpthread.so.0
>
> #2  0x0000000000409ed7 in glusterfs_sigwaiter ()
>
> #3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 2 (Thread 0x7f9ee8932700 (LWP 1758)):
>
> #0  0x00007f9ee9fd83b0 in nanosleep () from /lib64/libpthread.so.0
>
> #1  0x00007f9eeb224545 in gf_timer_proc (data=0x1808580) at timer.c:164
>
> #2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 1 (Thread 0x7f9eeb707780 (LWP 1757)):
>
> #0  0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from
> /lib64/libpthread.so.0
>
> #1  0x00007f9eeb282b09 in event_dispatch_epoll (event_pool=0x17feb00) at
> event-epoll.c:746
>
> #2  0x00007f9eeb246786 in event_dispatch (event_pool=0x17feb00) at
> event.c:124
>
> #3  0x000000000040ab95 in main ()
>
> (gdb)
>
> (gdb)
>
> (gdb) q!
>
> A syntax error in expression, near `'.
>
> (gdb) quit
>
>
>
> *From:* Sanju Rakonde <srakonde at redhat.com>
> *Sent:* Monday, April 08, 2019 4:58 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterd stuck for glusterfs with version
> 3.12.15
>
>
>
> Can you please capture output of "pstack $(pidof glusterd)" and send it to
> us? We need to capture this information when glusterd is struck.
>
>
>
> On Mon, Apr 8, 2019 at 8:05 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@
> \000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}},
> ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
>
> (gdb) quit
>
> A debugging session is active.
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
>
> --
>
> Thanks,
>
> Sanju
>


-- 
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190409/3574349c/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Tue Apr  9 07:18:30 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Tue, 9 Apr 2019 07:18:30 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CABj3WN10kB1ceXzH71N4gUcVaHGZq5yHRrJTqGMS7whr2efhGg@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CABj3WN2Jv0EPgUq3AcXHC02JtCGTwVysMb-cXk5y8ZrMRBQz0w@mail.gmail.com>
	<dbb1694cba2f4925a35b0ff6ba4f2c7b@nokia-sbell.com>
	<CABj3WN10kB1ceXzH71N4gUcVaHGZq5yHRrJTqGMS7whr2efhGg@mail.gmail.com>
Message-ID: <04bcfe872c314b7dbf7860dab74f8977@nokia-sbell.com>

Hi,
Gluster brick multiplex is not enabled in my env.
My point is that : socket notify another around of poll in too early actually this round of poll in is not finished, thread 9 has not finished poll in while thread 8 has begun another round of poll in, thread 8 has called iobref_unref and iobref_unref does not release
iobref->lock , so thread 9 may block.

cynthia
From: Sanju Rakonde <srakonde at redhat.com>
Sent: Tuesday, April 09, 2019 3:08 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15

Hello,

I'm unable to figure out the issue by just looking at the backtrace. You might be hitting https://bugzilla.redhat.com/show_bug.cgi?id=1650115. If you come across the same problem in future, please capture pstack output and share it with us.

On Mon, Apr 8, 2019 at 2:31 PM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
The env is not there anymore, but I have collected the thread stack trace of glusterd with command ?thread apply all bt?
Using host libthread_db library "/lib64/libthread_db.so.1".
0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
Missing separate debuginfos, use: dnf debuginfo-install rcp-pack-glusterfs-1.12.0_1_gc999db1-RCP2.wf29.x86_64
(gdb) thread apply all bt

Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f9ee49b3700 (LWP 1931)):
#0  0x00007f9ee9fd45bc in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9ee5e651b9 in hooks_worker (args=0x1813000) at glusterd-hooks.c:529
#2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f9ee692e700 (LWP 1762)):
#0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808e00) at syncop.c:603
#2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808e00) at syncop.c:695
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f9ee712f700 (LWP 1761)):
---Type <return> to continue, or q <return> to quit---
#0  0x00007f9ee9fd497a in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f9eeb25d904 in syncenv_task (proc=0x1808a40) at syncop.c:603
#2  0x00007f9eeb25db9f in syncenv_processor (thdata=0x1808a40) at syncop.c:695
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f9ee7930700 (LWP 1760)):
#0  0x00007f9ee98725d0 in nanosleep () from /lib64/libc.so.6
#1  0x00007f9ee98724aa in sleep () from /lib64/libc.so.6
#2  0x00007f9eeb247fdf in pool_sweeper (arg=0x0) at mem-pool.c:481
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 3 (Thread 0x7f9ee8131700 (LWP 1759)):
#0  0x00007f9ee97e3d7c in sigtimedwait () from /lib64/libc.so.6
#1  0x00007f9ee9fd8bac in sigwait () from /lib64/libpthread.so.0
#2  0x0000000000409ed7 in glusterfs_sigwaiter ()
#3  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#4  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9ee8932700 (LWP 1758)):
#0  0x00007f9ee9fd83b0 in nanosleep () from /lib64/libpthread.so.0
#1  0x00007f9eeb224545 in gf_timer_proc (data=0x1808580) at timer.c:164
#2  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#3  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f9eeb707780 (LWP 1757)):
#0  0x00007f9ee9fcfa3d in __pthread_timedjoin_ex () from /lib64/libpthread.so.0
#1  0x00007f9eeb282b09 in event_dispatch_epoll (event_pool=0x17feb00) at event-epoll.c:746
#2  0x00007f9eeb246786 in event_dispatch (event_pool=0x17feb00) at event.c:124
#3  0x000000000040ab95 in main ()
(gdb)
(gdb)
(gdb) q!
A syntax error in expression, near `'.
(gdb) quit

From: Sanju Rakonde <srakonde at redhat.com<mailto:srakonde at redhat.com>>
Sent: Monday, April 08, 2019 4:58 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterd stuck for glusterfs with version 3.12.15

Can you please capture output of "pstack $(pidof glusterd)" and send it to us? We need to capture this information when glusterd is struck.

On Mon, Apr 8, 2019 at 8:05 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
(gdb) quit
A debugging session is active.
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Thanks,
Sanju


--
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190409/ab5dbcb1/attachment-0001.html>

From rgowdapp at redhat.com  Tue Apr  9 07:52:26 2019
From: rgowdapp at redhat.com (Raghavendra Gowdappa)
Date: Tue, 9 Apr 2019 13:22:26 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
Message-ID: <CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>

On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177",
> __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced
> = -1, used = -1}
>

looks like the iobref is corrupted here. It seems to be a use-after-free
issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
>
> A debugging session is active.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190409/7589ef6b/attachment.html>

From cynthia.zhou at nokia-sbell.com  Tue Apr  9 07:57:15 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Tue, 9 Apr 2019 07:57:15 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
Message-ID: <ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>

Can you figure out some possible reason why iobref is corrupted, is it possible that thread 8 has called poll in and iobref has been relased, but the lock within it is not properly released (as I can not find any free lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy again, and so stuck on this lock
Also, there should not be two thread handling the same socket at the same time, although there has been a patch claimed to tackle this issue.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com>
Sent: Tuesday, April 09, 2019 3:52 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: gluster-devel at gluster.org
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

looks like the iobref is corrupted here. It seems to be a use-after-free issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190409/09786335/attachment-0001.html>

From spisla80 at gmail.com  Wed Apr 10 10:09:23 2019
From: spisla80 at gmail.com (David Spisla)
Date: Wed, 10 Apr 2019 12:09:23 +0200
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
References: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
Message-ID: <CAJyj9j_A-umyiLgqeJBvgGFBEeDH2CbU5iPUpedb+2JiMywHjg@mail.gmail.com>

Hello Martin,

look here:
https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/pdf/administration_guide/Red_Hat_Gluster_Storage-3.4-Administration_Guide-en-US.pdf
on page 324. There is a manual how to replace a brick in case of a hardware
failure

Regards
David Spisla

Am Mi., 10. Apr. 2019 um 11:42 Uhr schrieb Martin Toth <snowmailer at gmail.com
>:

> Hi all,
>
> I am running replica 3 gluster with 3 bricks. One of my servers failed -
> all disks are showing errors and raid is in fault state.
>
> Type: Replicate
> Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
> Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is down
> Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>
> So one of my bricks is totally failed (node2). It went down and all data
> are lost (failed raid on node2). Now I am running only two bricks on 2
> servers out from 3.
> This is really critical problem for us, we can lost all data. I want to
> add new disks to node2, create new raid array on them and try to replace
> failed brick on this node.
>
> What is the procedure of replacing Brick2 on node2, can someone advice? I
> can?t find anything relevant in documentation.
>
> Thanks in advance,
> Martin
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/718e869f/attachment.html>

From rkavunga at redhat.com  Wed Apr 10 10:16:25 2019
From: rkavunga at redhat.com (RAFI KC)
Date: Wed, 10 Apr 2019 15:46:25 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
 failed node (peer)
In-Reply-To: <CAJyj9j_A-umyiLgqeJBvgGFBEeDH2CbU5iPUpedb+2JiMywHjg@mail.gmail.com>
References: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
	<CAJyj9j_A-umyiLgqeJBvgGFBEeDH2CbU5iPUpedb+2JiMywHjg@mail.gmail.com>
Message-ID: <3292fe0e-f164-43c0-f922-fa2176158749@redhat.com>

reset brick is another way of replacing a brick. this usually helpful, 
when you want to replace the brick with same name. You can find the 
documentation here 
https://docs.gluster.org/en/latest/release-notes/3.9.0/#introducing-reset-brick-command.

In your case, I think you can use replace brick. So you can initiate a 
reset-brick start, then you have to replace your failed disk and create 
new brick with same name . Once you have healthy disk and brick, you can 
commit the reset-brick.


Let's know if you have any question,


Rafi KC


On 4/10/19 3:39 PM, David Spisla wrote:
> Hello Martin,
>
> look here:
> https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/pdf/administration_guide/Red_Hat_Gluster_Storage-3.4-Administration_Guide-en-US.pdf
> on page 324. There is a manual how to replace a brick in case of a 
> hardware failure
>
> Regards
> David Spisla
>
> Am Mi., 10. Apr. 2019 um 11:42?Uhr schrieb Martin Toth 
> <snowmailer at gmail.com <mailto:snowmailer at gmail.com>>:
>
>     Hi all,
>
>     I am running replica 3 gluster with 3 bricks. One of my servers
>     failed - all disks are showing errors and raid is in fault state.
>
>     Type: Replicate
>     Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
>     Status: Started
>     Number of Bricks: 1 x 3 = 3
>     Transport-type: tcp
>     Bricks:
>     Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
>     Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick
>     is down
>     Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>
>     So one of my bricks is totally failed (node2). It went down and
>     all data are lost (failed raid on node2). Now I am running only
>     two bricks on 2 servers out from 3.
>     This is really critical problem for us, we can lost all data. I
>     want to add new disks to node2, create new raid array on them and
>     try to replace failed brick on this node.
>
>     What is the procedure of replacing Brick2 on node2, can someone
>     advice? I can?t find anything relevant in documentation.
>
>     Thanks in advance,
>     Martin
>     _______________________________________________
>     Gluster-users mailing list
>     Gluster-users at gluster.org <mailto:Gluster-users at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/45a493c1/attachment.html>

From ksubrahm at redhat.com  Wed Apr 10 10:20:40 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Wed, 10 Apr 2019 15:50:40 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
References: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
Message-ID: <CAHRDaUHfXpjSOt81J16aEymnBqBJt-np5_p9erradu1Y5iV=NQ@mail.gmail.com>

Hi Martin,

After you add the new disks and creating raid array, you can run the
following command to replace the old brick with new one:

- If you are going to use a different name to the new brick you can run
gluster volume replace-brick <volname> <old-brick> <new-brick> commit force

- If you are planning to use the same name for the new brick as well then
you can use
gluster volume reset-brick <volname> <old-brick> <new-brick> commit force
Here old-brick & new-brick's hostname &  path should be same.

After replacing the brick, make sure the brick comes online using volume
status.
Heal should automatically start, you can check the heal status to see all
the files gets replicated to the newly added brick. If it does not start
automatically, you can manually start that by running gluster volume heal
<volname>.

HTH,
Karthik

On Wed, Apr 10, 2019 at 3:13 PM Martin Toth <snowmailer at gmail.com> wrote:

> Hi all,
>
> I am running replica 3 gluster with 3 bricks. One of my servers failed -
> all disks are showing errors and raid is in fault state.
>
> Type: Replicate
> Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
> Status: Started
> Number of Bricks: 1 x 3 = 3
> Transport-type: tcp
> Bricks:
> Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
> Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is down
> Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>
> So one of my bricks is totally failed (node2). It went down and all data
> are lost (failed raid on node2). Now I am running only two bricks on 2
> servers out from 3.
> This is really critical problem for us, we can lost all data. I want to
> add new disks to node2, create new raid array on them and try to replace
> failed brick on this node.
>
> What is the procedure of replacing Brick2 on node2, can someone advice? I
> can?t find anything relevant in documentation.
>
> Thanks in advance,
> Martin
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/a2b266d9/attachment.html>

From nbalacha at redhat.com  Wed Apr 10 11:37:52 2019
From: nbalacha at redhat.com (Nithya Balachandran)
Date: Wed, 10 Apr 2019 17:07:52 +0530
Subject: [Gluster-devel] SHD crash in
	https://build.gluster.org/job/centos7-regression/5510/consoleFull
Message-ID: <CAOUCJ=hXpwP5Vrvx=cVcEgDZHSK4sMv4=KLvjvuxKY7fgqw08w@mail.gmail.com>

Hi,

My patch is unlikely to have caused this as the changes are only in dht.
Can someone take a look?

Thanks,
Nithya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/d5a6a5f8/attachment.html>

From atin.mukherjee83 at gmail.com  Wed Apr 10 12:17:16 2019
From: atin.mukherjee83 at gmail.com (Atin Mukherjee)
Date: Wed, 10 Apr 2019 17:47:16 +0530
Subject: [Gluster-devel] SHD crash in
	https://build.gluster.org/job/centos7-regression/5510/consoleFull
In-Reply-To: <CAOUCJ=hXpwP5Vrvx=cVcEgDZHSK4sMv4=KLvjvuxKY7fgqw08w@mail.gmail.com>
References: <CAOUCJ=hXpwP5Vrvx=cVcEgDZHSK4sMv4=KLvjvuxKY7fgqw08w@mail.gmail.com>
Message-ID: <CAGkR8FPCpTFOKdWGY-MBWUHR=4NNtZs0XPwN4y368a=NmN3gnA@mail.gmail.com>

Rafi mentioned to me earlier that this will be fixed through
https://review.gluster.org/22468 . This crash is more often seen in the
nightly regression these days. Patch needs review and I'd request the
respective maintainers to take a look at it.

On Wed, Apr 10, 2019 at 5:08 PM Nithya Balachandran <nbalacha at redhat.com>
wrote:

> Hi,
>
> My patch is unlikely to have caused this as the changes are only in dht.
> Can someone take a look?
>
> Thanks,
> Nithya
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/3023905b/attachment-0001.html>

From ksubrahm at redhat.com  Wed Apr 10 12:26:36 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Wed, 10 Apr 2019 17:56:36 +0530
Subject: [Gluster-devel] [Gluster-users] [External] Replica 3 - how to
	replace failed node (peer)
In-Reply-To: <804C3826-0173-431C-A286-085E7E582212@gmail.com>
References: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
	<CAH2S_oReMcrObVq2Cr7yC7+CP-iqzxa1cCVt8pqeOSHyHUMQLQ@mail.gmail.com>
	<804C3826-0173-431C-A286-085E7E582212@gmail.com>
Message-ID: <CAHRDaUHRDTcPhURuzPibRd9qqE785YzDHyecCDKaiG5JSqzHqw@mail.gmail.com>

Hi Martin,

The reset-brick command is introduced in 3.9.0 and not present in 3.7.6.
You can try using the same replace-brick command with the force option even
if you want to use the same name for the brick being replaced.
3.7.6 is EOLed long back and glusterfs-6 is the latest version with lots of
improvements, bug fixes and new features. The release schedule can be found
at [1]. Upgrading to one of the maintained branch is highly recommended.

On Wed, Apr 10, 2019 at 4:14 PM Martin Toth <snowmailer at gmail.com> wrote:

> I?ve read this documentation but step 4 is really unclear to me. I don?t
> understand related mkdir/rmdir/setfattr and so on.
>
> Step 4:
>
> *Using the gluster volume fuse mount (In this example: /mnt/r2) set up
> metadata so that data will be synced to new brick (In this case it is
> from Server1:/home/gfs/r2_1 to Server1:/home/gfs/r2_5)*
>
> Why should I change trusted.non-existent-key on this volume?
> It is even more confusing because other mentioned howtos does not contain
> this step at all.
>
Those steps were needed in the older releases to set some metadata on the
good bricks so that heal should not happen from the replaced brick to good
bricks, which can lead to data loss. Since you are on 3.7.6, we have
automated all these steps for you in that branch. You just need to run the
replace-brick command, which will take care of all those things.

[1] https://www.gluster.org/release-schedule/

Regards,
Karthik

>
> BR,
> Martin
>
> On 10 Apr 2019, at 11:54, Davide Obbi <davide.obbi at booking.com> wrote:
>
>
> https://docs.gluster.org/en/v3/Administrator%20Guide/Managing%20Volumes/#replace-faulty-brick
>
> On Wed, Apr 10, 2019 at 11:42 AM Martin Toth <snowmailer at gmail.com> wrote:
>
>> Hi all,
>>
>> I am running replica 3 gluster with 3 bricks. One of my servers failed -
>> all disks are showing errors and raid is in fault state.
>>
>> Type: Replicate
>> Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
>> Status: Started
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
>> Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is down
>> Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>>
>> So one of my bricks is totally failed (node2). It went down and all data
>> are lost (failed raid on node2). Now I am running only two bricks on 2
>> servers out from 3.
>> This is really critical problem for us, we can lost all data. I want to
>> add new disks to node2, create new raid array on them and try to replace
>> failed brick on this node.
>>
>> What is the procedure of replacing Brick2 on node2, can someone advice? I
>> can?t find anything relevant in documentation.
>>
>> Thanks in advance,
>> Martin
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
> Davide Obbi
> Senior System Administrator
>
> Booking.com B.V.
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands
> Direct +31207031558
> [image: Booking.com] <https://www.booking.com/>
> Empowering people to experience the world since 1996
> 43 languages, 214+ offices worldwide, 141,000+ global destinations, 29
> million reported listings
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/56c4f3a9/attachment.html>

From amukherj at redhat.com  Wed Apr 10 13:59:28 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Wed, 10 Apr 2019 19:29:28 +0530
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
Message-ID: <CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>

And now for last 15 days:

https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10

./tests/bitrot/bug-1373520.t     18  ==> Fixed through
https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this failing
in brick mux post 5th April
./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux, needs
analysis.
./tests/basic/uss.t             15  ==> happens in both brick mux and non
brick mux runs, test just simply times out. Needs urgent analysis.
./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
https://review.gluster.org/#/c/22508/ , patch merged today.
./tests/basic/volfile-sanity.t      8  ==> Some race, though this succeeds
in second attempt every time.

There're plenty more with 5 instances of failure from many tests. We need
all maintainers/owners to look through these failures and fix them, we
certainly don't want to get into a stage where master is unstable and we
have to lock down the merges till all these failures are resolved. So
please help.

(Please note fstat stats show up the retries as failures too which in a way
is right)


On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com> wrote:

> [1] captures the test failures report since last 30 days and we'd need
> volunteers/component owners to see why the number of failures are so high
> against few tests.
>
> [1]
> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/40431bc5/attachment.html>

From atumball at redhat.com  Wed Apr 10 14:57:04 2019
From: atumball at redhat.com (Amar Tumballi Suryanarayan)
Date: Wed, 10 Apr 2019 20:27:04 +0530
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
Message-ID: <CAHxyDdOmaW+p1SykZ=D6UG_mqp=f_JNvM455rvYq=RAN+UKO1Q@mail.gmail.com>

Thanks for the summary Atin.

On Wed, Apr 10, 2019 at 7:30 PM Atin Mukherjee <amukherj at redhat.com> wrote:

> And now for last 15 days:
>
> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>
> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
> failing in brick mux post 5th April
> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux, needs
> analysis.
> ./tests/basic/uss.t             15  ==> happens in both brick mux and non
> brick mux runs, test just simply times out. Needs urgent analysis.
> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
> https://review.gluster.org/#/c/22508/ , patch merged today.
> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this succeeds
> in second attempt every time.
>
>
Can volfile-sanity.t be  failing because of the 'hang' in uss.t ? It is
possible as volfile-sanity.t runs after uss.t in regressions. I checked
volfile-sanity.t, but it has 'cleanup' at the beginning, but not sure if
there are any lingering things which caused these failures.


> There're plenty more with 5 instances of failure from many tests. We need
> all maintainers/owners to look through these failures and fix them, we
> certainly don't want to get into a stage where master is unstable and we
> have to lock down the merges till all these failures are resolved. So
> please help.
>
> (Please note fstat stats show up the retries as failures too which in a
> way is right)
>
>
> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>> [1] captures the test failures report since last 30 days and we'd need
>> volunteers/component owners to see why the number of failures are so high
>> against few tests.
>>
>> [1]
>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/82710ad3/attachment.html>

From jahernan at redhat.com  Wed Apr 10 17:25:06 2019
From: jahernan at redhat.com (Xavi Hernandez)
Date: Wed, 10 Apr 2019 19:25:06 +0200
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
Message-ID: <CALtvDK6ihuZjq0gu-RsqiRRBCCWH6ZsP+poj-wWCfRjfBGNOFw@mail.gmail.com>

On Wed, Apr 10, 2019 at 4:01 PM Atin Mukherjee <amukherj at redhat.com> wrote:

> And now for last 15 days:
>
> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>
> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
> failing in brick mux post 5th April
> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux, needs
> analysis.
>

I've identified the problem here, but not the cause yet. There's a stale
inodelk acquired by a process that is already dead, which causes inodelk
requests from self-heal and other processes to block.

The reason why it seemed to block in random places is that all commands are
executed with the working directory pointing to a gluster directory which
needs healing after the initial tests. Because of the stale inodelk, when
any application tries to open a file in the working directory, it's blocked.

I'll investigate what causes this.

Xavi

./tests/basic/uss.t             15  ==> happens in both brick mux and non
> brick mux runs, test just simply times out. Needs urgent analysis.
> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
> https://review.gluster.org/#/c/22508/ , patch merged today.
> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this succeeds
> in second attempt every time.
>
> There're plenty more with 5 instances of failure from many tests. We need
> all maintainers/owners to look through these failures and fix them, we
> certainly don't want to get into a stage where master is unstable and we
> have to lock down the merges till all these failures are resolved. So
> please help.
>
> (Please note fstat stats show up the retries as failures too which in a
> way is right)
>
>
> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>> [1] captures the test failures report since last 30 days and we'd need
>> volunteers/component owners to see why the number of failures are so high
>> against few tests.
>>
>> [1]
>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/22af728c/attachment.html>

From rabhat at redhat.com  Wed Apr 10 20:07:50 2019
From: rabhat at redhat.com (FNU Raghavendra Manjunath)
Date: Wed, 10 Apr 2019 16:07:50 -0400
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
Message-ID: <CAMRKmT8LDu9OOow-TRtsSfCZaijzADgvS_FD4m5c4o=8RfE3sg@mail.gmail.com>

On Wed, Apr 10, 2019 at 9:59 AM Atin Mukherjee <amukherj at redhat.com> wrote:

> And now for last 15 days:
>
> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>
> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
> failing in brick mux post 5th April
>

The above patch has been sent to fix the failure with brick mux enabled.


> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux, needs
> analysis.
> ./tests/basic/uss.t             15  ==> happens in both brick mux and non
> brick mux runs, test just simply times out. Needs urgent analysis.
>

Nothing has changed in snapview-server and snapview-client recently.
Looking into it.

./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
> https://review.gluster.org/#/c/22508/ , patch merged today.
> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this succeeds
> in second attempt every time.
>
> There're plenty more with 5 instances of failure from many tests. We need
> all maintainers/owners to look through these failures and fix them, we
> certainly don't want to get into a stage where master is unstable and we
> have to lock down the merges till all these failures are resolved. So
> please help.
>
> (Please note fstat stats show up the retries as failures too which in a
> way is right)
>
>
> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>> [1] captures the test failures report since last 30 days and we'd need
>> volunteers/component owners to see why the number of failures are so high
>> against few tests.
>>
>> [1]
>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190410/c409286f/attachment.html>

From ksubrahm at redhat.com  Thu Apr 11 04:34:23 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Thu, 11 Apr 2019 10:04:23 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <lsvsr70tih57amodoibls5bm.1554915620015@email.android.com>
References: <lsvsr70tih57amodoibls5bm.1554915620015@email.android.com>
Message-ID: <CAHRDaUF4GObPKUdKx7bWeBBTDHshfcRFsA4iThJ9hdRngm6QpA@mail.gmail.com>

Hi Strahil,

Thank you for sharing your experience with reset-brick option.
Since he is using the gluster version 3.7.6, we do not have the reset-brick
[1] option implemented there. It is introduced in 3.9.0. He has to go with
replace-brick with the force option if he wants to use the same path & name
for the new brick.
Yes, it is recommended to have the new brick to be of the same size as that
of the other bricks.

[1]
https://docs.gluster.org/en/latest/release-notes/3.9.0/#introducing-reset-brick-command

Regards,
Karthik

On Wed, Apr 10, 2019 at 10:31 PM Strahil <hunter86_bg at yahoo.com> wrote:

> I have used reset-brick - but I have just changed the brick layout.
> You may give it a try, but I guess you need your new brick to have same
> amount of space (or more).
>
> Maybe someone more experienced should share a more sound solution.
>
> Best Regards,
> Strahil NikolovOn Apr 10, 2019 12:42, Martin Toth <snowmailer at gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I am running replica 3 gluster with 3 bricks. One of my servers failed -
> all disks are showing errors and raid is in fault state.
> >
> > Type: Replicate
> > Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
> > Status: Started
> > Number of Bricks: 1 x 3 = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
> > Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is
> down
> > Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
> >
> > So one of my bricks is totally failed (node2). It went down and all data
> are lost (failed raid on node2). Now I am running only two bricks on 2
> servers out from 3.
> > This is really critical problem for us, we can lost all data. I want to
> add new disks to node2, create new raid array on them and try to replace
> failed brick on this node.
> >
> > What is the procedure of replacing Brick2 on node2, can someone advice?
> I can?t find anything relevant in documentation.
> >
> > Thanks in advance,
> > Martin
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/c2101be1/attachment.html>

From ksubrahm at redhat.com  Thu Apr 11 04:53:37 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Thu, 11 Apr 2019 10:23:37 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <o4gy6pyiy7qtc1se0ywi0l2n.1554957897973@email.android.com>
References: <o4gy6pyiy7qtc1se0ywi0l2n.1554957897973@email.android.com>
Message-ID: <CAHRDaUHztS6XRHFLJR8rAwcc8YYVofoQdMi2YPZVcpgm+-jQaQ@mail.gmail.com>

Hi Strahil,

Can you give us some more insights on
- the volume configuration you were using?
- why you wanted to replace your brick?
- which brick(s) you tried replacing?
- what problem(s) did you face?

Regards,
Karthik

On Thu, Apr 11, 2019 at 10:14 AM Strahil <hunter86_bg at yahoo.com> wrote:

> Hi Karthnik,
> I used only once the brick replace function when I wanted to change my
> Arbiter (v3.12.15 in oVirt 4.2.7)  and it was a complete disaster.
> Most probably I should have stopped the source arbiter before doing that,
> but the docs didn't mention it.
>
> Thus I always use reset-brick, as it never let me down.
>
> Best Regards,
> Strahil Nikolov
> On Apr 11, 2019 07:34, Karthik Subrahmanya <ksubrahm at redhat.com> wrote:
>
> Hi Strahil,
>
> Thank you for sharing your experience with reset-brick option.
> Since he is using the gluster version 3.7.6, we do not have the
> reset-brick [1] option implemented there. It is introduced in 3.9.0. He has
> to go with replace-brick with the force option if he wants to use the same
> path & name for the new brick.
> Yes, it is recommended to have the new brick to be of the same size as
> that of the other bricks.
>
> [1]
> https://docs.gluster.org/en/latest/release-notes/3.9.0/#introducing-reset-brick-command
>
> Regards,
> Karthik
>
> On Wed, Apr 10, 2019 at 10:31 PM Strahil <hunter86_bg at yahoo.com> wrote:
>
> I have used reset-brick - but I have just changed the brick layout.
> You may give it a try, but I guess you need your new brick to have same
> amount of space (or more).
>
> Maybe someone more experienced should share a more sound solution.
>
> Best Regards,
> Strahil NikolovOn Apr 10, 2019 12:42, Martin Toth <snowmailer at gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I am running replica 3 gluster with 3 bricks. One of my servers failed -
> all disks are showing errors and raid is in fault state.
> >
> > Type: Replicate
> > Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
> > Status: Started
> > Number of Bricks: 1 x 3 = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
> > Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is
> down
> > Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
> >
> > So one of my bricks is totally failed (node2). It went down and all data
> are lost (failed raid on node2). Now I am running only two bricks on 2
> servers out from 3.
> > This is really critical problem for us, we can lost all data. I want to
> add new disks to node2, create new raid array on them and try to replace
> failed brick on this node.
> >
> > What is the procedure of replacing Brick2 on node2, can someone advice?
> I can?t find anything relevant in documentation.
> >
> > Thanks in advance,
> > Martin
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/da8d768d/attachment-0001.html>

From ksubrahm at redhat.com  Thu Apr 11 04:55:37 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Thu, 11 Apr 2019 10:25:37 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <CAHRDaUHztS6XRHFLJR8rAwcc8YYVofoQdMi2YPZVcpgm+-jQaQ@mail.gmail.com>
References: <o4gy6pyiy7qtc1se0ywi0l2n.1554957897973@email.android.com>
	<CAHRDaUHztS6XRHFLJR8rAwcc8YYVofoQdMi2YPZVcpgm+-jQaQ@mail.gmail.com>
Message-ID: <CAHRDaUGou4fWVdDu2-E5tE0rgdeZ95p9wT5YfdXqg7Vzv4ZJHg@mail.gmail.com>

On Thu, Apr 11, 2019 at 10:23 AM Karthik Subrahmanya <ksubrahm at redhat.com>
wrote:

> Hi Strahil,
>
> Can you give us some more insights on
> - the volume configuration you were using?
> - why you wanted to replace your brick?
> - which brick(s) you tried replacing?
>
- if you remember the commands/steps that you followed, please give that as
well.

> - what problem(s) did you face?
>

> Regards,
> Karthik
>
> On Thu, Apr 11, 2019 at 10:14 AM Strahil <hunter86_bg at yahoo.com> wrote:
>
>> Hi Karthnik,
>> I used only once the brick replace function when I wanted to change my
>> Arbiter (v3.12.15 in oVirt 4.2.7)  and it was a complete disaster.
>> Most probably I should have stopped the source arbiter before doing that,
>> but the docs didn't mention it.
>>
>> Thus I always use reset-brick, as it never let me down.
>>
>> Best Regards,
>> Strahil Nikolov
>> On Apr 11, 2019 07:34, Karthik Subrahmanya <ksubrahm at redhat.com> wrote:
>>
>> Hi Strahil,
>>
>> Thank you for sharing your experience with reset-brick option.
>> Since he is using the gluster version 3.7.6, we do not have the
>> reset-brick [1] option implemented there. It is introduced in 3.9.0. He has
>> to go with replace-brick with the force option if he wants to use the same
>> path & name for the new brick.
>> Yes, it is recommended to have the new brick to be of the same size as
>> that of the other bricks.
>>
>> [1]
>> https://docs.gluster.org/en/latest/release-notes/3.9.0/#introducing-reset-brick-command
>>
>> Regards,
>> Karthik
>>
>> On Wed, Apr 10, 2019 at 10:31 PM Strahil <hunter86_bg at yahoo.com> wrote:
>>
>> I have used reset-brick - but I have just changed the brick layout.
>> You may give it a try, but I guess you need your new brick to have same
>> amount of space (or more).
>>
>> Maybe someone more experienced should share a more sound solution.
>>
>> Best Regards,
>> Strahil NikolovOn Apr 10, 2019 12:42, Martin Toth <snowmailer at gmail.com>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I am running replica 3 gluster with 3 bricks. One of my servers failed
>> - all disks are showing errors and raid is in fault state.
>> >
>> > Type: Replicate
>> > Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
>> > Status: Started
>> > Number of Bricks: 1 x 3 = 3
>> > Transport-type: tcp
>> > Bricks:
>> > Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
>> > Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is
>> down
>> > Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>> >
>> > So one of my bricks is totally failed (node2). It went down and all
>> data are lost (failed raid on node2). Now I am running only two bricks on 2
>> servers out from 3.
>> > This is really critical problem for us, we can lost all data. I want to
>> add new disks to node2, create new raid array on them and try to replace
>> failed brick on this node.
>> >
>> > What is the procedure of replacing Brick2 on node2, can someone advice?
>> I can?t find anything relevant in documentation.
>> >
>> > Thanks in advance,
>> > Martin
>> > _______________________________________________
>> > Gluster-users mailing list
>> > Gluster-users at gluster.org
>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/eac73ab1/attachment.html>

From dkhandel at redhat.com  Thu Apr 11 07:15:45 2019
From: dkhandel at redhat.com (Deepshikha Khandelwal)
Date: Thu, 11 Apr 2019 12:45:45 +0530
Subject: [Gluster-devel]  Unplanned Jenkins restart
Message-ID: <CABGtVXEsO4GnNQ3xzEhTrpm4q0wCgwjW_zOFWoNvEA5O9MRGBw@mail.gmail.com>

Hello,

I had to do an unplanned Jenkins restart. Jenkins was not responding to any
of the requests and was not giving back the regression votes.
I did update the vote verified values of regression jobs which seemed to
change to 0 all of a sudden and was not giving back the vote. I'm
investigating more on the root cause. I'll update on the bug[1] about the
root cause.

Centos regression jobs may have ended up canceled. Please retry them.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1698716
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/3dcd22b3/attachment.html>

From atumball at redhat.com  Thu Apr 11 08:54:14 2019
From: atumball at redhat.com (Amar Tumballi Suryanarayan)
Date: Thu, 11 Apr 2019 14:24:14 +0530
Subject: [Gluster-devel] [Gluster-users] Proposal: Changes in Gluster
	Community meetings
In-Reply-To: <CAHxyDdOhYfcOMh+LUV3dU+UqrU5AycNHaU21SRu=SVdZYaDuRg@mail.gmail.com>
References: <CAHxyDdN3NdNyGM3L-uX3zYFPxEYHvwNt7XQOSbA8y=1GH1uhaQ@mail.gmail.com>
	<62104B6F-99CF-4C22-80FC-9C177F73E897@onholyground.com>
	<CAHxyDdOhYfcOMh+LUV3dU+UqrU5AycNHaU21SRu=SVdZYaDuRg@mail.gmail.com>
Message-ID: <CAHxyDdOZz-aap+_o0ibucAAd202Mq1=y6kDuaJM02ciL-1WRcg@mail.gmail.com>

Hi All,

Below is the final details of our community meeting, and I will be sending
invites to mailing list following this email. You can add Gluster Community
Calendar so you can get notifications on the meetings.

We are starting the meetings from next week. For the first meeting, we need
1 volunteer from users to discuss the use case / what went well, and what
went bad, etc. preferrably in APAC region.  NA/EMEA region, next week.

Draft Content: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g
----
Gluster Community Meeting
<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Previous-Meeting-minutes>Previous
Meeting minutes:

   - http://github.com/gluster/community

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#DateTime-Check-the-community-calendar>Date/Time:
Check the community calendar
<https://calendar.google.com/calendar/b/1?cid=dmViajVibDBrbnNiOWQwY205ZWg5cGJsaTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ>
<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Bridge>Bridge

   - APAC friendly hours
      - Bridge: https://bluejeans.com/836554017
   - NA/EMEA
      - Bridge: https://bluejeans.com/486278655

------------------------------
<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Attendance>Attendance

   - Name, Company<Optional>

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Host>Host

   - Who will host next meeting?
      - Host will need to send out the agenda 24hr - 12hrs in advance to
      mailing list, and also make sure to send the meeting minutes.
      - Host will need to reach out to one user at least who can talk about
      their usecase, their experience, and their needs.
      - Host needs to send meeting minutes as PR to
      http://github.com/gluster/community

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#User-stories>User stories

   - Discuss 1 usecase from a user.
      - How was the architecture derived, what volume type used, options,
      etc?
      - What were the major issues faced ? How to improve them?
      - What worked good?
      - How can we all collaborate well, so it is win-win for the community
      and the user? How can we

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Community>Community

   -

   Any release updates?
   -

   Blocker issues across the project?
   -

   Metrics
   - Number of new bugs since previous meeting. How many are not triaged?
      - Number of emails, anything unanswered?

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Conferences--Meetups>Conferences
/ Meetups

   - Any conference in next 1 month where gluster-developers are going?
   gluster-users are going? So we can meet and discuss.

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Developer-focus>Developer
focus

   -

   Any design specs to discuss?
   -

   Metrics of the week?
   - Coverity
      - Clang-Scan
      - Number of patches from new developers.
      - Did we increase test coverage?
      - [Atin] Also talk about most frequent test failures in the CI and
      carve out an AI to get them fixed.

<https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#RoundTable>RoundTable

   - <Open for anything which is not covered in agenda>

----

Regards,
Amar

On Mon, Mar 25, 2019 at 8:53 PM Amar Tumballi Suryanarayan <
atumball at redhat.com> wrote:

> Thanks for the feedback Darrell,
>
> The new proposal is to have one in North America 'morning' time. (10AM
> PST), And another in ASIA day time, which is evening 7pm/6pm in Australia,
> 9pm Newzealand, 5pm Tokyo, 4pm Beijing.
>
> For example, if we choose Every other Tuesday for meeting, and 1st of the
> month is Tuesday, we would have North America time for 1st, and on 15th it
> would be ASIA/Pacific time.
>
> Hopefully, this way, we can cover all the timezones, and meeting minutes
> would be committed to github repo, so that way, it will be easier for
> everyone to be aware of what is happening.
>
> Regards,
> Amar
>
> On Mon, Mar 25, 2019 at 8:40 PM Darrell Budic <budic at onholyground.com>
> wrote:
>
>> As a user, I?d like to visit more of these, but the time slot is my 3AM.
>> Any possibility for a rolling schedule (move meeting +6 hours each week
>> with rolling attendance from maintainers?) or an occasional regional
>> meeting 12 hours opposed to the one you?re proposing?
>>
>>   -Darrell
>>
>> On Mar 25, 2019, at 4:25 AM, Amar Tumballi Suryanarayan <
>> atumball at redhat.com> wrote:
>>
>> All,
>>
>> We currently have 3 meetings which are public:
>>
>> 1. Maintainer's Meeting
>>
>> - Runs once in 2 weeks (on Mondays), and current attendance is around 3-5
>> on an avg, and not much is discussed.
>> - Without majority attendance, we can't take any decisions too.
>>
>> 2. Community meeting
>>
>> - Supposed to happen on #gluster-meeting, every 2 weeks, and is the only
>> meeting which is for 'Community/Users'. Others are for developers as of
>> now.
>> Sadly attendance is getting closer to 0 in recent times.
>>
>> 3. GCS meeting
>>
>> - We started it as an effort inside Red Hat gluster team, and opened it
>> up for community from Jan 2019, but the attendance was always from RHT
>> members, and haven't seen any traction from wider group.
>>
>> So, I have a proposal to call out for cancelling all these meeting, and
>> keeping just 1 weekly 'Community' meeting, where even topics related to
>> maintainers and GCS and other projects can be discussed.
>>
>> I have a template of a draft template @
>> https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g
>>
>> Please feel free to suggest improvements, both in agenda and in timings.
>> So, we can have more participation from members of community, which allows
>> more user - developer interactions, and hence quality of project.
>>
>> Waiting for feedbacks,
>>
>> Regards,
>> Amar
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>
> --
> Amar Tumballi (amarts)
>


-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/43f4d414/attachment-0001.html>

From amarts at gmail.com  Thu Apr 11 08:56:48 2019
From: amarts at gmail.com (amarts at gmail.com)
Date: Thu, 11 Apr 2019 08:56:48 +0000
Subject: [Gluster-devel] Invitation: Gluster Community Meeting (APAC
 friendly hours) @ Tue Apr 16,
 2019 11:30am - 12:30pm (IST) (gluster-devel@gluster.org)
Message-ID: <000000000000cc647605863d5d8c@google.com>

You have been invited to the following event.

Title: Gluster Community Meeting (APAC friendly hours)
Bridge: https://bluejeans.com/836554017

Meeting minutes: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both

Previous Meeting notes: http://github.com/gluster/community
When: Tue Apr 16, 2019 11:30am ? 12:30pm India Standard Time - Kolkata
Where: https://bluejeans.com/836554017
Calendar: gluster-devel at gluster.org
Who:
     * amarts at gmail.com - creator
     * gluster-users at gluster.org
     * maintainers at gluster.org
     * gluster-devel at gluster.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=MjU2dWllNDQyM2tqaGs0ZjhidGl2YmdtM2YgZ2x1c3Rlci1kZXZlbEBnbHVzdGVyLm9yZw&tok=NTIjdmViajVibDBrbnNiOWQwY205ZWg5cGJsaTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbTE4ODM2ZDY3Mzk4MjRjNDc2OWE3NmEyMTY0ODEwMDg0ODI5ODNlZmY&ctz=Asia%2FKolkata&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account  
gluster-devel at gluster.org because you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/047b5eaf/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 1821 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/047b5eaf/attachment.ics>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 1861 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/047b5eaf/attachment.bin>

From amarts at gmail.com  Thu Apr 11 08:57:51 2019
From: amarts at gmail.com (amarts at gmail.com)
Date: Thu, 11 Apr 2019 08:57:51 +0000
Subject: [Gluster-devel] Invitation: Gluster Community Meeting (NA/EMEA
 friendly hours) @ Tue Apr 23,
 2019 10:30pm - 11:30pm (IST) (gluster-devel@gluster.org)
Message-ID: <000000000000913c6405863d6169@google.com>

You have been invited to the following event.

Title: Gluster Community Meeting (NA/EMEA friendly hours)
Bridge: https://bluejeans.com/486278655


Meeting minutes: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both

Previous Meeting notes: http://github.com/gluster/community
When: Tue Apr 23, 2019 10:30pm ? 11:30pm India Standard Time - Kolkata
Where: https://bluejeans.com/486278655
Calendar: gluster-devel at gluster.org
Who:
     * amarts at gmail.com - creator
     * gluster-users at gluster.org
     * maintainers at gluster.org
     * gluster-devel at gluster.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=N3Y1NWZkZTkxNWQzc3QxcHR2OHJnNm4zNzYgZ2x1c3Rlci1kZXZlbEBnbHVzdGVyLm9yZw&tok=NTIjdmViajVibDBrbnNiOWQwY205ZWg5cGJsaTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbWYwYzdiMTk0ODRhYWY1MTBmNjU4NmQ0MGM2M2M1MWU3ZDg0ZDQzYzI&ctz=Asia%2FKolkata&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account  
gluster-devel at gluster.org because you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/184a2a0f/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 1828 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/184a2a0f/attachment-0001.ics>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 1868 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/184a2a0f/attachment-0001.bin>

From jahernan at redhat.com  Thu Apr 11 09:28:48 2019
From: jahernan at redhat.com (Xavi Hernandez)
Date: Thu, 11 Apr 2019 11:28:48 +0200
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CALtvDK6ihuZjq0gu-RsqiRRBCCWH6ZsP+poj-wWCfRjfBGNOFw@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
	<CALtvDK6ihuZjq0gu-RsqiRRBCCWH6ZsP+poj-wWCfRjfBGNOFw@mail.gmail.com>
Message-ID: <CALtvDK4_FYZ8V-Xs5j_Od-NO=NgnphZ68a5xhYV4braS3i_m7Q@mail.gmail.com>

On Wed, Apr 10, 2019 at 7:25 PM Xavi Hernandez <jahernan at redhat.com> wrote:

> On Wed, Apr 10, 2019 at 4:01 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>> And now for last 15 days:
>>
>>
>> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>>
>> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
>> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
>> failing in brick mux post 5th April
>> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux,
>> needs analysis.
>>
>
> I've identified the problem here, but not the cause yet. There's a stale
> inodelk acquired by a process that is already dead, which causes inodelk
> requests from self-heal and other processes to block.
>
> The reason why it seemed to block in random places is that all commands
> are executed with the working directory pointing to a gluster directory
> which needs healing after the initial tests. Because of the stale inodelk,
> when any application tries to open a file in the working directory, it's
> blocked.
>
> I'll investigate what causes this.
>

I think I've found the problem. This is a fragment of the brick log that
includes script steps, connections and disconnections of brick 0, and lock
requests to the problematic lock:

[2019-04-11 08:22:20.381398]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++++++++++
[2019-04-11 08:22:22.532646]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++++++++++
[2019-04-11 08:22:23.709655] I [MSGID: 115029]
[server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
from
CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2
(version: 7dev) with subvol /d/backends/patchy1
[2019-04-11 08:22:23.792204] I [common.c:234:pl_trace_in] 8-patchy-locks:
[REQUEST] Locker = {Pid=29710, lk-owner=68580998b47f0000,
Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
patchy-disperse-0, start=0, len=0, pid=0}
[2019-04-11 08:22:23.792299] I [common.c:285:pl_trace_out] 8-patchy-locks:
[GRANTED] Locker = {Pid=29710, lk-owner=68580998b47f0000,
Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
patchy-disperse-0, start=0, len=0, pid=0}
[2019-04-11 08:22:24.628478]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 68 5 online_brick_count ++++++++++
[2019-04-11 08:22:26.097092]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
[2019-04-11 08:22:26.333740]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 71 ec_test_make ++++++++++
[2019-04-11 08:22:27.718963] I [MSGID: 115029]
[server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
from
CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3
(version: 7dev) with subvol /d/backends/patchy1
[2019-04-11 08:22:27.801416] I [common.c:234:pl_trace_in] 8-patchy-locks:
[REQUEST] Locker = {Pid=29885, lk-owner=68580998b47f0000,
Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
patchy-disperse-0, start=0, len=0, pid=0}
[2019-04-11 08:22:27.801434] E [inodelk.c:513:__inode_unlock_lock]
8-patchy-locks:  Matching lock not found for unlock 0-9223372036854775807,
by 68580998b47f0000 on 0x7f0ed0029190
[2019-04-11 08:22:27.801446] I [common.c:285:pl_trace_out] 8-patchy-locks:
[Invalid argument] Locker = {Pid=29885, lk-owner=68580998b47f0000,
Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
patchy-disperse-0, start=0, len=0, pid=0}

This is a fragment of the client log:

[2019-04-11 08:22:20.381398]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++++++++++
[2019-04-11 08:22:20.675938] I [MSGID: 114018]
[client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
patchy-client-1. Client process will keep trying to connect to glusterd
until brick's port is available
[2019-04-11 08:22:21.674772] W [MSGID: 122035]
[ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
with some subvolumes unavailable. (6). FOP : 'INODELK' failed on '/test'
with gfid 35743386-b7c2-41c9-aafd-6b13de216704
[2019-04-11 08:22:22.532646]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++++++++++
[2019-04-11 08:22:23.691171] W [MSGID: 122035]
[ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
with some subvolumes unavailable. (8). FOP : 'INODELK' failed on '/test'
with gfid 35743386-b7c2-41c9-aafd-6b13de216704
[2019-04-11 08:22:23.710420] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
[2019-04-11 08:22:23.791635] W [MSGID: 122035]
[ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
with gfid 35743386-b7c2-41c9-aafd-6b13de216704
[2019-04-11 08:22:24.460529] I [MSGID: 114018]
[client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
patchy-client-1. Client process will keep trying to connect to glusterd
until brick's port is available
[2019-04-11 08:22:24.628478]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 68 5 online_brick_count ++++++++++
[2019-04-11 08:22:26.097092]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
[2019-04-11 08:22:26.333740]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
TEST: 71 ec_test_make ++++++++++
[2019-04-11 08:22:27.719299] I [MSGID: 114046]
[client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
[2019-04-11 08:22:27.840342] W [MSGID: 122035]
[ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
with gfid 35743386-b7c2-41c9-aafd-6b13de216704

The problem happens for two things:

1. Brick 0 gets disconnected randomly (apparently), but the server side is
not aware of these disconnections. This causes that at 08:22:24.460529, the
client has already sent a successful INODELK request to brick 0. At this
point the connection is broken on the client side, but server side doesn't
get any notification, so it doesn't clear the locks.
2. When client reconnects at 08:22:27.719299, a new connection is created,
and the servers does see this new connection (it creates a new client_t
structure). Then the client sends the unlock request, which fails on brick
0 because locks xlators checks if the client is the same by comparing the
pointers, but they are different because of the reconnection. So the lock
is not unlocked and remains there, blocking all future inodelk requests.

The first problem is why the client gets disconnected and the server
doesn't get any notification. The script is stopping bricks 2 and 3 when
this happens. Brick 0 shouldn't fail here. It seems related to the

The second problem is that when we receive a new connection from a client
we already consider connected, we don't cleanup the old connection, which
should take care of the stale locks.

The third problem is that locks xlator is checking if the client is the
same by comparing pointers of client_t structs instead of comparing
client_uid field, which remains the same.

Adding +Raghavendra Gowdappa <rgowdapp at redhat.com>, +Pranith Kumar Karampuri
<pkarampu at redhat.com>, +Krutika Dhananjay <kdhananj at redhat.com>, +Shyam
Ranganathan <srangana at redhat.com> and +Amar Tumballi <atumball at redhat.com> to
help me identify why this is happening and what's the best way to solve it.

Xavi


> Xavi
>
> ./tests/basic/uss.t             15  ==> happens in both brick mux and non
>> brick mux runs, test just simply times out. Needs urgent analysis.
>> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
>> https://review.gluster.org/#/c/22508/ , patch merged today.
>> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this
>> succeeds in second attempt every time.
>>
>> There're plenty more with 5 instances of failure from many tests. We need
>> all maintainers/owners to look through these failures and fix them, we
>> certainly don't want to get into a stage where master is unstable and we
>> have to lock down the merges till all these failures are resolved. So
>> please help.
>>
>> (Please note fstat stats show up the retries as failures too which in a
>> way is right)
>>
>>
>> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>> [1] captures the test failures report since last 30 days and we'd need
>>> volunteers/component owners to see why the number of failures are so high
>>> against few tests.
>>>
>>> [1]
>>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/15d96b2e/attachment.html>

From jahernan at redhat.com  Thu Apr 11 09:40:27 2019
From: jahernan at redhat.com (Xavi Hernandez)
Date: Thu, 11 Apr 2019 11:40:27 +0200
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CALtvDK4_FYZ8V-Xs5j_Od-NO=NgnphZ68a5xhYV4braS3i_m7Q@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
	<CALtvDK6ihuZjq0gu-RsqiRRBCCWH6ZsP+poj-wWCfRjfBGNOFw@mail.gmail.com>
	<CALtvDK4_FYZ8V-Xs5j_Od-NO=NgnphZ68a5xhYV4braS3i_m7Q@mail.gmail.com>
Message-ID: <CALtvDK4fG9JcqCwY0tWajidO_szXfj79TZQxePEpiD3oZ8nKDQ@mail.gmail.com>

On Thu, Apr 11, 2019 at 11:28 AM Xavi Hernandez <jahernan at redhat.com> wrote:

> On Wed, Apr 10, 2019 at 7:25 PM Xavi Hernandez <jahernan at redhat.com>
> wrote:
>
>> On Wed, Apr 10, 2019 at 4:01 PM Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>> And now for last 15 days:
>>>
>>>
>>> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>>>
>>> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
>>> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
>>> failing in brick mux post 5th April
>>> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux,
>>> needs analysis.
>>>
>>
>> I've identified the problem here, but not the cause yet. There's a stale
>> inodelk acquired by a process that is already dead, which causes inodelk
>> requests from self-heal and other processes to block.
>>
>> The reason why it seemed to block in random places is that all commands
>> are executed with the working directory pointing to a gluster directory
>> which needs healing after the initial tests. Because of the stale inodelk,
>> when any application tries to open a file in the working directory, it's
>> blocked.
>>
>> I'll investigate what causes this.
>>
>
> I think I've found the problem. This is a fragment of the brick log that
> includes script steps, connections and disconnections of brick 0, and lock
> requests to the problematic lock:
>
> [2019-04-11 08:22:20.381398]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++++++++++
> [2019-04-11 08:22:22.532646]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++++++++++
> [2019-04-11 08:22:23.709655] I [MSGID: 115029]
> [server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
> from
> CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2
> (version: 7dev) with subvol /d/backends/patchy1
> [2019-04-11 08:22:23.792204] I [common.c:234:pl_trace_in] 8-patchy-locks:
> [REQUEST] Locker = {Pid=29710, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
> Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
> [2019-04-11 08:22:23.792299] I [common.c:285:pl_trace_out] 8-patchy-locks:
> [GRANTED] Locker = {Pid=29710, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
> Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
> [2019-04-11 08:22:24.628478]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 68 5 online_brick_count ++++++++++
> [2019-04-11 08:22:26.097092]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
> 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
> [2019-04-11 08:22:26.333740]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 71 ec_test_make ++++++++++
> [2019-04-11 08:22:27.718963] I [MSGID: 115029]
> [server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
> from
> CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3
> (version: 7dev) with subvol /d/backends/patchy1
> [2019-04-11 08:22:27.801416] I [common.c:234:pl_trace_in] 8-patchy-locks:
> [REQUEST] Locker = {Pid=29885, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
> Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
> [2019-04-11 08:22:27.801434] E [inodelk.c:513:__inode_unlock_lock]
> 8-patchy-locks:  Matching lock not found for unlock 0-9223372036854775807,
> by 68580998b47f0000 on 0x7f0ed0029190
> [2019-04-11 08:22:27.801446] I [common.c:285:pl_trace_out] 8-patchy-locks:
> [Invalid argument] Locker = {Pid=29885, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
> Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
>
> This is a fragment of the client log:
>
> [2019-04-11 08:22:20.381398]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++++++++++
> [2019-04-11 08:22:20.675938] I [MSGID: 114018]
> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
> patchy-client-1. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-04-11 08:22:21.674772] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (6). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:22.532646]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++++++++++
> [2019-04-11 08:22:23.691171] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (8). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:23.710420] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
> [2019-04-11 08:22:23.791635] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:24.460529] I [MSGID: 114018]
> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
> patchy-client-1. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-04-11 08:22:24.628478]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 68 5 online_brick_count ++++++++++
> [2019-04-11 08:22:26.097092]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
> 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
> [2019-04-11 08:22:26.333740]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 71 ec_test_make ++++++++++
> [2019-04-11 08:22:27.719299] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
> [2019-04-11 08:22:27.840342] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>
> The problem happens for two things:
>
> 1. Brick 0 gets disconnected randomly (apparently), but the server side is
> not aware of these disconnections. This causes that at 08:22:24.460529, the
> client has already sent a successful INODELK request to brick 0. At this
> point the connection is broken on the client side, but server side doesn't
> get any notification, so it doesn't clear the locks.
> 2. When client reconnects at 08:22:27.719299, a new connection is created,
> and the servers does see this new connection (it creates a new client_t
> structure). Then the client sends the unlock request, which fails on brick
> 0 because locks xlators checks if the client is the same by comparing the
> pointers, but they are different because of the reconnection. So the lock
> is not unlocked and remains there, blocking all future inodelk requests.
>
> The first problem is why the client gets disconnected and the server
> doesn't get any notification. The script is stopping bricks 2 and 3 when
> this happens. Brick 0 shouldn't fail here. It seems related to the
>
> The second problem is that when we receive a new connection from a client
> we already consider connected, we don't cleanup the old connection, which
> should take care of the stale locks.
>
> The third problem is that locks xlator is checking if the client is the
> same by comparing pointers of client_t structs instead of comparing
> client_uid field, which remains the same.
>
> Adding +Raghavendra Gowdappa <rgowdapp at redhat.com>, +Pranith Kumar
> Karampuri <pkarampu at redhat.com>, +Krutika Dhananjay <kdhananj at redhat.com>
> , +Shyam Ranganathan <srangana at redhat.com> and +Amar Tumballi
> <atumball at redhat.com> to help me identify why this is happening and
> what's the best way to solve it.
>

BTW, while testing this I also hit another problem. I think it's not
related, but just in case...

When DEBUG or TRACE log levels are used sometimes (2 out of some tens) I
get an assertion failed when releasing memory because apparently it's
trying to release more memory than it was allocated (based on gluster's
memory accounting). So memory accounting says that there remain 14 bytes
allocated of "gf_common_mt_strdup" type but it's trying to free an object
that has size 30. At first sight it doesn't seem memory corruption. I'll
investigate it further.


> Xavi
>
>
>> Xavi
>>
>> ./tests/basic/uss.t             15  ==> happens in both brick mux and non
>>> brick mux runs, test just simply times out. Needs urgent analysis.
>>> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
>>> https://review.gluster.org/#/c/22508/ , patch merged today.
>>> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this
>>> succeeds in second attempt every time.
>>>
>>> There're plenty more with 5 instances of failure from many tests. We
>>> need all maintainers/owners to look through these failures and fix them, we
>>> certainly don't want to get into a stage where master is unstable and we
>>> have to lock down the merges till all these failures are resolved. So
>>> please help.
>>>
>>> (Please note fstat stats show up the retries as failures too which in a
>>> way is right)
>>>
>>>
>>> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
>>> wrote:
>>>
>>>> [1] captures the test failures report since last 30 days and we'd need
>>>> volunteers/component owners to see why the number of failures are so high
>>>> against few tests.
>>>>
>>>> [1]
>>>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/41fd76e2/attachment-0001.html>

From ksubrahm at redhat.com  Thu Apr 11 10:31:51 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Thu, 11 Apr 2019 16:01:51 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <69E7C95F-8A81-46CB-8BD8-F66B582144EC@gmail.com>
References: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
	<CAHRDaUHfXpjSOt81J16aEymnBqBJt-np5_p9erradu1Y5iV=NQ@mail.gmail.com>
	<1634978A-E849-48DB-A160-B1AC3DB56D38@gmail.com>
	<69E7C95F-8A81-46CB-8BD8-F66B582144EC@gmail.com>
Message-ID: <CAHRDaUFcb5Hpzj_7WkYbg8WHgz5TGL7h3_U4-gOB722RkFhZEA@mail.gmail.com>

On Thu, Apr 11, 2019 at 12:43 PM Martin Toth <snowmailer at gmail.com> wrote:

> Hi Karthik,
>
> more over, I would like to ask if there are some recommended
> settings/parameters for SHD in order to achieve good or fair I/O while
> volume will be healed when I will replace Brick (this should trigger
> healing process).
>
If I understand you concern correctly, you need to get fair I/O performance
for clients while healing takes place as part of  the replace brick
operation. For this you can turn off the "data-self-heal" and
"metadata-self-heal" options until the heal completes on the new brick.
Turning off client side healing doesn't compromise data integrity and
consistency. During the read request from client, pending xattr is
evaluated for replica copies and read is only served from correct copy.
During writes, IO will continue on both the replicas, SHD will take care of
healing files.
After replacing the brick, we strongly recommend you to consider upgrading
your gluster to one of the maintained versions. We have many stability
related fixes there, which can handle some critical issues and corner cases
which you could hit during these kind of scenarios.

Regards,
Karthik

> I had some problems in past when healing was triggered, VM disks became
> unresponsive because healing took most of I/O. My volume containing only
> big files with VM disks.
>
> Thanks for suggestions.
> BR,
> Martin
>
> On 10 Apr 2019, at 12:38, Martin Toth <snowmailer at gmail.com> wrote:
>
> Thanks, this looks ok to me, I will reset brick because I don't have any
> data anymore on failed node so I can use same path / brick name.
>
> Is reseting brick dangerous command? Should I be worried about some
> possible failure that will impact remaining two nodes? I am running really
> old 3.7.6 but stable version.
>
> Thanks,
> BR!
>
> Martin
>
>
> On 10 Apr 2019, at 12:20, Karthik Subrahmanya <ksubrahm at redhat.com> wrote:
>
> Hi Martin,
>
> After you add the new disks and creating raid array, you can run the
> following command to replace the old brick with new one:
>
> - If you are going to use a different name to the new brick you can run
> gluster volume replace-brick <volname> <old-brick> <new-brick> commit force
>
> - If you are planning to use the same name for the new brick as well then
> you can use
> gluster volume reset-brick <volname> <old-brick> <new-brick> commit force
> Here old-brick & new-brick's hostname &  path should be same.
>
> After replacing the brick, make sure the brick comes online using volume
> status.
> Heal should automatically start, you can check the heal status to see all
> the files gets replicated to the newly added brick. If it does not start
> automatically, you can manually start that by running gluster volume heal
> <volname>.
>
> HTH,
> Karthik
>
> On Wed, Apr 10, 2019 at 3:13 PM Martin Toth <snowmailer at gmail.com> wrote:
>
>> Hi all,
>>
>> I am running replica 3 gluster with 3 bricks. One of my servers failed -
>> all disks are showing errors and raid is in fault state.
>>
>> Type: Replicate
>> Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
>> Status: Started
>> Number of Bricks: 1 x 3 = 3
>> Transport-type: tcp
>> Bricks:
>> Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
>> Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is down
>> Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>>
>> So one of my bricks is totally failed (node2). It went down and all data
>> are lost (failed raid on node2). Now I am running only two bricks on 2
>> servers out from 3.
>> This is really critical problem for us, we can lost all data. I want to
>> add new disks to node2, create new raid array on them and try to replace
>> failed brick on this node.
>>
>> What is the procedure of replacing Brick2 on node2, can someone advice? I
>> can?t find anything relevant in documentation.
>>
>> Thanks in advance,
>> Martin
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/45172b74/attachment.html>

From ksubrahm at redhat.com  Thu Apr 11 10:45:52 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Thu, 11 Apr 2019 16:15:52 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <2030773662.1746728.1554970222668@mail.yahoo.com>
References: <o4gy6pyiy7qtc1se0ywi0l2n.1554957897973@email.android.com>
	<CAHRDaUHztS6XRHFLJR8rAwcc8YYVofoQdMi2YPZVcpgm+-jQaQ@mail.gmail.com>
	<2030773662.1746728.1554970222668@mail.yahoo.com>
Message-ID: <CAHRDaUFHDo-0vmYWunU3rFGnihy7iS1SMRYe60ZbnsrnEivmQw@mail.gmail.com>

On Thu, Apr 11, 2019 at 1:40 PM Strahil Nikolov <hunter86_bg at yahoo.com>
wrote:

> Hi Karthik,
>
> - the volume configuration you were using?
> I used oVirt 4.2.6 Gluster Wizard, so I guess - we need to involve the
> oVirt devs here.
> - why you wanted to replace your brick?
> I have deployed the arbiter on another location as I thought I can deploy
> the Thin Arbiter (still waiting the docs to be updated), but once I
> realized that GlusterD doesn't support Thin Arbiter, I had to build another
> machine for a local arbiter - thus a replacement was needed.
>
We are working on supporting Thin-arbiter with GlusterD. Once done, we will
update on the users list so that you can play with it and let us know your
experience.

> - which brick(s) you tried replacing?
> I was replacing the old arbiter with a new one
> - what problem(s) did you face?
> All oVirt VMs got paused due to I/O errors.
>
There could be many reasons for this. Without knowing the exact state of
the system at that time, I am afraid to make any comment on this.

>
> At the end, I have rebuild the whole setup and I never tried to replace
> the brick this way (used only reset-brick which didn't cause any issues).
>
> As I mentioned that was on v3.12, which is not the default for oVirt
> 4.3.x - so my guess is that it is OK now (current is v5.5).
>
I don't remember anyone complaining about this recently. This should work
in the latest releases.

>
> Just sharing my experience.
>
Highly appreciated.

Regards,
Karthik

>
> Best Regards,
> Strahil Nikolov
>
> ? ?????????, 11 ????? 2019 ?., 0:53:52 ?. ???????-4, Karthik Subrahmanya <
> ksubrahm at redhat.com> ??????:
>
>
> Hi Strahil,
>
> Can you give us some more insights on
> - the volume configuration you were using?
> - why you wanted to replace your brick?
> - which brick(s) you tried replacing?
> - what problem(s) did you face?
>
> Regards,
> Karthik
>
> On Thu, Apr 11, 2019 at 10:14 AM Strahil <hunter86_bg at yahoo.com> wrote:
>
> Hi Karthnik,
> I used only once the brick replace function when I wanted to change my
> Arbiter (v3.12.15 in oVirt 4.2.7)  and it was a complete disaster.
> Most probably I should have stopped the source arbiter before doing that,
> but the docs didn't mention it.
>
> Thus I always use reset-brick, as it never let me down.
>
> Best Regards,
> Strahil Nikolov
> On Apr 11, 2019 07:34, Karthik Subrahmanya <ksubrahm at redhat.com> wrote:
>
> Hi Strahil,
>
> Thank you for sharing your experience with reset-brick option.
> Since he is using the gluster version 3.7.6, we do not have the
> reset-brick [1] option implemented there. It is introduced in 3.9.0. He has
> to go with replace-brick with the force option if he wants to use the same
> path & name for the new brick.
> Yes, it is recommended to have the new brick to be of the same size as
> that of the other bricks.
>
> [1]
> https://docs.gluster.org/en/latest/release-notes/3.9.0/#introducing-reset-brick-command
>
> Regards,
> Karthik
>
> On Wed, Apr 10, 2019 at 10:31 PM Strahil <hunter86_bg at yahoo.com> wrote:
>
> I have used reset-brick - but I have just changed the brick layout.
> You may give it a try, but I guess you need your new brick to have same
> amount of space (or more).
>
> Maybe someone more experienced should share a more sound solution.
>
> Best Regards,
> Strahil NikolovOn Apr 10, 2019 12:42, Martin Toth <snowmailer at gmail.com>
> wrote:
> >
> > Hi all,
> >
> > I am running replica 3 gluster with 3 bricks. One of my servers failed -
> all disks are showing errors and raid is in fault state.
> >
> > Type: Replicate
> > Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
> > Status: Started
> > Number of Bricks: 1 x 3 = 3
> > Transport-type: tcp
> > Bricks:
> > Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
> > Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is
> down
> > Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
> >
> > So one of my bricks is totally failed (node2). It went down and all data
> are lost (failed raid on node2). Now I am running only two bricks on 2
> servers out from 3.
> > This is really critical problem for us, we can lost all data. I want to
> add new disks to node2, create new raid array on them and try to replace
> failed brick on this node.
> >
> > What is the procedure of replacing Brick2 on node2, can someone advice?
> I can?t find anything relevant in documentation.
> >
> > Thanks in advance,
> > Martin
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/940d6831/attachment-0001.html>

From ksubrahm at redhat.com  Thu Apr 11 13:40:30 2019
From: ksubrahm at redhat.com (Karthik Subrahmanya)
Date: Thu, 11 Apr 2019 19:10:30 +0530
Subject: [Gluster-devel] [Gluster-users] Replica 3 - how to replace
	failed node (peer)
In-Reply-To: <00009213-6BF3-4A7F-AFA7-AC076B04496C@gmail.com>
References: <0917AF4A-76EC-4A9E-820F-E0ADA2DA899A@gmail.com>
	<CAHRDaUHfXpjSOt81J16aEymnBqBJt-np5_p9erradu1Y5iV=NQ@mail.gmail.com>
	<1634978A-E849-48DB-A160-B1AC3DB56D38@gmail.com>
	<69E7C95F-8A81-46CB-8BD8-F66B582144EC@gmail.com>
	<CAHRDaUFcb5Hpzj_7WkYbg8WHgz5TGL7h3_U4-gOB722RkFhZEA@mail.gmail.com>
	<00009213-6BF3-4A7F-AFA7-AC076B04496C@gmail.com>
Message-ID: <CAHRDaUFF+z0Ca624ircfQY4gncKZciimJGfqWzKvaWJsmkFmrg@mail.gmail.com>

On Thu, Apr 11, 2019 at 6:38 PM Martin Toth <snowmailer at gmail.com> wrote:

> Hi Karthik,
>
> On Thu, Apr 11, 2019 at 12:43 PM Martin Toth <snowmailer at gmail.com> wrote:
>
>> Hi Karthik,
>>
>> more over, I would like to ask if there are some recommended
>> settings/parameters for SHD in order to achieve good or fair I/O while
>> volume will be healed when I will replace Brick (this should trigger
>> healing process).
>>
> If I understand you concern correctly, you need to get fair I/O
> performance for clients while healing takes place as part of  the replace
> brick operation. For this you can turn off the "data-self-heal" and
> "metadata-self-heal" options until the heal completes on the new brick.
>
>
> This is exactly what I mean. I am running VM disks on remaining 2 (out of
> 3 - one failed as mentioned) nodes and I need to ensure there will be fair
> I/O performance available on these two nodes while replace brick operation
> will heal volume.
> I will not run any VMs on node where replace brick operation will be
> running. So if I understand correctly, when I will set :
>
> # gluster volume set <volname> cluster.data-self-heal off
> # gluster volume set <volname> cluster.metadata-self-heal off
>
> this will tell Gluster clients (libgfapi and FUSE mount) not to read from
> node ?where replace brick operation? is in place but from remaing two
> healthy nodes. Is this correct ? Thanks for clarification.
>
The reads will be served from one of the good bricks since the file will
either be not present on the replaced brick at the time of read or it will
be present but marked for heal if it is not already healed. If already
healed by SHD, then it could be served from the new brick as well, but
there won't be any problem in reading from there in that scenario.
By setting these two options whenever a read comes from client it will not
try to heal the file for data/metadata. Otherwise it would try to heal (if
not already healed by SHD) when the read comes on this, hence slowing down
the client.

>
> Turning off client side healing doesn't compromise data integrity and
> consistency. During the read request from client, pending xattr is
> evaluated for replica copies and read is only served from correct copy.
> During writes, IO will continue on both the replicas, SHD will take care of
> healing files.
> After replacing the brick, we strongly recommend you to consider upgrading
> your gluster to one of the maintained versions. We have many stability
> related fixes there, which can handle some critical issues and corner cases
> which you could hit during these kind of scenarios.
>
>
> This will be first priority in infrastructure after fixing this cluster
> back to fully functional replica3. I will upgrade to 3.12.x and then to
> version 5 or 6.
>
Sounds good.

If you are planning to have the same name for the new brick and if you get
the error like "Brick may be containing or be contained by an existing
brick" even after using the force option, try  using a different name. That
should work.

Regards,
Karthik

>
> BR,
> Martin
>
> Regards,
> Karthik
>
>> I had some problems in past when healing was triggered, VM disks became
>> unresponsive because healing took most of I/O. My volume containing only
>> big files with VM disks.
>>
>> Thanks for suggestions.
>> BR,
>> Martin
>>
>> On 10 Apr 2019, at 12:38, Martin Toth <snowmailer at gmail.com> wrote:
>>
>> Thanks, this looks ok to me, I will reset brick because I don't have any
>> data anymore on failed node so I can use same path / brick name.
>>
>> Is reseting brick dangerous command? Should I be worried about some
>> possible failure that will impact remaining two nodes? I am running really
>> old 3.7.6 but stable version.
>>
>> Thanks,
>> BR!
>>
>> Martin
>>
>>
>> On 10 Apr 2019, at 12:20, Karthik Subrahmanya <ksubrahm at redhat.com>
>> wrote:
>>
>> Hi Martin,
>>
>> After you add the new disks and creating raid array, you can run the
>> following command to replace the old brick with new one:
>>
>> - If you are going to use a different name to the new brick you can run
>> gluster volume replace-brick <volname> <old-brick> <new-brick> commit
>> force
>>
>> - If you are planning to use the same name for the new brick as well then
>> you can use
>> gluster volume reset-brick <volname> <old-brick> <new-brick> commit force
>> Here old-brick & new-brick's hostname &  path should be same.
>>
>> After replacing the brick, make sure the brick comes online using volume
>> status.
>> Heal should automatically start, you can check the heal status to see all
>> the files gets replicated to the newly added brick. If it does not start
>> automatically, you can manually start that by running gluster volume heal
>> <volname>.
>>
>> HTH,
>> Karthik
>>
>> On Wed, Apr 10, 2019 at 3:13 PM Martin Toth <snowmailer at gmail.com> wrote:
>>
>>> Hi all,
>>>
>>> I am running replica 3 gluster with 3 bricks. One of my servers failed -
>>> all disks are showing errors and raid is in fault state.
>>>
>>> Type: Replicate
>>> Volume ID: 41d5c283-3a74-4af8-a55d-924447bfa59a
>>> Status: Started
>>> Number of Bricks: 1 x 3 = 3
>>> Transport-type: tcp
>>> Bricks:
>>> Brick1: node1.san:/tank/gluster/gv0imagestore/brick1
>>> Brick2: node2.san:/tank/gluster/gv0imagestore/brick1 <? this brick is
>>> down
>>> Brick3: node3.san:/tank/gluster/gv0imagestore/brick1
>>>
>>> So one of my bricks is totally failed (node2). It went down and all data
>>> are lost (failed raid on node2). Now I am running only two bricks on 2
>>> servers out from 3.
>>> This is really critical problem for us, we can lost all data. I want to
>>> add new disks to node2, create new raid array on them and try to replace
>>> failed brick on this node.
>>>
>>> What is the procedure of replacing Brick2 on node2, can someone advice?
>>> I can?t find anything relevant in documentation.
>>>
>>> Thanks in advance,
>>> Martin
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/b0bc33b6/attachment.html>

From dkhandel at redhat.com  Thu Apr 11 16:43:54 2019
From: dkhandel at redhat.com (Deepshikha Khandelwal)
Date: Thu, 11 Apr 2019 22:13:54 +0530
Subject: [Gluster-devel] [Gluster-infra] Upgrading build.gluster.org
Message-ID: <CABGtVXEdTnzbunisNg5MDyLM4iPdcE7pSjxkuZgznY6Ttr=Ebg@mail.gmail.com>

Hello,

I?ve planned to do an upgrade of build.gluster.org tomorrow morning so as
to install and pull in the latest security upgrade of the Jenkins plugins.
I?ll stop all the running jobs and re-trigger them once I'm done with the
upgrade.

The downtime window will be from :
UTC: 0330 to 0400
IST: 0900 to 0930

The outage is for 30 minutes. Please bear with us as we continue to ensure
the latest plugins and fixes for build.gluster.org

Thanks,
Deepshikha
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/4a831469/attachment.html>

From rabhat at redhat.com  Thu Apr 11 19:18:41 2019
From: rabhat at redhat.com (FNU Raghavendra Manjunath)
Date: Thu, 11 Apr 2019 15:18:41 -0400
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CAMRKmT8LDu9OOow-TRtsSfCZaijzADgvS_FD4m5c4o=8RfE3sg@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
	<CAMRKmT8LDu9OOow-TRtsSfCZaijzADgvS_FD4m5c4o=8RfE3sg@mail.gmail.com>
Message-ID: <CAMRKmT-iWA5+VMA1=_ZeNFVOGWKwpeaFn45i5tW1KK-J4EC_9g@mail.gmail.com>

While analysing the logs of the runs where uss.t failed made following
observations.

1) In the first iteration of uss.t, the time difference between the first
test of the .t file and the last test of the .t file is just within 1
minute.

But, I think it is the cleanup sequence which is taking more time. One of
the reasons I guess this is happening is, we dont see the brick process
shutting down message
in the logs.


2) In the 2nd iteration of uss.t (because 1st iteration failed because of
timeout) it fails because something has not been completed in the cleanup
sequence of the previous iteration.

The volume start command itself fails in the 2nd iteration. Because of that
the remaining tests also fail

This is from cmd_history.log

uster.org:/d/backends/2/patchy_snap_mnt
builder202.int.aws.gluster.org:/d/backends/3/patchy_snap_mnt
++++++++++
[2019-04-10 19:54:09.145086]  : volume create patchy
builder202.int.aws.gluster.org:/d/backends/1/patchy_snap_mnt
builder202.int.aws.gluster.org:/d/backends/2/patchy_snap_mnt
builder202.int.aws.gluster.org:/d/backends/3/patchy_snap_mnt : SUCCESS
[2019-04-10 19:54:09.156221]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 39
gluster --mode=script --wignore volume set patchy nfs.disable false
++++++++++
[2019-04-10 19:54:09.265138]  : volume set patchy nfs.disable false :
SUCCESS
[2019-04-10 19:54:09.274386]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 42
gluster --mode=script --wignore volume start patchy ++++++++++
[2019-04-10 19:54:09.565086]  : volume start patchy : FAILED : Commit
failed on localhost. Please check log file for details.
[2019-04-10 19:54:09.572753]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 44
_GFS --attribute-timeout=0 --entry-timeout=0 --volfile-server=
builder202.int.aws.gluster.org --volfile-id=patchy /mnt/glusterfs/0
++++++++++


And this is from the brick showing some issue with the export directory not
being present properly.

[2019-04-10 19:54:09.544476] I [MSGID: 100030] [glusterfsd.c:2857:main]
0-/build/install/sbin/glusterfsd: Started running
/build/install/sbin/glusterfsd version 7dev (args:
/build/install/sbin/glusterfsd -s buil
der202.int.aws.gluster.org --volfile-id
patchy.builder202.int.aws.gluster.org.d-backends-1-patchy_snap_mnt -p
/var/run/gluster/vols/patchy/builder202.int.aws.gluster.org-d-backends-1-patchy_snap_mnt.pid
-S /var/
run/gluster/7ac65190b72da80a.socket --brick-name
/d/backends/1/patchy_snap_mnt -l
/var/log/glusterfs/bricks/d-backends-1-patchy_snap_mnt.log --xlator-option
*-posix.glusterd-uuid=695c060d-74d3-440e-8cdb-327ec297
f2d2 --process-name brick --brick-port 49152 --xlator-option
patchy-server.listen-port=49152)
[2019-04-10 19:54:09.549394] I [socket.c:962:__socket_server_bind]
0-socket.glusterfsd: closing (AF_UNIX) reuse check socket 9
[2019-04-10 19:54:09.553190] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1
[2019-04-10 19:54:09.553209] I [MSGID: 101190]
[event-epoll.c:680:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 0
[2019-04-10 19:54:09.556932] I
[rpcsvc.c:2694:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured
rpc.outstanding-rpc-limit with value 64
[2019-04-10 19:54:09.557859] E [MSGID: 138001] [index.c:2392:init]
0-patchy-index: Failed to find parent dir
(/d/backends/1/patchy_snap_mnt/.glusterfs) of index basepath
/d/backends/1/patchy_snap_mnt/.glusterfs/
indices. [No such file or directory]        ============================>
(.glusterfs is absent)
[2019-04-10 19:54:09.557884] E [MSGID: 101019] [xlator.c:629:xlator_init]
0-patchy-index: Initialization of volume 'patchy-index' failed, review your
volfile again
[2019-04-10 19:54:09.557892] E [MSGID: 101066]
[graph.c:409:glusterfs_graph_init] 0-patchy-index: initializing translator
failed
[2019-04-10 19:54:09.557900] E [MSGID: 101176]
[graph.c:772:glusterfs_graph_activate] 0-graph: init failed
[2019-04-10 19:54:09.564154] I [io-stats.c:4033:fini] 0-patchy-io-stats:
io-stats translator unloaded
[2019-04-10 19:54:09.564748] W [glusterfsd.c:1592:cleanup_and_exit]
(-->/build/install/sbin/glusterfsd(mgmt_getspec_cbk+0x806) [0x411f32]
-->/build/install/sbin/glusterfsd(glusterfs_process_volfp+0x272) [0x40b9b
9] -->/build/install/sbin/glusterfsd(cleanup_and_exit+0x88) [0x4093a5] )
0-: received signum (-1), shutting down


And this is from the cmd_history.log file of the 2nd iteration uss.t from
another jenkins run of uss.t

[2019-04-10 15:35:51.927343]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 39
gluster --mode=script --wignore volume set patchy nfs.disable false
++++++++++
[2019-04-10 15:35:52.038072]  : volume set patchy nfs.disable false :
SUCCESS
[2019-04-10 15:35:52.057582]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 42
gluster --mode=script --wignore volume start patchy ++++++++++
[2019-04-10 15:35:52.104288]  : volume start patchy : FAILED : Failed to
find brick directory /d/backends/1/patchy_snap_mnt for volume patchy.
Reason : No such file or directory =========> (export directory is not
present)
[2019-04-10 15:35:52.117735]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 44
_GFS --attribute-timeout=0 --entry-timeout=0 --volfile-server=
builder205.int.aws.gluster.org --volfile-id=patchy /mnt/glusterfs/0
++++++++++


I suspect something wrong with the cleanup sequence which causes the
timeout of the test in the 1st iteration and the export directory issues in
the next iteration causes the failure of uss.t in the 2nd iteration.


Regards,
Raghavendra



On Wed, Apr 10, 2019 at 4:07 PM FNU Raghavendra Manjunath <rabhat at redhat.com>
wrote:

>
>
> On Wed, Apr 10, 2019 at 9:59 AM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>> And now for last 15 days:
>>
>>
>> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>>
>> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
>> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
>> failing in brick mux post 5th April
>>
>
> The above patch has been sent to fix the failure with brick mux enabled.
>
>
>> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux,
>> needs analysis.
>> ./tests/basic/uss.t             15  ==> happens in both brick mux and non
>> brick mux runs, test just simply times out. Needs urgent analysis.
>>
>
> Nothing has changed in snapview-server and snapview-client recently.
> Looking into it.
>
> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
>> https://review.gluster.org/#/c/22508/ , patch merged today.
>> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this
>> succeeds in second attempt every time.
>>
>> There're plenty more with 5 instances of failure from many tests. We need
>> all maintainers/owners to look through these failures and fix them, we
>> certainly don't want to get into a stage where master is unstable and we
>> have to lock down the merges till all these failures are resolved. So
>> please help.
>>
>> (Please note fstat stats show up the retries as failures too which in a
>> way is right)
>>
>>
>> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>> [1] captures the test failures report since last 30 days and we'd need
>>> volunteers/component owners to see why the number of failures are so high
>>> against few tests.
>>>
>>> [1]
>>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190411/0eba823b/attachment-0001.html>

From moagrawa at redhat.com  Fri Apr 12 02:56:06 2019
From: moagrawa at redhat.com (Mohit Agrawal)
Date: Fri, 12 Apr 2019 08:26:06 +0530
Subject: [Gluster-devel] Be careful before closing fd in a default case
Message-ID: <CALBz39wM0ztYb2+yptVoEAoNaw_PMgTQLesmLib0O9qsoo3XcA@mail.gmail.com>

Hi,

  I want to highlight recent bug(
https://bugzilla.redhat.com/show_bug.cgi?id=1699025) due to raised after
fixed one Coverity
   bug https://review.gluster.org/#/c/glusterfs/+/20720/
  As we know all gluster processes initially keeping open standard fd's (0,1,2)
at the time of daemonizing so that kernel
  don't assign these fd's to any fd open by gluster process. In this
Coverity bug, we closed fd in changelog fini if fd value is not equal to -1.
  As we know GF_CALLOC initializes to all structure members to 0 so initial
fd value was 0 and changelog_init did not open htime_fd
  because changelog was not active so at the time of calling changelog fini it
closes fd(0). After closing fd(0) by changelog fini if any
  client(shd) is trying to establish a connection with the server(in the
brick_mux environment), the server gets fd(0) as a socket fd.
  I have observed socket event framework (socket_event_handler) was not
working perfectly for fd(0) while
  volumes are stopped in a loop in brick_mux environment and bricks are not
detached successfully.

  So always we should careful at the time of closing fd, before closing fd in
default case we should check fd should not be zero.
  I have fixed the same from (
https://review.gluster.org/#/c/glusterfs/+/22549/) and upload a .t also.

Regards,
Mohit Agrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190412/bcb597b5/attachment.html>

From xhernandez at redhat.com  Fri Apr 12 12:34:54 2019
From: xhernandez at redhat.com (Xavi Hernandez)
Date: Fri, 12 Apr 2019 14:34:54 +0200
Subject: [Gluster-devel] Possible issues with shared threads
Message-ID: <CALtvDK76Td81bMX_3LmpX0yBthm1mLKbZX5nsXtOf3fv9=LC6A@mail.gmail.com>

Hi,

I've found some issues with memory accounting and I've written a patch [1]
to fix them. However during the tests I've found another problem:

In a brick-multiplexed environment, posix tries to start a single janitor
thread shared by all posix xlator instances, however there are two issues:

1. The creation is not atomic and it could happen that more than one
janitor thread is started (unless xlator init is serialized in some way)
2. Even though the thread is global, it's using information from a single
instance (through 'this'). This means that once the first instance of posix
xlator is stopped, 'this' can be destroyed, but the janitor will continue
using it. From the memory accounting point of view, it means that whatever
this thread does, is not tracked anymore.

Note that we only need to write a log message to access 'this' and use
dynamic memory.

I detected this problem in the posix xlator, but since there are other
threads that have been made global, maybe something similar could happen. I
think this need to be checked and fixed.

Xavi

[1] https://review.gluster.org/c/glusterfs/+/22554
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190412/a872f85b/attachment.html>

From jenkins at build.gluster.org  Mon Apr 15 01:45:02 2019
From: jenkins at build.gluster.org (jenkins at build.gluster.org)
Date: Mon, 15 Apr 2019 01:45:02 +0000 (UTC)
Subject: [Gluster-devel] Weekly Untriaged Bugs
Message-ID: <1991779699.5.1555292702672.JavaMail.jenkins@jenkins-el7.rht.gluster.org>

[...truncated 6 lines...]
https://bugzilla.redhat.com/1699023 / core: Brick is not able to detach successfully in brick_mux environment
https://bugzilla.redhat.com/1695416 / core: client log flooding with intentional socket shutdown message when a brick is down
https://bugzilla.redhat.com/1691833 / core: Client sends 128KByte network packet for 0 length file copy
https://bugzilla.redhat.com/1695480 / core: Global Thread Pool
https://bugzilla.redhat.com/1694943 / core: parallel-readdir slows down directory listing
https://bugzilla.redhat.com/1698861 / disperse: Renaming a directory when 2 bricks of multiple disperse subvols are down leaves both old and new dirs on the bricks.
https://bugzilla.redhat.com/1697293 / distribute: DHT: print hash and layout values in hexadecimal format in the logs
https://bugzilla.redhat.com/1697971 / fuse: Segfault in FUSE process, potential use after free
https://bugzilla.redhat.com/1696721 / geo-replication: geo-replication failing after upgrade from 5.5 to 6.0
https://bugzilla.redhat.com/1694637 / geo-replication: Geo-rep: Rename to an existing file name destroys its content on slave
https://bugzilla.redhat.com/1689981 / geo-replication: OSError: [Errno 1] Operation not permitted - failing with socket files?
https://bugzilla.redhat.com/1694139 / glusterd: Error waiting for job 'heketi-storage-copy-job' to complete on one-node k3s deployment.
https://bugzilla.redhat.com/1695099 / glusterd: The number of glusterfs processes keeps increasing, using all available resources
https://bugzilla.redhat.com/1690454 / posix-acl: mount-shared-storage.sh does not implement mount options
https://bugzilla.redhat.com/1696518 / project-infrastructure: builder203 does not have a valid hostname set
https://bugzilla.redhat.com/1697890 / project-infrastructure: centos-regression is not giving its vote
https://bugzilla.redhat.com/1697923 / project-infrastructure: CI: collect core file in a job artifacts
https://bugzilla.redhat.com/1691617 / project-infrastructure: clang-scan tests are failing nightly.
https://bugzilla.redhat.com/1691357 / project-infrastructure: core archive link from regression jobs throw not found error
https://bugzilla.redhat.com/1692349 / project-infrastructure: gluster-csi-containers job is failing
https://bugzilla.redhat.com/1698716 / project-infrastructure: Regression job did not vote for https://review.gluster.org/#/c/glusterfs/+/22366/
https://bugzilla.redhat.com/1698694 / project-infrastructure: regression job isn't voting back to gerrit
https://bugzilla.redhat.com/1693385 / project-infrastructure: request to change the version of fedora in fedora-smoke-job
https://bugzilla.redhat.com/1693295 / project-infrastructure: rpc.statd not started on builder204.aws.gluster.org
https://bugzilla.redhat.com/1691789 / project-infrastructure: rpc-statd service stops on AWS builders
https://bugzilla.redhat.com/1695484 / project-infrastructure: smoke fails with "Build root is locked by another process"
https://bugzilla.redhat.com/1693184 / replicate: A brick process(glusterfsd) died with 'memory violation'
https://bugzilla.redhat.com/1698566 / selfheal: shd crashed while executing ./tests/bugs/core/bug-1432542-mpx-restart-crash.t in CI
https://bugzilla.redhat.com/1699309 / snapshot: Gluster snapshot fails with systemd autmounted bricks
https://bugzilla.redhat.com/1696633 / tests: GlusterFs v4.1.5 Tests from /tests/bugs/ module failing on Intel
https://bugzilla.redhat.com/1697812 / website: mention a pointer to all the mailing lists available under glusterfs project(https://www.gluster.org/community/)
[...truncated 2 lines...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 3938 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/6fdd6a80/attachment.obj>

From cynthia.zhou at nokia-sbell.com  Mon Apr 15 02:40:48 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 15 Apr 2019 02:40:48 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
Message-ID: <6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>

Hi,
I made a patch and according to my test, this glusterd stuck issue disappear with my patch. Only need to move  event_handled to the end of socket_event_poll_in function.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
         }

-        if (notify_handled && (ret != -1))
-                event_handled (ctx->event_pool, priv->sock, priv->idx,
-                               priv->gen);
@@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
                 }
                 pthread_mutex_unlock (&priv->notify.lock);
         }
+        if (notify_handled && (ret != -1))
+                event_handled (ctx->event_pool, priv->sock, priv->idx,
+                               priv->gen);
         return ret;
}

cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Tuesday, April 09, 2019 3:57 PM
To: 'Raghavendra Gowdappa' <rgowdapp at redhat.com>
Cc: gluster-devel at gluster.org
Subject: RE: glusterd stuck for glusterfs with version 3.12.15

Can you figure out some possible reason why iobref is corrupted, is it possible that thread 8 has called poll in and iobref has been relased, but the lock within it is not properly released (as I can not find any free lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy again, and so stuck on this lock
Also, there should not be two thread handling the same socket at the same time, although there has been a patch claimed to tackle this issue.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Tuesday, April 09, 2019 3:52 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

looks like the iobref is corrupted here. It seems to be a use-after-free issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/6352b5c0/attachment-0001.html>

From rgowdapp at redhat.com  Mon Apr 15 03:51:48 2019
From: rgowdapp at redhat.com (Raghavendra Gowdappa)
Date: Mon, 15 Apr 2019 09:21:48 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
Message-ID: <CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>

Cynthia,

On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi,
>
> I made a patch and according to my test, this glusterd stuck issue
> disappear with my patch. Only need to move  event_handled to the end of
> socket_event_poll_in function.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>          }
>
>
>
> -        if (notify_handled && (ret != -1))
>
> -                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> -                               priv->gen);
>
> @@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>                  }
>
>                  pthread_mutex_unlock (&priv->notify.lock);
>
>          }
>
> +        if (notify_handled && (ret != -1))
>
> +                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> +                               priv->gen);
>

Thanks for this tip. Though this helps in fixing the hang, this change has
performance impact. Moving event_handled to end of poll_in means, socket
will be added back for polling of new events only _after_ the rpc is msg is
processed by higher layers (like EC) and higher layers can have significant
latency  for processing the msg. Which means, socket will be out of polling
for longer periods of time which decreases the throughput (number of msgs
read per second) affecting performance. However, this experiment definitely
indicates there is a codepath where event_handled is not called (and hence
causing the hang). I'll go through this codepath again.

Thanks for that experiment :).

         return ret;
>
> }
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Tuesday, April 09, 2019 3:57 PM
> *To:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* RE: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Can you figure out some possible reason why iobref is corrupted, is it
> possible that thread 8 has called poll in and iobref has been relased, but
> the lock within it is not properly released (as I can not find any free
> lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy
> again, and so stuck on this lock
>
> Also, there should not be two thread handling the same socket at the same
> time, although there has been a patch claimed to tackle this issue.
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Tuesday, April 09, 2019 3:52 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
>
>
>
>
> On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@
> \000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}},
> ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
>
>
>
> looks like the iobref is corrupted here. It seems to be a use-after-free
> issue. We need to dig into why a freed iobref is being accessed here.
>
>
>
> (gdb) quit
>
> A debugging session is active.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/bb30efd6/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 15 05:38:27 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 15 Apr 2019 05:38:27 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
Message-ID: <32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>

Ok, thanks for your comment!

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com>
Sent: Monday, April 15, 2019 11:52 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: gluster-devel at gluster.org
Subject: Re: glusterd stuck for glusterfs with version 3.12.15

Cynthia,

On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
I made a patch and according to my test, this glusterd stuck issue disappear with my patch. Only need to move  event_handled to the end of socket_event_poll_in function.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
         }

-        if (notify_handled && (ret != -1))
-                event_handled (ctx->event_pool, priv->sock, priv->idx,
-                               priv->gen);
@@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
                 }
                 pthread_mutex_unlock (&priv->notify.lock);
         }
+        if (notify_handled && (ret != -1))
+                event_handled (ctx->event_pool, priv->sock, priv->idx,
+                               priv->gen);

Thanks for this tip. Though this helps in fixing the hang, this change has performance impact. Moving event_handled to end of poll_in means, socket will be added back for polling of new events only _after_ the rpc is msg is processed by higher layers (like EC) and higher layers can have significant latency  for processing the msg. Which means, socket will be out of polling for longer periods of time which decreases the throughput (number of msgs read per second) affecting performance. However, this experiment definitely indicates there is a codepath where event_handled is not called (and hence causing the hang). I'll go through this codepath again.

Thanks for that experiment :).

         return ret;
}

cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Tuesday, April 09, 2019 3:57 PM
To: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: glusterd stuck for glusterfs with version 3.12.15

Can you figure out some possible reason why iobref is corrupted, is it possible that thread 8 has called poll in and iobref has been relased, but the lock within it is not properly released (as I can not find any free lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy again, and so stuck on this lock
Also, there should not be two thread handling the same socket at the same time, although there has been a patch claimed to tackle this issue.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Tuesday, April 09, 2019 3:52 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

looks like the iobref is corrupted here. It seems to be a use-after-free issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/5e23ac8c/attachment-0001.html>

From rgowdapp at redhat.com  Mon Apr 15 06:35:37 2019
From: rgowdapp at redhat.com (Raghavendra Gowdappa)
Date: Mon, 15 Apr 2019 12:05:37 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
	<32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
Message-ID: <CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>

On Mon, Apr 15, 2019 at 11:08 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Ok, thanks for your comment!
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Monday, April 15, 2019 11:52 AM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Cynthia,
>
>
>
> On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> I made a patch and according to my test, this glusterd stuck issue
> disappear with my patch. Only need to move  event_handled to the end of
> socket_event_poll_in function.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>          }
>
>
>
> -        if (notify_handled && (ret != -1))
>
> -                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> -                               priv->gen);
>
> @@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>                  }
>
>                  pthread_mutex_unlock (&priv->notify.lock);
>
>          }
>
> +        if (notify_handled && (ret != -1))
>
> +                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> +                               priv->gen);
>
>
>
> Thanks for this tip. Though this helps in fixing the hang, this change has
> performance impact. Moving event_handled to end of poll_in means, socket
> will be added back for polling of new events only _after_ the rpc is msg is
> processed by higher layers (like EC) and higher layers can have significant
> latency  for processing the msg. Which means, socket will be out of polling
> for longer periods of time which decreases the throughput (number of msgs
> read per second) affecting performance. However, this experiment definitely
> indicates there is a codepath where event_handled is not called (and hence
> causing the hang). I'll go through this codepath again.
>

Can you check whether patch [1] fixes the issue you are seeing?

[1] https://review.gluster.org/#/c/glusterfs/+/22566


>
> Thanks for that experiment :).
>
>
>
>          return ret;
>
> }
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Tuesday, April 09, 2019 3:57 PM
> *To:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* RE: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Can you figure out some possible reason why iobref is corrupted, is it
> possible that thread 8 has called poll in and iobref has been relased, but
> the lock within it is not properly released (as I can not find any free
> lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy
> again, and so stuck on this lock
>
> Also, there should not be two thread handling the same socket at the same
> time, although there has been a patch claimed to tackle this issue.
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Tuesday, April 09, 2019 3:52 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
>
>
>
>
> On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@
> \000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}},
> ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
>
>
>
> looks like the iobref is corrupted here. It seems to be a use-after-free
> issue. We need to dig into why a freed iobref is being accessed here.
>
>
>
> (gdb) quit
>
> A debugging session is active.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/775f826e/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 15 07:21:53 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 15 Apr 2019 07:21:53 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
	<32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
	<CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>
Message-ID: <a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>

Hi,
The reason why I move event_handled to the end of socket_event_poll_in is because if event_handled is called before rpc_transport_pollin_destroy, it allowed another round of event_dispatch_epoll_handler happen before  rpc_transport_pollin_destroy, in this way, when the latter poll in goes to rpc_transport_pollin_destroy , there is a chance that the pollin->iobref has already been destroyed by the first one(there is no lock destroy for iobref->lock in iobref_destroy by the way). That may cause stuck in ?LOCK (&iobref->lock);?
I find the one of recent patch
SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket

I think the point is to avoid the concurrent handling of the same socket at the same time, but after my test with this patch this problem also exists, so I think event_handled is still called too early to allow concurrent handling of the same socket happen, and after move it to the end of socket_event_poll this glusterd stuck issue disappeared.
cynthia
From: Raghavendra Gowdappa <rgowdapp at redhat.com>
Sent: Monday, April 15, 2019 2:36 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: gluster-devel at gluster.org
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 15, 2019 at 11:08 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Ok, thanks for your comment!

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Monday, April 15, 2019 11:52 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15

Cynthia,

On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
I made a patch and according to my test, this glusterd stuck issue disappear with my patch. Only need to move  event_handled to the end of socket_event_poll_in function.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
         }

-        if (notify_handled && (ret != -1))
-                event_handled (ctx->event_pool, priv->sock, priv->idx,
-                               priv->gen);
@@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
                 }
                 pthread_mutex_unlock (&priv->notify.lock);
         }
+        if (notify_handled && (ret != -1))
+                event_handled (ctx->event_pool, priv->sock, priv->idx,
+                               priv->gen);

Thanks for this tip. Though this helps in fixing the hang, this change has performance impact. Moving event_handled to end of poll_in means, socket will be added back for polling of new events only _after_ the rpc is msg is processed by higher layers (like EC) and higher layers can have significant latency  for processing the msg. Which means, socket will be out of polling for longer periods of time which decreases the throughput (number of msgs read per second) affecting performance. However, this experiment definitely indicates there is a codepath where event_handled is not called (and hence causing the hang). I'll go through this codepath again.

Can you check whether patch [1] fixes the issue you are seeing?

[1] https://review.gluster.org/#/c/glusterfs/+/22566


Thanks for that experiment :).

         return ret;
}

cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Tuesday, April 09, 2019 3:57 PM
To: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: glusterd stuck for glusterfs with version 3.12.15

Can you figure out some possible reason why iobref is corrupted, is it possible that thread 8 has called poll in and iobref has been relased, but the lock within it is not properly released (as I can not find any free lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy again, and so stuck on this lock
Also, there should not be two thread handling the same socket at the same time, although there has been a patch claimed to tackle this issue.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Tuesday, April 09, 2019 3:52 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

looks like the iobref is corrupted here. It seems to be a use-after-free issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/e44d839e/attachment-0001.html>

From rgowdapp at redhat.com  Mon Apr 15 08:35:54 2019
From: rgowdapp at redhat.com (Raghavendra Gowdappa)
Date: Mon, 15 Apr 2019 14:05:54 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
	<32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
	<CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>
	<a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>
Message-ID: <CAFkORY_G-in5zOLJqT-wFRxOZLUCJ3FdMZBWRaePP7P0yRFG4g@mail.gmail.com>

On Mon, Apr 15, 2019 at 12:52 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi,
>
> The reason why I move event_handled to the end of socket_event_poll_in is
> because if event_handled is called before rpc_transport_pollin_destroy, it
> allowed another round of event_dispatch_epoll_handler happen before
>  rpc_transport_pollin_destroy, in this way, when the latter poll in goes to
> rpc_transport_pollin_destroy , there is a chance that the pollin->iobref
> has already been destroyed by the first one(there is no lock destroy for
> iobref->lock in iobref_destroy by the way). That may cause stuck in ?LOCK
> (&iobref->lock);?
>
> I find the one of recent patch
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
> I think the point is to avoid the concurrent handling of the same socket
> at the same time, but after my test with this patch this problem also
> exists, so I think event_handled is still called too early to allow
> concurrent handling of the same socket happen,
>

But concurrent handling is required for performance. So, we cannot
serialize it.

and after move it to the end of socket_event_poll this glusterd stuck issue
> disappeared.
>

Ideally there shouldn't be a single instance of datastructure that should
be shared between two instances of pollin handling. My initial code-reading
didn't find any issues with the way iobref is handled even when there is
concurrent reading when the previous message was still not notified. I'll
continue to investigate how objects are shared across two instances of
pollin. Will post if I find anything interesting.

cynthia
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Monday, April 15, 2019 2:36 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
>
>
>
>
> On Mon, Apr 15, 2019 at 11:08 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Ok, thanks for your comment!
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Monday, April 15, 2019 11:52 AM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Cynthia,
>
>
>
> On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> I made a patch and according to my test, this glusterd stuck issue
> disappear with my patch. Only need to move  event_handled to the end of
> socket_event_poll_in function.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>          }
>
>
>
> -        if (notify_handled && (ret != -1))
>
> -                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> -                               priv->gen);
>
> @@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>                  }
>
>                  pthread_mutex_unlock (&priv->notify.lock);
>
>          }
>
> +        if (notify_handled && (ret != -1))
>
> +                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> +                               priv->gen);
>
>
>
> Thanks for this tip. Though this helps in fixing the hang, this change has
> performance impact. Moving event_handled to end of poll_in means, socket
> will be added back for polling of new events only _after_ the rpc is msg is
> processed by higher layers (like EC) and higher layers can have significant
> latency  for processing the msg. Which means, socket will be out of polling
> for longer periods of time which decreases the throughput (number of msgs
> read per second) affecting performance. However, this experiment definitely
> indicates there is a codepath where event_handled is not called (and hence
> causing the hang). I'll go through this codepath again.
>
>
>
> Can you check whether patch [1] fixes the issue you are seeing?
>
>
>
> [1] https://review.gluster.org/#/c/glusterfs/+/22566
>
>
>
>
>
> Thanks for that experiment :).
>
>
>
>          return ret;
>
> }
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Tuesday, April 09, 2019 3:57 PM
> *To:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* RE: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Can you figure out some possible reason why iobref is corrupted, is it
> possible that thread 8 has called poll in and iobref has been relased, but
> the lock within it is not properly released (as I can not find any free
> lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy
> again, and so stuck on this lock
>
> Also, there should not be two thread handling the same socket at the same
> time, although there has been a patch claimed to tackle this issue.
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Tuesday, April 09, 2019 3:52 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
>
>
>
>
> On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@
> \000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}},
> ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
>
>
>
> looks like the iobref is corrupted here. It seems to be a use-after-free
> issue. We need to dig into why a freed iobref is being accessed here.
>
>
>
> (gdb) quit
>
> A debugging session is active.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/46447893/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 15 08:38:41 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 15 Apr 2019 08:38:41 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CAFkORY_G-in5zOLJqT-wFRxOZLUCJ3FdMZBWRaePP7P0yRFG4g@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
	<32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
	<CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>
	<a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>
	<CAFkORY_G-in5zOLJqT-wFRxOZLUCJ3FdMZBWRaePP7P0yRFG4g@mail.gmail.com>
Message-ID: <5593f364bd7f4dbeb9d1db7f7d17eaf8@nokia-sbell.com>

Ok, I got your point, thanks for responding!

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com>
Sent: Monday, April 15, 2019 4:36 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: gluster-devel at gluster.org
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 15, 2019 at 12:52 PM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
The reason why I move event_handled to the end of socket_event_poll_in is because if event_handled is called before rpc_transport_pollin_destroy, it allowed another round of event_dispatch_epoll_handler happen before  rpc_transport_pollin_destroy, in this way, when the latter poll in goes to rpc_transport_pollin_destroy , there is a chance that the pollin->iobref has already been destroyed by the first one(there is no lock destroy for iobref->lock in iobref_destroy by the way). That may cause stuck in ?LOCK (&iobref->lock);?
I find the one of recent patch
SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket

I think the point is to avoid the concurrent handling of the same socket at the same time, but after my test with this patch this problem also exists, so I think event_handled is still called too early to allow concurrent handling of the same socket happen,

But concurrent handling is required for performance. So, we cannot serialize it.

and after move it to the end of socket_event_poll this glusterd stuck issue disappeared.

Ideally there shouldn't be a single instance of datastructure that should be shared between two instances of pollin handling. My initial code-reading didn't find any issues with the way iobref is handled even when there is concurrent reading when the previous message was still not notified. I'll continue to investigate how objects are shared across two instances of pollin. Will post if I find anything interesting.

cynthia
From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Monday, April 15, 2019 2:36 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 15, 2019 at 11:08 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Ok, thanks for your comment!

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Monday, April 15, 2019 11:52 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15

Cynthia,

On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
I made a patch and according to my test, this glusterd stuck issue disappear with my patch. Only need to move  event_handled to the end of socket_event_poll_in function.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
         }

-        if (notify_handled && (ret != -1))
-                event_handled (ctx->event_pool, priv->sock, priv->idx,
-                               priv->gen);
@@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
                 }
                 pthread_mutex_unlock (&priv->notify.lock);
         }
+        if (notify_handled && (ret != -1))
+                event_handled (ctx->event_pool, priv->sock, priv->idx,
+                               priv->gen);

Thanks for this tip. Though this helps in fixing the hang, this change has performance impact. Moving event_handled to end of poll_in means, socket will be added back for polling of new events only _after_ the rpc is msg is processed by higher layers (like EC) and higher layers can have significant latency  for processing the msg. Which means, socket will be out of polling for longer periods of time which decreases the throughput (number of msgs read per second) affecting performance. However, this experiment definitely indicates there is a codepath where event_handled is not called (and hence causing the hang). I'll go through this codepath again.

Can you check whether patch [1] fixes the issue you are seeing?

[1] https://review.gluster.org/#/c/glusterfs/+/22566


Thanks for that experiment :).

         return ret;
}

cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Tuesday, April 09, 2019 3:57 PM
To: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: glusterd stuck for glusterfs with version 3.12.15

Can you figure out some possible reason why iobref is corrupted, is it possible that thread 8 has called poll in and iobref has been relased, but the lock within it is not properly released (as I can not find any free lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy again, and so stuck on this lock
Also, there should not be two thread handling the same socket at the same time, although there has been a patch claimed to tackle this issue.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Tuesday, April 09, 2019 3:52 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

looks like the iobref is corrupted here. It seems to be a use-after-free issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/1b3d4c08/attachment-0001.html>

From pkarampu at redhat.com  Mon Apr 15 09:08:30 2019
From: pkarampu at redhat.com (Pranith Kumar Karampuri)
Date: Mon, 15 Apr 2019 14:38:30 +0530
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CALtvDK4_FYZ8V-Xs5j_Od-NO=NgnphZ68a5xhYV4braS3i_m7Q@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
	<CALtvDK6ihuZjq0gu-RsqiRRBCCWH6ZsP+poj-wWCfRjfBGNOFw@mail.gmail.com>
	<CALtvDK4_FYZ8V-Xs5j_Od-NO=NgnphZ68a5xhYV4braS3i_m7Q@mail.gmail.com>
Message-ID: <CAOgeEnZpKKYtoGLs7O-+Wf8EKM9-e4JmDqq_X7DqqbYTQJ9ZHQ@mail.gmail.com>

On Thu, Apr 11, 2019 at 2:59 PM Xavi Hernandez <jahernan at redhat.com> wrote:

> On Wed, Apr 10, 2019 at 7:25 PM Xavi Hernandez <jahernan at redhat.com>
> wrote:
>
>> On Wed, Apr 10, 2019 at 4:01 PM Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>> And now for last 15 days:
>>>
>>>
>>> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>>>
>>> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
>>> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
>>> failing in brick mux post 5th April
>>> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux,
>>> needs analysis.
>>>
>>
>> I've identified the problem here, but not the cause yet. There's a stale
>> inodelk acquired by a process that is already dead, which causes inodelk
>> requests from self-heal and other processes to block.
>>
>> The reason why it seemed to block in random places is that all commands
>> are executed with the working directory pointing to a gluster directory
>> which needs healing after the initial tests. Because of the stale inodelk,
>> when any application tries to open a file in the working directory, it's
>> blocked.
>>
>> I'll investigate what causes this.
>>
>
> I think I've found the problem. This is a fragment of the brick log that
> includes script steps, connections and disconnections of brick 0, and lock
> requests to the problematic lock:
>
> [2019-04-11 08:22:20.381398]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++++++++++
> [2019-04-11 08:22:22.532646]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++++++++++
> [2019-04-11 08:22:23.709655] I [MSGID: 115029]
> [server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
> from
> CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2
> (version: 7dev) with subvol /d/backends/patchy1
> [2019-04-11 08:22:23.792204] I [common.c:234:pl_trace_in] 8-patchy-locks:
> [REQUEST] Locker = {Pid=29710, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
> Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
> [2019-04-11 08:22:23.792299] I [common.c:285:pl_trace_out] 8-patchy-locks:
> [GRANTED] Locker = {Pid=29710, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
> Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
> [2019-04-11 08:22:24.628478]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 68 5 online_brick_count ++++++++++
> [2019-04-11 08:22:26.097092]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
> 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
> [2019-04-11 08:22:26.333740]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 71 ec_test_make ++++++++++
> [2019-04-11 08:22:27.718963] I [MSGID: 115029]
> [server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
> from
> CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3
> (version: 7dev) with subvol /d/backends/patchy1
> [2019-04-11 08:22:27.801416] I [common.c:234:pl_trace_in] 8-patchy-locks:
> [REQUEST] Locker = {Pid=29885, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
> Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
> [2019-04-11 08:22:27.801434] E [inodelk.c:513:__inode_unlock_lock]
> 8-patchy-locks:  Matching lock not found for unlock 0-9223372036854775807,
> by 68580998b47f0000 on 0x7f0ed0029190
> [2019-04-11 08:22:27.801446] I [common.c:285:pl_trace_out] 8-patchy-locks:
> [Invalid argument] Locker = {Pid=29885, lk-owner=68580998b47f0000,
> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
> Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
> patchy-disperse-0, start=0, len=0, pid=0}
>
> This is a fragment of the client log:
>
> [2019-04-11 08:22:20.381398]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 66 kill_brick patchy jahernan /d/backends/patchy2 ++++++++++
> [2019-04-11 08:22:20.675938] I [MSGID: 114018]
> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
> patchy-client-1. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-04-11 08:22:21.674772] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (6). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:22.532646]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 67 kill_brick patchy jahernan /d/backends/patchy3 ++++++++++
> [2019-04-11 08:22:23.691171] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (8). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:23.710420] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
> [2019-04-11 08:22:23.791635] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
> [2019-04-11 08:22:24.460529] I [MSGID: 114018]
> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
> patchy-client-1. Client process will keep trying to connect to glusterd
> until brick's port is available
> [2019-04-11 08:22:24.628478]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 68 5 online_brick_count ++++++++++
> [2019-04-11 08:22:26.097092]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o 14.o 15.o 16.o 17.o 18.o 19.o 1.o
> 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
> [2019-04-11 08:22:26.333740]:++++++++++ G_LOG:tests/bugs/ec/bug-1236065.t:
> TEST: 71 ec_test_make ++++++++++
> [2019-04-11 08:22:27.719299] I [MSGID: 114046]
> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
> [2019-04-11 08:22:27.840342] W [MSGID: 122035]
> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>
> The problem happens for two things:
>
> 1. Brick 0 gets disconnected randomly (apparently), but the server side is
> not aware of these disconnections. This causes that at 08:22:24.460529, the
> client has already sent a successful INODELK request to brick 0. At this
> point the connection is broken on the client side, but server side doesn't
> get any notification, so it doesn't clear the locks.
>
2. When client reconnects at 08:22:27.719299, a new connection is created,
> and the servers does see this new connection (it creates a new client_t
> structure). Then the client sends the unlock request, which fails on brick
> 0 because locks xlators checks if the client is the same by comparing the
> pointers, but they are different because of the reconnection. So the lock
> is not unlocked and remains there, blocking all future inodelk requests.
>
> The first problem is why the client gets disconnected and the server
> doesn't get any notification. The script is stopping bricks 2 and 3 when
> this happens. Brick 0 shouldn't fail here. It seems related to the
>
> The second problem is that when we receive a new connection from a client
> we already consider connected, we don't cleanup the old connection, which
> should take care of the stale locks.
>
> The third problem is that locks xlator is checking if the client is the
> same by comparing pointers of client_t structs instead of comparing
> client_uid field, which remains the same.
>
> Adding +Raghavendra Gowdappa <rgowdapp at redhat.com>, +Pranith Kumar
> Karampuri <pkarampu at redhat.com>, +Krutika Dhananjay <kdhananj at redhat.com>
> , +Shyam Ranganathan <srangana at redhat.com> and +Amar Tumballi
> <atumball at redhat.com> to help me identify why this is happening and
> what's the best way to solve it.
>

If server gets disconnect notification, then everything will be solved. I
think we need to find RC for that. Were you able to recreate it locally
even if it is happening once in a while that is fine.


>
> Xavi
>
>
>> Xavi
>>
>> ./tests/basic/uss.t             15  ==> happens in both brick mux and non
>>> brick mux runs, test just simply times out. Needs urgent analysis.
>>> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
>>> https://review.gluster.org/#/c/22508/ , patch merged today.
>>> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this
>>> succeeds in second attempt every time.
>>>
>>> There're plenty more with 5 instances of failure from many tests. We
>>> need all maintainers/owners to look through these failures and fix them, we
>>> certainly don't want to get into a stage where master is unstable and we
>>> have to lock down the merges till all these failures are resolved. So
>>> please help.
>>>
>>> (Please note fstat stats show up the retries as failures too which in a
>>> way is right)
>>>
>>>
>>> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
>>> wrote:
>>>
>>>> [1] captures the test failures report since last 30 days and we'd need
>>>> volunteers/component owners to see why the number of failures are so high
>>>> against few tests.
>>>>
>>>> [1]
>>>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>

-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/bf3960ea/attachment-0001.html>

From jahernan at redhat.com  Mon Apr 15 14:19:50 2019
From: jahernan at redhat.com (Xavi Hernandez)
Date: Mon, 15 Apr 2019 16:19:50 +0200
Subject: [Gluster-devel] test failure reports for last 15 days
In-Reply-To: <CAOgeEnZpKKYtoGLs7O-+Wf8EKM9-e4JmDqq_X7DqqbYTQJ9ZHQ@mail.gmail.com>
References: <CAGNCGH1=8G1ZAj-Ebyys8-hhfPAxsifHTa0DjkU3rDThU108HA@mail.gmail.com>
	<CAGNCGH1dV4q1v53SnZqw8_gQMKEyUXrHi=cAhtjsawpedBK2ew@mail.gmail.com>
	<CALtvDK6ihuZjq0gu-RsqiRRBCCWH6ZsP+poj-wWCfRjfBGNOFw@mail.gmail.com>
	<CALtvDK4_FYZ8V-Xs5j_Od-NO=NgnphZ68a5xhYV4braS3i_m7Q@mail.gmail.com>
	<CAOgeEnZpKKYtoGLs7O-+Wf8EKM9-e4JmDqq_X7DqqbYTQJ9ZHQ@mail.gmail.com>
Message-ID: <CALtvDK65n-sdxK3282h5EDABVoxrJbq3WBK3zkLPbnVH9b9ETw@mail.gmail.com>

On Mon, Apr 15, 2019 at 11:08 AM Pranith Kumar Karampuri <
pkarampu at redhat.com> wrote:

>
>
> On Thu, Apr 11, 2019 at 2:59 PM Xavi Hernandez <jahernan at redhat.com>
> wrote:
>
>> On Wed, Apr 10, 2019 at 7:25 PM Xavi Hernandez <jahernan at redhat.com>
>> wrote:
>>
>>> On Wed, Apr 10, 2019 at 4:01 PM Atin Mukherjee <amukherj at redhat.com>
>>> wrote:
>>>
>>>> And now for last 15 days:
>>>>
>>>>
>>>> https://fstat.gluster.org/summary?start_date=2019-03-25&end_date=2019-04-10
>>>>
>>>> ./tests/bitrot/bug-1373520.t     18  ==> Fixed through
>>>> https://review.gluster.org/#/c/glusterfs/+/22481/, I don't see this
>>>> failing in brick mux post 5th April
>>>> ./tests/bugs/ec/bug-1236065.t     17  ==> happens only in brick mux,
>>>> needs analysis.
>>>>
>>>
>>> I've identified the problem here, but not the cause yet. There's a stale
>>> inodelk acquired by a process that is already dead, which causes inodelk
>>> requests from self-heal and other processes to block.
>>>
>>> The reason why it seemed to block in random places is that all commands
>>> are executed with the working directory pointing to a gluster directory
>>> which needs healing after the initial tests. Because of the stale inodelk,
>>> when any application tries to open a file in the working directory, it's
>>> blocked.
>>>
>>> I'll investigate what causes this.
>>>
>>
>> I think I've found the problem. This is a fragment of the brick log that
>> includes script steps, connections and disconnections of brick 0, and lock
>> requests to the problematic lock:
>>
>> [2019-04-11 08:22:20.381398]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 66 kill_brick patchy jahernan
>> /d/backends/patchy2 ++++++++++
>> [2019-04-11 08:22:22.532646]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 67 kill_brick patchy jahernan
>> /d/backends/patchy3 ++++++++++
>> [2019-04-11 08:22:23.709655] I [MSGID: 115029]
>> [server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
>> from
>> CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2
>> (version: 7dev) with subvol /d/backends/patchy1
>> [2019-04-11 08:22:23.792204] I [common.c:234:pl_trace_in] 8-patchy-locks:
>> [REQUEST] Locker = {Pid=29710, lk-owner=68580998b47f0000,
>> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
>> Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
>> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
>> patchy-disperse-0, start=0, len=0, pid=0}
>> [2019-04-11 08:22:23.792299] I [common.c:285:pl_trace_out]
>> 8-patchy-locks: [GRANTED] Locker = {Pid=29710, lk-owner=68580998b47f0000,
>> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-2,
>> Frame=18676} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
>> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=WRITE, domain:
>> patchy-disperse-0, start=0, len=0, pid=0}
>> [2019-04-11 08:22:24.628478]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 68 5 online_brick_count ++++++++++
>> [2019-04-11 08:22:26.097092]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o
>> 14.o 15.o 16.o 17.o 18.o 19.o 1.o 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
>> [2019-04-11 08:22:26.333740]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 71 ec_test_make ++++++++++
>> [2019-04-11 08:22:27.718963] I [MSGID: 115029]
>> [server-handshake.c:550:server_setvolume] 0-patchy-server: accepted client
>> from
>> CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3
>> (version: 7dev) with subvol /d/backends/patchy1
>> [2019-04-11 08:22:27.801416] I [common.c:234:pl_trace_in] 8-patchy-locks:
>> [REQUEST] Locker = {Pid=29885, lk-owner=68580998b47f0000,
>> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
>> Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
>> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
>> patchy-disperse-0, start=0, len=0, pid=0}
>> [2019-04-11 08:22:27.801434] E [inodelk.c:513:__inode_unlock_lock]
>> 8-patchy-locks:  Matching lock not found for unlock 0-9223372036854775807,
>> by 68580998b47f0000 on 0x7f0ed0029190
>> [2019-04-11 08:22:27.801446] I [common.c:285:pl_trace_out]
>> 8-patchy-locks: [Invalid argument] Locker = {Pid=29885,
>> lk-owner=68580998b47f0000,
>> Client=CTX_ID:1c2952c2-e90f-4631-8712-170b8c05aa6e-GRAPH_ID:0-PID:28900-HOST:jahernan-PC_NAME:patchy-client-1-RECON_NO:-3,
>> Frame=19233} Lockee = {gfid=35743386-b7c2-41c9-aafd-6b13de216704, fd=(nil),
>> path=/test} Lock = {lock=INODELK, cmd=SETLK, type=UNLOCK, domain:
>> patchy-disperse-0, start=0, len=0, pid=0}
>>
>> This is a fragment of the client log:
>>
>> [2019-04-11 08:22:20.381398]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 66 kill_brick patchy jahernan
>> /d/backends/patchy2 ++++++++++
>> [2019-04-11 08:22:20.675938] I [MSGID: 114018]
>> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
>> patchy-client-1. Client process will keep trying to connect to glusterd
>> until brick's port is available
>> [2019-04-11 08:22:21.674772] W [MSGID: 122035]
>> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
>> with some subvolumes unavailable. (6). FOP : 'INODELK' failed on '/test'
>> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>> [2019-04-11 08:22:22.532646]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 67 kill_brick patchy jahernan
>> /d/backends/patchy3 ++++++++++
>> [2019-04-11 08:22:23.691171] W [MSGID: 122035]
>> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
>> with some subvolumes unavailable. (8). FOP : 'INODELK' failed on '/test'
>> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>> [2019-04-11 08:22:23.710420] I [MSGID: 114046]
>> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
>> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
>> [2019-04-11 08:22:23.791635] W [MSGID: 122035]
>> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
>> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
>> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>> [2019-04-11 08:22:24.460529] I [MSGID: 114018]
>> [client.c:2333:client_rpc_notify] 0-patchy-client-1: disconnected from
>> patchy-client-1. Client process will keep trying to connect to glusterd
>> until brick's port is available
>> [2019-04-11 08:22:24.628478]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 68 5 online_brick_count ++++++++++
>> [2019-04-11 08:22:26.097092]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 70 rm -f 0.o 10.o 11.o 12.o 13.o
>> 14.o 15.o 16.o 17.o 18.o 19.o 1.o 2.o 3.o 4.o 5.o 6.o 7.o 8.o 9.o ++++++++++
>> [2019-04-11 08:22:26.333740]:++++++++++
>> G_LOG:tests/bugs/ec/bug-1236065.t: TEST: 71 ec_test_make ++++++++++
>> [2019-04-11 08:22:27.719299] I [MSGID: 114046]
>> [client-handshake.c:1106:client_setvolume_cbk] 0-patchy-client-1: Connected
>> to patchy-client-1, attached to remote volume '/d/backends/patchy1'.
>> [2019-04-11 08:22:27.840342] W [MSGID: 122035]
>> [ec-common.c:654:ec_child_select] 0-patchy-disperse-0: Executing operation
>> with some subvolumes unavailable. (C). FOP : 'INODELK' failed on '/test'
>> with gfid 35743386-b7c2-41c9-aafd-6b13de216704
>>
>> The problem happens for two things:
>>
>> 1. Brick 0 gets disconnected randomly (apparently), but the server side
>> is not aware of these disconnections. This causes that at 08:22:24.460529,
>> the client has already sent a successful INODELK request to brick 0. At
>> this point the connection is broken on the client side, but server side
>> doesn't get any notification, so it doesn't clear the locks.
>>
> 2. When client reconnects at 08:22:27.719299, a new connection is created,
>> and the servers does see this new connection (it creates a new client_t
>> structure). Then the client sends the unlock request, which fails on brick
>> 0 because locks xlators checks if the client is the same by comparing the
>> pointers, but they are different because of the reconnection. So the lock
>> is not unlocked and remains there, blocking all future inodelk requests.
>>
>> The first problem is why the client gets disconnected and the server
>> doesn't get any notification. The script is stopping bricks 2 and 3 when
>> this happens. Brick 0 shouldn't fail here. It seems related to the
>>
>> The second problem is that when we receive a new connection from a client
>> we already consider connected, we don't cleanup the old connection, which
>> should take care of the stale locks.
>>
>> The third problem is that locks xlator is checking if the client is the
>> same by comparing pointers of client_t structs instead of comparing
>> client_uid field, which remains the same.
>>
>> Adding +Raghavendra Gowdappa <rgowdapp at redhat.com>, +Pranith Kumar
>> Karampuri <pkarampu at redhat.com>, +Krutika Dhananjay <kdhananj at redhat.com>
>> , +Shyam Ranganathan <srangana at redhat.com> and +Amar Tumballi
>> <atumball at redhat.com> to help me identify why this is happening and
>> what's the best way to solve it.
>>
>
> If server gets disconnect notification, then everything will be solved. I
> think we need to find RC for that. Were you able to recreate it locally
> even if it is happening once in a while that is fine.
>

I think the problem is not the a missed disconnection notification on the
server. The problem is that sometimes (not always but relatively frequently
running the test script), when the script kills one brick, another one also
gets disconnected from the client point of view. This is not correct. If
the script wants to kill one brick, only that one should die. Otherwise the
script can fail unexpectedly. Apparently there's no other problem that
could justify why the brick has been disconnected.

Xavi


>
>>
>> Xavi
>>
>>
>>> Xavi
>>>
>>> ./tests/basic/uss.t             15  ==> happens in both brick mux and
>>>> non brick mux runs, test just simply times out. Needs urgent analysis.
>>>> ./tests/basic/ec/ec-fix-openfd.t 13  ==> Fixed through
>>>> https://review.gluster.org/#/c/22508/ , patch merged today.
>>>> ./tests/basic/volfile-sanity.t      8  ==> Some race, though this
>>>> succeeds in second attempt every time.
>>>>
>>>> There're plenty more with 5 instances of failure from many tests. We
>>>> need all maintainers/owners to look through these failures and fix them, we
>>>> certainly don't want to get into a stage where master is unstable and we
>>>> have to lock down the merges till all these failures are resolved. So
>>>> please help.
>>>>
>>>> (Please note fstat stats show up the retries as failures too which in a
>>>> way is right)
>>>>
>>>>
>>>> On Tue, Feb 26, 2019 at 5:27 PM Atin Mukherjee <amukherj at redhat.com>
>>>> wrote:
>>>>
>>>>> [1] captures the test failures report since last 30 days and we'd need
>>>>> volunteers/component owners to see why the number of failures are so high
>>>>> against few tests.
>>>>>
>>>>> [1]
>>>>> https://fstat.gluster.org/summary?start_date=2019-01-26&end_date=2019-02-25&job=all
>>>>>
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>
> --
> Pranith
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190415/7573f3f9/attachment-0001.html>

From srangana at redhat.com  Tue Apr 16 13:53:04 2019
From: srangana at redhat.com (Shyam Ranganathan)
Date: Tue, 16 Apr 2019 09:53:04 -0400
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
Message-ID: <6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>

Status: Tagging pending

Waiting on patches:
(Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
  https://review.gluster.org/c/glusterfs/+/22579

Following patches will not be taken in if CentOS regression does not
pass by tomorrow morning Eastern TZ,
(Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
inodelk/entrylk
  https://review.gluster.org/c/glusterfs/+/22385
(Aravinda) - geo-rep: IPv6 support
  https://review.gluster.org/c/glusterfs/+/22488
(Aravinda) - geo-rep: fix integer config validation
  https://review.gluster.org/c/glusterfs/+/22489

Tracker bug status:
(Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
RHGSWA.
  All patches are merged, but none of the patches adds the "Fixes"
keyword, assume this is an oversight and that the bug is fixed in this
release.

(Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
the same brick, causing transport endpoint not connected
  No work has occurred post logs upload to bug, restart of bircks and
possibly glusterd is the existing workaround when the bug is hit. Moving
this out of the tracker for 6.1.

(Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
replace-brick is executed
  Very recent bug (15th April), does not seem to have any critical data
corruption or service availability issues, planning on not waiting for
the fix in 6.1

- Shyam
On 4/6/19 4:38 AM, Atin Mukherjee wrote:
> Hi Mohit,
> 
> https://review.gluster.org/22495 should get into 6.1 as it?s a
> regression. Can you please attach the respective bug to the tracker Ravi
> pointed out?
> 
> 
> On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
> <mailto:ravishankar at redhat.com>> wrote:
> 
>     Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in
>     case anyone wants to add blocker bugs.
> 
> 
>     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>     > Hi,
>     >
>     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>     >
>     > Please ensure required patches are backported and also are passing
>     > regressions and are appropriately reviewed for easy merging and
>     tagging
>     > on the date.
>     >
>     > Thanks,
>     > Shyam
>     > _______________________________________________
>     > Gluster-devel mailing list
>     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>     _______________________________________________
>     Gluster-devel mailing list
>     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>     https://lists.gluster.org/mailman/listinfo/gluster-devel
> 
> 
> -- 
> - Atin (atinm)
> 
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
> 

From amukherj at redhat.com  Tue Apr 16 15:49:47 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Tue, 16 Apr 2019 21:19:47 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
Message-ID: <CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>

On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
wrote:

> Status: Tagging pending
>
> Waiting on patches:
> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>   https://review.gluster.org/c/glusterfs/+/22579


The regression doesn't pass for the mainline patch. I believe master is
broken now. With latest master sdfs-sanity.t always fail. We either need to
fix it or mark it as bad test.

root at a5f81bd447c2:/home/glusterfs# prove -vf tests/basic/sdfs-sanity.t
tests/basic/sdfs-sanity.t ..
1..7
ok 1, LINENUM:8
ok 2, LINENUM:9
ok 3, LINENUM:11
ok 4, LINENUM:12
ok 5, LINENUM:13
ok 6, LINENUM:16
mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid argument
stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
not ok 7 , LINENUM:20
FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
Failed 1/7 subtests

Test Summary Report
-------------------
tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
  Failed test:  7
Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr  0.67
csys =  1.27 CPU)
Result: FAIL


>
> Following patches will not be taken in if CentOS regression does not
> pass by tomorrow morning Eastern TZ,
> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
> inodelk/entrylk
>   https://review.gluster.org/c/glusterfs/+/22385
> (Aravinda) - geo-rep: IPv6 support
>   https://review.gluster.org/c/glusterfs/+/22488
> (Aravinda) - geo-rep: fix integer config validation
>   https://review.gluster.org/c/glusterfs/+/22489
>
> Tracker bug status:
> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
> RHGSWA.
>   All patches are merged, but none of the patches adds the "Fixes"
> keyword, assume this is an oversight and that the bug is fixed in this
> release.
>
> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
> the same brick, causing transport endpoint not connected
>   No work has occurred post logs upload to bug, restart of bircks and
> possibly glusterd is the existing workaround when the bug is hit. Moving
> this out of the tracker for 6.1.
>
> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
> replace-brick is executed
>   Very recent bug (15th April), does not seem to have any critical data
> corruption or service availability issues, planning on not waiting for
> the fix in 6.1
>
> - Shyam
> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
> > Hi Mohit,
> >
> > https://review.gluster.org/22495 should get into 6.1 as it?s a
> > regression. Can you please attach the respective bug to the tracker Ravi
> > pointed out?
> >
> >
> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
> > <mailto:ravishankar at redhat.com>> wrote:
> >
> >     Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394,
> in
> >     case anyone wants to add blocker bugs.
> >
> >
> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
> >     > Hi,
> >     >
> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
> >     >
> >     > Please ensure required patches are backported and also are passing
> >     > regressions and are appropriately reviewed for easy merging and
> >     tagging
> >     > on the date.
> >     >
> >     > Thanks,
> >     > Shyam
> >     > _______________________________________________
> >     > Gluster-devel mailing list
> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >     _______________________________________________
> >     Gluster-devel mailing list
> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
> >
> >
> > --
> > - Atin (atinm)
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-devel
> >
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190416/e51d21b0/attachment.html>

From amukherj at redhat.com  Tue Apr 16 16:56:54 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Tue, 16 Apr 2019 22:26:54 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
Message-ID: <CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>

On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com> wrote:

>
>
> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
> wrote:
>
>> Status: Tagging pending
>>
>> Waiting on patches:
>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>   https://review.gluster.org/c/glusterfs/+/22579
>
>
> The regression doesn't pass for the mainline patch. I believe master is
> broken now. With latest master sdfs-sanity.t always fail. We either need to
> fix it or mark it as bad test.
>

commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
Author: Pranith Kumar K <pkarampu at redhat.com>
Date:   Mon Apr 1 11:14:56 2019 +0530

   features/locks: error-out {inode,entry}lk fops with all-zero lk-owner

   Problem:
   Sometimes we find that developers forget to assign lk-owner for an
   inodelk/entrylk/lk before writing code to wind these fops. locks
   xlator at the moment allows this operation. This leads to multiple
   threads in the same client being able to get locks on the inode
   because lk-owner is same and transport is same. So isolation
   with locks can't be achieved.

   Fix:
   Disallow locks with lk-owner zero.

   fixes bz#1624701
   Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
   Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>

With the above commit sdfs-sanity.t started failing. But when I looked at
the last regression vote at
https://build.gluster.org/job/centos7-regression/5568/consoleFull I saw it
voted back positive but the bell rang when I saw the overall regression
took less than 2 hours and when I opened the regression link I saw the test
actually failed but still this job voted back +1 at gerrit.

*Deepshika* - *This is a bad CI bug we have now and have to be addressed at
earliest. Please take a look at
https://build.gluster.org/job/centos7-regression/5568/consoleFull
<https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
investigate why the regression vote wasn't negative.*

Pranith - I request you to investigate on the sdfs-sanity.t failure because
of this patch.

*@Maintainers - Please open up every regression link to see the actual
status of the job and don't blindly trust on the +1 vote back at gerrit
till this is addressed.*

As per the policy, I'm going to revert this commit, watch out for the
patch. I request this to be directly pushed with out waiting for the
regression vote as we had done before in such breakage. Amar/Shyam - I
believe you have this permission?


> root at a5f81bd447c2:/home/glusterfs# prove -vf tests/basic/sdfs-sanity.t
> tests/basic/sdfs-sanity.t ..
> 1..7
> ok 1, LINENUM:8
> ok 2, LINENUM:9
> ok 3, LINENUM:11
> ok 4, LINENUM:12
> ok 5, LINENUM:13
> ok 6, LINENUM:16
> mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid
> argument
> stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
> tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
> not ok 7 , LINENUM:20
> FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
> Failed 1/7 subtests
>
> Test Summary Report
> -------------------
> tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
>   Failed test:  7
> Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr
> 0.67 csys =  1.27 CPU)
> Result: FAIL
>
>
>>
>> Following patches will not be taken in if CentOS regression does not
>> pass by tomorrow morning Eastern TZ,
>> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
>> inodelk/entrylk
>>   https://review.gluster.org/c/glusterfs/+/22385
>> (Aravinda) - geo-rep: IPv6 support
>>   https://review.gluster.org/c/glusterfs/+/22488
>> (Aravinda) - geo-rep: fix integer config validation
>>   https://review.gluster.org/c/glusterfs/+/22489
>>
>> Tracker bug status:
>> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
>> RHGSWA.
>>   All patches are merged, but none of the patches adds the "Fixes"
>> keyword, assume this is an oversight and that the bug is fixed in this
>> release.
>>
>> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
>> the same brick, causing transport endpoint not connected
>>   No work has occurred post logs upload to bug, restart of bircks and
>> possibly glusterd is the existing workaround when the bug is hit. Moving
>> this out of the tracker for 6.1.
>>
>> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
>> replace-brick is executed
>>   Very recent bug (15th April), does not seem to have any critical data
>> corruption or service availability issues, planning on not waiting for
>> the fix in 6.1
>>
>> - Shyam
>> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>> > Hi Mohit,
>> >
>> > https://review.gluster.org/22495 should get into 6.1 as it?s a
>> > regression. Can you please attach the respective bug to the tracker Ravi
>> > pointed out?
>> >
>> >
>> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>> > <mailto:ravishankar at redhat.com>> wrote:
>> >
>> >     Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394,
>> in
>> >     case anyone wants to add blocker bugs.
>> >
>> >
>> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>> >     > Hi,
>> >     >
>> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>> >     >
>> >     > Please ensure required patches are backported and also are passing
>> >     > regressions and are appropriately reviewed for easy merging and
>> >     tagging
>> >     > on the date.
>> >     >
>> >     > Thanks,
>> >     > Shyam
>> >     > _______________________________________________
>> >     > Gluster-devel mailing list
>> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>> >     _______________________________________________
>> >     Gluster-devel mailing list
>> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
>> >
>> >
>> > --
>> > - Atin (atinm)
>> >
>> > _______________________________________________
>> > Gluster-devel mailing list
>> > Gluster-devel at gluster.org
>> > https://lists.gluster.org/mailman/listinfo/gluster-devel
>> >
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190416/eb023b16/attachment-0001.html>

From amukherj at redhat.com  Tue Apr 16 17:11:10 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Tue, 16 Apr 2019 22:41:10 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
	<CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
Message-ID: <CAGNCGH2rZmKidX+RwaxYE18rcWP4dsxsqn-3FUeqNk2vW4jESg@mail.gmail.com>

On Tue, Apr 16, 2019 at 10:26 PM Atin Mukherjee <amukherj at redhat.com> wrote:

>
>
> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>>
>>
>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
>> wrote:
>>
>>> Status: Tagging pending
>>>
>>> Waiting on patches:
>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>   https://review.gluster.org/c/glusterfs/+/22579
>>
>>
>> The regression doesn't pass for the mainline patch. I believe master is
>> broken now. With latest master sdfs-sanity.t always fail. We either need to
>> fix it or mark it as bad test.
>>
>
> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
> Author: Pranith Kumar K <pkarampu at redhat.com>
> Date:   Mon Apr 1 11:14:56 2019 +0530
>
>    features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>
>    Problem:
>    Sometimes we find that developers forget to assign lk-owner for an
>    inodelk/entrylk/lk before writing code to wind these fops. locks
>    xlator at the moment allows this operation. This leads to multiple
>    threads in the same client being able to get locks on the inode
>    because lk-owner is same and transport is same. So isolation
>    with locks can't be achieved.
>
>    Fix:
>    Disallow locks with lk-owner zero.
>
>    fixes bz#1624701
>    Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>
> With the above commit sdfs-sanity.t started failing. But when I looked at
> the last regression vote at
> https://build.gluster.org/job/centos7-regression/5568/consoleFull I saw
> it voted back positive but the bell rang when I saw the overall regression
> took less than 2 hours and when I opened the regression link I saw the test
> actually failed but still this job voted back +1 at gerrit.
>
> *Deepshika* - *This is a bad CI bug we have now and have to be addressed
> at earliest. Please take a look at
> https://build.gluster.org/job/centos7-regression/5568/consoleFull
> <https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
> investigate why the regression vote wasn't negative.*
>
> Pranith - I request you to investigate on the sdfs-sanity.t failure
> because of this patch.
>
> *@Maintainers - Please open up every regression link to see the actual
> status of the job and don't blindly trust on the +1 vote back at gerrit
> till this is addressed.*
>
> As per the policy, I'm going to revert this commit, watch out for the
> patch.
>

https://review.gluster.org/#/c/glusterfs/+/22581/
Please review and merge it.

Also since we're already close to 23:00 in IST timezone, I need help from
folks from other timezone in getting
https://review.gluster.org/#/c/glusterfs/+/22578/ rebased and marked
verified +1 once the above fix is merged. This is a blocker to
glusterfs-6.1 as otherwise ctime feature option tuning isn't honoured.

I request this to be directly pushed with out waiting for the regression
> vote as we had done before in such breakage. Amar/Shyam - I believe you
> have this permission?
>
>
>> root at a5f81bd447c2:/home/glusterfs# prove -vf tests/basic/sdfs-sanity.t
>> tests/basic/sdfs-sanity.t ..
>> 1..7
>> ok 1, LINENUM:8
>> ok 2, LINENUM:9
>> ok 3, LINENUM:11
>> ok 4, LINENUM:12
>> ok 5, LINENUM:13
>> ok 6, LINENUM:16
>> mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid
>> argument
>> stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
>> tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
>> not ok 7 , LINENUM:20
>> FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
>> Failed 1/7 subtests
>>
>> Test Summary Report
>> -------------------
>> tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
>>   Failed test:  7
>> Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr
>> 0.67 csys =  1.27 CPU)
>> Result: FAIL
>>
>>
>>>
>>> Following patches will not be taken in if CentOS regression does not
>>> pass by tomorrow morning Eastern TZ,
>>> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
>>> inodelk/entrylk
>>>   https://review.gluster.org/c/glusterfs/+/22385
>>> (Aravinda) - geo-rep: IPv6 support
>>>   https://review.gluster.org/c/glusterfs/+/22488
>>> (Aravinda) - geo-rep: fix integer config validation
>>>   https://review.gluster.org/c/glusterfs/+/22489
>>>
>>> Tracker bug status:
>>> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
>>> RHGSWA.
>>>   All patches are merged, but none of the patches adds the "Fixes"
>>> keyword, assume this is an oversight and that the bug is fixed in this
>>> release.
>>>
>>> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
>>> the same brick, causing transport endpoint not connected
>>>   No work has occurred post logs upload to bug, restart of bircks and
>>> possibly glusterd is the existing workaround when the bug is hit. Moving
>>> this out of the tracker for 6.1.
>>>
>>> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
>>> replace-brick is executed
>>>   Very recent bug (15th April), does not seem to have any critical data
>>> corruption or service availability issues, planning on not waiting for
>>> the fix in 6.1
>>>
>>> - Shyam
>>> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>>> > Hi Mohit,
>>> >
>>> > https://review.gluster.org/22495 should get into 6.1 as it?s a
>>> > regression. Can you please attach the respective bug to the tracker
>>> Ravi
>>> > pointed out?
>>> >
>>> >
>>> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>>> > <mailto:ravishankar at redhat.com>> wrote:
>>> >
>>> >     Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394,
>>> in
>>> >     case anyone wants to add blocker bugs.
>>> >
>>> >
>>> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>>> >     > Hi,
>>> >     >
>>> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>>> >     >
>>> >     > Please ensure required patches are backported and also are
>>> passing
>>> >     > regressions and are appropriately reviewed for easy merging and
>>> >     tagging
>>> >     > on the date.
>>> >     >
>>> >     > Thanks,
>>> >     > Shyam
>>> >     > _______________________________________________
>>> >     > Gluster-devel mailing list
>>> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >     _______________________________________________
>>> >     Gluster-devel mailing list
>>> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> >
>>> > --
>>> > - Atin (atinm)
>>> >
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190416/3a1e4ee6/attachment.html>

From sankarshan.mukhopadhyay at gmail.com  Tue Apr 16 17:50:45 2019
From: sankarshan.mukhopadhyay at gmail.com (Sankarshan Mukhopadhyay)
Date: Tue, 16 Apr 2019 23:20:45 +0530
Subject: [Gluster-devel] [Gluster-Maintainers] Release 6.1: Expected
 tagging on April 10th
In-Reply-To: <CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
	<CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
Message-ID: <CAJWA-5bCUmT-Gvbh3zg1GdAWFXUWjridFPMZnjwcNkeWFtzhKg@mail.gmail.com>

On Tue, Apr 16, 2019 at 10:27 PM Atin Mukherjee <amukherj at redhat.com> wrote:
> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com> wrote:
>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com> wrote:
>>>
>>> Status: Tagging pending
>>>
>>> Waiting on patches:
>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>   https://review.gluster.org/c/glusterfs/+/22579
>>
>>
>> The regression doesn't pass for the mainline patch. I believe master is broken now. With latest master sdfs-sanity.t always fail. We either need to fix it or mark it as bad test.
>
>
> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
> Author: Pranith Kumar K <pkarampu at redhat.com>
> Date:   Mon Apr 1 11:14:56 2019 +0530
>
>    features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>
>    Problem:
>    Sometimes we find that developers forget to assign lk-owner for an
>    inodelk/entrylk/lk before writing code to wind these fops. locks
>    xlator at the moment allows this operation. This leads to multiple
>    threads in the same client being able to get locks on the inode
>    because lk-owner is same and transport is same. So isolation
>    with locks can't be achieved.
>
>    Fix:
>    Disallow locks with lk-owner zero.
>
>    fixes bz#1624701
>    Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>
> With the above commit sdfs-sanity.t started failing. But when I looked at the last regression vote at https://build.gluster.org/job/centos7-regression/5568/consoleFull I saw it voted back positive but the bell rang when I saw the overall regression took less than 2 hours and when I opened the regression link I saw the test actually failed but still this job voted back +1 at gerrit.
>
> Deepshika - This is a bad CI bug we have now and have to be addressed at earliest. Please take a look at https://build.gluster.org/job/centos7-regression/5568/consoleFull and investigate why the regression vote wasn't negative.

Atin, we (Deepshikha and I) agree with your assessment.

This is the kind of situation that reduces the trust in our
application build pipeline. This is a result of a minor change
introduced to fix the constant issue we have observed with non-voting.
This is something that should not have slipped through and it did. We
will be observing a random sampling of the jobs to ensure that we gate
any such incidents that reduce the utility value of the pipeline. We
will be reviewing the change to the scripts which have since also had
the fix for the issue which led to this situation in the first place.

From pkarampu at redhat.com  Tue Apr 16 19:02:56 2019
From: pkarampu at redhat.com (Pranith Kumar Karampuri)
Date: Wed, 17 Apr 2019 00:32:56 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
	<CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
Message-ID: <CAOgeEnYd3P18Vt29WPCkZUwfhDJNqbq=OaVrAXzKbArGo6ECEQ@mail.gmail.com>

On Tue, Apr 16, 2019 at 10:27 PM Atin Mukherjee <amukherj at redhat.com> wrote:

>
>
> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>>
>>
>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
>> wrote:
>>
>>> Status: Tagging pending
>>>
>>> Waiting on patches:
>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>   https://review.gluster.org/c/glusterfs/+/22579
>>
>>
>> The regression doesn't pass for the mainline patch. I believe master is
>> broken now. With latest master sdfs-sanity.t always fail. We either need to
>> fix it or mark it as bad test.
>>
>
> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
> Author: Pranith Kumar K <pkarampu at redhat.com>
> Date:   Mon Apr 1 11:14:56 2019 +0530
>
>    features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>
>    Problem:
>    Sometimes we find that developers forget to assign lk-owner for an
>    inodelk/entrylk/lk before writing code to wind these fops. locks
>    xlator at the moment allows this operation. This leads to multiple
>    threads in the same client being able to get locks on the inode
>    because lk-owner is same and transport is same. So isolation
>    with locks can't be achieved.
>
>    Fix:
>    Disallow locks with lk-owner zero.
>
>    fixes bz#1624701
>    Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>
> With the above commit sdfs-sanity.t started failing. But when I looked at
> the last regression vote at
> https://build.gluster.org/job/centos7-regression/5568/consoleFull I saw
> it voted back positive but the bell rang when I saw the overall regression
> took less than 2 hours and when I opened the regression link I saw the test
> actually failed but still this job voted back +1 at gerrit.
>
> *Deepshika* - *This is a bad CI bug we have now and have to be addressed
> at earliest. Please take a look at
> https://build.gluster.org/job/centos7-regression/5568/consoleFull
> <https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
> investigate why the regression vote wasn't negative.*
>
> Pranith - I request you to investigate on the sdfs-sanity.t failure
> because of this patch.
>

sdfs is supposed to serialize entry fops by doing entrylk, but all the
locks are being done with all-zero lk-owner. In essence sdfs doesn't
achieve its goal of mutual exclusion when conflicting operations are
executed by same client because two locks on same entry with same
all-zero-owner will get locks. The patch which lead to sdfs-sanity.t
failure treats inodelk/entrylk/lk fops with all-zero lk-owner as Invalid
request to prevent these kinds of bugs. So it exposed the bug in sdfs. I
sent a fix for sdfs @ https://review.gluster.org/#/c/glusterfs/+/22582


> *@Maintainers - Please open up every regression link to see the actual
> status of the job and don't blindly trust on the +1 vote back at gerrit
> till this is addressed.*
>
> As per the policy, I'm going to revert this commit, watch out for the
> patch. I request this to be directly pushed with out waiting for the
> regression vote as we had done before in such breakage. Amar/Shyam - I
> believe you have this permission?
>

>
>> root at a5f81bd447c2:/home/glusterfs# prove -vf tests/basic/sdfs-sanity.t
>> tests/basic/sdfs-sanity.t ..
>> 1..7
>> ok 1, LINENUM:8
>> ok 2, LINENUM:9
>> ok 3, LINENUM:11
>> ok 4, LINENUM:12
>> ok 5, LINENUM:13
>> ok 6, LINENUM:16
>> mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid
>> argument
>> stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
>> tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
>> not ok 7 , LINENUM:20
>> FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
>> Failed 1/7 subtests
>>
>> Test Summary Report
>> -------------------
>> tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
>>   Failed test:  7
>> Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr
>> 0.67 csys =  1.27 CPU)
>> Result: FAIL
>>
>>
>>>
>>> Following patches will not be taken in if CentOS regression does not
>>> pass by tomorrow morning Eastern TZ,
>>> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
>>> inodelk/entrylk
>>>   https://review.gluster.org/c/glusterfs/+/22385
>>> (Aravinda) - geo-rep: IPv6 support
>>>   https://review.gluster.org/c/glusterfs/+/22488
>>> (Aravinda) - geo-rep: fix integer config validation
>>>   https://review.gluster.org/c/glusterfs/+/22489
>>>
>>> Tracker bug status:
>>> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
>>> RHGSWA.
>>>   All patches are merged, but none of the patches adds the "Fixes"
>>> keyword, assume this is an oversight and that the bug is fixed in this
>>> release.
>>>
>>> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
>>> the same brick, causing transport endpoint not connected
>>>   No work has occurred post logs upload to bug, restart of bircks and
>>> possibly glusterd is the existing workaround when the bug is hit. Moving
>>> this out of the tracker for 6.1.
>>>
>>> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
>>> replace-brick is executed
>>>   Very recent bug (15th April), does not seem to have any critical data
>>> corruption or service availability issues, planning on not waiting for
>>> the fix in 6.1
>>>
>>> - Shyam
>>> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>>> > Hi Mohit,
>>> >
>>> > https://review.gluster.org/22495 should get into 6.1 as it?s a
>>> > regression. Can you please attach the respective bug to the tracker
>>> Ravi
>>> > pointed out?
>>> >
>>> >
>>> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>>> > <mailto:ravishankar at redhat.com>> wrote:
>>> >
>>> >     Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394,
>>> in
>>> >     case anyone wants to add blocker bugs.
>>> >
>>> >
>>> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>>> >     > Hi,
>>> >     >
>>> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>>> >     >
>>> >     > Please ensure required patches are backported and also are
>>> passing
>>> >     > regressions and are appropriately reviewed for easy merging and
>>> >     tagging
>>> >     > on the date.
>>> >     >
>>> >     > Thanks,
>>> >     > Shyam
>>> >     > _______________________________________________
>>> >     > Gluster-devel mailing list
>>> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >     _______________________________________________
>>> >     Gluster-devel mailing list
>>> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> >
>>> > --
>>> > - Atin (atinm)
>>> >
>>> > _______________________________________________
>>> > Gluster-devel mailing list
>>> > Gluster-devel at gluster.org
>>> > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>> >
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>

-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190417/1ae848ba/attachment-0001.html>

From amukherj at redhat.com  Wed Apr 17 02:34:13 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Wed, 17 Apr 2019 08:04:13 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAOgeEnYd3P18Vt29WPCkZUwfhDJNqbq=OaVrAXzKbArGo6ECEQ@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
	<CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
	<CAOgeEnYd3P18Vt29WPCkZUwfhDJNqbq=OaVrAXzKbArGo6ECEQ@mail.gmail.com>
Message-ID: <CAGNCGH0BZf2wUgajN_qEMOgxHw51jWNEpyo=8eYsJ44+ZnJ1+g@mail.gmail.com>

On Wed, Apr 17, 2019 at 12:33 AM Pranith Kumar Karampuri <
pkarampu at redhat.com> wrote:

>
>
> On Tue, Apr 16, 2019 at 10:27 PM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>>
>>
>> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
>>> wrote:
>>>
>>>> Status: Tagging pending
>>>>
>>>> Waiting on patches:
>>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>>   https://review.gluster.org/c/glusterfs/+/22579
>>>
>>>
>>> The regression doesn't pass for the mainline patch. I believe master is
>>> broken now. With latest master sdfs-sanity.t always fail. We either need to
>>> fix it or mark it as bad test.
>>>
>>
>> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
>> Author: Pranith Kumar K <pkarampu at redhat.com>
>> Date:   Mon Apr 1 11:14:56 2019 +0530
>>
>>    features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>>
>>    Problem:
>>    Sometimes we find that developers forget to assign lk-owner for an
>>    inodelk/entrylk/lk before writing code to wind these fops. locks
>>    xlator at the moment allows this operation. This leads to multiple
>>    threads in the same client being able to get locks on the inode
>>    because lk-owner is same and transport is same. So isolation
>>    with locks can't be achieved.
>>
>>    Fix:
>>    Disallow locks with lk-owner zero.
>>
>>    fixes bz#1624701
>>    Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>>    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>>
>> With the above commit sdfs-sanity.t started failing. But when I looked at
>> the last regression vote at
>> https://build.gluster.org/job/centos7-regression/5568/consoleFull I saw
>> it voted back positive but the bell rang when I saw the overall regression
>> took less than 2 hours and when I opened the regression link I saw the test
>> actually failed but still this job voted back +1 at gerrit.
>>
>> *Deepshika* - *This is a bad CI bug we have now and have to be addressed
>> at earliest. Please take a look at
>> https://build.gluster.org/job/centos7-regression/5568/consoleFull
>> <https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
>> investigate why the regression vote wasn't negative.*
>>
>> Pranith - I request you to investigate on the sdfs-sanity.t failure
>> because of this patch.
>>
>
> sdfs is supposed to serialize entry fops by doing entrylk, but all the
> locks are being done with all-zero lk-owner. In essence sdfs doesn't
> achieve its goal of mutual exclusion when conflicting operations are
> executed by same client because two locks on same entry with same
> all-zero-owner will get locks. The patch which lead to sdfs-sanity.t
> failure treats inodelk/entrylk/lk fops with all-zero lk-owner as Invalid
> request to prevent these kinds of bugs. So it exposed the bug in sdfs. I
> sent a fix for sdfs @ https://review.gluster.org/#/c/glusterfs/+/22582
>

Since this patch hasn't passed the regression and now that I see
tests/bugs/replicate/bug-1386188-sbrain-fav-child.t hanging and timing out
in the latest nightly regression runs because of the above commit (tested
locally and confirm) I still request that we first revert this commit, get
master back to stable and then put back the required fixes.


>
>> *@Maintainers - Please open up every regression link to see the actual
>> status of the job and don't blindly trust on the +1 vote back at gerrit
>> till this is addressed.*
>>
>> As per the policy, I'm going to revert this commit, watch out for the
>> patch. I request this to be directly pushed with out waiting for the
>> regression vote as we had done before in such breakage. Amar/Shyam - I
>> believe you have this permission?
>>
>
>>
>>> root at a5f81bd447c2:/home/glusterfs# prove -vf tests/basic/sdfs-sanity.t
>>> tests/basic/sdfs-sanity.t ..
>>> 1..7
>>> ok 1, LINENUM:8
>>> ok 2, LINENUM:9
>>> ok 3, LINENUM:11
>>> ok 4, LINENUM:12
>>> ok 5, LINENUM:13
>>> ok 6, LINENUM:16
>>> mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid
>>> argument
>>> stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
>>> tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
>>> not ok 7 , LINENUM:20
>>> FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
>>> Failed 1/7 subtests
>>>
>>> Test Summary Report
>>> -------------------
>>> tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
>>>   Failed test:  7
>>> Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr
>>> 0.67 csys =  1.27 CPU)
>>> Result: FAIL
>>>
>>>
>>>>
>>>> Following patches will not be taken in if CentOS regression does not
>>>> pass by tomorrow morning Eastern TZ,
>>>> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
>>>> inodelk/entrylk
>>>>   https://review.gluster.org/c/glusterfs/+/22385
>>>> (Aravinda) - geo-rep: IPv6 support
>>>>   https://review.gluster.org/c/glusterfs/+/22488
>>>> (Aravinda) - geo-rep: fix integer config validation
>>>>   https://review.gluster.org/c/glusterfs/+/22489
>>>>
>>>> Tracker bug status:
>>>> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
>>>> RHGSWA.
>>>>   All patches are merged, but none of the patches adds the "Fixes"
>>>> keyword, assume this is an oversight and that the bug is fixed in this
>>>> release.
>>>>
>>>> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
>>>> the same brick, causing transport endpoint not connected
>>>>   No work has occurred post logs upload to bug, restart of bircks and
>>>> possibly glusterd is the existing workaround when the bug is hit. Moving
>>>> this out of the tracker for 6.1.
>>>>
>>>> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
>>>> replace-brick is executed
>>>>   Very recent bug (15th April), does not seem to have any critical data
>>>> corruption or service availability issues, planning on not waiting for
>>>> the fix in 6.1
>>>>
>>>> - Shyam
>>>> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>>>> > Hi Mohit,
>>>> >
>>>> > https://review.gluster.org/22495 should get into 6.1 as it?s a
>>>> > regression. Can you please attach the respective bug to the tracker
>>>> Ravi
>>>> > pointed out?
>>>> >
>>>> >
>>>> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>>>> > <mailto:ravishankar at redhat.com>> wrote:
>>>> >
>>>> >     Tracker bug is
>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in
>>>> >     case anyone wants to add blocker bugs.
>>>> >
>>>> >
>>>> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>>>> >     > Hi,
>>>> >     >
>>>> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>>>> >     >
>>>> >     > Please ensure required patches are backported and also are
>>>> passing
>>>> >     > regressions and are appropriately reviewed for easy merging and
>>>> >     tagging
>>>> >     > on the date.
>>>> >     >
>>>> >     > Thanks,
>>>> >     > Shyam
>>>> >     > _______________________________________________
>>>> >     > Gluster-devel mailing list
>>>> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>> >     _______________________________________________
>>>> >     Gluster-devel mailing list
>>>> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>> >
>>>> >
>>>> > --
>>>> > - Atin (atinm)
>>>> >
>>>> > _______________________________________________
>>>> > Gluster-devel mailing list
>>>> > Gluster-devel at gluster.org
>>>> > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>> >
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>
> --
> Pranith
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190417/bea0c8de/attachment.html>

From atumball at redhat.com  Wed Apr 17 03:23:26 2019
From: atumball at redhat.com (Amar Tumballi Suryanarayan)
Date: Wed, 17 Apr 2019 08:53:26 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAGNCGH0BZf2wUgajN_qEMOgxHw51jWNEpyo=8eYsJ44+ZnJ1+g@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
	<CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
	<CAOgeEnYd3P18Vt29WPCkZUwfhDJNqbq=OaVrAXzKbArGo6ECEQ@mail.gmail.com>
	<CAGNCGH0BZf2wUgajN_qEMOgxHw51jWNEpyo=8eYsJ44+ZnJ1+g@mail.gmail.com>
Message-ID: <CAHxyDdMb6Hn1as1XSRP=hSBC_Bb9znQG=z0W6WgRzYYGD91SDg@mail.gmail.com>

My take is, lets disable sdfs for 6.1 (we also have issues with its
performance anyways). We will fix it properly by 6.2 or 7.0. Continue with
marking sdfs-sanity.t tests as bad in that case.

-Amar

On Wed, Apr 17, 2019 at 8:04 AM Atin Mukherjee <amukherj at redhat.com> wrote:

>
>
> On Wed, Apr 17, 2019 at 12:33 AM Pranith Kumar Karampuri <
> pkarampu at redhat.com> wrote:
>
>>
>>
>> On Tue, Apr 16, 2019 at 10:27 PM Atin Mukherjee <amukherj at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
>>>> wrote:
>>>>
>>>>> Status: Tagging pending
>>>>>
>>>>> Waiting on patches:
>>>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>>>   https://review.gluster.org/c/glusterfs/+/22579
>>>>
>>>>
>>>> The regression doesn't pass for the mainline patch. I believe master is
>>>> broken now. With latest master sdfs-sanity.t always fail. We either need to
>>>> fix it or mark it as bad test.
>>>>
>>>
>>> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
>>> Author: Pranith Kumar K <pkarampu at redhat.com>
>>> Date:   Mon Apr 1 11:14:56 2019 +0530
>>>
>>>    features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>>>
>>>    Problem:
>>>    Sometimes we find that developers forget to assign lk-owner for an
>>>    inodelk/entrylk/lk before writing code to wind these fops. locks
>>>    xlator at the moment allows this operation. This leads to multiple
>>>    threads in the same client being able to get locks on the inode
>>>    because lk-owner is same and transport is same. So isolation
>>>    with locks can't be achieved.
>>>
>>>    Fix:
>>>    Disallow locks with lk-owner zero.
>>>
>>>    fixes bz#1624701
>>>    Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>>>    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>>>
>>> With the above commit sdfs-sanity.t started failing. But when I looked
>>> at the last regression vote at
>>> https://build.gluster.org/job/centos7-regression/5568/consoleFull I saw
>>> it voted back positive but the bell rang when I saw the overall regression
>>> took less than 2 hours and when I opened the regression link I saw the test
>>> actually failed but still this job voted back +1 at gerrit.
>>>
>>> *Deepshika* - *This is a bad CI bug we have now and have to be
>>> addressed at earliest. Please take a look at
>>> https://build.gluster.org/job/centos7-regression/5568/consoleFull
>>> <https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
>>> investigate why the regression vote wasn't negative.*
>>>
>>> Pranith - I request you to investigate on the sdfs-sanity.t failure
>>> because of this patch.
>>>
>>
>> sdfs is supposed to serialize entry fops by doing entrylk, but all the
>> locks are being done with all-zero lk-owner. In essence sdfs doesn't
>> achieve its goal of mutual exclusion when conflicting operations are
>> executed by same client because two locks on same entry with same
>> all-zero-owner will get locks. The patch which lead to sdfs-sanity.t
>> failure treats inodelk/entrylk/lk fops with all-zero lk-owner as Invalid
>> request to prevent these kinds of bugs. So it exposed the bug in sdfs. I
>> sent a fix for sdfs @ https://review.gluster.org/#/c/glusterfs/+/22582
>>
>
> Since this patch hasn't passed the regression and now that I see
> tests/bugs/replicate/bug-1386188-sbrain-fav-child.t hanging and timing out
> in the latest nightly regression runs because of the above commit (tested
> locally and confirm) I still request that we first revert this commit, get
> master back to stable and then put back the required fixes.
>
>
>>
>>> *@Maintainers - Please open up every regression link to see the actual
>>> status of the job and don't blindly trust on the +1 vote back at gerrit
>>> till this is addressed.*
>>>
>>> As per the policy, I'm going to revert this commit, watch out for the
>>> patch. I request this to be directly pushed with out waiting for the
>>> regression vote as we had done before in such breakage. Amar/Shyam - I
>>> believe you have this permission?
>>>
>>
>>>
>>>> root at a5f81bd447c2:/home/glusterfs# prove -vf tests/basic/sdfs-sanity.t
>>>> tests/basic/sdfs-sanity.t ..
>>>> 1..7
>>>> ok 1, LINENUM:8
>>>> ok 2, LINENUM:9
>>>> ok 3, LINENUM:11
>>>> ok 4, LINENUM:12
>>>> ok 5, LINENUM:13
>>>> ok 6, LINENUM:16
>>>> mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid
>>>> argument
>>>> stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
>>>> tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
>>>> not ok 7 , LINENUM:20
>>>> FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
>>>> Failed 1/7 subtests
>>>>
>>>> Test Summary Report
>>>> -------------------
>>>> tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
>>>>   Failed test:  7
>>>> Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr
>>>> 0.67 csys =  1.27 CPU)
>>>> Result: FAIL
>>>>
>>>>
>>>>>
>>>>> Following patches will not be taken in if CentOS regression does not
>>>>> pass by tomorrow morning Eastern TZ,
>>>>> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
>>>>> inodelk/entrylk
>>>>>   https://review.gluster.org/c/glusterfs/+/22385
>>>>> (Aravinda) - geo-rep: IPv6 support
>>>>>   https://review.gluster.org/c/glusterfs/+/22488
>>>>> (Aravinda) - geo-rep: fix integer config validation
>>>>>   https://review.gluster.org/c/glusterfs/+/22489
>>>>>
>>>>> Tracker bug status:
>>>>> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
>>>>> RHGSWA.
>>>>>   All patches are merged, but none of the patches adds the "Fixes"
>>>>> keyword, assume this is an oversight and that the bug is fixed in this
>>>>> release.
>>>>>
>>>>> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
>>>>> the same brick, causing transport endpoint not connected
>>>>>   No work has occurred post logs upload to bug, restart of bircks and
>>>>> possibly glusterd is the existing workaround when the bug is hit.
>>>>> Moving
>>>>> this out of the tracker for 6.1.
>>>>>
>>>>> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
>>>>> replace-brick is executed
>>>>>   Very recent bug (15th April), does not seem to have any critical data
>>>>> corruption or service availability issues, planning on not waiting for
>>>>> the fix in 6.1
>>>>>
>>>>> - Shyam
>>>>> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>>>>> > Hi Mohit,
>>>>> >
>>>>> > https://review.gluster.org/22495 should get into 6.1 as it?s a
>>>>> > regression. Can you please attach the respective bug to the tracker
>>>>> Ravi
>>>>> > pointed out?
>>>>> >
>>>>> >
>>>>> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>>>>> > <mailto:ravishankar at redhat.com>> wrote:
>>>>> >
>>>>> >     Tracker bug is
>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in
>>>>> >     case anyone wants to add blocker bugs.
>>>>> >
>>>>> >
>>>>> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>>>>> >     > Hi,
>>>>> >     >
>>>>> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>>>>> >     >
>>>>> >     > Please ensure required patches are backported and also are
>>>>> passing
>>>>> >     > regressions and are appropriately reviewed for easy merging and
>>>>> >     tagging
>>>>> >     > on the date.
>>>>> >     >
>>>>> >     > Thanks,
>>>>> >     > Shyam
>>>>> >     > _______________________________________________
>>>>> >     > Gluster-devel mailing list
>>>>> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>>> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>> >     _______________________________________________
>>>>> >     Gluster-devel mailing list
>>>>> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>>> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>> >
>>>>> >
>>>>> > --
>>>>> > - Atin (atinm)
>>>>> >
>>>>> > _______________________________________________
>>>>> > Gluster-devel mailing list
>>>>> > Gluster-devel at gluster.org
>>>>> > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>> >
>>>>> _______________________________________________
>>>>> Gluster-devel mailing list
>>>>> Gluster-devel at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>
>>>>
>>
>> --
>> Pranith
>>
>

-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190417/3da9a5c5/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Wed Apr 17 05:22:59 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Wed, 17 Apr 2019 05:22:59 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable ssl
Message-ID: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>

Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option.
If I monitor glusterfsd process RSS with command:  pidstat -r -p <glusterfsd pid> 1
And at the same tme, execute command: gluster v heal <volume> info, i find that the RSS keep increasing until system Out of memory.
I have tried even with latest glusterfs5.5 version, this issue still exists,
Have you any idea about this issue?

The config in my env:
[root at mn-0:/var/lib/glusterd]
# gluster v info ccs
Volume Name: ccs
Type: Replicate
Volume ID: b565c958-2fef-4a39-811b-adce67aab20b
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: mn-0.local:/mnt/bricks/ccs/brick
Brick2: mn-1.local:/mnt/bricks/ccs/brick
Brick3: dbm-0.local:/mnt/bricks/ccs/brick
Options Reconfigured:
nfs.disable: on
transport.address-family: inet
cluster.server-quorum-type: none
cluster.quorum-reads: no
ssl.private-key: /var/opt/nokia/certs/glusterfs/glusterfs.key
cluster.favorite-child-policy: mtime
cluster.consistent-metadata: on
network.ping-timeout: 42
cluster.quorum-type: auto
server.allow-insecure: on
server.ssl: on
ssl.own-cert: /var/opt/nokia/certs/glusterfs/glusterfs.pem
ssl.ca-list: /var/opt/nokia/certs/glusterfs/glusterfs.ca
client.ssl: on
performance.client-io-threads: off
cluster.server-quorum-ratio: 51%
[root at mn-0:/var/lib/glusterd]
#
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190417/3daa916d/attachment.html>

From pkarampu at redhat.com  Wed Apr 17 05:25:11 2019
From: pkarampu at redhat.com (Pranith Kumar Karampuri)
Date: Wed, 17 Apr 2019 10:55:11 +0530
Subject: [Gluster-devel] Release 6.1: Expected tagging on April 10th
In-Reply-To: <CAHxyDdMb6Hn1as1XSRP=hSBC_Bb9znQG=z0W6WgRzYYGD91SDg@mail.gmail.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
	<CAGNCGH2UKGupfz8ONcb4mtG-vKUAoRpu2rutEV=-iz2UO2jXzQ@mail.gmail.com>
	<CAGNCGH2ttCDw9MiTyGKww7kBvt45RBsB3_W2GXs+OcHdWV5LdA@mail.gmail.com>
	<CAOgeEnYd3P18Vt29WPCkZUwfhDJNqbq=OaVrAXzKbArGo6ECEQ@mail.gmail.com>
	<CAGNCGH0BZf2wUgajN_qEMOgxHw51jWNEpyo=8eYsJ44+ZnJ1+g@mail.gmail.com>
	<CAHxyDdMb6Hn1as1XSRP=hSBC_Bb9znQG=z0W6WgRzYYGD91SDg@mail.gmail.com>
Message-ID: <CAOgeEnZRaLEZx_SYkDTCA4T8dyP2wBXh_V3P-pw481U_a98YEA@mail.gmail.com>

On Wed, Apr 17, 2019 at 8:53 AM Amar Tumballi Suryanarayan <
atumball at redhat.com> wrote:

> My take is, lets disable sdfs for 6.1 (we also have issues with its
> performance anyways). We will fix it properly by 6.2 or 7.0. Continue with
> marking sdfs-sanity.t tests as bad in that case.
>

It is better to revert the patch like Atin mentioned. The patch that was
merged was intended to find the existing bugs with lk-owner before getting
merged. We thought there were no bugs when regression passed. But that is
not the case. So better to revert, fix other bugs found by this patch and
then get it merged again.


>
> -Amar
>
> On Wed, Apr 17, 2019 at 8:04 AM Atin Mukherjee <amukherj at redhat.com>
> wrote:
>
>>
>>
>> On Wed, Apr 17, 2019 at 12:33 AM Pranith Kumar Karampuri <
>> pkarampu at redhat.com> wrote:
>>
>>>
>>>
>>> On Tue, Apr 16, 2019 at 10:27 PM Atin Mukherjee <amukherj at redhat.com>
>>> wrote:
>>>
>>>>
>>>>
>>>> On Tue, Apr 16, 2019 at 9:19 PM Atin Mukherjee <amukherj at redhat.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 16, 2019 at 7:24 PM Shyam Ranganathan <srangana at redhat.com>
>>>>> wrote:
>>>>>
>>>>>> Status: Tagging pending
>>>>>>
>>>>>> Waiting on patches:
>>>>>> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>>>>>>   https://review.gluster.org/c/glusterfs/+/22579
>>>>>
>>>>>
>>>>> The regression doesn't pass for the mainline patch. I believe master
>>>>> is broken now. With latest master sdfs-sanity.t always fail. We either need
>>>>> to fix it or mark it as bad test.
>>>>>
>>>>
>>>> commit 3883887427a7f2dc458a9773e05f7c8ce8e62301 (HEAD)
>>>> Author: Pranith Kumar K <pkarampu at redhat.com>
>>>> Date:   Mon Apr 1 11:14:56 2019 +0530
>>>>
>>>>    features/locks: error-out {inode,entry}lk fops with all-zero lk-owner
>>>>
>>>>    Problem:
>>>>    Sometimes we find that developers forget to assign lk-owner for an
>>>>    inodelk/entrylk/lk before writing code to wind these fops. locks
>>>>    xlator at the moment allows this operation. This leads to multiple
>>>>    threads in the same client being able to get locks on the inode
>>>>    because lk-owner is same and transport is same. So isolation
>>>>    with locks can't be achieved.
>>>>
>>>>    Fix:
>>>>    Disallow locks with lk-owner zero.
>>>>
>>>>    fixes bz#1624701
>>>>    Change-Id: I1c816280cffd150ebb392e3dcd4d21007cdd767f
>>>>    Signed-off-by: Pranith Kumar K <pkarampu at redhat.com>
>>>>
>>>> With the above commit sdfs-sanity.t started failing. But when I looked
>>>> at the last regression vote at
>>>> https://build.gluster.org/job/centos7-regression/5568/consoleFull I
>>>> saw it voted back positive but the bell rang when I saw the overall
>>>> regression took less than 2 hours and when I opened the regression link I
>>>> saw the test actually failed but still this job voted back +1 at gerrit.
>>>>
>>>> *Deepshika* - *This is a bad CI bug we have now and have to be
>>>> addressed at earliest. Please take a look at
>>>> https://build.gluster.org/job/centos7-regression/5568/consoleFull
>>>> <https://build.gluster.org/job/centos7-regression/5568/consoleFull> and
>>>> investigate why the regression vote wasn't negative.*
>>>>
>>>> Pranith - I request you to investigate on the sdfs-sanity.t failure
>>>> because of this patch.
>>>>
>>>
>>> sdfs is supposed to serialize entry fops by doing entrylk, but all the
>>> locks are being done with all-zero lk-owner. In essence sdfs doesn't
>>> achieve its goal of mutual exclusion when conflicting operations are
>>> executed by same client because two locks on same entry with same
>>> all-zero-owner will get locks. The patch which lead to sdfs-sanity.t
>>> failure treats inodelk/entrylk/lk fops with all-zero lk-owner as Invalid
>>> request to prevent these kinds of bugs. So it exposed the bug in sdfs. I
>>> sent a fix for sdfs @ https://review.gluster.org/#/c/glusterfs/+/22582
>>>
>>
>> Since this patch hasn't passed the regression and now that I see
>> tests/bugs/replicate/bug-1386188-sbrain-fav-child.t hanging and timing out
>> in the latest nightly regression runs because of the above commit (tested
>> locally and confirm) I still request that we first revert this commit, get
>> master back to stable and then put back the required fixes.
>>
>>
>>>
>>>> *@Maintainers - Please open up every regression link to see the actual
>>>> status of the job and don't blindly trust on the +1 vote back at gerrit
>>>> till this is addressed.*
>>>>
>>>> As per the policy, I'm going to revert this commit, watch out for the
>>>> patch. I request this to be directly pushed with out waiting for the
>>>> regression vote as we had done before in such breakage. Amar/Shyam - I
>>>> believe you have this permission?
>>>>
>>>
>>>>
>>>>> root at a5f81bd447c2:/home/glusterfs# prove -vf
>>>>> tests/basic/sdfs-sanity.t
>>>>> tests/basic/sdfs-sanity.t ..
>>>>> 1..7
>>>>> ok 1, LINENUM:8
>>>>> ok 2, LINENUM:9
>>>>> ok 3, LINENUM:11
>>>>> ok 4, LINENUM:12
>>>>> ok 5, LINENUM:13
>>>>> ok 6, LINENUM:16
>>>>> mkdir: cannot create directory ?/mnt/glusterfs/1/coverage?: Invalid
>>>>> argument
>>>>> stat: cannot stat '/mnt/glusterfs/1/coverage/dir': Invalid argument
>>>>> tests/basic/rpc-coverage.sh: line 61: test: ==: unary operator expected
>>>>> not ok 7 , LINENUM:20
>>>>> FAILED COMMAND: tests/basic/rpc-coverage.sh /mnt/glusterfs/1
>>>>> Failed 1/7 subtests
>>>>>
>>>>> Test Summary Report
>>>>> -------------------
>>>>> tests/basic/sdfs-sanity.t (Wstat: 0 Tests: 7 Failed: 1)
>>>>>   Failed test:  7
>>>>> Files=1, Tests=7, 14 wallclock secs ( 0.02 usr  0.00 sys +  0.58 cusr
>>>>> 0.67 csys =  1.27 CPU)
>>>>> Result: FAIL
>>>>>
>>>>>
>>>>>>
>>>>>> Following patches will not be taken in if CentOS regression does not
>>>>>> pass by tomorrow morning Eastern TZ,
>>>>>> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
>>>>>> inodelk/entrylk
>>>>>>   https://review.gluster.org/c/glusterfs/+/22385
>>>>>> (Aravinda) - geo-rep: IPv6 support
>>>>>>   https://review.gluster.org/c/glusterfs/+/22488
>>>>>> (Aravinda) - geo-rep: fix integer config validation
>>>>>>   https://review.gluster.org/c/glusterfs/+/22489
>>>>>>
>>>>>> Tracker bug status:
>>>>>> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
>>>>>> RHGSWA.
>>>>>>   All patches are merged, but none of the patches adds the "Fixes"
>>>>>> keyword, assume this is an oversight and that the bug is fixed in this
>>>>>> release.
>>>>>>
>>>>>> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched
>>>>>> for
>>>>>> the same brick, causing transport endpoint not connected
>>>>>>   No work has occurred post logs upload to bug, restart of bircks and
>>>>>> possibly glusterd is the existing workaround when the bug is hit.
>>>>>> Moving
>>>>>> this out of the tracker for 6.1.
>>>>>>
>>>>>> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
>>>>>> replace-brick is executed
>>>>>>   Very recent bug (15th April), does not seem to have any critical
>>>>>> data
>>>>>> corruption or service availability issues, planning on not waiting for
>>>>>> the fix in 6.1
>>>>>>
>>>>>> - Shyam
>>>>>> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>>>>>> > Hi Mohit,
>>>>>> >
>>>>>> > https://review.gluster.org/22495 should get into 6.1 as it?s a
>>>>>> > regression. Can you please attach the respective bug to the tracker
>>>>>> Ravi
>>>>>> > pointed out?
>>>>>> >
>>>>>> >
>>>>>> > On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>>>>>> > <mailto:ravishankar at redhat.com>> wrote:
>>>>>> >
>>>>>> >     Tracker bug is
>>>>>> https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in
>>>>>> >     case anyone wants to add blocker bugs.
>>>>>> >
>>>>>> >
>>>>>> >     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>>>>>> >     > Hi,
>>>>>> >     >
>>>>>> >     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>>>>>> >     >
>>>>>> >     > Please ensure required patches are backported and also are
>>>>>> passing
>>>>>> >     > regressions and are appropriately reviewed for easy merging
>>>>>> and
>>>>>> >     tagging
>>>>>> >     > on the date.
>>>>>> >     >
>>>>>> >     > Thanks,
>>>>>> >     > Shyam
>>>>>> >     > _______________________________________________
>>>>>> >     > Gluster-devel mailing list
>>>>>> >     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>>>> >     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>> >     _______________________________________________
>>>>>> >     Gluster-devel mailing list
>>>>>> >     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>>>>> >     https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > - Atin (atinm)
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Gluster-devel mailing list
>>>>>> > Gluster-devel at gluster.org
>>>>>> > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>> >
>>>>>> _______________________________________________
>>>>>> Gluster-devel mailing list
>>>>>> Gluster-devel at gluster.org
>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>>>
>>>>>
>>>
>>> --
>>> Pranith
>>>
>>
>
> --
> Amar Tumballi (amarts)
>


-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190417/4a4f48e4/attachment-0001.html>

From moagrawa at redhat.com  Wed Apr 17 06:06:09 2019
From: moagrawa at redhat.com (Mohit Agrawal)
Date: Wed, 17 Apr 2019 11:36:09 +0530
Subject: [Gluster-devel] Regards glusterd.service is not started
 automatically after reboot the node
Message-ID: <CALBz39yui0OtW3SN56g0gZVWnR4TSXAu-Cs+Nj0VXuA7gYDn7A@mail.gmail.com>

Hi,

  We are facing an issue after install glusterfs-6 rpms. After reboot, the
node glusterd.service is not started automatically because
   glusterd.service is not enabled by the installation script.I am not able
to find the patch from that we deleted command to enable service
   from glusterfs.spec.in.
  I have posted a patch(https://review.gluster.org/#/c/glusterfs/+/22584/)
to resolve the same.

Thanks,
Mohit Agrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190417/395e95d6/attachment.html>

From ndevos at redhat.com  Wed Apr 17 09:58:15 2019
From: ndevos at redhat.com (Niels de Vos)
Date: Wed, 17 Apr 2019 11:58:15 +0200
Subject: [Gluster-devel] Regards glusterd.service is not started
 automatically after reboot the node
In-Reply-To: <CALBz39yui0OtW3SN56g0gZVWnR4TSXAu-Cs+Nj0VXuA7gYDn7A@mail.gmail.com>
References: <CALBz39yui0OtW3SN56g0gZVWnR4TSXAu-Cs+Nj0VXuA7gYDn7A@mail.gmail.com>
Message-ID: <20190417095815.GB19605@ndevos-x270>

On Wed, Apr 17, 2019 at 11:36:09AM +0530, Mohit Agrawal wrote:
> Hi,
> 
>   We are facing an issue after install glusterfs-6 rpms. After reboot, the
> node glusterd.service is not started automatically because
>    glusterd.service is not enabled by the installation script.I am not able
> to find the patch from that we deleted command to enable service
>    from glusterfs.spec.in.
>   I have posted a patch(https://review.gluster.org/#/c/glusterfs/+/22584/)
> to resolve the same.

This is not a bug, it is expected behaviour.

Services are not allowed to get automatically enabled through RPM
scriptlets. Distributions that want to enable glusterd by default should
provide a systemd preset as explained in
https://www.freedesktop.org/wiki/Software/systemd/Preset/ . This is
something you could contribute to
https://github.com/CentOS-Storage-SIG/centos-release-gluster/tree/6

HTH,
Niels

From srangana at redhat.com  Wed Apr 17 18:31:55 2019
From: srangana at redhat.com (Shyam Ranganathan)
Date: Wed, 17 Apr 2019 14:31:55 -0400
Subject: [Gluster-devel] Release 6.1: Tagged!
In-Reply-To: <6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
References: <d3633382-29f1-0b51-a982-57d3f21d85ba@redhat.com>
	<e314ddb2-d0b0-4b73-a942-25a6d04a1b04@redhat.com>
	<CAGNCGH25L70sS0NRGFAQ3EN3g6nrme3fewZVz3zw3P6m0J7NBg@mail.gmail.com>
	<6866f19a-18f5-38cb-ab03-13de124f5ac5@redhat.com>
Message-ID: <f4dea969-425c-9d82-5b45-933277b7dedb@redhat.com>

Is now tagged and being packaged. If anyone gets a chance, please test
the packages from CentOS SIG, as I am unavailable for the next 4 days.

Thanks,
Shyam
On 4/16/19 9:53 AM, Shyam Ranganathan wrote:
> Status: Tagging pending
> 
> Waiting on patches:
> (Kotresh/Atin) - glusterd: fix loading ctime in client graph logic
>   https://review.gluster.org/c/glusterfs/+/22579
> 
> Following patches will not be taken in if CentOS regression does not
> pass by tomorrow morning Eastern TZ,
> (Pranith/KingLongMee) - cluster-syncop: avoid duplicate unlock of
> inodelk/entrylk
>   https://review.gluster.org/c/glusterfs/+/22385
> (Aravinda) - geo-rep: IPv6 support
>   https://review.gluster.org/c/glusterfs/+/22488
> (Aravinda) - geo-rep: fix integer config validation
>   https://review.gluster.org/c/glusterfs/+/22489
> 
> Tracker bug status:
> (Ravi) - Bug 1693155 - Excessive AFR messages from gluster showing in
> RHGSWA.
>   All patches are merged, but none of the patches adds the "Fixes"
> keyword, assume this is an oversight and that the bug is fixed in this
> release.
> 
> (Atin) - Bug 1698131 - multiple glusterfsd processes being launched for
> the same brick, causing transport endpoint not connected
>   No work has occurred post logs upload to bug, restart of bircks and
> possibly glusterd is the existing workaround when the bug is hit. Moving
> this out of the tracker for 6.1.
> 
> (Xavi) - Bug 1699917 - I/O error on writes to a disperse volume when
> replace-brick is executed
>   Very recent bug (15th April), does not seem to have any critical data
> corruption or service availability issues, planning on not waiting for
> the fix in 6.1
> 
> - Shyam
> On 4/6/19 4:38 AM, Atin Mukherjee wrote:
>> Hi Mohit,
>>
>> https://review.gluster.org/22495 should get into 6.1 as it?s a
>> regression. Can you please attach the respective bug to the tracker Ravi
>> pointed out?
>>
>>
>> On Sat, 6 Apr 2019 at 12:00, Ravishankar N <ravishankar at redhat.com
>> <mailto:ravishankar at redhat.com>> wrote:
>>
>>     Tracker bug is https://bugzilla.redhat.com/show_bug.cgi?id=1692394, in
>>     case anyone wants to add blocker bugs.
>>
>>
>>     On 05/04/19 8:03 PM, Shyam Ranganathan wrote:
>>     > Hi,
>>     >
>>     > Expected tagging date for release-6.1 is on April, 10th, 2019.
>>     >
>>     > Please ensure required patches are backported and also are passing
>>     > regressions and are appropriately reviewed for easy merging and
>>     tagging
>>     > on the date.
>>     >
>>     > Thanks,
>>     > Shyam
>>     > _______________________________________________
>>     > Gluster-devel mailing list
>>     > Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>     > https://lists.gluster.org/mailman/listinfo/gluster-devel
>>     _______________________________________________
>>     Gluster-devel mailing list
>>     Gluster-devel at gluster.org <mailto:Gluster-devel at gluster.org>
>>     https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>>
>> -- 
>> - Atin (atinm)
>>
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
> 

From amukherj at redhat.com  Thu Apr 18 05:19:01 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Thu, 18 Apr 2019 10:49:01 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
Message-ID: <CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>

On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi,
>
> In my recent test, I found that there are very severe glusterfsd memory
> leak when enable socket ssl option
>

What gluster version are you testing? Would you be able to continue your
investigation and share the root cause?

-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190418/4e5c21be/attachment.html>

From cynthia.zhou at nokia-sbell.com  Thu Apr 18 09:25:00 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Thu, 18 Apr 2019 09:25:00 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
Message-ID: <f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190418/83d75b6a/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Thu Apr 18 09:31:11 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Thu, 18 Apr 2019 09:31:11 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
Message-ID: <6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com>; gluster-devel at gluster.org
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190418/cfd50a08/attachment.html>

From jenkins at build.gluster.org  Mon Apr 22 01:45:02 2019
From: jenkins at build.gluster.org (jenkins at build.gluster.org)
Date: Mon, 22 Apr 2019 01:45:02 +0000 (UTC)
Subject: [Gluster-devel] Weekly Untriaged Bugs
Message-ID: <307671157.25.1555897502768.JavaMail.jenkins@jenkins-el7.rht.gluster.org>

[...truncated 6 lines...]
https://bugzilla.redhat.com/1699023 / core: Brick is not able to detach successfully in brick_mux environment
https://bugzilla.redhat.com/1695416 / core: client log flooding with intentional socket shutdown message when a brick is down
https://bugzilla.redhat.com/1695480 / core: Global Thread Pool
https://bugzilla.redhat.com/1694943 / core: parallel-readdir slows down directory listing
https://bugzilla.redhat.com/1700295 / core: The data couldn't be flushed immediately even with O_SYNC in glfs_create or with glfs_fsync/glfs_fdatasync after glfs_write.
https://bugzilla.redhat.com/1698861 / disperse: Renaming a directory when 2 bricks of multiple disperse subvols are down leaves both old and new dirs on the bricks.
https://bugzilla.redhat.com/1697293 / distribute: DHT: print hash and layout values in hexadecimal format in the logs
https://bugzilla.redhat.com/1701039 / distribute: gluster replica 3 arbiter Unfortunately data not distributed equally
https://bugzilla.redhat.com/1697971 / fuse: Segfault in FUSE process, potential use after free
https://bugzilla.redhat.com/1694139 / glusterd: Error waiting for job 'heketi-storage-copy-job' to complete on one-node k3s deployment.
https://bugzilla.redhat.com/1695099 / glusterd: The number of glusterfs processes keeps increasing, using all available resources
https://bugzilla.redhat.com/1692349 / project-infrastructure: gluster-csi-containers job is failing
https://bugzilla.redhat.com/1698716 / project-infrastructure: Regression job did not vote for https://review.gluster.org/#/c/glusterfs/+/22366/
https://bugzilla.redhat.com/1698694 / project-infrastructure: regression job isn't voting back to gerrit
https://bugzilla.redhat.com/1699712 / project-infrastructure: regression job is voting Success even in case of failure
https://bugzilla.redhat.com/1693385 / project-infrastructure: request to change the version of fedora in fedora-smoke-job
https://bugzilla.redhat.com/1695484 / project-infrastructure: smoke fails with "Build root is locked by another process"
https://bugzilla.redhat.com/1693184 / replicate: A brick process(glusterfsd) died with 'memory violation'
https://bugzilla.redhat.com/1698566 / selfheal: shd crashed while executing ./tests/bugs/core/bug-1432542-mpx-restart-crash.t in CI
https://bugzilla.redhat.com/1699309 / snapshot: Gluster snapshot fails with systemd autmounted bricks
https://bugzilla.redhat.com/1696633 / tests: GlusterFs v4.1.5 Tests from /tests/bugs/ module failing on Intel
https://bugzilla.redhat.com/1697812 / website: mention a pointer to all the mailing lists available under glusterfs project(https://www.gluster.org/community/)
[...truncated 2 lines...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 3020 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/20dfdb93/attachment-0001.obj>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 01:48:16 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 01:48:16 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
Message-ID: <b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>


Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com>; 'gluster-devel at gluster.org' <gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/e06a2e69/attachment-0001.html>

From rgowdapp at redhat.com  Mon Apr 22 02:08:37 2019
From: rgowdapp at redhat.com (Raghavendra Gowdappa)
Date: Mon, 22 Apr 2019 07:38:37 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
Message-ID: <CAFkORY8C=3rS5Pph_SMQS0+0NccGtZjrJMFoy+wW8vzFg92+Dw@mail.gmail.com>

On Mon, Apr 22, 2019 at 7:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

>
>
> Hi,
>
> From my code study it seems priv->ssl_ssl is not properly released, I made
> a patch and the glusterfsd memory leak is alleviated with my patch,
>

This is a legitimate leak. Can you post a patch to gerrit?

but some otherwhere is still leaking, I have no clue about the other leak
> points.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
>
>    memset(&priv->incoming, 0, sizeof(priv->incoming));
>
>    event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
>
> -
>
> +  if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>    priv->sock = -1;
>
>    priv->idx = -1;
>
>    priv->connected = -1;
>
> @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
>
>      pthread_mutex_destroy(&priv->out_lock);
>
>      pthread_mutex_destroy(&priv->cond_lock);
>
>      pthread_cond_destroy(&priv->cond);
>
> +     if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>      if (priv->ssl_private_key) {
>
>        GF_FREE(priv->ssl_private_key);
>
>      }
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:31 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>; '
> gluster-devel at gluster.org' <gluster-devel at gluster.org>
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> We scan it use memory-leak tool, there are following prints. We doubt some
> open ssl lib malloc is is not properly freed by glusterfs code.
>
> er+0x2af [*libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda* [*libpthread-2.27.so
> <http://libpthread-2.27.so>*]'
>         *13580* bytes in 175 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p*]'
>         *232904* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
> [15:41:56] Top 10 stacks with outstanding allocations:
>         *8792* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *9408* bytes in 42 allocations from stack
>                 b'CRYPTO_realloc+0x4d [*libcrypto.so.1.0.2p*]'
>         *9723* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *10696* bytes in 21 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11319* bytes in 602 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11431* bytes in 518 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11704* bytes in 371 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>
>
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:25 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after
> enable tls ssl socket, when execute gluster v heal <vol-name> info, will
> trigger glfshel to connect glusterfsd process, and cause glusterfsd process
> memory leak. Could you please try in your env?
>
>
>
> cynthia
>
>
>
>
>
> *From:* Atin Mukherjee <amukherj at redhat.com>
> *Sent:* Thursday, April 18, 2019 1:19 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
>
>
>
>
> On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> In my recent test, I found that there are very severe glusterfsd memory
> leak when enable socket ssl option
>
>
>
> What gluster version are you testing? Would you be able to continue your
> investigation and share the root cause?
>
>
>
> --
>
> - Atin (atinm)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/d0f2b5ff/attachment.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 02:14:00 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 02:14:00 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAFkORY8C=3rS5Pph_SMQS0+0NccGtZjrJMFoy+wW8vzFg92+Dw@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAFkORY8C=3rS5Pph_SMQS0+0NccGtZjrJMFoy+wW8vzFg92+Dw@mail.gmail.com>
Message-ID: <88c7d4971cdd4685a2b66f582ec1d9f6@nokia-sbell.com>

Ok, I will post it later.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com>
Sent: Monday, April 22, 2019 10:09 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Mon, Apr 22, 2019 at 7:18 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:

Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch,

This is a legitimate leak. Can you post a patch to gerrit?

but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; 'gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>' <gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/186fabe5/attachment-0001.html>

From mchangir at redhat.com  Mon Apr 22 02:20:34 2019
From: mchangir at redhat.com (Milind Changire)
Date: Mon, 22 Apr 2019 07:50:34 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
Message-ID: <CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>

After patch 22334 <https://review.gluster.org/c/glusterfs/+/22334>, the
priv->ssl_ctx is now maintained per socket connection and is no longer
shared.
So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx
to NULL.

There might be some strings that are duplicated (gf_strdup()) via the
socket_init() code path. Please take a look at those as well.

Sorry about that. I missed it.


On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

>
>
> Hi,
>
> From my code study it seems priv->ssl_ssl is not properly released, I made
> a patch and the glusterfsd memory leak is alleviated with my patch, but
> some otherwhere is still leaking, I have no clue about the other leak
> points.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
>
>    memset(&priv->incoming, 0, sizeof(priv->incoming));
>
>    event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
>
> -
>
> +  if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>    priv->sock = -1;
>
>    priv->idx = -1;
>
>    priv->connected = -1;
>
> @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
>
>      pthread_mutex_destroy(&priv->out_lock);
>
>      pthread_mutex_destroy(&priv->cond_lock);
>
>      pthread_cond_destroy(&priv->cond);
>
> +     if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>      if (priv->ssl_private_key) {
>
>        GF_FREE(priv->ssl_private_key);
>
>      }
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:31 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>; '
> gluster-devel at gluster.org' <gluster-devel at gluster.org>
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> We scan it use memory-leak tool, there are following prints. We doubt some
> open ssl lib malloc is is not properly freed by glusterfs code.
>
> er+0x2af [*libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda* [*libpthread-2.27.so
> <http://libpthread-2.27.so>*]'
>         *13580* bytes in 175 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p*]'
>         *232904* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
> [15:41:56] Top 10 stacks with outstanding allocations:
>         *8792* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *9408* bytes in 42 allocations from stack
>                 b'CRYPTO_realloc+0x4d [*libcrypto.so.1.0.2p*]'
>         *9723* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *10696* bytes in 21 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11319* bytes in 602 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11431* bytes in 518 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11704* bytes in 371 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>
>
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:25 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after
> enable tls ssl socket, when execute gluster v heal <vol-name> info, will
> trigger glfshel to connect glusterfsd process, and cause glusterfsd process
> memory leak. Could you please try in your env?
>
>
>
> cynthia
>
>
>
>
>
> *From:* Atin Mukherjee <amukherj at redhat.com>
> *Sent:* Thursday, April 18, 2019 1:19 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
>
>
>
>
> On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> In my recent test, I found that there are very severe glusterfsd memory
> leak when enable socket ssl option
>
>
>
> What gluster version are you testing? Would you be able to continue your
> investigation and share the root cause?
>
>
>
> --
>
> - Atin (atinm)
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/954d38d0/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 03:02:17 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 03:02:17 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
Message-ID: <9d40e0cab5e540faadd8b2885c45fb2e@nokia-sbell.com>

Hi,
Thanks for your quick responding!
My test env is without patch 22334, so priv->ssl_ctx is shared one. In socket_reset there is no need to free it. but if with patch 22334, it is absolutely needed to free priv->ssl_ctx as well.
Following is the log captured from glusterfsd side while connection establishment, when accept returns, a new  new_trans is allocated, and the socket_init input param is this new_trans, because this new_trans has no option now, so in socket_init   ssl_setup_connection_params is not called, so there should be no malloc done here, as showed from following log, ssl_setup_connection_params is called in socket_event_handler, and here the ssl_ctx had been assigned to the shared one.
This issue is very easy to reproduce in my env, just ?while true; do gluster v heal <vol-name> info;done? and check the memory of the corresponding glusterfsd , it is very obvious increase.

One more thing to confirm is that there is no need to free the ssl_sbio right? Ssl_free will handle it and just call ssl_free is enough?


[2019-04-21 05:02:17.820346] T [socket.c:2776:socket_server_event_handler] 0-tcp.ccs-server: XXX server:192.168.1.13:53952, client:192.168.1.13:63683
[2019-04-21 05:02:17.820370] T [socket.c:2820:socket_server_event_handler] 0-tcp.ccs-server: ### use non-blocking IO
[2019-04-21 05:02:17.820410] T [socket.c:2603:socket_event_handler] 0-tcp.ccs-server: server (sock:126) in:1, out:0, err:0
[2019-04-21 05:02:17.820431] T [socket.c:2610:socket_event_handler] 0-tcp.ccs-server: server (sock:126) socket is not connected, completing connection
[2019-04-21 05:02:17.820442] T [socket.c:3829:ssl_setup_connection_params] 0-tcp.ccs-server: found old SSL context!   // context is shared between listening socket and accepted socket
[2019-04-21 05:02:17.820451] T [socket.c:310:ssl_setup_connection_prefix] 0-tcp.ccs-server: + ssl_setup_connection_params() done!
[2019-04-21 05:02:17.822559] T [socket.c:2617:socket_event_handler] 0-tcp.ccs-server: (sock:126) socket_complete_connection() returned 1
[2019-04-21 05:02:17.822585] T [socket.c:2621:socket_event_handler] 0-tcp.ccs-server: (sock:126) returning to wait on socket
[2019-04-21 05:02:17.828455] T [socket.c:2603:socket_event_handler] 0-tcp.ccs-server: server (sock:126) in:1, out:0, err:0
[2019-04-21 05:02:17.828483] T [socket.c:2610:socket_event_handler] 0-tcp.ccs-server: server (sock:126) socket is not connected, completing connection
[2019-04-21 05:02:17.829130] D [socket.c:366:ssl_setup_connection_postfix] 0-tcp.ccs-server: peer CN = example ee certificate
[2019-04-21 05:02:17.829157] D [socket.c:369:ssl_setup_connection_postfix] 0-tcp.ccs-server: SSL verification succeeded (client: 192.168.1.13:63683) (server: 192.168.1.13:53952)
[2019-04-21 05:02:17.829171] T [socket.c:423:ssl_complete_connection] 0-tcp.ccs-server: ssl_accepted!
[2019-04-21 05:02:17.829183] T [socket.c:2617:socket_event_handler] 0-tcp.ccs-server: (sock:126) socket_complete_connection() returned 1
[2019-04-21 05:02:17.829192] T [socket.c:2621:socket_event_handler] 0-tcp.ccs-server: (sock:126) returning to wait on socket
[2019-04-21 05:02:17.829261] T [socket.c:2603:socket_event_handler] 0-tcp.ccs-server: server (sock:126) in:1, out:0, err:0
[2019-04-21 05:02:17.829282] T [socket.c:2628:socket_event_handler] 0-tcp.ccs-server: Server socket (126) is already connected
[2019-04-21 05:02:17.829294] T [socket.c:493:__socket_ssl_readv] 0-tcp.ccs-server: ***** reading over SSL
[2019-04-21 05:02:17.829311] T [socket.c:493:__socket_ssl_readv] 0-tcp.ccs-server: ***** reading over SSL
[2019-04-21 05:02:17.829337] T [socket.c:493:__socket_ssl_readv] 0-tcp.ccs-server: ***** reading over SSL


cynthia
From: Milind Changire <mchangir at redhat.com>
Sent: Monday, April 22, 2019 10:21 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

After patch 22334<https://review.gluster.org/c/glusterfs/+/22334>, the priv->ssl_ctx is now maintained per socket connection and is no longer shared.
So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx to NULL.

There might be some strings that are duplicated (gf_strdup()) via the socket_init() code path. Please take a look at those as well.

Sorry about that. I missed it.


On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:

Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; 'gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>' <gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/28e61eda/attachment-0001.html>

From mchangir at redhat.com  Mon Apr 22 03:46:07 2019
From: mchangir at redhat.com (Milind Changire)
Date: Mon, 22 Apr 2019 09:16:07 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <9d40e0cab5e540faadd8b2885c45fb2e@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<9d40e0cab5e540faadd8b2885c45fb2e@nokia-sbell.com>
Message-ID: <CAED=hWB-_F2NAOc+S0W4i8c86XcvSj8Nk=B0DeMbCBXK++Cbyw@mail.gmail.com>

On Mon, Apr 22, 2019 at 8:32 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi,
>
> Thanks for your quick responding!
>
> My test env is without patch 22334, so priv->ssl_ctx is shared one. In
> socket_reset there is no need to free it. but if with patch 22334, it is
> absolutely needed to free priv->ssl_ctx as well.
>
> Following is the log captured from glusterfsd side while connection
> establishment, when accept returns, a new  new_trans is allocated, and the
> socket_init input param is this new_trans, because this new_trans has no
> option now, so in socket_init   ssl_setup_connection_params is not called,
> so there should be no malloc done here, as showed from following log,
> ssl_setup_connection_params is called in socket_event_handler, and here the
> ssl_ctx had been assigned to the shared one.
>
> This issue is very easy to reproduce in my env, just ?while true; do
> gluster v heal <vol-name> info;done? and check the memory of the
> corresponding glusterfsd , it is very obvious increase.
>
> for pre-22334 code ssl_ctx free need not be done


>
> *One more thing to confirm is that there is no need to free the ssl_sbio
> right? Ssl_free will handle it and just call ssl_free is enough?*
>
You need to call BIO_free()
Check man bio.


>
>
>
> *[2019-04-21 05:02:17.820346] T
> [socket.c:2776:socket_server_event_handler] 0-tcp.ccs-server: XXX
> server:192.168.1.13:53952 <http://192.168.1.13:53952>,
> client:192.168.1.13:63683 <http://192.168.1.13:63683>*
>
> *[2019-04-21 05:02:17.820370] T
> [socket.c:2820:socket_server_event_handler] 0-tcp.ccs-server: ### use
> non-blocking IO*
>
> [2019-04-21 05:02:17.820410] T [socket.c:2603:socket_event_handler]
> 0-tcp.ccs-server: server (sock:126) in:1, out:0, err:0
>
> [2019-04-21 05:02:17.820431] T [socket.c:2610:socket_event_handler]
> 0-tcp.ccs-server: server (sock:126) socket is not connected, completing
> connection
>
> [2019-04-21 05:02:17.820442] T [socket.c:3829:
> *ssl_setup_connection_params*] 0-tcp.ccs-server: *found old SSL
> context!   // context is shared between listening socket and accepted
> socket*
>
> [2019-04-21 05:02:17.820451] T [socket.c:310:ssl_setup_connection_prefix]
> 0-tcp.ccs-server: + ssl_setup_connection_params() done!
>
> [2019-04-21 05:02:17.822559] T [socket.c:2617:socket_event_handler]
> 0-tcp.ccs-server: (sock:126) socket_complete_connection() returned 1
>
> [2019-04-21 05:02:17.822585] T [socket.c:2621:socket_event_handler]
> 0-tcp.ccs-server: (sock:126) returning to wait on socket
>
> [2019-04-21 05:02:17.828455] T [socket.c:2603:socket_event_handler]
> 0-tcp.ccs-server: server (sock:126) in:1, out:0, err:0
>
> [2019-04-21 05:02:17.828483] T [socket.c:2610:socket_event_handler]
> 0-tcp.ccs-server: server (sock:126) socket is not connected, completing
> connection
>
> [2019-04-21 05:02:17.829130] D [socket.c:366:ssl_setup_connection_postfix]
> 0-tcp.ccs-server: peer CN = example ee certificate
>
> [2019-04-21 05:02:17.829157] D [socket.c:369:ssl_setup_connection_postfix]
> 0-tcp.ccs-server: SSL verification succeeded (client: 192.168.1.13:63683)
> (server: 192.168.1.13:53952)
>
> [2019-04-21 05:02:17.829171] T [socket.c:423:ssl_complete_connection]
> 0-tcp.ccs-server: ssl_accepted!
>
> [2019-04-21 05:02:17.829183] T [socket.c:2617:socket_event_handler]
> 0-tcp.ccs-server: (sock:126) socket_complete_connection() returned 1
>
> [2019-04-21 05:02:17.829192] T [socket.c:2621:socket_event_handler]
> 0-tcp.ccs-server: (sock:126) returning to wait on socket
>
> [2019-04-21 05:02:17.829261] T [socket.c:2603:socket_event_handler]
> 0-tcp.ccs-server: server (sock:126) in:1, out:0, err:0
>
> [2019-04-21 05:02:17.829282] T [socket.c:2628:socket_event_handler]
> 0-tcp.ccs-server: Server socket (126) is already connected
>
> [2019-04-21 05:02:17.829294] T [socket.c:493:__socket_ssl_readv]
> 0-tcp.ccs-server: ***** reading over SSL
>
> [2019-04-21 05:02:17.829311] T [socket.c:493:__socket_ssl_readv]
> 0-tcp.ccs-server: ***** reading over SSL
>
> [2019-04-21 05:02:17.829337] T [socket.c:493:__socket_ssl_readv]
> 0-tcp.ccs-server: ***** reading over SSL
>
>
>
>
>
> cynthia
>
> *From:* Milind Changire <mchangir at redhat.com>
> *Sent:* Monday, April 22, 2019 10:21 AM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> After patch 22334 <https://review.gluster.org/c/glusterfs/+/22334>, the
> priv->ssl_ctx is now maintained per socket connection and is no longer
> shared.
>
> So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx
> to NULL.
>
>
>
> There might be some strings that are duplicated (gf_strdup()) via the
> socket_init() code path. Please take a look at those as well.
>
>
>
> Sorry about that. I missed it.
>
>
>
>
>
> On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
>
>
> Hi,
>
> From my code study it seems priv->ssl_ssl is not properly released, I made
> a patch and the glusterfsd memory leak is alleviated with my patch, but
> some otherwhere is still leaking, I have no clue about the other leak
> points.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
>
>    memset(&priv->incoming, 0, sizeof(priv->incoming));
>
>    event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
>
> -
>
> +  if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>    priv->sock = -1;
>
>    priv->idx = -1;
>
>    priv->connected = -1;
>
> @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
>
>      pthread_mutex_destroy(&priv->out_lock);
>
>      pthread_mutex_destroy(&priv->cond_lock);
>
>      pthread_cond_destroy(&priv->cond);
>
> +     if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>      if (priv->ssl_private_key) {
>
>        GF_FREE(priv->ssl_private_key);
>
>      }
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:31 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>; '
> gluster-devel at gluster.org' <gluster-devel at gluster.org>
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> We scan it use memory-leak tool, there are following prints. We doubt some
> open ssl lib malloc is is not properly freed by glusterfs code.
>
> er+0x2af [*libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda* [*libpthread-2.27.so
> <http://libpthread-2.27.so>*]'
>         *13580* bytes in 175 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p*]'
>         *232904* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
> [15:41:56] Top 10 stacks with outstanding allocations:
>         *8792* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *9408* bytes in 42 allocations from stack
>                 b'CRYPTO_realloc+0x4d [*libcrypto.so.1.0.2p*]'
>         *9723* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *10696* bytes in 21 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11319* bytes in 602 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11431* bytes in 518 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11704* bytes in 371 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>
>
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:25 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after
> enable tls ssl socket, when execute gluster v heal <vol-name> info, will
> trigger glfshel to connect glusterfsd process, and cause glusterfsd process
> memory leak. Could you please try in your env?
>
>
>
> cynthia
>
>
>
>
>
> *From:* Atin Mukherjee <amukherj at redhat.com>
> *Sent:* Thursday, April 18, 2019 1:19 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
>
>
>
>
> On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> In my recent test, I found that there are very severe glusterfsd memory
> leak when enable socket ssl option
>
>
>
> What gluster version are you testing? Would you be able to continue your
> investigation and share the root cause?
>
>
>
> --
>
> - Atin (atinm)
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
>
> Milind
>


-- 
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/9717e4a8/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 05:15:31 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 05:15:31 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
Message-ID: <3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>

Why there is no bio_free called in ssl_teardown_connection then?

cynthia

From: Milind Changire <mchangir at redhat.com>
Sent: Monday, April 22, 2019 10:21 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

After patch 22334<https://review.gluster.org/c/glusterfs/+/22334>, the priv->ssl_ctx is now maintained per socket connection and is no longer shared.
So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx to NULL.

There might be some strings that are duplicated (gf_strdup()) via the socket_init() code path. Please take a look at those as well.

Sorry about that. I missed it.


On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:

Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; 'gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>' <gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/a02ec0b0/attachment-0001.html>

From mchangir at redhat.com  Mon Apr 22 05:34:30 2019
From: mchangir at redhat.com (Milind Changire)
Date: Mon, 22 Apr 2019 11:04:30 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
Message-ID: <CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>

This probably went unnoticed until now.



On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Why there is no bio_free called in ssl_teardown_connection then?
>
>
>
> cynthia
>
>
>
> *From:* Milind Changire <mchangir at redhat.com>
> *Sent:* Monday, April 22, 2019 10:21 AM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> After patch 22334 <https://review.gluster.org/c/glusterfs/+/22334>, the
> priv->ssl_ctx is now maintained per socket connection and is no longer
> shared.
>
> So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx
> to NULL.
>
>
>
> There might be some strings that are duplicated (gf_strdup()) via the
> socket_init() code path. Please take a look at those as well.
>
>
>
> Sorry about that. I missed it.
>
>
>
>
>
> On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
>
>
> Hi,
>
> From my code study it seems priv->ssl_ssl is not properly released, I made
> a patch and the glusterfsd memory leak is alleviated with my patch, but
> some otherwhere is still leaking, I have no clue about the other leak
> points.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
>
>    memset(&priv->incoming, 0, sizeof(priv->incoming));
>
>    event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
>
> -
>
> +  if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>    priv->sock = -1;
>
>    priv->idx = -1;
>
>    priv->connected = -1;
>
> @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
>
>      pthread_mutex_destroy(&priv->out_lock);
>
>      pthread_mutex_destroy(&priv->cond_lock);
>
>      pthread_cond_destroy(&priv->cond);
>
> +     if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>      if (priv->ssl_private_key) {
>
>        GF_FREE(priv->ssl_private_key);
>
>      }
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:31 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>; '
> gluster-devel at gluster.org' <gluster-devel at gluster.org>
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> We scan it use memory-leak tool, there are following prints. We doubt some
> open ssl lib malloc is is not properly freed by glusterfs code.
>
> er+0x2af [*libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda* [*libpthread-2.27.so
> <http://libpthread-2.27.so>*]'
>         *13580* bytes in 175 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p*]'
>         *232904* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
> [15:41:56] Top 10 stacks with outstanding allocations:
>         *8792* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *9408* bytes in 42 allocations from stack
>                 b'CRYPTO_realloc+0x4d [*libcrypto.so.1.0.2p*]'
>         *9723* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *10696* bytes in 21 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11319* bytes in 602 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11431* bytes in 518 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11704* bytes in 371 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>
>
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:25 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after
> enable tls ssl socket, when execute gluster v heal <vol-name> info, will
> trigger glfshel to connect glusterfsd process, and cause glusterfsd process
> memory leak. Could you please try in your env?
>
>
>
> cynthia
>
>
>
>
>
> *From:* Atin Mukherjee <amukherj at redhat.com>
> *Sent:* Thursday, April 18, 2019 1:19 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
>
>
>
>
> On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> In my recent test, I found that there are very severe glusterfsd memory
> leak when enable socket ssl option
>
>
>
> What gluster version are you testing? Would you be able to continue your
> investigation and share the root cause?
>
>
>
> --
>
> - Atin (atinm)
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
>
> Milind
>


-- 
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/7a9fbed5/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 06:07:02 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 06:07:02 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
	<CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
Message-ID: <e0fa216ffa0f4e5fb0e79b13ac8215a0@nokia-sbell.com>

I tried to print priv->ssl_sbio after SSL_free() find this pointer is not null, so I add free ssl_sbio with BIO_free, however this cause glusterfd coredump
(gdb) bt
#0  0x00007f3047867f9b in raise () from /lib64/libc.so.6
#1  0x00007f3047869351 in abort () from /lib64/libc.so.6
#2  0x00007f30478aa8c7 in __libc_message () from /lib64/libc.so.6
#3  0x00007f30478b0e6a in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f30478b2835 in _int_free () from /lib64/libc.so.6
#5  0x00007f3047c5bbbd in CRYPTO_free () from /lib64/libcrypto.so.10
#6  0x00007f3047d07582 in BIO_free () from /lib64/libcrypto.so.10
#7  0x00007f3043f9ba4b in __socket_reset (this=0x7f303c1ae710) at socket.c:1032
#8  0x00007f3043f9c4aa in socket_event_poll_err (this=0x7f303c1ae710, gen=1, idx=17) at socket.c:1232
#9  0x00007f3043fa1b7d in socket_event_handler (fd=26, idx=17, gen=1, data=0x7f303c1ae710, poll_in=1, poll_out=0, poll_err=0) at socket.c:2669
#10 0x00007f3049307984 in event_dispatch_epoll_handler (event_pool=0x1035610, event=0x7f3043b14e84) at event-epoll.c:587
#11 0x00007f3049307c5b in event_dispatch_epoll_worker (data=0x107e3e0) at event-epoll.c:663
#12 0x00007f30480535da in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3047929eaf in clone () from /lib64/libc.so.6

@@ -1019,7 +1019,20 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_INFO,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p ",priv->sock,priv->ssl_sbio);
+     if(priv->ssl_sbio != NULL)
+         BIO_free(priv->ssl_sbio);
+    priv->ssl_ssl = NULL;
+     priv->ssl_sbio = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4251,20 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_INFO,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+     gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p ",priv->sock,priv->ssl_sbio);
+    if(priv->ssl_sbio != NULL)
+        BIO_free(priv->ssl_sbio);
+    priv->ssl_ssl = NULL;
+     priv->ssl_sbio = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);


From: Milind Changire <mchangir at redhat.com>
Sent: Monday, April 22, 2019 1:35 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

This probably went unnoticed until now.



On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Why there is no bio_free called in ssl_teardown_connection then?

cynthia

From: Milind Changire <mchangir at redhat.com<mailto:mchangir at redhat.com>>
Sent: Monday, April 22, 2019 10:21 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

After patch 22334<https://review.gluster.org/c/glusterfs/+/22334>, the priv->ssl_ctx is now maintained per socket connection and is no longer shared.
So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx to NULL.

There might be some strings that are duplicated (gf_strdup()) via the socket_init() code path. Please take a look at those as well.

Sorry about that. I missed it.


On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:

Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; 'gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>' <gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Milind


--
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/e46d4e16/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 06:19:49 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 06:19:49 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
	<CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
Message-ID: <bc0655239eb645f49647df10e7e89a62@nokia-sbell.com>

I do some google. Seems it is not needed to call BIO_free, SSL_free will free bio.

https://groups.google.com/forum/#!topic/mailing.openssl.users/8i9cRQGlfDM
From: Milind Changire <mchangir at redhat.com>
Sent: Monday, April 22, 2019 1:35 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

This probably went unnoticed until now.



On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Why there is no bio_free called in ssl_teardown_connection then?

Cynthia

From: Milind Changire <mchangir at redhat.com<mailto:mchangir at redhat.com>>
Sent: Monday, April 22, 2019 10:21 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

After patch 22334<https://review.gluster.org/c/glusterfs/+/22334>, the priv->ssl_ctx is now maintained per socket connection and is no longer shared.
So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx to NULL.

There might be some strings that are duplicated (gf_strdup()) via the socket_init() code path. Please take a look at those as well.

Sorry about that. I missed it.


On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:

Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; 'gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>' <gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Milind


--
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/02b7c419/attachment-0001.html>

From mchangir at redhat.com  Mon Apr 22 06:21:19 2019
From: mchangir at redhat.com (Milind Changire)
Date: Mon, 22 Apr 2019 11:51:19 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <e0fa216ffa0f4e5fb0e79b13ac8215a0@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
	<CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
	<e0fa216ffa0f4e5fb0e79b13ac8215a0@nokia-sbell.com>
Message-ID: <CAED=hWAwJ9Mbyw8g1YZ_R0YK7tPiR3Hc5K-0qJvreWiy4+ZPEA@mail.gmail.com>

Looks like using BIO_free() is not right. Here's what the SSL_set_bio() man
page says ...

SSL_set_bio() is similar to SSL_set0_rbio() and SSL_set0_wbio() except that
it connects both the rbio and the wbio at the same time, and *transfers the
ownership of rbio and wbio to ssl* according to the following set of rules:

So, I think you were right about SSL_free() doing the job for the bio.
However, SSL_free() has no reason to set the priv->ssl_sbio pointer to
NULL. I think priv->ssl_sbio should be set to NULL immediately after the
call to SSL_set_bio() is successful. And we need to add a comment while
setting priv->ssl_sbio to NULL that the ownership of the bio has now been
transferred to SSL and SSL will free the related memory appropriately.



On Mon, Apr 22, 2019 at 11:37 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> I tried to print priv->ssl_sbio after SSL_free() find this pointer is not
> null, so I add free ssl_sbio with BIO_free, however this cause glusterfd
> coredump
>
> (gdb) bt
>
> #0  0x00007f3047867f9b in raise () from /lib64/libc.so.6
>
> #1  0x00007f3047869351 in abort () from /lib64/libc.so.6
>
> #2  0x00007f30478aa8c7 in __libc_message () from /lib64/libc.so.6
>
> #3  0x00007f30478b0e6a in malloc_printerr () from /lib64/libc.so.6
>
> #4  0x00007f30478b2835 in _int_free () from /lib64/libc.so.6
>
> #5  0x00007f3047c5bbbd in CRYPTO_free () from /lib64/libcrypto.so.10
>
> #6  0x00007f3047d07582 in BIO_free () from /lib64/libcrypto.so.10
>
> #7  0x00007f3043f9ba4b in __socket_reset (this=0x7f303c1ae710) at
> socket.c:1032
>
> #8  0x00007f3043f9c4aa in socket_event_poll_err (this=0x7f303c1ae710,
> gen=1, idx=17) at socket.c:1232
>
> #9  0x00007f3043fa1b7d in socket_event_handler (fd=26, idx=17, gen=1,
> data=0x7f303c1ae710, poll_in=1, poll_out=0, poll_err=0) at socket.c:2669
>
> #10 0x00007f3049307984 in event_dispatch_epoll_handler
> (event_pool=0x1035610, event=0x7f3043b14e84) at event-epoll.c:587
>
> #11 0x00007f3049307c5b in event_dispatch_epoll_worker (data=0x107e3e0) at
> event-epoll.c:663
>
> #12 0x00007f30480535da in start_thread () from /lib64/libpthread.so.0
>
> #13 0x00007f3047929eaf in clone () from /lib64/libc.so.6
>
>
>
> @@ -1019,7 +1019,20 @@ static void __socket_reset(rpc_transport_t *this) {
>
>    memset(&priv->incoming, 0, sizeof(priv->incoming));
>
>    event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
>
> -
>
> +  if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_INFO,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p
> ",priv->sock,priv->ssl_sbio);
>
> +     if(priv->ssl_sbio != NULL)
>
> +         BIO_free(priv->ssl_sbio);
>
> +    priv->ssl_ssl = NULL;
>
> +     priv->ssl_sbio = NULL;
>
> +  }
>
>    priv->sock = -1;
>
>    priv->idx = -1;
>
>    priv->connected = -1;
>
> @@ -4238,6 +4251,20 @@ void fini(rpc_transport_t *this) {
>
>      pthread_mutex_destroy(&priv->out_lock);
>
>      pthread_mutex_destroy(&priv->cond_lock);
>
>      pthread_cond_destroy(&priv->cond);
>
> +     if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_INFO,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +     gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p
> ",priv->sock,priv->ssl_sbio);
>
> +    if(priv->ssl_sbio != NULL)
>
> +        BIO_free(priv->ssl_sbio);
>
> +    priv->ssl_ssl = NULL;
>
> +     priv->ssl_sbio = NULL;
>
> +  }
>
>      if (priv->ssl_private_key) {
>
>        GF_FREE(priv->ssl_private_key);
>
>
>
>
>
> *From:* Milind Changire <mchangir at redhat.com>
> *Sent:* Monday, April 22, 2019 1:35 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> This probably went unnoticed until now.
>
>
>
>
>
>
>
> On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Why there is no bio_free called in ssl_teardown_connection then?
>
>
>
> cynthia
>
>
>
> *From:* Milind Changire <mchangir at redhat.com>
> *Sent:* Monday, April 22, 2019 10:21 AM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> After patch 22334 <https://review.gluster.org/c/glusterfs/+/22334>, the
> priv->ssl_ctx is now maintained per socket connection and is no longer
> shared.
>
> So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx
> to NULL.
>
>
>
> There might be some strings that are duplicated (gf_strdup()) via the
> socket_init() code path. Please take a look at those as well.
>
>
>
> Sorry about that. I missed it.
>
>
>
>
>
> On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
>
>
> Hi,
>
> From my code study it seems priv->ssl_ssl is not properly released, I made
> a patch and the glusterfsd memory leak is alleviated with my patch, but
> some otherwhere is still leaking, I have no clue about the other leak
> points.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
>
>    memset(&priv->incoming, 0, sizeof(priv->incoming));
>
>    event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
>
> -
>
> +  if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>    priv->sock = -1;
>
>    priv->idx = -1;
>
>    priv->connected = -1;
>
> @@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
>
>      pthread_mutex_destroy(&priv->out_lock);
>
>      pthread_mutex_destroy(&priv->cond_lock);
>
>      pthread_cond_destroy(&priv->cond);
>
> +     if(priv->use_ssl&& priv->ssl_ssl)
>
> +  {
>
> +    gf_log(this->name, GF_LOG_TRACE,
>
> +           "clear and reset for socket(%d), free ssl ",
>
> +           priv->sock);
>
> +    SSL_shutdown(priv->ssl_ssl);
>
> +    SSL_clear(priv->ssl_ssl);
>
> +    SSL_free(priv->ssl_ssl);
>
> +    priv->ssl_ssl = NULL;
>
> +  }
>
>      if (priv->ssl_private_key) {
>
>        GF_FREE(priv->ssl_private_key);
>
>      }
>
>
>
>
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:31 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>; '
> gluster-devel at gluster.org' <gluster-devel at gluster.org>
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> We scan it use memory-leak tool, there are following prints. We doubt some
> open ssl lib malloc is is not properly freed by glusterfs code.
>
> er+0x2af [*libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda* [
> libpthread-2.27.so]'
>         *13580* bytes in 175 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p*]'
>         *232904* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
> [15:41:56] Top 10 stacks with outstanding allocations:
>         *8792* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *9408* bytes in 42 allocations from stack
>                 b'CRYPTO_realloc+0x4d [*libcrypto.so.1.0.2p*]'
>         *9723* bytes in 14 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *10696* bytes in 21 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11319* bytes in 602 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11431* bytes in 518 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>         *11704* bytes in 371 allocations from stack
>                 b'CRYPTO_malloc+0x58 [*libcrypto.so.1.0.2p]\n\t\t[unknown*
> ]'
>
>
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Thursday, April 18, 2019 5:25 PM
> *To:* 'Atin Mukherjee' <amukherj at redhat.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* RE: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
> I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after
> enable tls ssl socket, when execute gluster v heal <vol-name> info, will
> trigger glfshel to connect glusterfsd process, and cause glusterfsd process
> memory leak. Could you please try in your env?
>
>
>
> cynthia
>
>
>
>
>
> *From:* Atin Mukherjee <amukherj at redhat.com>
> *Sent:* Thursday, April 18, 2019 1:19 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* Raghavendra Gowdappa <rgowdapp at redhat.com>;
> gluster-devel at gluster.org
> *Subject:* Re: [Gluster-devel] glusterfsd memory leak issue found after
> enable ssl
>
>
>
>
>
>
>
> On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> In my recent test, I found that there are very severe glusterfsd memory
> leak when enable socket ssl option
>
>
>
> What gluster version are you testing? Would you be able to continue your
> investigation and share the root cause?
>
>
>
> --
>
> - Atin (atinm)
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
>
> Milind
>
>
>
> --
>
> Milind
>


-- 
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/8cc65874/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 06:30:45 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 06:30:45 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAED=hWAwJ9Mbyw8g1YZ_R0YK7tPiR3Hc5K-0qJvreWiy4+ZPEA@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
	<CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
	<e0fa216ffa0f4e5fb0e79b13ac8215a0@nokia-sbell.com>
	<CAED=hWAwJ9Mbyw8g1YZ_R0YK7tPiR3Hc5K-0qJvreWiy4+ZPEA@mail.gmail.com>
Message-ID: <b2d7a72386154bf79cd893e3c8dbfa50@nokia-sbell.com>

Ok ,another question, why priv->ssl_sbio = BIO_new_socket(priv->sock, BIO_NOCLOSE); use NOCLOSE mode instead of BIO_CLOSE?

cynthia

From: Milind Changire <mchangir at redhat.com>
Sent: Monday, April 22, 2019 2:21 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

Looks like using BIO_free() is not right. Here's what the SSL_set_bio() man page says ...

SSL_set_bio() is similar to SSL_set0_rbio() and SSL_set0_wbio() except that it connects both the rbio and the wbio at the same time, and transfers the ownership of rbio and wbio to ssl according to the following set of rules:

So, I think you were right about SSL_free() doing the job for the bio. However, SSL_free() has no reason to set the priv->ssl_sbio pointer to NULL. I think priv->ssl_sbio should be set to NULL immediately after the call to SSL_set_bio() is successful. And we need to add a comment while setting priv->ssl_sbio to NULL that the ownership of the bio has now been transferred to SSL and SSL will free the related memory appropriately.



On Mon, Apr 22, 2019 at 11:37 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
I tried to print priv->ssl_sbio after SSL_free() find this pointer is not null, so I add free ssl_sbio with BIO_free, however this cause glusterfd coredump
(gdb) bt
#0  0x00007f3047867f9b in raise () from /lib64/libc.so.6
#1  0x00007f3047869351 in abort () from /lib64/libc.so.6
#2  0x00007f30478aa8c7 in __libc_message () from /lib64/libc.so.6
#3  0x00007f30478b0e6a in malloc_printerr () from /lib64/libc.so.6
#4  0x00007f30478b2835 in _int_free () from /lib64/libc.so.6
#5  0x00007f3047c5bbbd in CRYPTO_free () from /lib64/libcrypto.so.10
#6  0x00007f3047d07582 in BIO_free () from /lib64/libcrypto.so.10
#7  0x00007f3043f9ba4b in __socket_reset (this=0x7f303c1ae710) at socket.c:1032
#8  0x00007f3043f9c4aa in socket_event_poll_err (this=0x7f303c1ae710, gen=1, idx=17) at socket.c:1232
#9  0x00007f3043fa1b7d in socket_event_handler (fd=26, idx=17, gen=1, data=0x7f303c1ae710, poll_in=1, poll_out=0, poll_err=0) at socket.c:2669
#10 0x00007f3049307984 in event_dispatch_epoll_handler (event_pool=0x1035610, event=0x7f3043b14e84) at event-epoll.c:587
#11 0x00007f3049307c5b in event_dispatch_epoll_worker (data=0x107e3e0) at event-epoll.c:663
#12 0x00007f30480535da in start_thread () from /lib64/libpthread.so.0
#13 0x00007f3047929eaf in clone () from /lib64/libc.so.6

@@ -1019,7 +1019,20 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_INFO,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p ",priv->sock,priv->ssl_sbio);
+     if(priv->ssl_sbio != NULL)
+         BIO_free(priv->ssl_sbio);
+    priv->ssl_ssl = NULL;
+     priv->ssl_sbio = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4251,20 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_INFO,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+     gf_log(this->name, GF_LOG_INFO,"priv->ssl_sbio of socket(%d)is %p ",priv->sock,priv->ssl_sbio);
+    if(priv->ssl_sbio != NULL)
+        BIO_free(priv->ssl_sbio);
+    priv->ssl_ssl = NULL;
+     priv->ssl_sbio = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);


From: Milind Changire <mchangir at redhat.com<mailto:mchangir at redhat.com>>
Sent: Monday, April 22, 2019 1:35 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

This probably went unnoticed until now.



On Mon, Apr 22, 2019 at 10:45 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Why there is no bio_free called in ssl_teardown_connection then?

cynthia

From: Milind Changire <mchangir at redhat.com<mailto:mchangir at redhat.com>>
Sent: Monday, April 22, 2019 10:21 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

After patch 22334<https://review.gluster.org/c/glusterfs/+/22334>, the priv->ssl_ctx is now maintained per socket connection and is no longer shared.
So you might want to SSL_free(priv->ssl_ctx) as well and set priv->ssl_ctx to NULL.

There might be some strings that are duplicated (gf_strdup()) via the socket_init() code path. Please take a look at those as well.

Sorry about that. I missed it.


On Mon, Apr 22, 2019 at 7:25 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:

Hi,
From my code study it seems priv->ssl_ssl is not properly released, I made a patch and the glusterfsd memory leak is alleviated with my patch, but some otherwhere is still leaking, I have no clue about the other leak points.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -1019,7 +1019,16 @@ static void __socket_reset(rpc_transport_t *this) {
   memset(&priv->incoming, 0, sizeof(priv->incoming));
   event_unregister_close(this->ctx->event_pool, priv->sock, priv->idx);
-
+  if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
   priv->sock = -1;
   priv->idx = -1;
   priv->connected = -1;
@@ -4238,6 +4250,16 @@ void fini(rpc_transport_t *this) {
     pthread_mutex_destroy(&priv->out_lock);
     pthread_mutex_destroy(&priv->cond_lock);
     pthread_cond_destroy(&priv->cond);
+     if(priv->use_ssl&& priv->ssl_ssl)
+  {
+    gf_log(this->name, GF_LOG_TRACE,
+           "clear and reset for socket(%d), free ssl ",
+           priv->sock);
+    SSL_shutdown(priv->ssl_ssl);
+    SSL_clear(priv->ssl_ssl);
+    SSL_free(priv->ssl_ssl);
+    priv->ssl_ssl = NULL;
+  }
     if (priv->ssl_private_key) {
       GF_FREE(priv->ssl_private_key);
     }


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:31 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; 'gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>' <gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

We scan it use memory-leak tool, there are following prints. We doubt some open ssl lib malloc is is not properly freed by glusterfs code.
er+0x2af [libglusterfs.so.0.0.1]\n\t\tstart_thread+0xda [libpthread-2.27.so<http://libpthread-2.27.so>]'
        13580 bytes in 175 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]'
        232904 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
[15:41:56] Top 10 stacks with outstanding allocations:
        8792 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        9408 bytes in 42 allocations from stack
                b'CRYPTO_realloc+0x4d [libcrypto.so.1.0.2p]'
        9723 bytes in 14 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        10696 bytes in 21 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11319 bytes in 602 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11431 bytes in 518 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'
        11704 bytes in 371 allocations from stack
                b'CRYPTO_malloc+0x58 [libcrypto.so.1.0.2p]\n\t\t[unknown]'


cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Thursday, April 18, 2019 5:25 PM
To: 'Atin Mukherjee' <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

I?ve test on glusterfs3.12.15 and glusterfs5.5 all have this issue, after enable tls ssl socket, when execute gluster v heal <vol-name> info, will trigger glfshel to connect glusterfsd process, and cause glusterfsd process memory leak. Could you please try in your env?

cynthia


From: Atin Mukherjee <amukherj at redhat.com<mailto:amukherj at redhat.com>>
Sent: Thursday, April 18, 2019 1:19 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>; gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl



On Wed, 17 Apr 2019 at 10:53, Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
In my recent test, I found that there are very severe glusterfsd memory leak when enable socket ssl option

What gluster version are you testing? Would you be able to continue your investigation and share the root cause?

--
- Atin (atinm)
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org<mailto:Gluster-devel at gluster.org>
https://lists.gluster.org/mailman/listinfo/gluster-devel


--
Milind


--
Milind


--
Milind
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/82f86b98/attachment-0001.html>

From mchangir at redhat.com  Mon Apr 22 06:36:04 2019
From: mchangir at redhat.com (Milind Changire)
Date: Mon, 22 Apr 2019 12:06:04 +0530
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
	ssl
In-Reply-To: <b2d7a72386154bf79cd893e3c8dbfa50@nokia-sbell.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
	<CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
	<e0fa216ffa0f4e5fb0e79b13ac8215a0@nokia-sbell.com>
	<CAED=hWAwJ9Mbyw8g1YZ_R0YK7tPiR3Hc5K-0qJvreWiy4+ZPEA@mail.gmail.com>
	<b2d7a72386154bf79cd893e3c8dbfa50@nokia-sbell.com>
Message-ID: <CAED=hWBeDLEA-FZXRaLVg6J_807MB3SY7eQZfPDdc7V1zBS=Rg@mail.gmail.com>

According to BIO_new_socket() man page ...

*If the close flag is set then the socket is shut down and closed when the
BIO is freed.*

For Gluster to have more control over the socket shutdown, the BIO_NOCLOSE
flag is set. Otherwise, SSL takes control of socket shutdown whenever BIO
is freed.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/3e50b39b/attachment.html>

From cynthia.zhou at nokia-sbell.com  Mon Apr 22 08:07:39 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Mon, 22 Apr 2019 08:07:39 +0000
Subject: [Gluster-devel] glusterfsd memory leak issue found after enable
 ssl
In-Reply-To: <CAED=hWBeDLEA-FZXRaLVg6J_807MB3SY7eQZfPDdc7V1zBS=Rg@mail.gmail.com>
References: <07cb1c3aa08b414dbe37442955ddad36@nokia-sbell.com>
	<CAGNCGH0kd7Rg8S+9DA_9JfyV8dWDAvErawfuWHOLc_9AycddFQ@mail.gmail.com>
	<f2fbb977a8f74fd79f5dc0b8774129e0@nokia-sbell.com>
	<6ce04fb69243465295a71b6953eafa19@nokia-sbell.com>
	<b1836856b974418d9215e0d86ba6b740@nokia-sbell.com>
	<CAED=hWDLjABGTRX8MDFVW7x52Wbd+k8o9WWRPLQaLvkDzPo+RQ@mail.gmail.com>
	<3cd91d1ce39541e7ad30c60ef15000aa@nokia-sbell.com>
	<CAED=hWAq+fG2Tdmk37=xGmezY8FN2Sm2Z9fsCaOLYErcihBZfw@mail.gmail.com>
	<e0fa216ffa0f4e5fb0e79b13ac8215a0@nokia-sbell.com>
	<CAED=hWAwJ9Mbyw8g1YZ_R0YK7tPiR3Hc5K-0qJvreWiy4+ZPEA@mail.gmail.com>
	<b2d7a72386154bf79cd893e3c8dbfa50@nokia-sbell.com>
	<CAED=hWBeDLEA-FZXRaLVg6J_807MB3SY7eQZfPDdc7V1zBS=Rg@mail.gmail.com>
Message-ID: <5d0c2ed30e884b86ba29bff5a47c960e@nokia-sbell.com>

Ok, I am clear now.
I?ve added ssl_free in socket reset and socket finish function, though glusterfsd memory leak is not that much, still it is leaking, from source code I can not find anything else,
Could you help to check if this issue exists in your env? If not I may have a try to merge your patch .
Step

1>   while true;do gluster v heal <vol-name> info,

2>   check the vol-name glusterfsd memory usage, it is obviously increasing.
cynthia

From: Milind Changire <mchangir at redhat.com>
Sent: Monday, April 22, 2019 2:36 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: Atin Mukherjee <amukherj at redhat.com>; gluster-devel at gluster.org
Subject: Re: [Gluster-devel] glusterfsd memory leak issue found after enable ssl

According to BIO_new_socket() man page ...

If the close flag is set then the socket is shut down and closed when the BIO is freed.

For Gluster to have more control over the socket shutdown, the BIO_NOCLOSE flag is set. Otherwise, SSL takes control of socket shutdown whenever BIO is freed.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/0e6d9f80/attachment.html>

From srangana at redhat.com  Mon Apr 22 13:30:45 2019
From: srangana at redhat.com (Shyam Ranganathan)
Date: Mon, 22 Apr 2019 09:30:45 -0400
Subject: [Gluster-devel] Announcing Gluster release 6.1
Message-ID: <e82bc1da-dd8a-c62e-a885-960fc95c7997@redhat.com>

The Gluster community is pleased to announce the release of Gluster
6.1 (packages available at [1]).

Release notes for the release can be found at [2].

Major changes, features and limitations addressed in this release:

None

Thanks,
Gluster community

[1] Packages for 6.1:
https://download.gluster.org/pub/gluster/glusterfs/6/6.1/

[2] Release notes for 6.1:
https://docs.gluster.org/en/latest/release-notes/6.1/

_______________________________________________
maintainers mailing list
maintainers at gluster.org
https://lists.gluster.org/mailman/listinfo/maintainers


From dkhandel at redhat.com  Mon Apr 22 14:09:46 2019
From: dkhandel at redhat.com (Deepshikha Khandelwal)
Date: Mon, 22 Apr 2019 19:39:46 +0530
Subject: [Gluster-devel] ./tests/basic/afr/tarissue.t is failing on
	regression runs
Message-ID: <CABGtVXF9jGwrR9Vxs4+NST2=Zz_e_gzKTE5M2szj5uoUO7M+4g@mail.gmail.com>

The above test is failing all day today because of rpc-statd going into
inactive state on builders.
I've been trying to enable this service going in dead state after every
run.
1. rpcbind  and rpcbind.socket is running on builders
2. ipv6 is disabled.
3. nfs-server is also working fine
4. is_nfs_export_available is failing for this test which means showmount
is failing. Hence some issue with mount.nfs

This test was not failing earlier. Does anyone know what has changed or how
this can be fixed on builder?

Jiffin and Soumya, can you please help me through this.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/ac9b63f2/attachment.html>

From amukherj at redhat.com  Mon Apr 22 17:27:57 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Mon, 22 Apr 2019 22:57:57 +0530
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <797512f6ff7f1b9fedbf8b7968dd86a6968d9105.camel@redhat.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
	<797512f6ff7f1b9fedbf8b7968dd86a6968d9105.camel@redhat.com>
Message-ID: <CAGNCGH28QdMgpSTqb6aH8Vo4kDOp3PF4fkow3BT2MF2_FBf+Lg@mail.gmail.com>

Is this back again? The recent patches are failing regression :-\ .

On Wed, 3 Apr 2019 at 19:26, Michael Scherer <mscherer at redhat.com> wrote:

> Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <jthottan at redhat.com>
> > wrote:
> >
> > > Hi,
> > >
> > > is_nfs_export_available is just a wrapper around "showmount"
> > > command AFAIR.
> > > I saw following messages in console output.
> > >  mount.nfs: rpc.statd is not running but is required for remote
> > > locking.
> > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local, or
> > > start
> > > statd.
> > > 05:06:55 mount.nfs: an incorrect mount option was specified
> > >
> > > For me it looks rpcbind may not be running on the machine.
> > > Usually rpcbind starts automatically on machines, don't know
> > > whether it
> > > can happen or not.
> > >
> >
> > That's precisely what the question is. Why suddenly we're seeing this
> > happening too frequently. Today I saw atleast 4 to 5 such failures
> > already.
> >
> > Deepshika - Can you please help in inspecting this?
>
> So we think (we are not sure) that the issue is a bit complex.
>
> What we were investigating was nightly run fail on aws. When the build
> crash, the builder is restarted, since that's the easiest way to clean
> everything (since even with a perfect test suite that would clean
> itself, we could always end in a corrupt state on the system, WRT
> mount, fs, etc).
>
> In turn, this seems to cause trouble on aws, since cloud-init or
> something rename eth0 interface to ens5, without cleaning to the
> network configuration.
>
> So the network init script fail (because the image say "start eth0" and
> that's not present), but fail in a weird way. Network is initialised
> and working (we can connect), but the dhclient process is not in the
> right cgroup, and network.service is in failed state. Restarting
> network didn't work. In turn, this mean that rpc-statd refuse to start
> (due to systemd dependencies), which seems to impact various NFS tests.
>
> We have also seen that on some builders, rpcbind pick some IP v6
> autoconfiguration, but we can't reproduce that, and there is no ip v6
> set up anywhere. I suspect the network.service failure is somehow
> involved, but fail to see how. In turn, rpcbind.socket not starting
> could cause NFS test troubles.
>
> Our current stop gap fix was to fix all the builders one by one. Remove
> the config, kill the rogue dhclient, restart network service.
>
> However, we can't be sure this is going to fix the problem long term
> since this only manifest after a crash of the test suite, and it
> doesn't happen so often. (plus, it was working before some day in the
> past, when something did make this fail, and I do not know if that's a
> system upgrade, or a test change, or both).
>
> So we are still looking at it to have a complete understanding of the
> issue, but so far, we hacked our way to make it work (or so do I
> think).
>
> Deepshika is working to fix it long term, by fixing the issue regarding
> eth0/ens5 with a new base image.
> --
> Michael Scherer
> Sysadmin, Community Infrastructure and Platform, OSAS
>
>
> --
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/b5802732/attachment-0001.html>

From rabhat at redhat.com  Mon Apr 22 21:20:58 2019
From: rabhat at redhat.com (FNU Raghavendra Manjunath)
Date: Mon, 22 Apr 2019 17:20:58 -0400
Subject: [Gluster-devel] [Gluster-users] Proposal: Changes in Gluster
	Community meetings
In-Reply-To: <CAHxyDdOZz-aap+_o0ibucAAd202Mq1=y6kDuaJM02ciL-1WRcg@mail.gmail.com>
References: <CAHxyDdN3NdNyGM3L-uX3zYFPxEYHvwNt7XQOSbA8y=1GH1uhaQ@mail.gmail.com>
	<62104B6F-99CF-4C22-80FC-9C177F73E897@onholyground.com>
	<CAHxyDdOhYfcOMh+LUV3dU+UqrU5AycNHaU21SRu=SVdZYaDuRg@mail.gmail.com>
	<CAHxyDdOZz-aap+_o0ibucAAd202Mq1=y6kDuaJM02ciL-1WRcg@mail.gmail.com>
Message-ID: <CAMRKmT_oUCj1fvVgpRXLGW87v8ajk73QP1aEMhi5-oekE+oaRA@mail.gmail.com>

Hi,

This is the agenda for tomorrow's community meeting for NA/EMEA timezone.

https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both
----



On Thu, Apr 11, 2019 at 4:56 AM Amar Tumballi Suryanarayan <
atumball at redhat.com> wrote:

> Hi All,
>
> Below is the final details of our community meeting, and I will be sending
> invites to mailing list following this email. You can add Gluster Community
> Calendar so you can get notifications on the meetings.
>
> We are starting the meetings from next week. For the first meeting, we
> need 1 volunteer from users to discuss the use case / what went well, and
> what went bad, etc. preferrably in APAC region.  NA/EMEA region, next week.
>
> Draft Content: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g
> ----
> Gluster Community Meeting
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Previous-Meeting-minutes>Previous
> Meeting minutes:
>
>    - http://github.com/gluster/community
>
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#DateTime-Check-the-community-calendar>Date/Time:
> Check the community calendar
> <https://calendar.google.com/calendar/b/1?cid=dmViajVibDBrbnNiOWQwY205ZWg5cGJsaTRAZ3JvdXAuY2FsZW5kYXIuZ29vZ2xlLmNvbQ>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Bridge>Bridge
>
>    - APAC friendly hours
>       - Bridge: https://bluejeans.com/836554017
>    - NA/EMEA
>       - Bridge: https://bluejeans.com/486278655
>
> ------------------------------
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Attendance>Attendance
>
>    - Name, Company<Optional>
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Host>Host
>
>    - Who will host next meeting?
>       - Host will need to send out the agenda 24hr - 12hrs in advance to
>       mailing list, and also make sure to send the meeting minutes.
>       - Host will need to reach out to one user at least who can talk
>       about their usecase, their experience, and their needs.
>       - Host needs to send meeting minutes as PR to
>       http://github.com/gluster/community
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#User-stories>User stories
>
>    - Discuss 1 usecase from a user.
>       - How was the architecture derived, what volume type used, options,
>       etc?
>       - What were the major issues faced ? How to improve them?
>       - What worked good?
>       - How can we all collaborate well, so it is win-win for the
>       community and the user? How can we
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Community>Community
>
>    -
>
>    Any release updates?
>    -
>
>    Blocker issues across the project?
>    -
>
>    Metrics
>    - Number of new bugs since previous meeting. How many are not triaged?
>       - Number of emails, anything unanswered?
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Conferences--Meetups>Conferences
> / Meetups
>
>    - Any conference in next 1 month where gluster-developers are going?
>    gluster-users are going? So we can meet and discuss.
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#Developer-focus>Developer
> focus
>
>    -
>
>    Any design specs to discuss?
>    -
>
>    Metrics of the week?
>    - Coverity
>       - Clang-Scan
>       - Number of patches from new developers.
>       - Did we increase test coverage?
>       - [Atin] Also talk about most frequent test failures in the CI and
>       carve out an AI to get them fixed.
>
> <https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both#RoundTable>RoundTable
>
>    - <Open for anything which is not covered in agenda>
>
> ----
>
> Regards,
> Amar
>
> On Mon, Mar 25, 2019 at 8:53 PM Amar Tumballi Suryanarayan <
> atumball at redhat.com> wrote:
>
>> Thanks for the feedback Darrell,
>>
>> The new proposal is to have one in North America 'morning' time. (10AM
>> PST), And another in ASIA day time, which is evening 7pm/6pm in Australia,
>> 9pm Newzealand, 5pm Tokyo, 4pm Beijing.
>>
>> For example, if we choose Every other Tuesday for meeting, and 1st of the
>> month is Tuesday, we would have North America time for 1st, and on 15th it
>> would be ASIA/Pacific time.
>>
>> Hopefully, this way, we can cover all the timezones, and meeting minutes
>> would be committed to github repo, so that way, it will be easier for
>> everyone to be aware of what is happening.
>>
>> Regards,
>> Amar
>>
>> On Mon, Mar 25, 2019 at 8:40 PM Darrell Budic <budic at onholyground.com>
>> wrote:
>>
>>> As a user, I?d like to visit more of these, but the time slot is my 3AM.
>>> Any possibility for a rolling schedule (move meeting +6 hours each week
>>> with rolling attendance from maintainers?) or an occasional regional
>>> meeting 12 hours opposed to the one you?re proposing?
>>>
>>>   -Darrell
>>>
>>> On Mar 25, 2019, at 4:25 AM, Amar Tumballi Suryanarayan <
>>> atumball at redhat.com> wrote:
>>>
>>> All,
>>>
>>> We currently have 3 meetings which are public:
>>>
>>> 1. Maintainer's Meeting
>>>
>>> - Runs once in 2 weeks (on Mondays), and current attendance is around
>>> 3-5 on an avg, and not much is discussed.
>>> - Without majority attendance, we can't take any decisions too.
>>>
>>> 2. Community meeting
>>>
>>> - Supposed to happen on #gluster-meeting, every 2 weeks, and is the
>>> only meeting which is for 'Community/Users'. Others are for developers
>>> as of now.
>>> Sadly attendance is getting closer to 0 in recent times.
>>>
>>> 3. GCS meeting
>>>
>>> - We started it as an effort inside Red Hat gluster team, and opened it
>>> up for community from Jan 2019, but the attendance was always from RHT
>>> members, and haven't seen any traction from wider group.
>>>
>>> So, I have a proposal to call out for cancelling all these meeting, and
>>> keeping just 1 weekly 'Community' meeting, where even topics related to
>>> maintainers and GCS and other projects can be discussed.
>>>
>>> I have a template of a draft template @
>>> https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g
>>>
>>> Please feel free to suggest improvements, both in agenda and in timings.
>>> So, we can have more participation from members of community, which allows
>>> more user - developer interactions, and hence quality of project.
>>>
>>> Waiting for feedbacks,
>>>
>>> Regards,
>>> Amar
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>
>> --
>> Amar Tumballi (amarts)
>>
>
>
> --
> Amar Tumballi (amarts)
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190422/68da931b/attachment-0001.html>

From mscherer at redhat.com  Tue Apr 23 14:14:49 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Tue, 23 Apr 2019 16:14:49 +0200
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <CAGNCGH28QdMgpSTqb6aH8Vo4kDOp3PF4fkow3BT2MF2_FBf+Lg@mail.gmail.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
	<797512f6ff7f1b9fedbf8b7968dd86a6968d9105.camel@redhat.com>
	<CAGNCGH28QdMgpSTqb6aH8Vo4kDOp3PF4fkow3BT2MF2_FBf+Lg@mail.gmail.com>
Message-ID: <b89a617416e1f4b64847b8d71b3916be8744ff0b.camel@redhat.com>

Le lundi 22 avril 2019 ? 22:57 +0530, Atin Mukherjee a ?crit :
> Is this back again? The recent patches are failing regression :-\ .

So, on builder206, it took me a while to find that the issue is that
nfs (the service) was running.

./tests/basic/afr/tarissue.t failed, because the nfs initialisation
failed with a rather cryptic message:

[2019-04-23 13:17:05.371733] I [socket.c:991:__socket_server_bind] 0-
socket.nfs-server: process started listening on port (38465)
[2019-04-23 13:17:05.385819] E [socket.c:972:__socket_server_bind] 0-
socket.nfs-server: binding to  failed: Address already in use
[2019-04-23 13:17:05.385843] E [socket.c:974:__socket_server_bind] 0-
socket.nfs-server: Port is already in use
[2019-04-23 13:17:05.385852] E [socket.c:3788:socket_listen] 0-
socket.nfs-server: __socket_server_bind failed;closing socket 14

I found where this came from, but a few stuff did surprised me:

- the order of print is different that the order in the code
- the message on "started listening" didn't take in account the fact
that bind failed on:


https://github.com/gluster/glusterfs/blob/master/rpc/rpc-transport/socket/src/socket.c#L967

The message about port 38465 also threw me off the track. The real
issue is that the service nfs was already running, and I couldn't find
anything listening on port 38465

once I do service nfs stop, it no longer failed.

So far, I do know why nfs.service was activated.

But at least, 206 should be fixed, and we know a bit more on what would
be causing some failure.

 

> On Wed, 3 Apr 2019 at 19:26, Michael Scherer <mscherer at redhat.com>
> wrote:
> 
> > Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> > > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <
> > > jthottan at redhat.com>
> > > wrote:
> > > 
> > > > Hi,
> > > > 
> > > > is_nfs_export_available is just a wrapper around "showmount"
> > > > command AFAIR.
> > > > I saw following messages in console output.
> > > >  mount.nfs: rpc.statd is not running but is required for remote
> > > > locking.
> > > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local,
> > > > or
> > > > start
> > > > statd.
> > > > 05:06:55 mount.nfs: an incorrect mount option was specified
> > > > 
> > > > For me it looks rpcbind may not be running on the machine.
> > > > Usually rpcbind starts automatically on machines, don't know
> > > > whether it
> > > > can happen or not.
> > > > 
> > > 
> > > That's precisely what the question is. Why suddenly we're seeing
> > > this
> > > happening too frequently. Today I saw atleast 4 to 5 such
> > > failures
> > > already.
> > > 
> > > Deepshika - Can you please help in inspecting this?
> > 
> > So we think (we are not sure) that the issue is a bit complex.
> > 
> > What we were investigating was nightly run fail on aws. When the
> > build
> > crash, the builder is restarted, since that's the easiest way to
> > clean
> > everything (since even with a perfect test suite that would clean
> > itself, we could always end in a corrupt state on the system, WRT
> > mount, fs, etc).
> > 
> > In turn, this seems to cause trouble on aws, since cloud-init or
> > something rename eth0 interface to ens5, without cleaning to the
> > network configuration.
> > 
> > So the network init script fail (because the image say "start eth0"
> > and
> > that's not present), but fail in a weird way. Network is
> > initialised
> > and working (we can connect), but the dhclient process is not in
> > the
> > right cgroup, and network.service is in failed state. Restarting
> > network didn't work. In turn, this mean that rpc-statd refuse to
> > start
> > (due to systemd dependencies), which seems to impact various NFS
> > tests.
> > 
> > We have also seen that on some builders, rpcbind pick some IP v6
> > autoconfiguration, but we can't reproduce that, and there is no ip
> > v6
> > set up anywhere. I suspect the network.service failure is somehow
> > involved, but fail to see how. In turn, rpcbind.socket not starting
> > could cause NFS test troubles.
> > 
> > Our current stop gap fix was to fix all the builders one by one.
> > Remove
> > the config, kill the rogue dhclient, restart network service.
> > 
> > However, we can't be sure this is going to fix the problem long
> > term
> > since this only manifest after a crash of the test suite, and it
> > doesn't happen so often. (plus, it was working before some day in
> > the
> > past, when something did make this fail, and I do not know if
> > that's a
> > system upgrade, or a test change, or both).
> > 
> > So we are still looking at it to have a complete understanding of
> > the
> > issue, but so far, we hacked our way to make it work (or so do I
> > think).
> > 
> > Deepshika is working to fix it long term, by fixing the issue
> > regarding
> > eth0/ens5 with a new base image.
> > --
> > Michael Scherer
> > Sysadmin, Community Infrastructure and Platform, OSAS
> > 
> > 
> > --
> 
> - Atin (atinm)
-- 
Michael Scherer
Sysadmin, Community Infrastructure



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190423/45ddd2e2/attachment.sig>

From nbalacha at redhat.com  Wed Apr 24 03:14:10 2019
From: nbalacha at redhat.com (Nithya Balachandran)
Date: Wed, 24 Apr 2019 08:44:10 +0530
Subject: [Gluster-devel] BZ updates
Message-ID: <CAOUCJ=i9EN9OoUp4jeZsU_GV8r6=MPsmN+pfcEXJQgg1pRX76w@mail.gmail.com>

All,

When working on a bug, please ensure that you update the BZ with any
relevant information as well as the RCA. I have seen several BZs in the
past which report crashes, however they do not have a bt or RCA captured.
Having this information in the BZ makes it much easier to see if a newly
reported issue has already been fixed.

I propose that maintainers merge patches only if the BZs are updated with
required information. It will take some time to make this a habit but it
will pay off in the end.

Regards,
Nithya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190424/6b3ea3f3/attachment.html>

From amukherj at redhat.com  Wed Apr 24 03:25:50 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Wed, 24 Apr 2019 08:55:50 +0530
Subject: [Gluster-devel] [Gluster-Maintainers] BZ updates
In-Reply-To: <CAOUCJ=i9EN9OoUp4jeZsU_GV8r6=MPsmN+pfcEXJQgg1pRX76w@mail.gmail.com>
References: <CAOUCJ=i9EN9OoUp4jeZsU_GV8r6=MPsmN+pfcEXJQgg1pRX76w@mail.gmail.com>
Message-ID: <CAGNCGH1YUmq97xbqQutdrxg8qJffVWdRBBP8MDYQg32UiZo7mQ@mail.gmail.com>

Absolutely agree and I definitely think this would help going forward.

On Wed, Apr 24, 2019 at 8:45 AM Nithya Balachandran <nbalacha at redhat.com>
wrote:

> All,
>
> When working on a bug, please ensure that you update the BZ with any
> relevant information as well as the RCA. I have seen several BZs in the
> past which report crashes, however they do not have a bt or RCA captured.
> Having this information in the BZ makes it much easier to see if a newly
> reported issue has already been fixed.
>
> I propose that maintainers merge patches only if the BZs are updated with
> required information. It will take some time to make this a habit but it
> will pay off in the end.
>
> Regards,
> Nithya
> _______________________________________________
> maintainers mailing list
> maintainers at gluster.org
> https://lists.gluster.org/mailman/listinfo/maintainers
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190424/d248e048/attachment.html>

From jthottan at redhat.com  Wed Apr 24 06:59:11 2019
From: jthottan at redhat.com (Jiffin Thottan)
Date: Wed, 24 Apr 2019 02:59:11 -0400 (EDT)
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <b89a617416e1f4b64847b8d71b3916be8744ff0b.camel@redhat.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
	<797512f6ff7f1b9fedbf8b7968dd86a6968d9105.camel@redhat.com>
	<CAGNCGH28QdMgpSTqb6aH8Vo4kDOp3PF4fkow3BT2MF2_FBf+Lg@mail.gmail.com>
	<b89a617416e1f4b64847b8d71b3916be8744ff0b.camel@redhat.com>
Message-ID: <527845347.24314753.1556089151794.JavaMail.zimbra@redhat.com>

Below looks like kernel nfs was started (may be enabled on the machine).

Did u start rpcbind manually on that machine, if yes can u please check kernel nfs status before and after that service?

--

Jiffin

----- Original Message -----
From: "Michael Scherer" <mscherer at redhat.com>
To: "Atin Mukherjee" <amukherj at redhat.com>
Cc: "Deepshikha Khandelwal" <dkhandel at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>, "Jiffin Thottan" <jthottan at redhat.com>, "gluster-infra" <gluster-infra at gluster.org>
Sent: Tuesday, April 23, 2019 7:44:49 PM
Subject: Re: [Gluster-infra] [Gluster-devel] is_nfs_export_available from nfs.rc failing too often?

Le lundi 22 avril 2019 ? 22:57 +0530, Atin Mukherjee a ?crit :
> Is this back again? The recent patches are failing regression :-\ .

So, on builder206, it took me a while to find that the issue is that
nfs (the service) was running.

./tests/basic/afr/tarissue.t failed, because the nfs initialisation
failed with a rather cryptic message:

[2019-04-23 13:17:05.371733] I [socket.c:991:__socket_server_bind] 0-
socket.nfs-server: process started listening on port (38465)
[2019-04-23 13:17:05.385819] E [socket.c:972:__socket_server_bind] 0-
socket.nfs-server: binding to  failed: Address already in use
[2019-04-23 13:17:05.385843] E [socket.c:974:__socket_server_bind] 0-
socket.nfs-server: Port is already in use
[2019-04-23 13:17:05.385852] E [socket.c:3788:socket_listen] 0-
socket.nfs-server: __socket_server_bind failed;closing socket 14

I found where this came from, but a few stuff did surprised me:

- the order of print is different that the order in the code
- the message on "started listening" didn't take in account the fact
that bind failed on:


https://github.com/gluster/glusterfs/blob/master/rpc/rpc-transport/socket/src/socket.c#L967

The message about port 38465 also threw me off the track. The real
issue is that the service nfs was already running, and I couldn't find
anything listening on port 38465

once I do service nfs stop, it no longer failed.

So far, I do know why nfs.service was activated.

But at least, 206 should be fixed, and we know a bit more on what would
be causing some failure.

 

> On Wed, 3 Apr 2019 at 19:26, Michael Scherer <mscherer at redhat.com>
> wrote:
> 
> > Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> > > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <
> > > jthottan at redhat.com>
> > > wrote:
> > > 
> > > > Hi,
> > > > 
> > > > is_nfs_export_available is just a wrapper around "showmount"
> > > > command AFAIR.
> > > > I saw following messages in console output.
> > > >  mount.nfs: rpc.statd is not running but is required for remote
> > > > locking.
> > > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local,
> > > > or
> > > > start
> > > > statd.
> > > > 05:06:55 mount.nfs: an incorrect mount option was specified
> > > > 
> > > > For me it looks rpcbind may not be running on the machine.
> > > > Usually rpcbind starts automatically on machines, don't know
> > > > whether it
> > > > can happen or not.
> > > > 
> > > 
> > > That's precisely what the question is. Why suddenly we're seeing
> > > this
> > > happening too frequently. Today I saw atleast 4 to 5 such
> > > failures
> > > already.
> > > 
> > > Deepshika - Can you please help in inspecting this?
> > 
> > So we think (we are not sure) that the issue is a bit complex.
> > 
> > What we were investigating was nightly run fail on aws. When the
> > build
> > crash, the builder is restarted, since that's the easiest way to
> > clean
> > everything (since even with a perfect test suite that would clean
> > itself, we could always end in a corrupt state on the system, WRT
> > mount, fs, etc).
> > 
> > In turn, this seems to cause trouble on aws, since cloud-init or
> > something rename eth0 interface to ens5, without cleaning to the
> > network configuration.
> > 
> > So the network init script fail (because the image say "start eth0"
> > and
> > that's not present), but fail in a weird way. Network is
> > initialised
> > and working (we can connect), but the dhclient process is not in
> > the
> > right cgroup, and network.service is in failed state. Restarting
> > network didn't work. In turn, this mean that rpc-statd refuse to
> > start
> > (due to systemd dependencies), which seems to impact various NFS
> > tests.
> > 
> > We have also seen that on some builders, rpcbind pick some IP v6
> > autoconfiguration, but we can't reproduce that, and there is no ip
> > v6
> > set up anywhere. I suspect the network.service failure is somehow
> > involved, but fail to see how. In turn, rpcbind.socket not starting
> > could cause NFS test troubles.
> > 
> > Our current stop gap fix was to fix all the builders one by one.
> > Remove
> > the config, kill the rogue dhclient, restart network service.
> > 
> > However, we can't be sure this is going to fix the problem long
> > term
> > since this only manifest after a crash of the test suite, and it
> > doesn't happen so often. (plus, it was working before some day in
> > the
> > past, when something did make this fail, and I do not know if
> > that's a
> > system upgrade, or a test change, or both).
> > 
> > So we are still looking at it to have a complete understanding of
> > the
> > issue, but so far, we hacked our way to make it work (or so do I
> > think).
> > 
> > Deepshika is working to fix it long term, by fixing the issue
> > regarding
> > eth0/ens5 with a new base image.
> > --
> > Michael Scherer
> > Sysadmin, Community Infrastructure and Platform, OSAS
> > 
> > 
> > --
> 
> - Atin (atinm)
-- 
Michael Scherer
Sysadmin, Community Infrastructure




From ndevos at redhat.com  Wed Apr 24 07:44:58 2019
From: ndevos at redhat.com (Niels de Vos)
Date: Wed, 24 Apr 2019 09:44:58 +0200
Subject: [Gluster-devel] [Gluster-Maintainers] BZ updates
In-Reply-To: <CAOUCJ=i9EN9OoUp4jeZsU_GV8r6=MPsmN+pfcEXJQgg1pRX76w@mail.gmail.com>
References: <CAOUCJ=i9EN9OoUp4jeZsU_GV8r6=MPsmN+pfcEXJQgg1pRX76w@mail.gmail.com>
Message-ID: <20190424074458.GG14100@ndevos-x270>

On Wed, Apr 24, 2019 at 08:44:10AM +0530, Nithya Balachandran wrote:
> All,
> 
> When working on a bug, please ensure that you update the BZ with any
> relevant information as well as the RCA. I have seen several BZs in the
> past which report crashes, however they do not have a bt or RCA captured.
> Having this information in the BZ makes it much easier to see if a newly
> reported issue has already been fixed.
> 
> I propose that maintainers merge patches only if the BZs are updated with
> required information. It will take some time to make this a habit but it
> will pay off in the end.

Great point! I really hope that most of the contributors know that
debugging steps in bugs are extremely valuable. When documented in a
bug, similar issues can be analyzed with the same techniques. As a
reminder for this, I'm proposing this addition to the Maintainer
Guidelines:

   https://github.com/gluster/glusterdocs/pull/471
   - Ensure the related Bug or GitHub Issue has sufficient details about the
     cause of the problem, or description of the introduction for the change.

I'd appreciate it when someone can approve and merge that. Of course,
suggestions for rephrasing are welcome too.

Niels

From ykaul at redhat.com  Wed Apr 24 11:59:11 2019
From: ykaul at redhat.com (Yaniv Kaul)
Date: Wed, 24 Apr 2019 14:59:11 +0300
Subject: [Gluster-devel] [Gluster-infra] is_nfs_export_available from
 nfs.rc failing too often?
In-Reply-To: <b89a617416e1f4b64847b8d71b3916be8744ff0b.camel@redhat.com>
References: <CAGNCGH3kQ-6v86FJs8L5bkRw9xfCs6ZPjaoaRVKp1+E-cf+Ybg@mail.gmail.com>
	<2056284426.17636953.1554272780313.JavaMail.zimbra@redhat.com>
	<CAGNCGH2QgF=qP19Brs4YVVDdaamtDm2QBT35ty01s+KbRACYFQ@mail.gmail.com>
	<797512f6ff7f1b9fedbf8b7968dd86a6968d9105.camel@redhat.com>
	<CAGNCGH28QdMgpSTqb6aH8Vo4kDOp3PF4fkow3BT2MF2_FBf+Lg@mail.gmail.com>
	<b89a617416e1f4b64847b8d71b3916be8744ff0b.camel@redhat.com>
Message-ID: <CAJgorsYN2gM+v=o45yzZL5CmcO9XyQedimV1KAabmo8wx9zK0w@mail.gmail.com>

On Tue, Apr 23, 2019 at 5:15 PM Michael Scherer <mscherer at redhat.com> wrote:

> Le lundi 22 avril 2019 ? 22:57 +0530, Atin Mukherjee a ?crit :
> > Is this back again? The recent patches are failing regression :-\ .
>
> So, on builder206, it took me a while to find that the issue is that
> nfs (the service) was running.
>
> ./tests/basic/afr/tarissue.t failed, because the nfs initialisation
> failed with a rather cryptic message:
>
> [2019-04-23 13:17:05.371733] I [socket.c:991:__socket_server_bind] 0-
> socket.nfs-server: process started listening on port (38465)
> [2019-04-23 13:17:05.385819] E [socket.c:972:__socket_server_bind] 0-
> socket.nfs-server: binding to  failed: Address already in use
> [2019-04-23 13:17:05.385843] E [socket.c:974:__socket_server_bind] 0-
> socket.nfs-server: Port is already in use
> [2019-04-23 13:17:05.385852] E [socket.c:3788:socket_listen] 0-
> socket.nfs-server: __socket_server_bind failed;closing socket 14
>
> I found where this came from, but a few stuff did surprised me:
>
> - the order of print is different that the order in the code
>

Indeed strange...

> - the message on "started listening" didn't take in account the fact
> that bind failed on:
>

Shouldn't it bail out if it failed to bind?
Some missing 'goto out' around line 975/976?
Y.

>
>
>
> https://github.com/gluster/glusterfs/blob/master/rpc/rpc-transport/socket/src/socket.c#L967
>
> The message about port 38465 also threw me off the track. The real
> issue is that the service nfs was already running, and I couldn't find
> anything listening on port 38465
>
> once I do service nfs stop, it no longer failed.
>
> So far, I do know why nfs.service was activated.
>
> But at least, 206 should be fixed, and we know a bit more on what would
> be causing some failure.
>
>
>
> > On Wed, 3 Apr 2019 at 19:26, Michael Scherer <mscherer at redhat.com>
> > wrote:
> >
> > > Le mercredi 03 avril 2019 ? 16:30 +0530, Atin Mukherjee a ?crit :
> > > > On Wed, Apr 3, 2019 at 11:56 AM Jiffin Thottan <
> > > > jthottan at redhat.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > is_nfs_export_available is just a wrapper around "showmount"
> > > > > command AFAIR.
> > > > > I saw following messages in console output.
> > > > >  mount.nfs: rpc.statd is not running but is required for remote
> > > > > locking.
> > > > > 05:06:55 mount.nfs: Either use '-o nolock' to keep locks local,
> > > > > or
> > > > > start
> > > > > statd.
> > > > > 05:06:55 mount.nfs: an incorrect mount option was specified
> > > > >
> > > > > For me it looks rpcbind may not be running on the machine.
> > > > > Usually rpcbind starts automatically on machines, don't know
> > > > > whether it
> > > > > can happen or not.
> > > > >
> > > >
> > > > That's precisely what the question is. Why suddenly we're seeing
> > > > this
> > > > happening too frequently. Today I saw atleast 4 to 5 such
> > > > failures
> > > > already.
> > > >
> > > > Deepshika - Can you please help in inspecting this?
> > >
> > > So we think (we are not sure) that the issue is a bit complex.
> > >
> > > What we were investigating was nightly run fail on aws. When the
> > > build
> > > crash, the builder is restarted, since that's the easiest way to
> > > clean
> > > everything (since even with a perfect test suite that would clean
> > > itself, we could always end in a corrupt state on the system, WRT
> > > mount, fs, etc).
> > >
> > > In turn, this seems to cause trouble on aws, since cloud-init or
> > > something rename eth0 interface to ens5, without cleaning to the
> > > network configuration.
> > >
> > > So the network init script fail (because the image say "start eth0"
> > > and
> > > that's not present), but fail in a weird way. Network is
> > > initialised
> > > and working (we can connect), but the dhclient process is not in
> > > the
> > > right cgroup, and network.service is in failed state. Restarting
> > > network didn't work. In turn, this mean that rpc-statd refuse to
> > > start
> > > (due to systemd dependencies), which seems to impact various NFS
> > > tests.
> > >
> > > We have also seen that on some builders, rpcbind pick some IP v6
> > > autoconfiguration, but we can't reproduce that, and there is no ip
> > > v6
> > > set up anywhere. I suspect the network.service failure is somehow
> > > involved, but fail to see how. In turn, rpcbind.socket not starting
> > > could cause NFS test troubles.
> > >
> > > Our current stop gap fix was to fix all the builders one by one.
> > > Remove
> > > the config, kill the rogue dhclient, restart network service.
> > >
> > > However, we can't be sure this is going to fix the problem long
> > > term
> > > since this only manifest after a crash of the test suite, and it
> > > doesn't happen so often. (plus, it was working before some day in
> > > the
> > > past, when something did make this fail, and I do not know if
> > > that's a
> > > system upgrade, or a test change, or both).
> > >
> > > So we are still looking at it to have a complete understanding of
> > > the
> > > issue, but so far, we hacked our way to make it work (or so do I
> > > think).
> > >
> > > Deepshika is working to fix it long term, by fixing the issue
> > > regarding
> > > eth0/ens5 with a new base image.
> > > --
> > > Michael Scherer
> > > Sysadmin, Community Infrastructure and Platform, OSAS
> > >
> > >
> > > --
> >
> > - Atin (atinm)
> --
> Michael Scherer
> Sysadmin, Community Infrastructure
>
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190424/833a9bb8/attachment-0001.html>

From rgowdapp at redhat.com  Thu Apr 25 06:07:23 2019
From: rgowdapp at redhat.com (Raghavendra Gowdappa)
Date: Thu, 25 Apr 2019 11:37:23 +0530
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
	<32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
	<CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>
	<a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>
Message-ID: <CAFkORY-HetB2i-dEYdkCt2ovF3KFybD2iARvcs9BZMfE+H94eg@mail.gmail.com>

On Mon, Apr 15, 2019 at 12:52 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:

> Hi,
>
> The reason why I move event_handled to the end of socket_event_poll_in is
> because if event_handled is called before rpc_transport_pollin_destroy, it
> allowed another round of event_dispatch_epoll_handler happen before
>  rpc_transport_pollin_destroy, in this way, when the latter poll in goes to
> rpc_transport_pollin_destroy , there is a chance that the pollin->iobref
> has already been destroyed by the first one(there is no lock destroy for
> iobref->lock in iobref_destroy by the way). That may cause stuck in ?LOCK
> (&iobref->lock);?
>

But, priv->incoming.iobref (from which pollin->iobref is initialized from)
is set to NULL in __socket_proto_state_machine:

    if (in->record_state == SP_STATE_COMPLETE) {
                in->record_state = SP_STATE_NADA;
        __socket_reset_priv (priv);
        }

And since pollin is an allocated object only one instance of
socket_event_poll_in will be aware of this object. IOW, multiple instances
of socket_event_poll_in will get different pollin objects. So, the only way
pollin->iobref could be shared across multiple invocations of
socket_event_poll_in is due to common shared object priv->incoming.iobref.
But that too is sanitized by the time __socket_proto_state_machine
completes and __socket_proto_state_machine is executed under lock. So, I
don't see how two different concurrent codepaths can get hold of same
iobref.

I find the one of recent patch
>
SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
> I think the point is to avoid the concurrent handling of the same socket
> at the same time, but after my test with this patch this problem also
> exists, so I think event_handled is still called too early to allow
> concurrent handling of the same socket happen, and after move it to the end
> of socket_event_poll this glusterd stuck issue disappeared.
>
> cynthia
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Monday, April 15, 2019 2:36 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
>
>
>
>
> On Mon, Apr 15, 2019 at 11:08 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Ok, thanks for your comment!
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Monday, April 15, 2019 11:52 AM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Cynthia,
>
>
>
> On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi,
>
> I made a patch and according to my test, this glusterd stuck issue
> disappear with my patch. Only need to move  event_handled to the end of
> socket_event_poll_in function.
>
>
>
> --- a/rpc/rpc-transport/socket/src/socket.c
>
> +++ b/rpc/rpc-transport/socket/src/socket.c
>
> @@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>          }
>
>
>
> -        if (notify_handled && (ret != -1))
>
> -                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> -                               priv->gen);
>
> @@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this,
> gf_boolean_t notify_handled)
>
>                  }
>
>                  pthread_mutex_unlock (&priv->notify.lock);
>
>          }
>
> +        if (notify_handled && (ret != -1))
>
> +                event_handled (ctx->event_pool, priv->sock, priv->idx,
>
> +                               priv->gen);
>
>
>
> Thanks for this tip. Though this helps in fixing the hang, this change has
> performance impact. Moving event_handled to end of poll_in means, socket
> will be added back for polling of new events only _after_ the rpc is msg is
> processed by higher layers (like EC) and higher layers can have significant
> latency  for processing the msg. Which means, socket will be out of polling
> for longer periods of time which decreases the throughput (number of msgs
> read per second) affecting performance. However, this experiment definitely
> indicates there is a codepath where event_handled is not called (and hence
> causing the hang). I'll go through this codepath again.
>
>
>
> Can you check whether patch [1] fixes the issue you are seeing?
>
>
>
> [1] https://review.gluster.org/#/c/glusterfs/+/22566
>
>
>
>
>
> Thanks for that experiment :).
>
>
>
>          return ret;
>
> }
>
>
>
> cynthia
>
> *From:* Zhou, Cynthia (NSB - CN/Hangzhou)
> *Sent:* Tuesday, April 09, 2019 3:57 PM
> *To:* 'Raghavendra Gowdappa' <rgowdapp at redhat.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* RE: glusterd stuck for glusterfs with version 3.12.15
>
>
>
> Can you figure out some possible reason why iobref is corrupted, is it
> possible that thread 8 has called poll in and iobref has been relased, but
> the lock within it is not properly released (as I can not find any free
> lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy
> again, and so stuck on this lock
>
> Also, there should not be two thread handling the same socket at the same
> time, although there has been a patch claimed to tackle this issue.
>
>
>
> cynthia
>
>
>
> *From:* Raghavendra Gowdappa <rgowdapp at redhat.com>
> *Sent:* Tuesday, April 09, 2019 3:52 PM
> *To:* Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
> *Cc:* gluster-devel at gluster.org
> *Subject:* Re: glusterd stuck for glusterfs with version 3.12.15
>
>
>
>
>
>
>
> On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <
> cynthia.zhou at nokia-sbell.com> wrote:
>
> Hi glusterfs experts,
>
> Good day!
>
> In my test env, sometimes glusterd stuck issue happened, and it is not
> responding to any gluster commands, when I checked this issue I find that
> glusterd thread 9 and thread 8 is dealing with the same socket, I thought
> following patch should be able to solve this issue, however after I merged
> this patch this issue still exist. When I looked into this code, it seems
> socket_event_poll_in called event_handled before
> rpc_transport_pollin_destroy, I think this gives the chance for another
> poll for the exactly the same socket. And caused this glusterd stuck issue,
> also, I find there is no   LOCK_DESTROY(&iobref->lock)
>
> In iobref_destroy, I think it is better to add destroy lock.
>
> Following is the gdb info when this issue happened, I would like to know
> your opinion on this issue, thanks!
>
>
>
> SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537
>
>
>
> ** socket: fix issue on concurrent handle of a socket*
>
>
>
>
>
>
>
> *GDB INFO:*
>
> Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in
> iobref_unref, I think
>
> Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from
> /lib64/libpthread.so.0
>
> #2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib64/libpthread.so.0
>
> #3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0,
> gen=4, idx=27) at socket.c:1201
>
> #4  0x00007f9ee4fbf99c in socket_event_handler (*fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0*) at socket.c:2480
>
> #5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
>
> #6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at
> event-epoll.c:659
>
> #7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
>
>
> (gdb) thread 9
>
> [Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> (gdb) bt
>
> #0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
>
> #1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> #3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy
> (pollin=0x7f9ed00452d0) at rpc-transport.c:123
>
> #4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0,
> notify_handled=_gf_true) at socket.c:2322
>
> #5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4,
> data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
>
> #6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler
> (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
>
> #7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at
> event-epoll.c:659
>
> #8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
>
> #9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
>
> (gdb) frame 2
>
> #2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at
> iobuf.c:944
>
> 944 iobuf.c: No such file or directory.
>
> (gdb) print *iobref
>
> $1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222,
> __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
>
>         __elision = 0, __list = {__prev = 0x4000, __next =
> 0x7f9ed00063b000}},
>
>       __size =
> "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@
> \000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}},
> ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}
>
>
>
> looks like the iobref is corrupted here. It seems to be a use-after-free
> issue. We need to dig into why a freed iobref is being accessed here.
>
>
>
> (gdb) quit
>
> A debugging session is active.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190425/c20f40e6/attachment-0001.html>

From atumball at redhat.com  Fri Apr 26 07:54:00 2019
From: atumball at redhat.com (Amar Tumballi Suryanarayan)
Date: Fri, 26 Apr 2019 13:24:00 +0530
Subject: [Gluster-devel] One more way to contact Gluster team - Slack
	(gluster.slack.com)
Message-ID: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>

Hi All,

We wanted to move to Slack from IRC for our official communication channel
from sometime, but couldn't as we didn't had a proper URL for us to
register. 'gluster' was taken and we didn't knew who had it registered.
Thanks to constant ask from Satish, Slack team has now agreed to let us use
https://gluster.slack.com and I am happy to invite you all there. (Use this
link
<https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA>
to
join)

Please note that, it won't be a replacement for mailing list. But can be
used by all developers and users for quick communication. Also note that,
no information there would be 'stored' beyond 10k lines as we are using the
free version of Slack.

Regards,
Amar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/0fa216a9/attachment.html>

From mscherer at redhat.com  Fri Apr 26 08:16:39 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Fri, 26 Apr 2019 10:16:39 +0200
Subject: [Gluster-devel] One more way to contact Gluster team - Slack
 (gluster.slack.com)
In-Reply-To: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
Message-ID: <e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>

Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan a
?crit :
> Hi All,
> 
> We wanted to move to Slack from IRC for our official communication
> channel
> from sometime, but couldn't as we didn't had a proper URL for us to
> register. 'gluster' was taken and we didn't knew who had it
> registered.
> Thanks to constant ask from Satish, Slack team has now agreed to let
> us use
> https://gluster.slack.com and I am happy to invite you all there.
> (Use this
> link
> <
> https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
> >
> to
> join)
> 
> Please note that, it won't be a replacement for mailing list. But can
> be
> used by all developers and users for quick communication. Also note
> that,
> no information there would be 'stored' beyond 10k lines as we are
> using the
> free version of Slack.

Aren't we concerned about the ToS of slack ? Last time I did read them,
they were quite scary (like, if you use your corporate email, you
engage your employer, and that wasn't the worst part). 

Also, to anticipate the question, my employer Legal department told me
to not setup a bridge between IRC and slack, due to the said ToS.
 
-- 
Michael Scherer
Sysadmin, Community Infrastructure



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/a8cf6748/attachment.sig>

From kkeithle at redhat.com  Fri Apr 26 12:57:43 2019
From: kkeithle at redhat.com (Kaleb Keithley)
Date: Fri, 26 Apr 2019 08:57:43 -0400
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <CAJMr+NnBfNAfxASXO3U+2AWw0kWOvo8KwY5sH=MGo=cbH_jSMg@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
	<CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
	<CAJMr+NnBfNAfxASXO3U+2AWw0kWOvo8KwY5sH=MGo=cbH_jSMg@mail.gmail.com>
Message-ID: <CAC+Jd5Aur44xSgCe0mad5KhG8G4487dOGvzD63O3kPJ+8p_rSw@mail.gmail.com>

On Fri, Apr 26, 2019 at 8:21 AM Harold Miller <harold at redhat.com> wrote:

> Has Red Hat security cleared the Slack systems for confidential / customer
> information?
>
> If not, it will make it difficult for support to collect/answer questions.
>

I'm pretty sure Amar meant as a replacement for the freenode #gluster and
#gluster-dev channels, given that he sent this to the public gluster
mailing lists @gluster.org. Nobody should have even been posting
confidential and/or customer information to any of those lists or channels.
And AFAIK nobody ever has.

Amar, would you like to clarify which IRC channels you meant?


> Harold Miller, Associate Manager,
> Red Hat, Enterprise Cloud Support
> Desk - US (650) 254-4346
>
>
>
> On Fri, Apr 26, 2019 at 6:00 AM Scott Worthington <
> scott.c.worthington at gmail.com> wrote:
>
>> Hello, are you not _BOTH_ Red Hat FTEs or contractors?
>>
>> On Fri, Apr 26, 2019, 3:16 AM Michael Scherer <mscherer at redhat.com>
>> wrote:
>>
>>> Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan a
>>> ?crit :
>>> > Hi All,
>>> >
>>> > We wanted to move to Slack from IRC for our official communication
>>> > channel
>>> > from sometime, but couldn't as we didn't had a proper URL for us to
>>> > register. 'gluster' was taken and we didn't knew who had it
>>> > registered.
>>> > Thanks to constant ask from Satish, Slack team has now agreed to let
>>> > us use
>>> > https://gluster.slack.com and I am happy to invite you all there.
>>> > (Use this
>>> > link
>>> > <
>>> >
>>> https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
>>> > >
>>> > to
>>> > join)
>>> >
>>> > Please note that, it won't be a replacement for mailing list. But can
>>> > be
>>> > used by all developers and users for quick communication. Also note
>>> > that,
>>> > no information there would be 'stored' beyond 10k lines as we are
>>> > using the
>>> > free version of Slack.
>>>
>>> Aren't we concerned about the ToS of slack ? Last time I did read them,
>>> they were quite scary (like, if you use your corporate email, you
>>> engage your employer, and that wasn't the worst part).
>>>
>>> Also, to anticipate the question, my employer Legal department told me
>>> to not setup a bridge between IRC and slack, due to the said ToS.
>>>
>>> --
>>> Michael Scherer
>>> Sysadmin, Community Infrastructure
>>>
>>>
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
> --
>
> HAROLD MILLER
>
> ASSOCIATE MANAGER, ENTERPRISE CLOUD SUPPORT
>
> Red Hat
>
> <https://www.redhat.com/>
>
> Harold at RedHat.com    T: (650)-254-4346
> <https://red.ht/sig>
> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/7c1699ed/attachment.html>

From atumball at redhat.com  Fri Apr 26 13:20:09 2019
From: atumball at redhat.com (Amar Tumballi Suryanarayan)
Date: Fri, 26 Apr 2019 18:50:09 +0530
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <CAC+Jd5Aur44xSgCe0mad5KhG8G4487dOGvzD63O3kPJ+8p_rSw@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
	<CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
	<CAJMr+NnBfNAfxASXO3U+2AWw0kWOvo8KwY5sH=MGo=cbH_jSMg@mail.gmail.com>
	<CAC+Jd5Aur44xSgCe0mad5KhG8G4487dOGvzD63O3kPJ+8p_rSw@mail.gmail.com>
Message-ID: <CAHxyDdNJkfdSwbr8ixgHynO=j73S+7iNp6suBtKRMGdQWNW==w@mail.gmail.com>

On Fri, Apr 26, 2019 at 6:27 PM Kaleb Keithley <kkeithle at redhat.com> wrote:

>
>
> On Fri, Apr 26, 2019 at 8:21 AM Harold Miller <harold at redhat.com> wrote:
>
>> Has Red Hat security cleared the Slack systems for confidential /
>> customer information?
>>
>> If not, it will make it difficult for support to collect/answer questions.
>>
>
> I'm pretty sure Amar meant as a replacement for the freenode #gluster and
> #gluster-dev channels, given that he sent this to the public gluster
> mailing lists @gluster.org. Nobody should have even been posting
> confidential and/or customer information to any of those lists or channels.
> And AFAIK nobody ever has.
>
>
Yep, I am only talking about IRC (from freenode, #gluster, #gluster-dev
etc).  Also, I am not saying we are 'replacing IRC'. Gluster as a project
started in pre-Slack era, and we have many users who prefer to stay in IRC.
So, for now, no pressure to make a statement calling Slack channel as a
'Replacement' to IRC.


> Amar, would you like to clarify which IRC channels you meant?
>
>

Thanks Kaleb. I was bit confused on why the concern of it came up in this
group.



>
>> On Fri, Apr 26, 2019 at 6:00 AM Scott Worthington <
>> scott.c.worthington at gmail.com> wrote:
>>
>>> Hello, are you not _BOTH_ Red Hat FTEs or contractors?
>>>
>>>
Yes! but come from very different internal teams.

Michael supports Gluster (the project) team's Infrastructure needs, and has
valid concerns from his perspective :-) I, on the other hand, bother more
about code, users, and how to make sure we are up-to-date with other
technologies and communities, from the engineering view point.


> On Fri, Apr 26, 2019, 3:16 AM Michael Scherer <mscherer at redhat.com> wrote:
>>>
>>>> Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan a
>>>> ?crit :
>>>> > Hi All,
>>>> >
>>>> > We wanted to move to Slack from IRC for our official communication
>>>> > channel
>>>> > from sometime, but couldn't as we didn't had a proper URL for us to
>>>> > register. 'gluster' was taken and we didn't knew who had it
>>>> > registered.
>>>> > Thanks to constant ask from Satish, Slack team has now agreed to let
>>>> > us use
>>>> > https://gluster.slack.com and I am happy to invite you all there.
>>>> > (Use this
>>>> > link
>>>> > <
>>>> >
>>>> https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
>>>> > >
>>>> > to
>>>> > join)
>>>> >
>>>> > Please note that, it won't be a replacement for mailing list. But can
>>>> > be
>>>> > used by all developers and users for quick communication. Also note
>>>> > that,
>>>> > no information there would be 'stored' beyond 10k lines as we are
>>>> > using the
>>>> > free version of Slack.
>>>>
>>>> Aren't we concerned about the ToS of slack ? Last time I did read them,
>>>> they were quite scary (like, if you use your corporate email, you
>>>> engage your employer, and that wasn't the worst part).
>>>>
>>>> Also, to anticipate the question, my employer Legal department told me
>>>> to not setup a bridge between IRC and slack, due to the said ToS.
>>>>
>>>>
Again, re-iterating here. Not planning to use any bridges from IRC to
Slack. I re-read the Slack API Terms and condition. And it makes sense.
They surely don't want us to build another slack, or abuse slack with too
many API requests made for collecting logs.

Currently, to start with, we are not adding any bots (other than github
bot). Hopefully, that will keep us under proper usage guidelines.

-Amar


> --
>>>> Michael Scherer
>>>> Sysadmin, Community Infrastructure
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>> --
>>
>> HAROLD MILLER
>>
>> ASSOCIATE MANAGER, ENTERPRISE CLOUD SUPPORT
>>
>> Red Hat
>>
>> <https://www.redhat.com/>
>>
>> Harold at RedHat.com    T: (650)-254-4346
>> <https://red.ht/sig>
>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>

-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/2b93b87b/attachment-0001.html>

From mscherer at redhat.com  Fri Apr 26 14:43:53 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Fri, 26 Apr 2019 16:43:53 +0200
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
	<CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
Message-ID: <52de190db15f5c1225029b6785f266d72751657e.camel@redhat.com>

Le vendredi 26 avril 2019 ? 04:59 -0500, Scott Worthington a ?crit :
> Hello, are you not _BOTH_ Red Hat FTEs or contractors?

We do, that was just a turn of phrase to be more generic.

> On Fri, Apr 26, 2019, 3:16 AM Michael Scherer <mscherer at redhat.com>
> wrote:
> 
> > Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan
> > a
> > ?crit :
> > > Hi All,
> > > 
> > > We wanted to move to Slack from IRC for our official
> > > communication
> > > channel
> > > from sometime, but couldn't as we didn't had a proper URL for us
> > > to
> > > register. 'gluster' was taken and we didn't knew who had it
> > > registered.
> > > Thanks to constant ask from Satish, Slack team has now agreed to
> > > let
> > > us use
> > > https://gluster.slack.com and I am happy to invite you all there.
> > > (Use this
> > > link
> > > <
> > > 
> > 
> > 
https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
> > > > 
> > > 
> > > to
> > > join)
> > > 
> > > Please note that, it won't be a replacement for mailing list. But
> > > can
> > > be
> > > used by all developers and users for quick communication. Also
> > > note
> > > that,
> > > no information there would be 'stored' beyond 10k lines as we are
> > > using the
> > > free version of Slack.
> > 
> > Aren't we concerned about the ToS of slack ? Last time I did read
> > them,
> > they were quite scary (like, if you use your corporate email, you
> > engage your employer, and that wasn't the worst part).
> > 
> > Also, to anticipate the question, my employer Legal department told
> > me
> > to not setup a bridge between IRC and slack, due to the said ToS.
> > 
> > --
> > Michael Scherer
> > Sysadmin, Community Infrastructure
> > 
> > 
> > 
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > https://lists.gluster.org/mailman/listinfo/gluster-users
-- 
Michael Scherer
Sysadmin, Community Infrastructure



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/9b8d1ea2/attachment.sig>

From junsongli at fb.com  Fri Apr 26 17:23:35 2019
From: junsongli at fb.com (Junsong Li)
Date: Fri, 26 Apr 2019 17:23:35 +0000
Subject: [Gluster-devel] questions on callstubs and "link-count" in index
	translator
Message-ID: <BYAPR15MB349599B0BB55B900D8194D37D03D0@BYAPR15MB3495.namprd15.prod.outlook.com>

Hello list,

I have a couple of questions on index translator implementation.

  *   Why does gluster need callstub and a different worker queue (and thread) to process those call stubs? Is it just to lower the priority of fops of internal inodes?
  *   What?s the purpose of ?link-count? in xdata? It?s being used only in index_fstat and index_lookup. I see sometimes the key is assigned 0/1 after callback, and sometimes AFR uses it to store flag GF_XATTROP_INDEX_COUNT. Is the code purposely reusing the key?
Thanks,
Junsong
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/273fad80/attachment.html>

From ravishankar at redhat.com  Sat Apr 27 06:20:01 2019
From: ravishankar at redhat.com (Ravishankar N)
Date: Sat, 27 Apr 2019 11:50:01 +0530
Subject: [Gluster-devel] questions on callstubs and "link-count" in
 index translator
In-Reply-To: <BYAPR15MB349599B0BB55B900D8194D37D03D0@BYAPR15MB3495.namprd15.prod.outlook.com>
References: <BYAPR15MB349599B0BB55B900D8194D37D03D0@BYAPR15MB3495.namprd15.prod.outlook.com>
Message-ID: <59a8ac38-4ae5-06e1-4b0d-e3e8efba16eb@redhat.com>


On 26/04/19 10:53 PM, Junsong Li wrote:
>
> Hello list,
>
> I have a couple of questions on index translator implementation.
>
>   * Why does gluster need callstub and a different worker queue (and
>     thread) to process those call stubs? Is it just to lower the
>     priority of fops of internal inodes?
>
As far as I know, this is to move the processing to background and free 
up the server thread to process other requests.
>
>  *
>
>
>   * What?s the purpose of ?link-count? in xdata? It?s being used only
>     in index_fstat and index_lookup. I see sometimes the key is
>     assigned 0/1 after callback, and sometimes AFR uses it to store
>     flag GF_XATTROP_INDEX_COUNT. Is the code purposely reusing the key?
>
A non-zero link count means there are entries that are pending heal. AFR 
requests this information in lookup and fstat fops and updates 
priv->need_heal in the fop-callbacks. It then uses that information to 
not nullify the inodes of the entries fetched during a readdirp call, 
improving readdirp performance.

https://review.gluster.org/#/c/glusterfs/+/12507/ is the patch that 
introduced it.

HTH,
Ravi
>
>  *
>
> Thanks,
>
> Junsong
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190427/1cbc0570/attachment-0001.html>

From cynthia.zhou at nokia-sbell.com  Sun Apr 28 02:12:10 2019
From: cynthia.zhou at nokia-sbell.com (Zhou, Cynthia (NSB - CN/Hangzhou))
Date: Sun, 28 Apr 2019 02:12:10 +0000
Subject: [Gluster-devel] glusterd stuck for glusterfs with version
	3.12.15
In-Reply-To: <CAFkORY-HetB2i-dEYdkCt2ovF3KFybD2iARvcs9BZMfE+H94eg@mail.gmail.com>
References: <2c963e64775f4a35b43d651906ce30ef@nokia-sbell.com>
	<CAFkORY-t9XrPhtzG8EQA+xtpM2_vcMj23VAUdq7RViXp5PuUqA@mail.gmail.com>
	<ab5b615eba2643d483cf1133b6a9f2e1@nokia-sbell.com>
	<6ec3136489ac4d119e43ec24256c1240@nokia-sbell.com>
	<CAFkORY-JoomM3OLupATLiSB6yEQDSdpG9c_pNqdP-F8kgKpHFg@mail.gmail.com>
	<32f4f1f20d344a0285fd7c17647b879e@nokia-sbell.com>
	<CAFkORY88w_byrf6qSohGQuHqMiBhQa_eoh=jtiAJ7wQwVcP4qA@mail.gmail.com>
	<a56f7c148d044bd0a876df754fc3755c@nokia-sbell.com>
	<CAFkORY-HetB2i-dEYdkCt2ovF3KFybD2iARvcs9BZMfE+H94eg@mail.gmail.com>
Message-ID: <22176e9b4f1746beb550f6b922e8940e@nokia-sbell.com>

Hi,
Ok, I see your point ,two rounds of pollin means two different iobref, so the first pollin will not affect the second. But from the stack the second poll in stuck in iobref_unref LOCK operation,
Do you think it is possible that the iobref_destroy does not deal with this iobref->lock will cause it to be some unsanitized value and cause this stuck?

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com>
Sent: Thursday, April 25, 2019 2:07 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com>
Cc: gluster-devel at gluster.org
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 15, 2019 at 12:52 PM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
The reason why I move event_handled to the end of socket_event_poll_in is because if event_handled is called before rpc_transport_pollin_destroy, it allowed another round of event_dispatch_epoll_handler happen before  rpc_transport_pollin_destroy, in this way, when the latter poll in goes to rpc_transport_pollin_destroy , there is a chance that the pollin->iobref has already been destroyed by the first one(there is no lock destroy for iobref->lock in iobref_destroy by the way). That may cause stuck in ?LOCK (&iobref->lock);?

But, priv->incoming.iobref (from which pollin->iobref is initialized from) is set to NULL in __socket_proto_state_machine:

    if (in->record_state == SP_STATE_COMPLETE) {
                in->record_state = SP_STATE_NADA;
        __socket_reset_priv (priv);
        }

And since pollin is an allocated object only one instance of socket_event_poll_in will be aware of this object. IOW, multiple instances of socket_event_poll_in will get different pollin objects. So, the only way pollin->iobref could be shared across multiple invocations of socket_event_poll_in is due to common shared object priv->incoming.iobref. But that too is sanitized by the time __socket_proto_state_machine completes and __socket_proto_state_machine is executed under lock. So, I don't see how two different concurrent codepaths can get hold of same iobref.

I find the one of recent patch
SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket

I think the point is to avoid the concurrent handling of the same socket at the same time, but after my test with this patch this problem also exists, so I think event_handled is still called too early to allow concurrent handling of the same socket happen, and after move it to the end of socket_event_poll this glusterd stuck issue disappeared.
cynthia
From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Monday, April 15, 2019 2:36 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 15, 2019 at 11:08 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Ok, thanks for your comment!

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Monday, April 15, 2019 11:52 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15

Cynthia,

On Mon, Apr 15, 2019 at 8:10 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi,
I made a patch and according to my test, this glusterd stuck issue disappear with my patch. Only need to move  event_handled to the end of socket_event_poll_in function.

--- a/rpc/rpc-transport/socket/src/socket.c
+++ b/rpc/rpc-transport/socket/src/socket.c
@@ -2305,9 +2305,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
         }

-        if (notify_handled && (ret != -1))
-                event_handled (ctx->event_pool, priv->sock, priv->idx,
-                               priv->gen);
@@ -2330,6 +2327,9 @@ socket_event_poll_in (rpc_transport_t *this, gf_boolean_t notify_handled)
                 }
                 pthread_mutex_unlock (&priv->notify.lock);
         }
+        if (notify_handled && (ret != -1))
+                event_handled (ctx->event_pool, priv->sock, priv->idx,
+                               priv->gen);

Thanks for this tip. Though this helps in fixing the hang, this change has performance impact. Moving event_handled to end of poll_in means, socket will be added back for polling of new events only _after_ the rpc is msg is processed by higher layers (like EC) and higher layers can have significant latency  for processing the msg. Which means, socket will be out of polling for longer periods of time which decreases the throughput (number of msgs read per second) affecting performance. However, this experiment definitely indicates there is a codepath where event_handled is not called (and hence causing the hang). I'll go through this codepath again.

Can you check whether patch [1] fixes the issue you are seeing?

[1] https://review.gluster.org/#/c/glusterfs/+/22566


Thanks for that experiment :).

         return ret;
}

cynthia
From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Tuesday, April 09, 2019 3:57 PM
To: 'Raghavendra Gowdappa' <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: RE: glusterd stuck for glusterfs with version 3.12.15

Can you figure out some possible reason why iobref is corrupted, is it possible that thread 8 has called poll in and iobref has been relased, but the lock within it is not properly released (as I can not find any free lock operation in iobref_destroy), then thread 9 called rpc_transport_pollin_destroy again, and so stuck on this lock
Also, there should not be two thread handling the same socket at the same time, although there has been a patch claimed to tackle this issue.

cynthia

From: Raghavendra Gowdappa <rgowdapp at redhat.com<mailto:rgowdapp at redhat.com>>
Sent: Tuesday, April 09, 2019 3:52 PM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>>
Cc: gluster-devel at gluster.org<mailto:gluster-devel at gluster.org>
Subject: Re: glusterd stuck for glusterfs with version 3.12.15



On Mon, Apr 8, 2019 at 7:42 AM Zhou, Cynthia (NSB - CN/Hangzhou) <cynthia.zhou at nokia-sbell.com<mailto:cynthia.zhou at nokia-sbell.com>> wrote:
Hi glusterfs experts,
Good day!
In my test env, sometimes glusterd stuck issue happened, and it is not responding to any gluster commands, when I checked this issue I find that glusterd thread 9 and thread 8 is dealing with the same socket, I thought following patch should be able to solve this issue, however after I merged this patch this issue still exist. When I looked into this code, it seems socket_event_poll_in called event_handled before rpc_transport_pollin_destroy, I think this gives the chance for another poll for the exactly the same socket. And caused this glusterd stuck issue, also, I find there is no   LOCK_DESTROY(&iobref->lock)
In iobref_destroy, I think it is better to add destroy lock.
Following is the gdb info when this issue happened, I would like to know your opinion on this issue, thanks!

SHA-1: f747d55a7fd364e2b9a74fe40360ab3cb7b11537

* socket: fix issue on concurrent handle of a socket



GDB INFO:
Thread 8 is blocked on pthread_cond_wait, and thread 9 is blocked in iobref_unref, I think
Thread 9 (Thread 0x7f9edf7fe700 (LWP 1933)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9edffff700 (LWP 1932)):
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fd2b42 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#2  0x00007f9ee9fd44a8 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#3  0x00007f9ee4fbadab in socket_event_poll_err (this=0x7f9ed0049cc0, gen=4, idx=27) at socket.c:1201
#4  0x00007f9ee4fbf99c in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2480
#5  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edfffee84) at event-epoll.c:583
#6  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180cf20) at event-epoll.c:659
#7  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#8  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6

(gdb) thread 9
[Switching to thread 9 (Thread 0x7f9edf7fe700 (LWP 1933))]
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
(gdb) bt
#0  0x00007f9ee9fd785c in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9ee9fda657 in __lll_lock_elision () from /lib64/libpthread.so.0
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
#3  0x00007f9eeafd2f29 in rpc_transport_pollin_destroy (pollin=0x7f9ed00452d0) at rpc-transport.c:123
#4  0x00007f9ee4fbf319 in socket_event_poll_in (this=0x7f9ed0049cc0, notify_handled=_gf_true) at socket.c:2322
#5  0x00007f9ee4fbf932 in socket_event_handler (fd=36, idx=27, gen=4, data=0x7f9ed0049cc0, poll_in=1, poll_out=0, poll_err=0) at socket.c:2471
#6  0x00007f9eeb2825d4 in event_dispatch_epoll_handler (event_pool=0x17feb00, event=0x7f9edf7fde84) at event-epoll.c:583
#7  0x00007f9eeb2828ab in event_dispatch_epoll_worker (data=0x180d0c0) at event-epoll.c:659
#8  0x00007f9ee9fce5da in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9ee98a4eaf in clone () from /lib64/libc.so.6
(gdb) frame 2
#2  0x00007f9eeb24cae6 in iobref_unref (iobref=0x7f9ed00063b0) at iobuf.c:944
944 iobuf.c: No such file or directory.
(gdb) print *iobref
$1 = {lock = {spinlock = 2, mutex = {__data = {__lock = 2, __count = 222, __owner = -2120437760, __nusers = 1, __kind = 8960, __spins = 512,
        __elision = 0, __list = {__prev = 0x4000, __next = 0x7f9ed00063b000}},
      __size = "\002\000\000\000\336\000\000\000\000\260\234\201\001\000\000\000\000#\000\000\000\002\000\000\000@\000\000\000\000\000\000\000\260c\000?\177", __align = 953482739714}}, ref = -256, iobrefs = 0xffffffffffffffff, alloced = -1, used = -1}

looks like the iobref is corrupted here. It seems to be a use-after-free issue. We need to dig into why a freed iobref is being accessed here.

(gdb) quit
A debugging session is active.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190428/3dd1a193/attachment-0001.html>

From amukherj at redhat.com  Sun Apr 28 13:13:22 2019
From: amukherj at redhat.com (Atin Mukherjee)
Date: Sun, 28 Apr 2019 18:43:22 +0530
Subject: [Gluster-devel] Weekly Untriaged Bugs
In-Reply-To: <307671157.25.1555897502768.JavaMail.jenkins@jenkins-el7.rht.gluster.org>
References: <307671157.25.1555897502768.JavaMail.jenkins@jenkins-el7.rht.gluster.org>
Message-ID: <CAGNCGH3hrBnOOKF3u6V6JAbfr7YM4TC6ED63WhWaziJ-_ZCdTw@mail.gmail.com>

While I understand this report captured bugs filed since last 1 week and do
not have ?Triaged? keyword, does it make better sense to exclude bugs which
aren?t in NEW state?

I believe the intention of this report is to check what all bugs haven?t
been looked at by maintainers/developers yet. BZs which are already fixed
or in ASSIGNED/POST state need not to feature in this list is what I
believe as otherwise it gives a false impression that too many bugs are
getting unnoticed which isn?t the reality. Thoughts?

On Mon, 22 Apr 2019 at 07:15, <jenkins at build.gluster.org> wrote:

> [...truncated 6 lines...]
> https://bugzilla.redhat.com/1699023 / core: Brick is not able to detach
> successfully in brick_mux environment
> https://bugzilla.redhat.com/1695416 / core: client log flooding with
> intentional socket shutdown message when a brick is down
> https://bugzilla.redhat.com/1695480 / core: Global Thread Pool
> https://bugzilla.redhat.com/1694943 / core: parallel-readdir slows down
> directory listing
> https://bugzilla.redhat.com/1700295 / core: The data couldn't be flushed
> immediately even with O_SYNC in glfs_create or with
> glfs_fsync/glfs_fdatasync after glfs_write.
> https://bugzilla.redhat.com/1698861 / disperse: Renaming a directory when
> 2 bricks of multiple disperse subvols are down leaves both old and new dirs
> on the bricks.
> https://bugzilla.redhat.com/1697293 / distribute: DHT: print hash and
> layout values in hexadecimal format in the logs
> https://bugzilla.redhat.com/1701039 / distribute: gluster replica 3
> arbiter Unfortunately data not distributed equally
> https://bugzilla.redhat.com/1697971 / fuse: Segfault in FUSE process,
> potential use after free
> https://bugzilla.redhat.com/1694139 / glusterd: Error waiting for job
> 'heketi-storage-copy-job' to complete on one-node k3s deployment.
> https://bugzilla.redhat.com/1695099 / glusterd: The number of glusterfs
> processes keeps increasing, using all available resources
> https://bugzilla.redhat.com/1692349 / project-infrastructure:
> gluster-csi-containers job is failing
> https://bugzilla.redhat.com/1698716 / project-infrastructure: Regression
> job did not vote for https://review.gluster.org/#/c/glusterfs/+/22366/
> https://bugzilla.redhat.com/1698694 / project-infrastructure: regression
> job isn't voting back to gerrit
> https://bugzilla.redhat.com/1699712 / project-infrastructure: regression
> job is voting Success even in case of failure
> https://bugzilla.redhat.com/1693385 / project-infrastructure: request to
> change the version of fedora in fedora-smoke-job
> https://bugzilla.redhat.com/1695484 / project-infrastructure: smoke fails
> with "Build root is locked by another process"
> https://bugzilla.redhat.com/1693184 / replicate: A brick
> process(glusterfsd) died with 'memory violation'
> https://bugzilla.redhat.com/1698566 / selfheal: shd crashed while
> executing ./tests/bugs/core/bug-1432542-mpx-restart-crash.t in CI
> https://bugzilla.redhat.com/1699309 / snapshot: Gluster snapshot fails
> with systemd autmounted bricks
> https://bugzilla.redhat.com/1696633 / tests: GlusterFs v4.1.5 Tests from
> /tests/bugs/ module failing on Intel
> https://bugzilla.redhat.com/1697812 / website: mention a pointer to all
> the mailing lists available under glusterfs project(
> https://www.gluster.org/community/)
> [...truncated 2 lines...]_______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel

-- 
- Atin (atinm)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190428/dac7d988/attachment.html>

From jenkins at build.gluster.org  Mon Apr 29 01:45:02 2019
From: jenkins at build.gluster.org (jenkins at build.gluster.org)
Date: Mon, 29 Apr 2019 01:45:02 +0000 (UTC)
Subject: [Gluster-devel] Weekly Untriaged Bugs
Message-ID: <468170951.44.1556502302538.JavaMail.jenkins@jenkins-el7.rht.gluster.org>

[...truncated 6 lines...]
https://bugzilla.redhat.com/1699023 / core: Brick is not able to detach successfully in brick_mux environment
https://bugzilla.redhat.com/1702316 / core: Cannot upgrade 5.x volume to 6.1 because of unused 'crypt' and 'bd' xlators
https://bugzilla.redhat.com/1695416 / core: client log flooding with intentional socket shutdown message when a brick is down
https://bugzilla.redhat.com/1695480 / core: Global Thread Pool
https://bugzilla.redhat.com/1694943 / core: parallel-readdir slows down directory listing
https://bugzilla.redhat.com/1700295 / core: The data couldn't be flushed immediately even with O_SYNC in glfs_create or with glfs_fsync/glfs_fdatasync after glfs_write.
https://bugzilla.redhat.com/1698861 / disperse: Renaming a directory when 2 bricks of multiple disperse subvols are down leaves both old and new dirs on the bricks.
https://bugzilla.redhat.com/1697293 / distribute: DHT: print hash and layout values in hexadecimal format in the logs
https://bugzilla.redhat.com/1703322 / doc: Need to document about fips-mode-rchecksum in gluster-7 release notes.
https://bugzilla.redhat.com/1702043 / fuse: Newly created files are inaccessible via FUSE
https://bugzilla.redhat.com/1695099 / glusterd: The number of glusterfs processes keeps increasing, using all available resources
https://bugzilla.redhat.com/1703007 / glusterd: The telnet or something would cause high memory usage for glusterd & glusterfsd
https://bugzilla.redhat.com/1703433 / project-infrastructure: gluster-block: setup GCOV & LCOV job
https://bugzilla.redhat.com/1703435 / project-infrastructure: gluster-block: Upstream Jenkins job which get triggered at PR level
https://bugzilla.redhat.com/1703329 / project-infrastructure: [gluster-infra]: Please create repo for plus one scale work
https://bugzilla.redhat.com/1698694 / project-infrastructure: regression job isn't voting back to gerrit
https://bugzilla.redhat.com/1695484 / project-infrastructure: smoke fails with "Build root is locked by another process"
https://bugzilla.redhat.com/1698566 / selfheal: shd crashed while executing ./tests/bugs/core/bug-1432542-mpx-restart-crash.t in CI
https://bugzilla.redhat.com/1699309 / snapshot: Gluster snapshot fails with systemd autmounted bricks
https://bugzilla.redhat.com/1696633 / tests: GlusterFs v4.1.5 Tests from /tests/bugs/ module failing on Intel
https://bugzilla.redhat.com/1702289 / tiering: Promotion failed for a0afd3e3-0109-49b7-9b74-ba77bf653aba.11229
https://bugzilla.redhat.com/1697812 / website: mention a pointer to all the mailing lists available under glusterfs project(https://www.gluster.org/community/)
[...truncated 2 lines...]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log
Type: application/octet-stream
Size: 2993 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/4063964a/attachment.obj>

From pkarampu at redhat.com  Mon Apr 29 05:15:48 2019
From: pkarampu at redhat.com (Pranith Kumar Karampuri)
Date: Mon, 29 Apr 2019 10:45:48 +0530
Subject: [Gluster-devel] questions on callstubs and "link-count" in
	index translator
In-Reply-To: <BYAPR15MB349599B0BB55B900D8194D37D03D0@BYAPR15MB3495.namprd15.prod.outlook.com>
References: <BYAPR15MB349599B0BB55B900D8194D37D03D0@BYAPR15MB3495.namprd15.prod.outlook.com>
Message-ID: <CAOgeEnYeSrE-Esz6Csg6hkyE1VP_XOfiuCv3=JkU85f6jHFMhQ@mail.gmail.com>

On Fri, Apr 26, 2019 at 10:55 PM Junsong Li <junsongli at fb.com> wrote:

> Hello list,
>
>
>
> I have a couple of questions on index translator implementation.
>
>    - Why does gluster need callstub and a different worker queue (and
>    thread) to process those call stubs? Is it just to lower the priority of
>    fops of internal inodes?
>
> It is to make sure parallel [f]xattrops on same inode doesn't happen.
Index xlator has to maintain indices in the internal directories based on
whether or not the xattrs it tracks on the file are non-zero.  When we
allow parallel [f]xattrops, callbacks of the fops may not reach index
xlator in the order they are wound to lower xlators. This can lead to index
xlator not maintain indices correctly.


>
>    -
>    - What?s the purpose of ?link-count? in xdata? It?s being used only in
>    index_fstat and index_lookup. I see sometimes the key is assigned 0/1 after
>    callback, and sometimes AFR uses it to store flag GF_XATTROP_INDEX_COUNT.
>    Is the code purposely reusing the key?
>
> Thanks,
>
> Junsong
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/7dcb14db/attachment.html>

From dkhandel at redhat.com  Mon Apr 29 05:22:26 2019
From: dkhandel at redhat.com (Deepshikha Khandelwal)
Date: Mon, 29 Apr 2019 10:52:26 +0530
Subject: [Gluster-devel] [Gluster-infra]  Weekly Untriaged Bugs
In-Reply-To: <CAGNCGH3hrBnOOKF3u6V6JAbfr7YM4TC6ED63WhWaziJ-_ZCdTw@mail.gmail.com>
References: <307671157.25.1555897502768.JavaMail.jenkins@jenkins-el7.rht.gluster.org>
	<CAGNCGH3hrBnOOKF3u6V6JAbfr7YM4TC6ED63WhWaziJ-_ZCdTw@mail.gmail.com>
Message-ID: <CABGtVXF6Jnp5R=RcxNMthmqPEqm=Fu4zt4ziajB+n99-X3h==A@mail.gmail.com>

This list also captures the BZs which are in NEW state. The search
description goes like this:

   - *Status:* NEW
   - *Product:* GlusterFS
   - *Changed:* (is greater than or equal to) -4w
   - *Creation date:* (changed after) -4w
   - *Keywords:* (does not contain the string) Triaged


On Sun, Apr 28, 2019 at 6:44 PM Atin Mukherjee <amukherj at redhat.com> wrote:

> While I understand this report captured bugs filed since last 1 week and
> do not have ?Triaged? keyword, does it make better sense to exclude bugs
> which aren?t in NEW state?
>
> I believe the intention of this report is to check what all bugs haven?t
> been looked at by maintainers/developers yet. BZs which are already fixed
> or in ASSIGNED/POST state need not to feature in this list is what I
> believe as otherwise it gives a false impression that too many bugs are
> getting unnoticed which isn?t the reality. Thoughts?
>
> On Mon, 22 Apr 2019 at 07:15, <jenkins at build.gluster.org> wrote:
>
>> [...truncated 6 lines...]
>> https://bugzilla.redhat.com/1699023 / core: Brick is not able to detach
>> successfully in brick_mux environment
>> https://bugzilla.redhat.com/1695416 / core: client log flooding with
>> intentional socket shutdown message when a brick is down
>> https://bugzilla.redhat.com/1695480 / core: Global Thread Pool
>> https://bugzilla.redhat.com/1694943 / core: parallel-readdir slows down
>> directory listing
>> https://bugzilla.redhat.com/1700295 / core: The data couldn't be flushed
>> immediately even with O_SYNC in glfs_create or with
>> glfs_fsync/glfs_fdatasync after glfs_write.
>> https://bugzilla.redhat.com/1698861 / disperse: Renaming a directory
>> when 2 bricks of multiple disperse subvols are down leaves both old and new
>> dirs on the bricks.
>> https://bugzilla.redhat.com/1697293 / distribute: DHT: print hash and
>> layout values in hexadecimal format in the logs
>> https://bugzilla.redhat.com/1701039 / distribute: gluster replica 3
>> arbiter Unfortunately data not distributed equally
>> https://bugzilla.redhat.com/1697971 / fuse: Segfault in FUSE process,
>> potential use after free
>> https://bugzilla.redhat.com/1694139 / glusterd: Error waiting for job
>> 'heketi-storage-copy-job' to complete on one-node k3s deployment.
>> https://bugzilla.redhat.com/1695099 / glusterd: The number of glusterfs
>> processes keeps increasing, using all available resources
>> https://bugzilla.redhat.com/1692349 / project-infrastructure:
>> gluster-csi-containers job is failing
>> https://bugzilla.redhat.com/1698716 / project-infrastructure: Regression
>> job did not vote for https://review.gluster.org/#/c/glusterfs/+/22366/
>> https://bugzilla.redhat.com/1698694 / project-infrastructure: regression
>> job isn't voting back to gerrit
>> https://bugzilla.redhat.com/1699712 / project-infrastructure: regression
>> job is voting Success even in case of failure
>> https://bugzilla.redhat.com/1693385 / project-infrastructure: request to
>> change the version of fedora in fedora-smoke-job
>> https://bugzilla.redhat.com/1695484 / project-infrastructure: smoke
>> fails with "Build root is locked by another process"
>> https://bugzilla.redhat.com/1693184 / replicate: A brick
>> process(glusterfsd) died with 'memory violation'
>> https://bugzilla.redhat.com/1698566 / selfheal: shd crashed while
>> executing ./tests/bugs/core/bug-1432542-mpx-restart-crash.t in CI
>> https://bugzilla.redhat.com/1699309 / snapshot: Gluster snapshot fails
>> with systemd autmounted bricks
>> https://bugzilla.redhat.com/1696633 / tests: GlusterFs v4.1.5 Tests from
>> /tests/bugs/ module failing on Intel
>> https://bugzilla.redhat.com/1697812 / website: mention a pointer to all
>> the mailing lists available under glusterfs project(
>> https://www.gluster.org/community/)
>> [...truncated 2 lines...]_______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
> --
> - Atin (atinm)
> _______________________________________________
> Gluster-infra mailing list
> Gluster-infra at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-infra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/86526612/attachment-0001.html>

From mscherer at redhat.com  Mon Apr 29 09:20:54 2019
From: mscherer at redhat.com (Michael Scherer)
Date: Mon, 29 Apr 2019 11:20:54 +0200
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <CAHxyDdNJkfdSwbr8ixgHynO=j73S+7iNp6suBtKRMGdQWNW==w@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
	<CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
	<CAJMr+NnBfNAfxASXO3U+2AWw0kWOvo8KwY5sH=MGo=cbH_jSMg@mail.gmail.com>
	<CAC+Jd5Aur44xSgCe0mad5KhG8G4487dOGvzD63O3kPJ+8p_rSw@mail.gmail.com>
	<CAHxyDdNJkfdSwbr8ixgHynO=j73S+7iNp6suBtKRMGdQWNW==w@mail.gmail.com>
Message-ID: <9ab07686a7beb07344fe39f76023dee4743619c8.camel@redhat.com>

Le vendredi 26 avril 2019 ? 18:50 +0530, Amar Tumballi Suryanarayan a
?crit :
> On Fri, Apr 26, 2019 at 6:27 PM Kaleb Keithley <kkeithle at redhat.com>
> wrote:
> 
> > 
> > 
> > On Fri, Apr 26, 2019 at 8:21 AM Harold Miller <harold at redhat.com>
> > wrote:
> > 
> > > Has Red Hat security cleared the Slack systems for confidential /
> > > customer information?
> > > 
> > > If not, it will make it difficult for support to collect/answer
> > > questions.
> > > 
> > 
> > I'm pretty sure Amar meant as a replacement for the freenode
> > #gluster and
> > #gluster-dev channels, given that he sent this to the public
> > gluster
> > mailing lists @gluster.org. Nobody should have even been posting
> > confidential and/or customer information to any of those lists or
> > channels.
> > And AFAIK nobody ever has.
> > 
> > 
> 
> Yep, I am only talking about IRC (from freenode, #gluster, #gluster-
> dev etc).  Also, I am not saying we are 'replacing IRC'. Gluster as a
> project started in pre-Slack era, and we have many users who prefer
> to stay in IRC.
> So, for now, no pressure to make a statement calling Slack channel as
> a 'Replacement' to IRC.
> 
> 
> > Amar, would you like to clarify which IRC channels you meant?
> > 
> > 
> 
> Thanks Kaleb. I was bit confused on why the concern of it came up in
> this group.

Well, unless people start to be on both irc and slack and everything,
that's fragmentation.

Also, since people can't access old logs (per design with the free plan
of slack), but they are still here on slack servers, how is it going to
work from a GDPR point of view ?

Shouldn't it requires a update to the privacy policy ?
-- 
Michael Scherer
Sysadmin, Community Infrastructure



-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: This is a digitally signed message part
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/f64631d6/attachment.sig>

From aspandey at redhat.com  Mon Apr 29 10:43:59 2019
From: aspandey at redhat.com (aspandey at redhat.com)
Date: Mon, 29 Apr 2019 10:43:59 +0000
Subject: [Gluster-devel] Invitation: Gluster Community Meeting (APAC
 friendly hours) @ Tue Apr 30,
 2019 11:30am - 12:30pm (IST) (gluster-devel@gluster.org)
Message-ID: <0000000000004638d70587a8f638@google.com>

You have been invited to the following event.

Title:  Gluster Community Meeting (APAC friendly hours)
Bridge: https://bluejeans.com/836554017

Meeting minutes: https://hackmd.io/OqZbh7gfQe6uvVUXUVKJ5g?both

Previous Meeting notes: http://github.com/gluster/community
When: Tue Apr 30, 2019 11:30am ? 12:30pm India Standard Time - Kolkata
Where: https://bluejeans.com/836554017
Calendar: gluster-devel at gluster.org
Who:
     * aspandey at redhat.com - organizer
     * gluster-users at gluster.org
     * maintainers at gluster.org
     * gluster-devel at gluster.org

Event details:  
https://www.google.com/calendar/event?action=VIEW&eid=N2NpMWp1YjRkbmZoYjhxNWMyZ2ZxdTB1dmUgZ2x1c3Rlci1kZXZlbEBnbHVzdGVyLm9yZw&tok=MTkjYXNwYW5kZXlAcmVkaGF0LmNvbWYzZDUzMmQ0MzQ5NmYxMDAwZThjNTk3ZmY3ZTY4M2Y1YTI3NzVlZDI&ctz=Asia%2FKolkata&hl=en&es=0

Invitation from Google Calendar: https://www.google.com/calendar/

You are receiving this courtesy email at the account  
gluster-devel at gluster.org because you are an attendee of this event.

To stop receiving future updates for this event, decline this event.  
Alternatively you can sign up for a Google account at  
https://www.google.com/calendar/ and control your notification settings for  
your entire calendar.

Forwarding this invitation could allow any recipient to send a response to  
the organizer and be added to the guest list, or invite others regardless  
of their own invitation status, or to modify your RSVP. Learn more at  
https://support.google.com/calendar/answer/37135#forwarding
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/f7e2161b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: text/calendar
Size: 1880 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/f7e2161b/attachment.ics>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: invite.ics
Type: application/ics
Size: 1921 bytes
Desc: not available
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/f7e2161b/attachment.bin>

From moagrawa at redhat.com  Mon Apr 29 14:03:47 2019
From: moagrawa at redhat.com (Mohit Agrawal)
Date: Mon, 29 Apr 2019 19:33:47 +0530
Subject: [Gluster-devel] Query regarding dictionary logic
Message-ID: <CALBz39y+0o5VmqSqPM4dk7UXGOPvfirsbn4konH0tK=13pDxEg@mail.gmail.com>

Hi All,

  I was just looking at the code of dict, I have one query current
dictionary logic.
  I am not able to understand why we use hash_size is 1 for a
dictionary.IMO with the
  hash_size of 1 dictionary always work like a list, not a hash, for every
lookup
  in dictionary complexity is O(n).

  Before optimizing the code I just want to know what was the exact reason
to define
  hash_size is 1?

  Please share your view on the same.

Thanks,
Mohit Agrawal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/46a09fbf/attachment-0001.html>

From vbellur at redhat.com  Tue Apr 30 06:14:35 2019
From: vbellur at redhat.com (Vijay Bellur)
Date: Mon, 29 Apr 2019 23:14:35 -0700
Subject: [Gluster-devel] Query regarding dictionary logic
In-Reply-To: <CALBz39y+0o5VmqSqPM4dk7UXGOPvfirsbn4konH0tK=13pDxEg@mail.gmail.com>
References: <CALBz39y+0o5VmqSqPM4dk7UXGOPvfirsbn4konH0tK=13pDxEg@mail.gmail.com>
Message-ID: <CAHn=sVBV9mcTkA921YWBPmOBNPYRZucb0mrsjcodQJYNsj_k2w@mail.gmail.com>

Hi Mohit,

On Mon, Apr 29, 2019 at 7:15 AM Mohit Agrawal <moagrawa at redhat.com> wrote:

> Hi All,
>
>   I was just looking at the code of dict, I have one query current
> dictionary logic.
>   I am not able to understand why we use hash_size is 1 for a
> dictionary.IMO with the
>   hash_size of 1 dictionary always work like a list, not a hash, for every
> lookup
>   in dictionary complexity is O(n).
>
>   Before optimizing the code I just want to know what was the exact reason
> to define
>   hash_size is 1?
>

This is a good question. I looked up the source in gluster's historic repo
[1] and hash_size is 1 even there. So, this could have been the case since
the first version of the dictionary code.

Would you be able to run some tests with a larger hash_size and share your
observations?

Thanks,
Vijay

[1] https://github.com/gluster/historic/blob/master/libglusterfs/src/dict.c



>
>   Please share your view on the same.
>
> Thanks,
> Mohit Agrawal
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190429/f5ab5a42/attachment.html>

From moagrawa at redhat.com  Tue Apr 30 06:31:13 2019
From: moagrawa at redhat.com (Mohit Agrawal)
Date: Tue, 30 Apr 2019 12:01:13 +0530
Subject: [Gluster-devel] Query regarding dictionary logic
In-Reply-To: <CAHn=sVBV9mcTkA921YWBPmOBNPYRZucb0mrsjcodQJYNsj_k2w@mail.gmail.com>
References: <CALBz39y+0o5VmqSqPM4dk7UXGOPvfirsbn4konH0tK=13pDxEg@mail.gmail.com>
	<CAHn=sVBV9mcTkA921YWBPmOBNPYRZucb0mrsjcodQJYNsj_k2w@mail.gmail.com>
Message-ID: <CALBz39xcQgR0dMi03Ud4SjdPj5yPyyhm7F=tRwNA-uLG=Y_2Jw@mail.gmail.com>

sure Vijay, I will try and update.

Regards,
Mohit Agrawal

On Tue, Apr 30, 2019 at 11:44 AM Vijay Bellur <vbellur at redhat.com> wrote:

> Hi Mohit,
>
> On Mon, Apr 29, 2019 at 7:15 AM Mohit Agrawal <moagrawa at redhat.com> wrote:
>
>> Hi All,
>>
>>   I was just looking at the code of dict, I have one query current
>> dictionary logic.
>>   I am not able to understand why we use hash_size is 1 for a
>> dictionary.IMO with the
>>   hash_size of 1 dictionary always work like a list, not a hash, for
>> every lookup
>>   in dictionary complexity is O(n).
>>
>>   Before optimizing the code I just want to know what was the exact
>> reason to define
>>   hash_size is 1?
>>
>
> This is a good question. I looked up the source in gluster's historic repo
> [1] and hash_size is 1 even there. So, this could have been the case since
> the first version of the dictionary code.
>
> Would you be able to run some tests with a larger hash_size and share your
> observations?
>
> Thanks,
> Vijay
>
> [1]
> https://github.com/gluster/historic/blob/master/libglusterfs/src/dict.c
>
>
>
>>
>>   Please share your view on the same.
>>
>> Thanks,
>> Mohit Agrawal
>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/e781b6ac/attachment.html>

From jthottan at redhat.com  Tue Apr 30 07:20:11 2019
From: jthottan at redhat.com (Jiffin Tony Thottan)
Date: Tue, 30 Apr 2019 12:50:11 +0530
Subject: [Gluster-devel] Proposing to previous ganesha HA cluster solution
 back to gluster code as gluster-7 feature
Message-ID: <b0ac05db-f0d7-143a-e8a9-48fff1189cdf@redhat.com>

Hi all,

Some of you folks may be familiar with HA solution provided for 
nfs-ganesha by gluster using pacemaker and corosync.

That feature was removed in glusterfs 3.10 in favour for common HA 
project "Storhaug". Even Storhaug was not progressed

much from last two years and current development is in halt state, hence 
planning to restore old HA ganesha solution back

to gluster code repository with some improvement and targetting for next 
gluster release 7.

I have opened up an issue [1] with details and posted initial set of 
patches [2]

Please share your thoughts on the same

Regards,

Jiffin

[1]https://github.com/gluster/glusterfs/issues/663 
<https://github.com/gluster/glusterfs/issues/663>

[2] 
https://review.gluster.org/#/q/topic:rfc-663+(status:open+OR+status:merged)

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/60189d01/attachment.html>

From atumball at redhat.com  Tue Apr 30 08:52:42 2019
From: atumball at redhat.com (Amar Tumballi Suryanarayan)
Date: Tue, 30 Apr 2019 14:22:42 +0530
Subject: [Gluster-devel] Query regarding dictionary logic
In-Reply-To: <CALBz39xcQgR0dMi03Ud4SjdPj5yPyyhm7F=tRwNA-uLG=Y_2Jw@mail.gmail.com>
References: <CALBz39y+0o5VmqSqPM4dk7UXGOPvfirsbn4konH0tK=13pDxEg@mail.gmail.com>
	<CAHn=sVBV9mcTkA921YWBPmOBNPYRZucb0mrsjcodQJYNsj_k2w@mail.gmail.com>
	<CALBz39xcQgR0dMi03Ud4SjdPj5yPyyhm7F=tRwNA-uLG=Y_2Jw@mail.gmail.com>
Message-ID: <CAHxyDdNg2BmqEM5o=SNaLEgNrxY+ctuyk3ogUMhyWc0se1kNQw@mail.gmail.com>

Shreyas/Kevin tried to address it some time back using
https://bugzilla.redhat.com/show_bug.cgi?id=1428049 (
https://review.gluster.org/16830)

I vaguely remember the reason to keep the hash value 1 was done during the
time when we had dictionary itself sent as on wire protocol, and in most
other places, number of entries in dictionary was on an avg, 3. So, we
felt, saving on a bit of memory for optimization was better at that time.

-Amar

On Tue, Apr 30, 2019 at 12:02 PM Mohit Agrawal <moagrawa at redhat.com> wrote:

> sure Vijay, I will try and update.
>
> Regards,
> Mohit Agrawal
>
> On Tue, Apr 30, 2019 at 11:44 AM Vijay Bellur <vbellur at redhat.com> wrote:
>
>> Hi Mohit,
>>
>> On Mon, Apr 29, 2019 at 7:15 AM Mohit Agrawal <moagrawa at redhat.com>
>> wrote:
>>
>>> Hi All,
>>>
>>>   I was just looking at the code of dict, I have one query current
>>> dictionary logic.
>>>   I am not able to understand why we use hash_size is 1 for a
>>> dictionary.IMO with the
>>>   hash_size of 1 dictionary always work like a list, not a hash, for
>>> every lookup
>>>   in dictionary complexity is O(n).
>>>
>>>   Before optimizing the code I just want to know what was the exact
>>> reason to define
>>>   hash_size is 1?
>>>
>>
>> This is a good question. I looked up the source in gluster's historic
>> repo [1] and hash_size is 1 even there. So, this could have been the case
>> since the first version of the dictionary code.
>>
>> Would you be able to run some tests with a larger hash_size and share
>> your observations?
>>
>> Thanks,
>> Vijay
>>
>> [1]
>> https://github.com/gluster/historic/blob/master/libglusterfs/src/dict.c
>>
>>
>>
>>>
>>>   Please share your view on the same.
>>>
>>> Thanks,
>>> Mohit Agrawal
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>
>> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel



-- 
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/5e87800f/attachment.html>

From moagrawa at redhat.com  Tue Apr 30 08:59:06 2019
From: moagrawa at redhat.com (Mohit Agrawal)
Date: Tue, 30 Apr 2019 14:29:06 +0530
Subject: [Gluster-devel] Query regarding dictionary logic
In-Reply-To: <CAHxyDdNg2BmqEM5o=SNaLEgNrxY+ctuyk3ogUMhyWc0se1kNQw@mail.gmail.com>
References: <CALBz39y+0o5VmqSqPM4dk7UXGOPvfirsbn4konH0tK=13pDxEg@mail.gmail.com>
	<CAHn=sVBV9mcTkA921YWBPmOBNPYRZucb0mrsjcodQJYNsj_k2w@mail.gmail.com>
	<CALBz39xcQgR0dMi03Ud4SjdPj5yPyyhm7F=tRwNA-uLG=Y_2Jw@mail.gmail.com>
	<CAHxyDdNg2BmqEM5o=SNaLEgNrxY+ctuyk3ogUMhyWc0se1kNQw@mail.gmail.com>
Message-ID: <CALBz39xeQK0JyN3tOcPm-Au9LduqPHwPaCmNOc+YF1ky8oTvMw@mail.gmail.com>

Thanks, Amar for sharing the patch, I will test and share the result.

On Tue, Apr 30, 2019 at 2:23 PM Amar Tumballi Suryanarayan <
atumball at redhat.com> wrote:

> Shreyas/Kevin tried to address it some time back using
> https://bugzilla.redhat.com/show_bug.cgi?id=1428049 (
> https://review.gluster.org/16830)
>
> I vaguely remember the reason to keep the hash value 1 was done during the
> time when we had dictionary itself sent as on wire protocol, and in most
> other places, number of entries in dictionary was on an avg, 3. So, we
> felt, saving on a bit of memory for optimization was better at that time.
>
> -Amar
>
> On Tue, Apr 30, 2019 at 12:02 PM Mohit Agrawal <moagrawa at redhat.com>
> wrote:
>
>> sure Vijay, I will try and update.
>>
>> Regards,
>> Mohit Agrawal
>>
>> On Tue, Apr 30, 2019 at 11:44 AM Vijay Bellur <vbellur at redhat.com> wrote:
>>
>>> Hi Mohit,
>>>
>>> On Mon, Apr 29, 2019 at 7:15 AM Mohit Agrawal <moagrawa at redhat.com>
>>> wrote:
>>>
>>>> Hi All,
>>>>
>>>>   I was just looking at the code of dict, I have one query current
>>>> dictionary logic.
>>>>   I am not able to understand why we use hash_size is 1 for a
>>>> dictionary.IMO with the
>>>>   hash_size of 1 dictionary always work like a list, not a hash, for
>>>> every lookup
>>>>   in dictionary complexity is O(n).
>>>>
>>>>   Before optimizing the code I just want to know what was the exact
>>>> reason to define
>>>>   hash_size is 1?
>>>>
>>>
>>> This is a good question. I looked up the source in gluster's historic
>>> repo [1] and hash_size is 1 even there. So, this could have been the case
>>> since the first version of the dictionary code.
>>>
>>> Would you be able to run some tests with a larger hash_size and share
>>> your observations?
>>>
>>> Thanks,
>>> Vijay
>>>
>>> [1]
>>> https://github.com/gluster/historic/blob/master/libglusterfs/src/dict.c
>>>
>>>
>>>
>>>>
>>>>   Please share your view on the same.
>>>>
>>>> Thanks,
>>>> Mohit Agrawal
>>>> _______________________________________________
>>>> Gluster-devel mailing list
>>>> Gluster-devel at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>> _______________________________________________
>> Gluster-devel mailing list
>> Gluster-devel at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
>
>
> --
> Amar Tumballi (amarts)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/56a543dc/attachment-0001.html>

From skoduri at redhat.com  Tue Apr 30 10:45:17 2019
From: skoduri at redhat.com (Soumya Koduri)
Date: Tue, 30 Apr 2019 16:15:17 +0530
Subject: [Gluster-devel] Backlog/Improvements tracking
Message-ID: <d74cde0b-da92-5d20-36c4-86aa6e05503a@redhat.com>

Hi,

To track any new feature or improvements we are currently using github . 
I assume those issues refer to the ones which are actively being worked 
upon. How do we track backlogs which may not get addressed (at least in 
the near future)?

For eg., I am planning to close couple of RFE BZs [1]..[3] which were 
filed to improve upcall mechanism, as there is no active development 
happening in those aspects. But at the same time I like to retain the 
list for future reference (in case any new member like to take up).

Can we use github itself to track all the feature-gaps of a component in 
one issue (note: it may be in open state forever) or is it better to 
document these as limitations in the admin/developer guide & close BZ/issue?

Thanks,
Soumya

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1214654
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214644
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1200264

From kkeithle at redhat.com  Tue Apr 30 11:06:35 2019
From: kkeithle at redhat.com (Kaleb Keithley)
Date: Tue, 30 Apr 2019 07:06:35 -0400
Subject: [Gluster-devel] Backlog/Improvements tracking
In-Reply-To: <d74cde0b-da92-5d20-36c4-86aa6e05503a@redhat.com>
References: <d74cde0b-da92-5d20-36c4-86aa6e05503a@redhat.com>
Message-ID: <CAC+Jd5ACvQ9qMAV2jCzy1UoryL7H1ng_XPPPGLjutRStLAYnFQ@mail.gmail.com>

Yes, please open github issues for these RFEs and close the BZs.

Thanks

On Tue, Apr 30, 2019 at 6:46 AM Soumya Koduri <skoduri at redhat.com> wrote:

> Hi,
>
> To track any new feature or improvements we are currently using github .
> I assume those issues refer to the ones which are actively being worked
> upon. How do we track backlogs which may not get addressed (at least in
> the near future)?
>
> For eg., I am planning to close couple of RFE BZs [1]..[3] which were
> filed to improve upcall mechanism, as there is no active development
> happening in those aspects. But at the same time I like to retain the
> list for future reference (in case any new member like to take up).
>
> Can we use github itself to track all the feature-gaps of a component in
> one issue (note: it may be in open state forever) or is it better to
> document these as limitations in the admin/developer guide & close
> BZ/issue?
>
> Thanks,
> Soumya
>
> [1] https://bugzilla.redhat.com/show_bug.cgi?id=1214654
> [2] https://bugzilla.redhat.com/show_bug.cgi?id=1214644
> [3] https://bugzilla.redhat.com/show_bug.cgi?id=1200264
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/f72550ed/attachment.html>

From srakonde at redhat.com  Tue Apr 30 12:37:25 2019
From: srakonde at redhat.com (Sanju Rakonde)
Date: Tue, 30 Apr 2019 18:07:25 +0530
Subject: [Gluster-devel] ./tests/basic/uss.t is timing out in release-6
	branch
Message-ID: <CABj3WN11=p_8y-SawHnxZxKo83NG96xxDf7pe--zL8U=bWJ7Yw@mail.gmail.com>

Hi Raghavendra,

./tests/basic/uss.t is timing out in release-6 branch consistently. One
such instance is https://review.gluster.org/#/c/glusterfs/+/22641/. Can you
please look into this?

-- 
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/304c0ea4/attachment.html>

From rabhat at redhat.com  Tue Apr 30 14:42:34 2019
From: rabhat at redhat.com (FNU Raghavendra Manjunath)
Date: Tue, 30 Apr 2019 10:42:34 -0400
Subject: [Gluster-devel] ./tests/basic/uss.t is timing out in release-6
	branch
In-Reply-To: <CABj3WN11=p_8y-SawHnxZxKo83NG96xxDf7pe--zL8U=bWJ7Yw@mail.gmail.com>
References: <CABj3WN11=p_8y-SawHnxZxKo83NG96xxDf7pe--zL8U=bWJ7Yw@mail.gmail.com>
Message-ID: <CAMRKmT9Me4isEM55bS80-aYa=BN3T4jQ2sY6AbEhCS6usSa5Jg@mail.gmail.com>

The failure looks similar to the issue I had mentioned in [1]

In short for some reason the cleanup (the cleanup function that we call in
our .t files) seems to be taking more time and also not cleaning up
properly. This leads to problems for the 2nd iteration (where basic things
such as volume creation or volume start itself fails due to ENODATA or
ENOENT errors).

The 2nd iteration of the uss.t ran had the following errors.

"[2019-04-29 09:08:15.275773]:++++++++++ G_LOG:./tests/basic/uss.t: TEST:
39 gluster --mode=script --wignore volume set patchy nfs.disable false
++++++++++
[2019-04-29 09:08:15.390550]  : volume set patchy nfs.disable false :
SUCCESS
[2019-04-29 09:08:15.404624]:++++++++++ G_LOG:./tests/basic/uss.t: TEST: 42
gluster --mode=script --wignore volume start patchy ++++++++++
[2019-04-29 09:08:15.468780]  : volume start patchy : FAILED : Failed to
get extended attribute trusted.glusterfs.volume-id for brick dir
/d/backends/3/patchy_snap_mnt. Reason : No data available
"

These are the initial steps to create and start volume. Why
trusted.glusterfs.volume-id extended attribute is absent is not sure. The
analysis in [1] had errors of ENOENT (i.e. export directory itself was
absent).
I suspect this to be because of some issue with the cleanup mechanism at
the end of the tests.

[1] https://lists.gluster.org/pipermail/gluster-devel/2019-April/056104.html

On Tue, Apr 30, 2019 at 8:37 AM Sanju Rakonde <srakonde at redhat.com> wrote:

> Hi Raghavendra,
>
> ./tests/basic/uss.t is timing out in release-6 branch consistently. One
> such instance is https://review.gluster.org/#/c/glusterfs/+/22641/. Can
> you please look into this?
>
> --
> Thanks,
> Sanju
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/08a7ddba/attachment.html>

From rabhat at redhat.com  Tue Apr 30 18:16:14 2019
From: rabhat at redhat.com (FNU Raghavendra Manjunath)
Date: Tue, 30 Apr 2019 14:16:14 -0400
Subject: [Gluster-devel] ./tests/basic/uss.t is timing out in release-6
	branch
In-Reply-To: <CAMRKmT9Me4isEM55bS80-aYa=BN3T4jQ2sY6AbEhCS6usSa5Jg@mail.gmail.com>
References: <CABj3WN11=p_8y-SawHnxZxKo83NG96xxDf7pe--zL8U=bWJ7Yw@mail.gmail.com>
	<CAMRKmT9Me4isEM55bS80-aYa=BN3T4jQ2sY6AbEhCS6usSa5Jg@mail.gmail.com>
Message-ID: <CAMRKmT8Zg9J4U=dQg5LUVpO1V9q0YELDoB0xTfxdHjyE_q9GBA@mail.gmail.com>

To make things relatively easy for the cleanup () function in the test
framework, I think it would be better to ensure that uss.t itself deletes
snapshots and the volume once the tests are done. Patch [1] has been
submitted for review.

[1] https://review.gluster.org/#/c/glusterfs/+/22649/

Regards,
Raghavendra

On Tue, Apr 30, 2019 at 10:42 AM FNU Raghavendra Manjunath <
rabhat at redhat.com> wrote:

>
> The failure looks similar to the issue I had mentioned in [1]
>
> In short for some reason the cleanup (the cleanup function that we call in
> our .t files) seems to be taking more time and also not cleaning up
> properly. This leads to problems for the 2nd iteration (where basic things
> such as volume creation or volume start itself fails due to ENODATA or
> ENOENT errors).
>
> The 2nd iteration of the uss.t ran had the following errors.
>
> "[2019-04-29 09:08:15.275773]:++++++++++ G_LOG:./tests/basic/uss.t: TEST:
> 39 gluster --mode=script --wignore volume set patchy nfs.disable false
> ++++++++++
> [2019-04-29 09:08:15.390550]  : volume set patchy nfs.disable false :
> SUCCESS
> [2019-04-29 09:08:15.404624]:++++++++++ G_LOG:./tests/basic/uss.t: TEST:
> 42 gluster --mode=script --wignore volume start patchy ++++++++++
> [2019-04-29 09:08:15.468780]  : volume start patchy : FAILED : Failed to
> get extended attribute trusted.glusterfs.volume-id for brick dir
> /d/backends/3/patchy_snap_mnt. Reason : No data available
> "
>
> These are the initial steps to create and start volume. Why
> trusted.glusterfs.volume-id extended attribute is absent is not sure. The
> analysis in [1] had errors of ENOENT (i.e. export directory itself was
> absent).
> I suspect this to be because of some issue with the cleanup mechanism at
> the end of the tests.
>
> [1]
> https://lists.gluster.org/pipermail/gluster-devel/2019-April/056104.html
>
> On Tue, Apr 30, 2019 at 8:37 AM Sanju Rakonde <srakonde at redhat.com> wrote:
>
>> Hi Raghavendra,
>>
>> ./tests/basic/uss.t is timing out in release-6 branch consistently. One
>> such instance is https://review.gluster.org/#/c/glusterfs/+/22641/. Can
>> you please look into this?
>>
>> --
>> Thanks,
>> Sanju
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/d0637088/attachment.html>

From rabhat at redhat.com  Tue Apr 30 19:58:21 2019
From: rabhat at redhat.com (FNU Raghavendra Manjunath)
Date: Tue, 30 Apr 2019 15:58:21 -0400
Subject: [Gluster-devel] inode table destruction
Message-ID: <CAMRKmT-vuJU_u3KmPqS=oCi7h8ZET=58e8JW7OxjpYsfZ5HE_w@mail.gmail.com>

Hi All,

There is a good chance that, the inode on which unref came has already been
 zero refed and added to the purge list. This can happen when inode table is
 being destroyed (glfs_fini is something which destroys the inode table).

 Consider a directory 'a' which has a file 'b'. Now as part of inode table
destruction
 zero refing of inodes does not happen from leaf to the root. It happens in
the order
 inodes are present in the list. So, in this example, the dentry of 'b'
would have its
 parent set to the inode of 'a'. So if 'a' gets zero refed first (as part of
 inode table cleanup) and then 'b' has to zero refed, then dentry_unset is
called on
 the dentry of 'b' and it further goes on to call inode_unref on b's parent
which is
 'a'. In this situation, GF_ASSERT would be called as the refcount of 'a'
has been
 already set to zero.

Below is a snippet of the core file generated from such ASSERT call in one
of the regression test runs.

"

*14:39:49* No symbol table info available.*14:39:49* #1
0x00007f2a539fc8f8 in abort () from /lib64/libc.so.6*14:39:49* No
symbol table info available.*14:39:49* #2  0x00007f2a539f4026 in
__assert_fail_base () from /lib64/libc.so.6*14:39:49* No symbol table
info available.*14:39:49* #3  0x00007f2a539f40d2 in __assert_fail ()
from /lib64/libc.so.6*14:39:49* No symbol table info
available.*14:39:49* #4  0x00007f2a553e3208 in __inode_unref
(inode=0x7f2a3c05edf8, clear=false) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:483*14:39:49*
        index = 0*14:39:49*         this = 0x7f2a3c03e840*14:39:49*
     nlookup = 0*14:39:49*         __PRETTY_FUNCTION__ =
"__inode_unref"*14:39:49* #5  0x00007f2a553e2745 in __dentry_unset
(dentry=0x7f2a3c064e48) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:212*14:39:49*
No locals.*14:39:49* #6  0x00007f2a553e308a in __inode_retire
(inode=0x7f2a3c05ebc8) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:442*14:39:49*
        dentry = 0x7f2a3c064e48*14:39:49*         t =
0x7f2a3c064398*14:39:49* #7  0x00007f2a553e392f in
__inode_ref_reduce_by_n (inode=0x7f2a3c05ebc8, nref=0) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:708*14:39:49*
        nlookup = 0*14:39:49*         __PRETTY_FUNCTION__ =
"__inode_ref_reduce_by_n"*14:39:49* #8  0x00007f2a553e61d5 in
inode_table_destroy (inode_table=0x7f2a28007f90) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1867*14:39:49*
        trav = 0x7f2a3c05ebc8*14:39:49*         __FUNCTION__ =
"inode_table_destroy"*14:39:49* #9  0x00007f2a553e600c in
inode_table_destroy_all (ctx=0x7f2a3c001170) at
/home/jenkins/root/workspace/centos7-regression/libglusterfs/src/inode.c:1791*14:39:49*
        trav_graph = 0x7f2a240041a0*14:39:49*         tmp =
0x7f2a3c0013c8*14:39:49*         tree = 0x7f2a24022af0*14:39:49*
  inode_table = 0x7f2a28007f90*14:39:49* #10 0x00007f2a46a83390 in
pub_glfs_fini (fs=0x7f2a3c000ff0) at
/home/jenkins/root/workspace/centos7-regression/api/src/glfs.c:1346*14:39:49*
        ret = 0*14:39:49*         countdown = 98*14:39:49*
subvol = 0x7f2a24022af0*14:39:49*         ctx =
0x7f2a3c001170*14:39:49*         graph = 0x7f2a240041a0*14:39:49*
   call_pool = 0x7f2a3c000df0*14:39:49*         fs_init = 1*14:39:49*
       err = 0*14:39:49*         old_THIS = 0x7f2a380084c0*14:39:49*
      __FUNCTION__ = "pub_glfs_fini"

"


IIUC the solution for it would be to add a flag in the inode table which
tells whether the
cleanup of inode table has started or not. Do not call GF_ASSRT
(inode->ref) if the inode
table to which the inode being unrefed belongs to, is already getting
cleaned up.

A patch [1] has been submitted for review with the change mentioned above.

[1] https://review.gluster.org/#/c/glusterfs/+/22650/

Regards,
Raghavendra
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/dd5b7c9a/attachment-0001.html>

From scott.c.worthington at gmail.com  Fri Apr 26 09:59:38 2019
From: scott.c.worthington at gmail.com (Scott Worthington)
Date: Fri, 26 Apr 2019 09:59:38 -0000
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
Message-ID: <CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>

Hello, are you not _BOTH_ Red Hat FTEs or contractors?

On Fri, Apr 26, 2019, 3:16 AM Michael Scherer <mscherer at redhat.com> wrote:

> Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan a
> ?crit :
> > Hi All,
> >
> > We wanted to move to Slack from IRC for our official communication
> > channel
> > from sometime, but couldn't as we didn't had a proper URL for us to
> > register. 'gluster' was taken and we didn't knew who had it
> > registered.
> > Thanks to constant ask from Satish, Slack team has now agreed to let
> > us use
> > https://gluster.slack.com and I am happy to invite you all there.
> > (Use this
> > link
> > <
> >
> https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
> > >
> > to
> > join)
> >
> > Please note that, it won't be a replacement for mailing list. But can
> > be
> > used by all developers and users for quick communication. Also note
> > that,
> > no information there would be 'stored' beyond 10k lines as we are
> > using the
> > free version of Slack.
>
> Aren't we concerned about the ToS of slack ? Last time I did read them,
> they were quite scary (like, if you use your corporate email, you
> engage your employer, and that wasn't the worst part).
>
> Also, to anticipate the question, my employer Legal department told me
> to not setup a bridge between IRC and slack, due to the said ToS.
>
> --
> Michael Scherer
> Sysadmin, Community Infrastructure
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/1b0952f2/attachment-0001.html>

From harold at redhat.com  Fri Apr 26 12:20:33 2019
From: harold at redhat.com (Harold Miller)
Date: Fri, 26 Apr 2019 12:20:33 -0000
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
	<CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
Message-ID: <CAJMr+NnBfNAfxASXO3U+2AWw0kWOvo8KwY5sH=MGo=cbH_jSMg@mail.gmail.com>

Has Red Hat security cleared the Slack systems for confidential / customer
information?

If not, it will make it difficult for support to collect/answer questions.


Harold Miller, Associate Manager,
Red Hat, Enterprise Cloud Support
Desk - US (650) 254-4346



On Fri, Apr 26, 2019 at 6:00 AM Scott Worthington <
scott.c.worthington at gmail.com> wrote:

> Hello, are you not _BOTH_ Red Hat FTEs or contractors?
>
> On Fri, Apr 26, 2019, 3:16 AM Michael Scherer <mscherer at redhat.com> wrote:
>
>> Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan a
>> ?crit :
>> > Hi All,
>> >
>> > We wanted to move to Slack from IRC for our official communication
>> > channel
>> > from sometime, but couldn't as we didn't had a proper URL for us to
>> > register. 'gluster' was taken and we didn't knew who had it
>> > registered.
>> > Thanks to constant ask from Satish, Slack team has now agreed to let
>> > us use
>> > https://gluster.slack.com and I am happy to invite you all there.
>> > (Use this
>> > link
>> > <
>> >
>> https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
>> > >
>> > to
>> > join)
>> >
>> > Please note that, it won't be a replacement for mailing list. But can
>> > be
>> > used by all developers and users for quick communication. Also note
>> > that,
>> > no information there would be 'stored' beyond 10k lines as we are
>> > using the
>> > free version of Slack.
>>
>> Aren't we concerned about the ToS of slack ? Last time I did read them,
>> they were quite scary (like, if you use your corporate email, you
>> engage your employer, and that wasn't the worst part).
>>
>> Also, to anticipate the question, my employer Legal department told me
>> to not setup a bridge between IRC and slack, due to the said ToS.
>>
>> --
>> Michael Scherer
>> Sysadmin, Community Infrastructure
>>
>>
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> https://lists.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users



-- 

HAROLD MILLER

ASSOCIATE MANAGER, ENTERPRISE CLOUD SUPPORT

Red Hat

<https://www.redhat.com/>

Harold at RedHat.com    T: (650)-254-4346
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/3d9c4fd8/attachment-0001.html>

From harold at redhat.com  Fri Apr 26 13:26:47 2019
From: harold at redhat.com (Harold Miller)
Date: Fri, 26 Apr 2019 13:26:47 -0000
Subject: [Gluster-devel] [Gluster-users] One more way to contact Gluster
 team - Slack (gluster.slack.com)
In-Reply-To: <CAHxyDdNJkfdSwbr8ixgHynO=j73S+7iNp6suBtKRMGdQWNW==w@mail.gmail.com>
References: <CAHxyDdOthr0kzN+k_YR0yacvfKNMM0OK8vYjzNJ7DNqUuABpmw@mail.gmail.com>
	<e347c9c8c8455c2cff3d0bc0f3e262c3a828c74c.camel@redhat.com>
	<CAK6EcdmHrjEkCyS2bZn8=-BLtukmaJ8JS=QqotAkw624RmQjqg@mail.gmail.com>
	<CAJMr+NnBfNAfxASXO3U+2AWw0kWOvo8KwY5sH=MGo=cbH_jSMg@mail.gmail.com>
	<CAC+Jd5Aur44xSgCe0mad5KhG8G4487dOGvzD63O3kPJ+8p_rSw@mail.gmail.com>
	<CAHxyDdNJkfdSwbr8ixgHynO=j73S+7iNp6suBtKRMGdQWNW==w@mail.gmail.com>
Message-ID: <CAJMr+NkWMQ6Lm4SbuayAMUFwtav0cTc6g1k2BxjBnoC802U=sg@mail.gmail.com>

Amar,

Thanks for the clarification.  I'll go climb back into my cave. :)

Harold

On Fri, Apr 26, 2019 at 9:20 AM Amar Tumballi Suryanarayan <
atumball at redhat.com> wrote:

>
>
> On Fri, Apr 26, 2019 at 6:27 PM Kaleb Keithley <kkeithle at redhat.com>
> wrote:
>
>>
>>
>> On Fri, Apr 26, 2019 at 8:21 AM Harold Miller <harold at redhat.com> wrote:
>>
>>> Has Red Hat security cleared the Slack systems for confidential /
>>> customer information?
>>>
>>> If not, it will make it difficult for support to collect/answer
>>> questions.
>>>
>>
>> I'm pretty sure Amar meant as a replacement for the freenode #gluster and
>> #gluster-dev channels, given that he sent this to the public gluster
>> mailing lists @gluster.org. Nobody should have even been posting
>> confidential and/or customer information to any of those lists or channels.
>> And AFAIK nobody ever has.
>>
>>
> Yep, I am only talking about IRC (from freenode, #gluster, #gluster-dev
> etc).  Also, I am not saying we are 'replacing IRC'. Gluster as a project
> started in pre-Slack era, and we have many users who prefer to stay in IRC.
> So, for now, no pressure to make a statement calling Slack channel as a
> 'Replacement' to IRC.
>
>
>> Amar, would you like to clarify which IRC channels you meant?
>>
>>
>
> Thanks Kaleb. I was bit confused on why the concern of it came up in this
> group.
>
>
>
>>
>>> On Fri, Apr 26, 2019 at 6:00 AM Scott Worthington <
>>> scott.c.worthington at gmail.com> wrote:
>>>
>>>> Hello, are you not _BOTH_ Red Hat FTEs or contractors?
>>>>
>>>>
> Yes! but come from very different internal teams.
>
> Michael supports Gluster (the project) team's Infrastructure needs, and
> has valid concerns from his perspective :-) I, on the other hand, bother
> more about code, users, and how to make sure we are up-to-date with other
> technologies and communities, from the engineering view point.
>
>
>> On Fri, Apr 26, 2019, 3:16 AM Michael Scherer <mscherer at redhat.com>
>>>> wrote:
>>>>
>>>>> Le vendredi 26 avril 2019 ? 13:24 +0530, Amar Tumballi Suryanarayan a
>>>>> ?crit :
>>>>> > Hi All,
>>>>> >
>>>>> > We wanted to move to Slack from IRC for our official communication
>>>>> > channel
>>>>> > from sometime, but couldn't as we didn't had a proper URL for us to
>>>>> > register. 'gluster' was taken and we didn't knew who had it
>>>>> > registered.
>>>>> > Thanks to constant ask from Satish, Slack team has now agreed to let
>>>>> > us use
>>>>> > https://gluster.slack.com and I am happy to invite you all there.
>>>>> > (Use this
>>>>> > link
>>>>> > <
>>>>> >
>>>>> https://join.slack.com/t/gluster/shared_invite/enQtNjIxMTA1MTk3MDE1LWIzZWZjNzhkYWEwNDdiZWRiOTczMTc4ZjdiY2JiMTc3MDE5YmEyZTRkNzg0MWJiMWM3OGEyMDU2MmYzMTViYTA
>>>>> > >
>>>>> > to
>>>>> > join)
>>>>> >
>>>>> > Please note that, it won't be a replacement for mailing list. But can
>>>>> > be
>>>>> > used by all developers and users for quick communication. Also note
>>>>> > that,
>>>>> > no information there would be 'stored' beyond 10k lines as we are
>>>>> > using the
>>>>> > free version of Slack.
>>>>>
>>>>> Aren't we concerned about the ToS of slack ? Last time I did read them,
>>>>> they were quite scary (like, if you use your corporate email, you
>>>>> engage your employer, and that wasn't the worst part).
>>>>>
>>>>> Also, to anticipate the question, my employer Legal department told me
>>>>> to not setup a bridge between IRC and slack, due to the said ToS.
>>>>>
>>>>>
> Again, re-iterating here. Not planning to use any bridges from IRC to
> Slack. I re-read the Slack API Terms and condition. And it makes sense.
> They surely don't want us to build another slack, or abuse slack with too
> many API requests made for collecting logs.
>
> Currently, to start with, we are not adding any bots (other than github
> bot). Hopefully, that will keep us under proper usage guidelines.
>
> -Amar
>
>
>> --
>>>>> Michael Scherer
>>>>> Sysadmin, Community Infrastructure
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>
>>>
>>>
>>> --
>>>
>>> HAROLD MILLER
>>>
>>> ASSOCIATE MANAGER, ENTERPRISE CLOUD SUPPORT
>>>
>>> Red Hat
>>>
>>> <https://www.redhat.com/>
>>>
>>> Harold at RedHat.com    T: (650)-254-4346
>>> <https://red.ht/sig>
>>> TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
> --
> Amar Tumballi (amarts)
>


-- 

HAROLD MILLER

ASSOCIATE MANAGER, ENTERPRISE CLOUD SUPPORT

Red Hat

<https://www.redhat.com/>

Harold at RedHat.com    T: (650)-254-4346
<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190426/61f5c6ed/attachment-0001.html>

From jim.kinney at gmail.com  Tue Apr 30 12:19:47 2019
From: jim.kinney at gmail.com (Jim Kinney)
Date: Tue, 30 Apr 2019 12:19:47 -0000
Subject: [Gluster-devel] [Gluster-users] Proposing to previous ganesha
	HA cluster solution back to gluster code as gluster-7 feature
In-Reply-To: <b0ac05db-f0d7-143a-e8a9-48fff1189cdf@redhat.com>
References: <b0ac05db-f0d7-143a-e8a9-48fff1189cdf@redhat.com>
Message-ID: <9BE7F129-DE42-46A5-896B-81460E605E9E@gmail.com>

+1!
I'm using nfs-ganesha in my next upgrade so my client systems can use NFS instead of fuse mounts. Having an integrated, designed in process to coordinate multiple nodes into an HA cluster will very welcome.

On April 30, 2019 3:20:11 AM EDT, Jiffin Tony Thottan <jthottan at redhat.com> wrote:
>Hi all,
>
>Some of you folks may be familiar with HA solution provided for 
>nfs-ganesha by gluster using pacemaker and corosync.
>
>That feature was removed in glusterfs 3.10 in favour for common HA 
>project "Storhaug". Even Storhaug was not progressed
>
>much from last two years and current development is in halt state,
>hence 
>planning to restore old HA ganesha solution back
>
>to gluster code repository with some improvement and targetting for
>next 
>gluster release 7.
>
>I have opened up an issue [1] with details and posted initial set of 
>patches [2]
>
>Please share your thoughts on the same
>
>Regards,
>
>Jiffin
>
>[1]https://github.com/gluster/glusterfs/issues/663 
><https://github.com/gluster/glusterfs/issues/663>
>
>[2] 
>https://review.gluster.org/#/q/topic:rfc-663+(status:open+OR+status:merged)

-- 
Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/bc40d5d6/attachment-0001.html>

From hunter86_bg at yahoo.com  Tue Apr 30 13:04:29 2019
From: hunter86_bg at yahoo.com (Strahil)
Date: Tue, 30 Apr 2019 13:04:29 -0000
Subject: [Gluster-devel] [Gluster-users] Proposing to previous ganesha
 HA clustersolution back to gluster code as gluster-7 feature
Message-ID: <geuj12ndtqkroi36h5xmrydi.1556629490318@email.android.com>

Keep in mind that corosync/pacemaker  is hard for proper setup by new admins/users.

I'm still trying to remediate the effects of poor configuration at work.
Also, storhaug is nice for hyperconverged setups where the host is not only hosting bricks, but  other  workloads.
Corosync/pacemaker require proper fencing to be setup and most of the stonith resources 'shoot the other node in the head'.
I would be happy to see an easy to deploy (let say 'cluster.enable-ha-ganesha true') and gluster to be bringing up the Floating IPs and taking care of the NFS locks, so no disruption will be felt by the clients.

Still, this will be a lot of work to achieve.

Best Regards,
Strahil NikolovOn Apr 30, 2019 15:19, Jim Kinney <jim.kinney at gmail.com> wrote:
>
> +1!
> I'm using nfs-ganesha in my next upgrade so my client systems can use NFS instead of fuse mounts. Having an integrated, designed in process to coordinate multiple nodes into an HA cluster will very welcome.
>
> On April 30, 2019 3:20:11 AM EDT, Jiffin Tony Thottan <jthottan at redhat.com> wrote:
>>
>> Hi all,
>>
>> Some of you folks may be familiar with HA solution provided for nfs-ganesha by gluster using pacemaker and corosync.
>>
>> That feature was removed in glusterfs 3.10 in favour for common HA project "Storhaug". Even Storhaug was not progressed
>>
>> much from last two years and current development is in halt state, hence planning to restore old HA ganesha solution back
>>
>> to gluster code repository with some improvement and targetting for next gluster release 7.
>>
>> I have opened up an issue [1] with details and posted initial set of patches [2]
>>
>> Please share your thoughts on the same
>>
>> Regards,
>>
>> Jiffin??
>>
>> [1] https://github.com/gluster/glusterfs/issues/663
>>
>> [2] https://review.gluster.org/#/q/topic:rfc-663+(status:open+OR+status:merged)
>
>
> -- 
> Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/d94f3b1c/attachment-0001.html>

From Renaud.Fortier at fsaa.ulaval.ca  Tue Apr 30 13:21:26 2019
From: Renaud.Fortier at fsaa.ulaval.ca (Renaud Fortier)
Date: Tue, 30 Apr 2019 13:21:26 -0000
Subject: [Gluster-devel] [Gluster-users] Proposing to previous ganesha
 HA cluster	solution back to gluster code as gluster-7 feature
In-Reply-To: <9BE7F129-DE42-46A5-896B-81460E605E9E@gmail.com>
References: <b0ac05db-f0d7-143a-e8a9-48fff1189cdf@redhat.com>
	<9BE7F129-DE42-46A5-896B-81460E605E9E@gmail.com>
Message-ID: <7d75b62f0eb0495782c46ef8521790d5@ul-exc-pr-mbx13.ulaval.ca>

IMO, you should keep storhaug and maintain it. At the beginning, we were with pacemaker and corosync. Then we move to storhaug with the upgrade to gluster 4.1.x. Now you are talking about going back like it was. Maybe it will be better with pacemake and corosync but the important is to have a solution that will be stable and maintained.

thanks
Renaud

De : gluster-users-bounces at gluster.org [mailto:gluster-users-bounces at gluster.org] De la part de Jim Kinney
Envoy? : 30 avril 2019 08:20
? : gluster-users at gluster.org; Jiffin Tony Thottan <jthottan at redhat.com>; gluster-users at gluster.org; Gluster Devel <gluster-devel at gluster.org>; gluster-maintainers at gluster.org; nfs-ganesha <nfs-ganesha at redhat.com>; devel at lists.nfs-ganesha.org
Objet : Re: [Gluster-users] Proposing to previous ganesha HA cluster solution back to gluster code as gluster-7 feature

+1!
I'm using nfs-ganesha in my next upgrade so my client systems can use NFS instead of fuse mounts. Having an integrated, designed in process to coordinate multiple nodes into an HA cluster will very welcome.
On April 30, 2019 3:20:11 AM EDT, Jiffin Tony Thottan <jthottan at redhat.com<mailto:jthottan at redhat.com>> wrote:

Hi all,

Some of you folks may be familiar with HA solution provided for nfs-ganesha by gluster using pacemaker and corosync.

That feature was removed in glusterfs 3.10 in favour for common HA project "Storhaug". Even Storhaug was not progressed

much from last two years and current development is in halt state, hence planning to restore old HA ganesha solution back

to gluster code repository with some improvement and targetting for next gluster release 7.

I have opened up an issue [1] with details and posted initial set of patches [2]

Please share your thoughts on the same

Regards,

Jiffin

[1] https://github.com/gluster/glusterfs/issues/663

[2] https://review.gluster.org/#/q/topic:rfc-663+(status:open+OR+status:merged)

--
Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190430/d83a772b/attachment-0001.html>

From hunter86_bg at yahoo.com  Tue Apr 30 13:29:56 2019
From: hunter86_bg at yahoo.com (Strahil Nikolov)
Date: Tue, 30 Apr 2019 13:29:56 -0000
Subject: [Gluster-devel] [Gluster-users] Proposing to previous ganesha
 HA clustersolution back to gluster code as gluster-7 feature
In-Reply-To: <geuj12ndtqkroi36h5xmrydi.1556629490318@email.android.com>
References: <geuj12ndtqkroi36h5xmrydi.1556629490318@email.android.com>
Message-ID: <1028413072.2343069.1556630991785@mail.yahoo.com>

Hi,

I'm posting this again as it got bounced.
Keep in mind that corosync/pacemaker? is hard for proper setup by new admins/users.

I'm still trying to remediate the effects of poor configuration at work.
Also, storhaug is nice for hyperconverged setups where the host is not only hosting bricks, but? other? workloads.
Corosync/pacemaker require proper fencing to be setup and most of the stonith resources 'shoot the other node in the head'.
I would be happy to see an easy to deploy (let say 'cluster.enable-ha-ganesha true') and gluster to be bringing up the Floating IPs and taking care of the NFS locks, so no disruption will be felt by the clients.

Still, this will be a lot of work to achieve.

Best Regards,
Strahil Nikolov

On Apr 30, 2019 15:19, Jim Kinney <jim.kinney at gmail.com> wrote:
>??
> +1!
> I'm using nfs-ganesha in my next upgrade so my client systems can use NFS instead of fuse mounts. Having an integrated, designed in process to coordinate multiple nodes into an HA cluster will very welcome.
> 
> On April 30, 2019 3:20:11 AM EDT, Jiffin Tony Thottan <jthottan at redhat.com> wrote:
>>??
>> Hi all,
>> 
>> Some of you folks may be familiar with HA solution provided for nfs-ganesha by gluster using pacemaker and corosync.
>> 
>> That feature was removed in glusterfs 3.10 in favour for common HA project "Storhaug". Even Storhaug was not progressed
>> 
>> much from last two years and current development is in halt state, hence planning to restore old HA ganesha solution back
>> 
>> to gluster code repository with some improvement and targetting for next gluster release 7.
>> 
>>??I have opened up an issue [1] with details and posted initial set of patches [2]
>> 
>> Please share your thoughts on the same
>> 
>> 
>> Regards,
>> 
>> Jiffin??
>> 
>> [1] https://github.com/gluster/glusterfs/issues/663
>> 
>> [2] https://review.gluster.org/#/q/topic:rfc-663+(status:open+OR+status:merged)
>> 
>> 
> 
> -- 
> Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.
Keep in mind that corosync/pacemaker? is hard for proper setup by new admins/users.

I'm still trying to remediate the effects of poor configuration at work.
Also, storhaug is nice for hyperconverged setups where the host is not only hosting bricks, but? other? workloads.
Corosync/pacemaker require proper fencing to be setup and most of the stonith resources 'shoot the other node in the head'.
I would be happy to see an easy to deploy (let say 'cluster.enable-ha-ganesha true') and gluster to be bringing up the Floating IPs and taking care of the NFS locks, so no disruption will be felt by the clients.

Still, this will be a lot of work to achieve.

Best Regards,
Strahil NikolovOn Apr 30, 2019 15:19, Jim Kinney <jim.kinney at gmail.com> wrote:
>
> +1!
> I'm using nfs-ganesha in my next upgrade so my client systems can use NFS instead of fuse mounts. Having an integrated, designed in process to coordinate multiple nodes into an HA cluster will very welcome.
>
> On April 30, 2019 3:20:11 AM EDT, Jiffin Tony Thottan <jthottan at redhat.com> wrote:
>>
>> Hi all,
>>
>> Some of you folks may be familiar with HA solution provided for nfs-ganesha by gluster using pacemaker and corosync.
>>
>> That feature was removed in glusterfs 3.10 in favour for common HA project "Storhaug". Even Storhaug was not progressed
>>
>> much from last two years and current development is in halt state, hence planning to restore old HA ganesha solution back
>>
>> to gluster code repository with some improvement and targetting for next gluster release 7.
>>
>> I have opened up an issue [1] with details and posted initial set of patches [2]
>>
>> Please share your thoughts on the same
>>
>> Regards,
>>
>> Jiffin??
>>
>> [1] https://github.com/gluster/glusterfs/issues/663
>>
>> [2] https://review.gluster.org/#/q/topic:rfc-663+(status:open+OR+status:merged)
>
>
> -- 
> Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.