[Bugs] [Bug 1402621] High load one node, gluster fuse clients hang, heal info does not complete

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1402621



--- Comment #3 from denmat <tu2Bgone at gmail.com> ---
$ sudo gluster volume heal marketplace_nfs info
Brick 10.90.5.105:/data/data0/marketplace_nfs
Status: Transport endpoint is not connected
Number of entries: -

Brick 10.90.3.14:/data/data3/marketplace_nfs
<gfid:5bba3981-5a34-4fae-9efc-12dc4638baaa>
...
<output removed>
...
Status: Connected
Number of entries: 146

Brick 10.90.4.195:/data/data0/marketplace_nfs
<gfid:53834b40-8bb6-4d79-a393-46daaaf36f13>
...
<output removed>
...
Status: Connected
Number of entries: 142

Brick 10.90.5.105:/data/data1/marketplace_nfs
Status: Connected
Number of entries: 0

Brick 10.90.3.14:/data/data1/marketplace_nfs
Status: Connected
Number of entries: 0

Brick 10.90.4.195:/data/data1/marketplace_nfs
Status: Connected
Number of entries: 0

$ sudo gluster v status
Status of volume: marketplace_nfs
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.90.5.105:/data/data0/marketplace_n
fs                                          49152     0          Y       3426
Brick 10.90.3.14:/data/data3/marketplace_nf
s                                           49154     0          Y       3402
Brick 10.90.4.195:/data/data0/marketplace_n
fs                                          49152     0          Y       4868
Brick 10.90.5.105:/data/data1/marketplace_n
fs                                          49153     0          Y       31636
Brick 10.90.3.14:/data/data1/marketplace_nf
s                                           49153     0          Y       348
Brick 10.90.4.195:/data/data1/marketplace_n
fs                                          49153     0          Y       31238
NFS Server on localhost                     2049      0          Y       20526
Self-heal Daemon on localhost               N/A       N/A        Y       20534
NFS Server on ip-10-90-5-105.ec2.internal   2049      0          Y       1488
Self-heal Daemon on ip-10-90-5-105.ec2.inte
rnal                                        N/A       N/A        Y       1496
NFS Server on 10.90.3.14                    2049      0          Y       3999
Self-heal Daemon on 10.90.3.14              N/A       N/A        Y       4008

Task Status of Volume marketplace_nfs
------------------------------------------------------------------------------
There are no active volume tasks

Status of volume: marketplace_uploads
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.90.4.195:/data/data2/uploads       49154     0          Y       20506
Brick 10.90.3.14:/data/data2/uploads        49155     0          Y       3976
Brick 10.90.5.105:/data/data2/uploads       49154     0          Y       1468
NFS Server on localhost                     2049      0          Y       20526
Self-heal Daemon on localhost               N/A       N/A        Y       20534
NFS Server on ip-10-90-5-105.ec2.internal   2049      0          Y       1488
Self-heal Daemon on ip-10-90-5-105.ec2.inte
rnal                                        N/A       N/A        Y       1496
NFS Server on 10.90.3.14                    2049      0          Y       3999
Self-heal Daemon on 10.90.3.14              N/A       N/A        Y       4008

Task Status of Volume marketplace_uploads
------------------------------------------------------------------------------
There are no active volume tasks


It has happened again. We shutdown some hosts and the heal info started
completing.
You can see the host with the high load is reporting
Brick 10.90.5.105:/data/data0/marketplace_nfs
Status: Transport endpoint is not connected
Number of entries: -

Will attach another statedump from this node.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.