[Gluster-users] Rebalance failing on fix-layout

[Jarsulic, Michael [CRI]]

Hello,

The past couple of weeks I had some issues with firmware on the OS hard drives in my gluster cluster. I have recently fixed the issue, and am bringing my bricks back into the volume. I am running gluster 3.7.6 and am running into the following issue:

When I add the brick and rebalance, the operation fails after a couple minutes. The errors I find in the rebalance log is this:

[2017-06-05 13:38:40.441671] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C gfid not present
[2017-06-05 13:38:40.450341] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C/NoCov_NoImm gfid not present
[2017-06-05 13:38:40.450380] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C/simulate gfid not present
[2017-06-05 13:38:40.459365] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C/NoCov_NoImm/fits_generate gfid not present
[2017-06-05 13:38:40.468756] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/code/C/NoCov_NoImm/fits_generate/N_0_vector.dat gfid not present
[2017-06-05 13:38:40.495645] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: LV_Fitting/code/C/simulate/RK45_Integrate.c gfid not present
[2017-06-05 13:38:40.512336] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht: /LV_Fitting/output/line_search gfid not present
[2017-06-05 13:38:40.512373] E [MSGID: 109010] [dht-rebalance.c:2259:gf_defrag_get_entry] 0-hpcscratch-dht:  /LV_Fitting/output/mcmc gfid not present
[2017-06-05 13:38:40.517808] E [dht-rebalance.c:2992:gf_defrag_fix_layout] 0-hpcscratch-dht: Setxattr failed for /LV_Fitting/output/line_search
[2017-06-05 13:38:40.518025] E [MSGID: 109016] [dht-rebalance.c:3006:gf_defrag_fix_layout] 0-hpcscratch-dht: Fix layout failed for /LV_Fitting/output
[2017-06-05 13:38:40.518136] E [MSGID: 109016] [dht-rebalance.c:3006:gf_defrag_fix_layout] 0-hpcscratch-dht: Fix layout failed for /LV_Fitting


There are about 102,000 of the gfid error, but only a few errors for the fix layout failed. Is there any way to recover from this issue?

--
Mike Jarsulic
Sr. HPC Administrator
Center for Research Informatics | University of Chicago
773.702.2066