[Gluster-users] Rebalance task fails

[Szymon Miotk]

SOLVED:
Problem was caused by bug affecting servers with more than 40 CPUs
visible - https://review.gluster.org/#/c/15062/
Rebalance works after upgrading to 3.10.4.

Many thanks to Nithya for help!

Best regards,
Szymon Miotk

On Thu, Jul 13, 2017 at 10:32 AM, Szymon Miotk <szymon.miotk at gmail.com> wrote:
> Hi Nithya,
> I see index in context:
> [2017-07-07 10:07:18.230202] E [MSGID: 106062]
> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict]
> 0-glusterd: failed to get index
>
> I wonder if there is anything I can do to fix it.
> I was trying to strace gluster process but still have no clue what
> exactly is gluster index.
>
> Best regards,
> Szymon Miotk
>
> On Thu, Jul 13, 2017 at 10:12 AM, Nithya Balachandran
> <nbalacha at redhat.com> wrote:
>> Hi Szymon,
>>
>> I have received the files and will take a look and get back to you.
>>
>> In what context are you seeing index?
>>
>> Thanks,
>> Nithya
>>
>> On 11 July 2017 at 01:15, Szymon Miotk <szymon.miotk at gmail.com> wrote:
>>>
>>> Hi Nithya,
>>>
>>> the files were sent to priv to avoid spamming the list with large
>>> attachments.
>>> Could someone explain what is index in Gluster?
>>> Unfortunately index is popular word, so googling is not very helpful.
>>>
>>> Best regards,
>>> Szymon Miotk
>>>
>>> On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <nbalacha at redhat.com>
>>> wrote:
>>> >
>>> > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk at gmail.com> wrote:
>>> >>
>>> >> Hello everyone,
>>> >>
>>> >>
>>> >> I have problem rebalancing Gluster volume.
>>> >> Gluster version is 3.7.3.
>>> >> My 1x3 replicated volume become full, so I've added three more bricks
>>> >> to make it 2x3 and wanted to rebalance.
>>> >> But every time I start rebalancing, it fails immediately.
>>> >> Rebooting Gluster nodes doesn't help.
>>> >>
>>> >> # gluster volume rebalance  gsae_artifactory_cluster_storage start
>>> >> volume rebalance: gsae_artifactory_cluster_storage: success: Rebalance
>>> >> on gsae_artifactory_cluster_storage has been started successfully. Use
>>> >> rebalance status command to check status of the rebalance process.
>>> >> ID: b22572ff-7575-4557-8317-765f7e52d445
>>> >>
>>> >> # gluster volume rebalance  gsae_artifactory_cluster_storage status
>>> >>                                     Node Rebalanced-files
>>> >> size       scanned      failures       skipped               status
>>> >> run time in secs
>>> >>                                ---------      -----------
>>> >> -----------   -----------   -----------   -----------
>>> >> ------------     --------------
>>> >>                                localhost                0
>>> >> 0Bytes             0             0             0               failed
>>> >>              0.00
>>> >>                              10.239.40.9                0
>>> >> 0Bytes             0             0             0               failed
>>> >>              0.00
>>> >>                              10.239.40.8                0
>>> >> 0Bytes             0             0             0               failed
>>> >>              0.00
>>> >> volume rebalance: gsae_artifactory_cluster_storage: success:
>>> >>
>>> >> The messages in logfiles mention 'failed to get index':
>>> >> [2017-07-07 10:07:18.230202] E [MSGID: 106062]
>>> >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict]
>>> >> 0-glusterd: failed to get index
>>> >>
>>> >> and then the rebalance process crashes:
>>> >> pending frames:
>>> >> frame : type(0) op(0)
>>> >> frame : type(0) op(0)
>>> >> frame : type(0) op(0)
>>> >> patchset: git://git.gluster.com/glusterfs.git
>>> >> signal received: 4
>>> >> time of crash:
>>> >> 2017-07-07 10:07:23
>>> >> configuration details:
>>> >> argp 1
>>> >> backtrace 1
>>> >> dlfcn 1
>>> >> libpthread 1
>>> >> llistxattr 1
>>> >> setfsid 1
>>> >> spinlock 1
>>> >> epoll.h 1
>>> >> xattr.h 1
>>> >> st_atim.tv_nsec 1
>>> >> package-string: glusterfs 3.7.3
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_backtrace_nomem+0x92)[0x7f24de214502]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_trace+0x31d)[0x7f24de23059d]
>>> >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40]
>>> >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+0x19)[0x7f24de234e69]
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+0x4a)[0x7f24de21291a]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_lookup+0x1a2)[0x7f24de258fc2]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3]
>>> >>
>>> >>
>>> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_wrap+0x12)[0x7f24de255822]
>>> >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0]
>>> >>
>>> >>
>>> >> Anybody has a clue how to fix 'failed to get index' error?
>>> >
>>> >
>>> > Do you have the coredump? If yes, please send that and the rebalance
>>> > logs
>>> > across.
>>> >
>>> > Regards,
>>> > Nithya
>>> >>
>>> >>
>>> >> Thank you in advance!
>>> >> Szymon Miotk
>>> >> _______________________________________________
>>> >> Gluster-users mailing list
>>> >> Gluster-users at gluster.org
>>> >> http://lists.gluster.org/mailman/listinfo/gluster-users
>>> >
>>> >
>>
>>