[Gluster-users] Rebalance task fails
Nithya Balachandran
nbalacha at redhat.com
Thu Jul 13 08:45:43 UTC 2017
On 13 July 2017 at 14:02, Szymon Miotk <szymon.miotk at gmail.com> wrote:
> Hi Nithya,
> I see index in context:
> [2017-07-07 10:07:18.230202] E [MSGID: 106062]
> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict]
> 0-glusterd: failed to get index
>
> I wonder if there is anything I can do to fix it.
> I was trying to strace gluster process but still have no clue what
> exactly is gluster index.
>
>
This message looks like a glusterd log message (not rebalance) and
glusterd_volume_rebalance_use_rsp_dict possibly failed because the
rebalance process had already crashed. The index here is simply the value
of the number of nodes on which the rebalance process should be running -
it is used to track the rebalance status on all nodes.
Best regards,
> Szymon Miotk
>
> On Thu, Jul 13, 2017 at 10:12 AM, Nithya Balachandran
> <nbalacha at redhat.com> wrote:
> > Hi Szymon,
> >
> > I have received the files and will take a look and get back to you.
> >
> > In what context are you seeing index?
> >
> > Thanks,
> > Nithya
> >
> > On 11 July 2017 at 01:15, Szymon Miotk <szymon.miotk at gmail.com> wrote:
> >>
> >> Hi Nithya,
> >>
> >> the files were sent to priv to avoid spamming the list with large
> >> attachments.
> >> Could someone explain what is index in Gluster?
> >> Unfortunately index is popular word, so googling is not very helpful.
> >>
> >> Best regards,
> >> Szymon Miotk
> >>
> >> On Sun, Jul 9, 2017 at 6:37 PM, Nithya Balachandran <
> nbalacha at redhat.com>
> >> wrote:
> >> >
> >> > On 7 July 2017 at 15:42, Szymon Miotk <szymon.miotk at gmail.com> wrote:
> >> >>
> >> >> Hello everyone,
> >> >>
> >> >>
> >> >> I have problem rebalancing Gluster volume.
> >> >> Gluster version is 3.7.3.
> >> >> My 1x3 replicated volume become full, so I've added three more bricks
> >> >> to make it 2x3 and wanted to rebalance.
> >> >> But every time I start rebalancing, it fails immediately.
> >> >> Rebooting Gluster nodes doesn't help.
> >> >>
> >> >> # gluster volume rebalance gsae_artifactory_cluster_storage start
> >> >> volume rebalance: gsae_artifactory_cluster_storage: success:
> Rebalance
> >> >> on gsae_artifactory_cluster_storage has been started successfully.
> Use
> >> >> rebalance status command to check status of the rebalance process.
> >> >> ID: b22572ff-7575-4557-8317-765f7e52d445
> >> >>
> >> >> # gluster volume rebalance gsae_artifactory_cluster_storage status
> >> >> Node Rebalanced-files
> >> >> size scanned failures skipped status
> >> >> run time in secs
> >> >> --------- -----------
> >> >> ----------- ----------- ----------- -----------
> >> >> ------------ --------------
> >> >> localhost 0
> >> >> 0Bytes 0 0 0 failed
> >> >> 0.00
> >> >> 10.239.40.9 0
> >> >> 0Bytes 0 0 0 failed
> >> >> 0.00
> >> >> 10.239.40.8 0
> >> >> 0Bytes 0 0 0 failed
> >> >> 0.00
> >> >> volume rebalance: gsae_artifactory_cluster_storage: success:
> >> >>
> >> >> The messages in logfiles mention 'failed to get index':
> >> >> [2017-07-07 10:07:18.230202] E [MSGID: 106062]
> >> >> [glusterd-utils.c:7997:glusterd_volume_rebalance_use_rsp_dict]
> >> >> 0-glusterd: failed to get index
> >> >>
> >> >> and then the rebalance process crashes:
> >> >> pending frames:
> >> >> frame : type(0) op(0)
> >> >> frame : type(0) op(0)
> >> >> frame : type(0) op(0)
> >> >> patchset: git://git.gluster.com/glusterfs.git
> >> >> signal received: 4
> >> >> time of crash:
> >> >> 2017-07-07 10:07:23
> >> >> configuration details:
> >> >> argp 1
> >> >> backtrace 1
> >> >> dlfcn 1
> >> >> libpthread 1
> >> >> llistxattr 1
> >> >> setfsid 1
> >> >> spinlock 1
> >> >> epoll.h 1
> >> >> xattr.h 1
> >> >> st_atim.tv_nsec 1
> >> >> package-string: glusterfs 3.7.3
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(_gf_msg_
> backtrace_nomem+0x92)[0x7f24de214502]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(gf_print_
> trace+0x31d)[0x7f24de23059d]
> >> >> /lib/x86_64-linux-gnu/libc.so.6(+0x36d40)[0x7f24dd612d40]
> >> >> /lib/x86_64-linux-gnu/libpthread.so.0(+0x11f6b)[0x7f24dd9b2f6b]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(inode_ref+
> 0x19)[0x7f24de234e69]
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(loc_copy+
> 0x4a)[0x7f24de21291a]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/
> cluster/distribute.so(dht_local_init+0x4b)[0x7f24d851f51b]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/
> cluster/distribute.so(dht_lookup+0x91)[0x7f24d8550521]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(syncop_
> lookup+0x1a2)[0x7f24de258fc2]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/
> cluster/distribute.so(gf_defrag_fix_layout+0x87)[0x7f24d85289e7]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/glusterfs/3.7.3/xlator/
> cluster/distribute.so(gf_defrag_start_crawl+0x6d3)[0x7f24d8529ce3]
> >> >>
> >> >>
> >> >> /usr/lib/x86_64-linux-gnu/libglusterfs.so.0(synctask_
> wrap+0x12)[0x7f24de255822]
> >> >> /lib/x86_64-linux-gnu/libc.so.6(+0x498b0)[0x7f24dd6258b0]
> >> >>
> >> >>
> >> >> Anybody has a clue how to fix 'failed to get index' error?
> >> >
> >> >
> >> > Do you have the coredump? If yes, please send that and the rebalance
> >> > logs
> >> > across.
> >> >
> >> > Regards,
> >> > Nithya
> >> >>
> >> >>
> >> >> Thank you in advance!
> >> >> Szymon Miotk
> >> >> _______________________________________________
> >> >> Gluster-users mailing list
> >> >> Gluster-users at gluster.org
> >> >> http://lists.gluster.org/mailman/listinfo/gluster-users
> >> >
> >> >
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170713/b87f937a/attachment.html>
More information about the Gluster-users
mailing list