[Gluster-devel] 【replace-brick failed but make there’re two same client-id of the gluster cluster, which lead can’t mount the gluster anymore】

[Frank Yu]

Hi guys,


I met a terrible situations need all your helps. I have a production
cluster running well at first. the version of gluster is
3.12.15-1.el7.x86_64, the cluster has 12 nodes, 12 brick(disk) per nodes,
there is one distributed-replicate volume, with 144 bricks(48 * 3).


then there is a node crushed(the node named nodeA), and all it’s disk can’t
be used anymore, but since the os of nodes run on kvm machine, so it came
back with 12 new disks.


I try to replace the first brick of nodeA with cmd ‘gluster volume
replace-brick VOLUMENAME nodeA:/mnt/data-1/data  nodeA:/mnt/data-1/data01
commit force’, after some times, it failed with error “Error : Request
timed out”.


here came the problem, both ‘nodeA:/mnt/data-1/data’ and
‘nodeA:/mnt/data-1/data01’ show in the output of cmd ‘gluster volume info’


When I try to mount gluster to client with fuse, it report error like
below:


[2019-08-12 12:27:42.395440] I [MSGID: 100030] [glusterfsd.c:2511:main]
0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.12.15
(args: /usr/sbin/glusterfs --volfile-server=xxxxx
--volfile-id=/training-data-ali /mnt/glusterfs)

[2019-08-12 12:27:42.400015] W [MSGID: 101002]
[options.c:995:xl_opt_validate] 0-glusterfs: option 'address-family' is
deprecated, preferred is 'transport.address-family', continuing with
correction

[2019-08-12 12:27:42.404994] I [MSGID: 101190]
[event-epoll.c:613:event_dispatch_epoll_worker] 0-epoll: Started thread
with index 1

*[2019-08-12 12:27:42.415971] E [MSGID: 101179] [graph.y:153:new_volume]
0-parser: Line 1381: volume ‘VOLUME-NAME-client-74' defined again*

[2019-08-12 12:27:42.416124] E [MSGID: 100026]
[glusterfsd.c:2358:glusterfs_process_volfp] 0-: failed to construct the
graph

[2019-08-12 12:27:42.416376] E [graph.c:1102:glusterfs_graph_destroy]
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092]
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x150) [0x55898e357da0]
-->/lib64/libglusterfs.so.0(glusterfs_graph_destroy+0x84) [0x7f95f7318754]
) 0-graph: invalid argument: graph [Invalid argument]

[2019-08-12 12:27:42.416425] W [glusterfsd.c:1375:cleanup_and_exit]
(-->/usr/sbin/glusterfs(mgmt_getspec_cbk+0x532) [0x55898e35e092]
-->/usr/sbin/glusterfs(glusterfs_process_volfp+0x163) [0x55898e357db3]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-:
received signum (-1), shutting down

[2019-08-12 12:27:42.416455] I [fuse-bridge.c:5852:fini] 0-fuse: Unmounting
'/mnt/glusterfs'.

[2019-08-12 12:27:42.429655] I [fuse-bridge.c:5857:fini] 0-fuse: Closing
fuse connection to '/mnt/glusterfs-aliyun'.

[2019-08-12 12:27:42.429759] W [glusterfsd.c:1375:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7e25) [0x7f95f6140e25]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xe5) [0x55898e3574b5]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x6b) [0x55898e35732b] ) 0-:
received signum (15), shutting down


So, how can I solve error *“Line 1381: volume ‘VOLUME-NAME-client-74'
defined again” *

-- 
Regards
Frank Yu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20190812/bb176b28/attachment.html>