[Gluster-users] Replace Brick fails (gluster 3.2.5)
Niels de Vos
ndevos at redhat.com
Mon Dec 1 11:08:07 UTC 2014
On Sat, Nov 29, 2014 at 11:23:07PM +0100, Heiko Schröter wrote:
> sorry, at least for the next 18 month i'am stuck with 3.2.5 (Lifetime of
> project).
> But maybe someone from the dev team could give a hint where to start
> debugging with these error messages.
> It seems to be related to the "rpc XDR" encoding mechanism. The only thing
> i've found so far is that it might be a "LOCALE" probelm. I checked the
> locale settings. They where indeed different.
> So recompiled everything with correct locale but still get the error.
> "strace" gives me no real clue.
If there is an issue with the locale, recompiling will likely not be
helping. You would need to set the locale for the running processes in
their environment. The easiest is probably to add 'export LC_ALL=C' in
the startup scripts (maybe more, see 'man locale').
Other than that, the locale that is set in the environment of where the
'gluster' commandline is started may be important too, depending on
where the error originates.
Potentially related:
- http://review.gluster.org/8299
- https://bugzilla.redhat.com/1117951
HTH,
Niels
>
> Is there a way to debug or trace the gluster RPC XDR encoding mechanism ?
>
> Heiko
>
>
> [2014-11-29 23:10:52.912545] W [rpc-common.c:38:xdr_serialize_generic] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_op_send_cli_response+0x97)
> [0x7f801881d7e7] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_submit_reply+0x65)
> [0x7f801880f695] (-->/usr/lib/glusterfs/3.2.5qa9/xlator/mgmt/glusterd.so(glusterd_serialize_reply+0x56)
> [0x7f801880f576]))) 0-xdr: XDR encoding failed
> [2014-11-29 23:10:52.912556] E
> [glusterd-utils.c:404:glusterd_serialize_reply] 0-: Failed to encode message
> [2014-11-29 23:10:52.912561] E [glusterd-utils.c:446:glusterd_submit_reply]
> 0-: Failed to serialize reply
>
>
> strace end:
> .....
> read(5, "# /etc/services\n#\n# Network serv"..., 4096) = 4096
> read(5, " private\t77/tcp\t\t\t\t# any private"..., 4096) = 4096
> read(5, "e\nemfis-cntl\t141/udp\nimap\t\t143/t"..., 4096) = 4096
> read(5, "dialog\t360/tcp\t\t\t\t# scoi2odialog"..., 4096) = 4096
> read(5, "\t\tdqs313_intercell\ncryptoadmin\t6"..., 4096) = 4096
> read(5, "# Citrix ICA Client\nica\t\t1494/ud"..., 4096) = 4096
> read(5, "05/udp\nlstp\t\t2559/tcp\t\t\t# \nlstp\t"..., 4096) = 4096
> read(5, "t-pmp\t\t5351/udp\ndns-llq\t\t5352/tc"..., 4096) = 4096
> read(5, "p\t\t\t# OpenPGP HTTP Keyserver\nhkp"..., 4096) = 3373
> read(5, "", 4096) = 0
> close(5) = 0
> munmap(0x7f8cf229a000, 4096) = 0
> socket(PF_INET, SOCK_STREAM, IPPROTO_IP) = 5
> setsockopt(5, SOL_SOCKET, SO_RCVBUF, [524288], 4) = 0
> setsockopt(5, SOL_SOCKET, SO_SNDBUF, [524288], 4) = 0
> setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0
> fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)
> fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0
> setsockopt(5, SOL_SOCKET, SO_KEEPALIVE, [1], 4) = 0
> setsockopt(5, SOL_TCP, TCP_KEEPIDLE, [20], 4) = 0
> setsockopt(5, SOL_TCP, TCP_KEEPINTVL, [2], 4) = 0
> bind(5, {sa_family=AF_INET, sin_port=htons(1023),
> sin_addr=inet_addr("0.0.0.0")}, 16) = 0
> connect(5, {sa_family=AF_INET, sin_port=htons(24007),
> sin_addr=inet_addr("127.0.0.1")}, 16) = -1 EINPROGRESS (Operation now in
> progress)
> epoll_ctl(3, EPOLL_CTL_ADD, 5, {EPOLLIN|EPOLLPRI|EPOLLOUT, {u32=5, u64=5}})
> = 0
> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f8cef018000
> mprotect(0x7f8cef018000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7f8cef817ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7f8cef8189d0, tls=0x7f8cef818700,
> child_tidptr=0x7f8cef8189d0) = 2840
> mmap(NULL, 8392704, PROT_READ|PROT_WRITE,
> MAP_PRIVATE|MAP_ANONYMOUS|MAP_STACK, -1, 0) = 0x7f8cee817000
> mprotect(0x7f8cee817000, 4096, PROT_NONE) = 0
> clone(child_stack=0x7f8cef016ff0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID,
> parent_tidptr=0x7f8cef0179d0, tls=0x7f8cef017700,
> child_tidptr=0x7f8cef0179d0) = 2841
> epoll_wait(3, {{EPOLLOUT, {u32=5, u64=5}}}, 257, -1) = 1
> getsockopt(5, SOL_SOCKET, SO_ERROR, [0], [4]) = 0
> getsockname(5, {sa_family=AF_INET, sin_port=htons(1023),
> sin_addr=inet_addr("127.0.0.1")}, [16]) = 0
> futex(0x621ea4, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x621e60, 2) = 1
> epoll_ctl(3, EPOLL_CTL_MOD, 5, {EPOLLIN|EPOLLPRI, {u32=5, u64=5}}) = 0
> epoll_wait(3, {{EPOLLIN, {u32=5, u64=5}}}, 257, -1) = 1
> readv(5, [{"\200\2\0\30", 4}], 1) = 4
> readv(5, [{"\0\0\0\1\0\0\0\1", 8}], 1) = 8
> readv(5,
> [{"\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\1\0\0\0\1\0\0\0\0\0\0\0\0"...,
> 131088}], 1) = 131088
> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2309, ...}) = 0
> write(4, "[2014-11-29 23:21:41.942744] I ["..., 118) = 118
> fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
> mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
> 0x7f8cf229a000
> write(1, "replace-brick failed to start\n", 30replace-brick failed to start
> ) = 30
> futex(0x621e24, FUTEX_CMP_REQUEUE_PRIVATE, 1, 2147483647, 0x621de0, 2) = 1
> epoll_wait(3, <unfinished ...>
> +++ exited with 1 +++
>
>
>
>
>
>
> Am 28.11.2014 um 12:50 schrieb Atin Mukherjee:
> >3.2.5 is too old, can you please upgrade your cluster to recent version
> >of glusterfs bits and try it out?
> >
> >~Atin
> >
> >On 11/28/2014 05:17 PM, Heiko Schröter wrote:
> >>Unable to set cli op
> >
> >
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 181 bytes
Desc: not available
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20141201/f87ac5b8/attachment.sig>
More information about the Gluster-users
mailing list