[Gluster-users] Not able to start glusterd

Wed Mar 6 07:28:08 UTC 2019

Abhishek,

We need below information on investigate this issue.
1. gluster --version
2. Please run glusterd in gdb, so that we can capture the backtrace. I see
some rpc errors in log, but backtrace will be more helpful.
    To run glusterd in gdb, you need start glusterd in gdb (i.e. gdb
glusterd, and then give the command "run -N"). when you see a segmentation
         fault, please capture the backtrace and paste it here.

On Wed, Mar 6, 2019 at 10:07 AM ABHISHEK PALIWAL <abhishpaliwal at gmail.com>
wrote:

> Hi Team,
>
> I am facing the issue where at the time of starting the glusterd
> segmentation fault is reported.
>
> Below are the logs
>
> root at 128:/usr/sbin# ./glusterd  --debug
> [1970-01-01 15:19:43.940386] I [MSGID: 100030] [glusterfsd.c:2691:main]
> 0-./glusterd: Started running ./glusterd version 5.0 (args: ./glusterd
> --debug)
> [1970-01-01 15:19:43.940855] D
> [logging.c:1833:__gf_log_inject_timer_event] 0-logging-infra: Starting
> timer now. Timeout = 120, current buf size = 5
> [1970-01-01 15:19:43.941736] D [MSGID: 0] [glusterfsd.c:747:get_volfp]
> 0-glusterfsd: loading volume file /etc/glusterfs/glusterd.vol
> [1970-01-01 15:19:43.945796] D [MSGID: 101097]
> [xlator.c:341:xlator_dynload_newway] 0-xlator: dlsym(xlator_api) on
> /usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so: undefined symbol:
> xlator_api. Fall back to old symbols
> [1970-01-01 15:19:43.946279] I [MSGID: 106478] [glusterd.c:1435:init]
> 0-management: Maximum allowed open file descriptors set to 65536
> [1970-01-01 15:19:43.946419] I [MSGID: 106479] [glusterd.c:1491:init]
> 0-management: Using /var/lib/glusterd as working directory
> [1970-01-01 15:19:43.946515] I [MSGID: 106479] [glusterd.c:1497:init]
> 0-management: Using /var/run/gluster as pid file working directory
> [1970-01-01 15:19:43.946968] D [MSGID: 0]
> [glusterd.c:458:glusterd_rpcsvc_options_build] 0-glusterd: listen-backlog
> value: 10
> [1970-01-01 15:19:43.947139] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service:
> RPC service inited.
> [1970-01-01 15:19:43.947241] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1,
> Port: 0
> [1970-01-01 15:19:43.947379] D [rpc-transport.c:269:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/5.0/rpc-transport/socket.so
> [1970-01-01 15:19:43.955198] D [socket.c:4464:socket_init]
> 0-socket.management: Configued transport.tcp-user-timeout=0
> [1970-01-01 15:19:43.955316] D [socket.c:4482:socket_init]
> 0-socket.management: Reconfigued transport.keepalivecnt=9
> [1970-01-01 15:19:43.955415] D [socket.c:4167:ssl_setup_connection_params]
> 0-socket.management: SSL support on the I/O path is NOT enabled
> [1970-01-01 15:19:43.955504] D [socket.c:4170:ssl_setup_connection_params]
> 0-socket.management: SSL support for glusterd is NOT enabled
> [1970-01-01 15:19:43.955612] D [name.c:572:server_fill_address_family]
> 0-socket.management: option address-family not specified, defaulting to
> inet6
> [1970-01-01 15:19:43.955928] D [rpc-transport.c:269:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so
> [1970-01-01 15:19:43.956079] E [rpc-transport.c:273:rpc_transport_load]
> 0-rpc-transport: /usr/lib64/glusterfs/5.0/rpc-transport/rdma.so: cannot
> open shared object file: No such file or directory
> [1970-01-01 15:19:43.956177] W [rpc-transport.c:277:rpc_transport_load]
> 0-rpc-transport: volume 'rdma.management': transport-type 'rdma' is not
> valid or not found on this machine
> [1970-01-01 15:19:43.956270] W [rpcsvc.c:1789:rpcsvc_create_listener]
> 0-rpc-service: cannot create listener, initing the transport failed
> [1970-01-01 15:19:43.956362] E [MSGID: 106244] [glusterd.c:1798:init]
> 0-management: creation of 1 listeners failed, continuing with succeeded
> transport
> [1970-01-01 15:19:43.956459] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc peer, Num: 1238437,
> Ver: 2, Port: 0
> [1970-01-01 15:19:43.956561] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc cli read-only, Num:
> 1238463, Ver: 2, Port: 0
> [1970-01-01 15:19:43.956666] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc mgmt, Num: 1238433,
> Ver: 2, Port: 0
> [1970-01-01 15:19:43.956758] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc mgmt v3, Num: 1238433,
> Ver: 3, Port: 0
> [1970-01-01 15:19:43.956853] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster Portmap, Num: 34123456, Ver:
> 1, Port: 0
> [1970-01-01 15:19:43.956946] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster Handshake, Num: 14398633,
> Ver: 2, Port: 0
> [1970-01-01 15:19:43.957062] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster MGMT Handshake, Num:
> 1239873, Ver: 1, Port: 0
> [1970-01-01 15:19:43.957205] D [rpcsvc.c:2607:rpcsvc_init] 0-rpc-service:
> RPC service inited.
> [1970-01-01 15:19:43.957303] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GF-DUMP, Num: 123451501, Ver: 1,
> Port: 0
> [1970-01-01 15:19:43.957408] D [rpc-transport.c:269:rpc_transport_load]
> 0-rpc-transport: attempt to load file
> /usr/lib64/glusterfs/5.0/rpc-transport/socket.so
> [1970-01-01 15:19:43.957563] D [socket.c:4424:socket_init]
> 0-socket.management: disabling nodelay
> [1970-01-01 15:19:43.957650] D [socket.c:4464:socket_init]
> 0-socket.management: Configued transport.tcp-user-timeout=0
> [1970-01-01 15:19:43.957738] D [socket.c:4482:socket_init]
> 0-socket.management: Reconfigued transport.keepalivecnt=9
> [1970-01-01 15:19:43.957830] D [socket.c:4167:ssl_setup_connection_params]
> 0-socket.management: SSL support on the I/O path is NOT enabled
> [1970-01-01 15:19:43.957922] D [socket.c:4170:ssl_setup_connection_params]
> 0-socket.management: SSL support for glusterd is NOT enabled
> [1970-01-01 15:19:43.958186] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: GlusterD svc cli, Num: 1238463, Ver:
> 2, Port: 0
> [1970-01-01 15:19:43.958280] D [rpcsvc.c:2146:rpcsvc_program_register]
> 0-rpc-service: New program registered: Gluster Handshake (CLI Getspec),
> Num: 14398633, Ver: 2, Port: 0
> [1970-01-01 15:19:43.958461] D [MSGID: 0]
> [glusterd-utils.c:7878:glusterd_sm_tr_log_init] 0-glusterd: returning 0
> [1970-01-01 15:19:43.958557] D [MSGID: 0] [glusterd.c:1875:init]
> 0-management: cannot get run-with-valgrind value
> [1970-01-01 15:19:43.960895] E [MSGID: 101032]
> [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [1970-01-01 15:19:43.961016] D [MSGID: 0]
> [store.c:452:gf_store_handle_retrieve] 0-: Returning -1
> [1970-01-01 15:19:43.961108] D [MSGID: 0]
> [glusterd-store.c:2169:glusterd_retrieve_op_version] 0-management: Unable
> to get store handle!
> [1970-01-01 15:19:43.961216] E [MSGID: 101032]
> [store.c:447:gf_store_handle_retrieve] 0-: Path corresponding to
> /var/lib/glusterd/glusterd.info. [No such file or directory]
> [1970-01-01 15:19:43.961325] D [MSGID: 0]
> [store.c:452:gf_store_handle_retrieve] 0-: Returning -1
> [1970-01-01 15:19:43.961428] D [MSGID: 0]
> [glusterd-store.c:2345:glusterd_retrieve_uuid] 0-management: Unable to get
> storehandle!
> [1970-01-01 15:19:43.961523] D [MSGID: 0]
> [glusterd-store.c:2366:glusterd_retrieve_uuid] 0-management: Returning -1
> [1970-01-01 15:19:43.961617] I [MSGID: 106514]
> [glusterd-store.c:2304:glusterd_restore_op_version] 0-management: Detected
> new install. Setting op-version to maximum : 50000
> [1970-01-01 15:19:43.962658] D [MSGID: 0]
> [store.c:432:gf_store_handle_new] 0-: Returning 0
> [1970-01-01 15:19:43.962750] D [MSGID: 0]
> [store.c:452:gf_store_handle_retrieve] 0-: Returning 0
> [1970-01-01 15:19:43.963047] D [MSGID: 0] [store.c:515:gf_store_iter_new]
> 0-: Returning with 0
> [1970-01-01 15:19:43.963194] D [MSGID: 0]
> [store.c:632:gf_store_iter_get_next] 0-: Returning with 0
> [1970-01-01 15:19:43.963318] D [MSGID: 0]
> [store.c:632:gf_store_iter_get_next] 0-: Returning with -1
> [1970-01-01 15:19:43.963455] D [MSGID: 0]
> [store.c:473:gf_store_handle_destroy] 0-: Returning 0
> [1970-01-01 15:19:43.963757] D [MSGID: 0]
> [glusterd-store.c:3546:glusterd_store_retrieve_volumes] 0-management:
> Returning with 0
> [1970-01-01 15:19:43.964159] D [MSGID: 0]
> [glusterd-store.c:4662:glusterd_store_retrieve_peers] 0-management:
> Returning with 0
> [1970-01-01 15:19:43.964471] I [MSGID: 106194]
> [glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list]
> 0-management: No missed snaps list.
> [1970-01-01 15:19:43.964580] D [MSGID: 0]
> [glusterd-store.c:4104:glusterd_store_retrieve_snaps] 0-management:
> Returning with 0
> [1970-01-01 15:19:43.964680] D [MSGID: 0]
> [glusterd-store.c:4894:glusterd_restore] 0-management: Returning 0
> [1970-01-01 15:19:43.965060] D [MSGID: 0]
> [options.c:1225:xlator_option_init_int32] 0-management: option
> event-threads using set value 1
> Final graph:
>
> +------------------------------------------------------------------------------+
>   1: volume management
>   2:     type mgmt/glusterd
>   3:     option rpc-auth.auth-glusterfs on
>   4:     option rpc-auth.auth-unix on
>   5:     option rpc-auth.auth-null on
>   6:     option rpc-auth-allow-insecure on
>   7:     option transport.listen-backlog 10
>   8:     option event-threads 1
>   9:     option ping-timeout 0
>  10:     option transport.socket.read-fail-log off
>  11:     option transport.socket.keepalive-interval 2
>  12:     option transport.socket.keepalive-time 10
>  13:     option transport-type rdma
>  14:     option working-directory /var/lib/glusterd
>  15: end-volume
>  16:
>
> +------------------------------------------------------------------------------+
> [1970-01-01 15:19:43.966808] I [MSGID: 101190]
> [event-epoll.c:622:event_dispatch_epoll_worker] 0-epoll: Started thread
> with index 1
> [1970-01-01 15:19:44.840454] E [rpcsvc.c:513:rpcsvc_request_create]
> 0-rpc-service: RPC version not supported (XID: 0x0, Ver: 0, Program: 0,
> ProgVers: 0, Proc: 2) from trans (socket.management)
> [1970-01-01 15:19:44.840884] D [rpcsvc.c:1416:rpcsvc_error_reply] (-->
> /usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4]
> (--> /usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4] (-->
> /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00] (-->
> /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224] (-->
> /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c] ))))) 0-: sending a RPC
> error reply
> [1970-01-01 15:19:44.841055] D [logging.c:1805:gf_log_flush_extra_msgs]
> 0-logging-infra: Log buffer size reduced. About to flush 5 extra log
> messages
> [1970-01-01 15:19:44.841156] D [logging.c:1808:gf_log_flush_extra_msgs]
> 0-logging-infra: Just flushed 5 extra log messages
> pending frames:
> patchset: git://git.gluster.org/glusterfs.git
> signal received: 11
> time of crash:
> 1970-01-01 15:19:44
> configuration details:
> argp 1
> backtrace 1
> dlfcn 1
> libpthread 1
> llistxattr 1
> setfsid 1
> spinlock 1
> epoll.h 1
> xattr.h 1
> st_atim.tv_nsec 1
> package-string: glusterfs 5.0
> /usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4]
> /usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0]
> ./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec]
> linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478]
> /lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc]
> /lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c]
> /lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4]
> /lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0]
> /lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120]
> /usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c]
> /usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c]
> /usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614]
> /usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00]
> /usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224]
> /usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c]
> /usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c]
> /usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c]
> /usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc]
> /lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30]
> /lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c]
> ---------
> Segmentation fault (core dumped)
>
> Could you  please help me, what actually the problem?
>
>
> --
>
>
>
>
> Regards
> Abhishek Paliwal
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users

-- 
Thanks,
Sanju
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190306/51e35ec5/attachment.html>