<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi Abhishek,</p>
<p>Good to know that you have resolved your problem. Do you think
any more information is required to add in the upgrade doc for a
smooth upgrade flow. It would be great to see a PR to the repo
<a class="moz-txt-link-freetext" href="https://github.com/gluster/glusterdocs/tree/master/docs/Upgrade-Guide">https://github.com/gluster/glusterdocs/tree/master/docs/Upgrade-Guide</a>
for updating the doc with the information.</p>
<p><br>
</p>
<p>Regards</p>
<p>Rafi KC<br>
</p>
<div class="moz-cite-prefix">On 3/6/19 2:03 PM, ABHISHEK PALIWAL
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CA+15cFNM+8wd66biaONyV_HvXzchG=rJo8EL0yOknXAY0x1bKQ@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr">Hi Sanju,
<div><br>
</div>
<div>Thanks for the response.</div>
<div><br>
</div>
<div>I have resolved the issue, actually I have updated from
3.7.6 to 5.0, in new version RPC is coming from libtirpb , but
I forgot to enable "--with-libtirpc" in configuration.</div>
<div><br>
</div>
<div>After enabling able to start glusterd.</div>
<div><br>
</div>
<div>Regards,</div>
<div>Abhishek</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Mar 6, 2019 at 12:58
PM Sanju Rakonde <<a href="mailto:srakonde@redhat.com"
moz-do-not-send="true">srakonde@redhat.com</a>> wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">Abhishek,
<div><br>
</div>
<div>We need below information on investigate this issue.</div>
<div>1. gluster --version</div>
<div>2. Please run glusterd in gdb, so that we can capture
the backtrace. I see some rpc errors in log, but backtrace
will be more helpful.</div>
<div> To run glusterd in gdb, you need start glusterd in
gdb (i.e. gdb glusterd, and then give the command "run
-N"). when you see a segmentation fault, please
capture the backtrace and paste it here.</div>
</div>
<br>
<div class="gmail_quote">
<div dir="ltr" class="gmail_attr">On Wed, Mar 6, 2019 at
10:07 AM ABHISHEK PALIWAL <<a
href="mailto:abhishpaliwal@gmail.com" target="_blank"
moz-do-not-send="true">abhishpaliwal@gmail.com</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div dir="ltr">Hi Team,
<div><br>
</div>
<div>I am facing the issue where at the time of
starting the glusterd segmentation fault is
reported.</div>
<div><br>
</div>
<div>Below are the logs</div>
<div><br>
</div>
<div>
<div>root@128:/usr/sbin# ./glusterd --debug</div>
<div>[1970-01-01 15:19:43.940386] I [MSGID: 100030]
[glusterfsd.c:2691:main] 0-./glusterd: Started
running ./glusterd version 5.0 (args: ./glusterd
--debug)</div>
<div>[1970-01-01 15:19:43.940855] D
[logging.c:1833:__gf_log_inject_timer_event]
0-logging-infra: Starting timer now. Timeout =
120, current buf size = 5</div>
<div>[1970-01-01 15:19:43.941736] D [MSGID: 0]
[glusterfsd.c:747:get_volfp] 0-glusterfsd: loading
volume file /etc/glusterfs/glusterd.vol</div>
<div>[1970-01-01 15:19:43.945796] D [MSGID: 101097]
[xlator.c:341:xlator_dynload_newway] 0-xlator:
dlsym(xlator_api) on
/usr/lib64/glusterfs/5.0/xlator/mgmt/glusterd.so:
undefined symbol: xlator_api. Fall back to old
symbols</div>
<div>[1970-01-01 15:19:43.946279] I [MSGID: 106478]
[glusterd.c:1435:init] 0-management: Maximum
allowed open file descriptors set to 65536</div>
<div>[1970-01-01 15:19:43.946419] I [MSGID: 106479]
[glusterd.c:1491:init] 0-management: Using
/var/lib/glusterd as working directory</div>
<div>[1970-01-01 15:19:43.946515] I [MSGID: 106479]
[glusterd.c:1497:init] 0-management: Using
/var/run/gluster as pid file working directory</div>
<div>[1970-01-01 15:19:43.946968] D [MSGID: 0]
[glusterd.c:458:glusterd_rpcsvc_options_build]
0-glusterd: listen-backlog value: 10</div>
<div>[1970-01-01 15:19:43.947139] D
[rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC
service inited.</div>
<div>[1970-01-01 15:19:43.947241] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GF-DUMP,
Num: 123451501, Ver: 1, Port: 0</div>
<div>[1970-01-01 15:19:43.947379] D
[rpc-transport.c:269:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/5.0/rpc-transport/socket.so</div>
<div>[1970-01-01 15:19:43.955198] D
[socket.c:4464:socket_init] 0-socket.management:
Configued transport.tcp-user-timeout=0</div>
<div>[1970-01-01 15:19:43.955316] D
[socket.c:4482:socket_init] 0-socket.management:
Reconfigued transport.keepalivecnt=9</div>
<div>[1970-01-01 15:19:43.955415] D
[socket.c:4167:ssl_setup_connection_params]
0-socket.management: SSL support on the I/O path
is NOT enabled</div>
<div>[1970-01-01 15:19:43.955504] D
[socket.c:4170:ssl_setup_connection_params]
0-socket.management: SSL support for glusterd is
NOT enabled</div>
<div>[1970-01-01 15:19:43.955612] D
[name.c:572:server_fill_address_family]
0-socket.management: option address-family not
specified, defaulting to inet6</div>
<div>[1970-01-01 15:19:43.955928] D
[rpc-transport.c:269:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/5.0/rpc-transport/rdma.so</div>
<div>[1970-01-01 15:19:43.956079] E
[rpc-transport.c:273:rpc_transport_load]
0-rpc-transport:
/usr/lib64/glusterfs/5.0/rpc-transport/rdma.so:
cannot open shared object file: No such file or
directory</div>
<div>[1970-01-01 15:19:43.956177] W
[rpc-transport.c:277:rpc_transport_load]
0-rpc-transport: volume 'rdma.management':
transport-type 'rdma' is not valid or not found on
this machine</div>
<div>[1970-01-01 15:19:43.956270] W
[rpcsvc.c:1789:rpcsvc_create_listener]
0-rpc-service: cannot create listener, initing the
transport failed</div>
<div>[1970-01-01 15:19:43.956362] E [MSGID: 106244]
[glusterd.c:1798:init] 0-management: creation of 1
listeners failed, continuing with succeeded
transport</div>
<div>[1970-01-01 15:19:43.956459] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GlusterD
svc peer, Num: 1238437, Ver: 2, Port: 0</div>
<div>[1970-01-01 15:19:43.956561] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GlusterD
svc cli read-only, Num: 1238463, Ver: 2, Port: 0</div>
<div>[1970-01-01 15:19:43.956666] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GlusterD
svc mgmt, Num: 1238433, Ver: 2, Port: 0</div>
<div>[1970-01-01 15:19:43.956758] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GlusterD
svc mgmt v3, Num: 1238433, Ver: 3, Port: 0</div>
<div>[1970-01-01 15:19:43.956853] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: Gluster
Portmap, Num: 34123456, Ver: 1, Port: 0</div>
<div>[1970-01-01 15:19:43.956946] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: Gluster
Handshake, Num: 14398633, Ver: 2, Port: 0</div>
<div>[1970-01-01 15:19:43.957062] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: Gluster
MGMT Handshake, Num: 1239873, Ver: 1, Port: 0</div>
<div>[1970-01-01 15:19:43.957205] D
[rpcsvc.c:2607:rpcsvc_init] 0-rpc-service: RPC
service inited.</div>
<div>[1970-01-01 15:19:43.957303] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GF-DUMP,
Num: 123451501, Ver: 1, Port: 0</div>
<div>[1970-01-01 15:19:43.957408] D
[rpc-transport.c:269:rpc_transport_load]
0-rpc-transport: attempt to load file
/usr/lib64/glusterfs/5.0/rpc-transport/socket.so</div>
<div>[1970-01-01 15:19:43.957563] D
[socket.c:4424:socket_init] 0-socket.management:
disabling nodelay</div>
<div>[1970-01-01 15:19:43.957650] D
[socket.c:4464:socket_init] 0-socket.management:
Configued transport.tcp-user-timeout=0</div>
<div>[1970-01-01 15:19:43.957738] D
[socket.c:4482:socket_init] 0-socket.management:
Reconfigued transport.keepalivecnt=9</div>
<div>[1970-01-01 15:19:43.957830] D
[socket.c:4167:ssl_setup_connection_params]
0-socket.management: SSL support on the I/O path
is NOT enabled</div>
<div>[1970-01-01 15:19:43.957922] D
[socket.c:4170:ssl_setup_connection_params]
0-socket.management: SSL support for glusterd is
NOT enabled</div>
<div>[1970-01-01 15:19:43.958186] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: GlusterD
svc cli, Num: 1238463, Ver: 2, Port: 0</div>
<div>[1970-01-01 15:19:43.958280] D
[rpcsvc.c:2146:rpcsvc_program_register]
0-rpc-service: New program registered: Gluster
Handshake (CLI Getspec), Num: 14398633, Ver: 2,
Port: 0</div>
<div>[1970-01-01 15:19:43.958461] D [MSGID: 0]
[glusterd-utils.c:7878:glusterd_sm_tr_log_init]
0-glusterd: returning 0</div>
<div>[1970-01-01 15:19:43.958557] D [MSGID: 0]
[glusterd.c:1875:init] 0-management: cannot get
run-with-valgrind value</div>
<div>[1970-01-01 15:19:43.960895] E [MSGID: 101032]
[store.c:447:gf_store_handle_retrieve] 0-: Path
corresponding to /var/lib/glusterd/<a
href="http://glusterd.info" target="_blank"
moz-do-not-send="true">glusterd.info</a>. [No
such file or directory]</div>
<div>[1970-01-01 15:19:43.961016] D [MSGID: 0]
[store.c:452:gf_store_handle_retrieve] 0-:
Returning -1</div>
<div>[1970-01-01 15:19:43.961108] D [MSGID: 0]
[glusterd-store.c:2169:glusterd_retrieve_op_version]
0-management: Unable to get store handle!</div>
<div>[1970-01-01 15:19:43.961216] E [MSGID: 101032]
[store.c:447:gf_store_handle_retrieve] 0-: Path
corresponding to /var/lib/glusterd/<a
href="http://glusterd.info" target="_blank"
moz-do-not-send="true">glusterd.info</a>. [No
such file or directory]</div>
<div>[1970-01-01 15:19:43.961325] D [MSGID: 0]
[store.c:452:gf_store_handle_retrieve] 0-:
Returning -1</div>
<div>[1970-01-01 15:19:43.961428] D [MSGID: 0]
[glusterd-store.c:2345:glusterd_retrieve_uuid]
0-management: Unable to get storehandle!</div>
<div>[1970-01-01 15:19:43.961523] D [MSGID: 0]
[glusterd-store.c:2366:glusterd_retrieve_uuid]
0-management: Returning -1</div>
<div>[1970-01-01 15:19:43.961617] I [MSGID: 106514]
[glusterd-store.c:2304:glusterd_restore_op_version] 0-management:
Detected new install. Setting op-version to
maximum : 50000</div>
<div>[1970-01-01 15:19:43.962658] D [MSGID: 0]
[store.c:432:gf_store_handle_new] 0-: Returning 0</div>
<div>[1970-01-01 15:19:43.962750] D [MSGID: 0]
[store.c:452:gf_store_handle_retrieve] 0-:
Returning 0</div>
<div>[1970-01-01 15:19:43.963047] D [MSGID: 0]
[store.c:515:gf_store_iter_new] 0-: Returning with
0</div>
<div>[1970-01-01 15:19:43.963194] D [MSGID: 0]
[store.c:632:gf_store_iter_get_next] 0-: Returning
with 0</div>
<div>[1970-01-01 15:19:43.963318] D [MSGID: 0]
[store.c:632:gf_store_iter_get_next] 0-: Returning
with -1</div>
<div>[1970-01-01 15:19:43.963455] D [MSGID: 0]
[store.c:473:gf_store_handle_destroy] 0-:
Returning 0</div>
<div>[1970-01-01 15:19:43.963757] D [MSGID: 0]
[glusterd-store.c:3546:glusterd_store_retrieve_volumes]
0-management: Returning with 0</div>
<div>[1970-01-01 15:19:43.964159] D [MSGID: 0]
[glusterd-store.c:4662:glusterd_store_retrieve_peers]
0-management: Returning with 0</div>
<div>[1970-01-01 15:19:43.964471] I [MSGID: 106194]
[glusterd-store.c:3983:glusterd_store_retrieve_missed_snaps_list]
0-management: No missed snaps list.</div>
<div>[1970-01-01 15:19:43.964580] D [MSGID: 0]
[glusterd-store.c:4104:glusterd_store_retrieve_snaps]
0-management: Returning with 0</div>
<div>[1970-01-01 15:19:43.964680] D [MSGID: 0]
[glusterd-store.c:4894:glusterd_restore]
0-management: Returning 0</div>
<div>[1970-01-01 15:19:43.965060] D [MSGID: 0]
[options.c:1225:xlator_option_init_int32]
0-management: option event-threads using set value
1</div>
<div>Final graph:</div>
<div>+------------------------------------------------------------------------------+</div>
<div> 1: volume management</div>
<div> 2: type mgmt/glusterd</div>
<div> 3: option rpc-auth.auth-glusterfs on</div>
<div> 4: option rpc-auth.auth-unix on</div>
<div> 5: option rpc-auth.auth-null on</div>
<div> 6: option rpc-auth-allow-insecure on</div>
<div> 7: option transport.listen-backlog 10</div>
<div> 8: option event-threads 1</div>
<div> 9: option ping-timeout 0</div>
<div> 10: option transport.socket.read-fail-log
off</div>
<div> 11: option
transport.socket.keepalive-interval 2</div>
<div> 12: option transport.socket.keepalive-time
10</div>
<div> 13: option transport-type rdma</div>
<div> 14: option working-directory
/var/lib/glusterd</div>
<div> 15: end-volume</div>
<div> 16: </div>
<div>+------------------------------------------------------------------------------+</div>
<div>[1970-01-01 15:19:43.966808] I [MSGID: 101190]
[event-epoll.c:622:event_dispatch_epoll_worker]
0-epoll: Started thread with index 1</div>
<div>[1970-01-01 15:19:44.840454] E
[rpcsvc.c:513:rpcsvc_request_create]
0-rpc-service: RPC version not supported (XID:
0x0, Ver: 0, Program: 0, ProgVers: 0, Proc: 2)
from trans (socket.management)</div>
<div>[1970-01-01 15:19:44.840884] D
[rpcsvc.c:1416:rpcsvc_error_reply] (-->
/usr/lib64/libglusterfs.so.0(_gf_log_callingfn-0xffbac)[0x3fffa12acbe4]
(-->
/usr/lib64/libgfrpc.so.0(+0xc5f4)[0x3fffa12525f4]
(-->
/usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00]
(-->
/usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224]
(-->
/usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c]
))))) 0-: sending a RPC error reply</div>
<div>[1970-01-01 15:19:44.841055] D
[logging.c:1805:gf_log_flush_extra_msgs]
0-logging-infra: Log buffer size reduced. About to
flush 5 extra log messages</div>
<div>[1970-01-01 15:19:44.841156] D
[logging.c:1808:gf_log_flush_extra_msgs]
0-logging-infra: Just flushed 5 extra log messages</div>
<div>pending frames:</div>
<div>patchset: git://<a
href="http://git.gluster.org/glusterfs.git"
target="_blank" moz-do-not-send="true">git.gluster.org/glusterfs.git</a></div>
<div>signal received: 11</div>
<div>time of crash: </div>
<div>1970-01-01 15:19:44</div>
<div>configuration details:</div>
<div>argp 1</div>
<div>backtrace 1</div>
<div>dlfcn 1</div>
<div>libpthread 1</div>
<div>llistxattr 1</div>
<div>setfsid 1</div>
<div>spinlock 1</div>
<div>epoll.h 1</div>
<div>xattr.h 1</div>
<div>st_atim.tv_nsec 1</div>
<div>package-string: glusterfs 5.0</div>
<div>/usr/lib64/libglusterfs.so.0(+0x422a4)[0x3fffa12ab2a4]</div>
<div>/usr/lib64/libglusterfs.so.0(gf_print_trace-0xf5080)[0x3fffa12b82e0]</div>
<div>./glusterd(glusterfsd_print_trace-0x22fa4)[0x100067ec]</div>
<div>linux-vdso64.so.1(__kernel_sigtramp_rt64+0x0)[0x3fffa13f0478]</div>
<div>/lib64/libc.so.6(xdr_accepted_reply-0x72d3c)[0x3fffa11375cc]</div>
<div>/lib64/libc.so.6(xdr_accepted_reply-0x72d9c)[0x3fffa113756c]</div>
<div>/lib64/libc.so.6(xdr_union-0x63a94)[0x3fffa1147dd4]</div>
<div>/lib64/libc.so.6(xdr_replymsg-0x72c58)[0x3fffa11376e0]</div>
<div>/lib64/libc.so.6(xdr_sizeof-0x62a78)[0x3fffa1149120]</div>
<div>/usr/lib64/libgfrpc.so.0(+0x9b0c)[0x3fffa124fb0c]</div>
<div>/usr/lib64/libgfrpc.so.0(rpcsvc_submit_generic-0x149f4)[0x3fffa125228c]</div>
<div>/usr/lib64/libgfrpc.so.0(+0xc614)[0x3fffa1252614]</div>
<div>/usr/lib64/libgfrpc.so.0(+0xcf00)[0x3fffa1252f00]</div>
<div>/usr/lib64/libgfrpc.so.0(+0xd224)[0x3fffa1253224]</div>
<div>/usr/lib64/libgfrpc.so.0(+0xd84c)[0x3fffa125384c]</div>
<div>/usr/lib64/libgfrpc.so.0(rpc_transport_notify-0x10eec)[0x3fffa125610c]</div>
<div>/usr/lib64/glusterfs/5.0/rpc-transport/socket.so(+0xc09c)[0x3fff9d51709c]</div>
<div>/usr/lib64/libglusterfs.so.0(+0xb84bc)[0x3fffa13214bc]</div>
<div>/lib64/libpthread.so.0(+0xbb30)[0x3fffa11bdb30]</div>
<div>/lib64/libc.so.6(clone-0x9e964)[0x3fffa110817c]</div>
<div>---------</div>
<div>Segmentation fault (core dumped)</div>
<div><br>
</div>
<div>Could you please help me, what actually the
problem?</div>
<div><br>
</div>
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_131193042669896152gmail-m_-5776870226640831026gmail_signature">
<div dir="ltr"><br>
<br>
<br>
<br>
Regards<br>
Abhishek Paliwal<br>
</div>
</div>
</div>
</div>
</div>
_______________________________________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank"
moz-do-not-send="true">Gluster-users@gluster.org</a><br>
<a
href="https://lists.gluster.org/mailman/listinfo/gluster-users"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://lists.gluster.org/mailman/listinfo/gluster-users</a></blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr"
class="gmail-m_131193042669896152gmail_signature">
<div dir="ltr">
<div>Thanks,<br>
</div>
Sanju<br>
</div>
</div>
</blockquote>
</div>
<br clear="all">
<div><br>
</div>
-- <br>
<div dir="ltr" class="gmail_signature">
<div dir="ltr"><br>
<br>
<br>
<br>
Regards<br>
Abhishek Paliwal<br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<pre class="moz-quote-pre" wrap="">_______________________________________________
Gluster-users mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a>
<a class="moz-txt-link-freetext" href="https://lists.gluster.org/mailman/listinfo/gluster-users">https://lists.gluster.org/mailman/listinfo/gluster-users</a></pre>
</blockquote>
</body>
</html>