<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:宋体;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=ZH-CN link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>Yes, I could ensure that no other daemon used this port at that time.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>I have encountered this situation several times, some need multiple restarts to bring all bricks back. Before restart, I killed all the daemons, and used netstat to ensure the ports begin with 4915 were closed , and also 24007 . <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'>Likun<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:10.5pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div><div><p class=MsoNormal><span lang=EN-US>On Mon, Jan 16, 2017 at 10:19 AM, likun <<a href="mailto:kun.li@ucarinc.com" target="_blank">kun.li@ucarinc.com</a>> wrote:<o:p></o:p></span></p><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Hi, everybody.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>I have glusterfs 3.8.5 setuped in a kubernetes enveironment.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Recently, one of my glusterfs server rebooted, glusterfs service didn’t come up successfully.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Out of totally 6 bricks, 3 came up, 3 didn’t.<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>Here is the logs of the brick that have failed:<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:46:50.240032] I [MSGID: 100030] [glusterfsd.c:2454:main] 0-/usr/sbin/glusterfsd: Started running /usr/sbin/glusterfsd version 3.8.5 (args: /usr/sbin/glusterfsd -s 10.32.3.23 --volfile-id gvol0.10.32.3.23.mnt-brick1-vol -p /var/lib/glusterd/vols/gvol0/run/10.32.3.23-mnt-brick1-vol.pid -S /var/run/gluster/08da045f3e66eefc50c0ff9a035c6794.socket --brick-name /mnt/brick1/vol -l /var/log/glusterfs/bricks/mnt-brick1-vol.log --xlator-option *-posix.glusterd-uuid=58c3b462-a4b6-4655-b2ac-d0502e278e03 --brick-port 49152 --xlator-option gvol0-server.listen-port=49152)<o:p></o:p></span></p></div></div></blockquote><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>So 49152 port was allocated for this brick.<br> <o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:46:50.258772] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 1<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.153575] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-gvol0-server: adding option 'listen-port' for volume 'gvol0-server' with value '49152'<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.153613] I [MSGID: 101173] [graph.c:269:gf_add_cmdline_options] 0-gvol0-posix: adding option 'glusterd-uuid' for volume 'gvol0-posix' with value '58c3b462-a4b6-4655-b2ac-d0502e278e03'<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.153777] I [MSGID: 115034] [server.c:398:_check_for_auth_option] 0-gvol0-decompounder: skip format check for non-addr auth option auth.login./mnt/brick1/vol.allow<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.153785] I [MSGID: 115034] [server.c:398:_check_for_auth_option] 0-gvol0-decompounder: skip format check for non-addr auth option auth.login.94bedfd1-619d-402a-9826-67dab7600f43.password<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.153895] I [MSGID: 101190] [event-epoll.c:628:event_dispatch_epoll_worker] 0-epoll: Started thread with index 2<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159776] I [rpcsvc.c:2214:rpcsvc_set_outstanding_rpc_limit] 0-rpc-service: Configured rpc.outstanding-rpc-limit with value 64<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159829] W [MSGID: 101002] [options.c:954:xl_opt_validate] 0-gvol0-server: option 'listen-port' is deprecated, preferred is 'transport.socket.listen-port', continuing with correction<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159893] E [socket.c:793:__socket_server_bind] 0-tcp.gvol0-server: binding to failed: Address already in use<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159899] E [socket.c:796:__socket_server_bind] 0-tcp.gvol0-server: Port is already in use<o:p></o:p></span></p></div></div></blockquote><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><span lang=EN-US>What does 'netstat -nap | grep 49152' says at that time? Are we sure no other application has accidentally raced to consume the same port ?<o:p></o:p></span></p></div><blockquote style='border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm'><div><div><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159907] W [rpcsvc.c:1645:rpcsvc_create_listener] 0-rpc-service: listening on transport failed<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159913] W [MSGID: 115045] [server.c:1061:init] 0-gvol0-server: creation of listener failed<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159919] E [MSGID: 101019] [xlator.c:433:xlator_init] 0-gvol0-server: Initialization of volume 'gvol0-server' failed, review your volfile again<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159924] E [MSGID: 101066] [graph.c:324:glusterfs_graph_init] 0-gvol0-server: initializing translator failed<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.159929] E [MSGID: 101176] [graph.c:673:glusterfs_graph_activate] 0-graph: init failed<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>[2016-12-29 03:47:08.160764] W [glusterfsd.c:1327:cleanup_and_exit] (-->/usr/sbin/glusterfsd(mgmt_getspec_cbk+0x3c1) [0x55ead22bee51] -->/usr/sbin/glusterfsd(glusterfs_process_volfp+0x172) [0x55ead22b95d2] -->/usr/sbin/glusterfsd(cleanup_and_exit+0x6b) [0x55ead22b8b4b] ) 0-: received signum (98), shutting down<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>What’s wrong with it ? “port is already in use” ? the system was just rebooted, it’s clear, no other daemon was using this port .<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US>After restarted the glusterfs container, all six bricks came up. Sometime, two restart or more needed to take all the bricks back to work. Any idea ?<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='color:#888888'> <o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='color:#888888'>Likun<o:p></o:p></span></p><p class=MsoNormal style='mso-margin-top-alt:auto;mso-margin-bottom-alt:auto'><span lang=EN-US style='color:#888888'> <o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-US><br>_______________________________________________<br>Gluster-users mailing list<br><a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br><a href="http://lists.gluster.org/mailman/listinfo/gluster-users" target="_blank">http://lists.gluster.org/mailman/listinfo/gluster-users</a><o:p></o:p></span></p></blockquote></div><p class=MsoNormal><span lang=EN-US><br><br clear=all><br>-- <o:p></o:p></span></p><div><div><div><div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div><div><p class=MsoNormal><span lang=EN-US>~ Atin (atinm)<o:p></o:p></span></p></div></div></div></div></div><p class=MsoNormal><span lang=EN-US><o:p> </o:p></span></p></div></div></body></html>