<div dir="ltr">i have a 12 nodes cluster with 2 nodes downgraded and currently down. The procedure of taking down 3.13 nodes means a cluster downtime?<br><br>if it matters /var/lib/glusterd/<a href="http://glusterd.info">glusterd.info</a> is empty on the 3.12.9 nodes.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 15, 2018 at 2:47 PM, Kaleb S. KEITHLEY <span dir="ltr"><<a href="mailto:kkeithle@redhat.com" target="_blank">kkeithle@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 05/15/2018 08:08 AM, Davide Obbi wrote:<br>
> Thanks Kaleb,<br>
> <br>
> any chance i can make the node working after the downgrade?<br>
> thanks<br>
<br>
</span>Without knowing what doesn't work, I'll go out on a limb and guess that<br>
it's an op-version problem.<br>
<br>
Shut down your 3.13 nodes, change their op-version to one of the valid<br>
3.12 op-versions (e.g. 31203) and restart. Then the 3.12 nodes should<br>
work with 3.13, and/or the formerly 3.13 nodes should work with 3.12.<br>
<br>
JoeJulian in #gluster may have other ideas.<br>
<span class=""><br>
<br>
> <br>
> On Tue, May 15, 2018 at 2:02 PM, Kaleb S. KEITHLEY <<a href="mailto:kkeithle@redhat.com">kkeithle@redhat.com</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:kkeithle@redhat.com">kkeithle@redhat.com</a>>> wrote:<br>
> <br>
> <br>
> You can still get them from<br>
> <a href="https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.13/" rel="noreferrer" target="_blank">https://buildlogs.centos.org/<wbr>centos/7/storage/x86_64/<wbr>gluster-3.13/</a><br>
> <<a href="https://buildlogs.centos.org/centos/7/storage/x86_64/gluster-3.13/" rel="noreferrer" target="_blank">https://buildlogs.centos.org/<wbr>centos/7/storage/x86_64/<wbr>gluster-3.13/</a>><br>
> <br>
> (I don't know how much longer they'll be there. I suggest you copy them<br>
> if you think you're going to need them in the future.)<br>
> <br>
> <br>
> <br>
> n 05/15/2018 04:58 AM, Davide Obbi wrote:<br>
> > hi,<br>
> > <br>
> > i noticed that this repo for glusterfs 3.13 does not exists anymore at:<br>
> > <br>
> > <a href="http://mirror.centos.org/centos/7/storage/x86_64/" rel="noreferrer" target="_blank">http://mirror.centos.org/<wbr>centos/7/storage/x86_64/</a><br>
> <<a href="http://mirror.centos.org/centos/7/storage/x86_64/" rel="noreferrer" target="_blank">http://mirror.centos.org/<wbr>centos/7/storage/x86_64/</a>><br>
> > <br>
> > i knew was not going to be long term supported however the downgrade to<br>
> > 3.12 breaks the server node i believe the issue is with:<br>
> > *[2018-05-15 08:54:39.981101] E [MSGID: 101019]<br>
> > [xlator.c:503:xlator_init] 0-management: Initialization of volume<br>
> > 'management' failed, review your volfile again <wbr> <br>
> > [2018-05-15 08:54:39.981113] E [MSGID: 101066]<br>
> > [graph.c:327:glusterfs_graph_<wbr>init] 0-management: initializing translator<br>
> > failed <wbr> <wbr> <br>
> > [2018-05-15 08:54:39.981121] E [MSGID: 101176]<br>
> > [graph.c:698:glusterfs_graph_<wbr>activate] 0-graph: init<br>
> > failed <wbr> <wbr> <br>
> > *<br>
> > <br>
> > Any help appreciated, thanks<br>
> > <br>
> > Details:<br>
> > *Installed Packages*<br>
> ><br>
> glusterfs.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-api.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-cli.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-client-xlators.x86_<wbr>64 <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-events.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-fuse.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-libs.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> glusterfs-server.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> python2-gluster.x86_64 <wbr> <wbr> <wbr> <br>
> ><br>
> 3.12.9-1.el7 <wbr> <wbr> <br>
> > @gluster-3.12<br>
> ><br>
> > *# journalctl -u<br>
> ><br>
> glusterd * <wbr> <wbr> <wbr> <wbr> <br>
> ><br>
> > -- Logs begin at Wed 2018-03-21 15:06:46 CET, end at Tue 2018-05-15<br>
> > 10:48:01 CEST. --<br>
> > Mar 21 15:06:58 glusternode-1003 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > Mar 21 15:07:01 glusternode-1 systemd[1]: Started GlusterFS, a<br>
> clustered<br>
> > file-system<br>
> ><br>
> server. <wbr> <wbr> <br>
> ><br>
> > Mar 21 15:25:07 glusternode-1 systemd[1]: Stopping GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > Mar 21 15:25:07 glusternode-1 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > Mar 21 15:25:09 glusternode-1 systemd[1]: Started GlusterFS, a<br>
> clustered<br>
> > file-system<br>
> ><br>
> server. <wbr> <wbr> <br>
> ><br>
> > May 10 12:29:11 glusternode-1 systemd[1]: Stopping GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > May 10 12:29:11 glusternode-1 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > May 10 12:29:13 glusternode-1 systemd[1]: Started GlusterFS, a<br>
> clustered<br>
> > file-system<br>
> ><br>
> server. <wbr> <wbr> <br>
> ><br>
> > May 15 10:21:01 glusternode-1 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > May 15 10:21:01 glusternode-1 systemd[1]: glusterd.service: control<br>
> > process exited, code=exited<br>
> > status=1 <wbr> <wbr> <br>
> > May 15 10:21:01 glusternode-1 systemd[1]: Failed to start GlusterFS, a<br>
> > clustered file-system<br>
> > server. <wbr> <wbr> <br>
> > May 15 10:21:01 glusternode-1 systemd[1]: Unit glusterd.service<br>
> entered<br>
> > failed<br>
> ><br>
> state. <wbr> <wbr> <br>
> ><br>
> > May 15 10:21:01 glusternode-1 systemd[1]: glusterd.service failed.<br>
> > May 15 10:22:14 glusternode-1 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > May 15 10:22:14 glusternode-1 systemd[1]: glusterd.service: control<br>
> > process exited, code=exited<br>
> > status=1 <wbr> <wbr> <br>
> > May 15 10:22:14 glusternode-1 systemd[1]: Failed to start GlusterFS, a<br>
> > clustered file-system<br>
> > server. <wbr> <wbr> <br>
> > May 15 10:22:14 glusternode-1 systemd[1]: Unit glusterd.service<br>
> entered<br>
> > failed<br>
> ><br>
> state. <wbr> <wbr> <br>
> ><br>
> > May 15 10:22:14 glusternode-1 systemd[1]: glusterd.service failed.<br>
> > May 15 10:22:49 glusternode-1 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > May 15 10:22:49 glusternode-1 systemd[1]: glusterd.service: control<br>
> > process exited, code=exited<br>
> > status=1 <wbr> <wbr> <br>
> > May 15 10:22:49 glusternode-1 systemd[1]: Failed to start GlusterFS, a<br>
> > clustered file-system<br>
> > server. <wbr> <wbr> <br>
> > May 15 10:22:49 glusternode-1 systemd[1]: Unit glusterd.service<br>
> entered<br>
> > failed<br>
> ><br>
> state. <wbr> <wbr> <br>
> ><br>
> > May 15 10:22:49 glusternode-1 systemd[1]: glusterd.service failed.<br>
> > May 15 10:23:36 glusternode-1 systemd[1]: Starting GlusterFS, a<br>
> > clustered file-system<br>
> ><br>
> server... <wbr> <wbr> <br>
> ><br>
> > May 15 10:23:36 glusternode-1 systemd[1]: glusterd.service: control<br>
> > process exited, code=exited<br>
> > status=1 <wbr> <wbr> <br>
> > May 15 10:23:36 glusternode-1 systemd[1]: Failed to start GlusterFS, a<br>
> > clustered file-system<br>
> > server. <wbr> <wbr> <br>
> > May 15 10:23:36 glusternode-1 systemd[1]: Unit glusterd.service<br>
> entered<br>
> > failed<br>
> ><br>
> state. <wbr> <wbr> <br>
> ><br>
> > May 15 10:23:36 glusternode-1 systemd[1]: glusterd.service failed.<br>
> > [2018-05-15 08:54:38.354520] I [MSGID: 100030]<br>
> [glusterfsd.c:2511:main]<br>
> > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version<br>
> 3.12.9<br>
> > (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)<br>
> > [2018-05-15 08:54:38.359267] I [MSGID: 106478] [glusterd.c:1423:init]<br>
> > 0-management: Maximum allowed open file descriptors set to<br>
> > 65536 <wbr> <br>
> > [2018-05-15 08:54:38.359305] I [MSGID: 106479] [glusterd.c:1481:init]<br>
> > 0-management: Using /var/lib/glusterd as working<br>
> > directory <wbr> <wbr> <br>
> > [2018-05-15 08:54:38.359313] I [MSGID: 106479] [glusterd.c:1486:init]<br>
> > 0-management: Using /var/run/gluster as pid file working<br>
> > directory <wbr> <wbr> <br>
> > [2018-05-15 08:54:38.363363] E<br>
> [rpc-transport.c:283:rpc_<wbr>transport_load]<br>
> > 0-rpc-transport: /usr/lib64/glusterfs/3.12.9/<wbr>rpc-transport/rdma.so:<br>
> > cannot open shared object file: No such file or directory<br>
> > [2018-05-15 08:54:38.363381] W<br>
> [rpc-transport.c:287:rpc_<wbr>transport_load]<br>
> > 0-rpc-transport: volume 'rdma.management': transport-type 'rdma'<br>
> is not<br>
> > valid or not found on this machine <br>
> > [2018-05-15 08:54:38.363391] W [rpcsvc.c:1682:rpcsvc_create_<wbr>listener]<br>
> > 0-rpc-service: cannot create listener, initing the transport<br>
> > failed <wbr> <br>
> > [2018-05-15 08:54:38.363398] E [MSGID: 106243] [glusterd.c:1769:init]<br>
> > 0-management: creation of 1 listeners failed, continuing with<br>
> succeeded<br>
> > transport <wbr> <br>
> > [2018-05-15 08:54:39.981028] E [MSGID: 106022]<br>
> > [glusterd-store.c:2234:<wbr>glusterd_restore_op_version] 0-management:<br>
> wrong<br>
> > op-version (31300) retrieved [Invalid argument] <wbr> <br>
> > [2018-05-15 08:54:39.981071] E [MSGID: 106244] [glusterd.c:1928:init]<br>
> > 0-management: Failed to restore<br>
> ><br>
> op_version <wbr> <wbr> <br>
> ><br>
> > [2018-05-15 08:54:39.981101] E [MSGID: 101019]<br>
> > [xlator.c:503:xlator_init] 0-management: Initialization of volume<br>
> > 'management' failed, review your volfile<br>
> again <wbr> <br>
> > [2018-05-15 08:54:39.981113] E [MSGID: 101066]<br>
> > [graph.c:327:glusterfs_graph_<wbr>init] 0-management: initializing<br>
> translator<br>
> > failed <wbr> <wbr> <br>
> > [2018-05-15 08:54:39.981121] E [MSGID: 101176]<br>
> > [graph.c:698:glusterfs_graph_<wbr>activate] 0-graph: init<br>
> ><br>
> failed <wbr> <wbr> <br>
> ><br>
> > [2018-05-15 08:54:39.981720] W [glusterfsd.c:1375:cleanup_<wbr>and_exit]<br>
> > (-->/usr/sbin/glusterd(<wbr>glusterfs_volumes_init+0xfd) [0x559572dbcf5d]<br>
> > -->/usr/sbin/glusterd(<wbr>glusterfs_process_volfp+0x163)<br>
> > [0x559572dbce03] -->/usr/sbin/glusterd(cleanup_<wbr>and_exit+0x6b)<br>
> > [0x559572dbc37b] ) 0-: received signum (-1), shutting<br>
> > down <wbr> <wbr> <br>
> > */var/log/glusterfs/glusterd.<wbr>log*<br>
> > [2018-05-15 08:54:38.354520] I [MSGID: 100030]<br>
> [glusterfsd.c:2511:main]<br>
> > 0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version<br>
> 3.12.9<br>
> > (args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO)<br>
> > [2018-05-15 08:54:38.359267] I [MSGID: 106478] [glusterd.c:1423:init]<br>
> > 0-management: Maximum allowed open file descriptors set to<br>
> > 65536 <wbr> <br>
> > [2018-05-15 08:54:38.359305] I [MSGID: 106479] [glusterd.c:1481:init]<br>
> > 0-management: Using /var/lib/glusterd as working<br>
> > directory <wbr> <wbr> <br>
> > [2018-05-15 08:54:38.359313] I [MSGID: 106479] [glusterd.c:1486:init]<br>
> > 0-management: Using /var/run/gluster as pid file working<br>
> > directory <wbr> <wbr> <br>
> > [2018-05-15 08:54:38.363363] E<br>
> [rpc-transport.c:283:rpc_<wbr>transport_load]<br>
> > 0-rpc-transport: /usr/lib64/glusterfs/3.12.9/<wbr>rpc-transport/rdma.so:<br>
> > cannot open shared object file: No such file or directory<br>
> > [2018-05-15 08:54:38.363381] W<br>
> [rpc-transport.c:287:rpc_<wbr>transport_load]<br>
> > 0-rpc-transport: volume 'rdma.management': transport-type 'rdma'<br>
> is not<br>
> > valid or not found on this machine <br>
> > [2018-05-15 08:54:38.363391] W [rpcsvc.c:1682:rpcsvc_create_<wbr>listener]<br>
> > 0-rpc-service: cannot create listener, initing the transport<br>
> > failed <wbr> <br>
> > [2018-05-15 08:54:38.363398] E [MSGID: 106243] [glusterd.c:1769:init]<br>
> > 0-management: creation of 1 listeners failed, continuing with<br>
> succeeded<br>
> > transport <wbr> <br>
> > [2018-05-15 08:54:39.981028] E [MSGID: 106022]<br>
> > [glusterd-store.c:2234:<wbr>glusterd_restore_op_version] 0-management:<br>
> wrong<br>
> > op-version (31300) retrieved [Invalid argument] <wbr> <br>
> > [2018-05-15 08:54:39.981071] E [MSGID: 106244] [glusterd.c:1928:init]<br>
> > 0-management: Failed to restore<br>
> ><br>
> op_version <wbr> <wbr> <br>
> ><br>
> > *[2018-05-15 08:54:39.981101] E [MSGID: 101019]<br>
> > [xlator.c:503:xlator_init] 0-management: Initialization of volume<br>
> > 'management' failed, review your volfile again <wbr> <br>
> > [2018-05-15 08:54:39.981113] E [MSGID: 101066]<br>
> > [graph.c:327:glusterfs_graph_<wbr>init] 0-management: initializing translator<br>
> > failed <wbr> <wbr> <br>
> > [2018-05-15 08:54:39.981121] E [MSGID: 101176]<br>
> > [graph.c:698:glusterfs_graph_<wbr>activate] 0-graph: init<br>
> > failed <wbr> <wbr> <br>
> > *<br>
> > [2018-05-15 08:54:39.981720] W [glusterfsd.c:1375:cleanup_<wbr>and_exit]<br>
> > (-->/usr/sbin/glusterd(<wbr>glusterfs_volumes_init+0xfd) [0x559572dbcf5d]<br>
> > -->/usr/sbin/glusterd(<wbr>glusterfs_process_volfp+0x163)<br>
> > [0x559572dbce03] -->/usr/sbin/glusterd(cleanup_<wbr>and_exit+0x6b)<br>
> > [0x559572dbc37b] ) 0-: received signum (-1), shutting<br>
> > down <wbr> <wbr> <br>
> > <br>
> > <br>
> > Kind regards<br>
> > Davide<br>
> > <br>
> > <br>
> > ______________________________<wbr>_________________<br>
> > Gluster-users mailing list<br>
</div></div>> > <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a> <mailto:<a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.<wbr>org</a>><br>
> > <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
<span class="">> <<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><wbr>><br>
> ><br>
> <br>
> <br>
> <br>
> <br>
> <br>
> -- <br>
> Davide Obbi<br>
> System Administrator<br>
> <br>
> Booking.com B.V.<br>
> Vijzelstraat 66-80 Amsterdam 1017HL Netherlands<br>
> Direct +31207031558<br>
</span>> Booking.com <<a href="http://www.booking.com/" rel="noreferrer" target="_blank">http://www.booking.com/</a>><br>
<div class="HOEnZb"><div class="h5">> The world's #1 accommodation site <br>
> 43 languages, 198+ offices worldwide, 120,000+ global destinations,<br>
> 1,550,000+ room nights booked every day <br>
> No booking fees, best price always guaranteed <br>
> Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)<br>
<br>
<br>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><span style="display:block;font-size:11.0pt;font-family:Century Gothic;color:#003580"><div style="color:rgb(0,53,128);font-family:Arial,Helvetica,sans-serif;font-weight:bold;font-size:15px"><div>Davide Obbi</div><div style="font-weight:normal;font-size:13px;color:rgb(0,174,239)">System Administrator<br><br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)">Booking.com B.V.<br>Vijzelstraat 66-80 Amsterdam 1017HL Netherlands</div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><span style="color:rgb(0,174,239)">Direct </span>+31207031558<br></div><div style="font-weight:normal;font-size:13px;color:rgb(102,102,102)"><div style="font-weight:bold;font-size:16px;color:rgb(0,53,128)"><a href="http://www.booking.com/" style="color:rgb(0,127,255);background-image:initial;background-position:initial;background-repeat:initial" target="_blank"><img src="http://bstatic.com/static/img/siglogo.jpg" alt="Booking.com" title="Booking.com"></a></div><span style="font-size:11px">The world's #1 accommodation site <br>43 languages, 198+ offices worldwide, 120,000+ global destinations, 1,550,000+ room nights booked every day <br>No booking fees, best price always guaranteed <br></span><span style="font-size:11px">Subsidiary of Booking Holdings Inc. (NASDAQ: BKNG)</span><span style="font-size:11px"><br></span></div></div></span></div>
</div>