<div dir="ltr"><div dir="ltr">Thanks for the heads up Humble. This would help many of the gluster community users, who may not be following k8s threads actively to be planning their migration plans actively.<br><br>For all the users who are currently running heketi + glusterfs, starting from k8s v1.26, you CANNOT use heketi + glusterfs based storage in k8s.<br><br>Below are my personal suggestions for the users. Please treat these options as my personal opinion, and not an official stand of the gluster community.<br><br>0. Use an older (< 1.25) version of k8s, and keep using the setup :-)<br><br>1. Use current storage nodes as part of storage, but managed separately, and expose NFS and use NFS CSI to get the data in the pods. (Note the change over to new PV through CSI based PVC, which means applications need a migration). - I haven't tested this setup, hence can't vouch for this.<br><br>2. Use kadalu [6] operator to manage currently deployed glusterfs nodes as 'External' storage class, and use kadalu CSI (which uses native glusterfs mount as part of CSI node plugin) to get PV for your application pods. NOTE: here too, there is an application migration needed to use kadalu CSI based PVC. Suggested for those users with bigger PVs used in k8s setup already. There is a team to help with this migration if you wish to.<br><br>3. Use kadalu (or any 'other' CSI providers), provision a new storage, and copy over the data set to new storage: Would be an option if the storage is smaller in size. In this case, there would be extra time to do a copy of data through starting a pod with both existing PV and new PV added as mounts in the same pod, so you can copy off the data quickly.</div><div><br>In any case, considering you do not have a lot of time before kubernetes v1.26 comes out, please do start your migration plans soon.</div><div><br></div><div>For the developers of the glusterfs community, what are the thoughts you have on this? I know there is some effort started on keeping glusterfs-containers repo relevant, and I see PRs coming out. Happy to open up a discussion on the same.</div><div><br>Regards, <br>Amar (@amarts)<br><br></div>[6] - <a href="https://github.com/kadalu/kadalu">https://github.com/kadalu/kadalu</a><br><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Aug 11, 2022 at 5:47 PM Humble Chirammal <<a href="mailto:hchiramm@redhat.com">hchiramm@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt" id="gmail-m_-3428702550769678219gmail-docs-internal-guid-62a53032-7fff-f34d-e7f9-9e41b9ea7015"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Hey Gluster Community,</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">As you might be aware, there is an effort in the kubernetes community to remove in-tree storage plugins to reduce external dependencies and security concerns in the core Kubernetes. Thus, we are in a process to gradually deprecate all the in-tree external storage plugins and eventually remove them from the core Kubernetes codebase.  GlusterFS is one of the very first dynamic provisioners which was made into kubernetes </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">v1.4 ( 2016 ) </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">release via </span><a href="https://github.com/kubernetes/kubernetes/pull/30888" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/kubernetes/kubernetes/pull/30888</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> . From then on many deployments were/are making use of this driver to consume GlusterFS volumes in Kubernetes/Openshift clusters. </span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">As part of this effort, we are planning to deprecate GlusterFS intree plugin in </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">1.25 </span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">release and planning to take out Heketi code from Kubernetes Code base in subsequent release. This code removal will not be following kubernetes' normal deprecation policy [1] and will be treated as an exception [2]. The main reason for this exception is that, Heketi is in "Deep Maintenance" [3], also please see [4] for the latest push back from the Heketi team on changes we would need to keep vendoring heketi into kubernetes/kubernetes. We cannot keep heketi in the kubernetes code base as heketi itself is literally going away. The current plan is to start declaring the deprecation in kubernetes v1.25 and code removal in v1.26.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">If you are using GlusterFS driver in your cluster setup, please reply with  below info before 16-Aug-2022 to </span><a href="mailto:dev@kubernetes.io" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">dev@kubernetes.io</span></a><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> ML on thread ( </span><span style="font-size:10pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:700;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Deprecation of intree GlusterFS driver in 1.25</span><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">) or to this thread which can help us to make a decision on when to completely remove this code base from the repo.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">- what version of kubernetes are you running in your setup ?</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">- how often do you upgrade your cluster?</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">- what vendor or distro you are using ? Is it any (downstream) product offering or upstream GlusterFS driver directly used in your setup?</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Awaiting your feedback.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Thanks,</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">Humble</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[1]</span><a href="https://kubernetes.io/docs/reference/using-api/deprecation-policy/" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://kubernetes.io/docs/reference/using-api/deprecation-policy/</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[2]</span><a href="https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://kubernetes.io/docs/reference/using-api/deprecation-policy/#exceptions</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[3]</span><a href="https://github.com/heketi/heketi#maintenance-status" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/heketi/heketi#maintenance-status</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[4]</span><a href="https://github.com/heketi/heketi/pull/1904#issuecomment-1197100513" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/heketi/heketi/pull/1904#issuecomment-1197100513</span></a></p><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap">[5]</span><a href="https://github.com/kubernetes/kubernetes/issues/100897" style="text-decoration:none" target="_blank"><span style="font-size:11pt;font-family:Arial;color:rgb(0,0,0);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:none;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:Arial;color:rgb(17,85,204);background-color:transparent;font-weight:400;font-style:normal;font-variant:normal;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/kubernetes/kubernetes/issues/100897</span></a><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div><br><br></div><br></div></div></div></div></div></div></div></div>
</blockquote></div></div>