<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 26, 2018 at 8:30 PM, John Strunk <span dir="ltr"><<a href="mailto:jstrunk@redhat.com" target="_blank">jstrunk@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It is configurable. Use the default as a notion of scale... 5s may become 30s; It won't be 5m.<div>Also remember, this is the maximum, not minimum. A change to a watched kube resource will cause an immediate reconcile. The periodic, timer-based loop is just a fallback to catch state changes not represented in the kube API.</div></div></blockquote><div><br></div><div>Cool, got it. Let us wait if anyone sees any objections to the solution proposed.</div><div><br></div><div>Request everyone to comment if they see any issues with <a href="https://github.com/gluster/glusterd2/issues/1069">https://github.com/gluster/glusterd2/issues/1069</a></div><div>I think EC/AFR/Quota components will definitely be affected with this approach. CCing them.</div><div>Please feel free to CC anyone who works on commands that require a mount to give status.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="gmail-HOEnZb"><div class="gmail-h5"><br><div class="gmail_quote"><div dir="ltr">On Thu, Jul 26, 2018 at 12:57 AM Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Jul 26, 2018 at 9:59 AM, Pranith Kumar Karampuri <span dir="ltr"><<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote"><span class="gmail-m_6679871537803201214m_2317293645654858914gmail-">On Wed, Jul 25, 2018 at 10:48 PM, John Strunk <span dir="ltr"><<a href="mailto:jstrunk@redhat.com" target="_blank">jstrunk@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I have not put together a list. Perhaps the following will help w/ the context though...<div><br></div><div>The "reconcile loop" of the operator will take the cluster CRs and reconcile them against the actual cluster config. At the 20k foot level, this amounts to something like determining there should be 8 gluster pods running, and making the appropriate changes if that doesn't match reality. In practical terms, the construction of this reconciliation loop can be thought of as a set (array) of 3-tuples: [{should_act() -> bool, can_act -> bool, action() -> ok, error}, {..., ..., ...}, ...]</div><div><br></div><div>Each capability of the operator would be expressed as one of these tuples.</div><div>should_act() : true if the action() should be taken</div><div>can_act() : true if the prerequisites for taking the action are met</div><div>action() : make the change. Only run if should && can.</div><div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">(note that I believe should_act() and can_act() should not be separate in the implementation, for reasons I'll not go into here)</span><br></div><div><br></div><div>An example action might be "upgrade the container image for pod X". The associated should_act would be triggered if the "image=" of the pod doesn't match the desired "image=" in the operator CRs. The can_act evaluation would be verifying that it's ok to do this... Thinking from the top of my head:</div><div>- All volumes w/ a brick on this pod should be fully healed</div><div>- Sufficient cluster nodes should be up such that quorum is not lost when this node goes down (does this matter?)</div><div>- The proposed image is compatible with the current version of the CSI driver(s), the operator, and other gluster pods</div><div>- Probably some other stuff</div><div>The action() would update the "image=" in the Deployment to trigger the rollout</div><div><br></div><div>The idea is that queries would be made, both to the kube API and the gluster cluster to verify the necessary preconditions for an action prior to that action being invoked. There would obviously be commonality among the preconditions for various actions, so the results should be fetched exactly once per reconcile cycle. Also note, 1 cycle == at most 1 action() due to the action changing the state of the system.</div><div><br></div><div>Given that we haven't designed (or even listed) all the potential action()s, I can't give you a list of everything to query. I guarantee we'll need to know the up/down status, heal counts, and free capacity for each brick and node.</div></div></blockquote><div><br></div></span><div>Thanks for the detailed explanation. This helps. One question though, is 5 seconds a hard limit or is there a possibility to configure it?<br></div></div></div></div></blockquote><div><br></div><div>I put together an idea for reducing the mgmt operation latency involving mounts at <a href="https://github.com/gluster/glusterd2/issues/1069" target="_blank">https://github.com/gluster/<wbr>glusterd2/issues/1069</a>, comments welcome.</div><div>@john Still want to know if there exists a way to find if the hard limit can be configured...<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div></div><div><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-h5"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501HOEnZb"><font color="#888888"><div><br></div><div>-John</div></font></span></div><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501HOEnZb"><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501h5"><br><div class="gmail_quote"><div dir="ltr">On Wed, Jul 25, 2018 at 11:56 AM Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Jul 25, 2018 at 8:17 PM, John Strunk <span dir="ltr"><<a href="mailto:jstrunk@redhat.com" target="_blank">jstrunk@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">To add an additional data point... The operator will need to regularly reconcile the true state of the gluster cluster with the desired state stored in kubernetes. This task will be required frequently (i.e., operator-framework defaults to every 5s even if there are no config changes).<div>The actual amount of data we will need to query from the cluster is currently TBD and likely significantly affected by Heketi/GD1 vs. GD2 choice.</div></div></blockquote><div><br></div><div>Do we have any partial list of data we will gather? Just want to understand what this might entail already...<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><span class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992HOEnZb"><font color="#888888"><div><br></div><div>-John</div><div><br></div></font></span></div><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992HOEnZb"><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992h5"><br><div class="gmail_quote"><div dir="ltr">On Wed, Jul 25, 2018 at 5:45 AM Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jul 24, 2018 at 10:10 PM, Sankarshan Mukhopadhyay <span dir="ltr"><<a href="mailto:sankarshan.mukhopadhyay@gmail.com" target="_blank">sankarshan.mukhopadhyay@<wbr>gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On Tue, Jul 24, 2018 at 9:48 PM, Pranith Kumar Karampuri<br>
<<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br>
> hi,<br>
> Quite a few commands to monitor gluster at the moment take almost a<br>
> second to give output.<br>
<br>
</span>Is this at the (most) minimum recommended cluster size?<br></blockquote><div><br></div><div>Yes, with a single volume with 3 bricks i.e. 3 nodes in cluster.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span><br>
> Some categories of these commands:<br>
> 1) Any command that needs to do some sort of mount/glfs_init.<br>
> Examples: 1) heal info family of commands 2) statfs to find<br>
> space-availability etc (On my laptop replica 3 volume with all local bricks,<br>
> glfs_init takes 0.3 seconds on average)<br>
> 2) glusterd commands that need to wait for the previous command to unlock.<br>
> If the previous command is something related to lvm snapshot which takes<br>
> quite a few seconds, it would be even more time consuming.<br>
><br>
> Nowadays container workloads have hundreds of volumes if not thousands. If<br>
> we want to serve any monitoring solution at this scale (I have seen<br>
> customers use upto 600 volumes at a time, it will only get bigger) and lets<br>
> say collecting metrics per volume takes 2 seconds per volume(Let us take the<br>
> worst example which has all major features enabled like<br>
> snapshot/geo-rep/quota etc etc), that will mean that it will take 20 minutes<br>
> to collect metrics of the cluster with 600 volumes. What are the ways in<br>
> which we can make this number more manageable? I was initially thinking may<br>
> be it is possible to get gd2 to execute commands in parallel on different<br>
> volumes, so potentially we could get this done in ~2 seconds. But quite a<br>
> few of the metrics need a mount or equivalent of a mount(glfs_init) to<br>
> collect different information like statfs, number of pending heals, quota<br>
> usage etc. This may lead to high memory usage as the size of the mounts tend<br>
> to be high.<br>
><br>
<br>
</span>I am not sure if starting from the "worst example" (it certainly is<br>
not) is a good place to start from.</blockquote><div><br></div><div>I didn't understand your statement. Are you saying 600 volumes is a worst example?<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> That said, for any environment<br>
with that number of disposable volumes, what kind of metrics do<br>
actually make any sense/impact?<br></blockquote><div><br></div><div>Same metrics you track for long running volumes. It is just that the way the metrics</div><div>are interpreted will be different. On a long running volume, you would look at the metrics</div><div>and try to find why is the volume not giving performance as expected in the last 1 hour. Where as</div><div>in this case, you would look at metrics and find the reason why volumes that were</div><div>created and deleted in the last hour didn't give performance as expected. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992m_-1140664742746247274m_2965230752172933805HOEnZb"><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992m_-1140664742746247274m_2965230752172933805h5"><br>
> I wanted to seek suggestions from others on how to come to a conclusion<br>
> about which path to take and what problems to solve.<br>
><br>
> I will be happy to raise github issues based on our conclusions on this mail<br>
> thread.<br>
><br>
> --<br>
> Pranith<br>
><br>
<br>
<br>
<br>
<br>
<br>
</div></div><span class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992m_-1140664742746247274m_2965230752172933805HOEnZb"><font color="#888888">-- <br>
sankarshan mukhopadhyay<br>
<<a href="https://about.me/sankarshan.mukhopadhyay" rel="noreferrer" target="_blank">https://about.me/sankarshan.<wbr>mukhopadhyay</a>><br>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a><br>
</font></span></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992m_-1140664742746247274m_2965230752172933805gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>
______________________________<wbr>_________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/<wbr>mailman/listinfo/gluster-devel</a></blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501m_-6004866552137779909m_5444167284388512992gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>
</blockquote></div>
</div></div></blockquote></div></div></div><span class="gmail-m_6679871537803201214m_2317293645654858914gmail-HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="gmail-m_6679871537803201214m_2317293645654858914gmail-m_-7073584694057990501gmail_signature"><div dir="ltr">Pranith<br></div></div>
</font></span></div></div>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail-m_6679871537803201214m_2317293645654858914gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>
</blockquote></div>
</div></div></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div></div>