<div dir="ltr"><br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><br></div></div></div></div></div></div><br><br><div class="gmail_quote"><div dir="ltr">On Wed, Jul 25, 2018 at 11:54 PM Yaniv Kaul <<a href="mailto:ykaul@redhat.com">ykaul@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Jul 24, 2018, 7:20 PM Pranith Kumar Karampuri <<a href="mailto:pkarampu@redhat.com" target="_blank">pkarampu@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>hi,<br></div><div> Quite a few commands to monitor gluster at the moment take almost a second to give output.</div><div>Some categories of these commands:</div><div>1) Any command that needs to do some sort of mount/glfs_init.</div><div> Examples: 1) heal info family of commands 2) statfs to find space-availability etc (On my laptop replica 3 volume with all local bricks, glfs_init takes 0.3 seconds on average)<br></div><div>2) glusterd commands that need to wait for the previous command to unlock. If the previous command is something related to lvm snapshot which takes quite a few seconds, it would be even more time consuming.</div><div><br></div><div>Nowadays container workloads have hundreds of volumes if not thousands. If we want to serve any monitoring solution at this scale (I have seen customers use upto 600 volumes at a time, it will only get bigger) and lets say collecting metrics per volume takes 2 seconds per volume(Let us take the worst example which has all major features enabled like snapshot/geo-rep/quota etc etc), that will mean that it will take 20 minutes to collect metrics of the cluster with 600 volumes. What are the ways in which we can make this number more manageable? I was initially thinking may be it is possible to get gd2 to execute commands in parallel on different volumes, so potentially we could get this done in ~2 seconds. But quite a few of the metrics need a mount or equivalent of a mount(glfs_init) to collect different information like statfs, number of pending heals, quota usage etc. This may lead to high memory usage as the size of the mounts tend to be high.<br> </div><div><br></div><div>I wanted to seek suggestions from others on how to come to a conclusion about which path to take and what problems to solve.<br></div></div></blockquote></div></div><div dir="auto"><br></div><div dir="auto">I would imagine that in gd2 world:</div><div dir="auto">1. All stats would be in etcd. </div></div></blockquote><div><br></div><div>Only static state information stored in etcd by gd2. For real-time status gd2 still has to reach respective nodes to collect the details. For example, Volume utilization is changed by multiple mounts which are external to gd2, to keep track of real-time status gd2 has to poll bricks utilization on every node and update etcd.</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="auto">2. There will be a single API call for GetALLVolumesStats or something and we won't be asking the client to loop, or there will be a similar efficient single API to query and deliver stats for some volumes in a batch ('all bricks in host X' for example). </div></div></blockquote><div><br></div><div>Single API available for Volume stats, but this API is expensive because the real-time stats not stored in etcd.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><div dir="auto"><br></div><div dir="auto">Worth looking how it's implemented elsewhere in K8S.</div><div dir="auto"><br></div><div dir="auto">In any case, when asking for metrics I assume the latest already available would be returned and we are not going to fetch them when queried. This is both fragile (imagine an entity that doesn't respond well) and adds latency and will be inaccurate anyway a split second later. </div><div dir="auto"><br></div><div dir="auto">Y. </div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div></div><div><br></div>I will be happy to raise github issues based on our conclusions on this mail thread.<br><div><br></div>-- <br><div class="m_2901045493560639446m_307996942811566914gmail_signature"><div dir="ltr">Pranith<br></div></div>
</div>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" rel="noreferrer" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a></blockquote></div></div></div>
_______________________________________________<br>
Gluster-devel mailing list<br>
<a href="mailto:Gluster-devel@gluster.org" target="_blank">Gluster-devel@gluster.org</a><br>
<a href="https://lists.gluster.org/mailman/listinfo/gluster-devel" rel="noreferrer" target="_blank">https://lists.gluster.org/mailman/listinfo/gluster-devel</a></blockquote><div><br></div><div><div style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial">--<br>regards<br></div><span style="font-size:small;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Aravinda VK</span> </div></div></div>