<div dir="ltr"><div>Hi,</div><div> </div><div> Yes, you can use cgroup to improve speed limit for gluster operation.</div><div> To use the cgroup for any specific glusterd(selfheald etc) first you have to change the glusterd unit file and start the service (glusterd and volumes).</div><div> </div><div> I would like to suggest before apply the step on large cluster please do run the same steps on test vm and </div><div> if it is working fine on test vm then apply same on your large cluster.</div><div><br></div><div> 1) Add directive "Delegate=yes" in Service section in glusterd unit file as below </div><div><br></div><div>[Service]</div><div>Type=forking</div><div>PIDFile=/var/run/glusterd.pid</div><div>LimitNOFILE=65536</div><div>Delegate=yes</div><div>Environment="LOG_LEVEL=INFO"</div><div>EnvironmentFile=-${prefix}/etc/sysconfig/glusterd</div><div>ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS</div><div>KillMode=process</div><div>SuccessExitStatus=15</div><div> </div><div> 2) stop gluster services (kill volume also)</div><div> 3) do daemon reload systemctl daemon-reload</div><div> 4) start glusterd service</div><div> 5) Create cgroup in subsystems (cpu,cpuacct,memory etc) whatever you want to restrict for specific daemon</div><div> here i am trying to control CPU for selfheald so i have created mycgroup directory in that subsystem </div><div> mkdir -p /sys/fs/cgroup/cpu,cpuacct/system.slice/glusterd.service/mycgroup</div><div> 6) Set quota for this cgroup as below</div><div> echo 25000 > /sys/fs/cgroup/cpu,cpuacct/system.slice/glusterd.service/mycgroup/cpu.cfs_quota_us</div><div> here the cpu.cfs_quota_us means total available run-time within a period (microsecond) and length</div><div> of period is 100 ms , here 25000 represent 25 ms, it means kernel will allocate cpu for these tasks</div><div> for 25ms in every 100 ms</div><div> 7) Attach the daemon tasks for that you want to control CPU</div><div> here i am trying to control selfheal daemon , suppose pid of selfheald is 576, run below command to move</div><div> all selfheald tasks to mycgroup</div><div> for thid in `ps -T -p 576 | grep gluster | awk -F " " '{print $2}'`; do echo $thid > /sys/fs/cgroup/cpu,cpuacct/system.slice/glusterd.service/mycgroup/tasks ; done </div><div> </div><div> 8) After attached all selfheald tasks to mygcgroup to ensure task is successfully attached or not you can check</div><div> tasks file in glusterd.service/tasks in the same hierarchy, all tasks will be move from glusterd tasks to mycgroup</div><div> tasks.</div><div> 9) Check top output, cpu usage for selfheald will be lower.Let me know if u face any problem to run the steps.</div><div><br></div><div> </div><div>Regards</div><div>Mohit Agrawal</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Aug 1, 2017 at 4:19 PM, Ravishankar N <span dir="ltr"><<a href="mailto:ravishankar@redhat.com" target="_blank">ravishankar@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Adding Mohit who is experimenting with cgroups has found some way to restrict glustershd's CPU usage using cgroups. Mohit maybe you want to share the steps we need to follow to apply cgroups only to glustershd.<br>
<br>
Thanks.<br>
<br>
Ravi<br>
<br>
<br>
On 08/01/2017 03:46 PM, Alexey Zakurin wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi, community<br>
<br>
I have a large distributed-replicated Glusterfs volume, that contains few hundreds VM's images. Between servers 20Gb/sec link.<br>
When I start some operations like healing or removing, storage performance becomes too low for a few days and server load becomes like this:<br>
<br>
13:06:32 up 13 days, 20:02, 3 users, load average: 43.62, 31.75, 23.53.<br>
<br>
Is it possible to set limit on this operations? Actually, VM's on my cluster becomes offline, when I start healing, rebalance or removing brick.<br>
<br>
Please, help.<br>
<br>
</blockquote>
<br>
</blockquote></div><br></div>