[Gluster-users] Gluster operations speed limit

[Mohit Agrawal]

Hi,

   Yes, you can use cgroup to improve speed limit for gluster operation.
   To use the cgroup for any specific glusterd(selfheald etc) first  you
have to change the glusterd unit file and start the service (glusterd and
volumes).

   I would like to suggest before apply the step on large cluster please do
run the same steps on test vm and
   if it is working fine on test vm then apply same on your large cluster.

   1) Add directive "Delegate=yes" in Service section in glusterd unit file
as below

[Service]
Type=forking
PIDFile=/var/run/glusterd.pid
LimitNOFILE=65536
Delegate=yes
Environment="LOG_LEVEL=INFO"
EnvironmentFile=-${prefix}/etc/sysconfig/glusterd
ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid  --log-level
$LOG_LEVEL $GLUSTERD_OPTIONS
KillMode=process
SuccessExitStatus=15

   2) stop gluster services (kill volume also)
   3) do daemon reload systemctl daemon-reload
   4) start glusterd service
   5) Create cgroup in subsystems (cpu,cpuacct,memory etc) whatever you
want to restrict for specific daemon
      here i am trying to control CPU for selfheald so i have created
mycgroup directory in that subsystem
      mkdir -p
/sys/fs/cgroup/cpu,cpuacct/system.slice/glusterd.service/mycgroup
   6) Set quota for this cgroup as below
      echo 25000 >
/sys/fs/cgroup/cpu,cpuacct/system.slice/glusterd.service/mycgroup/cpu.cfs_quota_us
      here the cpu.cfs_quota_us means total available run-time within a
period (microsecond) and length
      of period is 100 ms , here 25000 represent 25 ms, it means kernel
will allocate cpu for these tasks
      for 25ms in every 100 ms
   7) Attach the daemon tasks for that you want to control CPU
      here i am trying to control selfheal daemon , suppose pid of
selfheald is 576, run below command to move
      all selfheald tasks to mycgroup
      for thid in `ps -T -p 576 | grep gluster | awk -F " " '{print $2}'`;
do echo $thid >
/sys/fs/cgroup/cpu,cpuacct/system.slice/glusterd.service/mycgroup/tasks ;
done

   8) After attached all selfheald tasks to mygcgroup to ensure task is
successfully attached or not you can check
      tasks file in glusterd.service/tasks in the same hierarchy, all tasks
will be move from glusterd tasks to mycgroup
      tasks.
   9) Check top output, cpu usage for selfheald will be lower.Let me know
if u face any problem to run the steps.


Regards
Mohit Agrawal

On Tue, Aug 1, 2017 at 4:19 PM, Ravishankar N <ravishankar at redhat.com>
wrote:

> Adding Mohit who is experimenting with cgroups  has found some way to
> restrict glustershd's CPU usage using cgroups. Mohit maybe you want to
> share the steps we need to follow to apply cgroups only to glustershd.
>
> Thanks.
>
> Ravi
>
>
> On 08/01/2017 03:46 PM, Alexey Zakurin wrote:
>
>> Hi, community
>>
>> I have a large distributed-replicated Glusterfs volume, that contains few
>> hundreds VM's images. Between servers 20Gb/sec link.
>> When I start some operations like healing or removing, storage
>> performance becomes too low for a few days and server load becomes like
>> this:
>>
>> 13:06:32 up 13 days, 20:02,  3 users,  load average: 43.62, 31.75, 23.53.
>>
>> Is it possible to set limit on this operations? Actually, VM's on my
>> cluster becomes offline, when I start healing, rebalance or removing brick.
>>
>> Please, help.
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20170803/88b7a00a/attachment.html>