[Bugs] [Bug 1787993] New: etcd is giving warning key took too long to execute with glusterfs as backend storage when running in production environment

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1787993

            Bug ID: 1787993
           Summary: etcd is giving warning key took too long to execute
                    with glusterfs as backend storage when running in
                    production environment
           Product: GlusterFS
           Version: 5
          Hardware: x86_64
                OS: Linux
            Status: NEW
         Component: io-cache
          Severity: urgent
          Assignee: bugs at gluster.org
          Reporter: sharmaakshay890 at gmail.com
                CC: bugs at gluster.org
  Target Milestone: ---
    Classification: Community



Tagged component(io-cache) as i don't know below issue related to which
component in glusterfs.


Description of problem:
We are running etcd in kubernetes with glusterfs as backend storage.

Version-Release number of selected component (if applicable):
Glusterfs 5.9
Etcd - 3.4.3
kubernetes- 1.9.1


How reproducible:
Mount etcd-pod to glusterfs in kubernetes.
Continuously keep writing key to etcd database using the following command.

while [ 1 ]; do dd if=/dev/urandom bs=1024 count=1024 | ETCDCTL_API=3 etcdctl
put key || break; done

Check the logs of etcd using kubectl logs -f <etcd-pod>
you will get to see below warning.

2020-01-03 11:50:06.591364 W | etcdserver: request
"header:<ID:2814309323035058006 > put:<key:"key" value_size:1048576 >" with
result "size:5" took too long (1.052778581s) to execute
2020-01-03 11:50:07.529154 W | etcdserver: request
"header:<ID:2814309323035058007 > put:<key:"key" value_size:1048576 >" with
result "size:5" took too long (775.427962ms) to execute

Actual results:
Performance is decreasing when running for long time.


Expected results:
Shouldn't see above warning , writing to glusterfs mount should be fast and
smooth.

Additional info:
We are running kafka , logstash and influxdb also in kubernetes with glusterfs
as backed storage .

we have tested below scenarios

case 1- we have mounted etcd to hostpath and kafka to glusterfs
we are not seeing above issue(warning) , performance was good.

case2 - etcd with glusterfs and kafka hostpath
seeing above issue , performance is very bad.

Note: we haven't not tunned any glusterfs parameters explicitly 
we are running glusterfs with default configurations( as what glusterfs comes
up with ).

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.