[Bugs] [Bug 1273333] New: [Tier]: lookup from client takes too long {~7m for 18k files}

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1273333

            Bug ID: 1273333
           Summary: [Tier]: lookup from client takes too long {~7m for 18k
                    files}
           Product: GlusterFS
           Version: 3.7.5
         Component: tiering
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: rhinduja at redhat.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org



Description of problem:
=======================

On virtual longevity setup, performed stat on each file from fuse client. For
18k files, it took more than 7min. 

Following is the pattern:
 <1> stat is successful on few files
 <2> It gets paused for 30sec-1min
 <3> It resumes and follow step 1 and step 2 again until completion

Once ctr is disabled, the lookup on each file completed in ~3min.

default ctr-enabled:
====================

[root at mia fuse]# find . | wc -l 
18733
[root at mia fuse]# 

[root at dj fuse]# time find . | xargs stat
:        :              :            :             :           :             :
:        :              :            :             :           :             :
:        :              :            :             :           :             :
Modify: 2015-10-19 02:53:53.228000000 +0530
Change: 2015-10-20 00:32:21.784000000 +0530
 Birth: -
  File: ‘./s.644’
  Size: 1048576       Blocks: 2048       IO Block: 131072 regular file
Device: 24h/36d    Inode: 12947151012847994339  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:fusefs_t:s0
Access: 2015-10-19 02:54:03.722000000 +0530
Modify: 2015-10-19 02:54:04.446000000 +0530
Change: 2015-10-20 00:32:46.345000000 +0530
 Birth: -
  File: ‘./dd.972’
  Size: 1048576       Blocks: 2048       IO Block: 131072 regular file
Device: 24h/36d    Inode: 9723623769459433783  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:fusefs_t:s0
Access: 2015-10-15 04:45:57.534000000 +0530
Modify: 2015-10-15 04:45:57.570000000 +0530
Change: 2015-10-20 00:33:14.208000000 +0530
 Birth: -

real    7m41.900s
user    0m1.510s
sys    0m3.217s
[root at dj fuse]# 


Disabled ctr: "ctr-enabled off"
===============================


[root at dj fuse]# 
[root at dj fuse]# time find . | xargs stat
:        :              :            :             :           :             :
:        :              :            :             :           :             :
:        :              :            :             :           :             :
Modify: 2015-10-19 03:21:01.914000000 +0530
Change: 2015-10-20 01:54:02.067000000 +0530
 Birth: -
  File: ‘./dd.259’
  Size: 1048576       Blocks: 2048       IO Block: 131072 regular file
Device: 24h/36d    Inode: 12399192322619829042  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:fusefs_t:s0
Access: 2015-10-15 02:38:20.231000000 +0530
Modify: 2015-10-15 02:38:20.281000000 +0530
Change: 2015-10-20 01:54:05.228000000 +0530
 Birth: -
  File: ‘./s.246’
  Size: 1048576       Blocks: 2048       IO Block: 131072 regular file
Device: 24h/36d    Inode: 12784199456817109142  Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Context: system_u:object_r:fusefs_t:s0
Access: 2015-10-19 01:39:07.175000000 +0530
Modify: 2015-10-19 01:39:07.848000000 +0530
Change: 2015-10-20 01:54:12.182000000 +0530
 Birth: -

real    3m11.742s
user    0m1.244s
sys    0m2.482s
[root at dj fuse]# 



Version-Release number of selected component (if applicable):
=============================================================

glusterfs-3.7.5-0.19.git0f5c3e8.el7.centos.x86_64


How reproducible:
=================

Always


Steps Carried:
==============
1. Created 12 node cluster
2. Create tiered volume with Hot tier as (6 x 2) and Cold tier as (2 x (6 + 2)
= 16)
3. Fuse Mount the volume on 3 clients RHEL7.2,RHEl7.1 and RHEL6.7
4. Start creating data from each client:

Client 1:
=========
[root at dj ~]# crefi --multi -n 10 -b 10 -d 10 --max=1024k --min=5k --random -T 5
-t text -I 5 --fop=create /mnt/fuse/

Client 2:
=========
[root at mia ~]# cd /mnt/fuse/
[root at mia fuse]# for i in {1..10}; do cp -rf /etc etc.$i ; sleep 100 ; done

Client 3:
=========
[root at wingo fuse]# for i in {1..999}; do dd if=/dev/zero of=dd.$i bs=1M count=1
; sleep 10 ; done

5. After a while, the data creation of client 1 and client 2 should be
completed while the data creation from client 3 will still be inprogress

6. At this point the data creation will be of only 1 file from client 3 in
every 10 sec.

7. Once data creation is completed. Perform "find . | xargs stat" from any of
one client . Note the time it takes to complete this task.

8. disable ctr "ctr-enabled off"

9. Perform "find . | xargs stat" from any of one client . Note the time it
takes to complete this task.


Actual results:
===============

With default ctr-enabled, the lookup from clients are too slow.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.