[Gluster-users] Extremely slow cluster performance
Patrick Rennie
patrickmrennie at gmail.com
Sun Apr 21 14:24:34 UTC 2019
Hi Strahil,
Thank you for your reply and your suggestions. I'm not sure which logs
would be most relevant to be checking to diagnose this issue, we have the
brick logs, the cluster mount logs, the shd logs or something else? I have
posted a few that I have seen repeated a few times already. I will continue
to post anything further that I see.
I am working on migrating data to some new storage, so this will slowly
free up space, although this is a production cluster and new data is being
uploaded every day, sometimes faster than I can migrate it off. I have
several other similar clusters and none of them have the same problem, one
the others is actually at 98-99% right now (big problem, I know) but still
performs perfectly fine compared to this cluster, I am not sure low space
is the root cause here.
I currently have 13 VMs accessing this cluster, I have checked each one and
all of them use one of the two options below to mount the cluster in fstab
HOSTNAME:/gvAA01 /mountpoint glusterfs
defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable,use-readdirp=no
0 0
HOSTNAME:/gvAA01 /mountpoint glusterfs
defaults,_netdev,rw,log-level=WARNING,direct-io-mode=disable
I also have a few other VMs which use NFS to access the cluster, and these
machines appear to be significantly quicker, initially I get a similar
delay with NFS but if I cancel the first "ls" and try it again I get < 1
sec lookups, this can take over 10 minutes by FUSE/gluster client, but the
same trick of cancelling and trying again doesn't work for FUSE/gluster.
Sometimes the NFS queries have no delay at all, so this is a bit strange to
me.
HOSTNAME:/gvAA01 /mountpoint/ nfs
defaults,_netdev,vers=3,async,noatime 0 0
Example:
user at VM:~$ time ls /cluster/folder
^C
real 9m49.383s
user 0m0.001s
sys 0m0.010s
user at VM:~$ time ls /cluster/folder
<results>
real 0m0.069s
user 0m0.001s
sys 0m0.007s
---
I have checked the profiling as you suggested, I let it run for around a
minute, then cancelled it and saved the profile info.
root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 start
Starting volume profile on gvAA01 has been successful
root at HOSTNAME:/var/log/glusterfs# time ls /cluster/folder
^C
real 1m1.660s
user 0m0.000s
sys 0m0.002s
root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 info >>
~/profile.txt
root at HOSTNAME:/var/log/glusterfs# gluster volume profile gvAA01 stop
I will attach the results to this email as it's over 1000 lines.
Unfortunately, I'm not sure what I'm looking at but possibly somebody will
be able to help me make sense of it and let me know if it highlights any
specific issues.
Happy to try any further suggestions. Thank you,
-Patrick
On Sun, Apr 21, 2019 at 7:55 PM Strahil <hunter86_bg at yahoo.com> wrote:
> By the way, can you provide the 'volume info' and the mount options on all
> clients?
> Maybe , there is an option that uses a lot of resources due to some
> client's mount options.
>
> Best Regards,
> Strahil Nikolov
> On Apr 21, 2019 10:55, Patrick Rennie <patrickmrennie at gmail.com> wrote:
>
> Just another small update, I'm continuing to watch my brick logs and I
> just saw these errors come up in the recent events too. I am going to
> continue to post any errors I see in the hope of finding the right one to
> try and fix..
> This is from the logs on brick1, seems to be occurring on both nodes on
> brick1, although at different times. I'm not sure what this means, can
> anyone shed any light?
> I guess I am looking for some kind of specific error which may indicate
> something is broken or stuck and locking up and causing the extreme latency
> I'm seeing in the cluster.
>
> [2019-04-21 07:25:55.064497] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c700c, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 29) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.064612] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e58a)
> [0x7f3b3e93158a]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17d45)
> [0x7f3b3e4c5d45]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.064675] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c70af, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.064705] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.064742] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c723c, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.064768] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.064812] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c72b4, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.064837] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.064880] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c740b, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.064905] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.064939] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c7441, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.064962] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.064996] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c74d5, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.065020] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.065052] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c7551, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.065076] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
> [2019-04-21 07:25:55.065110] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x7c76d1, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 30) to rpc-transport (tcp.gvAA01-server)
> [2019-04-21 07:25:55.065133] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/debug/io-stats.so(+0x1e8fa)
> [0x7f3b3e9318fa]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x17f35)
> [0x7f3b3e4c5f35]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.15/xlator/protocol/server.so(+0x92cd)
> [0x7f3b3e4b72cd] ) 0-: Reply submission failed
>
> Thanks again,
>
> -Patrick
>
> On Sun, Apr 21, 2019 at 3:50 PM Patrick Rennie <patrickmrennie at gmail.com>
> wrote:
>
> Hi Darrell,
>
> Thanks again for your advice, I've left it for a while but unfortunately
> it's still just as slow and causing more problems for our operations now. I
> will need to try and take some steps to at least bring performance back to
> normal while continuing to investigate the issue longer term. I can
> definitely see one node with heavier CPU than the other, almost double,
> which I am OK with, but I think the heal process is going to take forever,
> trying to check the "gluster volume heal info" shows thousands and
> thousands of files which may need healing, I have no idea how many in total
> the command is still running after hours, so I am not sure what has gone so
> wrong to cause this.
>
> I've checked cluster.op-version and cluster.max-op-version and it looks
> like I'm on the latest version there.
>
> I have no idea how long the healing is going to take on this cluster, we
> have around 560TB of data on here, but I don't think I can wait that long
> to try and restore performance to normal.
>
> Can anyone think of anything else I can try in the meantime to work out
> what's causing the extreme latency?
>
> I've been going through cluster client the logs of some of our VMs and on
> some of our FTP servers I found this in the cluster mount log, but I am not
> seeing it on any of our other servers, just our FTP servers.
>
> [2019-04-21 07:16:19.925388] E [MSGID: 101046]
> [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null
> [2019-04-21 07:19:43.413834] W [MSGID: 114031]
> [client-rpc-fops.c:2203:client3_3_setattr_cbk] 0-gvAA01-client-19: remote
> operation failed [No such file or directory]
> [2019-04-21 07:19:43.414153] W [MSGID: 114031]
> [client-rpc-fops.c:2203:client3_3_setattr_cbk] 0-gvAA01-client-20: remote
> operation failed [No such file or directory]
> [2019-04-21 07:23:33.154717] E [MSGID: 101046]
> [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null
> [2019-04-21 07:33:24.943913] E [MSGID: 101046]
> [dht-common.c:1904:dht_revalidate_cbk] 0-gvAA01-dht: dict is null
>
> Any ideas what this could mean? I am basically just grasping at straws
> here.
>
> I am going to hold off on the version upgrade until I know there are no
> files which need healing, which could be a while, from some reading I've
> done there shouldn't be any issues with this as both are on v3.12.x
>
> I've free'd up a small amount of space, but I still need to work on this
> further.
>
> I've read of a command "find .glusterfs -type f -links -2 -exec rm {} \;"
> which could be run on each brick and it would potentially clean up any
> files which were deleted straight from the bricks, but not via the client,
> I have a feeling this could help me free up about 5-10TB per brick from
> what I've been told about the history of this cluster. Can anyone confirm
> if this is actually safe to run?
>
> At this stage, I'm open to any suggestions as to how to proceed, thanks
> again for any advice.
>
> Cheers,
>
> - Patrick
>
> On Sun, Apr 21, 2019 at 1:22 AM Darrell Budic <budic at onholyground.com>
> wrote:
>
> Patrick,
>
> Sounds like progress. Be aware that gluster is expected to max out the
> CPUs on at least one of your servers while healing. This is normal and
> won’t adversely affect overall performance (any more than having bricks in
> need of healing, at any rate) unless you’re overdoing it. shd threads <= 4
> should not do that on your hardware. Other tunings may have also increased
> overall performance, so you may see higher CPU than previously anyway. I’d
> recommend upping those thread counts and letting it heal as fast as
> possible, especially if these are dedicated Gluster storage servers (Ie:
> not also running VMs, etc). You should see “normal” CPU use one heals are
> completed. I see ~15-30% overall normally, 95-98% while healing (x my 20
> cores). It’s also likely to be different between your servers, in a pure
> replica, one tends to max and one tends to be a little higher, in a
> distributed-replica, I’d expect more than one to run harder while healing.
>
> Keep the differences between doing an ls on a brick and doing an ls on a
> gluster mount in mind. When you do a ls on a gluster volume, it isn’t just
> doing a ls on one brick, it’s effectively doing it on ALL of your bricks,
> and they all have to return data before the ls succeeds. In a distributed
> volume, it’s figuring out where on each volume things live and getting the
> stat() from each to assemble the whole thing. And if things are in need of
> healing, it will take even longer to decide which version is current and
> use it (shd triggers a heal anytime it encounters this). Any of these
> things being slow slows down the overall response.
>
> At this point, I’d get some sleep too, and let your cluster heal while you
> do. I’d really want it fully healed before I did any updates anyway, so let
> it use CPU and get itself sorted out. Expect it to do a round of healing
> after you upgrade each machine too, this is normal so don’t let the CPU
> spike surprise you, It’s just catching up from the downtime incurred by the
> update and/or reboot if you did one.
>
> That reminds me, check your gluster cluster.op-version and
> cluster.max-op-version (gluster vol get all all | grep op-version). If
> op-version isn’t at the max-op-verison, set it to it so you’re taking
> advantage of the latest features available to your version.
>
> -Darrell
>
> On Apr 20, 2019, at 11:54 AM, Patrick Rennie <patrickmrennie at gmail.com>
> wrote:
>
> Hi Darrell,
>
> Thanks again for your advice, I've applied the acltype=posixacl on my
> zpools and I think that has reduced some of the noise from my brick logs.
> I also bumped up some of the thread counts you suggested but my CPU load
> skyrocketed, so I dropped it back down to something slightly lower, but
> still higher than it was before, and will see how that goes for a while.
>
> Although low space is a definite issue, if I run an ls anywhere on my
> bricks directly it's instant, <1 second, and still takes several minutes
> via gluster, so there is still a problem in my gluster configuration
> somewhere. We don't have any snapshots, but I am trying to work out if any
> data on there is safe to delete, or if there is any way I can safely find
> and delete data which has been removed directly from the bricks in the
> past. I also have lz4 compression already enabled on each zpool which does
> help a bit, we get between 1.05 and 1.08x compression on this data.
> I've tried to go through each client and checked it's cluster mount logs
> and also my brick logs and looking for errors, so far nothing is jumping
> out at me, but there are some warnings and errors here and there, I am
> trying to work out what they mean.
>
> It's already 1 am here and unfortunately, I'm still awake working on this
> issue, but I think that I will have to leave the version upgrades until
> tomorrow.
>
> Thanks again for your advice so far. If anyone has any ideas on where I
> can look for errors other than brick logs or the cluster mount logs to help
> resolve this issue, it would be much appreciated.
>
> Cheers,
>
> - Patrick
>
> On Sat, Apr 20, 2019 at 11:57 PM Darrell Budic <budic at onholyground.com>
> wrote:
>
> See inline:
>
> On Apr 20, 2019, at 10:09 AM, Patrick Rennie <patrickmrennie at gmail.com>
> wrote:
>
> Hi Darrell,
>
> Thanks for your reply, this issue seems to be getting worse over the last
> few days, really has me tearing my hair out. I will do as you have
> suggested and get started on upgrading from 3.12.14 to 3.12.15.
> I've checked the zfs properties and all bricks have "xattr=sa" set, but
> none of them has "acltype=posixacl" set, currently the acltype property
> shows "off", if I make these changes will it apply retroactively to the
> existing data? I'm unfamiliar with what this will change so I may need to
> look into that before I proceed.
>
>
> It is safe to apply that now, any new set/get calls will then use it if
> new posixacls exist, and use older if not. ZFS is good that way. It should
> clear up your posix_acl and posix errors over time.
>
> I understand performance is going to slow down as the bricks get full, I
> am currently trying to free space and migrate data to some newer storage, I
> have fresh several hundred TB storage I just setup recently but with these
> performance issues it's really slow. I also believe there is significant
> data which has been deleted directly from the bricks in the past, so if I
> can reclaim this space in a safe manner then I will have at least around
> 10-15% free space.
>
>
> Full ZFS volumes will have a much larger impact on performance than you’d
> think, I’d prioritize this. If you have been taking zfs snapshots, consider
> deleting them to get the overall volume free space back up. And just to be
> sure it’s been said, delete from within the mounted volumes, don’t delete
> directly from the bricks (gluster will just try and heal it later,
> compounding your issues). Does not apply to deleting other data from the
> ZFS volume if it’s not part of the brick directory, of course.
>
> These servers have dual 8 core Xeon (E5-2620v4) and 512GB of RAM so
> generally they have plenty of resources available, currently only using
> around 330/512GB of memory.
>
> I will look into what your suggested settings will change, and then will
> probably go ahead with your recommendations, for our specs as stated above,
> what would you suggest for performance.io-thread-count ?
>
>
> I run single 2630v4s on my servers, which have a smaller storage footprint
> than yours. I’d go with 32 for performance.io-thread-count. I’d try 4 for
> the shd thread settings on that gear. Your memory use sounds fine, so no
> worries there.
>
> Our workload is nothing too extreme, we have a few VMs which write backup
> data to this storage nightly for our clients, our VMs don't live on this
> cluster, but just write to it.
>
>
> If they are writing compressible data, you’ll get immediate benefit by
> setting compression=lz4 on your ZFS volumes. It won’t help any old data, of
> course, but it will compress new data going forward. This is another one
> that’s safe to enable on the fly.
>
> I've been going through all of the logs I can, below are some slightly
> sanitized errors I've come across, but I'm not sure what to make of them.
> The main error I am seeing is the first one below, across several of my
> bricks, but possibly only for specific folders on the cluster, I'm not 100%
> about that yet though.
>
> [2019-04-20 05:56:59.512649] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not
> supported]
> [2019-04-20 05:59:06.084333] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not
> supported]
> [2019-04-20 05:59:43.289030] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not
> supported]
> [2019-04-20 05:59:50.582257] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not
> supported]
> [2019-04-20 06:01:42.501701] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick7/xxxxxxxxxxxxxxxxxxxx: system.posix_acl_default [Operation not
> supported]
> [2019-04-20 06:01:51.665354] W [posix.c:4929:posix_getxattr]
> 0-gvAA01-posix: Extended attributes not supported (try remounting brick
> with 'user_xattr' flag)
>
>
> [2019-04-20 13:12:36.131856] E [MSGID: 113002]
> [posix-helpers.c:893:posix_gfid_set] 0-gvAA01-posix: gfid is null for
> /xxxxxxxxxxxxxxxxxxxx [Invalid argument]
> [2019-04-20 13:12:36.131959] E [MSGID: 113002] [posix.c:362:posix_lookup]
> 0-gvAA01-posix: buf->ia_gfid is null for
> /brick2/xxxxxxxxxxxxxxxxxxxx_62906_tmp [No data available]
> [2019-04-20 13:12:36.132016] E [MSGID: 115050]
> [server-rpc-fops.c:175:server_lookup_cbk] 0-gvAA01-server: 24274759: LOOKUP
> /xxxxxxxxxxxxxxxxxxxx (a7c9b4a0-b7ee-4d01-a79e-576013c8ac87/Cloud
> Backup_clone1.vbm_62906_tmp), client:
> 00-A-16217-2019/04/08-21:23:03:692424-gvAA01-client-4-0-3, error-xlator:
> gvAA01-posix [No data available]
> [2019-04-20 13:12:38.093719] E [MSGID: 115050]
> [server-rpc-fops.c:175:server_lookup_cbk] 0-gvAA01-server: 24276491: LOOKUP
> /xxxxxxxxxxxxxxxxxxxx (a7c9b4a0-b7ee-4d01-a79e-576013c8ac87/Cloud
> Backup_clone1.vbm_62906_tmp), client:
> 00-A-16217-2019/04/08-21:23:03:692424-gvAA01-client-4-0-3, error-xlator:
> gvAA01-posix [No data available]
> [2019-04-20 13:12:38.093660] E [MSGID: 113002]
> [posix-helpers.c:893:posix_gfid_set] 0-gvAA01-posix: gfid is null for
> /xxxxxxxxxxxxxxxxxxxx [Invalid argument]
> [2019-04-20 13:12:38.093696] E [MSGID: 113002] [posix.c:362:posix_lookup]
> 0-gvAA01-posix: buf->ia_gfid is null for /brick2/xxxxxxxxxxxxxxxxxxxx [No
> data available]
>
>
> posixacls should clear those up, as mentioned.
>
>
> [2019-04-20 14:25:59.654576] E [inodelk.c:404:__inode_unlock_lock]
> 0-gvAA01-locks: Matching lock not found for unlock 0-9223372036854775807,
> by 980fdbbd367f0000 on 0x7fc4f0161440
> [2019-04-20 14:25:59.654668] E [MSGID: 115053]
> [server-rpc-fops.c:295:server_inodelk_cbk] 0-gvAA01-server: 6092928:
> INODELK /xxxxxxxxxxxxxxxxxxxx.cdr$ (25b14631-a179-4274-8243-6e272d4f2ad8),
> client:
> cb-per-worker18-53637-2019/04/19-14:25:37:927673-gvAA01-client-1-0-4,
> error-xlator: gvAA01-locks [Invalid argument]
>
>
> [2019-04-20 13:35:07.495495] E [rpcsvc.c:1364:rpcsvc_submit_generic]
> 0-rpc-service: failed to submit message (XID: 0x247c644, Program: GlusterFS
> 3.3, ProgVers: 330, Proc: 27) to rpc-transport (tcp.gvAA01-server)
> [2019-04-20 13:35:07.495619] E [server.c:195:server_submit_reply]
> (-->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/debug/io-stats.so(+0x1696a)
> [0x7ff4ae6f796a]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/protocol/server.so(+0x2d6e8)
> [0x7ff4ae2a96e8]
> -->/usr/lib/x86_64-linux-gnu/glusterfs/3.12.14/xlator/protocol/server.so(+0x928d)
> [0x7ff4ae28528d] ) 0-: Reply submission failed
>
>
> Fix the posix acls and see if these clear up over time as well, I’m
> unclear on what the overall effect of running without the posix acls will
> be to total gluster health. Your biggest problem sounds like you need to
> free up space on the volumes and get the overall volume health back up to
> par and see if that doesn’t resolve the symptoms you’re seeing.
>
>
>
> Thank you again for your assistance. It is greatly appreciated.
>
> - Patrick
>
>
>
> On Sat, Apr 20, 2019 at 10:50 PM Darrell Budic <budic at onholyground.com>
> wrote:
>
> Patrick,
>
> I would definitely upgrade your two nodes from 3.12.14 to 3.12.15. You
> also mention ZFS, and that error you show makes me think you need to check
> to be sure you have “xattr=sa” and “acltype=posixacl” set on your ZFS
> volumes.
>
> You also observed your bricks are crossing the 95% full line, ZFS
> performance will degrade significantly the closer you get to full. In my
> experience, this starts somewhere between 10% and 5% free space remaining,
> so you’re in that realm.
>
> How’s your free memory on the servers doing? Do you have your zfs arc
> cache limited to something less than all the RAM? It shares pretty well,
> but I’ve encountered situations where other things won’t try and take ram
> back properly if they think it’s in use, so ZFS never gets the opportunity
> to give it up.
>
> Since your volume is a disperse-replica, you might try tuning
> disperse.shd-max-threads, default is 1, I’d try it at 2, 4, or even more if
> the CPUs are beefy enough. And setting server.event-threads to 4 and
> client.event-threads to 8 has proven helpful in many cases. After you get
> upgraded to 3.12.15, enabling performance.stat-prefetch may help as well. I
> don’t know if it matters, but I’d also recommend resetting
> performance.least-prio-threads to the default of 1 (or try 2 or 4) and/or
> also setting performance.io-thread-count to 32 if those have beefy CPUs.
>
> Beyond those general ideas, more info about your hardware (CPU and RAM)
> and workload (VMs, direct storage for web servers or enders, etc) may net
> you some more ideas. Then you’re going to have to do more digging into
> brick logs looking for errors and/or warnings to see what’s going on.
>
> -Darrell
>
>
> On Apr 20, 2019, at 8:22 AM, Patrick Rennie <patrickmrennie at gmail.com>
> wrote:
>
> Hello Gluster Users,
>
> I am hoping someone can help me with resolving an ongoing issue I've been
> having, I'm new to mailing lists so forgive me if I have gotten anything
> wrong. We have noticed our performance deteriorating over the last few
> weeks, easily measured by trying to do an ls on one of our top-level
> folders, and timing it, which usually would take 2-5 seconds, and now takes
> up to 20 minutes, which obviously renders our cluster basically unusable.
> This has been intermittent in the past but is now almost constant and I am
> not sure how to work out the exact cause. We have noticed some errors in
> the brick logs, and have noticed that if we kill the right brick process,
> performance instantly returns back to normal, this is not always the same
> brick, but it indicates to me something in the brick processes or
> background tasks may be causing extreme latency. Due to this ability to fix
> it by killing the right brick process off, I think it's a specific file, or
> folder, or operation which may be hanging and causing the increased
> latency, but I am not sure how to work it out. One last thing to add is
> that our bricks are getting quite full (~95% full), we are trying to
> migrate data off to new storage but that is going slowly, not helped by
> this issue. I am currently trying to run a full heal as there appear to be
> many files needing healing, and I have all brick processes running so they
> have an opportunity to heal, but this means performance is very poor. It
> currently takes over 15-20 minutes to do an ls of one of our top-level
> folders, which just contains 60-80 other folders, this should take 2-5
> seconds. This is all being checked by FUSE mount locally on the storage
> node itself, but it is the same for other clients and VMs accessing the
> cluster. Initially, it seemed our NFS mounts were not affected and operated
> at normal speed, but testing over the last day has shown that our NFS
> clients are also extremely slow, so it doesn't seem specific to FUSE as I
> first thought it might be.
>
> I am not sure how to proceed from here, I am fairly new to gluster having
> inherited this setup from my predecessor and trying to keep it going. I
> have included some info below to try and help with diagnosis, please let me
> know if any further info would be helpful. I would really appreciate any
> advice on what I could try to work out the cause. Thank you in advance for
> reading this, and any suggestions you might be able to offer.
>
> - Patrick
>
> This is an example of the main error I see in our brick logs, there have
> been others, I can post them when I see them again too:
> [2019-04-20 04:54:43.055680] E [MSGID: 113001]
> [posix.c:4940:posix_getxattr] 0-gvAA01-posix: getxattr failed on
> /brick1/<filename> library: system.posix_acl_default [Operation not
> supported]
> [2019-04-20 05:01:29.476313] W [posix.c:4929:posix_getxattr]
> 0-gvAA01-posix: Extended attributes not supported (try remounting brick
> with 'user_xattr' flag)
>
> Our setup consists of 2 storage nodes and an arbiter node. I have noticed
> our nodes are on slightly different versions, I'm not sure if this could be
> an issue. We have 9 bricks on each node, made up of ZFS RAIDZ2 pools -
> total capacity is around 560TB.
> We have bonded 10gbps NICS on each node, and I have tested bandwidth with
> iperf and found that it's what would be expected from this config.
> Individual brick performance seems ok, I've tested several bricks using dd
> and can write a 10GB files at 1.7GB/s.
>
> # dd if=/dev/zero of=/brick1/test/test.file bs=1M count=10000
> 10000+0 records in
> 10000+0 records out
> 10485760000 bytes (10 GB, 9.8 GiB) copied, 6.20303 s, 1.7 GB/s
>
> Node 1:
> # glusterfs --version
> glusterfs 3.12.15
>
> Node 2:
> # glusterfs --version
> glusterfs 3.12.14
>
> Arbiter:
> # glusterfs --version
> glusterfs 3.12.14
>
> Here is our gluster volume status:
>
> # gluster volume status
> Status of volume: gvAA01
> Gluster process TCP Port RDMA Port Online
> Pid
>
> ------------------------------------------------------------------------------
> Brick 01-B:/brick1/gvAA01/brick 49152 0 Y 7219
> Brick 02-B:/brick1/gvAA01/brick 49152 0 Y 21845
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck1 49152 0 Y
> 6931
> Brick 01-B:/brick2/gvAA01/brick 49153 0 Y 7239
> Brick 02-B:/brick2/gvAA01/brick 49153 0 Y 9916
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck2 49153 0 Y
> 6939
> Brick 01-B:/brick3/gvAA01/brick 49154 0 Y 7235
> Brick 02-B:/brick3/gvAA01/brick 49154 0 Y 21858
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck3 49154 0 Y
> 6947
> Brick 01-B:/brick4/gvAA01/brick 49155 0 Y 31840
> Brick 02-B:/brick4/gvAA01/brick 49155 0 Y 9933
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck4 49155 0 Y
> 6956
> Brick 01-B:/brick5/gvAA01/brick 49156 0 Y 7233
> Brick 02-B:/brick5/gvAA01/brick 49156 0 Y 9942
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck5 49156 0 Y
> 6964
> Brick 01-B:/brick6/gvAA01/brick 49157 0 Y 7234
> Brick 02-B:/brick6/gvAA01/brick 49157 0 Y 9952
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck6 49157 0 Y
> 6974
> Brick 01-B:/brick7/gvAA01/brick 49158 0 Y 7248
> Brick 02-B:/brick7/gvAA01/brick 49158 0 Y 9960
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck7 49158 0 Y
> 6984
> Brick 01-B:/brick8/gvAA01/brick 49159 0 Y 7253
> Brick 02-B:/brick8/gvAA01/brick 49159 0 Y 9970
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck8 49159 0 Y
> 6993
> Brick 01-B:/brick9/gvAA01/brick 49160 0 Y 7245
> Brick 02-B:/brick9/gvAA01/brick 49160 0 Y 9984
> Brick 00-A:/arbiterAA01/gvAA01/bri
> ck9 49160 0 Y
> 7001
> NFS Server on localhost 2049 0 Y
> 17276
> Self-heal Daemon on localhost N/A N/A Y
> 25245
> NFS Server on 02-B 2049 0 Y 9089
> Self-heal Daemon on 02-B N/A N/A Y 17838
> NFS Server on 00-a 2049 0 Y 15660
> Self-heal Daemon on 00-a N/A N/A Y 16218
>
> Task Status of Volume gvAA01
>
> ------------------------------------------------------------------------------
> There are no active volume tasks
>
> And gluster volume info:
>
> # gluster volume info
>
> Volume Name: gvAA01
> Type: Distributed-Replicate
> Volume ID: ca4ece2c-13fe-414b-856c-2878196d6118
> Status: Started
> Snapshot Count: 0
> Number of Bricks: 9 x (2 + 1) = 27
> Transport-type: tcp
> Bricks:
> Brick1: 01-B:/brick1/gvAA01/brick
> Brick2: 02-B:/brick1/gvAA01/brick
> Brick3: 00-A:/arbiterAA01/gvAA01/brick1 (arbiter)
> Brick4: 01-B:/brick2/gvAA01/brick
> Brick5: 02-B:/brick2/gvAA01/brick
> Brick6: 00-A:/arbiterAA01/gvAA01/brick2 (arbiter)
> Brick7: 01-B:/brick3/gvAA01/brick
> Brick8: 02-B:/brick3/gvAA01/brick
> Brick9: 00-A:/arbiterAA01/gvAA01/brick3 (arbiter)
> Brick10: 01-B:/brick4/gvAA01/brick
> Brick11: 02-B:/brick4/gvAA01/brick
> Brick12: 00-A:/arbiterAA01/gvAA01/brick4 (arbiter)
> Brick13: 01-B:/brick5/gvAA01/brick
> Brick14: 02-B:/brick5/gvAA01/brick
> Brick15: 00-A:/arbiterAA01/gvAA01/brick5 (arbiter)
> Brick16: 01-B:/brick6/gvAA01/brick
> Brick17: 02-B:/brick6/gvAA01/brick
> Brick18: 00-A:/arbiterAA01/gvAA01/brick6 (arbiter)
> Brick19: 01-B:/brick7/gvAA01/brick
> Brick20: 02-B:/brick7/gvAA01/brick
> Brick21: 00-A:/arbiterAA01/gvAA01/brick7 (arbiter)
> Brick22: 01-B:/brick8/gvAA01/brick
> Brick23: 02-B:/brick8/gvAA01/brick
> Brick24: 00-A:/arbiterAA01/gvAA01/brick8 (arbiter)
> Brick25: 01-B:/brick9/gvAA01/brick
> Brick26: 02-B:/brick9/gvAA01/brick
> Brick27: 00-A:/arbiterAA01/gvAA01/brick9 (arbiter)
> Options Reconfigured:
> cluster.shd-max-threads: 4
> performance.least-prio-threads: 16
> cluster.readdir-optimize: on
> performance.quick-read: off
> performance.stat-prefetch: off
> cluster.data-self-heal: on
> cluster.lookup-unhashed: auto
> cluster.lookup-optimize: on
> cluster.favorite-child-policy: mtime
> server.allow-insecure: on
> transport.address-family: inet
> client.bind-insecure: on
> cluster.entry-self-heal: off
> cluster.metadata-self-heal: off
> performance.md-cache-timeout: 600
> cluster.self-heal-daemon: enable
> performance.readdir-ahead: on
> diagnostics.brick-log-level: INFO
> nfs.disable: off
>
> Thank you for any assistance.
>
> - Patrick
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20190421/97818bce/attachment.html>
-------------- next part --------------
Brick: HOSTNAME-02-B:/brick1/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 32b+ 64b+ 128b+
No. of Reads: 1 0 0
No. of Writes: 138 7 45
Block Size: 256b+ 512b+ 1024b+
No. of Reads: 0 0 0
No. of Writes: 1 588 8321
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 2 5
No. of Writes: 9294 88957 21544
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 13 22 41
No. of Writes: 253312 24305 207953
Block Size: 131072b+
No. of Reads: 415121
No. of Writes: 632261
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1656 FORGET
0.00 0.00 us 0.00 us 0.00 us 3098 RELEASE
0.00 0.00 us 0.00 us 0.00 us 30293 RELEASEDIR
0.00 38.00 us 38.00 us 38.00 us 1 FLUSH
0.00 881.00 us 881.00 us 881.00 us 1 UNLINK
0.00 252705.00 us 252705.00 us 252705.00 us 1 LK
0.00 45581.06 us 275.00 us 760038.00 us 17 SETXATTR
0.00 178264.40 us 1180.00 us 567795.00 us 5 MKNOD
0.01 52932.15 us 269.00 us 710452.00 us 34 SETATTR
0.01 56835.03 us 22.00 us 875479.00 us 34 GETXATTR
0.01 33009.92 us 55.00 us 690160.00 us 59 READ
0.01 130547.06 us 199.00 us 746238.00 us 17 REMOVEXATTR
0.01 101512.69 us 45.00 us 898760.00 us 35 READDIR
0.02 176200.21 us 79.00 us 767224.00 us 24 READDIRP
0.03 149224.76 us 1210.00 us 847273.00 us 50 MKDIR
0.04 477780.00 us 33.00 us 894975.00 us 20 FSTAT
0.04 294297.09 us 766.00 us 1055615.00 us 33 XATTROP
0.10 121732.71 us 19.00 us 1134288.00 us 211 ENTRYLK
0.10 61611.54 us 470.00 us 1132038.00 us 439 FSYNC
0.11 308381.30 us 56.00 us 1117021.00 us 97 OPENDIR
0.26 352045.54 us 25.00 us 1117112.00 us 200 STATFS
0.54 59824.41 us 66.00 us 1050503.00 us 2449 WRITE
0.62 42784.77 us 135.00 us 982688.00 us 3920 FXATTROP
1.09 48299.64 us 12.00 us 1475231.00 us 6113 FINODELK
1.99 74036.50 us 337.00 us 1504736.00 us 7270 RCHECKSUM
5.02 91592.07 us 14.00 us 1727644.00 us 14800 INODELK
18.22 391339.18 us 13.00 us 1118801.00 us 12565 STAT
71.77 447238.86 us 65.00 us 1520575.00 us 43295 LOOKUP
Duration: 78214 seconds
Data Read: 54416941626 bytes
Data Written: 113658424203 bytes
Interval 1 Stats:
Block Size: 4096b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 10 1 37
Block Size: 65536b+ 131072b+
No. of Reads: 0 99
No. of Writes: 364 2494
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 115 RELEASEDIR
0.00 159976.00 us 1747.00 us 318205.00 us 2 MKNOD
0.01 43052.35 us 72.00 us 711323.00 us 17 READDIR
0.01 95923.62 us 306.00 us 760038.00 us 8 SETXATTR
0.01 66875.50 us 283.00 us 703396.00 us 16 SETATTR
0.01 99516.19 us 22.00 us 875479.00 us 16 GETXATTR
0.01 186339.50 us 268.00 us 670244.00 us 10 READDIRP
0.01 38121.54 us 55.00 us 690160.00 us 49 READ
0.02 276285.38 us 199.00 us 746238.00 us 8 REMOVEXATTR
0.03 234487.19 us 766.00 us 992806.00 us 16 XATTROP
0.04 160636.43 us 1337.00 us 847273.00 us 35 MKDIR
0.05 445612.14 us 33.00 us 885814.00 us 14 FSTAT
0.10 318281.76 us 56.00 us 1117021.00 us 42 OPENDIR
0.11 77332.15 us 512.00 us 1132038.00 us 184 FSYNC
0.13 140282.40 us 19.00 us 1134288.00 us 121 ENTRYLK
0.26 371711.56 us 25.00 us 1117112.00 us 94 STATFS
0.50 65120.02 us 77.00 us 1048875.00 us 1037 WRITE
0.52 44151.04 us 145.00 us 982688.00 us 1588 FXATTROP
0.96 53136.48 us 15.00 us 1131465.00 us 2444 FINODELK
2.04 75382.00 us 337.00 us 1135632.00 us 3653 RCHECKSUM
4.89 89311.00 us 14.00 us 1727644.00 us 7403 INODELK
19.46 432093.99 us 13.00 us 1118801.00 us 6093 STAT
70.86 461973.08 us 65.00 us 1520575.00 us 20751 LOOKUP
Duration: 246 seconds
Data Read: 12976128 bytes
Data Written: 374076928 bytes
Brick: HOSTNAME-02-B:/brick7/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 32b+ 64b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 3 1 2
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 5 6 1174
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 1
No. of Writes: 354 415 1133
Block Size: 65536b+ 131072b+
No. of Reads: 0 5403
No. of Writes: 7199 16939
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 59 FORGET
0.00 0.00 us 0.00 us 0.00 us 414 RELEASE
0.00 0.00 us 0.00 us 0.00 us 812 RELEASEDIR
0.00 51.00 us 51.00 us 51.00 us 1 LK
0.00 811.00 us 811.00 us 811.00 us 1 LINK
0.00 1962.67 us 116.00 us 5155.00 us 3 OPEN
0.00 19450.00 us 19450.00 us 19450.00 us 1 RENAME
0.00 425385.00 us 3485.00 us 847285.00 us 2 MKNOD
0.00 514930.00 us 94.00 us 1029766.00 us 2 FTRUNCATE
0.01 355550.20 us 457.00 us 713035.00 us 5 UNLINK
0.01 1192925.00 us 955005.00 us 1430845.00 us 2 XATTROP
0.01 493677.80 us 1865.00 us 1133142.00 us 5 CREATE
0.02 429648.22 us 26.00 us 1005252.00 us 9 FLUSH
0.02 375096.13 us 31.00 us 743624.00 us 15 FSTAT
0.03 553294.91 us 124.00 us 2047492.00 us 11 SETATTR
0.04 241321.34 us 210.00 us 1130863.00 us 35 READDIRP
0.11 308798.10 us 37.00 us 1090701.00 us 80 OPENDIR
0.14 437895.92 us 20.00 us 2390710.00 us 76 ENTRYLK
0.24 274161.42 us 20.00 us 1131003.00 us 206 STATFS
0.26 380771.70 us 111.00 us 2217146.00 us 156 FSYNC
0.62 608724.21 us 98.00 us 2805007.00 us 234 RCHECKSUM
0.68 436962.29 us 65.00 us 2218008.00 us 359 READ
0.88 292235.22 us 16.00 us 1860745.00 us 696 FINODELK
1.16 517693.60 us 20.00 us 3188822.00 us 516 INODELK
3.10 1007821.69 us 185.00 us 8062558.00 us 710 FXATTROP
5.47 459404.39 us 60.00 us 2424925.00 us 2747 WRITE
19.30 343135.28 us 11.00 us 1508447.00 us 12965 STAT
67.89 449890.07 us 56.00 us 3189202.00 us 34791 LOOKUP
Duration: 2760 seconds
Data Read: 708235477 bytes
Data Written: 3108339591 bytes
Interval 1 Stats:
Block Size: 32b+ 2048b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 1 2 87
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 33 46 155
Block Size: 65536b+ 131072b+
No. of Reads: 0 384
No. of Writes: 898 1821
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 7 FORGET
0.00 0.00 us 0.00 us 0.00 us 5 RELEASE
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 811.00 us 811.00 us 811.00 us 1 LINK
0.00 3485.00 us 3485.00 us 3485.00 us 1 MKNOD
0.00 1962.67 us 116.00 us 5155.00 us 3 OPEN
0.01 364139.50 us 35.00 us 728244.00 us 2 FLUSH
0.01 955005.00 us 955005.00 us 955005.00 us 1 XATTROP
0.01 514930.00 us 94.00 us 1029766.00 us 2 FTRUNCATE
0.01 568496.50 us 3851.00 us 1133142.00 us 2 CREATE
0.02 443739.00 us 457.00 us 713035.00 us 4 UNLINK
0.03 340935.25 us 31.00 us 700342.00 us 8 FSTAT
0.03 701433.25 us 124.00 us 2047492.00 us 4 SETATTR
0.03 319603.10 us 1904.00 us 1130863.00 us 10 READDIRP
0.10 277164.95 us 40.00 us 1090701.00 us 39 OPENDIR
0.18 558491.82 us 25.00 us 2390710.00 us 34 ENTRYLK
0.25 275750.60 us 20.00 us 1131003.00 us 96 STATFS
0.26 362599.55 us 111.00 us 2217146.00 us 77 FSYNC
0.78 761765.55 us 98.00 us 2805007.00 us 110 RCHECKSUM
0.81 519364.02 us 65.00 us 2218008.00 us 168 READ
0.86 292915.32 us 16.00 us 1860745.00 us 313 FINODELK
1.36 618592.78 us 23.00 us 3188822.00 us 235 INODELK
3.03 1037876.55 us 220.00 us 8062558.00 us 313 FXATTROP
6.12 518399.40 us 67.00 us 2424925.00 us 1265 WRITE
21.23 369324.98 us 11.00 us 1496678.00 us 6163 STAT
64.89 406497.92 us 60.00 us 3189202.00 us 17113 LOOKUP
Duration: 246 seconds
Data Read: 50331648 bytes
Data Written: 349692935 bytes
Brick: HOSTNAME-02-B:/brick4/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 1 0 1
No. of Writes: 2 1 7
Block Size: 16b+ 32b+ 64b+
No. of Reads: 4 392 51
No. of Writes: 12 135 57
Block Size: 128b+ 256b+ 512b+
No. of Reads: 443 127 10
No. of Writes: 140 304 1444
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 90 261 364
No. of Writes: 219068 121020 345512
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 514 856 1478
No. of Writes: 118098 171851 108390
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 2306 8288758 0
No. of Writes: 371871 702539 10
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 45329 FORGET
0.00 0.00 us 0.00 us 0.00 us 48418 RELEASE
0.00 0.00 us 0.00 us 0.00 us 311777 RELEASEDIR
0.00 38.00 us 38.00 us 38.00 us 1 LK
0.00 49.50 us 45.00 us 54.00 us 2 ENTRYLK
0.00 191.00 us 191.00 us 191.00 us 1 SETATTR
0.00 114.50 us 82.00 us 212.00 us 10 OPEN
0.00 53.83 us 28.00 us 258.00 us 29 FINODELK
0.00 32180.00 us 32180.00 us 32180.00 us 1 MKNOD
0.05 13870.29 us 44.00 us 565518.00 us 80 OPENDIR
0.11 84408.74 us 164.00 us 483823.00 us 27 READDIRP
0.15 15813.07 us 27.00 us 642324.00 us 200 STATFS
0.21 10819.69 us 30.00 us 2037193.00 us 404 FSTAT
0.34 49.82 us 10.00 us 159830.00 us 141050 INODELK
0.47 155572.08 us 67.00 us 469682.00 us 62 WRITE
1.93 376836.99 us 162.00 us 3309818.00 us 105 FXATTROP
9.35 2726.16 us 388.00 us 708391.00 us 70513 RCHECKSUM
17.39 6686.00 us 36.00 us 689525.00 us 53448 READ
18.05 29067.22 us 14.00 us 1708922.00 us 12766 STAT
51.94 33265.70 us 59.00 us 3308011.00 us 32089 LOOKUP
Duration: 87061 seconds
Data Read: 1086749393549 bytes
Data Written: 147537171706 bytes
Interval 1 Stats:
Block Size: 8192b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 1 3 11
Block Size: 131072b+
No. of Reads: 85818
No. of Writes: 47
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 77.00 us 33.00 us 258.00 us 7 FINODELK
0.00 109.12 us 46.00 us 397.00 us 34 OPENDIR
0.06 120138.50 us 837.00 us 483823.00 us 6 READDIRP
0.10 12370.53 us 31.00 us 642324.00 us 94 STATFS
0.29 54.74 us 10.00 us 159830.00 us 62641 INODELK
0.32 12202.91 us 30.00 us 2037193.00 us 313 FSTAT
0.42 166359.60 us 67.00 us 469682.00 us 30 WRITE
1.76 424134.63 us 162.00 us 3309818.00 us 49 FXATTROP
9.44 3566.54 us 396.00 us 708391.00 us 31320 RCHECKSUM
21.56 41576.42 us 14.00 us 1708922.00 us 6138 STAT
22.70 6488.02 us 36.00 us 689525.00 us 41414 READ
43.34 33595.04 us 81.00 us 3308011.00 us 15268 LOOKUP
Duration: 246 seconds
Data Read: 11248336896 bytes
Data Written: 7489024 bytes
Brick: HOSTNAME-02-B:/brick8/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 32b+ 512b+ 1024b+
No. of Reads: 0 1 0
No. of Writes: 2 415 835
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 4 2 4
No. of Writes: 1788 22766 9151
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 12 28 129
No. of Writes: 12333 23668 124056
Block Size: 131072b+ 262144b+
No. of Reads: 1565084 0
No. of Writes: 331920 2
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 99 FORGET
0.00 0.00 us 0.00 us 0.00 us 1590 RELEASE
0.00 0.00 us 0.00 us 0.00 us 32065 RELEASEDIR
0.00 158.00 us 158.00 us 158.00 us 1 SETATTR
0.00 42.75 us 34.00 us 57.00 us 4 ENTRYLK
0.00 95.00 us 91.00 us 99.00 us 2 OPEN
0.00 36.90 us 31.00 us 49.00 us 10 INODELK
0.00 70.67 us 23.00 us 139.00 us 6 GETXATTR
0.03 3341.00 us 3341.00 us 3341.00 us 1 UNLINK
0.03 4044.00 us 4044.00 us 4044.00 us 1 MKNOD
0.09 135.14 us 2.00 us 1219.00 us 83 OPENDIR
0.30 187.68 us 24.00 us 7544.00 us 200 STATFS
0.67 4177.55 us 263.00 us 28261.00 us 20 READDIRP
1.02 21242.50 us 153.00 us 122468.00 us 6 READDIR
14.15 138.01 us 16.00 us 21061.00 us 12765 STAT
83.70 336.20 us 35.00 us 30375.00 us 31007 LOOKUP
Duration: 87061 seconds
Data Read: 205153204736 bytes
Data Written: 59695163000 bytes
Interval 1 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 41.00 us 33.00 us 49.00 us 2 INODELK
0.00 158.00 us 158.00 us 158.00 us 1 SETATTR
0.00 42.75 us 34.00 us 57.00 us 4 ENTRYLK
0.05 3341.00 us 3341.00 us 3341.00 us 1 UNLINK
0.06 116.15 us 43.00 us 427.00 us 34 OPENDIR
0.06 4044.00 us 4044.00 us 4044.00 us 1 MKNOD
0.25 176.27 us 31.00 us 7544.00 us 94 STATFS
0.34 4370.40 us 3093.00 us 5741.00 us 5 READDIRP
14.33 152.05 us 17.00 us 21061.00 us 6138 STAT
84.90 366.58 us 73.00 us 30375.00 us 15084 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-02-B:/brick5/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 675 2015 5669
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 3 0 2
No. of Writes: 86933 18437 13563
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 13 11 2226145
No. of Writes: 28736 150681 348421
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 114 FORGET
0.00 0.00 us 0.00 us 0.00 us 1644 RELEASE
0.00 0.00 us 0.00 us 0.00 us 29197 RELEASEDIR
0.00 93.00 us 93.00 us 93.00 us 1 OPEN
0.00 121.00 us 111.00 us 131.00 us 2 SETATTR
0.00 62.43 us 22.00 us 134.00 us 7 GETXATTR
0.00 25.81 us 17.00 us 74.00 us 70 ENTRYLK
0.00 5628.50 us 2325.00 us 8932.00 us 2 MKNOD
0.01 124665.17 us 231.00 us 624079.00 us 6 READDIR
0.02 101333.57 us 165.00 us 520857.00 us 30 READDIRP
0.14 237618.40 us 1.00 us 763629.00 us 85 OPENDIR
0.24 170075.12 us 22.00 us 762654.00 us 200 STATFS
0.28 1494.00 us 387.00 us 658594.00 us 26497 RCHECKSUM
0.38 67376.52 us 67.00 us 592219.00 us 782 WRITE
0.39 36555.77 us 136.00 us 970840.00 us 1506 FXATTROP
0.41 76461.54 us 1824.00 us 784619.00 us 751 FSYNC
0.44 27318.09 us 14.00 us 779529.00 us 2256 FINODELK
1.39 3660.22 us 13.00 us 727062.00 us 53075 INODELK
18.68 205180.63 us 12.00 us 767356.00 us 12736 STAT
77.62 351003.58 us 30.00 us 971220.00 us 30942 LOOKUP
Duration: 78214 seconds
Data Read: 291787269120 bytes
Data Written: 65192541184 bytes
Interval 1 Stats:
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 2 1 5
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 10 19 31
Block Size: 65536b+ 131072b+
No. of Reads: 0 0
No. of Writes: 206 686
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.02 264573.67 us 103764.00 us 499127.00 us 6 READDIRP
0.09 197846.12 us 43.00 us 651148.00 us 34 OPENDIR
0.23 173971.46 us 27.00 us 663534.00 us 94 STATFS
0.25 1340.23 us 403.00 us 609386.00 us 13504 RCHECKSUM
0.33 30710.89 us 136.00 us 970840.00 us 774 FXATTROP
0.33 59568.81 us 79.00 us 592219.00 us 400 WRITE
0.40 74721.05 us 1894.00 us 645840.00 us 386 FSYNC
0.42 25870.04 us 17.00 us 641253.00 us 1160 FINODELK
1.35 3568.56 us 13.00 us 677567.00 us 27047 INODELK
20.17 235917.45 us 12.00 us 668351.00 us 6109 STAT
76.39 365037.23 us 61.00 us 971220.00 us 14951 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 116390912 bytes
Brick: HOSTNAME-02-B:/brick6/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 64b+ 512b+ 1024b+
No. of Reads: 2 1 0
No. of Writes: 0 249 554
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 0 0 7
No. of Writes: 1147 5601 4339
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 9 14 52
No. of Writes: 8060 16044 85807
Block Size: 131072b+ 262144b+
No. of Reads: 57503 0
No. of Writes: 233443 2
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 177 FORGET
0.00 0.00 us 0.00 us 0.00 us 267 RELEASE
0.00 0.00 us 0.00 us 0.00 us 32347 RELEASEDIR
0.00 77.80 us 18.00 us 140.00 us 5 GETXATTR
0.01 110.21 us 28.00 us 884.00 us 14 ENTRYLK
0.01 2082.00 us 2082.00 us 2082.00 us 1 RENAME
0.10 185.46 us 2.00 us 1256.00 us 84 OPENDIR
0.20 148.97 us 30.00 us 2192.00 us 200 STATFS
0.64 16299.00 us 163.00 us 95266.00 us 6 READDIR
0.83 5265.21 us 315.00 us 28553.00 us 24 READDIRP
15.27 181.48 us 14.00 us 30885.00 us 12765 STAT
82.92 409.24 us 29.00 us 31576.00 us 30731 LOOKUP
Duration: 87061 seconds
Data Read: 7542971594 bytes
Data Written: 41717849088 bytes
Interval 1 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.01 275.25 us 65.00 us 884.00 us 4 ENTRYLK
0.03 2082.00 us 2082.00 us 2082.00 us 1 RENAME
0.10 217.26 us 63.00 us 1044.00 us 34 OPENDIR
0.17 134.26 us 30.00 us 939.00 us 94 STATFS
0.33 4005.17 us 2273.00 us 6785.00 us 6 READDIRP
14.30 171.33 us 16.00 us 19199.00 us 6138 STAT
85.06 413.91 us 68.00 us 31576.00 us 15114 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-02-B:/brick2/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 8b+ 32b+ 64b+
No. of Reads: 0 0 0
No. of Writes: 1 54 46
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 0
No. of Writes: 180 277 721
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 3
No. of Writes: 20590 17995 62409
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 4 26 28
No. of Writes: 68162 107367 71851
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 59 542702 0
No. of Writes: 218072 310390 8
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 573 FORGET
0.00 0.00 us 0.00 us 0.00 us 2644 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28568 RELEASEDIR
0.00 33.00 us 33.00 us 33.00 us 1 FLUSH
0.00 67.50 us 56.00 us 79.00 us 2 LK
0.00 167.00 us 167.00 us 167.00 us 1 OPEN
0.01 144.96 us 42.00 us 1139.00 us 80 OPENDIR
0.01 111.08 us 24.00 us 2868.00 us 200 STATFS
0.03 157.12 us 73.00 us 9048.00 us 429 WRITE
0.86 135.98 us 15.00 us 13652.00 us 12765 STAT
1.18 81686.03 us 231.00 us 851745.00 us 29 READDIRP
3.98 45.61 us 14.00 us 11734.00 us 175875 INODELK
8.95 577.68 us 74.00 us 224862.00 us 31188 LOOKUP
84.98 1946.12 us 330.00 us 811977.00 us 87936 RCHECKSUM
Duration: 78214 seconds
Data Read: 71140696438 bytes
Data Written: 71129235470 bytes
Interval 1 Stats:
Block Size: 131072b+
No. of Reads: 0
No. of Writes: 551
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1 FORGET
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.01 150.06 us 42.00 us 1139.00 us 34 OPENDIR
0.01 90.35 us 31.00 us 735.00 us 94 STATFS
0.04 202.23 us 81.00 us 9048.00 us 144 WRITE
1.11 133.66 us 16.00 us 9674.00 us 6138 STAT
1.90 234569.83 us 1225.00 us 851745.00 us 6 READDIRP
5.01 46.95 us 14.00 us 11257.00 us 78964 INODELK
11.66 568.83 us 74.00 us 114583.00 us 15164 LOOKUP
80.26 1503.60 us 330.00 us 558433.00 us 39481 RCHECKSUM
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 72220672 bytes
Brick: HOSTNAME-02-B:/brick9/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 64b+ 512b+ 1024b+
No. of Reads: 1 2 0
No. of Writes: 0 1488 3133
Block Size: 2048b+ 4096b+ 8192b+
No. of Reads: 2 12 6
No. of Writes: 6241 20337 17112
Block Size: 16384b+ 32768b+ 65536b+
No. of Reads: 41 40 109
No. of Writes: 36536 97167 400752
Block Size: 131072b+ 262144b+
No. of Reads: 1392662 0
No. of Writes: 734025 9
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 245 FORGET
0.00 0.00 us 0.00 us 0.00 us 1040 RELEASE
0.00 0.00 us 0.00 us 0.00 us 32052 RELEASEDIR
0.00 40.50 us 32.00 us 49.00 us 2 FLUSH
0.00 64.50 us 54.00 us 75.00 us 2 ENTRYLK
0.00 45.25 us 14.00 us 92.00 us 4 GETXATTR
0.00 66.33 us 43.00 us 94.00 us 3 LK
0.00 230.00 us 230.00 us 230.00 us 1 OPEN
0.06 14688.00 us 14688.00 us 14688.00 us 1 UNLINK
0.13 358.22 us 2.00 us 6730.00 us 83 OPENDIR
0.16 178.78 us 26.00 us 4082.00 us 200 STATFS
0.57 21934.83 us 164.00 us 122756.00 us 6 READDIR
2.74 24249.73 us 122.00 us 185638.00 us 26 READDIRP
16.00 287.83 us 14.00 us 53088.00 us 12765 STAT
80.33 596.96 us 60.00 us 55814.00 us 30911 LOOKUP
Duration: 87061 seconds
Data Read: 182552716865 bytes
Data Written: 146735579648 bytes
Interval 1 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 64.50 us 54.00 us 75.00 us 2 ENTRYLK
0.14 14688.00 us 14688.00 us 14688.00 us 1 UNLINK
0.16 174.14 us 26.00 us 3573.00 us 94 STATFS
0.17 541.12 us 45.00 us 6730.00 us 34 OPENDIR
2.16 45567.60 us 951.00 us 139415.00 us 5 READDIRP
14.97 257.11 us 14.00 us 30619.00 us 6138 STAT
82.40 576.00 us 70.00 us 49717.00 us 15086 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-02-B:/brick3/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 4b+ 16b+ 32b+
No. of Reads: 0 0 0
No. of Writes: 1 1 13
Block Size: 64b+ 128b+ 256b+
No. of Reads: 0 0 0
No. of Writes: 10 125 37
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 0 0 1
No. of Writes: 324 326351 4542
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 4 2 9
No. of Writes: 56358 18819 27128
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 17 42 3816011
No. of Writes: 35096 76153 521853
Block Size: 262144b+
No. of Reads: 0
No. of Writes: 12
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 175 FORGET
0.00 0.00 us 0.00 us 0.00 us 2519 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28490 RELEASEDIR
0.00 27.00 us 27.00 us 27.00 us 1 FLUSH
0.00 512.00 us 512.00 us 512.00 us 1 UNLINK
0.00 414686.00 us 414686.00 us 414686.00 us 1 LK
0.00 253232.00 us 39.00 us 506425.00 us 2 ENTRYLK
0.05 497784.80 us 124.00 us 1570288.00 us 30 READDIRP
0.11 7015.25 us 336.00 us 1565069.00 us 4505 RCHECKSUM
0.13 494818.47 us 52.00 us 1557805.00 us 80 OPENDIR
0.15 21618.97 us 19.00 us 1410202.00 us 2051 FSTAT
0.23 68568.80 us 1781.00 us 1573119.00 us 984 FSYNC
0.27 415087.87 us 24.00 us 1423645.00 us 196 STATFS
0.33 10851.24 us 16.00 us 1570142.00 us 9075 INODELK
0.60 85769.69 us 152.00 us 2328777.00 us 2065 FXATTROP
0.63 104161.26 us 53.00 us 1573331.00 us 1808 WRITE
0.71 24037.55 us 37.00 us 1570577.00 us 8788 READ
1.34 123004.33 us 13.00 us 1572980.00 us 3244 FINODELK
20.01 469342.63 us 11.00 us 1570713.00 us 12689 STAT
75.44 675578.60 us 98.00 us 1963137.00 us 33238 LOOKUP
Duration: 78214 seconds
Data Read: 500177719808 bytes
Data Written: 79883172260 bytes
Interval 1 Stats:
Block Size: 128b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 1 566 14
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 623 191 123
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 0 0 10368
No. of Writes: 81 126 445
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 512.00 us 512.00 us 512.00 us 1 UNLINK
0.00 253232.00 us 39.00 us 506425.00 us 2 ENTRYLK
0.02 567538.67 us 61455.00 us 1096240.00 us 6 READDIRP
0.10 425877.85 us 52.00 us 1412618.00 us 34 OPENDIR
0.10 6947.28 us 340.00 us 1416193.00 us 2133 RCHECKSUM
0.17 24302.80 us 20.00 us 1393421.00 us 1016 FSTAT
0.20 79981.08 us 1784.00 us 1444418.00 us 364 FSYNC
0.27 432331.46 us 24.00 us 1393263.00 us 90 STATFS
0.34 11557.54 us 16.00 us 1441854.00 us 4303 INODELK
0.51 94985.64 us 162.00 us 2328777.00 us 775 FXATTROP
0.63 128079.62 us 58.00 us 1442703.00 us 712 WRITE
0.73 23597.75 us 37.00 us 1416135.00 us 4456 READ
0.86 109550.04 us 13.00 us 1443792.00 us 1140 FINODELK
19.86 470022.73 us 14.00 us 1412768.00 us 6122 STAT
76.20 702701.46 us 98.00 us 1963137.00 us 15711 LOOKUP
Duration: 246 seconds
Data Read: 1358954496 bytes
Data Written: 85018520 bytes
Brick: HOSTNAME-01-B:/brick8/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 148 9
No. of Writes: 2198 856 1414
Block Size: 8b+ 16b+ 32b+
No. of Reads: 17 52 2454
No. of Writes: 2883 23076 27561
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2418 1055 843
No. of Writes: 148757 245049 274675
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 8942 12954 30505
No. of Writes: 1985063 409642431 64134867
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 112381 168949 285944
No. of Writes: 92567046 63502163 116489833
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 743469 1210030 816372704
No. of Writes: 17318963 70068548 142121633
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 1088 11 111
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1179035 FORGET
0.00 0.00 us 0.00 us 0.00 us 2293524 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26725606 RELEASEDIR
0.00 292.00 us 292.00 us 292.00 us 1 LINK
0.00 36.47 us 24.00 us 45.00 us 15 FLUSH
0.00 428.50 us 358.00 us 499.00 us 2 RENAME
0.00 166.80 us 121.00 us 220.00 us 10 SETATTR
0.00 70.97 us 35.00 us 326.00 us 30 LK
0.00 58.15 us 37.00 us 105.00 us 55 FINODELK
0.00 1624.25 us 770.00 us 3488.00 us 8 MKNOD
0.00 61.19 us 27.00 us 334.00 us 214 ENTRYLK
0.01 242.92 us 154.00 us 560.00 us 78 SETXATTR
0.03 85.31 us 28.00 us 2293.00 us 894 STATFS
0.05 26925.20 us 294.00 us 90304.00 us 5 UNLINK
0.06 3003.38 us 297.00 us 57010.00 us 55 FXATTROP
0.07 2474.06 us 475.00 us 36988.00 us 78 MKDIR
0.10 68.06 us 27.00 us 2641.00 us 4293 FSTAT
0.11 82114.75 us 351.00 us 310420.00 us 4 READDIR
0.21 97.59 us 3.00 us 3330.00 us 6260 OPENDIR
0.39 134.22 us 47.00 us 26252.00 us 8479 OPEN
0.44 128.85 us 12.00 us 25805.00 us 9866 STAT
0.80 151.41 us 75.00 us 69114.00 us 15222 WRITE
0.83 69.77 us 18.00 us 599472.00 us 34079 INODELK
3.07 476.92 us 48.00 us 59897.00 us 18573 GETXATTR
8.90 505.04 us 47.00 us 120915.00 us 50768 READ
10.30 284.59 us 20.00 us 120764.00 us 104241 LOOKUP
74.63 19961.45 us 52.00 us 5262484.00 us 10772 READDIRP
Duration: 4279793 seconds
Data Read: 107167648980220 bytes
Data Written: 31737926706873 bytes
Interval 5 Stats:
Block Size: 131072b+
No. of Reads: 2903
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 85.00 us 62.00 us 108.00 us 2 ENTRYLK
0.00 185.00 us 185.00 us 185.00 us 1 SETATTR
0.03 1168.50 us 77.00 us 2260.00 us 2 INODELK
0.07 93.63 us 40.00 us 165.00 us 62 FSTAT
0.07 177.49 us 54.00 us 955.00 us 35 OPENDIR
0.14 135.89 us 28.00 us 2293.00 us 91 STATFS
0.49 43269.00 us 43269.00 us 43269.00 us 1 UNLINK
2.25 1365.04 us 164.00 us 22604.00 us 145 READDIRP
5.95 153.65 us 18.00 us 9302.00 us 3397 STAT
36.13 2655.90 us 61.00 us 101625.00 us 1194 READ
54.87 321.63 us 74.00 us 33476.00 us 14974 LOOKUP
Duration: 246 seconds
Data Read: 380502016 bytes
Data Written: 0 bytes
Brick: HOSTNAME-01-B:/brick6/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 1 0
No. of Writes: 2161 831 1435
Block Size: 8b+ 16b+ 32b+
No. of Reads: 0 0 2903
No. of Writes: 4166 5588026 44908
Block Size: 64b+ 128b+ 256b+
No. of Reads: 3536 5 80
No. of Writes: 87870 241581 376594
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 8020 7205 19984
No. of Writes: 7805307 539972646 80512909
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 62273 100742 210754
No. of Writes: 102909970 76529387 166357551
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 551103 735158 761717336
No. of Writes: 15712591 54848367 117383734
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 5 360612
No. of Writes: 3717 5306 1855286
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 835383 FORGET
0.00 0.00 us 0.00 us 0.00 us 817859 RELEASE
0.00 0.00 us 0.00 us 0.00 us 28832755 RELEASEDIR
0.00 42.75 us 30.00 us 67.00 us 4 FLUSH
0.00 65.38 us 43.00 us 120.00 us 8 LK
0.00 172.33 us 123.00 us 238.00 us 9 SETATTR
0.00 548.14 us 264.00 us 1315.00 us 7 UNLINK
0.00 1554.89 us 596.00 us 4101.00 us 9 MKNOD
0.00 59.64 us 24.00 us 520.00 us 238 ENTRYLK
0.00 245.50 us 150.00 us 493.00 us 78 SETXATTR
0.00 20433.00 us 20433.00 us 20433.00 us 1 RENAME
0.00 2955.50 us 81.00 us 22895.00 us 8 FSTAT
0.01 7511.17 us 216.00 us 42274.00 us 6 READDIR
0.01 85.94 us 26.00 us 2189.00 us 894 STATFS
0.01 4187.81 us 2124.00 us 20241.00 us 21 FSYNC
0.04 1532.93 us 223.00 us 66019.00 us 202 FXATTROP
0.05 4863.85 us 568.00 us 145709.00 us 78 MKDIR
0.11 131.42 us 61.00 us 3438.00 us 6721 OPEN
0.11 91.46 us 2.00 us 29957.00 us 10315 OPENDIR
0.17 142.09 us 15.00 us 12445.00 us 9865 STAT
0.35 107.33 us 18.00 us 774166.00 us 27066 INODELK
1.21 480.50 us 23.00 us 121342.00 us 21084 GETXATTR
2.71 145604.99 us 38.00 us 1277202.00 us 155 FINODELK
3.14 247.45 us 14.00 us 64731.00 us 105762 LOOKUP
3.36 282.18 us 68.00 us 131903.00 us 99206 READ
15.81 7013.42 us 49.00 us 4783006.00 us 18800 READDIRP
72.91 61747.58 us 84.00 us 612823.00 us 9846 WRITE
Duration: 4279793 seconds
Data Read: 100321376892903 bytes
Data Written: 29652026734449 bytes
Interval 5 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.01 157.00 us 50.00 us 436.00 us 4 ENTRYLK
0.12 177.06 us 58.00 us 1407.00 us 35 OPENDIR
0.24 138.63 us 39.00 us 1793.00 us 91 STATFS
0.38 20433.00 us 20433.00 us 20433.00 us 1 RENAME
2.46 1564.36 us 149.00 us 18439.00 us 84 READDIRP
8.49 133.50 us 19.00 us 7417.00 us 3397 STAT
88.30 313.58 us 81.00 us 20208.00 us 15040 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-01-B:/brick4/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 2b+ 4b+ 8b+
No. of Reads: 0 0 0
No. of Writes: 3 1 7
Block Size: 16b+ 32b+ 64b+
No. of Reads: 0 3 0
No. of Writes: 14 176 55
Block Size: 128b+ 256b+ 512b+
No. of Reads: 0 0 6
No. of Writes: 551 409 1327
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 25 20
No. of Writes: 163855 98807 281768
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 52 81 142
No. of Writes: 105053 164786 104540
Block Size: 65536b+ 131072b+ 262144b+
No. of Reads: 311 2267877 0
No. of Writes: 353531 7948711 10
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 61063 FORGET
0.00 0.00 us 0.00 us 0.00 us 14884 RELEASE
0.00 0.00 us 0.00 us 0.00 us 95159 RELEASEDIR
0.00 70.00 us 70.00 us 70.00 us 1 LK
0.00 59.50 us 56.00 us 63.00 us 2 ENTRYLK
0.00 174.00 us 174.00 us 174.00 us 1 SETATTR
0.00 142.19 us 70.00 us 229.00 us 26 OPEN
0.00 264.95 us 47.00 us 3405.00 us 81 OPENDIR
0.01 5896.86 us 27.00 us 127468.00 us 22 FINODELK
0.02 199117.00 us 199117.00 us 199117.00 us 1 MKNOD
0.12 5566.34 us 20.00 us 545166.00 us 197 STATFS
0.25 312.26 us 26.00 us 700937.00 us 7444 FSTAT
0.86 63127.77 us 162.00 us 952344.00 us 125 READDIRP
0.91 77367.29 us 212.00 us 1039173.00 us 107 FXATTROP
1.42 209840.71 us 121.00 us 620314.00 us 62 WRITE
1.48 95.93 us 15.00 us 722187.00 us 141085 INODELK
4.15 2543.50 us 57.00 us 721863.00 us 14886 READ
13.69 15652.88 us 16.00 us 721267.00 us 7986 STAT
17.78 2302.89 us 363.00 us 834931.00 us 70497 RCHECKSUM
59.29 16876.75 us 68.00 us 1036627.00 us 32076 LOOKUP
Duration: 77227 seconds
Data Read: 297294236329 bytes
Data Written: 1094314598941 bytes
Interval 1 Stats:
Block Size: 8192b+ 32768b+ 65536b+
No. of Reads: 0 0 0
No. of Writes: 1 3 11
Block Size: 131072b+
No. of Reads: 17261
No. of Writes: 47
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 193.43 us 54.00 us 1791.00 us 35 OPENDIR
0.02 554.58 us 20.00 us 31421.00 us 91 STATFS
0.06 12897.40 us 32.00 us 127468.00 us 10 FINODELK
1.02 312.28 us 26.00 us 700937.00 us 7443 FSTAT
1.47 44044.13 us 162.00 us 917890.00 us 76 READDIRP
2.55 92.91 us 16.00 us 722187.00 us 62715 INODELK
2.99 227428.40 us 121.00 us 534197.00 us 30 WRITE
3.12 139535.71 us 259.00 us 1039173.00 us 51 FXATTROP
8.24 5538.18 us 18.00 us 721267.00 us 3397 STAT
16.69 2558.28 us 57.00 us 721863.00 us 14886 READ
22.42 1632.14 us 363.00 us 718808.00 us 31356 RCHECKSUM
41.42 6177.50 us 76.00 us 1036627.00 us 15304 LOOKUP
Duration: 246 seconds
Data Read: 2262433792 bytes
Data Written: 7489024 bytes
Brick: HOSTNAME-01-B:/brick5/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 0 2
No. of Writes: 482 547 864
Block Size: 8b+ 16b+ 32b+
No. of Reads: 12 39 3645
No. of Writes: 1717 18223 16043
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2441 542 390
No. of Writes: 607834 219970 204446
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 6262 5164 13211
No. of Writes: 1064280 203635430 61786149
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 70712 123308 170598
No. of Writes: 80206120 104132098 110568574
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 493854 698297 533801800
No. of Writes: 18319266 76209603 132443069
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 2
No. of Writes: 837 1 11
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 583048 FORGET
0.00 0.00 us 0.00 us 0.00 us 1791579 RELEASE
0.00 0.00 us 0.00 us 0.00 us 25457262 RELEASEDIR
0.00 322.17 us 205.00 us 808.00 us 6 SETATTR
0.00 2292.58 us 31.00 us 16199.00 us 12 FLUSH
0.00 28625.50 us 10170.00 us 47081.00 us 2 UNLINK
0.00 2983.42 us 51.00 us 37006.00 us 24 LK
0.00 15029.17 us 1096.00 us 44572.00 us 6 MKNOD
0.00 4302.60 us 150.00 us 66649.00 us 78 SETXATTR
0.01 13873.86 us 808.00 us 107723.00 us 78 MKDIR
0.01 755.09 us 121.00 us 211768.00 us 1698 FXATTROP
0.01 4013.43 us 23.00 us 71921.00 us 334 ENTRYLK
0.02 220483.93 us 257.00 us 516216.00 us 15 READDIR
0.03 4518.96 us 31.00 us 76937.00 us 894 STATFS
0.04 7217.03 us 1872.00 us 1382417.00 us 811 FSYNC
0.09 5690.68 us 2.00 us 2641373.00 us 2147 OPENDIR
0.10 1368.02 us 15.00 us 134024.00 us 9865 STAT
0.16 1421.97 us 48.00 us 3165390.00 us 15489 OPEN
0.17 2595.75 us 26.00 us 2281303.00 us 8854 GETXATTR
0.21 2879.52 us 22.00 us 132848.00 us 9957 FSTAT
0.21 11982.66 us 54.00 us 2340814.00 us 2438 READDIRP
1.42 2157.46 us 39.00 us 3980667.00 us 90417 LOOKUP
2.01 1851.28 us 13.00 us 4353557.00 us 149524 INODELK
2.09 6592.01 us 333.00 us 2340514.00 us 43669 RCHECKSUM
3.92 8208.32 us 37.00 us 284665.00 us 65801 READ
13.14 726280.54 us 26.00 us 4778309.00 us 2490 FINODELK
76.36 304426.35 us 47.00 us 3078703.00 us 34522 WRITE
Duration: 4279793 seconds
Data Read: 70063314642216 bytes
Data Written: 30894365505026 bytes
Interval 5 Stats:
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 0 0 0
No. of Writes: 2 1 5
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 10 19 31
Block Size: 65536b+ 131072b+
No. of Reads: 0 19354
No. of Writes: 206 675
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 142.03 us 51.00 us 558.00 us 35 OPENDIR
0.00 207.80 us 37.00 us 4278.00 us 91 STATFS
0.00 79.99 us 31.00 us 141.00 us 290 FSTAT
0.01 1539.89 us 170.00 us 8853.00 us 54 READDIRP
0.02 260.63 us 121.00 us 1374.00 us 776 FXATTROP
0.11 3223.89 us 2100.00 us 5998.00 us 388 FSYNC
0.14 444.37 us 15.00 us 15289.00 us 3397 STAT
0.83 593.88 us 76.00 us 47077.00 us 15531 LOOKUP
2.89 83325.97 us 86.00 us 502479.00 us 387 WRITE
5.95 8661.33 us 47.00 us 267392.00 us 7668 READ
8.17 3374.06 us 18.00 us 3834382.00 us 27019 INODELK
10.53 8711.57 us 411.00 us 496077.00 us 13488 RCHECKSUM
71.35 688576.36 us 37.00 us 4310436.00 us 1156 FINODELK
Duration: 246 seconds
Data Read: 2536767488 bytes
Data Written: 114949120 bytes
Brick: HOSTNAME-01-B:/brick1/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 32b+ 128b+ 512b+
No. of Reads: 10 16 0
No. of Writes: 2 0 3
Block Size: 1024b+ 2048b+ 4096b+
No. of Reads: 2 2 4
No. of Writes: 4 1 230
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 2 8 10
No. of Writes: 62 111 342
Block Size: 65536b+ 131072b+
No. of Reads: 12 39099
No. of Writes: 1999 14756
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 110 FORGET
0.00 0.00 us 0.00 us 0.00 us 1552 RELEASE
0.00 0.00 us 0.00 us 0.00 us 5486 RELEASEDIR
0.00 33.00 us 33.00 us 33.00 us 1 FLUSH
0.00 75.00 us 75.00 us 75.00 us 1 LK
0.00 177.50 us 93.00 us 313.00 us 10 FSTAT
0.00 23790.00 us 23790.00 us 23790.00 us 1 UNLINK
0.00 308.52 us 28.00 us 9424.00 us 197 STATFS
0.01 5786.87 us 124.00 us 196553.00 us 102 READDIRP
0.01 166.50 us 54.00 us 37190.00 us 3921 FXATTROP
0.02 1713.96 us 155.00 us 57875.00 us 440 FSYNC
0.03 13994.76 us 26.00 us 115223.00 us 86 GETXATTR
0.03 9081.56 us 33.00 us 119977.00 us 134 OPENDIR
0.04 58848.03 us 100.00 us 709195.00 us 33 READDIR
0.05 262.36 us 15.00 us 16663.00 us 8244 STAT
0.08 40910.06 us 598.00 us 174924.00 us 88 XATTROP
0.16 20063.18 us 25.00 us 165004.00 us 355 ENTRYLK
0.21 17832.65 us 77.00 us 171068.00 us 545 READ
0.31 7303.16 us 65.00 us 559590.00 us 1961 WRITE
2.79 2933.18 us 67.00 us 208762.00 us 43381 LOOKUP
5.41 33917.36 us 430.00 us 842225.00 us 7262 RCHECKSUM
13.39 41023.35 us 20.00 us 13484708.00 us 14865 INODELK
77.46 603032.78 us 17.00 us 14543086.00 us 5850 FINODELK
Duration: 2952 seconds
Data Read: 5126633408 bytes
Data Written: 2195986825 bytes
Interval 1 Stats:
Block Size: 4096b+ 16384b+ 32768b+
No. of Reads: 0 0 0
No. of Writes: 10 1 37
Block Size: 65536b+ 131072b+
No. of Reads: 0 678
No. of Writes: 364 1888
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 17 FORGET
0.00 0.00 us 0.00 us 0.00 us 159 RELEASEDIR
0.00 175.83 us 93.00 us 313.00 us 6 FSTAT
0.00 372.52 us 34.00 us 9424.00 us 91 STATFS
0.01 155.47 us 54.00 us 15012.00 us 1586 FXATTROP
0.02 1662.29 us 603.00 us 19925.00 us 185 FSYNC
0.02 5773.04 us 124.00 us 196553.00 us 56 READDIRP
0.03 11697.47 us 56.00 us 119977.00 us 55 OPENDIR
0.04 19956.53 us 41.00 us 115223.00 us 36 GETXATTR
0.04 48883.81 us 100.00 us 119392.00 us 16 READDIR
0.06 344.76 us 15.00 us 16663.00 us 3514 STAT
0.10 39258.90 us 800.00 us 140015.00 us 52 XATTROP
0.17 19739.70 us 28.00 us 165004.00 us 169 ENTRYLK
0.29 19225.55 us 85.00 us 154901.00 us 299 READ
0.38 9418.15 us 74.00 us 180442.00 us 795 WRITE
3.27 3127.60 us 74.00 us 208762.00 us 20788 LOOKUP
6.44 35274.85 us 431.00 us 310417.00 us 3637 RCHECKSUM
13.94 37449.35 us 25.00 us 12546026.00 us 7412 INODELK
75.20 635507.62 us 19.00 us 14543086.00 us 2356 FINODELK
Duration: 246 seconds
Data Read: 88866816 bytes
Data Written: 294647296 bytes
Brick: HOSTNAME-01-B:/brick9/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 55 9
No. of Writes: 1491 1000 2102
Block Size: 8b+ 16b+ 32b+
No. of Reads: 64 50 4812
No. of Writes: 4696 7222905 44892
Block Size: 64b+ 128b+ 256b+
No. of Reads: 4319 4022 3248
No. of Writes: 293891 418033 542883
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 12528 18194 54796
No. of Writes: 8494202 664144673 121431414
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 216483 292636 465186
No. of Writes: 206999000 164921303 247436952
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1428505 2116990 1026270560
No. of Writes: 33487910 131100929 250121202
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 3112 1649 486445
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 1466412 FORGET
0.00 0.00 us 0.00 us 0.00 us 1649265 RELEASE
0.00 0.00 us 0.00 us 0.00 us 26523053 RELEASEDIR
0.00 41.00 us 18.00 us 134.00 us 13 FLUSH
0.00 61.28 us 37.00 us 155.00 us 25 LK
0.00 6961.00 us 6961.00 us 6961.00 us 1 RENAME
0.00 1194.50 us 237.00 us 4764.00 us 6 READDIR
0.00 28213.00 us 28213.00 us 28213.00 us 1 UNLINK
0.00 6334.43 us 126.00 us 25589.00 us 7 SETATTR
0.00 12989.29 us 800.00 us 37294.00 us 7 MKNOD
0.00 1259.36 us 128.00 us 32443.00 us 78 SETXATTR
0.01 1803.81 us 25.00 us 44751.00 us 204 ENTRYLK
0.01 7197.82 us 506.00 us 90051.00 us 78 MKDIR
0.04 3359.54 us 21.00 us 216237.00 us 894 STATFS
0.05 25889.78 us 600.00 us 236221.00 us 130 FSYNC
0.06 1592.46 us 2.00 us 381435.00 us 2539 OPENDIR
0.06 463.18 us 60.00 us 539032.00 us 8960 OPEN
0.12 841.19 us 14.00 us 217750.00 us 9865 STAT
0.25 1861.16 us 20.00 us 247927.00 us 9467 FSTAT
0.28 544.06 us 17.00 us 1376223.00 us 36022 INODELK
0.30 13330.61 us 159.00 us 399369.00 us 1583 FXATTROP
0.33 2010.68 us 29.00 us 537519.00 us 11519 GETXATTR
0.66 35582.06 us 18.00 us 983226.00 us 1290 FINODELK
1.93 1499.23 us 47.00 us 1543777.00 us 89791 LOOKUP
4.19 87790.99 us 52.00 us 11504765.00 us 3329 READDIRP
13.62 2616.76 us 35.00 us 534914.00 us 363070 READ
78.09 65133.76 us 49.00 us 1024146.00 us 83658 WRITE
Duration: 4279793 seconds
Data Read: 134801101778949 bytes
Data Written: 58578267669735 bytes
Interval 5 Stats:
Block Size: 131072b+
No. of Reads: 1129
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 81.00 us 77.00 us 85.00 us 2 ENTRYLK
0.10 197.94 us 61.00 us 1277.00 us 35 OPENDIR
0.17 127.99 us 37.00 us 1322.00 us 91 STATFS
0.40 827.09 us 58.00 us 24347.00 us 33 FSTAT
0.41 28213.00 us 28213.00 us 28213.00 us 1 UNLINK
7.75 156.57 us 17.00 us 16695.00 us 3397 STAT
8.23 3870.01 us 138.00 us 284848.00 us 146 READDIRP
10.90 1623.22 us 50.00 us 23865.00 us 461 READ
72.04 327.04 us 87.00 us 60571.00 us 15125 LOOKUP
Duration: 246 seconds
Data Read: 147980288 bytes
Data Written: 0 bytes
Brick: HOSTNAME-01-B:/brick3/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 12 6
No. of Writes: 2567 781 1340
Block Size: 8b+ 16b+ 32b+
No. of Reads: 31 66 4036
No. of Writes: 2948 440590 38913
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2734 1728 1167
No. of Writes: 75799 347793 470159
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 9774 16256 29003
No. of Writes: 2265947 446846351 86298960
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 118753 170352 279731
No. of Writes: 147940171 119810004 134459511
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1011892 1204426 639618284
No. of Writes: 29300584 134801413 251115548
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 3 368642
No. of Writes: 1808 14 312
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 556431 FORGET
0.00 0.00 us 0.00 us 0.00 us 646004 RELEASE
0.00 0.00 us 0.00 us 0.00 us 24454045 RELEASEDIR
0.00 308.00 us 308.00 us 308.00 us 1 RENAME
0.00 39.80 us 27.00 us 67.00 us 15 FLUSH
0.00 452.50 us 304.00 us 601.00 us 2 READDIR
0.00 62.24 us 43.00 us 88.00 us 29 LK
0.00 167.55 us 133.00 us 195.00 us 11 SETATTR
0.00 230.03 us 138.00 us 610.00 us 78 SETXATTR
0.00 252.74 us 27.00 us 40634.00 us 205 ENTRYLK
0.00 4938.91 us 750.00 us 39272.00 us 11 MKNOD
0.00 24506.75 us 379.00 us 59086.00 us 4 UNLINK
0.00 64.45 us 26.00 us 1244.00 us 1721 FSTAT
0.00 93.36 us 3.00 us 6609.00 us 5953 OPENDIR
0.00 725.79 us 29.00 us 556857.00 us 894 STATFS
0.00 151.83 us 73.00 us 40777.00 us 4636 OPEN
0.00 9566.51 us 531.00 us 85953.00 us 78 MKDIR
0.05 192.95 us 40.00 us 96090.00 us 46282 READ
0.05 2198.09 us 349.00 us 123915.00 us 4724 RCHECKSUM
0.12 2289.39 us 15.00 us 574911.00 us 9870 STAT
0.16 2874.39 us 29.00 us 725497.00 us 10731 GETXATTR
0.22 5318.90 us 112.00 us 613165.00 us 7887 FXATTROP
0.62 630.24 us 49.00 us 436551.00 us 188339 WRITE
1.04 19727.28 us 54.00 us 2411484.00 us 10085 READDIRP
1.71 3781.30 us 21.00 us 613196.00 us 86277 LOOKUP
1.86 120602.70 us 146.00 us 22966912.00 us 2935 FSYNC
4.43 29711.50 us 19.00 us 36688198.00 us 28412 INODELK
89.72 1630926.05 us 19.00 us 39459053.00 us 10488 FINODELK
Duration: 4279793 seconds
Data Read: 84395109496663 bytes
Data Written: 55381384251629 bytes
Interval 5 Stats:
Block Size: 128b+ 1024b+ 2048b+
No. of Reads: 0 0 0
No. of Writes: 1 566 14
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 0 0 0
No. of Writes: 623 191 123
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 6 2 19
No. of Writes: 81 126 445
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 71.00 us 44.00 us 98.00 us 3 ENTRYLK
0.00 117.00 us 102.00 us 132.00 us 2 FSTAT
0.00 160.97 us 54.00 us 989.00 us 35 OPENDIR
0.00 182.71 us 37.00 us 2711.00 us 91 STATFS
0.00 37863.00 us 37863.00 us 37863.00 us 1 UNLINK
0.01 10687.38 us 93.00 us 96090.00 us 24 READ
0.04 906.65 us 112.00 us 140471.00 us 793 FXATTROP
0.04 10671.96 us 130.00 us 156901.00 us 74 READDIRP
0.05 248.06 us 17.00 us 26637.00 us 3400 STAT
0.06 2929.57 us 1830.00 us 42843.00 us 365 FSYNC
0.20 1782.90 us 377.00 us 114465.00 us 2135 RCHECKSUM
0.62 699.57 us 87.00 us 44929.00 us 16450 LOOKUP
0.91 23643.10 us 80.00 us 227392.00 us 716 WRITE
7.92 34236.55 us 19.00 us 21085778.00 us 4296 INODELK
90.15 1542280.19 us 21.00 us 23438747.00 us 1086 FINODELK
Duration: 246 seconds
Data Read: 2984448 bytes
Data Written: 85018520 bytes
Brick: HOSTNAME-01-B:/brick2/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 6 1
No. of Writes: 5690 1158 1507
Block Size: 8b+ 16b+ 32b+
No. of Reads: 3 8 4220
No. of Writes: 4257 653093 38884
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2057 143 237
No. of Writes: 138246 284967 301583
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 9964 14318 24878
No. of Writes: 2799071 412706426 75565460
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 111870 177302 261376
No. of Writes: 190929249 79365618 255744283
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1009823 1146396 686781597
No. of Writes: 27530847 83970418 194993443
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 1 0
No. of Writes: 1208 19 69
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 723412 FORGET
0.00 0.00 us 0.00 us 0.00 us 731255 RELEASE
0.00 0.00 us 0.00 us 0.00 us 24362990 RELEASEDIR
0.00 374.71 us 20.00 us 4282.00 us 21 FLUSH
0.00 262.48 us 35.00 us 2950.00 us 40 LK
0.00 2522.33 us 138.00 us 13040.00 us 15 SETATTR
0.00 8113.44 us 298.00 us 28903.00 us 9 RENAME
0.00 400.89 us 71.00 us 85670.00 us 1193 OPEN
0.00 6982.42 us 138.00 us 31124.00 us 78 SETXATTR
0.00 40219.80 us 955.00 us 114403.00 us 15 MKNOD
0.00 807.49 us 25.00 us 37564.00 us 894 STATFS
0.01 586.83 us 27.00 us 52770.00 us 1780 FSTAT
0.01 94259.50 us 1264.00 us 259989.00 us 12 UNLINK
0.01 238.64 us 15.00 us 418394.00 us 5290 ENTRYLK
0.03 20173.97 us 302.00 us 1014766.00 us 195 XATTROP
0.05 90005.90 us 1022.00 us 480876.00 us 78 MKDIR
0.06 908.45 us 15.00 us 46094.00 us 10388 STAT
0.06 2539.71 us 38.00 us 981906.00 us 3719 OPENDIR
0.29 11452.34 us 20.00 us 5427128.00 us 3916 GETXATTR
0.95 1026081.59 us 632.00 us 18320824.00 us 141 FSYNC
1.19 554.78 us 14.00 us 9904525.00 us 327694 INODELK
1.67 3739.81 us 67.00 us 9923314.00 us 68290 LOOKUP
1.76 2587364.42 us 45.00 us 134512764.00 us 104 READDIR
4.39 208181.93 us 58.00 us 39750208.00 us 3224 READDIRP
4.59 4377.20 us 259.00 us 10053127.00 us 160191 RCHECKSUM
7.95 5493.00 us 38.00 us 1437578.00 us 221339 READ
21.05 110957.49 us 151.00 us 15390646.00 us 29002 FXATTROP
27.49 29140.19 us 47.00 us 5380701.00 us 144248 WRITE
28.45 103576.62 us 12.00 us 39311904.00 us 41993 FINODELK
Duration: 4279793 seconds
Data Read: 90182870915545 bytes
Data Written: 43304871600094 bytes
Interval 5 Stats:
Block Size: 131072b+
No. of Reads: 585
No. of Writes: 0
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 390.00 us 390.00 us 390.00 us 1 FSTAT
0.01 363.34 us 54.00 us 4960.00 us 35 OPENDIR
0.01 175.43 us 30.00 us 3792.00 us 91 STATFS
0.22 3670.76 us 65.00 us 173128.00 us 148 READ
0.54 18823.38 us 146.00 us 428923.00 us 72 READDIRP
1.75 55.66 us 16.00 us 63524.00 us 78812 INODELK
2.19 1615.88 us 16.00 us 28474.00 us 3397 STAT
12.31 2043.55 us 96.00 us 127094.00 us 15129 LOOKUP
82.99 5290.98 us 359.00 us 998606.00 us 39404 RCHECKSUM
Duration: 246 seconds
Data Read: 76677120 bytes
Data Written: 0 bytes
Brick: HOSTNAME-01-B:/brick7/gvAA01/brick
-----------------------------------------
Cumulative Stats:
Block Size: 1b+ 2b+ 4b+
No. of Reads: 0 13 4
No. of Writes: 1196 734 1040
Block Size: 8b+ 16b+ 32b+
No. of Reads: 1 36 3299
No. of Writes: 2062 300726 37753
Block Size: 64b+ 128b+ 256b+
No. of Reads: 2023 496 525
No. of Writes: 78098 422006 308797
Block Size: 512b+ 1024b+ 2048b+
No. of Reads: 18849 14540 35638
No. of Writes: 1846989 429384177 84605233
Block Size: 4096b+ 8192b+ 16384b+
No. of Reads: 150787 230288 420816
No. of Writes: 137132537 56105000 83554076
Block Size: 32768b+ 65536b+ 131072b+
No. of Reads: 1193840 1821978 1328519682
No. of Writes: 23571537 108072617 188243753
Block Size: 262144b+ 524288b+ 1048576b+
No. of Reads: 0 0 0
No. of Writes: 1439 45 467
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 584854 FORGET
0.00 0.00 us 0.00 us 0.00 us 847666 RELEASE
0.00 0.00 us 0.00 us 0.00 us 25363464 RELEASEDIR
0.00 64.12 us 40.00 us 118.00 us 8 LK
0.00 55.38 us 28.00 us 195.00 us 13 FLUSH
0.00 584.50 us 296.00 us 873.00 us 2 XATTROP
0.00 191.79 us 117.00 us 302.00 us 14 SETATTR
0.00 861.00 us 700.00 us 921.00 us 6 MKNOD
0.00 15796.00 us 15796.00 us 15796.00 us 1 RENAME
0.00 242.94 us 137.00 us 827.00 us 78 SETXATTR
0.00 80.85 us 29.00 us 2109.00 us 280 ENTRYLK
0.00 40533.00 us 40533.00 us 40533.00 us 1 LINK
0.00 12168.00 us 3663.00 us 29808.00 us 5 CREATE
0.00 85.45 us 27.00 us 4758.00 us 894 STATFS
0.00 1005.23 us 84.00 us 6375.00 us 157 FSYNC
0.00 2083.33 us 570.00 us 29605.00 us 78 MKDIR
0.00 46997.20 us 1143.00 us 78119.00 us 5 UNLINK
0.00 57905.17 us 236.00 us 344520.00 us 6 READDIR
0.01 76.43 us 28.00 us 86185.00 us 8213 FSTAT
0.02 155.31 us 76.00 us 61948.00 us 9793 OPEN
0.02 7686.27 us 146.00 us 89509.00 us 231 RCHECKSUM
0.04 100.69 us 3.00 us 41331.00 us 31020 OPENDIR
0.09 3927.64 us 81.00 us 852406.00 us 1732 FXATTROP
0.13 238.47 us 21.00 us 36324.00 us 39834 GETXATTR
0.20 217.49 us 42.00 us 104309.00 us 68968 READ
0.55 1549.97 us 62.00 us 186554.00 us 25900 WRITE
1.14 8343.75 us 13.00 us 877090.00 us 10010 STAT
2.36 2870.24 us 55.00 us 133764.00 us 60231 READDIRP
3.39 1648.07 us 15.00 us 1386407.00 us 150815 LOOKUP
18.82 34568.94 us 20.00 us 255108468.00 us 39875 INODELK
73.20 2950233.38 us 22.00 us 238932201.00 us 1817 FINODELK
Duration: 4279793 seconds
Data Read: 174378591384147 bytes
Data Written: 42067381218610 bytes
Interval 5 Stats:
Block Size: 32b+ 2048b+ 4096b+
No. of Reads: 1 0 0
No. of Writes: 0 2 90
Block Size: 8192b+ 16384b+ 32768b+
No. of Reads: 0 0 1
No. of Writes: 34 59 174
Block Size: 65536b+ 131072b+
No. of Reads: 0 102
No. of Writes: 1045 1863
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 8 FORGET
0.00 0.00 us 0.00 us 0.00 us 5 RELEASE
0.00 0.00 us 0.00 us 0.00 us 93 RELEASEDIR
0.00 48.00 us 35.00 us 61.00 us 2 FLUSH
0.00 182.00 us 79.00 us 285.00 us 2 FSTAT
0.00 188.50 us 184.00 us 193.00 us 2 SETATTR
0.00 873.00 us 873.00 us 873.00 us 1 XATTROP
0.00 492.00 us 154.00 us 1073.00 us 3 OPEN
0.00 314.67 us 128.00 us 658.00 us 6 GETXATTR
0.00 115.02 us 29.00 us 1537.00 us 44 ENTRYLK
0.00 168.39 us 45.00 us 861.00 us 38 OPENDIR
0.00 166.43 us 39.00 us 4758.00 us 91 STATFS
0.00 40533.00 us 40533.00 us 40533.00 us 1 LINK
0.00 23496.50 us 17185.00 us 29808.00 us 2 CREATE
0.00 1991.03 us 209.00 us 17197.00 us 35 READ
0.00 1044.09 us 84.00 us 3060.00 us 74 FSYNC
0.00 1268.33 us 159.00 us 8139.00 us 70 READDIRP
0.00 39216.75 us 1143.00 us 60962.00 us 4 UNLINK
0.01 149.32 us 16.00 us 55158.00 us 3479 STAT
0.01 7967.72 us 146.00 us 76151.00 us 106 RCHECKSUM
0.02 2188.39 us 81.00 us 87414.00 us 500 FXATTROP
0.08 296.47 us 80.00 us 20575.00 us 16886 LOOKUP
0.24 10693.63 us 85.00 us 186554.00 us 1365 WRITE
20.68 5275482.42 us 32.00 us 255108468.00 us 239 INODELK
78.95 8146074.61 us 22.00 us 238932201.00 us 591 FINODELK
Duration: 246 seconds
Data Read: 13408908 bytes
Data Written: 373263308 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick2
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 4595631725
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 5210160 FORGET
0.00 0.00 us 0.00 us 0.00 us 19833857 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121845659 RELEASEDIR
0.00 47.77 us 29.00 us 72.00 us 22 FLUSH
0.00 67.21 us 32.00 us 161.00 us 42 LK
0.00 190.20 us 142.00 us 368.00 us 15 SETATTR
0.00 355.78 us 255.00 us 408.00 us 9 RENAME
0.01 660.58 us 239.00 us 988.00 us 12 UNLINK
0.01 223.95 us 142.00 us 652.00 us 78 SETXATTR
0.02 1351.87 us 991.00 us 1807.00 us 15 MKNOD
0.11 1625.97 us 1013.00 us 3225.00 us 78 MKDIR
0.14 134.56 us 73.00 us 1296.00 us 1194 OPEN
0.18 1095.03 us 265.00 us 17673.00 us 195 XATTROP
0.25 56.99 us 18.00 us 3303.00 us 5290 ENTRYLK
0.57 286.73 us 46.00 us 28784.00 us 2379 OPENDIR
2.47 69.67 us 17.00 us 31954.00 us 42231 FINODELK
3.17 26765.84 us 73.00 us 243526.00 us 141 FSYNC
6.30 73.60 us 22.00 us 18488.00 us 101758 WRITE
9.51 389.77 us 167.00 us 21472.00 us 29019 FXATTROP
17.26 62.64 us 16.00 us 37178.00 us 327699 INODELK
59.99 1046.54 us 81.00 us 83922.00 us 68182 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 4595631725 bytes
Interval 5 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.11 1109.06 us 64.00 us 28784.00 us 35 OPENDIR
14.97 64.35 us 16.00 us 36247.00 us 78809 INODELK
84.92 1903.39 us 101.00 us 57323.00 us 15117 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick5
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 3725579480
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 4675824 FORGET
0.00 0.00 us 0.00 us 0.00 us 3501480 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121321048 RELEASEDIR
0.00 43.60 us 26.00 us 60.00 us 5 GETXATTR
0.00 183.50 us 136.00 us 254.00 us 6 SETATTR
0.00 94.50 us 34.00 us 360.00 us 12 FLUSH
0.00 637.50 us 378.00 us 897.00 us 2 UNLINK
0.00 152.08 us 54.00 us 1647.00 us 24 LK
0.01 923.40 us 352.00 us 2505.00 us 5 READDIR
0.02 1971.33 us 952.00 us 4079.00 us 6 MKNOD
0.03 278.36 us 158.00 us 1290.00 us 78 SETXATTR
0.03 70.37 us 27.00 us 906.00 us 334 ENTRYLK
0.06 147.48 us 83.00 us 1130.00 us 311 OPEN
0.18 1747.03 us 991.00 us 14702.00 us 78 MKDIR
0.28 178.04 us 33.00 us 9265.00 us 1190 OPENDIR
0.31 92.93 us 27.00 us 28425.00 us 2510 FINODELK
2.11 1980.44 us 117.00 us 116330.00 us 812 FSYNC
2.48 1109.55 us 141.00 us 193978.00 us 1702 FXATTROP
3.54 77.89 us 21.00 us 19627.00 us 34622 WRITE
7.06 60.51 us 21.00 us 43675.00 us 88861 INODELK
83.89 470.13 us 46.00 us 227967.00 us 135916 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 3725579480 bytes
Interval 5 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 949
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.09 532.37 us 59.00 us 4354.00 us 35 OPENDIR
0.15 79.14 us 42.00 us 813.00 us 388 WRITE
0.44 75.07 us 27.00 us 1936.00 us 1164 FINODELK
0.94 241.59 us 141.00 us 3508.00 us 776 FXATTROP
1.86 957.12 us 124.00 us 9367.00 us 388 FSYNC
8.59 63.47 us 21.00 us 43675.00 us 27031 INODELK
87.92 1125.41 us 103.00 us 60160.00 us 15599 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 949 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick9
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 13824276920
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 11714931 FORGET
0.00 0.00 us 0.00 us 0.00 us 12574659 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121307214 RELEASEDIR
0.00 612.00 us 612.00 us 612.00 us 1 RENAME
0.00 53.69 us 32.00 us 75.00 us 13 FLUSH
0.00 146.43 us 118.00 us 157.00 us 7 SETATTR
0.00 1053.00 us 1053.00 us 1053.00 us 1 UNLINK
0.00 90.76 us 39.00 us 575.00 us 25 LK
0.01 1011.83 us 280.00 us 3498.00 us 6 READDIR
0.01 1154.50 us 873.00 us 1578.00 us 6 MKNOD
0.01 65.37 us 32.00 us 609.00 us 202 ENTRYLK
0.02 212.12 us 124.00 us 561.00 us 78 SETXATTR
0.05 408.86 us 31.00 us 989.00 us 111 GETXATTR
0.07 66.08 us 25.00 us 1177.00 us 1094 FINODELK
0.11 1413.50 us 793.00 us 6939.00 us 78 MKDIR
0.17 145.79 us 3.00 us 16855.00 us 1188 OPENDIR
0.90 566.03 us 178.00 us 101200.00 us 1583 FXATTROP
1.17 132.19 us 70.00 us 15892.00 us 8843 OPEN
2.28 63.93 us 25.00 us 14064.00 us 35560 INODELK
3.30 25327.51 us 77.00 us 275527.00 us 130 FSYNC
6.02 71.86 us 20.00 us 29539.00 us 83532 WRITE
85.86 950.72 us 45.00 us 234956.00 us 89984 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 13824276920 bytes
Interval 5 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 107.50 us 84.00 us 131.00 us 2 ENTRYLK
0.00 1053.00 us 1053.00 us 1053.00 us 1 UNLINK
0.13 946.06 us 58.00 us 16855.00 us 35 OPENDIR
99.87 1700.08 us 89.00 us 67917.00 us 15086 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick1
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 2407865216
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 3885977 FORGET
0.00 0.00 us 0.00 us 0.00 us 1908766 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121336583 RELEASEDIR
0.00 41.33 us 37.00 us 44.00 us 3 FLUSH
0.00 107.40 us 60.00 us 216.00 us 5 LK
0.00 197.00 us 144.00 us 367.00 us 9 SETATTR
0.00 559.25 us 269.00 us 1318.00 us 4 UNLINK
0.01 117.55 us 34.00 us 426.00 us 49 GETXATTR
0.01 729.10 us 111.00 us 2755.00 us 10 FSTAT
0.02 283.04 us 169.00 us 1903.00 us 78 SETXATTR
0.04 71.01 us 24.00 us 3460.00 us 547 ENTRYLK
0.05 5719.78 us 1045.00 us 39386.00 us 9 MKNOD
0.10 1199.26 us 338.00 us 2419.00 us 88 XATTROP
0.10 18965.50 us 244.00 us 111009.00 us 6 READDIR
0.13 1842.74 us 740.00 us 17409.00 us 78 MKDIR
0.22 196.96 us 3.00 us 13973.00 us 1241 OPENDIR
0.37 848.02 us 97.00 us 22427.00 us 478 FSYNC
0.58 142.08 us 80.00 us 9771.00 us 4495 OPEN
0.71 79.24 us 30.00 us 8519.00 us 9794 WRITE
0.72 197.05 us 83.00 us 36661.00 us 4020 FXATTROP
2.42 80.26 us 16.00 us 20655.00 us 33176 INODELK
4.57 791.78 us 19.00 us 854250.00 us 6345 FINODELK
89.97 1130.79 us 47.00 us 716773.00 us 87554 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 2407865216 bytes
Interval 5 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 2300
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 159 RELEASEDIR
0.02 993.33 us 161.00 us 2755.00 us 6 FSTAT
0.04 85.40 us 24.00 us 3460.00 us 169 ENTRYLK
0.14 957.84 us 61.00 us 13973.00 us 55 OPENDIR
0.16 76.25 us 38.00 us 1284.00 us 795 WRITE
0.17 1246.94 us 468.00 us 2063.00 us 52 XATTROP
0.36 755.13 us 97.00 us 10173.00 us 185 FSYNC
0.68 165.35 us 92.00 us 6470.00 us 1586 FXATTROP
1.75 90.50 us 16.00 us 7633.00 us 7475 INODELK
6.20 972.86 us 23.00 us 854250.00 us 2464 FINODELK
90.50 1686.49 us 89.00 us 78332.00 us 20759 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 2300 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick8
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 5492133233
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 6710878 FORGET
0.00 0.00 us 0.00 us 0.00 us 5709109 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121308841 RELEASEDIR
0.00 381.00 us 381.00 us 381.00 us 1 LINK
0.00 321.50 us 309.00 us 334.00 us 2 RENAME
0.00 50.13 us 31.00 us 92.00 us 15 FLUSH
0.00 68.93 us 43.00 us 136.00 us 30 LK
0.00 207.60 us 137.00 us 379.00 us 10 SETATTR
0.00 456.20 us 247.00 us 837.00 us 5 UNLINK
0.00 67.16 us 39.00 us 233.00 us 56 FINODELK
0.01 1313.12 us 943.00 us 1632.00 us 8 MKNOD
0.01 61.85 us 34.00 us 384.00 us 214 ENTRYLK
0.02 260.12 us 141.00 us 1053.00 us 78 SETXATTR
0.04 688.39 us 327.00 us 2844.00 us 56 FXATTROP
0.11 1360.77 us 862.00 us 3735.00 us 78 MKDIR
0.22 177.44 us 49.00 us 11813.00 us 1185 OPENDIR
0.90 147.72 us 78.00 us 17203.00 us 5732 OPEN
1.30 79.82 us 37.00 us 11963.00 us 15230 WRITE
1.65 67.26 us 23.00 us 13182.00 us 23091 INODELK
95.71 796.62 us 69.00 us 131536.00 us 112767 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 5492133233 bytes
Interval 5 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 106.00 us 84.00 us 128.00 us 2 ENTRYLK
0.00 379.00 us 379.00 us 379.00 us 1 SETATTR
0.00 513.00 us 513.00 us 513.00 us 1 UNLINK
0.00 665.50 us 662.00 us 669.00 us 2 INODELK
0.15 1173.54 us 86.00 us 11813.00 us 35 OPENDIR
99.84 1831.00 us 105.00 us 93579.00 us 15150 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick3
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 4572691669
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 4594149 FORGET
0.00 0.00 us 0.00 us 0.00 us 3305802 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121331753 RELEASEDIR
0.00 351.00 us 351.00 us 351.00 us 1 RENAME
0.00 52.20 us 31.00 us 74.00 us 15 FLUSH
0.00 150.27 us 116.00 us 192.00 us 11 SETATTR
0.00 70.48 us 40.00 us 155.00 us 29 LK
0.00 698.00 us 294.00 us 1204.00 us 4 UNLINK
0.01 101.02 us 45.00 us 170.00 us 123 GETXATTR
0.02 226.96 us 132.00 us 661.00 us 78 SETXATTR
0.03 2449.45 us 981.00 us 14588.00 us 11 MKNOD
0.03 133.80 us 33.00 us 14384.00 us 205 ENTRYLK
0.08 13713.00 us 250.00 us 79575.00 us 6 READDIR
0.11 1403.08 us 856.00 us 3307.00 us 78 MKDIR
0.15 130.34 us 3.00 us 7373.00 us 1188 OPENDIR
0.63 135.12 us 75.00 us 3913.00 us 4698 OPEN
1.58 151.39 us 20.00 us 720477.00 us 10441 FINODELK
2.03 70.30 us 26.00 us 31988.00 us 28857 INODELK
3.61 457.95 us 148.00 us 96986.00 us 7889 FXATTROP
13.93 74.03 us 19.00 us 38536.00 us 188198 WRITE
15.54 5296.92 us 87.00 us 285482.00 us 2936 FSYNC
62.24 723.17 us 19.00 us 1037869.00 us 86100 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 4572691669 bytes
Interval 5 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 2170
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.00 105.67 us 65.00 us 164.00 us 3 ENTRYLK
0.01 950.00 us 950.00 us 950.00 us 1 UNLINK
0.07 398.43 us 64.00 us 5324.00 us 35 OPENDIR
0.34 88.92 us 36.00 us 4283.00 us 716 WRITE
0.52 88.23 us 22.00 us 6820.00 us 1112 FINODELK
1.21 289.14 us 156.00 us 6432.00 us 793 FXATTROP
1.53 796.04 us 104.00 us 2668.00 us 365 FSYNC
1.55 68.03 us 29.00 us 1880.00 us 4317 INODELK
94.77 1086.38 us 76.00 us 101220.00 us 16536 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 2170 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick6
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 3299040333
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 4224301 FORGET
0.00 0.00 us 0.00 us 0.00 us 2341147 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121317598 RELEASEDIR
0.00 75.25 us 39.00 us 117.00 us 4 FLUSH
0.00 70.62 us 47.00 us 100.00 us 8 LK
0.00 156.11 us 125.00 us 189.00 us 9 SETATTR
0.00 1980.00 us 1980.00 us 1980.00 us 1 RENAME
0.00 364.29 us 311.00 us 434.00 us 7 UNLINK
0.01 134.92 us 31.00 us 662.00 us 53 GETXATTR
0.01 1307.44 us 1005.00 us 1850.00 us 9 MKNOD
0.01 79.36 us 26.00 us 665.00 us 157 FINODELK
0.02 275.58 us 162.00 us 1593.00 us 78 SETXATTR
0.03 108.84 us 33.00 us 4176.00 us 238 ENTRYLK
0.12 1469.38 us 947.00 us 7741.00 us 78 MKDIR
0.16 17107.56 us 88.00 us 150018.00 us 9 READDIR
0.22 170.89 us 3.00 us 12533.00 us 1191 OPENDIR
0.24 1117.26 us 216.00 us 79180.00 us 202 FXATTROP
0.85 38142.10 us 498.00 us 126241.00 us 21 FSYNC
0.89 85.41 us 21.00 us 27516.00 us 9833 WRITE
0.98 138.12 us 74.00 us 3527.00 us 6670 OPEN
1.87 65.50 us 24.00 us 18157.00 us 26861 INODELK
94.56 840.11 us 19.00 us 150836.00 us 105706 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 3299040333 bytes
Interval 5 Stats:
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.01 1980.00 us 1980.00 us 1980.00 us 1 RENAME
0.04 2164.25 us 113.00 us 4176.00 us 4 ENTRYLK
0.14 780.77 us 62.00 us 12533.00 us 35 OPENDIR
99.81 1300.71 us 65.00 us 96686.00 us 15135 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 0 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick4
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 4188179800
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 4843776 FORGET
0.00 0.00 us 0.00 us 0.00 us 4734700 RELEASE
0.00 0.00 us 0.00 us 0.00 us 122187144 RELEASEDIR
0.00 453.00 us 453.00 us 453.00 us 1 RENAME
0.00 2935.00 us 2935.00 us 2935.00 us 1 MKNOD
0.00 314.83 us 275.00 us 348.00 us 12 LINK
0.00 46.91 us 22.00 us 394.00 us 116 FLUSH
0.00 78.42 us 37.00 us 725.00 us 78 LK
0.00 152.00 us 66.00 us 2303.00 us 41 FTRUNCATE
0.01 120.96 us 27.00 us 211.00 us 80 GETXATTR
0.01 1123.27 us 400.00 us 1955.00 us 11 XATTROP
0.01 264.98 us 89.00 us 4166.00 us 59 SETATTR
0.01 522.77 us 219.00 us 1988.00 us 31 UNLINK
0.01 3116.33 us 335.00 us 15814.00 us 6 READDIR
0.02 194.27 us 87.00 us 1585.00 us 146 OPEN
0.02 404.47 us 146.00 us 10053.00 us 78 SETXATTR
0.03 94.93 us 24.00 us 12262.00 us 407 ENTRYLK
0.04 380.95 us 82.00 us 30028.00 us 169 FSTAT
0.08 1825.35 us 1237.00 us 3241.00 us 62 CREATE
0.11 2054.41 us 1177.00 us 7853.00 us 78 MKDIR
0.11 137.97 us 4.00 us 4662.00 us 1188 OPENDIR
0.93 140.91 us 18.00 us 368752.00 us 9912 FINODELK
2.28 404.58 us 88.00 us 26154.00 us 8431 FXATTROP
20.16 9694.89 us 64.00 us 288694.00 us 3111 FSYNC
20.76 68.66 us 19.00 us 45520.00 us 452355 WRITE
26.67 59.16 us 16.00 us 26108.00 us 674404 INODELK
28.72 687.59 us 46.00 us 85079.00 us 62473 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 4188179800 bytes
Interval 5 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 62
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 90 RELEASEDIR
0.01 71.83 us 46.00 us 116.00 us 30 WRITE
0.03 150.81 us 33.00 us 1289.00 us 37 FINODELK
0.08 489.54 us 63.00 us 4603.00 us 35 OPENDIR
0.20 874.41 us 240.00 us 2977.00 us 51 FXATTROP
19.05 66.56 us 17.00 us 14272.00 us 62733 INODELK
80.63 1162.50 us 99.00 us 85079.00 us 15201 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 62 bytes
Brick: HOSTNAME-00-A:/arbiterAA01/gvAA01/brick7
-----------------------------------------------
Cumulative Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 4459447660
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 4903888 FORGET
0.00 0.00 us 0.00 us 0.00 us 4906100 RELEASE
0.00 0.00 us 0.00 us 0.00 us 121313760 RELEASEDIR
0.00 157.00 us 157.00 us 157.00 us 1 FSTAT
0.00 334.00 us 334.00 us 334.00 us 1 LINK
0.00 492.00 us 492.00 us 492.00 us 1 RENAME
0.00 81.56 us 50.00 us 121.00 us 9 LK
0.00 600.00 us 433.00 us 767.00 us 2 XATTROP
0.00 284.69 us 26.00 us 1664.00 us 13 FLUSH
0.00 343.79 us 162.00 us 1811.00 us 14 SETATTR
0.00 1061.67 us 236.00 us 3527.00 us 6 UNLINK
0.01 1371.00 us 997.00 us 2107.00 us 6 MKNOD
0.01 109.50 us 33.00 us 166.00 us 103 GETXATTR
0.01 3619.60 us 1046.00 us 6783.00 us 5 CREATE
0.02 258.63 us 161.00 us 1183.00 us 78 SETXATTR
0.05 214.24 us 27.00 us 6443.00 us 280 ENTRYLK
0.09 1455.38 us 615.00 us 7156.00 us 78 MKDIR
0.11 24019.00 us 296.00 us 140960.00 us 6 READDIR
0.17 181.83 us 4.00 us 11536.00 us 1191 OPENDIR
0.20 1608.87 us 48.00 us 21829.00 us 157 FSYNC
1.05 138.38 us 73.00 us 1645.00 us 9762 OPEN
1.27 945.52 us 99.00 us 35935.00 us 1730 FXATTROP
2.19 71.12 us 25.00 us 12922.00 us 39768 INODELK
2.31 115.41 us 18.00 us 17045.00 us 25869 WRITE
10.45 6550.60 us 19.00 us 5168504.00 us 2062 FINODELK
82.07 703.56 us 22.00 us 235243.00 us 150709 LOOKUP
Duration: 19297631 seconds
Data Read: 0 bytes
Data Written: 4459447660 bytes
Interval 5 Stats:
Block Size: 1b+
No. of Reads: 0
No. of Writes: 3267
%-latency Avg-latency Min-Latency Max-Latency No. of calls Fop
--------- ----------- ----------- ----------- ------------ ----
0.00 0.00 us 0.00 us 0.00 us 9 FORGET
0.00 0.00 us 0.00 us 0.00 us 5 RELEASE
0.00 0.00 us 0.00 us 0.00 us 93 RELEASEDIR
0.00 157.00 us 157.00 us 157.00 us 1 FSTAT
0.00 334.00 us 334.00 us 334.00 us 1 LINK
0.00 187.00 us 170.00 us 204.00 us 2 SETATTR
0.00 272.50 us 54.00 us 491.00 us 2 FLUSH
0.00 767.00 us 767.00 us 767.00 us 1 XATTROP
0.01 654.00 us 153.00 us 1645.00 us 3 OPEN
0.03 1135.40 us 236.00 us 3527.00 us 5 UNLINK
0.04 3975.50 us 1713.00 us 6238.00 us 2 CREATE
0.05 305.97 us 56.00 us 2057.00 us 38 OPENDIR
0.12 607.39 us 27.00 us 6443.00 us 44 ENTRYLK
0.59 1786.74 us 48.00 us 21829.00 us 74 FSYNC
0.73 661.93 us 33.00 us 12922.00 us 245 INODELK
3.02 1345.12 us 101.00 us 31614.00 us 500 FXATTROP
3.23 526.99 us 18.00 us 14746.00 us 1365 WRITE
22.47 6998.94 us 19.00 us 2596341.00 us 715 FINODELK
69.70 916.77 us 78.00 us 51010.00 us 16930 LOOKUP
Duration: 246 seconds
Data Read: 0 bytes
Data Written: 3267 bytes
More information about the Gluster-users
mailing list