[Bugs] [Bug 1576411] New: when running a one liner recursive find with awks, gluster client gets unstable

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1576411

            Bug ID: 1576411
           Summary: when running a one liner recursive find with awks,
                    gluster client gets unstable
           Product: GlusterFS
           Version: 3.12
         Component: fuse
          Severity: medium
          Assignee: bugs at gluster.org
          Reporter: avishay.aton at gmail.com
                CC: bugs at gluster.org



Description of problem: when running a one liner recursive find with awks,
gluster client gets unstable


Version-Release number of selected component (if applicable): 12.6.1 and 12.9.1
on CentOS 7.4



How reproducible:


Steps to Reproduce:
1. mount gluster:/volume /mnt/gluster
2. cd /mnt/gluster/FOLDER
3. find . -type f -print0 | xargs -0 ls -l | awk '{ n=int(log($5)/log(2)); if
(n<10) { n=10; } size[n]++ } END { for (i in size) printf("%d %d\n", 2^i,
size[i]) }' | sort -n | awk 'function human(x) { x[1]/=1024; if (x[1]>=1024) {
x[2]++; human(x) } } { a[1]=$1; a[2]=0; human(a); printf("%3d%s: %6d\n",
a[1],substr("kMGTEPYZ",a[2]+1,1),$2) }'

4. it runs find for once or twice. but then we get the following errors:


ls: cannot access ./short/GF1/j1/xc_work/xc_make: Transport endpoint is not
connected
ls: cannot access ./short/GF1/j1/xc_work/test23.log: Transport endpoint is not
connected
ls: cannot access ./short/GF1/j1/xc_work/pbc.vlog: Transport endpoint is not
connected
ls: cannot access ./short/GF1/j1/xc_work/hdl.var: Transport endpoint is not
connected
ls: cannot access ./short/GF1/j1/xc_work/xc_top.xdb: Transport endpoint is not
connected
ls: cannot access ./short/GF1/jl/xc_work/xc_top.ccf: Transport endpoint is not
connected
ls: cannot access ./short/GF1/j1/xc_work/pdb.ucdb: Transport endpoint is not
connected
ls: cannot access ./short/GF1/j1/1cl-1cpu/.design: Transport endpoint is not
connected


then, any "ls" or "cd" gives the following :
 ls
ls: cannot open directory .: Transport endpoint is not connected


only umount and mount solves the problem, until we run the find command again




Actual results:


Expected results:(file size histogram)

 1k:     79
  2k:      7
  4k:   1402
  8k:   3552
 16k:   3534
 32k:   3973
 64k:   1648
128k:   1313
256k:    684
512k:    485
  1M:    195
  2M:     79
  4M:     25
  8M:     20
 16M:     27
 32M:     13
 64M:      6
128M:      1
256M:      5



Additional info:
gluster distributed-replicated (9x2) with 18 ec2 instances. lots of cpu, ram
and EBS bandwidth.
typical workload : small files
volume optimized for small file:

performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
performance.parallel-readdir: on
performance.cache-size: 1024MB
performance.io-thread-count: 32
cluster.lookup-optimize: on
network.inode-lru-limit: 90000
performance.stat-prefetch: on
client.event-threads: 6
server.event-threads: 6
features.cache-invalidation: on
features.cache-invalidation-timeout: 900
performance.cache-invalidation: on
performance.md-cache-timeout: 600

-- 
You are receiving this mail because:
You are on the CC list for the bug.
You are the assignee for the bug.