[Gluster-users] Possible workaround for a problem with a "permissions" or "file access" error

[Joe Landman]

Hi folks

  We saw this at two different customer sites, one running 3.0.5 and one 
running 3.1.1.  This is a distribute layout, no replication.  Pretty 
intensive loads on the siCluster.

Every now and then there was a hiccup:

258% ls -l /x/y/.z/t.xml
-rw-r--r-- 1 a b 574 Dec 9 07:10
  /x/y/.z/t.xml
259% cat !$
cat /x/y/.z/t.xml
cat: /x/y/.z/t.xml: Input/output error
260% df -h !$
df -h /x/y/.z/t.xml
Filesystem Size Used Avail Use% Mounted on
a.b.c.d:/nfs 15T 100G 15T 1% /x

The file was really there.  And it was really inaccessible.  Remounts 
sometimes helped.  Logs contained access errors.  File system on backend 
was fine, no issues.  Network was fine, no issues.

Way back in 2.0.x days, there were caching bugs.  stat-prefetch and 
others were problematic.  I remembered this.  So we tried this at the 
3.0.5 customer site (set timeout to zero and remove stat-prefetch)

volume quickread
     type performance/quick-read
     option cache-timeout 0
     option max-file-size 64kB
     subvolumes iocache
end-volume

#volume statprefetch
#    type performance/stat-prefetch
#    subvolumes quickread
#end-volume

and it seemed to fix it (changed cache-timeout to 0 and commented out 
stat-prefetch).  Still testing, but it seems to have made that bug less 
problematic.

On 3.1.1, there are no config files.  And there is no comprehensive list 
of volume/translator switches, nor any real clue as to what the 
translator layers are in use.

So we experimented until we found this:

[root at manager ~]# gluster volume set nfs performance.cache-refresh-timeout 0
Set volume successful

[root at manager ~]# gluster volume set nfs performance.stat-prefetch 0
Set volume successful

These seem to have worked.  And we are testing to see if they fix the 
problem.

Has someone somewhere compiled a complete list of the settable config 
elements and their meanings?  I don't know if the settings simply set 
key-value pairs, or if they actually impact things.  A listing of these 
would be nice, though I found what we found by some creative guessing 
based upon older configurations.

Regards,

Joe

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics Inc.
email: landman at scalableinformatics.com
web  : http://scalableinformatics.com
        http://scalableinformatics.com/sicluster
phone: +1 734 786 8423 x121
fax  : +1 866 888 3112
cell : +1 734 612 4615