[Gluster-users] forrtl: severe (51): inconsistent file organization error on Gluster

[Neil Van Lysel]

Hello,

I recently setup a SLURM cluster with a shared filesystem using Gluster. 
The Gluster nodes are connected to the rest of the cluster with a 56Gb 
InfiniBand Interconnect.

Some of our users are receiving the following error when they run VASP 
jobs that access files on Gluster:

     forrtl: severe (51): inconsistent file organization, unit 12 
/path/to/file/WAVECAR

Is this an error with VASP or Gluster? If it is an error with Gluster 
how do I fix it? I do not know much about Gluster so I need some help.

Here are some relevant specs:
     [root at aci-storage-1 ~]# gluster --version
     glusterfs 3.4.0beta2 built on May 24 2013 14:11:16

     [root at aci-storage-1 ~]# gluster volume info
     Volume Name: scratch
     Type: Distribute
     Volume ID: 2d30a015-0452-45a3-9a1d-42cee619d35f
     Status: Started
     Number of Bricks: 8
     Transport-type: tcp
     Bricks:
     Brick1: 10.129.40.21:/data/glusterfs/brick1/scratch
     Brick2: 10.129.40.21:/data/glusterfs/brick2/scratch
     Brick3: 10.129.40.22:/data/glusterfs/brick1/scratch
     Brick4: 10.129.40.22:/data/glusterfs/brick2/scratch
     Brick5: 10.129.40.23:/data/glusterfs/brick1/scratch
     Brick6: 10.129.40.23:/data/glusterfs/brick2/scratch
     Brick7: 10.129.40.24:/data/glusterfs/brick1/scratch
     Brick8: 10.129.40.24:/data/glusterfs/brick2/scratch
     Options Reconfigured:
     features.quota: on
     features.limit-usage: /:80TB

     Volume Name: home
     Type: Distribute
     Volume ID: 711465cf-db6c-4407-9b02-43e44ee4779b
     Status: Started
     Number of Bricks: 8
     Transport-type: tcp
     Bricks:
     Brick1: 10.129.40.21:/data/glusterfs/brick1/home
     Brick2: 10.129.40.21:/data/glusterfs/brick2/home
     Brick3: 10.129.40.22:/data/glusterfs/brick1/home
     Brick4: 10.129.40.22:/data/glusterfs/brick2/home
     Brick5: 10.129.40.23:/data/glusterfs/brick1/home
     Brick6: 10.129.40.23:/data/glusterfs/brick2/home
     Brick7: 10.129.40.24:/data/glusterfs/brick1/home
     Brick8: 10.129.40.24:/data/glusterfs/brick2/home
     Options Reconfigured:
     features.limit-usage: /:30TB
     features.quota: on

There doesn't appear to be any significant errors in the log files, but 
/var/log/glusterfs/scratch.log does have a lot of these types of messages:
     [2013-06-27 21:57:21.399355] W [quota.c:2167:quota_fstat_cbk] 
0-scratch-quota: quota context not set in inode 
(gfid:0b855d43-2a51-42bc-8707-fbe010cfe5b9)
     [2013-06-27 21:59:29.188686] E [io-cache.c:557:ioc_open_cbk] 
0-scratch-io-cache: inode context is NULL 
(5555d554-41ff-44be-be88-af3b0d570876)
     [2013-06-27 21:59:29.189095] W [quota.c:2301:quota_readv_cbk] 
0-scratch-quota: quota context not set in inode 
(gfid:5555d554-41ff-44be-be88-af3b0d570876)
     [2013-06-27 21:59:34.296190] E [io-cache.c:557:ioc_open_cbk] 
0-scratch-io-cache: inode context is NULL 
(5555d554-41ff-44be-be88-af3b0d570876)
     [2013-06-27 21:59:34.296686] W [quota.c:2301:quota_readv_cbk] 
0-scratch-quota: quota context not set in inode 
(gfid:5555d554-41ff-44be-be88-af3b0d570876)
     [2013-06-27 22:01:41.415542] E [io-cache.c:557:ioc_open_cbk] 
0-scratch-io-cache: inode context is NULL 
(bb9a4fba-3cc9-4d2a-a937-00752ec6c5d2)
     [2013-06-27 22:01:41.416062] W [quota.c:2301:quota_readv_cbk] 
0-scratch-quota: quota context not set in inode 
(gfid:bb9a4fba-3cc9-4d2a-a937-00752ec6c5d2)
     [2013-06-27 22:01:43.570357] W [quota.c:1253:quota_unlink_cbk] 
0-scratch-quota: quota context not set in inode 
(gfid:bb9a4fba-3cc9-4d2a-a937-00752ec6c5d2)
     [2013-06-27 22:01:43.571182] W [quota.c:1253:quota_unlink_cbk] 
0-scratch-quota: quota context not set in inode 
(gfid:592ca6e8-31f9-4e97-9fe3-68ecaa806f22)

Please let me know if you need anything else.

Thanks much,

Neil Van Lysel
van-lyse at cs.wisc.edu
UNIX Systems Administrator
Center for High Throughput Computing
University of Wisconsin - Madison
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130627/42fd41c0/attachment.html>