[Gluster-users] ls -alR problems on mounted glusterFS

[Larry Bates]

I've set up everything and have moved close to 2Tb of data to the glusterFS 
volume without incident.  I thought that I try a recursive long listing just
to make sure everything was working properly (in anticipation of doing just
that when I turn on AFR soon).

Doing ls -alR on mounted GlusterFS volume.  Everything looks good, but every 
so often I get output that looks like:

-rw-------   1 zope zope   30512 Jan 27  2008 efb1f22f2312af6ed5fbef014ee3
?---------   ? ?    ?          ?            ? /mnt/BACKUPS/blobdata/00/66/1ac485
0e731fed2e0eb2fdbc8d05
?---------   ? ?    ?          ?            ? /mnt/BACKUPS/blobdata/00/66/1d7d22
6f5f6845d62fddcadb4ca9
?---------   ? ?    ?          ?            ? /mnt/BACKUPS/blobdata/00/66/284371
d43c24d0d3267109c674e7


If I stop the listing and check the files individually I get:

[zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1ac4850e731fed2e0
eb2fdbc8d05
-rw------- 1 zope zope 23328 Aug 12  2008 /mnt/BACKUPS/blobdata/00/66/1ac4850e73
1fed2e0eb2fdbc8d05
[zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6845d62
fddcadb4ca9
-rw-rw-r-- 1 zope zope 1616 Nov  2  2007 /mnt/BACKUPS/blobdata/00/66/1d7d226f5f6
845d62fddcadb4ca9
[zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/66/284371d43c24d0d32
67109c674e7
-rw-rw-r-- 1 zope zope 34928 Dec 28  2007 /mnt/BACKUPS/blobdata/00/66/284371d43c
24d0d3267109c674e7
[zope at zodb01 prodscripts]$

Nothing appears to be wrong.

Hint: These errors ALWAYS appear to affect the last few files listed in an 
individual directory (that is it nevery happens in the middle of a folder).


Occasionally I get and error like the following:

ls: /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc36324a41227: Permission denied

Stopping the listing to check this file results in:

[zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/84/7590a35e615c736cc
36324a41227
-rw-rw-r-- 1 zope zope 512 Nov 22  2007 /mnt/BACKUPS/blobdata/00/84/7590a35e615c
736cc36324a41227
[zope at zodb01 prodscripts]$

Again, nothing appears to be wrong.

Hint: These errors ALWAYS appear to affect the last file or files listed in an 
individual directory (that is it nevery happens in the middle of a folder).

Sometimes it affects entire folders not just individual files:

ls: /mnt/BACKUPS/blobdata/00/e1: Permission denied

Stopping the recursive listing and just executing on this single folder works
perfectly.

[zope at zodb01 prodscripts]$ ls -al /mnt/BACKUPS/blobdata/00/e1
total 15976
drwxrwxr-x   2 zope zope    8192 Dec 15 16:46 .
drwxrwxr-x 258 zope zope   16384 Dec 15 16:46 ..
-rw-------   1 zope zope  909456 Sep 15  2008 00fe85dffc1830fe10f34942c2c5
-rw-------   1 zope zope  399136 Oct 15  2008 0effcac8a7fb6819f17fcaea296f
-rw-------   1 zope zope  149456 Jan 27  2008 1ebd7783357503e7b553dcfae23a
-rw-------   1 zope zope   19472 Sep 29 10:27 2d3ada857f0e3692146f98cc923b
-rw-------   1 zope zope    1440 Jun 21 10:18 302ad12cda83dcc7a7f241f9d7d4
-rw-------   1 zope zope     896 Mar  5  2008 3298c88694ee904cfd0ade6715fa
-rw-------   1 zope zope      16 Jun 21 22:06 3ebb81573577e37fff643fbb7dcc
-rw-rw-r--   1 zope zope  132624 Dec 28  2007 415255d42503872b3a88cfe70d6c
-rw-------   1 zope zope     416 Jun 26  2008 489e5ef508e56e2d76ba67c306ea
-rw-------   1 zope zope   29200 Mar 17  2008 4a9f16e818309cc601bd886631cd
-rw-------   1 zope zope  554272 Jan 26  2008 4ac36056b24a9b46e659abc44f49
-rw-rw-r--   1 zope zope  468160 Jan  8  2008 5204e6d98d70e74240ec91263a46
-rw-------   1 zope zope  401616 Jan 26  2008 5255ecacc6709b64b7633cfb9199
-rw-------   1 zope zope   22096 Aug  4 02:49 590c4138783609d0d11b5d7d04a7
-rw-------   1 zope zope 2362752 Nov  8 19:48 631542f36a07601d678975374428
-rw-------   1 zope zope    1248 Feb  8  2009 6335e89fef0bdb300491a246e489
-rw-------   1 zope zope  875024 Jun 29  2008 6439dde68e0d91909518dce73e65
-rw-------   1 zope zope    1248 Feb 20  2008 6818b0f5686f2f4f7d892f42c5bf
-rw-------   1 zope zope   27184 Jul  2  2008 693c281e4f7ae9ced89d0c326411
-rw-------   1 zope zope 1057392 Jan 26  2008 6a0ac5e30a06f3ee7723e4a1eb13
-rw-rw-r--   1 zope zope 2871776 Dec 28  2007 6d932ed95810a51d7a5dd3852ad9
-rw-rw-r--   1 zope zope 2996896 Nov  2  2007 73000cfd294042e5fcdc322e1f3d
-rw-------   1 zope zope    3392 Oct 28 20:23 77638a97c972a1c1a395324d3d68
-rw-rw-r--   1 zope zope   21584 Dec 29  2007 78fdd154c8c45aa793c2a4fdfac5
-rw-------   1 zope zope   21008 Jul 26 19:29 79ec9ef2f3937effc6af0f25d5c8
-rw-rw-r--   1 zope zope   19872 Jan  8  2008 8044fb8c6b68f4484716c9757bfa
-rw-rw-r--   1 zope zope   21168 Jan 10  2008 83074f956ab297c62b3c1a41574c
-rw-------   1 zope zope     784 Oct 16 08:56 8c72c89d9d2d86efc8fe1ed71cfe
-rw-rw-r--   1 zope zope     144 Dec 29  2007 99c5f01f3ed0dbdafed07ae38205
-rw-------   1 zope zope      16 Feb 12  2008 a2a02f4aa6ffc7f36103576c36e2
-rw-------   1 zope zope   26640 Sep 12 14:03 a2e55ba90942c56738e5c704c545
-rw-------   1 zope zope   95760 Jul  2  2008 a3c63b279c7edc4b4c797b8f332e
-rw-rw-r--   1 zope zope   77712 Jan  8  2008 ab93c709881a668b6f9f8a92c964
-rw-rw-r--   1 zope zope   16544 Jan  8  2008 aed5db4a22f81825165b86a99aa1
-rw-------   1 zope zope    5040 Aug 20  2008 c9e589908be1c448737a5fca842c
-rw-------   1 zope zope    3184 Jun 30  2008 cd45ed25b3ccea39d0091bbad4d3
-rw-rw-r--   1 zope zope  534976 Dec 29  2007 cd4a58908bb8b8a8c84c004e258d
-rw-rw-r--   1 zope zope     144 Jan 10  2008 d84cfb4389497fd679bbb7ce4077
-rw-------   1 zope zope    4800 May 30  2008 ed0c4a25941795c2d216355af1d4
-rw-------   1 zope zope 1362768 Oct  7 06:12 edec7b082feab6b5ab923094954c
-rw-------   1 zope zope   37392 Aug  4 17:48 ef358bc52a503dc9302526af3ce1
-rw-------   1 zope zope    1616 Jun 26  2008 f221db1d18f01ebe93caf29fe730
-rw-------   1 zope zope     400 Jan 20  2008 f4c43212930051c45da5163fba15
-rw-------   1 zope zope   10064 Aug 24  2008 f612a2284de15ad71b166132bb8b
-rw-rw-r--   1 zope zope  416704 Dec 28  2007 fbfca80d116b0d092aacd6712260
-rw-------   1 zope zope     640 Nov 26 16:13 fec8f41e8306758f238983948c64
-rw-------   1 zope zope   30224 Mar 16  2008 ffbda6d769a1bb8e73a8e966914f

Ideas?

Larry