[Gluster-devel] GlusterFS vs xfs w/ inode64 (was: Missing files?)

[Liam Slusser]

Matthias,

Just to run a few more tests....

Centos 5.3 x86_64 on all systems in my test

I just created 1000 directories with a file in each directory and everything
worked fine.  All the directories shows up on each node and on the client.

[root at store01 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'
icount = 37786752
ifree = 1133409
[root at store02 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'
icount = 36526592
ifree = 269

[root at client testliam]# for i in `seq 1 1000`; do mkdir $i; done
[root at client testliam]# for i in `seq 1 1000`; do echo "woohoo" >
$i/$i.myfileyay; done
[root at client testliam]# ls |wc -l
1000
[root at client testliam]# ls -R | wc -l
4001

(after the test)

[root at store01 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'
icount = 37786752
ifree = 1133409
[root at store02 /]# xfs_db -r -c sb -c p /dev/sdb1| egrep 'ifree|icount'
icount = 36526592
ifree = 269

[root at server01 testliam]# ls | wc -l
1000
[root at server01 testliam]# ls -R | wc -l
4001
[root at server02 testliam]# ls | wc -l
1000
[root at server02 testliam]# ls -R | wc -l
4001

The ifree number didnt change after i ran the tests, all the files look to
be fine and intact - so as far as i can tell everything is working fine.

Did your tests show something else?

liam


On Tue, Jul 28, 2009 at 1:14 PM, Matthias Saou <
thias at spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net> wrote:

> Liam Slusser <lslusser at gmail.com> wrote:
>
> > I worked on this issue this morning and could find nothing that would
> > indicate it wouldn't work.  I was down to 45 free inodes (says
> > xfs_db) so i brought down one of the nodes and applied the inode64
> > option to /etc/fstab and remount the partition and restarted
> > gluster.  Everything appears to be working normally so i applied the
> > same option to my other server, and again, everything is working
> > normally. I'll let you know after we run with this for a few days but
> > so far everything is fine and working normally.  I'm on Centos 5.3
> > x86_64 btw.
> >
> > An interesting note, after applying the inode64 option the "ifree"
> > output after running xfs_db didn't actually change but the filesystem
> > is working normally.  I found a bunch of posts on the interweb of
> > people who had that exact experience.
>
> It's not mounting with the inode64 option which is an issue, it's once
> you get files or directories allocated with inodes > 32bits from our
> experience. So what you need to do is test with files or directories
> created once the filesystem has been mounted with the inode64 options
> and do have their inode beyond the 32bit limit.
>
> Here's an example of what we have on the server :
> [...]
>  3221490663 drwxrwsr-x  5 flumotion file   46 Mar 11 19:07 cust5
>  3221491044 drwxrwsr-x  6 flumotion file   56 May 12 12:27 cust1
>  3221495387 drwxrwsr-x  9 flumotion file  109 Mar 26  2008 cust2
>  3221500135 drwxrwsr-x  5 flumotion file   46 May 22 11:36 cust8
>  3221500510 drwxrwsr-x  3 flumotion file   21 Jan  2  2008 cust23
>  3221500956 drwxrwsr-x  5 flumotion file   46 Jun 25 16:23 cust7
> 72107183169 drwxrwsr-x  3 flumotion file   26 Jul 10 13:38 cust4
> 98784439192 drwxrwsr-x  3 flumotion file   29 Jul  2 16:55 cust12
>
> The last two directories are the ones which aren't accessible from the
> glusterfs clients.
>
> So in your case, if you had only 45 free inodes left, you should be
> able to create 46 directories and get the 46th reproduce the problems.
>
> Matthias
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090728/ee429d85/attachment-0003.html>