[Gluster-devel] GlusterFS vs xfs w/ inode64 (was: Missing files?)

Liam Slusser lslusser at gmail.com
Tue Jul 28 19:20:36 UTC 2009


I worked on this issue this morning and could find nothing that would
indicate it wouldn't work.  I was down to 45 free inodes (says xfs_db) so i
brought down one of the nodes and applied the inode64 option to /etc/fstab
and remount the partition and restarted gluster.  Everything appears to be
working normally so i applied the same option to my other server, and again,
everything is working normally.
I'll let you know after we run with this for a few days but so far
everything is fine and working normally.  I'm on Centos 5.3 x86_64 btw.

An interesting note, after applying the inode64 option the "ifree" output
after running xfs_db didn't actually change but the filesystem is working
normally.  I found a bunch of posts on the interweb of people who had that
exact experience.

liam

On Tue, Jul 28, 2009 at 2:55 AM, Roger Torrentsgenerós <
rtorrents at flumotion.com> wrote:

>
> Any light on this issue?
>
> To sum up, I need to know if GlusterFS is capable of dealing with an xfs
> filesystem mounted with the inode64 option, or if it will be shortly.
>
> Sorry but it's an urgent matter to me.
>
> Thanks a lot.
>
> Roger
>
> On Thu, 2009-07-23 at 04:13 -0700, Liam Slusser wrote:
> >
> >
> > This brings up an interesting
> > question.  I just had a look on our large cluster (which is using xfs)
> and i've eaten up 98% of my free inodes.  I am mounting the filesystem
> without any options so i assume i am not using the inode64 option. (i dont
> believe it is default even on 64bit systems).
> >
> >
> > Is gluster giggy with using the inode64 mount option on xfs?  My
> understanding is as long as the binary is 64bit it shouldnt have any issue
> (im using 64bit gluster binary i compiled myself).
> >
> >
> > Anybody have some insight into this?
> >
> >
> > liam
> >
> > On Thu, Jul 23, 2009 at 2:59 AM, Matthias Saou
> > <thias at spam.spam.spam.spam.spam.spam.spam.egg.and.spam.freshrpms.net>
> > wrote:
> >         Hi,
> >
> >         Replying to myself with some more details : The servers are
> >         64bit
> >         (x86_64) whereas the clients are 32bit (ix86). It seems like
> >         this could
> >         be the cause of this problem...
> >
> >         http://oss.sgi.com/archives/xfs/2009-07/msg00044.html
> >
> >         But if the glusterfs client doesn't know about the original
> >         inodes of
> >         the files, then it should be possible to fix, right?
> >
> >         Matthias
> >
> >         Matthias Saou wrote :
> >
> >
> >         > Hi,
> >         >
> >         > (Note: I have access to the systems referenced in the
> >         initial post)
> >         >
> >         > I think I've found the problem. It's the filesystem, XFS,
> >         which has
> >         > been mounted with the "inode64" option, as it can't be
> >         mounted without
> >         > since it has been grown to 39TB. I've just checked this :
> >         >
> >         > # ls -1 -ai /file/data/cust | sort -n
> >         >
> >         > And the last few lines are like this :
> >         >
> >         > [...]
> >         >  2148235729 cust2
> >         >  2148236297 cust6
> >         >  2148236751 cust5
> >         >  2148236974 cust7
> >         >  2148237729 cust3
> >         >  2148239365 cust4
> >         >  2156210172 cust8
> >         > 61637541899 cust1
> >         > 96636784146 cust9
> >         >
> >         > Note that "cust1" here is the one where the problem has been
> >         seen
> >         > initially. I've just checked, and the "cust9" directory is
> >         affected in
> >         > the exact same way.
> >         >
> >         > So it seems like the glusterfs build being used has problems
> >         with 64bit
> >         > inodes. Is this a known limitation? Is it easy to fix or
> >         work around?
> >         >
> >         > Matthias
> >         >
> >         > Roger Torrentsgenerós wrote :
> >         >
> >         > >
> >         > > We have 2 servers, let's name them file01 and file02. They
> >         are synced
> >         > > very frequently, so we can assume contents are the same.
> >         Then we have
> >         > > lots of clients, everyone of each has two glusterfs
> >         mountings, one
> >         > > against every file server.
> >         > >
> >         > > Before you ask, let me say the clients are in a production
> >         environment,
> >         > > where I can't afford any downtime. To make the migration
> >         from glusterfs
> >         > > v1.3 to glusterfs v2.0 as smooth as possible, I recompiled
> >         the packages
> >         > > to run under glusterfs2 name. Servers are running two
> >         instances of the
> >         > > glusterfs daemon, and the old one is to be stopped when
> >         all the
> >         > > migration is complete. So you'll be seeing some glusterfs2
> >         and build
> >         > > dates that may not be normal, but you'll also see this has
> >         nothing to do
> >         > > with this matter.
> >         > >
> >         > > file01 server log:
> >         > >
> >         > >
> >
> ================================================================================
> >         > > Version      : glusterfs 2.0.1 built on May 26 2009
> >         05:11:51
> >         > > TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
> >         > > Starting Time: 2009-07-14 18:07:12
> >         > > Command line : /usr/sbin/glusterfsd2
> >         -p /var/run/glusterfsd2.pid
> >         > > PID          : 6337
> >         > > System name  : Linux
> >         > > Nodename     : file01
> >         > > Kernel Release : 2.6.18-128.1.14.el5
> >         > > Hardware Identifier: x86_64
> >         > >
> >         > > Given volfile:
> >         > >
> >
> +------------------------------------------------------------------------------+
> >         > >   1: # The data store directory to serve
> >         > >   2: volume filedata-ds
> >         > >   3:   type storage/posix
> >         > >   4:   option directory /file/data
> >         > >   5: end-volume
> >         > >   6:
> >         > >   7: # Make the data store read-only
> >         > >   8: volume filedata-readonly
> >         > >   9:   type testing/features/filter
> >         > > 10:   option read-only on
> >         > > 11:   subvolumes filedata-ds
> >         > > 12: end-volume
> >         > > 13:
> >         > > 14: # Optimize
> >         > > 15: volume filedata-iothreads
> >         > > 16:   type performance/io-threads
> >         > > 17:   option thread-count 64
> >         > > 18: #  option autoscaling on
> >         > > 19: #  option min-threads 16
> >         > > 20: #  option max-threads 256
> >         > > 21:   subvolumes filedata-readonly
> >         > > 22: end-volume
> >         > > 23:
> >         > > 24: # Add readahead feature
> >         > > 25: volume filedata
> >         > > 26:   type performance/read-ahead   # cache per file =
> >         (page-count x
> >         > > page-size)
> >         > > 27: #  option page-size 256kB        # 256KB is the
> >         default option ?
> >         > > 28: #  option page-count 8           # 16 is default
> >         option ?
> >         > > 29:   subvolumes filedata-iothreads
> >         > > 30: end-volume
> >         > > 31:
> >         > > 32: # Main server section
> >         > > 33: volume server
> >         > > 34:   type protocol/server
> >         > > 35:   option transport-type tcp
> >         > > 36:   option transport.socket.listen-port 6997
> >         > > 37:   subvolumes filedata
> >         > > 38:   option auth.addr.filedata.allow 192.168.128.* #
> >         streamers
> >         > > 39:   option verify-volfile-checksum off # don't have
> >         clients complain
> >         > > 40: end-volume
> >         > > 41:
> >         > >
> >         > >
> >
> +------------------------------------------------------------------------------+
> >         > > [2009-07-14 18:07:12] N [glusterfsd.c:1152:main]
> >         glusterfs: Successfully
> >         > > started
> >         > >
> >         > > file02 server log:
> >         > >
> >         > >
> >
> ================================================================================
> >         > > Version      : glusterfs 2.0.1 built on May 26 2009
> >         05:11:51
> >         > > TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
> >         > > Starting Time: 2009-06-28 08:42:13
> >         > > Command line : /usr/sbin/glusterfsd2
> >         -p /var/run/glusterfsd2.pid
> >         > > PID          : 5846
> >         > > System name  : Linux
> >         > > Nodename     : file02
> >         > > Kernel Release : 2.6.18-92.1.10.el5
> >         > > Hardware Identifier: x86_64
> >         > >
> >         > > Given volfile:
> >         > >
> >
> +------------------------------------------------------------------------------+
> >         > >   1: # The data store directory to serve
> >         > >   2: volume filedata-ds
> >         > >   3:   type storage/posix
> >         > >   4:   option directory /file/data
> >         > >   5: end-volume
> >         > >   6:
> >         > >   7: # Make the data store read-only
> >         > >   8: volume filedata-readonly
> >         > >   9:   type testing/features/filter
> >         > > 10:   option read-only on
> >         > > 11:   subvolumes filedata-ds
> >         > > 12: end-volume
> >         > > 13:
> >         > > 14: # Optimize
> >         > > 15: volume filedata-iothreads
> >         > > 16:   type performance/io-threads
> >         > > 17:   option thread-count 64
> >         > > 18: #  option autoscaling on
> >         > > 19: #  option min-threads 16
> >         > > 20: #  option max-threads 256
> >         > > 21:   subvolumes filedata-readonly
> >         > > 22: end-volume
> >         > > 23:
> >         > > 24: # Add readahead feature
> >         > > 25: volume filedata
> >         > > 26:   type performance/read-ahead   # cache per file =
> >         (page-count x
> >         > > page-size)
> >         > > 27: #  option page-size 256kB        # 256KB is the
> >         default option ?
> >         > > 28: #  option page-count 8           # 16 is default
> >         option ?
> >         > > 29:   subvolumes filedata-iothreads
> >         > > 30: end-volume
> >         > > 31:
> >         > > 32: # Main server section
> >         > > 33: volume server
> >         > > 34:   type protocol/server
> >         > > 35:   option transport-type tcp
> >         > > 36:   option transport.socket.listen-port 6997
> >         > > 37:   subvolumes filedata
> >         > > 38:   option auth.addr.filedata.allow 192.168.128.* #
> >         streamers
> >         > > 39:   option verify-volfile-checksum off # don't have
> >         clients complain
> >         > > 40: end-volume
> >         > > 41:
> >         > >
> >         > >
> >
> +------------------------------------------------------------------------------+
> >         > > [2009-06-28 08:42:13] N [glusterfsd.c:1152:main]
> >         glusterfs: Successfully
> >         > > started
> >         > >
> >         > > Now let's pick a random client, for example streamer013,
> >         and see its
> >         > > log:
> >         > >
> >         > >
> >
> ================================================================================
> >         > > Version      : glusterfs 2.0.1 built on May 26 2009
> >         05:23:52
> >         > > TLA Revision : 5c1d9108c1529a1155963cb1911f8870a674ab5b
> >         > > Starting Time: 2009-07-22 18:34:31
> >         > > Command line : /usr/sbin/glusterfs2 --log-level=NORMAL
> >         > > --volfile-server=file02.priv
> >         --volfile-server-port=6997 /mnt/file02
> >         > > PID          : 14519
> >         > > System name  : Linux
> >         > > Nodename     : streamer013
> >         > > Kernel Release : 2.6.18-92.1.10.el5PAE
> >         > > Hardware Identifier: i686
> >         > >
> >         > > Given volfile:
> >         > >
> >
> +------------------------------------------------------------------------------+
> >         > >   1: # filedata
> >         > >   2: volume filedata
> >         > >   3:   type protocol/client
> >         > >   4:   option transport-type tcp
> >         > >   5:   option remote-host file02.priv
> >         > >   6:   option remote-port 6997          # use non default
> >         to run in
> >         > > parallel
> >         > >   7:   option remote-subvolume filedata
> >         > >   8: end-volume
> >         > >   9:
> >         > > 10: # Add readahead feature
> >         > > 11: volume readahead
> >         > > 12:   type performance/read-ahead   # cache per file =
> >         (page-count x
> >         > > page-size)
> >         > > 13: #  option page-size 256kB        # 256KB is the
> >         default option ?
> >         > > 14: #  option page-count 2           # 16 is default
> >         option ?
> >         > > 15:   subvolumes filedata
> >         > > 16: end-volume
> >         > > 17:
> >         > > 18: # Add threads
> >         > > 19: volume iothreads
> >         > > 20:   type performance/io-threads
> >         > > 21:   option thread-count 8
> >         > > 22: #  option autoscaling on
> >         > > 23: #  option min-threads 16
> >         > > 24: #  option max-threads 256
> >         > > 25:   subvolumes readahead
> >         > > 26: end-volume
> >         > > 27:
> >         > > 28: # Add IO-Cache feature
> >         > > 29: volume iocache
> >         > > 30:   type performance/io-cache
> >         > > 31:   option cache-size 64MB        # default is 32MB (in
> >         1.3)
> >         > > 32:   option page-size 256KB        # 128KB is default
> >         option (in 1.3)
> >         > > 33:   subvolumes iothreads
> >         > > 34: end-volume
> >         > > 35:
> >         > >
> >         > >
> >
> +------------------------------------------------------------------------------+
> >         > > [2009-07-22 18:34:31] N [glusterfsd.c:1152:main]
> >         glusterfs: Successfully
> >         > > started
> >         > > [2009-07-22 18:34:31] N
> >         [client-protocol.c:5557:client_setvolume_cbk]
> >         > > filedata: Connected to 192.168.128.232:6997, attached to
> >         remote volume
> >         > > 'filedata'.
> >         > > [2009-07-22 18:34:31] N
> >         [client-protocol.c:5557:client_setvolume_cbk]
> >         > > filedata: Connected to 192.168.128.232:6997, attached to
> >         remote volume
> >         > > 'filedata'.
> >         > >
> >         > > The mountings seem ok:
> >         > >
> >         > > [root at streamer013 /]# mount
> >         > > /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
> >         > > proc on /proc type proc (rw)
> >         > > sysfs on /sys type sysfs (rw)
> >         > > devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> >         > > /dev/sda1 on /boot type ext3 (rw)
> >         > > tmpfs on /dev/shm type tmpfs (rw)
> >         > > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> >         > > glusterfs#file01.priv on /mnt/file01 type fuse
> >         > > (rw,max_read=131072,allow_other,default_permissions)
> >         > > glusterfs#file02.priv on /mnt/file02 type fuse
> >         > > (rw,max_read=131072,allow_other,default_permissions)
> >         > >
> >         > > They work:
> >         > >
> >         > > [root at streamer013 /]# ls /mnt/file01/
> >         > > cust
> >         > > [root at streamer013 /]# ls /mnt/file02/
> >         > > cust
> >         > >
> >         > > And they are seen by both servers:
> >         > >
> >         > > file01:
> >         > >
> >         > > [2009-07-22 18:34:19] N
> >         [server-helpers.c:723:server_connection_destroy]
> >         > > server: destroyed connection of streamer013.
> >         > >
> >         p4.bt.bcn.flumotion.net-14335-2009/07/22-18:34:13:210609-filedata
> >         > > [2009-07-22 18:34:31] N [server-protocol.c:7796:notify]
> >         server:
> >         > > 192.168.128.213:1017 disconnected
> >         > > [2009-07-22 18:34:31] N [server-protocol.c:7796:notify]
> >         server:
> >         > > 192.168.128.213:1018 disconnected
> >         > > [2009-07-22 18:34:31] N
> >         [server-protocol.c:7035:mop_setvolume] server:
> >         > > accepted client from 192.168.128.213:1017
> >         > > [2009-07-22 18:34:31] N
> >         [server-protocol.c:7035:mop_setvolume] server:
> >         > > accepted client from 192.168.128.213:1018
> >         > >
> >         > > file02:
> >         > >
> >         > > [2009-07-22 18:34:20] N
> >         [server-helpers.c:723:server_connection_destroy]
> >         > > server: destroyed connection of streamer013.
> >         > >
> >         p4.bt.bcn.flumotion.net-14379-2009/07/22-18:34:13:267495-filedata
> >         > > [2009-07-22 18:34:31] N [server-protocol.c:7796:notify]
> >         server:
> >         > > 192.168.128.213:1014 disconnected
> >         > > [2009-07-22 18:34:31] N [server-protocol.c:7796:notify]
> >         server:
> >         > > 192.168.128.213:1015 disconnected
> >         > > [2009-07-22 18:34:31] N
> >         [server-protocol.c:7035:mop_setvolume] server:
> >         > > accepted client from 192.168.128.213:1015
> >         > > [2009-07-22 18:34:31] N
> >         [server-protocol.c:7035:mop_setvolume] server:
> >         > > accepted client from 192.168.128.213:1014
> >         > >
> >         > > Now let's see the funny things. First, a content listing
> >         of a particular
> >         > > directory, locally from both servers:
> >         > >
> >         > > [root at file01 ~]# ls /file/data/cust/cust1
> >         > > configs  files  outgoing  reports
> >         > >
> >         > > [root at file02 ~]# ls /file/data/cust/cust1
> >         > > configs  files  outgoing  reports
> >         > >
> >         > > Now let's try to see the same from the client side:
> >         > >
> >         > > [root at streamer013 /]# ls /mnt/file01/cust/cust1
> >         > > ls: /mnt/file01/cust/cust1: No such file or directory
> >         > > [root at streamer013 /]# ls /mnt/file02/cust/cust1
> >         > > configs  files  outgoing  reports
> >         > >
> >         > > Oops :( And the client log says:
> >         > >
> >         > > [2009-07-22 18:41:22] W [fuse-bridge.c:1651:fuse_opendir]
> >         > > glusterfs-fuse: 64: OPENDIR (null) (fuse_loc_fill()
> >         failed)
> >         > >
> >         > > While none of the servers logs say anything.
> >         > >
> >         > > So files really exist in the servers, but the same client
> >         can see them
> >         > > in one of the filers but not in the other, although both
> >         are running
> >         > > exactly the same software. But there's more. It seems it
> >         only happens
> >         > > for certain directories (I can't show you the contents due
> >         to privacity,
> >         > > but I guess you'll figure it out):
> >         > >
> >         > > [root at streamer013 /]# ls /mnt/file01/cust/|wc -l
> >         > > 95
> >         > > [root at streamer013 /]# ls /mnt/file02/cust/|wc -l
> >         > > 95
> >         > > [root at streamer013 /]# for i in `ls /mnt/file01/cust/`; do
> >         > > ls /mnt/file01/cust/$i; done|grep such
> >         > > ls: /mnt/file01/cust/cust1: No such file or directory
> >         > > ls: /mnt/file01/cust/cust2: No such file or directory
> >         > > [root at streamer013 /]# for i in `ls /mnt/file02/cust/`; do
> >         > > ls /mnt/file02/cust/$i; done|grep such
> >         > > [root at streamer013 /]#
> >         > >
> >         > > And of course, our client log error twice:
> >         > >
> >         > > [2009-07-22 18:49:21] W [fuse-bridge.c:1651:fuse_opendir]
> >         > > glusterfs-fuse: 2119: OPENDIR (null) (fuse_loc_fill()
> >         failed)
> >         > > [2009-07-22 18:49:21] W [fuse-bridge.c:1651:fuse_opendir]
> >         > > glusterfs-fuse: 2376: OPENDIR (null) (fuse_loc_fill()
> >         failed)
> >         > >
> >         > >
> >         > > I hope having been clear enough this time. If you need
> >         more data just
> >         > > let me know and I'll see what I can do.
> >         > >
> >         > > And thanks again for your help.
> >         > >
> >         > > Roger
> >         > >
> >         > >
> >         > > On Wed, 2009-07-22 at 09:10 -0700, Anand Avati wrote:
> >         > > > > I have been witnessing some strange behaviour with my
> >         GlusterFS system.
> >         > > > > Fact is there are some files which exist and are
> >         completely accessible
> >         > > > > in the server, while they can't be accessed from a
> >         client, while other
> >         > > > > files do.
> >         > > > >
> >         > > > > To be sure, I copied the same files to another
> >         directory and I still was
> >         > > > > unable to see them from the client. To be sure it
> >         wasn't any kind of
> >         > > > > file permissions, selinux or whatever issue, I created
> >         a copy from a
> >         > > > > working directory, and still wasn't seen from the
> >         client. All I get is
> >         > > > > a:
> >         > > > >
> >         > > > > ls: .: No such file or directory
> >         > > > >
> >         > > > > And the client log says:
> >         > > > >
> >         > > > > [2009-07-22 14:04:18] W
> >         [fuse-bridge.c:1651:fuse_opendir]
> >         > > > > glusterfs-fuse: 104778: OPENDIR (null)
> >         (fuse_loc_fill() failed)
> >         > > > >
> >         > > > > While the server log says nothing.
> >         > > > >
> >         > > > > Funniest thing is the same client has another
> >         GlusterFS mount to another
> >         > > > > server, which has exactly the same contents as the
> >         first one, and this
> >         > > > > mount does work.
> >         > > > >
> >         > > > > Some data:
> >         > > > >
> >         > > > > [root at streamer001 /]# ls /mnt/file01/cust/cust1/
> >         > > > > ls: /mnt/file01/cust/cust1/: No such file or directory
> >         > > > >
> >         > > > > [root at streamer001 /]# ls /mnt/file02/cust/cust1/
> >         > > > > configs  files  outgoing  reports
> >         > > > >
> >         > > > > [root at streamer001 /]# mount
> >         > > > > /dev/mapper/VolGroup00-LogVol00 on / type ext3 (rw)
> >         > > > > proc on /proc type proc (rw)
> >         > > > > sysfs on /sys type sysfs (rw)
> >         > > > > devpts on /dev/pts type devpts (rw,gid=5,mode=620)
> >         > > > > /dev/sda1 on /boot type ext3 (rw)
> >         > > > > tmpfs on /dev/shm type tmpfs (rw)
> >         > > > > none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
> >         > > > > sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
> >         > > > > glusterfs#file01.priv on /mnt/file01 type fuse
> >         > > > > (rw,max_read=131072,allow_other,default_permissions)
> >         > > > > glusterfs#file02.priv on /mnt/file02 type fuse
> >         > > > > (rw,max_read=131072,allow_other,default_permissions)
> >         > > > >
> >         > > > > [root at file01 /]# ls /file/data/cust/cust1
> >         > > > > configs  files  outgoing  reports
> >         > > > >
> >         > > > > [root at file02 /]# ls /file/data/cust/cust1
> >         > > > > configs  files  outgoing  reports
> >         > > > >
> >         > > > > Any ideas?
> >         > > >
> >         > > > Can you please post all your client and server logs and
> >         volfiles? Are
> >         > > > you quite certain that this is not a result of some
> >         misconfiguration?
> >         > > >
> >         > > > Avati
> >         > >
> >         > >
> >         > > _______________________________________________
> >         > > Gluster-devel mailing list
> >         > > Gluster-devel at nongnu.org
> >         > > http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >         >
> >         >
> >
> >
> >         --
> >         Clean custom Red Hat Linux rpm packages :
> >         http://freshrpms.net/
> >         Fedora release 10 (Cambridge) - Linux kernel
> >
> >         2.6.27.25-170.2.72.fc10.x86_64 Load : 0.50 3.32 2.58
> >
> >
> >
> >         _______________________________________________
> >         Gluster-devel mailing list
> >         Gluster-devel at nongnu.org
> >         http://lists.nongnu.org/mailman/listinfo/gluster-devel
> >
> >
> >
> > _______________________________________________
> > Gluster-devel mailing list
> > Gluster-devel at nongnu.org
> > http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
>
> _______________________________________________
> Gluster-devel mailing list
> Gluster-devel at nongnu.org
> http://lists.nongnu.org/mailman/listinfo/gluster-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-devel/attachments/20090728/6d2e9230/attachment-0003.html>


More information about the Gluster-devel mailing list