[Gluster-devel] GlusterFS Root

Sun Nov 23 01:55:33 UTC 2008

I'm trying to add GlusterFS support to the Open Shared Root project, and 
I'm hitting a problem that is a bit difficult to debug. GlusterFS works 
fine in the normal full-fat environment (RHEL5), but I can't seem to get 
it to mount the volume in the init-root environment.

Things I have checked so far:

All the libraries and dependencies that glusterfs package lists are 
included in the initroot:

# rpm -q --requires glusterfs
/bin/sh
/bin/sh
config(glusterfs) = 1.3.12-1
libc.so.6()(64bit)
libc.so.6(GLIBC_2.2.5)(64bit)
libc.so.6(GLIBC_2.3)(64bit)
libc.so.6(GLIBC_2.3.2)(64bit)
libc.so.6(GLIBC_2.3.3)(64bit)
libdl.so.2()(64bit)
libdl.so.2(GLIBC_2.2.5)(64bit)
libfuse.so.2()(64bit)
libfuse.so.2(FUSE_2.4)(64bit)
libfuse.so.2(FUSE_2.5)(64bit)
libfuse.so.2(FUSE_2.6)(64bit)
libglusterfs.so.0()(64bit)
libibverbs.so.1()(64bit)
libibverbs.so.1(IBVERBS_1.0)(64bit)
libibverbs.so.1(IBVERBS_1.1)(64bit)
libpthread.so.0()(64bit)
libpthread.so.0(GLIBC_2.2.5)(64bit)
libpthread.so.0(GLIBC_2.3.2)(64bit)
rpmlib(CompressedFileNames) <= 3.0.4-1
rpmlib(PayloadFilesHavePrefix) <= 4.0-1
rtld(GNU_HASH)

The fuse driver is loaded and the device node is there:
# ls -la dev/fuse
crw------- 1 root root 10, 229 Nov 22 22:59 dev/fuse

There are no errors reported on the console.

Here is what the debug level log reports:
# cat glusterfs.log
2008-11-23 01:34:16 D [glusterfs.c:167:get_spec_fp] glusterfs: loading 
spec from /etc/glusterfs/root.vol
2008-11-23 01:34:16 D [spec.y:107:new_section] parser: New node for 'root2'
2008-11-23 01:34:16 D [xlator.c:115:xlator_set_type] xlator: attempt to 
load file /usr/lib64/glusterfs/1.3.12/xlator/protocol/client.so
2008-11-23 01:34:16 D [spec.y:127:section_type] parser: 
Type:root2:protocol/client
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:root2:transport-type:tcp/client
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:root2:remote-host:192.168.10.2
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:root2:remote-subvolume:root2
2008-11-23 01:34:16 D [spec.y:209:section_end] parser: end:root2
2008-11-23 01:34:16 D [spec.y:107:new_section] parser: New node for 
'root-store'
2008-11-23 01:34:16 D [xlator.c:115:xlator_set_type] xlator: attempt to 
load file /usr/lib64/glusterfs/1.3.12/xlator/storage/posix.so
2008-11-23 01:34:16 D [spec.y:127:section_type] parser: 
Type:root-store:storage/posix
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:root-store:directory:/mnt/tmproot/gluster/root
2008-11-23 01:34:16 D [spec.y:209:section_end] parser: end:root-store
2008-11-23 01:34:16 D [spec.y:107:new_section] parser: New node for 'root1'
2008-11-23 01:34:16 D [xlator.c:115:xlator_set_type] xlator: attempt to 
load file /usr/lib64/glusterfs/1.3.12/xlator/features/posix-locks.so
2008-11-23 01:34:16 D [xlator.c:145:xlator_set_type] xlator: 
dlsym(notify) on 
/usr/lib64/glusterfs/1.3.12/xlator/features/posix-locks.so: undefined 
symbol: notify -- neglecting
2008-11-23 01:34:16 D [spec.y:127:section_type] parser: 
Type:root1:features/posix-locks
2008-11-23 01:34:16 D [spec.y:196:section_sub] parser: 
child:root1->root-store
2008-11-23 01:34:16 D [spec.y:209:section_end] parser: end:root1
2008-11-23 01:34:16 D [spec.y:107:new_section] parser: New node for 'server'
2008-11-23 01:34:16 D [xlator.c:115:xlator_set_type] xlator: attempt to 
load file /usr/lib64/glusterfs/1.3.12/xlator/protocol/server.so
2008-11-23 01:34:16 D [spec.y:127:section_type] parser: 
Type:server:protocol/server
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:server:transport-type:tcp/server
2008-11-23 01:34:16 D [spec.y:196:section_sub] parser: child:server->root1
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:server:auth.ip.root1.allow:127.0.0.1,192.168.*
2008-11-23 01:34:16 D [spec.y:209:section_end] parser: end:server
2008-11-23 01:34:16 D [spec.y:107:new_section] parser: New node for 'root'
2008-11-23 01:34:16 D [xlator.c:115:xlator_set_type] xlator: attempt to 
load file /usr/lib64/glusterfs/1.3.12/xlator/cluster/afr.so
2008-11-23 01:34:16 D [spec.y:127:section_type] parser: 
Type:root:cluster/afr
2008-11-23 01:34:16 D [spec.y:196:section_sub] parser: child:root->root1
2008-11-23 01:34:16 D [spec.y:196:section_sub] parser: child:root->root2
2008-11-23 01:34:16 D [spec.y:141:section_option] parser: 
Option:root:read-subvolume:root1
2008-11-23 01:34:16 D [spec.y:209:section_end] parser: end:root
2008-11-23 01:34:16 D [glusterfs.c:128:fuse_graph] glusterfs: setting 
option mount-point to /mnt/newroot
2008-11-23 01:34:16 D [xlator.c:115:xlator_set_type] xlator: attempt to 
load file /usr/lib64/glusterfs/1.3.12/xlator/mount/fuse.so
2008-11-23 01:34:16 E [fuse-bridge.c:2699:init] glusterfs-fuse: 
fuse_mount failed (Inappropriate ioctl for device)

2008-11-23 01:34:16 E [glusterfs.c:547:main] glusterfs: Initializing 
FUSE failed

And here is the root volume spec:
# cat /etc/glusterfs/root.vol
volume root2
         type protocol/client
         option transport-type tcp/client
         #option transport-type socket
         #option address-family inet
         option remote-host 192.168.10.2
         option remote-subvolume root2
end-volume

volume root-store
         type storage/posix
         option directory /mnt/tmproot/gluster/root
end-volume

volume root1
         type features/posix-locks
         subvolumes root-store
end-volume

volume server
         type protocol/server
         option transport-type tcp/server
         #option transport-type socket
         #option address-family inet
         subvolumes root1
         option auth.ip.root1.allow 127.0.0.1,192.168.*
         #option auth.addr.root1.allow 127.0.0.1,192.168.*
end-volume

volume root
         type cluster/afr
         subvolumes root1 root2
         option read-subvolume root1
end-volume

Any suggestions on where I should even begin to look for the cause of 
this problem? I tested it in single-user mode, and it seems to mount OK 
there, so it has that feel of a missing dependency to it, but I can't 
think of what it could be.

I can provide an image of the kernel/initrd I'm using which could be 
dropped into a virtual machine for testing, if it'd help (I'm testing it 
on a VM myself).

Many thanks.

Gordan