[Gluster-users] glusterfs bdb backend problem

William Hanwoody hanwoody at gmail.com
Wed Mar 18 01:45:56 UTC 2009 

I cannot touch or cp files to glusterfs filesystem with bdb backend.

orion31 is glusterfs server, orion28 is glusterfs client.

-------------------------------------------------------------------------------------------

[root at orion28 ~]# mount -t glusterfs /etc/glusterfs/glusterfs-bdb.vol /mnt
[root at orion28 ~]# touch /mnt/a
touch: cannot touch `/mnt/a': No such file or directory
[root at orion28 ~]# ls
a  b
[root at orion28 ~]# cp a /mnt/a
cp: writing `/mnt/a': File descriptor in bad state
cp: closing `/mnt/a': File descriptor in bad state
-------------------------------------------------------------------------------------------

[root at orion28 ~]# tail /var/log/glusterfs/glusterfs.log
2009-03-18 09:31:21 E [fuse-bridge.c:539:fuse_attr_cbk] glusterfs-fuse: 13: 
UTIMENS() /a => -1 (Operation not permitted)
2009-03-18 09:32:17 E [fuse-bridge.c:1606:fuse_writev_cbk] glusterfs-fuse: 17: 
WRITE => -1 (File descriptor in bad state)
2009-03-18 09:32:17 E [fuse-bridge.c:924:fuse_err_cbk] glusterfs-fuse: 18: 
FLUSH() ERR => -1 (File descriptor in bad state)
-------------------------------------------------------------------------------------------

My glusterfs server log:
[root at orion31 glusterfs]# tail glusterfsd.log -n 500
.........................
2009-03-18 09:30:26 W [xlator.c:426:validate_xlator_volume_options] brick: 
option 'mandatory' is deprecated, preferred is 'mandatory-locks', continuing 
with
 correction
2009-03-18 09:30:26 W [glusterfsd.c:451:_log_if_option_is_invalid] bdb: 
option 'logdir' is not recognized
2009-03-18 09:30:26 W [glusterfsd.c:451:_log_if_option_is_invalid] bdb: 
option 'errfile' is not recognized
2009-03-18 09:30:26 W [glusterfsd.c:451:_log_if_option_is_invalid] bdb: 
option 'lru-limit' is not recognized
2009-03-18 09:30:26 W [glusterfsd.c:451:_log_if_option_is_invalid] bdb: 
option 'checkpoint-timeout' is not recognized
2009-03-18 09:30:26 W [glusterfsd.c:451:_log_if_option_is_invalid] bdb: 
option 'cache' is not recognized
2009-03-18 09:30:26 W [glusterfsd.c:451:_log_if_option_is_invalid] bdb: 
option 'directory' is not recognized
2009-03-18 09:30:26 N [glusterfsd.c:1134:main] glusterfs: Successfully started
2009-03-18 09:31:14 N [server-protocol.c:7295:mop_setvolume] server: accepted 
client from 10.55.22.28:1023
2009-03-18 09:31:14 N [server-protocol.c:7295:mop_setvolume] server: accepted 
client from 10.55.22.28:1022
2009-03-18 09:31:17 E [bdb.c:2750:bdb_readdir] bdb: invalid argument: bfd
2009-03-18 09:31:17 E [bdb.c:1498:bdb_releasedir] bdb: failed to extract fd 
data from fd=0x513880
2009-03-18 09:31:21 E [bdb.c:123:bdb_mknod] bdb: bdb_db_get() failed for 
path: /a
2009-03-18 09:31:21 E [bdb.c:2022:bdb_utimens] bdb: failed to lstat 
on /data1/bdb-export/a (Operation not permitted)
2009-03-18 09:32:17 E [bdb.c:789:bdb_writev] bdb: invalid argument: bfd
2009-03-18 09:32:17 E [bdb.c:868:bdb_flush] bdb: invalid argument: bfd
2009-03-18 09:32:17 E [bdb.c:868:bdb_flush] bdb: invalid argument: bfd
2009-03-18 09:32:17 E [bdb.c:891:bdb_release] bdb: failed to extract bdb 
specific information from fd:0x513a80
-------------------------------------------------------------------------------------------

My server vol file:
[root at orion32 bdb-export]# cat /etc/glusterfs/glusterfsd-bdb.vol
volume bdb
  type storage/bdb
  option directory /data1/bdb-export
  #option transaction off   # default is on
  option cache on          # default is off
  #option access-mode btree # default will be hash
  option checkpoint-timeout 10 # default is 30seconds
  #option file-mode 0644  # default is 0644
  #option dir-mode 0755   # default is 0755
  option lru-limit 200    # default is 100
  option errfile /var/log/glusterfs/bdberrlog # default is /dev/null?
  option logdir /var/log/glusterfs
end-volume

volume brick
  type features/locks
  option mandatory on          # enables mandatory locking on all files
  subvolumes bdb
end-volume

volume server
  type protocol/server
  option transport-type tcp                      # For TCP/IP transport
  #option transport.socket.listen-port 6996       # Default is 6996
  subvolumes brick
  option auth.addr.brick.allow *               # access to "brick" volume
end-volume
-------------------------------------------------------------------------------------------

My client vol:
[root at orion28 ~]# cat /etc/glusterfs/glusterfs-bdb.vol
volume orion31
  type protocol/client
  option transport-type tcp
  option remote-host orion31         # IP address of the remote brick

# option transport-timeout 30          # seconds to wait for a reply
                                       # from server for each request
  option remote-subvolume brick # name of the remote volume
end-volume
-------------------------------------------------------------------------------------------

My system envirement:

centos4.4 linux  x86_64
dkms-fuse-2.7.4-1.nodist.rf
fuse-2.7.4-1.el4.rf
glusterfs 2.0.0rc4
bdb 4.7.25

my bdb 4.7.25:
[root at orion32 bdb-export]# rpm -ql db
/usr/local/bdb4.7/lib64/libdb-4.7.so
/usr/local/bdb4.7/lib64/libdb_cxx-4.7.so

[root at orion32 bdb-export]#rpm -ql db-devel
/usr/local/bdb4.7/include/db.h
/usr/local/bdb4.7/include/db_cxx.h
/usr/local/bdb4.7/lib/libdb-4.7.a
/usr/local/bdb4.7/lib/libdb-4.7.la
/usr/local/bdb4.7/lib/libdb-4.so
/usr/local/bdb4.7/lib/libdb.a
/usr/local/bdb4.7/lib/libdb.so
/usr/local/bdb4.7/lib/libdb_cxx-4.7.a
/usr/local/bdb4.7/lib/libdb_cxx-4.7.la
/usr/local/bdb4.7/lib/libdb_cxx-4.so
/usr/local/bdb4.7/lib/libdb_cxx.a
/usr/local/bdb4.7/lib/libdb_cxx.so

[root at orion32 bdb-export]# ls -l /lib64/libdb-4.7.so
lrwxrwxrwx  1 root root 36 Mar 17 
16:59 /lib64/libdb-4.7.so -> /usr/local/bdb4.7/lib64/libdb-4.7.so
[root at orion32 bdb-export]# ls -l /usr/lib64/libdb-4.7.so
lrwxrwxrwx  1 root root 36 Mar 17 
16:59 /usr/lib64/libdb-4.7.so -> /usr/local/bdb4.7/lib64/libdb-4.7.so
[root at orion32 bdb-export]#  

for bdb's version in centos4 is 4.2, and difficult to upgrade to 4.7.25, then 
I installed bdb4.7 in /usr/local/bdb/4.7 

and compile glusterfs 2.0.0rc4 with libdb-4.7 through modifing 
xlators/storage/bdb/src/Makefile.am/Makefile.in.
and link /usr/include/db.h to /usr/local/bdb4.7/include/db.h

-------------------------------------------------------------------------------------------

More information about the Gluster-users mailing list