[Gluster-users] Advice for building samba-glusterfs-vfs

Dan Mons dmons at cuttingedge.com.au
Fri Oct 11 04:50:56 UTC 2013 

Following up on this:

* GlusterFS 3.4.1 solves the problem of renaming over CIFS from a
Windows client (via Samba3 and vfs_glusterfs/libgfapi).  Happy days!

* I still see 4 lines of this sort of thing for each file read in
/var/log/glusterfs/bricks/*.log:

"[2013-10-11 04:01:37.554826] E [posix.c:2668:posix_getxattr]
0-prodbackup-posix: getxattr failed on /gback_brick0/vfx:
system.posix_acl_default (No data available)"

I have the following Gluster options set:

diagnostics.client-log-level: ERROR
diagnostics.brick-log-level: ERROR

When Samba3 is configured to read from a FUSE mount instead (which has
been configured at mount time with "-o log-level=CRITICAL"), the
errors do no appear in the brick log.

Next up is trying to find a way to tell Samba not to pass on getxattr
when vfs_glusterfs is used.  Stacking VFS module "xattr_tdb" has no
effect with respect to this error (xattr_tdb works, but the errors
still fill the brick log).

-Dan

On 4 October 2013 09:21, Dan Mons <dmons at cuttingedge.com.au> wrote:
> Thanks for the pointers Dave.   That got me to compiling the module.
>
> I'm still using the RPM-installed GlusterFS and Samba, but have
> compiled/installed matching source versions into a --prefix=/localdev
> folder and used that to build the samba-glusterfs-vfs module with
> relevant paths during the "./configure" step.
>
> The module was copied to /usr/lib64/samba/vfs, vfs_glusterfs.so was
> symlinked to glusterfs.so, and appears to be working.
>
> Relevant smb.conf
>
> [sharename]
>         vfs object = glusterfs
>         glusterfs:volume = volumename
>         path = /
>
> Testing so far has been much faster for read/browse operations, so
> that much is good.
>
> However, bugs that already make it unusable for me at the moment:
>
> * renaming files and folders from a Windows host gives a "does not
> exist" error in the Explorer GUI, and a "duplicate filename exists"
> error from cmd.exe.
>
> * Brick logs fill up with 4 lines of "getxattr failed" messages for
> each file or folder read, and this is with all GlusterFS error log
> levels already set to "critical".  With a network full of users, I'm
> going to fill disks with logs in minutes.  I had seen this reported in
> 3.3.1 and marked as fixed in 3.4.0, but it appears to still be around.
>  I have XFS formatted disk backing each brick with 512 byte inodes and
> default mount options.  I don't wish to turn on user_xattr, as I want
> to just discard any of that sort of data coming from Mac and Windows
> hosts, as it's unnecessary for our production data.  But at the same
> time, I don't want GlusterFS telling me it's an error when it's not
> there (INFO level would suffice, so I can ignore that in my logs).
>
> -Dan
>
>
> On 3 October 2013 10:59, David Gibbons <david.c.gibbons at gmail.com> wrote:
>> Dan,
>>
>> I had the same trouble yesterday. As it happens, I created a doc to help
>> script installs of future nodes. I did not snip out the portions that apply
>> to what you're looking for, but the script below works for me.
>>
>> The biggest issue was that some modules were apparently installed in lib and
>> the build process was looking for them in lib64. In any event, the vfs
>> module builds, installs and runs cleanly after this.
>>
>> ** The big win for me was finding this command, that allowed me to figure
>> out where it was looking for modules in the wrong lib directory.
>> ldd /usr/local/samba/lib/vfs/glusterfs.so
>>
>> I'm sure there is any easier way to to do this :)...
>>
>> Cheers,
>> Dave
>>
>>
>> --
>>
>> #!/bin/bash
>>
>> yum groupinstall "Development Tools"
>>
>> yum install git openssl-devel wget
>>
>> yum install libtalloc libtdb
>>
>>
>> # set up gluster
>>
>> cd /usr/src && git clone https://github.com/gluster/glusterfs.git
>>
>> cd /usr/src/glusterfs && ./autogen.sh && ./configure && make
>>
>> make install
>>
>>
>> # set up samba 3.6.9
>>
>> cd /usr/src && wget http://ftp.samba.org/pub/samba/stable/samba-3.6.9.tar.gz
>> && tar -zxvf samba-3.6.9.tar.gz
>>
>> cd /usr/src/samba-3.6.9/source3 && ./configure && make
>>
>> make install
>>
>>
>> ln -s /usr/local/samba/lib/libwbclient.so.0 /usr/lib64/libwbclient.so.0
>>
>>
>> # then install the RPM samba version
>>
>> yum install samba
>>
>>
>> # set up vfs_glusterfs
>>
>> cd /usr/src && git clone
>> git://forge.gluster.org/samba-glusterfs/samba-glusterfs-vfs.git
>>
>>
>> ln -s /usr/local/include/glusterfs /usr/include/glusterfs
>>
>>
>> cd /usr/src/samba-vfs/glusterfs
>>
>> ./configure --with-samba-source=/usr/src/samba-3.6.9/source3
>>
>>
>> ln -s /usr/local/samba/lib/vfs/glusterfs.so
>> /usr/lib64/samba/vfs/glusterfs.so
>>
>>
>> # link the other modules
>>
>> ln -s /usr/local/lib/libgfapi.so /usr/lib64/
>>
>> ln -s /usr/local/lib/libgfapi.so.0 /usr/lib64/
>>
>> ln -s /usr/local/lib/libgfapi.la /usr/lib64/
>>
>> ln -s /usr/local/lib/libglusterfs.la /usr/lib64/
>>
>> ln -s /usr/local/lib/libglusterfs.so /usr/lib64/
>>
>> ln -s /usr/local/lib/libglusterfs.so.0 /usr/lib64/
>>
>> ln -s /usr/local/lib/libglusterfs.so.0.0.0 /usr/lib64/
>>
>> EOF
>>
>>
>> On Tue, Oct 1, 2013 at 10:38 PM, Dan Mons <dmons at cuttingedge.com.au> wrote:
>>>
>>> Hi folks,
>>>
>>> I've got CentOS6.4 with Samba 3.6.9 installed from the standard CentOS
>>> repos via yum.  I also have GlusterFS 3.4.0 GA installed from RPMs
>>> direct from gluster.org.
>>>
>>> I'm trying to build the glusterfs VFS module for Samba to take
>>> advantage of libgfapi for our Windows users, and migrate them off the
>>> current Samba-on-FUSE setup we have currently.
>>>
>>> I've downloaded the appropriate source trees for all projects
>>> (GlusterFS from gluster.org, Samba from the matching CentOS6 SRPM, and
>>> samba-glusterfs-vfs from the git repo), but am facing troubles early
>>> on just finding appropriate headers.
>>>
>>> [root at bne-gback000 samba-glusterfs-vfs]# find
>>> /usr/local/src/glusterfs-3.4.0 -type f -name glfs.h
>>> /usr/local/src/glusterfs-3.4.0/api/src/glfs.h
>>>
>>> [root at bne-gback000 samba-glusterfs-vfs]# ./configure
>>> --with-glusterfs=/usr/local/src/glusterfs-3.4.0
>>> *snip*
>>> checking api/glfs.h usability... no
>>> checking api/glfs.h presence... no
>>> checking for api/glfs.h... no
>>> Cannot find api/glfs.h. Please specify --with-glusterfs=dir if necessary
>>>
>>> If I install glusterfs-api-devel-3.4.0-8.el6.x86_64.rpm, I need to
>>> copy /usr/include/glusterfs/api/glfs.h to /usr/include for it to be
>>> found (even using --with-glusterfs= doesn't work), and then I get
>>> further errors about not being able to link to glfs_init:
>>>
>>> [root at bne-gback000 samba-glusterfs-vfs]# rpm -ivh
>>> /tmp/glusterfs-api-devel-3.4.0-8.el6.x86_64.rpm
>>> [root at bne-gback000 samba-glusterfs-vfs]# cp
>>> /usr/include/glusterfs/api/glfs.h /usr/include/
>>> [root at bne-gback000 samba-glusterfs-vfs]# ./configure
>>> *snip*
>>> checking api/glfs.h usability... yes
>>> checking api/glfs.h presence... yes
>>> checking for api/glfs.h... yes
>>> checking for glfs_init... no
>>> Cannot link to gfapi (glfs_init). Please specify --with-glusterfs=dir
>>> if necessary
>>>
>>> If anyone can point me in the right direction, that would be greatly
>>> appreciated.
>>>
>>> Cheers,
>>>
>>> -Dan
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>
>
>
> --
> Dan Mons
> R&D SysAdmin
> Cutting Edge
> http://cuttingedge.com.au



-- 
Dan Mons
R&D SysAdmin
Cutting Edge
http://cuttingedge.com.au

More information about the Gluster-users mailing list