[Gluster-users] URGENT: Update issues from 3.6.6 to 3.10.2 Accessing files via samba come up with permission denied

Wed May 31 16:59:20 UTC 2017

The vfs gluster plugin for samba is linked against libglusterfs.so.0
which is provided by glusterfs-libs-3.10.2-1.el7.x86_64, please see
below.

# ldd /usr/lib64/samba/vfs/glusterfs.so | grep glus
       libglusterfs.so.0 => /lib64/libglusterfs.so.0 (0x00007f61da858000)

]# yum provides /lib64/libglusterfs.so.0
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: centos.vwtonline.net
* extras: mirror.cs.vt.edu
* updates: centosv.centos.org
glusterfs-libs-3.10.2-1.el7.x86_64 : GlusterFS common libraries
Repo        : @centos-gluster310
Matched from:
Filename    : /lib64/libglusterfs.so.0

On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <dijuremo at gmail.com> wrote:
> Samba is running in the same machine as glusterd. The machines were
> rebooted after the upgrades and samba has been restarted a few times.
>
> # rpm -qa | grep gluster
> glusterfs-client-xlators-3.10.2-1.el7.x86_64
> glusterfs-server-3.10.2-1.el7.x86_64
> glusterfs-api-3.10.2-1.el7.x86_64
> glusterfs-3.10.2-1.el7.x86_64
> glusterfs-cli-3.10.2-1.el7.x86_64
> centos-release-gluster310-1.0-1.el7.centos.noarch
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> glusterfs-fuse-3.10.2-1.el7.x86_64
> glusterfs-libs-3.10.2-1.el7.x86_64
> glusterfs-rdma-3.10.2-1.el7.x86_64
>
> # rpm -qa | grep samba
> samba-common-libs-4.4.4-14.el7_3.x86_64
> samba-common-tools-4.4.4-14.el7_3.x86_64
> samba-libs-4.4.4-14.el7_3.x86_64
> samba-4.4.4-14.el7_3.x86_64
> samba-client-libs-4.4.4-14.el7_3.x86_64
> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
> samba-common-4.4.4-14.el7_3.noarch
>
> # cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
>
> I also raised the op version.
>
> # gluster volume get all cluster.op-version
> Option                                  Value
> ------                                  -----
> cluster.op-version                      31000
>
> # gluster volume get all cluster.max-op-version
> Option                                  Value
> ------                                  -----
> cluster.max-op-version                  31000
>
> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <rtalur at redhat.com> wrote:
>> Also, please attach your smb.conf. You can directly attach in the list
>> and need not have a google drive link.
>>
>>
>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur <rtalur at redhat.com> wrote:
>>> Diego,
>>>
>>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did
>>> not restart smb after upgrading gluster(if samba is on same machine as
>>> Gluster) or if you forgot to update Gluster client packages on Samba
>>> node?
>>>
>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina <dijuremo at gmail.com> wrote:
>>>> Please download the log file from this link:
>>>>
>>>> https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU
>>>>
>>>> Let me know if you need any other log files.
>>>>
>>>> Diego
>>>>
>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur <rtalur at redhat.com> wrote:
>>>>> If possible please share the glusterfs-* log files from /var/log/samba.
>>>>>
>>>>> This might be because of cluster.lookup-optimize. Adding Poornima and
>>>>> Raghavendra Gowdappa to help with this.
>>>>>
>>>>>
>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina <dijuremo at gmail.com> wrote:
>>>>>> This is a bit puzzling, not sure what difference it would make, but:
>>>>>>
>>>>>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt
>>>>>> Revit opens and shows a window that says access denied.
>>>>>>
>>>>>> 2. Rename file, i.e from windows explorer right click, rename to
>>>>>> MyRevitFile2.rvt
>>>>>>
>>>>>> 3. Without opening the file, rename file back to the original, i.e
>>>>>> MyRevitFile.rvt
>>>>>>
>>>>>> 4. Double click on file and now it will open just fine without the
>>>>>> Access Denied error.
>>>>>>
>>>>>> Any explanation for this? Could the rename operation be forcing or
>>>>>> updating some attributes that then allow the file to open?
>>>>>>
>>>>>> Diego
>>>>>>
>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina <dijuremo at gmail.com> wrote:
>>>>>>> This is what I see in the logs set from smb.conf via line ->
>>>>>>> glusterfs:logfile = /var/log/samba/glusterfs-projects.log
>>>>>>>
>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]
>>>>>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL
>>>>>>> (a97bc9bb-68cf-4a69-aef7-39766b323c14) [Invalid argument]
>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]
>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-0:
>>>>>>> remote operation failed. Path:
>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>>>>>> a directory]
>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]
>>>>>>> [client-rpc-fops.c:1100:client3_3_getxattr_cbk] 0-export-client-1:
>>>>>>> remote operation failed. Path:
>>>>>>> /projects/INACTIVE/WESTCOAST/Automotive/Acura/AS-Acura of Richmond/02
>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-ArchModel.rvt (a97bc9bb-68cf-4a69-
>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:desktop.ini [Not
>>>>>>> a directory]
>>>>>>>
>>>>>>>
>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina <dijuremo at gmail.com> wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Over the weekend we updated a two server glusterfs 3.6.6 install to
>>>>>>>> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I
>>>>>>>> enabled several of the newer caching features from gluster 3.9 for
>>>>>>>> small file performance and samba, and we now seem to have some issues
>>>>>>>> with accessing files from glusterfs. When users try to access some
>>>>>>>> files, they get a Permission denied message. This seems to be only via
>>>>>>>> samba as I am able to su - username and then do strings on the file.
>>>>>>>>
>>>>>>>> [root at ysmha02 gluster-backups]# rpm -qa | grep gluster
>>>>>>>> glusterfs-client-xlators-3.10.2-1.el7.x86_64
>>>>>>>> glusterfs-server-3.10.2-1.el7.x86_64
>>>>>>>> glusterfs-api-3.10.2-1.el7.x86_64
>>>>>>>> glusterfs-3.10.2-1.el7.x86_64
>>>>>>>> glusterfs-cli-3.10.2-1.el7.x86_64
>>>>>>>> centos-release-gluster310-1.0-1.el7.centos.noarch
>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>> glusterfs-fuse-3.10.2-1.el7.x86_64
>>>>>>>> glusterfs-libs-3.10.2-1.el7.x86_64
>>>>>>>> glusterfs-rdma-3.10.2-1.el7.x86_64
>>>>>>>>
>>>>>>>> [root at ysmha02 gluster-backups]# rpm -qa | grep samba
>>>>>>>> samba-common-libs-4.4.4-14.el7_3.x86_64
>>>>>>>> samba-common-tools-4.4.4-14.el7_3.x86_64
>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_64
>>>>>>>> samba-4.4.4-14.el7_3.x86_64
>>>>>>>> samba-client-libs-4.4.4-14.el7_3.x86_64
>>>>>>>> samba-vfs-glusterfs-4.4.4-14.el7_3.x86_64
>>>>>>>> samba-common-4.4.4-14.el7_3.noarch
>>>>>>>>
>>>>>>>> On the samba logs for the machine I notice something weird, samba
>>>>>>>> seems to be trying to stat the file we are trying as a directory to
>>>>>>>> see if it contains desktop.ini:
>>>>>>>>
>>>>>>>> [2017/05/30 10:13:07.297026,  0]
>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>>>>>>>> Not a directory
>>>>>>>> [2017/05/30 10:13:07.298155,  0]
>>>>>>>> ../source3/modules/vfs_glusterfs.c:870(vfs_gluster_stat)
>>>>>>>>  glfs_stat(ACTIVE/Automotive/FORD/AN - Ford East/02
>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/desktop.ini) failed:
>>>>>>>> Not a directory
>>>>>>>>
>>>>>>>> This seems to be happening only with files with the .rvt extension.
>>>>>>>> Though these files are usually larger in size vs other smaller excel,
>>>>>>>> power point, etc files.
>>>>>>>>
>>>>>>>> Here are the complete options for the volume:
>>>>>>>>
>>>>>>>> https://pastebin.com/ZH2vMsMN
>>>>>>>>
>>>>>>>> I turned off performance.cache-samba-metadata again to see if that
>>>>>>>> would help, but seems it does not help.
>>>>>>>>
>>>>>>>> I really appreciate any help with this.
>>>>>>>>
>>>>>>>> DIego
>>>>>> _______________________________________________
>>>>>> Gluster-users mailing list
>>>>>> Gluster-users at gluster.org
>>>>>> http://lists.gluster.org/mailman/listinfo/gluster-users