<div dir="auto"><div><br><div class="gmail_extra"><br><div class="gmail_quote">On 03-Jun-2017 3:27 AM, "Diego Remolina" <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br type="attribution"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br>
<br>
Is there anything else we could do to check on this problem and try to<br>
fix it? The issue is definitively related to either the samba vfs<br>
gluster plugin or gluster itself. I am not sure how to pin it down<br>
futher.<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">I don't think it is vfs plugin because you haven't updated samba packages and vfs plugin is same as before. </div><div dir="auto"><br></div><div dir="auto">It is either gfapi in Gluster or one of the lower xlators. </div><div dir="auto"><br></div><div dir="auto">Changing to fuse based shares might make the problem go away and can be used as a workaround. </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br><br>
I went ahead and created a new share in the samba server which is on a<br>
local filesystem where the OS is installed, not part of gluster:<br>
<br>
# mount | grep home<br>
]# ls -ld /home<br>
drwxr-xr-x. 14 root root 4096 Jun 2 17:21 /home<br>
<br>
This shows this is not mounted from anywhere. so I created a folder<br>
and shared it:<br>
<br>
]# ls -ld /home/rvtsharetest/<br>
drwxrwx---. 3 dijuremo Staff 95 Jun 2 17:31 /home/rvtsharetest/<br>
<br>
]# tail -7 /etc/samba/smb.conf<br>
[rvtsharetest]<br>
path=/home/rvtsharetest<br>
browseable = yes<br>
write list = @Staff,dijuremo<br>
guest ok = yes<br>
create mask = 664<br>
directory mask = 775<br>
<br>
When accessing Revit files in this new share the problem is *not* observed.<br>
<br>
When accessing Revit files on any of the samba shares that use the vfs<br>
gluster plugin and are stored in the gluster volume, we see the<br>
problems.<br>
<br>
Further analysis of the issue is even more disconcerting. As you may<br>
remember, I have found a workaround about renaming the file and back<br>
to the original, where things work. Here is where it gets more<br>
interesting. This seems to be workstation dependent, not user<br>
dependent.<br>
<br>
1. User1 logs into PC1 and tries to access the file and gets the error<br>
message from Revit "ACCESS DENIED".<br>
<br>
2. User1 uses windows explorer to go to the file location (we tried<br>
doing this from the server itself on the command line and it did not<br>
change anything, i.e. su - User1 then mv command). User1 renames the<br>
file, Revit.rvt -> Revit2.rvt, and clicks away for the rename to take<br>
place. User1 immediately renames the file to the original, Revit2.rvt<br>
-> Revit.rvt<br>
<br>
3. User1 opens the file and everything works properly.<br>
<br>
4. User2 logs into PC1 and tries to open the file. The file opens and<br>
works properly.<br>
<br>
5. User2 logs into PC2 and tries to open the file, the problem comes<br>
up. User2 uses the rename trick and this fixes the problem on PC2.<br>
Even if User1 now comes to use PC2, User1 will have no problems with<br>
the file.<br>
<br>
6. User1 now goes to PC3 where nobody has used the file rename trick,<br>
and experiences the problem. Only solution is to play the rename trick<br>
again in PC3.<br>
<br>
So it seems you have to play the rename trick at least one per<br>
workstation and that "fixes" the issue for any user who logs into that<br>
workstation.<br>
<br>
What other suggestions do you have? What debugging can I do next? I am<br>
planning once everyone leaves the office today on changing the share<br>
to bypass vfs gluster plugin and access the file directly from the<br>
fuse mount, ie.:<br>
<br>
# mount | grep export<br>
10.0.1.7:/export on /export type fuse.glusterfs<br>
(rw,relatime,user_id=0,group_<wbr>id=0,allow_other,max_read=<wbr>131072)<br>
<br>
Then set samba to share without using the vfs gluster plugin as follows<br>
<br>
[Projects]<br>
path = /export/projects<br>
browseable = yes<br>
write list = @Staff,root,@Admin,@Managers<br>
writeable = yes<br>
guest ok = no<br>
create mask = 660<br>
directory mask = 770<br>
<br>
This test will determine if the issue is the samba vfs gluster plugin<br>
or if it is the fact that the file is stored in the gluster volume.<br>
<br>
Any other thoughts?<br></blockquote></div></div></div><div dir="auto"><br></div><div dir="auto">I suspect this has to do with locking and such. </div><div dir="auto">Can create a new volume and share through vfs plugin.</div><div dir="auto">Set logging to number 5 for samba and Gluster volume set client-log-level and server-log-level to TRACE.</div><div dir="auto">Copy one rvt file. </div><div dir="auto">Performs open, this is like fail. </div><div dir="auto">Perform rename </div><div dir="auto">Perform open, this would pass </div><div dir="auto"><br></div><div dir="auto">Send us the logs. </div><div dir="auto"><br></div><div dir="auto">I know it is troublesome but detailed logs are the only things that would help us analyze the issue. </div><div dir="auto"><br></div><div dir="auto">Talur </div><div dir="auto"><br></div><div dir="auto"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<font color="#888888"><br>
Diego<br>
</font><div class="elided-text"><br>
On Wed, May 31, 2017 at 12:59 PM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
> The vfs gluster plugin for samba is linked against libglusterfs.so.0<br>
> which is provided by glusterfs-libs-3.10.2-1.el7.<wbr>x86_64, please see<br>
> below.<br>
><br>
> # ldd /usr/lib64/samba/vfs/<wbr>glusterfs.so | grep glus<br>
> libglusterfs.so.0 => /lib64/libglusterfs.so.0 (0x00007f61da858000)<br>
><br>
> ]# yum provides /lib64/libglusterfs.so.0<br>
> Loaded plugins: fastestmirror<br>
> Loading mirror speeds from cached hostfile<br>
> * base: <a href="http://centos.vwtonline.net" rel="noreferrer" target="_blank">centos.vwtonline.net</a><br>
> * extras: <a href="http://mirror.cs.vt.edu" rel="noreferrer" target="_blank">mirror.cs.vt.edu</a><br>
> * updates: <a href="http://centosv.centos.org" rel="noreferrer" target="_blank">centosv.centos.org</a><br>
> glusterfs-libs-3.10.2-1.el7.<wbr>x86_64 : GlusterFS common libraries<br>
> Repo : @centos-gluster310<br>
> Matched from:<br>
> Filename : /lib64/libglusterfs.so.0<br>
><br>
><br>
><br>
> On Wed, May 31, 2017 at 12:39 PM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
>> Samba is running in the same machine as glusterd. The machines were<br>
>> rebooted after the upgrades and samba has been restarted a few times.<br>
>><br>
>> # rpm -qa | grep gluster<br>
>> glusterfs-client-xlators-3.10.<wbr>2-1.el7.x86_64<br>
>> glusterfs-server-3.10.2-1.el7.<wbr>x86_64<br>
>> glusterfs-api-3.10.2-1.el7.<wbr>x86_64<br>
>> glusterfs-3.10.2-1.el7.x86_64<br>
>> glusterfs-cli-3.10.2-1.el7.<wbr>x86_64<br>
>> centos-release-gluster310-1.0-<wbr>1.el7.centos.noarch<br>
>> samba-vfs-glusterfs-4.4.4-14.<wbr>el7_3.x86_64<br>
>> glusterfs-fuse-3.10.2-1.el7.<wbr>x86_64<br>
>> glusterfs-libs-3.10.2-1.el7.<wbr>x86_64<br>
>> glusterfs-rdma-3.10.2-1.el7.<wbr>x86_64<br>
>><br>
>> # rpm -qa | grep samba<br>
>> samba-common-libs-4.4.4-14.<wbr>el7_3.x86_64<br>
>> samba-common-tools-4.4.4-14.<wbr>el7_3.x86_64<br>
>> samba-libs-4.4.4-14.el7_3.x86_<wbr>64<br>
>> samba-4.4.4-14.el7_3.x86_64<br>
>> samba-client-libs-4.4.4-14.<wbr>el7_3.x86_64<br>
>> samba-vfs-glusterfs-4.4.4-14.<wbr>el7_3.x86_64<br>
>> samba-common-4.4.4-14.el7_3.<wbr>noarch<br>
>><br>
>> # cat /etc/redhat-release<br>
>> CentOS Linux release 7.3.1611 (Core)<br>
>><br>
>> I also raised the op version.<br>
>><br>
>> # gluster volume get all cluster.op-version<br>
>> Option Value<br>
>> ------ -----<br>
>> cluster.op-version 31000<br>
>><br>
>> # gluster volume get all cluster.max-op-version<br>
>> Option Value<br>
>> ------ -----<br>
>> cluster.max-op-version 31000<br>
>><br>
>> On Wed, May 31, 2017 at 12:21 PM, Raghavendra Talur <<a href="mailto:rtalur@redhat.com">rtalur@redhat.com</a>> wrote:<br>
>>> Also, please attach your smb.conf. You can directly attach in the list<br>
>>> and need not have a google drive link.<br>
>>><br>
>>><br>
>>> On Wed, May 31, 2017 at 9:37 PM, Raghavendra Talur <<a href="mailto:rtalur@redhat.com">rtalur@redhat.com</a>> wrote:<br>
>>>> Diego,<br>
>>>><br>
>>>> I see that Samba is still using 3.6.6 Gluster. Is it possible you did<br>
>>>> not restart smb after upgrading gluster(if samba is on same machine as<br>
>>>> Gluster) or if you forgot to update Gluster client packages on Samba<br>
>>>> node?<br>
>>>><br>
>>>> On Wed, May 31, 2017 at 9:04 PM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
>>>>> Please download the log file from this link:<br>
>>>>><br>
>>>>> <a href="https://drive.google.com/open?id=0B8EAPWIe4oyKN0h0X1pZVkRWVEU" rel="noreferrer" target="_blank">https://drive.google.com/open?<wbr>id=<wbr>0B8EAPWIe4oyKN0h0X1pZVkRWVEU</a><br>
>>>>><br>
>>>>> Let me know if you need any other log files.<br>
>>>>><br>
>>>>> Diego<br>
>>>>><br>
>>>>> On Wed, May 31, 2017 at 11:19 AM, Raghavendra Talur <<a href="mailto:rtalur@redhat.com">rtalur@redhat.com</a>> wrote:<br>
>>>>>> If possible please share the glusterfs-* log files from /var/log/samba.<br>
>>>>>><br>
>>>>>> This might be because of cluster.lookup-optimize. Adding Poornima and<br>
>>>>>> Raghavendra Gowdappa to help with this.<br>
>>>>>><br>
>>>>>><br>
>>>>>> On Wed, May 31, 2017 at 1:03 AM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
>>>>>>> This is a bit puzzling, not sure what difference it would make, but:<br>
>>>>>>><br>
>>>>>>> 1. Try to open file that has a problem, ie. MyRevitFile.rvt<br>
>>>>>>> Revit opens and shows a window that says access denied.<br>
>>>>>>><br>
>>>>>>> 2. Rename file, i.e from windows explorer right click, rename to<br>
>>>>>>> MyRevitFile2.rvt<br>
>>>>>>><br>
>>>>>>> 3. Without opening the file, rename file back to the original, i.e<br>
>>>>>>> MyRevitFile.rvt<br>
>>>>>>><br>
>>>>>>> 4. Double click on file and now it will open just fine without the<br>
>>>>>>> Access Denied error.<br>
>>>>>>><br>
>>>>>>> Any explanation for this? Could the rename operation be forcing or<br>
>>>>>>> updating some attributes that then allow the file to open?<br>
>>>>>>><br>
>>>>>>> Diego<br>
>>>>>>><br>
>>>>>>> On Tue, May 30, 2017 at 10:57 AM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
>>>>>>>> This is what I see in the logs set from smb.conf via line -><br>
>>>>>>>> glusterfs:logfile = /var/log/samba/glusterfs-<wbr>projects.log<br>
>>>>>>>><br>
>>>>>>>> [2017-05-30 14:52:31.051524] E [MSGID: 123001]<br>
>>>>>>>> [io-cache.c:564:ioc_open_cbk] 0-export-io-cache: inode context is NULL<br>
>>>>>>>> (a97bc9bb-68cf-4a69-aef7-<wbr>39766b323c14) [Invalid argument]<br>
>>>>>>>> [2017-05-30 14:52:31.241850] W [MSGID: 114031]<br>
>>>>>>>> [client-rpc-fops.c:1100:<wbr>client3_3_getxattr_cbk] 0-export-client-0:<br>
>>>>>>>> remote operation failed. Path:<br>
>>>>>>>> /projects/INACTIVE/WESTCOAST/<wbr>Automotive/Acura/AS-Acura of Richmond/02<br>
>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-<wbr>ArchModel.rvt (a97bc9bb-68cf-4a69-<br>
>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:<wbr>desktop.ini [Not<br>
>>>>>>>> a directory]<br>
>>>>>>>> [2017-05-30 14:52:31.242956] W [MSGID: 114031]<br>
>>>>>>>> [client-rpc-fops.c:1100:<wbr>client3_3_getxattr_cbk] 0-export-client-1:<br>
>>>>>>>> remote operation failed. Path:<br>
>>>>>>>> /projects/INACTIVE/WESTCOAST/<wbr>Automotive/Acura/AS-Acura of Richmond/02<br>
>>>>>>>> DRAWINGS/02 ARCH/CrownAcura-SD02-<wbr>ArchModel.rvt (a97bc9bb-68cf-4a69-<br>
>>>>>>>> aef7-39766b323c14). Key: glusterfs.get_real_filename:<wbr>desktop.ini [Not<br>
>>>>>>>> a directory]<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> On Tue, May 30, 2017 at 10:37 AM, Diego Remolina <<a href="mailto:dijuremo@gmail.com">dijuremo@gmail.com</a>> wrote:<br>
>>>>>>>>> Hi,<br>
>>>>>>>>><br>
>>>>>>>>> Over the weekend we updated a two server glusterfs 3.6.6 install to<br>
>>>>>>>>> 3.10.2 We also updated samba and samba-vfs to the latest in CentOS. I<br>
>>>>>>>>> enabled several of the newer caching features from gluster 3.9 for<br>
>>>>>>>>> small file performance and samba, and we now seem to have some issues<br>
>>>>>>>>> with accessing files from glusterfs. When users try to access some<br>
>>>>>>>>> files, they get a Permission denied message. This seems to be only via<br>
>>>>>>>>> samba as I am able to su - username and then do strings on the file.<br>
>>>>>>>>><br>
>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep gluster<br>
>>>>>>>>> glusterfs-client-xlators-3.10.<wbr>2-1.el7.x86_64<br>
>>>>>>>>> glusterfs-server-3.10.2-1.el7.<wbr>x86_64<br>
>>>>>>>>> glusterfs-api-3.10.2-1.el7.<wbr>x86_64<br>
>>>>>>>>> glusterfs-3.10.2-1.el7.x86_64<br>
>>>>>>>>> glusterfs-cli-3.10.2-1.el7.<wbr>x86_64<br>
>>>>>>>>> centos-release-gluster310-1.0-<wbr>1.el7.centos.noarch<br>
>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.<wbr>el7_3.x86_64<br>
>>>>>>>>> glusterfs-fuse-3.10.2-1.el7.<wbr>x86_64<br>
>>>>>>>>> glusterfs-libs-3.10.2-1.el7.<wbr>x86_64<br>
>>>>>>>>> glusterfs-rdma-3.10.2-1.el7.<wbr>x86_64<br>
>>>>>>>>><br>
>>>>>>>>> [root@ysmha02 gluster-backups]# rpm -qa | grep samba<br>
>>>>>>>>> samba-common-libs-4.4.4-14.<wbr>el7_3.x86_64<br>
>>>>>>>>> samba-common-tools-4.4.4-14.<wbr>el7_3.x86_64<br>
>>>>>>>>> samba-libs-4.4.4-14.el7_3.x86_<wbr>64<br>
>>>>>>>>> samba-4.4.4-14.el7_3.x86_64<br>
>>>>>>>>> samba-client-libs-4.4.4-14.<wbr>el7_3.x86_64<br>
>>>>>>>>> samba-vfs-glusterfs-4.4.4-14.<wbr>el7_3.x86_64<br>
>>>>>>>>> samba-common-4.4.4-14.el7_3.<wbr>noarch<br>
>>>>>>>>><br>
>>>>>>>>> On the samba logs for the machine I notice something weird, samba<br>
>>>>>>>>> seems to be trying to stat the file we are trying as a directory to<br>
>>>>>>>>> see if it contains desktop.ini:<br>
>>>>>>>>><br>
>>>>>>>>> [2017/05/30 10:13:07.297026, 0]<br>
>>>>>>>>> ../source3/modules/vfs_<wbr>glusterfs.c:870(vfs_gluster_<wbr>stat)<br>
>>>>>>>>> glfs_stat(ACTIVE/Automotive/<wbr>FORD/AN - Ford East/02<br>
>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/<wbr>desktop.ini) failed:<br>
>>>>>>>>> Not a directory<br>
>>>>>>>>> [2017/05/30 10:13:07.298155, 0]<br>
>>>>>>>>> ../source3/modules/vfs_<wbr>glusterfs.c:870(vfs_gluster_<wbr>stat)<br>
>>>>>>>>> glfs_stat(ACTIVE/Automotive/<wbr>FORD/AN - Ford East/02<br>
>>>>>>>>> DRAWINGS/CURRENT/AN-FORD EAST_04-05-17_CD_R17.rvt/<wbr>desktop.ini) failed:<br>
>>>>>>>>> Not a directory<br>
>>>>>>>>><br>
>>>>>>>>> This seems to be happening only with files with the .rvt extension.<br>
>>>>>>>>> Though these files are usually larger in size vs other smaller excel,<br>
>>>>>>>>> power point, etc files.<br>
>>>>>>>>><br>
>>>>>>>>> Here are the complete options for the volume:<br>
>>>>>>>>><br>
>>>>>>>>> <a href="https://pastebin.com/ZH2vMsMN" rel="noreferrer" target="_blank">https://pastebin.com/ZH2vMsMN</a><br>
>>>>>>>>><br>
>>>>>>>>> I turned off performance.cache-samba-<wbr>metadata again to see if that<br>
>>>>>>>>> would help, but seems it does not help.<br>
>>>>>>>>><br>
>>>>>>>>> I really appreciate any help with this.<br>
>>>>>>>>><br>
>>>>>>>>> DIego<br>
>>>>>>> ______________________________<wbr>_________________<br>
>>>>>>> Gluster-users mailing list<br>
>>>>>>> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
>>>>>>> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
</div></blockquote></div><br></div></div></div>