[Bugs] [Bug 1479655] New: Permission denied errors when appending files after readdir

bugzilla at redhat.com bugzilla at redhat.com
Wed Aug 9 04:46:16 UTC 2017


https://bugzilla.redhat.com/show_bug.cgi?id=1479655

            Bug ID: 1479655
           Summary: Permission denied errors when appending files after
                    readdir
           Product: GlusterFS
           Version: 3.12
         Component: libgfapi
          Keywords: Triaged
          Severity: high
          Assignee: bugs at gluster.org
          Reporter: pgurusid at redhat.com
        QA Contact: bugs at gluster.org
                CC: amukherj at redhat.com, anoopcs at redhat.com,
                    bugs at gluster.org, ksandha at redhat.com,
                    rhinduja at redhat.com, rhs-smb at redhat.com
        Depends On: 1457713, 1477488



+++ This bug was initially created as a clone of Bug #1477488 +++

+++ This bug was initially created as a clone of Bug #1457713 +++

Description of problem:
Running readdir (scan directories only, don't read files or their metadata) 
operation and then doing append to files results in permission denied errors on
cifs 

Version-Release number of selected component (if applicable):
3.8.4-26

How reproducible:
100%

Steps to Reproduce:
mount the volume using cifs3
1. Create files using small-file tool
2. Run readdir operation on the mount.
3. After completion , run append operation. 

Actual results:
Permission denied errors seen on tool result in failure.

Expected results:
No errors should be observed. 

Additional info:

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-01
03:48:25 EDT ---

This bug is automatically being proposed for the current release of Red Hat
Gluster Storage 3 under active development, by setting the release flag
'rhgs‑3.3.0' to '?'. 

If this bug should be proposed for a different release, please manually change
the proposed release flag.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-01
03:51:48 EDT ---

This bug report has Keywords: Regression or TestBlocker.

Since no regressions or test blockers are allowed between releases, it is also
being identified as a blocker for this release.

Please resolve ASAP.

--- Additional comment from Red Hat Bugzilla Rules Engine on 2017-06-09
03:48:54 EDT ---


--- Additional comment from Poornima G on 2017-06-20 05:18:44 EDT ---

As discussed, the issue wasn't reproduced on setup with 4node samba and 4
clients when ctdb was added. The issue originally was reported with samba
cluster but no ctdb which is not a valid setup. Hence deferring it from 3.3.

But would like to retain the BZ and see if there is any other bug that is being
exposed only in non CTDB setup as the same was working earlier.

--- Additional comment from Karan Sandha on 2017-06-27 05:06:18 EDT ---

Hi,

I am able to reproduce this on ctdb setup also.

--- Additional comment from Atin Mukherjee on 2017-06-27 14:24:02 EDT ---


--- Additional comment from Karan Sandha on 2017-07-03 09:06:13 EDT ---

Hi Anoop,

I have seen this issue repetitively on Linux and windows clients. Attaching the
screen shot of the issue and the logs. 


The issue has started apperaring on delete-renamed operation too. 

Thanks & Regards
Karan Sandha


--- Additional comment from Karan Sandha on 2017-07-03 09:12 EDT ---



--- Additional comment from Anoop C S on 2017-07-06 03:23:07 EDT ---

Hi Karan,

I looked over the from comment #9. It's weird that I am not seeing any
'Permission denied' or related errors in any of the
logs(samba/glusterfs-client/brick logs). Since you mentioned it as 100%
reproducible, can you please make the following changes to /etc/samba/smb.conf
and re-run the tests in hope of getting some useful logs.

* Change 'max log size = 50' --> 'max log size = 0'
* Add 'log level = 7'
* And in [glusterfs-<VOLNAME>] section change 'glusterfs:loglevel = 7' -->
'glusterfs:loglevel = 8'

Additionally, you can also increase the brick log level to DEBUG.

--- Additional comment from Karan Sandha on 2017-07-11 09:13:10 EDT ---

Hi Anoop,
The permission denied issue was seen on application side. as seen in the
screenshot. 
[root at gqac006 ~]# qeSyncDropCaches.sh; python
/small-files/smallfile/smallfile_cli.py --operation append --threads 8
--file-size 64 --files 2000 --top /gluster-mount  --host-set "`echo $CLIENT |
tr ' ' ','`"Dropping cache on 
smallfile version 3.0
                           hosts in test :
['gqac006.sbu.lab.eng.bos.redhat.com', 'gqac025.sbu.lab.eng.bos.redhat.com',
'gqac026.sbu.lab.eng.bos.redhat.com', 'gqac027.sbu.lab.eng.bos.redhat.com']
                   top test directory(s) : ['/gluster-mount']
                               operation : append
                            files/thread : 2000
                                 threads : 8
           record size (KB, 0 = maximum) : 0
                          file size (KB) : 64
                  file size distribution : fixed
                           files per dir : 100
                            dirs per dir : 10
              threads share directories? : N
                         filename prefix : 
                         filename suffix : 
             hash file number into dir.? : N
                     fsync after modify? : N
          pause between files (microsec) : 0
                    finish all requests? : Y
                              stonewall? : Y
                 measure response times? : N
                            verify read? : Y
                                verbose? : False
                          log to stderr? : False
                           ext.attr.size : 0
                          ext.attr.count : 0
               permute host directories? : N
                remote program directory : /small-files/smallfile
               network thread sync. dir. : /gluster-mount/network_shared
starting all threads by creating starting gate file
/gluster-mount/network_shared/starting_gate.tmp
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 00,elapsed =
-1499777081.965670,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 01,elapsed =
-1499777081.976919,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 02,elapsed =
-1499777081.976804,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 03,elapsed =
-1499777081.976993,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 04,elapsed =
-1499777081.977319,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 05,elapsed =
-1499777081.977191,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 06,elapsed =
-1499777081.977044,files = 0,records = 0,status = ERR: Permission denied
host = gqac006.sbu.lab.eng.bos.redhat.com,thr = 07,elapsed =
-1499777081.977497,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 00,elapsed =
-1499777083.199119,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 01,elapsed =
-1499777083.199166,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 02,elapsed =
-1499777083.199351,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 03,elapsed =
-1499777083.199384,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 04,elapsed =
-1499777083.199855,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 05,elapsed =
-1499777083.200094,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 06,elapsed =
-1499777083.200609,files = 0,records = 0,status = ERR: Permission denied
host = gqac025.sbu.lab.eng.bos.redhat.com,thr = 07,elapsed =
-1499777083.200980,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 00,elapsed =
-1499777083.377028,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 01,elapsed =
-1499777083.377371,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 02,elapsed =
-1499777083.377461,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 03,elapsed =
-1499777083.377809,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 04,elapsed =
-1499777083.378299,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 05,elapsed =
-1499777083.378584,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 06,elapsed =
-1499777083.379124,files = 0,records = 0,status = ERR: Permission denied
host = gqac026.sbu.lab.eng.bos.redhat.com,thr = 07,elapsed =
-1499777083.379233,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 00,elapsed =
-1499777083.312093,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 01,elapsed =
-1499777083.312346,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 02,elapsed =
-1499777083.312302,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 03,elapsed =
-1499777083.312386,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 04,elapsed =
-1499777083.312767,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 05,elapsed =
-1499777083.313294,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 06,elapsed =
-1499777083.313696,files = 0,records = 0,status = ERR: Permission denied
host = gqac027.sbu.lab.eng.bos.redhat.com,thr = 07,elapsed =
-1499777083.313883,files = 0,records = 0,status = ERR: Permission denied
total threads = 32
total files = 0
  0.00% of requested files processed, minimum is  70.00
at least one thread encountered error, test may be incomplete
[root at gqac006 ~]#

--- Additional comment from Worker Ant on 2017-08-02 04:49:27 EDT ---

REVIEW: https://review.gluster.org/17956 (md-cache: Fix use after free) posted
(#1) for review on master by Poornima G (pgurusid at redhat.com)

--- Additional comment from Worker Ant on 2017-08-03 08:26:12 EDT ---

REVIEW: https://review.gluster.org/17967 (gfapi: Duplicate the buffer sent in
setxattr calls) posted (#1) for review on master by Poornima G
(pgurusid at redhat.com)

--- Additional comment from Worker Ant on 2017-08-04 01:12:59 EDT ---

REVIEW: https://review.gluster.org/17967 (gfapi: Duplicate the buffer sent in
setxattr calls) posted (#2) for review on master by Poornima G
(pgurusid at redhat.com)

--- Additional comment from Worker Ant on 2017-08-04 06:14:43 EDT ---

REVIEW: https://review.gluster.org/17967 (gfapi: Duplicate the buffer sent in
setxattr calls) posted (#3) for review on master by Poornima G
(pgurusid at redhat.com)

--- Additional comment from Worker Ant on 2017-08-06 08:03:16 EDT ---

REVIEW: https://review.gluster.org/17983 (core: remove memdup() and use
gf_memdup() instead) posted (#1) for review on master by Niels de Vos
(ndevos at redhat.com)

--- Additional comment from Worker Ant on 2017-08-06 08:36:47 EDT ---

REVIEW: https://review.gluster.org/17983 (core: remove memdup() and use
gf_memdup() instead) posted (#2) for review on master by Niels de Vos
(ndevos at redhat.com)

--- Additional comment from Worker Ant on 2017-08-07 01:19:33 EDT ---

REVIEW: https://review.gluster.org/17967 (gfapi: Duplicate the buffer sent in
setxattr calls) posted (#4) for review on master by Poornima G
(pgurusid at redhat.com)

--- Additional comment from Worker Ant on 2017-08-07 05:16:57 EDT ---

REVIEW: https://review.gluster.org/17967 (gfapi: Duplicate the buffer sent in
setxattr calls) posted (#5) for review on master by Poornima G
(pgurusid at redhat.com)

--- Additional comment from Worker Ant on 2017-08-07 12:28:49 EDT ---

COMMIT: https://review.gluster.org/17967 committed in master by Jeff Darcy
(jeff at pl.atyp.us) 
------
commit e11296f8e52b7e3b13d21b41d4fa34baea878edf
Author: Poornima G <pgurusid at redhat.com>
Date:   Thu Aug 3 17:43:22 2017 +0530

    gfapi: Duplicate the buffer sent in setxattr calls

    Issue:
    The caller of glfs_setxattr sends a buffer to set as the value.
    We create a dict in which the pointer to the value is set.
    Underlying layers like md-cache take a ref on this dict to store
    the value for a longer time. But the moment setxattr is complete,
    the caller of glfs_setxattr can free the value memory.

    Solution:
    memcpy the setxattr value to the gluster buffer.

    Change-Id: I58753fe702e8b7d0f6c4f058714c65d0ad5d7a0a
    BUG: 1477488
    Signed-off-by: Poornima G <pgurusid at redhat.com>
    Reviewed-on: https://review.gluster.org/17967
    Reviewed-by: soumya k <skoduri at redhat.com>
    Smoke: Gluster Build System <jenkins at build.gluster.org>
    CentOS-regression: Gluster Build System <jenkins at build.gluster.org>
    Reviewed-by: Jeff Darcy <jeff at pl.atyp.us>


Referenced Bugs:

https://bugzilla.redhat.com/show_bug.cgi?id=1457713
[Bug 1457713] Permission denied errors when appending files after readdir
https://bugzilla.redhat.com/show_bug.cgi?id=1477488
[Bug 1477488] Permission denied errors when appending files after readdir
-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.


More information about the Bugs mailing list