[Gluster-users] files got sticky permissions T--------- after gluster volume rebalance

[Joe Julian]

If you change permissions on a 0 length mode 1000 file with 
trusted.dht.linkto attributes set, you'll cause all sorts of issues. If 
you /need/ to do something with those files, just delete them.

On 08/22/2013 03:53 PM, ??? wrote:
> Sure there's a bug about fd leak during rebalance. It has been 
> fixed and I have back port it in our production cluster. About the 
> permission issue I have to run find cmd too.
>
> ? 2013?8?23????,Justin Dossey ??:
>
>     I had the same problem after a rebalance (going from 2 bricks to 6
>     bricks).  It took about a week to get everything straightened out
>     (and I reported details on what I did to fix it in this mailing
>     list).
>
>     I dread the next rebalance (going from 6 bricks to 8 bricks)! For
>     this rollout, I still have five rebalances remaining before I can
>     declare the GlusterFS migration complete.
>
>     To recap,
>     1. After a rebalance in which one of my nodes (not the "master",
>     which I initiated the rebalance from) had to be rebooted due to
>     too many open files on system (caused by the rebalance), many
>     files appeared to clients to have 000 or 1000 (---------- or
>     T---------) permissions.  Many of these files could not even be
>     chmodded by root over NFS, returning error 576 when I tried.
>     2. I found that in many cases, files which had this problem had
>     entries on more than two bricks (and my replica count is 2).  The
>     entries had different permissions and some were zero-length files.
>      It appears that different clients got different entries at
>     different times, so one might see a file as inaccessible while
>     another could read it without issues.
>     3. I wrote a script to remove the zero-length files (and their
>     .glusterfs shadow links), and set permissions properly on all the
>     files.  Luckily, all the files on my volume have uniform
>     permissions (files are all 0644, directories are all 0755).
>     4. I ran a find command every ten minutes to find and correct bad
>     permissions.  The script in (3) didn't appear to have gotten them
>     all for some reason.
>     5. No more files have appeared with this problem since August 6th.
>      I'm still running the find every day.
>     6. After the permissions problems appeared to be resolved, I ran a
>     check to verify that all the files present on the volume before
>     the rebalance were present after the rebalance.  Thankfully, the
>     data appears to have all survived.
>
>     The only feedback I got on this mailing list was that nothing was
>     wrong.
>
>
>     On Wed, Aug 21, 2013 at 11:21 PM, Vijay Bellur <vbellur at redhat.com
>     <javascript:_e({}, 'cvml', 'vbellur at redhat.com');>> wrote:
>
>         On 08/22/2013 09:12 AM, ??? wrote:
>
>             Hi Joe thank you but the sticky permissions is exposed to
>             client side
>             due to potential bug related to glusterfs rebalance.
>
>
>
>         Can you please provide output of ls -l that shows these files
>         after rebalance?
>
>         -Vijay
>
>
>
>             2013/8/20 Joe Julian <joe at julianfamily.org
>             <javascript:_e({}, 'cvml', 'joe at julianfamily.org');>
>             <mailto:joe at julianfamily.org <javascript:_e({}, 'cvml',
>             'joe at julianfamily.org');>>>
>
>
>                 Sticky pointers are normal. See the extended
>             attributes on them to
>                 see where they point,
>
>                 getfattr -m trusted.* -d $filename
>
>                 To diagnose your client issue, look in your client log.
>
>
>                 "???" <yongtaofu at gmail.com <javascript:_e({}, 'cvml',
>             'yongtaofu at gmail.com');> <mailto:yongtaofu at gmail.com
>             <javascript:_e({}, 'cvml', 'yongtaofu at gmail.com');>>> wrote:
>
>                     Dear gluster experts,
>
>                     We're running glusterfs 3.3 and we have met file
>             permission
>                     probelems after gluster volume rebalance. Files
>             got stick
>                     permissions T--------- after rebalance which break
>             our client
>                     normal fops unexpectedly.
>                     Any one known this issue?
>                     Thank you for your help.
>
>
>                 --
>                 Sent from my Android device with K-9 Mail. Please
>             excuse my brevity.
>
>
>
>
>             --
>             ???
>
>
>             _______________________________________________
>             Gluster-users mailing list
>             Gluster-users at gluster.org <javascript:_e({}, 'cvml',
>             'Gluster-users at gluster.org');>
>             http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>         _______________________________________________
>         Gluster-users mailing list
>         Gluster-users at gluster.org <javascript:_e({}, 'cvml',
>         'Gluster-users at gluster.org');>
>         http://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
>     -- 
>     Justin Dossey
>     CTO, PodOmatic
>
>
>
> -- 
> ???
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130822/6e8a8da5/attachment.html>