[Gluster-users] files got sticky permissions T--------- after gluster volume rebalance

[符永涛]

Thank you Joe but I didn't change permissions of files on the brick
back-end. Some of the file permissions changed to T--------- unexpectedly
after rebalance from client side view. I only run find and try to repair
the permissions from client side.



2013/8/23 Joe Julian <joe at julianfamily.org>

>  If you change permissions on a 0 length mode 1000 file with
> trusted.dht.linkto attributes set, you'll cause all sorts of issues. If you
> /need/ to do something with those files, just delete them.
>
>
> On 08/22/2013 03:53 PM, 符永涛 wrote:
>
> Sure there's a bug about fd leak during rebalance. It has been fixed and I
> have back port it in our production cluster. About the permission issue I
> have to run find cmd too.
>
> 在 2013年8月23日星期五，Justin Dossey 写道：
>
>> I had the same problem after a rebalance (going from 2 bricks to 6
>> bricks).  It took about a week to get everything straightened out (and I
>> reported details on what I did to fix it in this mailing list).
>>
>>  I dread the next rebalance (going from 6 bricks to 8 bricks)! For this
>> rollout, I still have five rebalances remaining before I can declare the
>> GlusterFS migration complete.
>>
>>  To recap,
>> 1. After a rebalance in which one of my nodes (not the "master", which I
>> initiated the rebalance from) had to be rebooted due to too many open files
>> on system (caused by the rebalance), many files appeared to clients to have
>> 000 or 1000 (---------- or T---------) permissions.  Many of these files
>> could not even be chmodded by root over NFS, returning error 576 when I
>> tried.
>> 2. I found that in many cases, files which had this problem had entries
>> on more than two bricks (and my replica count is 2).  The entries had
>> different permissions and some were zero-length files.  It appears that
>> different clients got different entries at different times, so one might
>> see a file as inaccessible while another could read it without issues.
>> 3. I wrote a script to remove the zero-length files (and their .glusterfs
>> shadow links), and set permissions properly on all the files.  Luckily, all
>> the files on my volume have uniform permissions (files are all 0644,
>> directories are all 0755).
>> 4. I ran a find command every ten minutes to find and correct bad
>> permissions.  The script in (3) didn't appear to have gotten them all for
>> some reason.
>> 5. No more files have appeared with this problem since August 6th.  I'm
>> still running the find every day.
>> 6. After the permissions problems appeared to be resolved, I ran a check
>> to verify that all the files present on the volume before the rebalance
>> were present after the rebalance.  Thankfully, the data appears to have all
>> survived.
>>
>>  The only feedback I got on this mailing list was that nothing was wrong.
>>
>>
>> On Wed, Aug 21, 2013 at 11:21 PM, Vijay Bellur <vbellur at redhat.com>wrote:
>>
>>> On 08/22/2013 09:12 AM, 符永涛 wrote:
>>>
>>>> Hi Joe thank you but the sticky permissions is exposed to client side
>>>> due to potential bug related to glusterfs rebalance.
>>>>
>>>
>>>
>>>  Can you please provide output of ls -l that shows these files after
>>> rebalance?
>>>
>>> -Vijay
>>>
>>>>
>>>>
>>>> 2013/8/20 Joe Julian <joe at julianfamily.org <mailto:joe at julianfamily.org>>
>>>>
>>>>
>>>>
>>>>     Sticky pointers are normal. See the extended attributes on them to
>>>>     see where they point,
>>>>
>>>>     getfattr -m trusted.* -d $filename
>>>>
>>>>     To diagnose your client issue, look in your client log.
>>>>
>>>>
>>>>      "符永涛" <yongtaofu at gmail.com <mailto:yongtaofu at gmail.com>> wrote:
>>>>
>>>>         Dear gluster experts,
>>>>
>>>>         We're running glusterfs 3.3 and we have met file permission
>>>>         probelems after gluster volume rebalance. Files got stick
>>>>         permissions T--------- after rebalance which break our client
>>>>         normal fops unexpectedly.
>>>>         Any one known this issue?
>>>>         Thank you for your help.
>>>>
>>>>
>>>>     --
>>>>     Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> 符永涛
>>>>
>>>>
>>>>  _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
>>
>>
>>
>>  --
>> Justin Dossey
>> CTO, PodOmatic
>>
>>
>
> --
> 符永涛
>
>
> _______________________________________________
> Gluster-users mailing listGluster-users at gluster.orghttp://supercolony.gluster.org/mailman/listinfo/gluster-users
>
>
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>



-- 
符永涛
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20130823/a55b4153/attachment.html>