[Gluster-devel] gfid and volume-id extended attributes lost
Sanoj Unnikrishnan
sunnikri at redhat.com
Mon Jul 10 11:49:22 UTC 2017
Please use the systemtap script(
https://paste.fedoraproject.org/paste/EGDa0ErwX0LV3y-gBYpfNA) to check
which process is invoking remove xattr calls.
It prints the pid, tid and arguments of all removexattr calls.
I have checked for these fops at the protocol/client and posix translators.
To run the script ..
1) install systemtap and dependencies.
2) install glusterfs-debuginfo
3) change the path of the translator in the systemtap script to appropriate
values for your system
(change "/usr/lib64/glusterfs/3.12dev/xlator/protocol/client.so" and
"/usr/lib64/glusterfs/3.12dev/xlator/storage/posix.so")
4) run the script as follows
#stap -v fop_trace.stp
The o/p would look like these .. additionally arguments will also be dumped
if glusterfs-debuginfo is also installed (i had not done it here.)
pid-958: 0 glusterfsd(3893):->posix_setxattr
pid-958: 47 glusterfsd(3893):<-posix_setxattr
pid-966: 0 glusterfsd(5033):->posix_setxattr
pid-966: 57 glusterfsd(5033):<-posix_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423: 37 glusterfs(1431):<-client_setxattr
pid-1423: 0 glusterfs(1431):->client_setxattr
pid-1423: 41 glusterfs(1431):<-client_setxattr
Regards,
Sanoj
On Mon, Jul 10, 2017 at 2:56 PM, Sanoj Unnikrishnan <sunnikri at redhat.com>
wrote:
> @ pranith , yes . we can get the pid on all removexattr call and also
> print the backtrace of the glusterfsd process when trigerring removing
> xattr.
> I will write the script and reply back.
>
> On Sat, Jul 8, 2017 at 7:06 AM, Pranith Kumar Karampuri <
> pkarampu at redhat.com> wrote:
>
>> Ram,
>> As per the code, self-heal was the only candidate which *can* do
>> it. Could you check logs of self-heal daemon and the mount to check if
>> there are any metadata heals on root?
>>
>>
>> +Sanoj
>>
>> Sanoj,
>> Is there any systemtap script we can use to detect which process
>> is removing these xattrs?
>>
>> On Sat, Jul 8, 2017 at 2:58 AM, Ankireddypalle Reddy <
>> areddy at commvault.com> wrote:
>>
>>> We lost the attributes on all the bricks on servers glusterfs2 and
>>> glusterfs3 again.
>>>
>>>
>>>
>>> [root at glusterfs2 Log_Files]# gluster volume info
>>>
>>>
>>>
>>> Volume Name: StoragePool
>>>
>>> Type: Distributed-Disperse
>>>
>>> Volume ID: 149e976f-4e21-451c-bf0f-f5691208531f
>>>
>>> Status: Started
>>>
>>> Number of Bricks: 20 x (2 + 1) = 60
>>>
>>> Transport-type: tcp
>>>
>>> Bricks:
>>>
>>> Brick1: glusterfs1sds:/ws/disk1/ws_brick
>>>
>>> Brick2: glusterfs2sds:/ws/disk1/ws_brick
>>>
>>> Brick3: glusterfs3sds:/ws/disk1/ws_brick
>>>
>>> Brick4: glusterfs1sds:/ws/disk2/ws_brick
>>>
>>> Brick5: glusterfs2sds:/ws/disk2/ws_brick
>>>
>>> Brick6: glusterfs3sds:/ws/disk2/ws_brick
>>>
>>> Brick7: glusterfs1sds:/ws/disk3/ws_brick
>>>
>>> Brick8: glusterfs2sds:/ws/disk3/ws_brick
>>>
>>> Brick9: glusterfs3sds:/ws/disk3/ws_brick
>>>
>>> Brick10: glusterfs1sds:/ws/disk4/ws_brick
>>>
>>> Brick11: glusterfs2sds:/ws/disk4/ws_brick
>>>
>>> Brick12: glusterfs3sds:/ws/disk4/ws_brick
>>>
>>> Brick13: glusterfs1sds:/ws/disk5/ws_brick
>>>
>>> Brick14: glusterfs2sds:/ws/disk5/ws_brick
>>>
>>> Brick15: glusterfs3sds:/ws/disk5/ws_brick
>>>
>>> Brick16: glusterfs1sds:/ws/disk6/ws_brick
>>>
>>> Brick17: glusterfs2sds:/ws/disk6/ws_brick
>>>
>>> Brick18: glusterfs3sds:/ws/disk6/ws_brick
>>>
>>> Brick19: glusterfs1sds:/ws/disk7/ws_brick
>>>
>>> Brick20: glusterfs2sds:/ws/disk7/ws_brick
>>>
>>> Brick21: glusterfs3sds:/ws/disk7/ws_brick
>>>
>>> Brick22: glusterfs1sds:/ws/disk8/ws_brick
>>>
>>> Brick23: glusterfs2sds:/ws/disk8/ws_brick
>>>
>>> Brick24: glusterfs3sds:/ws/disk8/ws_brick
>>>
>>> Brick25: glusterfs4sds.commvault.com:/ws/disk1/ws_brick
>>>
>>> Brick26: glusterfs5sds.commvault.com:/ws/disk1/ws_brick
>>>
>>> Brick27: glusterfs6sds.commvault.com:/ws/disk1/ws_brick
>>>
>>> Brick28: glusterfs4sds.commvault.com:/ws/disk10/ws_brick
>>>
>>> Brick29: glusterfs5sds.commvault.com:/ws/disk10/ws_brick
>>>
>>> Brick30: glusterfs6sds.commvault.com:/ws/disk10/ws_brick
>>>
>>> Brick31: glusterfs4sds.commvault.com:/ws/disk11/ws_brick
>>>
>>> Brick32: glusterfs5sds.commvault.com:/ws/disk11/ws_brick
>>>
>>> Brick33: glusterfs6sds.commvault.com:/ws/disk11/ws_brick
>>>
>>> Brick34: glusterfs4sds.commvault.com:/ws/disk12/ws_brick
>>>
>>> Brick35: glusterfs5sds.commvault.com:/ws/disk12/ws_brick
>>>
>>> Brick36: glusterfs6sds.commvault.com:/ws/disk12/ws_brick
>>>
>>> Brick37: glusterfs4sds.commvault.com:/ws/disk2/ws_brick
>>>
>>> Brick38: glusterfs5sds.commvault.com:/ws/disk2/ws_brick
>>>
>>> Brick39: glusterfs6sds.commvault.com:/ws/disk2/ws_brick
>>>
>>> Brick40: glusterfs4sds.commvault.com:/ws/disk3/ws_brick
>>>
>>> Brick41: glusterfs5sds.commvault.com:/ws/disk3/ws_brick
>>>
>>> Brick42: glusterfs6sds.commvault.com:/ws/disk3/ws_brick
>>>
>>> Brick43: glusterfs4sds.commvault.com:/ws/disk4/ws_brick
>>>
>>> Brick44: glusterfs5sds.commvault.com:/ws/disk4/ws_brick
>>>
>>> Brick45: glusterfs6sds.commvault.com:/ws/disk4/ws_brick
>>>
>>> Brick46: glusterfs4sds.commvault.com:/ws/disk5/ws_brick
>>>
>>> Brick47: glusterfs5sds.commvault.com:/ws/disk5/ws_brick
>>>
>>> Brick48: glusterfs6sds.commvault.com:/ws/disk5/ws_brick
>>>
>>> Brick49: glusterfs4sds.commvault.com:/ws/disk6/ws_brick
>>>
>>> Brick50: glusterfs5sds.commvault.com:/ws/disk6/ws_brick
>>>
>>> Brick51: glusterfs6sds.commvault.com:/ws/disk6/ws_brick
>>>
>>> Brick52: glusterfs4sds.commvault.com:/ws/disk7/ws_brick
>>>
>>> Brick53: glusterfs5sds.commvault.com:/ws/disk7/ws_brick
>>>
>>> Brick54: glusterfs6sds.commvault.com:/ws/disk7/ws_brick
>>>
>>> Brick55: glusterfs4sds.commvault.com:/ws/disk8/ws_brick
>>>
>>> Brick56: glusterfs5sds.commvault.com:/ws/disk8/ws_brick
>>>
>>> Brick57: glusterfs6sds.commvault.com:/ws/disk8/ws_brick
>>>
>>> Brick58: glusterfs4sds.commvault.com:/ws/disk9/ws_brick
>>>
>>> Brick59: glusterfs5sds.commvault.com:/ws/disk9/ws_brick
>>>
>>> Brick60: glusterfs6sds.commvault.com:/ws/disk9/ws_brick
>>>
>>> Options Reconfigured:
>>>
>>> performance.readdir-ahead: on
>>>
>>> diagnostics.client-log-level: INFO
>>>
>>> auth.allow: glusterfs1sds,glusterfs2sds,glusterfs3sds,glusterfs4sds.comm
>>> vault.com,glusterfs5sds.commvault.com,glusterfs6sds.commvault.com
>>>
>>>
>>>
>>> Thanks and Regards,
>>>
>>> Ram
>>>
>>> *From:* Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
>>> *Sent:* Friday, July 07, 2017 12:15 PM
>>>
>>> *To:* Ankireddypalle Reddy
>>> *Cc:* Gluster Devel (gluster-devel at gluster.org);
>>> gluster-users at gluster.org
>>> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes
>>> lost
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 7, 2017 at 9:25 PM, Ankireddypalle Reddy <
>>> areddy at commvault.com> wrote:
>>>
>>> 3.7.19
>>>
>>>
>>>
>>> These are the only callers for removexattr and only _posix_remove_xattr
>>> has the potential to do removexattr as posix_removexattr already makes sure
>>> that it is not gfid/volume-id. And surprise surprise _posix_remove_xattr
>>> happens only from healing code of afr/ec. And this can only happen if the
>>> source brick doesn't have gfid, which doesn't seem to match with the
>>> situation you explained.
>>>
>>> # line filename / context / line
>>> 1 1234 xlators/mgmt/glusterd/src/glusterd-quota.c
>>> <<glusterd_remove_quota_limit>>
>>> ret = sys_lremovexattr (abspath, QUOTA_LIMIT_KEY);
>>> 2 1243 xlators/mgmt/glusterd/src/glusterd-quota.c
>>> <<glusterd_remove_quota_limit>>
>>> ret = sys_lremovexattr (abspath, QUOTA_LIMIT_OBJECTS_KEY);
>>> 3 6102 xlators/mgmt/glusterd/src/glusterd-utils.c
>>> <<glusterd_check_and_set_brick_xattr>>
>>> sys_lremovexattr (path, "trusted.glusterfs.test");
>>> 4 80 xlators/storage/posix/src/posix-handle.h
>>> <<REMOVE_PGFID_XATTR>>
>>> op_ret = sys_lremovexattr (path, key); \
>>> 5 5026 xlators/storage/posix/src/posix.c <<_posix_remove_xattr>>
>>> op_ret = sys_lremovexattr (filler->real_path, key);
>>> 6 5101 xlators/storage/posix/src/posix.c <<posix_removexattr>>
>>> op_ret = sys_lremovexattr (real_path, name);
>>> 7 6811 xlators/storage/posix/src/posix.c <<init>>
>>> sys_lremovexattr (dir_data->data, "trusted.glusterfs.test");
>>>
>>> So there are only two possibilities:
>>>
>>> 1) Source directory in ec/afr doesn't have gfid
>>>
>>> 2) Something else removed these xattrs.
>>>
>>> What is your volume info? May be that will give more clues.
>>>
>>>
>>>
>>> PS: sys_fremovexattr is called only from posix_fremovexattr(), so that
>>> doesn't seem to be the culprit as it also have checks to guard against
>>> gfid/volume-id removal.
>>>
>>>
>>>
>>> Thanks and Regards,
>>>
>>> Ram
>>>
>>> *From:* Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
>>> *Sent:* Friday, July 07, 2017 11:54 AM
>>>
>>>
>>> *To:* Ankireddypalle Reddy
>>> *Cc:* Gluster Devel (gluster-devel at gluster.org);
>>> gluster-users at gluster.org
>>> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes
>>> lost
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 7, 2017 at 9:20 PM, Ankireddypalle Reddy <
>>> areddy at commvault.com> wrote:
>>>
>>> Pranith,
>>>
>>> Thanks for looking in to the issue. The bricks were
>>> mounted after the reboot. One more thing that I noticed was when the
>>> attributes were manually set when glusterd was up then on starting the
>>> volume the attributes were again lost. Had to stop glusterd set attributes
>>> and then start glusterd. After that the volume start succeeded.
>>>
>>>
>>>
>>> Which version is this?
>>>
>>>
>>>
>>>
>>>
>>> Thanks and Regards,
>>>
>>> Ram
>>>
>>>
>>>
>>> *From:* Pranith Kumar Karampuri [mailto:pkarampu at redhat.com]
>>> *Sent:* Friday, July 07, 2017 11:46 AM
>>> *To:* Ankireddypalle Reddy
>>> *Cc:* Gluster Devel (gluster-devel at gluster.org);
>>> gluster-users at gluster.org
>>> *Subject:* Re: [Gluster-devel] gfid and volume-id extended attributes
>>> lost
>>>
>>>
>>>
>>> Did anything special happen on these two bricks? It can't happen in the
>>> I/O path:
>>> posix_removexattr() has:
>>> 0 if (!strcmp (GFID_XATTR_KEY, name))
>>> {
>>>
>>>
>>> 1 gf_msg (this->name, GF_LOG_WARNING, 0,
>>> P_MSG_XATTR_NOT_REMOVED,
>>> 2 "Remove xattr called on gfid for file %s",
>>> real_path);
>>> 3 op_ret = -1;
>>>
>>> 4 goto out;
>>>
>>> 5 }
>>>
>>> 6 if (!strcmp (GF_XATTR_VOL_ID_KEY, name))
>>> {
>>> 7 gf_msg (this->name, GF_LOG_WARNING, 0,
>>> P_MSG_XATTR_NOT_REMOVED,
>>> 8 "Remove xattr called on volume-id for file
>>> %s",
>>> 9 real_path);
>>>
>>> 10 op_ret = -1;
>>>
>>> 11 goto out;
>>>
>>> 12 }
>>>
>>> I just found that op_errno is not set correctly, but it can't happen in
>>> the I/O path, so self-heal/rebalance are off the hook.
>>>
>>> I also grepped for any removexattr of trusted.gfid from glusterd and
>>> didn't find any.
>>>
>>> So one thing that used to happen was that sometimes when machines
>>> reboot, the brick mounts wouldn't happen and this would lead to absence of
>>> both trusted.gfid and volume-id. So at the moment this is my wild guess.
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jul 7, 2017 at 8:39 PM, Ankireddypalle Reddy <
>>> areddy at commvault.com> wrote:
>>>
>>> Hi,
>>>
>>> We faced an issue in the production today. We had to stop the
>>> volume and reboot all the servers in the cluster. Once the servers
>>> rebooted starting of the volume failed because the following extended
>>> attributes were not present on all the bricks on 2 servers.
>>>
>>> 1) trusted.gfid
>>>
>>> 2) trusted.glusterfs.volume-id
>>>
>>>
>>>
>>> We had to manually set these extended attributes to start the volume.
>>> Are there any such known issues.
>>>
>>>
>>>
>>> Thanks and Regards,
>>>
>>> Ram
>>>
>>> ***************************Legal Disclaimer***************************
>>>
>>> "This communication may contain confidential and privileged material for
>>> the
>>>
>>> sole use of the intended recipient. Any unauthorized review, use or
>>> distribution
>>>
>>> by others is strictly prohibited. If you have received the message by
>>> mistake,
>>>
>>> please advise the sender by reply email and delete the message. Thank
>>> you."
>>>
>>> **********************************************************************
>>>
>>>
>>> _______________________________________________
>>> Gluster-devel mailing list
>>> Gluster-devel at gluster.org
>>> http://lists.gluster.org/mailman/listinfo/gluster-devel
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Pranith
>>>
>>> ***************************Legal Disclaimer***************************
>>>
>>> "This communication may contain confidential and privileged material for
>>> the
>>>
>>> sole use of the intended recipient. Any unauthorized review, use or
>>> distribution
>>>
>>> by others is strictly prohibited. If you have received the message by
>>> mistake,
>>>
>>> please advise the sender by reply email and delete the message. Thank
>>> you."
>>>
>>> **********************************************************************
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Pranith
>>>
>>> ***************************Legal Disclaimer***************************
>>>
>>> "This communication may contain confidential and privileged material for
>>> the
>>>
>>> sole use of the intended recipient. Any unauthorized review, use or
>>> distribution
>>>
>>> by others is strictly prohibited. If you have received the message by
>>> mistake,
>>>
>>> please advise the sender by reply email and delete the message. Thank
>>> you."
>>>
>>> **********************************************************************
>>>
>>>
>>>
>>>
>>> --
>>>
>>> Pranith
>>> ***************************Legal Disclaimer***************************
>>> "This communication may contain confidential and privileged material for
>>> the
>>> sole use of the intended recipient. Any unauthorized review, use or
>>> distribution
>>> by others is strictly prohibited. If you have received the message by
>>> mistake,
>>> please advise the sender by reply email and delete the message. Thank
>>> you."
>>> **********************************************************************
>>>
>>
>>
>>
>> --
>> Pranith
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20170710/7eee2de1/attachment-0001.html>
More information about the Gluster-devel
mailing list