[Gluster-devel] remaining entry in gluster volume heal info command even after reboot
Pranith Kumar Karampuri
pkarampu at redhat.com
Wed Sep 5 08:55:32 UTC 2018
On Wed, Sep 5, 2018 at 1:27 PM Zhou, Cynthia (NSB - CN/Hangzhou) <
cynthia.zhou at nokia-sbell.com> wrote:
> Hi glusterfs experts:
>
> Good day!
>
> Recently when I do some test on my gluster env, I found that there
> are some remaining entries in command “gluster v heal mstate info” *even
> after reboot*.
>
> fstest_035ffc492ec43551a64087f9280ffe3e is a folder in /mnt/mstate
> and in this folder only one file(fstest_458bb82d8884ed5c9dadec4ed93bec4e)
> exists.
>
> When I dbg by gdb on sn-0 I found that:
>
> Parent dir changelog(fstest_035ffc492ec43551a64087f9280ffe3e) says need
> to heal entry, but the changelog/gfid/filetype of the only entry in parent
> dir shows there is nothing to be healed, so glustershd does nothing every
> round of heal. And this entry will remain.
>
> My gdb shows that each round of heal on sn-0 , it exits in function
> __afr_selfheal_entry (if (AFR_COUNT(healed_sinks, priv->child_count) ==
> 0)), because in this case all three healed_sinks are zero.
>
What is the return value of this function in gdb?
> Have you any idea how to solve this issue from glusterfs pov?
> Thanks!
>
>
>
> [test steps]
>
> Reboot three sn nodes( sn-0, sn-1, sn-2(arbiter)) sequentially, and
> on another node (with glusterfs clients) run fstest.
>
>
>
> [problem description]
>
>
>
> Remaining entries in “gluster v heal mstate info” command even after
> reboot sn-0 many times, the entries are still there!
>
>
>
> [root at sn-0:/home/robot]
>
> # gluster v heal mstate info
>
> Brick sn-0.local:/mnt/bricks/mstate/brick
>
> /fstest_035ffc492ec43551a64087f9280ffe3e
>
> Status: Connected
>
> Number of entries: 1
>
>
>
> Brick sn-1.local:/mnt/bricks/mstate/brick
>
> Status: Connected
>
> Number of entries: 0
>
>
>
> Brick sn-2.local:/mnt/bricks/mstate/brick
>
> /fstest_035ffc492ec43551a64087f9280ffe3e
>
> Status: Connected
>
> Number of entries: 1
>
>
>
>
>
>
>
> ////////////////////////////////////////////////////////////////////some
> env informations///////////////////////////////////////////////////////
>
> # gluster v info mstate
>
> Volume Name: mstate
>
> Type: Replicate
>
> Volume ID: 1d896674-17a2-4ae7-aa7c-c6e22013df99
>
> Status: Started
>
> Snapshot Count: 0
>
> Number of Bricks: 1 x (2 + 1) = 3
>
> Transport-type: tcp
>
> Bricks:
>
> Brick1: sn-0.local:/mnt/bricks/mstate/brick
>
> Brick2: sn-1.local:/mnt/bricks/mstate/brick
>
> Brick3: sn-2.local:/mnt/bricks/mstate/brick (arbiter)
>
> Options Reconfigured:
>
> performance.client-io-threads: off
>
> nfs.disable: on
>
> transport.address-family: inet
>
> cluster.server-quorum-type: none
>
> cluster.quorum-reads: no
>
> cluster.favorite-child-policy: mtime
>
> cluster.consistent-metadata: on
>
> network.ping-timeout: 42
>
> cluster.quorum-type: auto
>
> server.allow-insecure: on
>
> cluster.server-quorum-ratio: 51%
>
> [root at sn-1:/home/robot]
>
>
>
>
>
> [root at sn-2:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.mstate-client-0=0x000000010000000000000003
>
> trusted.gfid=0xa0975560eaef4cb299467101de00446a
>
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>
>
>
> [root at sn-2:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> getfattr: Removing leading '/' from absolute path names
>
> # file:
> mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> trusted.afr.mstate-client-0=0x000000000000000000000000
>
> trusted.gfid=0x9fc20f587f094182816390f056f7370f
>
>
> trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465
>
>
>
> # cd /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e
>
> [root at sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> # ls
>
> fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> [root at sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> # stat fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> File: fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> Size: 0 Blocks: 8 IO Block: 4096 fifo
>
> Device: fd31h/64817d Inode: 22086 Links: 2
>
> Access: (0644/prw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>
> Access: 2018-08-30 04:33:17.552870661 +0300
>
> Modify: 2018-08-30 04:33:17.552870661 +0300
>
> Change: 2018-08-30 04:33:17.553870661 +0300
>
> Birth: -
>
> [root at sn-2
> :/mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e]
>
> [root at sn-2:/root]
>
> # exit
>
> logout
>
> Connection to sn-2.local closed.
>
> [root at sn-0:/home/robot]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.mstate-client-2=0x00000000000000000000001c
>
> trusted.gfid=0xa0975560eaef4cb299467101de00446a
>
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>
>
>
> [root at sn-0:/home/robot]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> getfattr: Removing leading '/' from absolute path names
>
> # file:
> mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> trusted.gfid=0x9fc20f587f094182816390f056f7370f
>
>
> trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465
>
>
>
> # ls /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e
>
> fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> [root at sn-0:/home/robot]
>
> # stat
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> File:
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> Size: 0 Blocks: 8 IO Block: 4096 fifo
>
> Device: fd31h/64817d Inode: 21899 Links: 2
>
> Access: (0644/prw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>
> Access: 2018-08-30 04:33:17.552870661 +0300
>
> Modify: 2018-08-30 04:33:17.552870661 +0300
>
> Change: 2018-08-30 04:33:17.263809048 +0300
>
> Birth: -
>
> [root at sn-0:/home/robot]
>
> [root at sn-0:/home/robot]
>
> # ssh sn-1.local
>
>
>
> USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR
> LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish)
> AND/OR LIMITED BASH SHELL.
>
>
>
> Read /opt/nokia/share/security/readme_root.txt for more details.
>
>
>
> [root at sn-1:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> getfattr: Removing leading '/' from absolute path names
>
> # file: mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/
>
> trusted.afr.dirty=0x000000000000000000000000
>
> trusted.afr.mstate-client-0=0x000000000000000000000000
>
> trusted.afr.mstate-client-2=0x000000000000000000000000
>
> trusted.gfid=0xa0975560eaef4cb299467101de00446a
>
> trusted.glusterfs.dht=0x000000010000000000000000ffffffff
>
>
>
> [root at sn-1:/root]
>
> # getfattr -m . -d -e hex
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> getfattr: Removing leading '/' from absolute path names
>
> # file:
> mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> trusted.afr.mstate-client-0=0x000000000000000000000000
>
> trusted.gfid=0x9fc20f587f094182816390f056f7370f
>
>
> trusted.gfid2path.864159d77373ad5f=0x61303937353536302d656165662d346362322d393934362d3731303164653030343436612f6673746573745f3435386262383264383838346564356339646164656334656439336265633465
>
>
>
> [root at sn-1:/root]
>
> [root at sn-1:/root]
>
> # ls /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e
>
> fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> [root at sn-1:/root]
>
> # stat
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
>
> File:
> /mnt/bricks/mstate/brick/fstest_035ffc492ec43551a64087f9280ffe3e/fstest_458bb82d8884ed5c9dadec4ed93bec4e
>
> Size: 0 Blocks: 8 IO Block: 4096 fifo
>
> Device: fd31h/64817d Inode: 22168 Links: 2
>
> Access: (0644/prw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
>
> Access: 2018-08-30 04:33:17.552870661 +0300
>
> Modify: 2018-08-30 04:33:17.552870661 +0300
>
> Change: 2018-08-30 04:33:17.037673648 +0300
>
> Birth: -
>
> [root at sn-1:/root]
>
>
>
>
>
>
>
--
Pranith
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-devel/attachments/20180905/6dd4fc2a/attachment-0001.html>
More information about the Gluster-devel
mailing list