[Bugs] [Bug 1274595] Data Tiering:getting failed to fsync on germany-hot-dht (Structure needs cleaning) warning

Tue Dec 1 16:22:32 UTC 2015

https://bugzilla.redhat.com/show_bug.cgi?id=1274595

nchilaka <nchilaka at redhat.com> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|ON_QA                       |VERIFIED


--- Comment #4 from nchilaka <nchilaka at redhat.com> ---
I have validated the fix and found that the issue is not seen.
1)Checked with both brick log level as default(info mode) and trace mode
Tested with watermarks disabled
Tested on 3.7.5-7

Volume Name: fsync
Type: Tier
Volume ID: 862b28d6-329e-4ad4-8e32-0dd5e62a2670
Status: Started
Number of Bricks: 8
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distributed-Replicate
Number of Bricks: 2 x 2 = 4
Brick1: yarrow:/dummy/brick104/fsync_hot
Brick2: zod:/dummy/brick104/fsync_hot
Brick3: yarrow:/dummy/brick105/fsync_hot
Brick4: zod:/dummy/brick105/fsync_hot
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 4
Brick5: zod:/rhs/brick1/fsync
Brick6: yarrow:/rhs/brick1/fsync
Brick7: zod:/rhs/brick2/fsync
Brick8: yarrow:/rhs/brick2/fsync
Options Reconfigured:
cluster.tier-mode: test
features.ctr-enabled: on
performance.readdir-ahead: on


Following were the logs seen:
============================
[2015-12-01 13:52:00.876106] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//trans.avi)
[2015-12-01 13:52:00.898462] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//gola.avi)
[2015-12-01 13:52:26.694075] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//mm700.avi)
=====================
[2015-12-01 13:57:51.919390] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt:
Volume file changed
[2015-12-01 13:57:52.542343] I [glusterfsd-mgmt.c:58:mgmt_cbk_spec] 0-mgmt:
Volume file changed
[2015-12-01 13:57:52.592790] I [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk]
0-glusterfs: No change in volfile, continuing
[2015-12-01 13:57:52.620886] I [glusterfsd-mgmt.c:1596:mgmt_getspec_cbk]
0-glusterfs: No change in volfile, continuing
 ########retrying after setting to trace log lveel
 ########retrying after setting to trace log lveel
 ########retrying after setting to trace log lveel
 ########retrying after setting to trace log lveel
 ########retrying after setting to trace log lveel
[2015-12-01 13:58:29.922001] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 741.00 secs
[2015-12-01 13:58:29.922042] I [MSGID: 109028]
[dht-rebalance.c:3612:gf_defrag_status_get] 0-glusterfs: Files migrated: 11,
size: 0, lookups: 11, failures: 0, skipped: 0
[2015-12-01 13:58:33.370249] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 745.00 secs
[2015-12-01 13:58:36.972573] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 748.00 secs
[2015-12-01 14:00:00.212550] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//trans.avi)
[2015-12-01 14:00:00.240233] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//gola.avi)
[2015-12-01 14:00:00.245615] I [MSGID: 109038]
[tier.c:530:tier_migrate_using_query_file] 0-fsync-tier-dht: Reached cycle
migration limit.migrated bytes 1486189890 files 2
[2015-12-01 13:58:36.994649] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 748.00 secs
The message "I [MSGID: 109028] [dht-rebalance.c:3612:gf_defrag_status_get]
0-glusterfs: Files migrated: 11, size: 0, lookups: 11, failures: 0, skipped: 0"
repeated 3 times between [2015-12-01 13:58:29.922042] and [2015-12-01
13:58:36.994651]
[2015-12-01 14:01:52.535352] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 944.00 secs
[2015-12-01 14:01:52.535397] I [MSGID: 109028]
[dht-rebalance.c:3612:gf_defrag_status_get] 0-glusterfs: Files migrated: 15,
size: 0, lookups: 18, failures: 0, skipped: 0
[2015-12-01 14:01:52.561891] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 944.00 secs
[2015-12-01 14:01:52.561893] I [MSGID: 109028]
[dht-rebalance.c:3612:gf_defrag_status_get] 0-glusterfs: Files migrated: 15,
size: 0, lookups: 18, failures: 0, skipped: 0
[2015-12-01 14:02:52.845777] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 1004.00 secs
[2015-12-01 14:02:52.845819] I [MSGID: 109028]
[dht-rebalance.c:3612:gf_defrag_status_get] 0-glusterfs: Files migrated: 18,
size: 0, lookups: 21, failures: 0, skipped: 0
-=========================
[2015-12-01 14:02:52.876059] I [MSGID: 109028]
[dht-rebalance.c:3608:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 1004.00 secs
[2015-12-01 14:02:52.876061] I [MSGID: 109028]
[dht-rebalance.c:3612:gf_defrag_status_get] 0-glusterfs: Files migrated: 18,
size: 0, lookups: 21, failures: 0, skipped: 0
===###################
[2015-12-01 14:06:00.312002] E [MSGID: 108008]
[afr-read-txn.c:89:afr_read_txn_refresh_done] 0-fsync-replicate-0: Failing
GETXATTR on gfid c4038106-02b9-48d1-8c11-6826a6b408bb: split-brain observed.
[Input/output error]
[2015-12-01 14:06:00.312269] W [MSGID: 109023]
[dht-rebalance.c:1281:dht_migrate_file] 0-fsync-tier-dht: Migrate file
failed://gtrans.avi: failed to get xattr from fsync-hot-dht (Input/output
error)
[2015-12-01 14:06:00.314588] W [dict.c:612:dict_ref]
(-->/lib64/libglusterfs.so.0(syncop_fsetxattr+0x1a4) [0x7feebcc6fe14]
-->/usr/lib64/glusterfs/3.7.5/xlator/cluster/distribute.so(dht_fsetxattr+0xcb)
[0x7feeaeee93fb] -->/lib64/libglusterfs.so.0(dict_ref+0x79) [0x7feebcc202a9] )
0-dict: dict is NULL [Invalid argument]
[2015-12-01 14:06:00.315395] W [MSGID: 114031]
[client-rpc-fops.c:1980:client3_3_fsetxattr_cbk] 0-fsync-client-1: remote
operation failed
[2015-12-01 14:06:00.316094] W [MSGID: 114031]
[client-rpc-fops.c:1980:client3_3_fsetxattr_cbk] 0-fsync-client-6: remote
operation failed [No space left on device]
[2015-12-01 14:06:00.316206] W [MSGID: 114031]
[client-rpc-fops.c:1980:client3_3_fsetxattr_cbk] 0-fsync-client-7: remote
operation failed [No space left on device]
[2015-12-01 14:06:00.316680] W [MSGID: 109023]
[dht-rebalance.c:592:__dht_rebalance_create_dst_file] 0-fsync-tier-dht:
//trans.avi: failed to set xattr on fsync-hot-dht (No space left on device)
[2015-12-01 14:06:00.322215] E [MSGID: 108008]
[afr-transaction.c:1981:afr_transaction] 0-fsync-replicate-0: Failing SETXATTR
on gfid c4038106-02b9-48d1-8c11-6826a6b408bb: split-brain observed.
[Input/output error]
[2015-12-01 14:06:00.322521] E [MSGID: 109023]
[dht-rebalance.c:907:__dht_rebalance_open_src_file] 0-fsync-tier-dht: failed to
set xattr on //gtrans.avi in fsync-hot-dht (Input/output error)
[2015-12-01 14:06:00.322545] E [MSGID: 109023]
[dht-rebalance.c:1306:dht_migrate_file] 0-fsync-tier-dht: Migrate file failed:
failed to open //gtrans.avi on fsync-hot-dht
[2015-12-01 14:06:00.329383] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//trans.avi)
[2015-12-01 14:06:00.354235] E [MSGID: 109023]
[dht-rebalance.c:721:__dht_check_free_space] 0-fsync-tier-dht: data movement
attempted from node (fsync-cold-dht) to node (fsync-hot-dht) which does not
have required free space for (//gola.avi)
[2015-12-01 14:06:00.358709] I [MSGID: 109038]
[tier.c:530:tier_migrate_using_query_file] 0-fsync-tier-dht: Reached cycle
migration limit.migrated bytes 1486189890 files 2


Hence moving to verified

-- 
You are receiving this mail because:
You are on the CC list for the bug.
Unsubscribe from this bug https://bugzilla.redhat.com/token.cgi?t=0ooVss743H&a=cc_unsubscribe