[Bugs] [Bug 1240926] New: Data Tiering: file marked for demotion is getting demoted even if the file is being accessed( a heating operation)
bugzilla at redhat.com
bugzilla at redhat.com
Wed Jul 8 07:20:14 UTC 2015
https://bugzilla.redhat.com/show_bug.cgi?id=1240926
Bug ID: 1240926
Summary: Data Tiering: file marked for demotion is getting
demoted even if the file is being accessed( a heating
operation)
Product: GlusterFS
Version: mainline
Component: tiering
Severity: urgent
Assignee: bugs at gluster.org
Reporter: nchilaka at redhat.com
QA Contact: bugs at gluster.org
CC: bugs at gluster.org
RAISING THIS BUG FOR UPSTREAM TRACKING OF DOWNSTREAM BUG 1240925
Description:
============
When I file in hot tier is idle for sometime, it is marked for demotion. But
before the demote happens, if the file is accessed again, by say reading the
file or changing file metadata, then the file ideally should continue to stay
in hot tier. But this is not the case. It is currently getting demoted, once
marked for demotion irrespective of file being re-accessed or not
Steps:
=====
A file marked for demotion is getting demoted even if the file is being
accessed:
1) i created a hot file as below
[root at rhs-client6 vol2]# touch xyz
[root at rhs-client6 vol2]# stat xyz
File: `xyz'
Size: 0 Blocks: 0 IO Block: 1048576 regular empty file
Device: 1bh/27d Inode: 13353154674754277901 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-08 01:25:09.000000000 +0530
Modify: 2015-07-08 01:25:09.000000000 +0530
Change: 2015-07-08 01:25:09.097999988 +0530
2)checked backend brick, file was in hot tier and epoch was stamped
[root at nchilaka-tier03 ~]# echo "== c1 ==" ; echo "select * from gf_file_tb;
select * from gf_flink_tb;" | sqlite3 /rhs/brick1/vol2/.glusterfs/vol2.db; echo
"== c2 ==" ; echo "select * from gf_file_tb; select * from gf_flink_tb;" |
sqlite3 /rhs/brick2/vol2/.glusterfs/vol2.db; echo "== h1 ==" ; echo "select *
from gf_file_tb; select * from gf_flink_tb;" | sqlite3
/rhs/brick3/vol2/.glusterfs/vol2.db
== c1 ==
Error: unable to open database "/rhs/brick1/vol2/.glusterfs/vol2.db": unable to
open database file
== c2 ==
Error: unable to open database "/rhs/brick2/vol2/.glusterfs/vol2.db": unable to
open database file
== h1 ==
68ab61db-c15f-497c-b94f-ef6dbb93520d|1436298909|97064|0|0|0|0|0|0|1|1
68ab61db-c15f-497c-b94f-ef6dbb93520d|00000000-0000-0000-0000-000000000001|xyz|/xyz|0|0
3)used touch to re-access file again after 30 sec
[root at rhs-client6 vol2]# touch xyz
[root at rhs-client6 vol2]# stat xyz
File: `xyz'
Size: 0 Blocks: 0 IO Block: 1048576 regular empty file
Device: 1bh/27d Inode: 13353154674754277901 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-08 01:25:38.000000000 +0530
Modify: 2015-07-08 01:25:38.000000000 +0530
Change: 2015-07-08 01:25:38.882000001 +0530
4)saw brickend and ctime was stamped successfully as below
[root at nchilaka-tier03 ~]# echo "== c1 ==" ; echo "select * from gf_file_tb;
select * from gf_flink_tb;" | sqlite3 /rhs/brick1/vol2/.glusterfs/vol2.db; echo
"== c2 ==" ; echo "select * from gf_file_tb; select * from gf_flink_tb;" |
sqlite3 /rhs/brick2/vol2/.glusterfs/vol2.db; echo "== h1 ==" ; echo "select *
from gf_file_tb; select * from gf_flink_tb;" | sqlite3
/rhs/brick3/vol2/.glusterfs/vol2.db
== c1 ==
Error: unable to open database "/rhs/brick1/vol2/.glusterfs/vol2.db": unable to
open database file
== c2 ==
Error: unable to open database "/rhs/brick2/vol2/.glusterfs/vol2.db": unable to
open database file
== h1 ==
68ab61db-c15f-497c-b94f-ef6dbb93520d|1436298938|880419|0|0|0|0|0|0|1|1
68ab61db-c15f-497c-b94f-ef6dbb93520d|00000000-0000-0000-0000-000000000001|xyz|/xyz|0|0
5)HOWEVER, Now within a few seconds the file got demoted (below was the ctime)
[root at rhs-client6 vol2]# stat xyz
File: `xyz'
Size: 0 Blocks: 0 IO Block: 1048576 regular empty file
Device: 1bh/27d Inode: 13353154674754277901 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2015-07-08 01:25:38.000000000 +0530
Modify: 2015-07-08 01:25:38.000000000 +0530
Change: 2015-07-08 01:26:00.583000149 +0530
6)brick backend
[root at nchilaka-tier01 ~]# echo "== c1 ==" ; echo "select * from gf_file_tb;
select * from gf_flink_tb;" | sqlite3 /rhs/brick1/vol2/.glusterfs/vol2.db; echo
"== c2 ==" ; echo "select * from gf_file_tb; select * from gf_flink_tb;" |
sqlite3 /rhs/brick2/vol2/.glusterfs/vol2.db; echo "== h1 ==" ; echo "select *
from gf_file_tb; select * from gf_flink_tb;" | sqlite3
/rhs/brick3/vol2/.glusterfs/vol2.db
== c1 ==
== c2 ==
52ef1723-f985-4d47-9f6f-0fd302e3553d|0|0|0|0|0|0|0|0|1|1
68ab61db-c15f-497c-b94f-ef6dbb93520d|0|0|0|0|0|0|0|0|1|1
90bc6ab0-e836-415f-940a-c955b2a4544c|0|0|0|0|0|0|0|0|1|1
52ef1723-f985-4d47-9f6f-0fd302e3553d|00000000-0000-0000-0000-000000000001|f1|/f1|0|0
68ab61db-c15f-497c-b94f-ef6dbb93520d|00000000-0000-0000-0000-000000000001|xyz|/xyz|0|0
90bc6ab0-e836-415f-940a-c955b2a4544c|00000000-0000-0000-0000-000000000001|newnew|/newnew|0|0
== h1 ==
[root at nchilaka-tier01 ~]#
[root at nchilaka-tier03 ~]# echo "== c1 ==" ; echo "select * from gf_file_tb;
select * from gf_flink_tb;" | sqlite3 /rhs/brick1/vol2/.glusterfs/vol2.db; echo
"== c2 ==" ; echo "select * from gf_file_tb; select * from gf_flink_tb;" |
sqlite3 /rhs/brick2/vol2/.glusterfs/vol2.db; echo "== h1 ==" ; echo "select *
from gf_file_tb; select * from gf_flink_tb;" | sqlite3
/rhs/brick3/vol2/.glusterfs/vol2.db
== c1 ==
Error: unable to open database "/rhs/brick1/vol2/.glusterfs/vol2.db": unable to
open database file
== c2 ==
Error: unable to open database "/rhs/brick2/vol2/.glusterfs/vol2.db": unable to
open database file
== h1 ==
[root at nchilaka-tier03 ~]#
####################
Tiering reblancer logs
[root at nchilaka-tier03 glusterfs]# ########################################
[root at nchilaka-tier03 glusterfs]# ########################################
[root at nchilaka-tier03 glusterfs]# ########################################
[root at nchilaka-tier03 glusterfs]# ########################################
[root at nchilaka-tier03 glusterfs]# tail -f vol2-tier.log
[2015-07-07 19:32:00.967420] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:32:00.967534] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:32:00.968528] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
########deleted all files #############
[2015-07-07 19:32:58.421870] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89069.00 secs
[2015-07-07 19:32:58.421942] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 26,
size: 0, lookups: 27, failures: 1, skipped: 0
[2015-07-07 19:33:11.970021] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89082.00 secs
[2015-07-07 19:33:12.222025] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89083.00 secs
[2015-07-07 19:33:48.813434] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89119.00 secs
The message "I [MSGID: 109028] [dht-rebalance.c:3033:gf_defrag_status_get]
0-glusterfs: Files migrated: 26, size: 0, lookups: 27, failures: 1, skipped: 0"
repeated 3 times between [2015-07-07 19:32:58.421942] and [2015-07-07
19:33:48.813561]
[2015-07-07 19:33:49.094588] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89120.00 secs
[2015-07-07 19:33:49.094739] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 26,
size: 0, lookups: 27, failures: 1, skipped: 0
[2015-07-07 19:34:00.981909] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:34:00.982043] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:34:00.984646] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 0 src_subvol
vol2-hot-dht file f1
[2015-07-07 19:34:00.985517] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /f1: attempting to move from vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:34:00.998607] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-3: remote
operation failed [Permission denied]
[2015-07-07 19:34:00.998702] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-2: remote
operation failed [Permission denied]
[2015-07-07 19:34:00.998997] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /f1:
failed to set xattr on vol2-cold-dht (Permission denied)
[2015-07-07 19:34:01.031010] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/f1 from subvolume vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:34:01.031799] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
#######created file f1############
[2015-07-07 19:34:55.168130] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89186.00 secs
[2015-07-07 19:34:55.168185] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 27,
size: 0, lookups: 28, failures: 1, skipped: 0
[2015-07-07 19:36:00.047138] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:36:00.047231] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:36:00.048014] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:34:55.404911] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89186.00 secs
[2015-07-07 19:34:55.404915] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 27,
size: 0, lookups: 28, failures: 1, skipped: 0
[2015-07-07 19:38:00.060926] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:38:00.061097] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:38:00.062975] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:38:47.855203] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89418.00 secs
[2015-07-07 19:38:47.855383] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 27,
size: 0, lookups: 28, failures: 1, skipped: 0
[2015-07-07 19:38:48.081794] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89419.00 secs
[2015-07-07 19:40:00.077018] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:40:00.077103] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:40:00.078110] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:38:48.081922] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 27,
size: 0, lookups: 28, failures: 1, skipped: 0
[2015-07-07 19:42:00.091865] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:42:00.091960] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:42:00.094502] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 0 src_subvol
vol2-hot-dht file f1
[2015-07-07 19:42:00.095208] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /f1: attempting to move from vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:42:00.108946] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-3: remote
operation failed [Permission denied]
[2015-07-07 19:42:00.109035] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-2: remote
operation failed [Permission denied]
[2015-07-07 19:42:00.109545] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /f1:
failed to set xattr on vol2-cold-dht (Permission denied)
[2015-07-07 19:42:00.147199] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/f1 from subvolume vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:42:00.148794] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:44:00.161117] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:44:00.161200] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:44:00.162053] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:46:00.173749] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:46:00.173838] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:46:00.174854] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:48:00.188318] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:48:00.188476] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:48:00.190805] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 0 src_subvol
vol2-hot-dht file f1
[2015-07-07 19:48:00.191778] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /f1: attempting to move from vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:48:00.200294] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-3: remote
operation failed [Permission denied]
[2015-07-07 19:48:00.200387] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-2: remote
operation failed [Permission denied]
[2015-07-07 19:48:00.200686] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /f1:
failed to set xattr on vol2-cold-dht (Permission denied)
[2015-07-07 19:48:00.230904] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/f1 from subvolume vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:48:00.231830] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:50:00.246291] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:50:00.246492] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:50:00.247128] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:52:00.260568] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:52:00.260683] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:52:00.261636] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:54:00.273972] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:54:00.274134] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:54:00.276288] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 0 src_subvol
vol2-hot-dht file newnew
[2015-07-07 19:54:00.277311] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /newnew: attempting to move from vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:54:00.284817] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-3: remote
operation failed [Permission denied]
[2015-07-07 19:54:00.284820] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-2: remote
operation failed [Permission denied]
[2015-07-07 19:54:00.285127] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /newnew:
failed to set xattr on vol2-cold-dht (Permission denied)
[2015-07-07 19:54:00.310826] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/newnew from subvolume vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:54:00.311582] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:56:00.326270] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:56:00.326356] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:56:00.328671] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 0 src_subvol
vol2-hot-dht file xyz
[2015-07-07 19:56:00.329502] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /xyz: attempting to move from vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:56:00.338673] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-2: remote
operation failed [Permission denied]
[2015-07-07 19:56:00.338809] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-3: remote
operation failed [Permission denied]
[2015-07-07 19:56:00.339200] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /xyz:
failed to set xattr on vol2-cold-dht (Permission denied)
[2015-07-07 19:56:00.370221] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/xyz from subvolume vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:56:00.372545] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 0 src_subvol
vol2-hot-dht file newnew
[2015-07-07 19:56:00.374297] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /newnew: attempting to move from vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:56:00.382836] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-2: remote
operation failed [Permission denied]
[2015-07-07 19:56:00.382904] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-3: remote
operation failed [Permission denied]
[2015-07-07 19:56:00.383281] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /newnew:
failed to set xattr on vol2-cold-dht (Permission denied)
[2015-07-07 19:56:00.412371] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/newnew from subvolume vol2-hot-dht to vol2-cold-dht
[2015-07-07 19:56:00.413245] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:58:00.424964] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 19:58:00.425047] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 19:58:00.426146] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 20:00:00.438396] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 20:00:00.438488] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 20:00:00.439240] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 20:02:00.531611] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 20:02:00.531727] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 20:02:00.532694] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 20:04:00.546901] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6709
[2015-07-07 20:04:00.546984] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/promotequeryfile-6709 for migration
[2015-07-07 20:04:00.547872] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[root at nchilaka-tier01 glusterfs]# ########################################
[root at nchilaka-tier01 glusterfs]# ########################################
[root at nchilaka-tier01 glusterfs]# ########################################
[root at nchilaka-tier01 glusterfs]# ########################################
[root at nchilaka-tier01 glusterfs]# tail -f vol2-tier.log
########deleted all files #############
[2015-07-07 19:32:58.639229] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89069.00 secs
[2015-07-07 19:32:58.639314] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 37,
size: 0, lookups: 37, failures: 0, skipped: 0
[2015-07-07 19:33:12.187151] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89083.00 secs
[2015-07-07 19:33:49.032889] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89120.00 secs
#######created file f1############
[2015-07-07 19:33:12.438199] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89083.00 secs
[2015-07-07 19:33:49.311872] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89120.00 secs
The message "I [MSGID: 109028] [dht-rebalance.c:3033:gf_defrag_status_get]
0-glusterfs: Files migrated: 37, size: 0, lookups: 37, failures: 0, skipped: 0"
repeated 4 times between [2015-07-07 19:32:58.639314] and [2015-07-07
19:33:49.311875]
[2015-07-07 19:34:55.386510] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89186.00 secs
[2015-07-07 19:34:55.386576] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 37,
size: 0, lookups: 37, failures: 0, skipped: 0
[2015-07-07 19:34:55.623864] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89186.00 secs
[2015-07-07 19:34:55.623867] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 37,
size: 0, lookups: 37, failures: 0, skipped: 0
[2015-07-07 19:38:48.072710] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89419.00 secs
[2015-07-07 19:38:48.072768] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 37,
size: 0, lookups: 37, failures: 0, skipped: 0
[2015-07-07 19:40:00.843725] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol
vol2-cold-dht file f1
[2015-07-07 19:40:00.844920] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /f1: attempting to move from vol2-cold-dht to vol2-hot-dht
[2015-07-07 19:40:00.855922] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-5: remote
operation failed [Permission denied]
[2015-07-07 19:40:00.856003] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /f1:
failed to set xattr on vol2-hot-dht (Permission denied)
[2015-07-07 19:40:00.890948] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/f1 from subvolume vol2-cold-dht to vol2-hot-dht
[2015-07-07 19:38:48.300146] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89419.00 secs
[2015-07-07 19:38:48.300150] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 37,
size: 0, lookups: 37, failures: 0, skipped: 0
[2015-07-07 19:46:00.931646] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol
vol2-cold-dht file f1
[2015-07-07 19:46:00.932667] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /f1: attempting to move from vol2-cold-dht to vol2-hot-dht
[2015-07-07 19:46:00.939392] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-5: remote
operation failed [Permission denied]
[2015-07-07 19:46:00.939492] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /f1:
failed to set xattr on vol2-hot-dht (Permission denied)
[2015-07-07 19:46:00.968211] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/f1 from subvolume vol2-cold-dht to vol2-hot-dht
[2015-07-07 19:56:00.021155] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol
vol2-cold-dht file newnew
[2015-07-07 19:56:00.022004] I [dht-rebalance.c:1002:dht_migrate_file]
0-vol2-tier-dht: /newnew: attempting to move from vol2-cold-dht to vol2-hot-dht
[2015-07-07 19:56:00.029370] W [MSGID: 114031]
[client-rpc-fops.c:1977:client3_3_fsetxattr_cbk] 0-vol2-client-5: remote
operation failed [Permission denied]
[2015-07-07 19:56:00.029449] W [MSGID: 109023]
[dht-rebalance.c:546:__dht_rebalance_create_dst_file] 0-vol2-tier-dht: /newnew:
failed to set xattr on vol2-hot-dht (Permission denied)
[2015-07-07 19:56:00.060819] I [MSGID: 109022]
[dht-rebalance.c:1282:dht_migrate_file] 0-vol2-tier-dht: completed migration of
/newnew from subvolume vol2-cold-dht to vol2-hot-dht
[root at nchilaka-tier02 glusterfs]# ########################################
[root at nchilaka-tier02 glusterfs]# ########################################
[root at nchilaka-tier02 glusterfs]# ########################################
[root at nchilaka-tier02 glusterfs]# ########################################
[root at nchilaka-tier02 glusterfs]# tail -f vol2-tier.log
[2015-07-07 19:32:00.080967] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:32:00.081057] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:32:00.081121] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
########deleted all files #############
[2015-07-07 19:32:58.954230] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89068.00 secs
[2015-07-07 19:32:58.954307] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0,
size: 0, lookups: 22, failures: 22, skipped: 0
[2015-07-07 19:33:12.502584] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89082.00 secs
[2015-07-07 19:33:49.345711] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89119.00 secs
[2015-07-07 19:34:00.094622] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:34:00.094711] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:34:00.094767] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
#######created file f1############
[2015-07-07 19:33:12.753532] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89082.00 secs
[2015-07-07 19:33:49.627265] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89119.00 secs
The message "I [MSGID: 109028] [dht-rebalance.c:3033:gf_defrag_status_get]
0-glusterfs: Files migrated: 0, size: 0, lookups: 22, failures: 22, skipped: 0"
repeated 4 times between [2015-07-07 19:32:58.954307] and [2015-07-07
19:33:49.627268]
[2015-07-07 19:34:55.699793] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89185.00 secs
[2015-07-07 19:34:55.699856] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0,
size: 0, lookups: 22, failures: 22, skipped: 0
[2015-07-07 19:36:00.108488] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:36:00.108650] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:36:00.108773] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:34:55.936619] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89185.00 secs
[2015-07-07 19:34:55.936622] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0,
size: 0, lookups: 22, failures: 22, skipped: 0
[2015-07-07 19:38:00.122471] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:38:00.122588] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:38:00.122698] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:38:48.385543] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89418.00 secs
[2015-07-07 19:38:48.385595] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0,
size: 0, lookups: 22, failures: 22, skipped: 0
[2015-07-07 19:40:00.138322] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:40:00.138515] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:40:00.138643] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:40:00.148329] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol
vol2-cold-dht file f1
[2015-07-07 19:40:00.148938] I [MSGID: 109038]
[tier.c:109:tier_check_same_node] 0-vol2-tier-dht: /f1 does not belong to this
node
[2015-07-07 19:40:00.149330] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:38:48.615115] I [MSGID: 109028]
[dht-rebalance.c:3029:gf_defrag_status_get] 0-glusterfs: Rebalance is in
progress. Time taken is 89418.00 secs
[2015-07-07 19:38:48.615118] I [MSGID: 109028]
[dht-rebalance.c:3033:gf_defrag_status_get] 0-glusterfs: Files migrated: 0,
size: 0, lookups: 22, failures: 22, skipped: 0
[2015-07-07 19:42:00.162120] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:42:00.162136] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6736
[2015-07-07 19:42:00.162471] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:42:00.162615] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:44:00.178019] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:44:00.178118] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:44:00.178182] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:46:00.192907] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:46:00.193013] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:46:00.193077] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:46:00.198343] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol
vol2-cold-dht file f1
[2015-07-07 19:46:00.198752] I [MSGID: 109038]
[tier.c:109:tier_check_same_node] 0-vol2-tier-dht: /f1 does not belong to this
node
[2015-07-07 19:46:00.198989] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:48:00.212514] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6736
[2015-07-07 19:48:00.212514] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:48:00.212662] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:48:00.212750] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:50:00.229031] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:50:00.229138] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:50:00.229285] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:52:00.242040] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:52:00.242198] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:52:00.242369] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:54:00.256207] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:54:00.256336] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:54:00.256428] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:56:00.270555] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:56:00.270645] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:56:00.270707] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 19:56:00.276106] I [MSGID: 109038]
[tier.c:350:tier_migrate_using_query_file] 0-vol2-tier-dht: Tier 1 src_subvol
vol2-cold-dht file newnew
[2015-07-07 19:56:00.276498] I [MSGID: 109038]
[tier.c:109:tier_check_same_node] 0-vol2-tier-dht: /newnew does not belong to
this node
[2015-07-07 19:56:00.276751] E [MSGID: 109037] [tier.c:908:tier_start]
0-vol2-tier-dht: Promotion failed!
[2015-07-07 19:58:00.291193] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 19:58:00.291327] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 19:58:00.291225] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/promotequeryfile-6736
[2015-07-07 19:58:00.291475] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 20:00:00.306421] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 20:00:00.306515] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 20:00:00.306574] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 20:02:00.319567] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 20:02:00.319670] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 20:02:00.319726] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
[2015-07-07 20:04:00.335432] I [MSGID: 109038]
[tier.c:572:tier_build_migration_qfile] 0-vol2-tier-dht: Failed to remove
/var/run/gluster/demotequeryfile-6736
[2015-07-07 20:04:00.335522] E [MSGID: 109051]
[tier.c:614:tier_migrate_files_using_qfile] 0-tier: Failed opening
/var/run/gluster/demotequeryfile-6736 for migration
[2015-07-07 20:04:00.335590] E [MSGID: 109037] [tier.c:898:tier_start]
0-vol2-tier-dht: Demotion failed!
--
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.
More information about the Bugs
mailing list