[Gluster-users] Hot Tier exceeding watermark-hi
Hari Gowtham
hgowtham at redhat.com
Sun Sep 30 18:35:15 UTC 2018
Hi,
The demotion has failed, so it's not moving files to cold tier any more.
Transport endpoint usually refers to the bricks or servers. Need to see if
the brick processes are up. The demotion failed because of the bricks being
unavailable.
Can you send the "gluster volume status" output again, the log files,
version of gluster and the operation performed on the volume.
>From the status output you sent earlier, it looks like all the processes
are running. But the log file says the other way. The log files and the
operations performed on the volume will help to identify what happened here.
If the bricks are down then the tier process might have gone down too. then
you can use " gluster volume tier <volname> start" to start the tier
process.
On Sun 30 Sep, 2018, 6:03 PM David Brown, <dbccemtp at gmail.com> wrote:
> Just found this in the tierd.log Not sure what it means or how to fix it
> tho, but I assume it may be the cause of my problem with files not being
> demoted from the hot tier..
>
>
> [2018-09-30 12:25:56.438821] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.440940] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
> (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.444633] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
> (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.448347] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
> (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.451919] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test-march-3-bc-file-501
> (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.456198] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //krishna
> (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.459702] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test1
> (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.463164] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
> (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.466601] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
> (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.470129] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
> (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.473758] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
> (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.477237] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
> (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.480696] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
> (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.484354] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
> (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.487884] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
> [2018-09-30 12:25:56.490973] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
> (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.494618] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
> (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.498059] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
> (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.501428] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //DBTest
> (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.504736] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //testCliff
> (cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file]
> 0-FFPrimary-tier-dht: Demotion failed for
> 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)"
> repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30
> 12:25:57.515127]
> [2018-09-30 12:25:57.520933] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
> [2018-09-30 12:25:57.521545] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
> [2018-09-30 12:25:57.521854] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.524079] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
> (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.527993] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
> (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.531829] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
> (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.535529] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test-march-3-bc-file-501
> (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.539852] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //krishna
> (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.543468] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test1
> (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.547051] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
> (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.550694] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
> (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.554498] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
> (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.558086] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
> (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.561568] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
> (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.565094] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
> (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.568562] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
> (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.571974] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
> [2018-09-30 12:25:57.574925] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
> (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.578639] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
> (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.582118] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
> (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.585547] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //DBTest
> (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> The message "E [MSGID: 109037] [tier.c:2532:tier_run]
> 0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30
> 12:25:54.323716] and [2018-09-30 12:25:57.592361]
> [2018-09-30 12:25:58.002724] I [MSGID: 0]
> [dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht:
> local subvol: FFPrimary-cold-dht,cnt = 272288612352
> [2018-09-30 12:25:58.018840] I [MSGID: 0]
> [dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht:
> Total size files = 272288612352
> [2018-09-30 12:25:58.018854] I
> [dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size
> =272288612352
> The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file]
> 0-FFPrimary-tier-dht: Failed to lookup file
> 6bed3d1e-17b8-11e8-a5dc-902b3450f388
> [No such file or directory]" repeated 4 times between [2018-09-30
> 12:25:54.247352] and [2018-09-30 12:25:58.599151]
> [2018-09-30 12:25:58.599162] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for
> 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
> [2018-09-30 12:25:58.604808] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
> [2018-09-30 12:25:58.605428] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
> [2018-09-30 12:25:58.605750] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.607865] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
> (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.611433] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
> (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.615059] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
> (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.618669] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test-march-3-bc-file-501
> (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.622667] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //krishna
> (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.626029] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test1
> (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.629459] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
> (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.632994] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
> (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.636669] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
> (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.640155] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
> (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.643516] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
> (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.648787] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
> (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.652106] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
> (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.655577] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
> [2018-09-30 12:25:58.658482] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
> (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.661703] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
> (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.665051] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
> (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285
> (edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]" repeated 3 times between [2018-09-30
> 12:25:55.406782] and [2018-09-30 12:25:58.674842]
> [2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run]
> 0-FFPrimary-tier-dht: Demotion failed
> [root at Glus1 FFPrimary]#
>
> On Sat, Sep 29, 2018 at 3:54 PM David Brown <dbccemtp at gmail.com> wrote:
>
>> 4 hours later, no files have been demoted
>>
>> [root at Glus1 ~]# gluster volume status FFPrimary detail
>> Status of volume: FFPrimary
>> Hot Bricks:
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
>> TCP Port : 49155
>> RDMA Port : 0
>> Online : Y
>> Pid : 24177
>> File System : xfs
>> Device : /dev/nvme0n1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 24.3GB
>> Total Disk Space : 476.7GB
>> Inode Count : 50877088
>> Free Inodes : 50874696
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
>> TCP Port : 49155
>> RDMA Port : 0
>> Online : Y
>> Pid : 17994
>> File System : xfs
>> Device : /dev/nvme0n1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 15.5GB
>> Total Disk Space : 476.7GB
>> Inode Count : 32560288
>> Free Inodes : 32557896
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
>> TCP Port : 49154
>> RDMA Port : 0
>> Online : Y
>> Pid : 23573
>> File System : xfs
>> Device : /dev/nvme0n1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 13.8GB
>> Total Disk Space : 476.7GB
>> Inode Count : 29027000
>> Free Inodes : 29024515
>> Cold Bricks:
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
>> TCP Port : 49152
>> RDMA Port : 0
>> Online : Y
>> Pid : 23442
>> File System : xfs
>> Device : /dev/sdb1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.5TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969488
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
>> TCP Port : 49153
>> RDMA Port : 0
>> Online : Y
>> Pid : 17856
>> File System : xfs
>> Device : /dev/sdb1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.5TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969489
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
>> TCP Port : 49153
>> RDMA Port : 0
>> Online : Y
>> Pid : 24018
>> File System : xfs
>> Device : /dev/sdb1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.5TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969488
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
>> TCP Port : 49153
>> RDMA Port : 0
>> Online : Y
>> Pid : 23518
>> File System : xfs
>> Device : /dev/sdc1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.7TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969607
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
>> TCP Port : 49154
>> RDMA Port : 0
>> Online : Y
>> Pid : 17943
>> File System : xfs
>> Device : /dev/sdc1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.7TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969607
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
>> TCP Port : 49154
>> RDMA Port : 0
>> Online : Y
>> Pid : 24108
>> File System : xfs
>> Device : /dev/sdc1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.7TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969604
>>
>> [root at Glus1 ~]# gluster volume info
>>
>> Volume Name: FFPrimary
>> Type: Tier
>> Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 9
>> Transport-type: tcp
>> Hot Tier :
>> Hot Tier Type : Replicate
>> Number of Bricks: 1 x 3 = 3
>> Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
>> Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
>> Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
>> Cold Tier:
>> Cold Tier Type : Distributed-Replicate
>> Number of Bricks: 2 x 3 = 6
>> Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
>> Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
>> Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
>> Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
>> Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
>> Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
>> Options Reconfigured:
>> cluster.tier-promote-frequency: 1800
>> cluster.tier-demote-frequency: 120
>> cluster.watermark-low: 60
>> cluster.watermark-hi: 80
>> performance.flush-behind: on
>> performance.cache-max-file-size: 128MB
>> performance.cache-size: 25GB
>> diagnostics.count-fop-hits: off
>> diagnostics.latency-measurement: off
>> cluster.tier-mode: cache
>> features.ctr-enabled: on
>> transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: off
>>
>> [root at Glus1 ~]# gluster volume tier FFPrimary status
>> Node Promoted files Demoted files Status
>> run time in h:m:s
>> --------- --------- --------- ---------
>> ---------
>> localhost 49 0 in
>> progress 5159:15:50
>> Glus2 0 0 in progress
>> 5159:15:50
>> Glus3 0 2075 in progress
>> 5159:15:52
>> Tiering Migration Functionality: FFPrimary: success
>> [root at Glus1 ~]#
>>
>>
>>
>> On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp at gmail.com> wrote:
>>
>>> Thank you Hari,
>>>
>>> I have set:
>>> cluster.tier-promote-frequency: 1800
>>> cluster.tier-demote-frequency: 120
>>>
>>> I will let you know if it makes a difference after some time. So far (10
>>> minutes), nothing has changed.
>>> I would agree with you, that by looking at the result of 'gluster volume
>>> tier FFPrimary status' it would seem that demoting is happening. However,
>>> for the last 24hrs, nothing has changed in the tier status report except
>>> the time. Could it be stuck? How would I know? Is there a way to restart it
>>> without restarting the cluster?
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham at redhat.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I can see that the demotion is happening from the status provided by
>>>> you.
>>>> Do verify it.
>>>> I would recommend you to change the cluster.tier-demote-frequency to 120
>>>> and cluster.tier-promote-frequency to 1800 to increase the demotions
>>>> until the
>>>> hot tier is emptied to a certain extent. Later you can use the values
>>>> existing now.
>>>> On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp at gmail.com> wrote:
>>>> >
>>>> > Hey Everyone,
>>>> >
>>>> > I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD
>>>> cold tier.
>>>> > I recently ran into some problems when the hot tier became full with
>>>> df-h showing 100%.
>>>> >
>>>> > I did not have a watermark-hi set, but it is my understanding that
>>>> 90% is the default. In an attempt to get the cluster to demote some files,
>>>> I set cluster.watermark-hi: 80 but it is still not demoting.
>>>> >
>>>> >
>>>> > [root at Glus1 ~]# gluster volume info
>>>> >
>>>> > Volume Name: FFPrimary
>>>> > Type: Tier
>>>> > Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
>>>> > Status: Started
>>>> > Snapshot Count: 0
>>>> > Number of Bricks: 9
>>>> > Transport-type: tcp
>>>> > Hot Tier :
>>>> > Hot Tier Type : Replicate
>>>> > Number of Bricks: 1 x 3 = 3
>>>> > Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
>>>> > Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
>>>> > Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
>>>> > Cold Tier:
>>>> > Cold Tier Type : Distributed-Replicate
>>>> > Number of Bricks: 2 x 3 = 6
>>>> > Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
>>>> > Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
>>>> > Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
>>>> > Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
>>>> > Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
>>>> > Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
>>>> > Options Reconfigured:
>>>> > cluster.tier-promote-frequency: 120
>>>> > cluster.tier-demote-frequency: 1800
>>>> > cluster.watermark-low: 60
>>>> > cluster.watermark-hi: 80
>>>> > performance.flush-behind: on
>>>> > performance.cache-max-file-size: 128MB
>>>> > performance.cache-size: 25GB
>>>> > diagnostics.count-fop-hits: off
>>>> > diagnostics.latency-measurement: off
>>>> > cluster.tier-mode: cache
>>>> > features.ctr-enabled: on
>>>> > transport.address-family: inet
>>>> > nfs.disable: on
>>>> > performance.client-io-threads: off
>>>> > [root at Glus1 ~]# gluster volume tier FFPrimary status
>>>> > Node Promoted files Demoted files
>>>> Status run time in h:m:s
>>>> > --------- --------- ---------
>>>> --------- ---------
>>>> > localhost 49 0 in
>>>> progress 5151:30:45
>>>> > Glus2 0 0 in progress
>>>> 5151:30:45
>>>> > Glus3 0 2075 in progress
>>>> 5151:30:47
>>>> > Tiering Migration Functionality: FFPrimary: success
>>>> > [root at Glus1 ~]#
>>>> >
>>>> > What can cause GlusterFS to stop demoting files and allow it to
>>>> completely fill the Hot Tier?
>>>> >
>>>> > Thank you!
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Gluster-users mailing list
>>>> > Gluster-users at gluster.org
>>>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Hari Gowtham.
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181001/22de888f/attachment.html>
More information about the Gluster-users
mailing list