[Gluster-users] Hot Tier exceeding watermark-hi
Amar Tumballi
atumball at redhat.com
Tue Oct 9 12:44:34 UTC 2018
Hi David,
Just for your information, as a project, we are not currently taking any
development focus on Tiering feature!
Please refer to email thread @
https://lists.gluster.org/pipermail//gluster-devel/2018-July/055017.html
It is recommended to use 'dmcache' on your disks to get best performance
out of your backend instead! Also note that this may get retired in next
upcoming release: https://review.gluster.org/21331
Hope this email will save you lot of time!
Regards,
Amar
On Sun, Sep 30, 2018 at 6:04 PM David Brown <dbccemtp at gmail.com> wrote:
> Just found this in the tierd.log Not sure what it means or how to fix it
> tho, but I assume it may be the cause of my problem with files not being
> demoted from the hot tier..
>
>
> [2018-09-30 12:25:56.438821] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.440940] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
> (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.444633] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
> (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.448347] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
> (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.451919] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test-march-3-bc-file-501
> (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.456198] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //krishna
> (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.459702] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test1
> (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.463164] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
> (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.466601] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
> (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.470129] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
> (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.473758] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
> (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.477237] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
> (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.480696] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
> (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.484354] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
> (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.487884] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
> [2018-09-30 12:25:56.490973] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
> (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.494618] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
> (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.498059] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
> (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.501428] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //DBTest
> (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:56.504736] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //testCliff
> (cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file]
> 0-FFPrimary-tier-dht: Demotion failed for
> 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)"
> repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30
> 12:25:57.515127]
> [2018-09-30 12:25:57.520933] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
> [2018-09-30 12:25:57.521545] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
> [2018-09-30 12:25:57.521854] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.524079] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
> (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.527993] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
> (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.531829] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
> (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.535529] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test-march-3-bc-file-501
> (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.539852] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //krishna
> (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.543468] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test1
> (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.547051] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
> (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.550694] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
> (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.554498] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
> (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.558086] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
> (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.561568] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
> (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.565094] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
> (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.568562] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
> (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.571974] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
> [2018-09-30 12:25:57.574925] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
> (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.578639] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
> (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.582118] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
> (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:57.585547] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //DBTest
> (c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> The message "E [MSGID: 109037] [tier.c:2532:tier_run]
> 0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30
> 12:25:54.323716] and [2018-09-30 12:25:57.592361]
> [2018-09-30 12:25:58.002724] I [MSGID: 0]
> [dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht:
> local subvol: FFPrimary-cold-dht,cnt = 272288612352
> [2018-09-30 12:25:58.018840] I [MSGID: 0]
> [dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht:
> Total size files = 272288612352
> [2018-09-30 12:25:58.018854] I
> [dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size
> =272288612352
> The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file]
> 0-FFPrimary-tier-dht: Failed to lookup file
> 6bed3d1e-17b8-11e8-a5dc-902b3450f388
> [No such file or directory]" repeated 4 times between [2018-09-30
> 12:25:54.247352] and [2018-09-30 12:25:58.599151]
> [2018-09-30 12:25:58.599162] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for
> 6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
> [2018-09-30 12:25:58.604808] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
> [2018-09-30 12:25:58.605428] W [MSGID: 114031]
> [client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> (10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
> [2018-09-30 12:25:58.605750] E [MSGID: 114031]
> [client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
> operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.607865] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
> (10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.611433] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
> (32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.615059] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
> (389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.618669] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test-march-3-bc-file-501
> (41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.622667] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //krishna
> (49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.626029] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //test1
> (5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.629459] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
> (5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.632994] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
> (6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.636669] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
> (7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.640155] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
> (8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.643516] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
> (91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.648787] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
> (972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.652106] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
> (a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.655577] I [MSGID: 109038]
> [tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
> failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
> [2018-09-30 12:25:58.658482] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
> (ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.661703] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
> (ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> [2018-09-30 12:25:58.665051] W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
> (c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]
> The message "W [MSGID: 114031]
> [client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
> remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285
> (edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid
> [Transport endpoint is not connected]" repeated 3 times between [2018-09-30
> 12:25:55.406782] and [2018-09-30 12:25:58.674842]
> [2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run]
> 0-FFPrimary-tier-dht: Demotion failed
> [root at Glus1 FFPrimary]#
>
> On Sat, Sep 29, 2018 at 3:54 PM David Brown <dbccemtp at gmail.com> wrote:
>
>> 4 hours later, no files have been demoted
>>
>> [root at Glus1 ~]# gluster volume status FFPrimary detail
>> Status of volume: FFPrimary
>> Hot Bricks:
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
>> TCP Port : 49155
>> RDMA Port : 0
>> Online : Y
>> Pid : 24177
>> File System : xfs
>> Device : /dev/nvme0n1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 24.3GB
>> Total Disk Space : 476.7GB
>> Inode Count : 50877088
>> Free Inodes : 50874696
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
>> TCP Port : 49155
>> RDMA Port : 0
>> Online : Y
>> Pid : 17994
>> File System : xfs
>> Device : /dev/nvme0n1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 15.5GB
>> Total Disk Space : 476.7GB
>> Inode Count : 32560288
>> Free Inodes : 32557896
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
>> TCP Port : 49154
>> RDMA Port : 0
>> Online : Y
>> Pid : 23573
>> File System : xfs
>> Device : /dev/nvme0n1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 13.8GB
>> Total Disk Space : 476.7GB
>> Inode Count : 29027000
>> Free Inodes : 29024515
>> Cold Bricks:
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
>> TCP Port : 49152
>> RDMA Port : 0
>> Online : Y
>> Pid : 23442
>> File System : xfs
>> Device : /dev/sdb1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.5TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969488
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
>> TCP Port : 49153
>> RDMA Port : 0
>> Online : Y
>> Pid : 17856
>> File System : xfs
>> Device : /dev/sdb1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.5TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969489
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
>> TCP Port : 49153
>> RDMA Port : 0
>> Online : Y
>> Pid : 24018
>> File System : xfs
>> Device : /dev/sdb1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.5TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969488
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
>> TCP Port : 49153
>> RDMA Port : 0
>> Online : Y
>> Pid : 23518
>> File System : xfs
>> Device : /dev/sdc1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.7TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969607
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
>> TCP Port : 49154
>> RDMA Port : 0
>> Online : Y
>> Pid : 17943
>> File System : xfs
>> Device : /dev/sdc1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.7TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969607
>>
>> ------------------------------------------------------------------------------
>> Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
>> TCP Port : 49154
>> RDMA Port : 0
>> Online : Y
>> Pid : 24108
>> File System : xfs
>> Device : /dev/sdc1
>> Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
>> Inode Size : 512
>> Disk Space Free : 2.7TB
>> Total Disk Space : 2.7TB
>> Inode Count : 292971904
>> Free Inodes : 292969604
>>
>> [root at Glus1 ~]# gluster volume info
>>
>> Volume Name: FFPrimary
>> Type: Tier
>> Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
>> Status: Started
>> Snapshot Count: 0
>> Number of Bricks: 9
>> Transport-type: tcp
>> Hot Tier :
>> Hot Tier Type : Replicate
>> Number of Bricks: 1 x 3 = 3
>> Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
>> Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
>> Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
>> Cold Tier:
>> Cold Tier Type : Distributed-Replicate
>> Number of Bricks: 2 x 3 = 6
>> Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
>> Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
>> Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
>> Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
>> Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
>> Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
>> Options Reconfigured:
>> cluster.tier-promote-frequency: 1800
>> cluster.tier-demote-frequency: 120
>> cluster.watermark-low: 60
>> cluster.watermark-hi: 80
>> performance.flush-behind: on
>> performance.cache-max-file-size: 128MB
>> performance.cache-size: 25GB
>> diagnostics.count-fop-hits: off
>> diagnostics.latency-measurement: off
>> cluster.tier-mode: cache
>> features.ctr-enabled: on
>> transport.address-family: inet
>> nfs.disable: on
>> performance.client-io-threads: off
>>
>> [root at Glus1 ~]# gluster volume tier FFPrimary status
>> Node Promoted files Demoted files Status
>> run time in h:m:s
>> --------- --------- --------- ---------
>> ---------
>> localhost 49 0 in
>> progress 5159:15:50
>> Glus2 0 0 in progress
>> 5159:15:50
>> Glus3 0 2075 in progress
>> 5159:15:52
>> Tiering Migration Functionality: FFPrimary: success
>> [root at Glus1 ~]#
>>
>>
>>
>> On Sat, Sep 29, 2018 at 11:56 AM David Brown <dbccemtp at gmail.com> wrote:
>>
>>> Thank you Hari,
>>>
>>> I have set:
>>> cluster.tier-promote-frequency: 1800
>>> cluster.tier-demote-frequency: 120
>>>
>>> I will let you know if it makes a difference after some time. So far (10
>>> minutes), nothing has changed.
>>> I would agree with you, that by looking at the result of 'gluster volume
>>> tier FFPrimary status' it would seem that demoting is happening. However,
>>> for the last 24hrs, nothing has changed in the tier status report except
>>> the time. Could it be stuck? How would I know? Is there a way to restart it
>>> without restarting the cluster?
>>>
>>>
>>>
>>>
>>>
>>> On Sat, Sep 29, 2018 at 11:08 AM Hari Gowtham <hgowtham at redhat.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I can see that the demotion is happening from the status provided by
>>>> you.
>>>> Do verify it.
>>>> I would recommend you to change the cluster.tier-demote-frequency to 120
>>>> and cluster.tier-promote-frequency to 1800 to increase the demotions
>>>> until the
>>>> hot tier is emptied to a certain extent. Later you can use the values
>>>> existing now.
>>>> On Sat, Sep 29, 2018 at 5:39 PM David Brown <dbccemtp at gmail.com> wrote:
>>>> >
>>>> > Hey Everyone,
>>>> >
>>>> > I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD
>>>> cold tier.
>>>> > I recently ran into some problems when the hot tier became full with
>>>> df-h showing 100%.
>>>> >
>>>> > I did not have a watermark-hi set, but it is my understanding that
>>>> 90% is the default. In an attempt to get the cluster to demote some files,
>>>> I set cluster.watermark-hi: 80 but it is still not demoting.
>>>> >
>>>> >
>>>> > [root at Glus1 ~]# gluster volume info
>>>> >
>>>> > Volume Name: FFPrimary
>>>> > Type: Tier
>>>> > Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
>>>> > Status: Started
>>>> > Snapshot Count: 0
>>>> > Number of Bricks: 9
>>>> > Transport-type: tcp
>>>> > Hot Tier :
>>>> > Hot Tier Type : Replicate
>>>> > Number of Bricks: 1 x 3 = 3
>>>> > Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
>>>> > Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
>>>> > Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
>>>> > Cold Tier:
>>>> > Cold Tier Type : Distributed-Replicate
>>>> > Number of Bricks: 2 x 3 = 6
>>>> > Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
>>>> > Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
>>>> > Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
>>>> > Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
>>>> > Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
>>>> > Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
>>>> > Options Reconfigured:
>>>> > cluster.tier-promote-frequency: 120
>>>> > cluster.tier-demote-frequency: 1800
>>>> > cluster.watermark-low: 60
>>>> > cluster.watermark-hi: 80
>>>> > performance.flush-behind: on
>>>> > performance.cache-max-file-size: 128MB
>>>> > performance.cache-size: 25GB
>>>> > diagnostics.count-fop-hits: off
>>>> > diagnostics.latency-measurement: off
>>>> > cluster.tier-mode: cache
>>>> > features.ctr-enabled: on
>>>> > transport.address-family: inet
>>>> > nfs.disable: on
>>>> > performance.client-io-threads: off
>>>> > [root at Glus1 ~]# gluster volume tier FFPrimary status
>>>> > Node Promoted files Demoted files
>>>> Status run time in h:m:s
>>>> > --------- --------- ---------
>>>> --------- ---------
>>>> > localhost 49 0 in
>>>> progress 5151:30:45
>>>> > Glus2 0 0 in progress
>>>> 5151:30:45
>>>> > Glus3 0 2075 in progress
>>>> 5151:30:47
>>>> > Tiering Migration Functionality: FFPrimary: success
>>>> > [root at Glus1 ~]#
>>>> >
>>>> > What can cause GlusterFS to stop demoting files and allow it to
>>>> completely fill the Hot Tier?
>>>> >
>>>> > Thank you!
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Gluster-users mailing list
>>>> > Gluster-users at gluster.org
>>>> > https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Hari Gowtham.
>>>>
>>> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users
--
Amar Tumballi (amarts)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20181009/ddc96d3e/attachment.html>
More information about the Gluster-users
mailing list