<div dir="ltr">Milind - Thank you for your help, I appreciate it..<div><br></div><div>It appears that the tiering behaves the same when quota is turned off, info:</div><div><br></div><div><div># gluster vol info <vol></div><div> </div><div>Volume Name: <vol></div><div>Type: Tier</div><div>Volume ID: 7710ed2f-775e-4dd9-92ad-66407c72b0ad</div><div>Status: Started</div><div>Snapshot Count: 0</div><div>Number of Bricks: 8</div><div>Transport-type: tcp</div><div>Hot Tier :</div><div>Hot Tier Type : Distributed-Replicate</div><div>Number of Bricks: 2 x 2 = 4</div><div>Brick1: <node2>:/mnt/brick_nvme1/brick</div><div>Brick2: <node1>:/mnt/brick_nvme2/brick</div><div>Brick3: <node2>:/mnt/brick_nvme2/brick</div><div>Brick4: <node1>:/mnt/brick_nvme1/brick</div><div>Cold Tier:</div><div>Cold Tier Type : Distributed-Replicate</div><div>Number of Bricks: 2 x 2 = 4</div><div>Brick5: <node1>:/mnt/brick1/brick</div><div>Brick6: <node2>:/mnt/brick2/brick</div><div>Brick7: <node1>:/mnt/brick2/brick</div><div>Brick8: <node2>:/mnt/brick1/brick</div><div>Options Reconfigured:</div><div>cluster.lookup-optimize: on</div><div>client.event-threads: 4</div><div>server.event-threads: 4</div><div>performance.write-behind-window-size: 4MB</div><div>performance.cache-size: 16GB</div><div>features.inode-quota: off</div><div>features.quota: off</div><div>nfs.disable: on</div><div>transport.address-family: inet</div><div>features.ctr-enabled: on</div><div>cluster.tier-mode: cache</div><div>performance.io-cache: off</div><div>performance.quick-read: off</div><div>cluster.tier-max-files: 1000000</div></div><div><br></div><div>Errors in /var/log/glusterfs/tier/<vol>/tierd.log on node1 after turning off quota:</div><div><br></div><div><div>[2017-10-27 18:38:08.880502] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/83540503.jpg</div><div>[2017-10-27 18:38:08.880686] E [MSGID: 109023] [dht-rebalance.c:757:__dht_rebalance_create_dst_file] 0-<vol>-tier-dht: failed to create /path/to/83540503.jpg on <vol>-hot-dht [Input/output error]</div><div>[2017-10-27 18:38:08.880717] E [MSGID: 0] [dht-rebalance.c:1696:dht_migrate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - /path/to/83540503.jpg</div><div>[2017-10-27 18:38:08.881101] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate /path/to/83540503.jpg [No space left on device]</div><div>[2017-10-27 18:38:08.881145] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for 83540503.jpg(gfid:00cf352a-0a21-42d3-91ae-fe6fc63fac9d)</div><div>[2017-10-27 18:38:08.891692] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/152640504.jpg</div><div>[2017-10-27 18:38:08.891876] E [MSGID: 109023] [dht-rebalance.c:757:__dht_rebalance_create_dst_file] 0-<vol>-tier-dht: failed to create /path/to/152640504.jpg on <vol>-hot-dht [Input/output error]</div><div>[2017-10-27 18:38:08.891899] E [MSGID: 0] [dht-rebalance.c:1696:dht_migrate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - /path/to/152640504.jpg</div><div>[2017-10-27 18:38:08.920077] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate /path/to/152640504.jpg [No space left on device]</div><div>[2017-10-27 18:38:08.920121] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for 152640504.jpg(gfid:0436b8b5-2e15-411e-acfa-a5870cf125bf)</div><div>[2017-10-27 18:38:08.952939] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/89240318.jpg</div><div>[2017-10-27 18:38:08.953121] E [MSGID: 109023] [dht-rebalance.c:757:__dht_rebalance_create_dst_file] 0-<vol>-tier-dht: failed to create /path/to/89240318.jpg on <vol>-hot-dht [Input/output error]</div><div>[2017-10-27 18:38:08.953147] E [MSGID: 0] [dht-rebalance.c:1696:dht_migrate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - /path/to/89240318.jpg</div><div>[2017-10-27 18:38:08.959510] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate /path/to/89240318.jpg [No space left on device]</div><div>[2017-10-27 18:38:08.959560] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for 89240318.jpg(gfid:1143c9bb-ea79-4c15-ad03-97a611d53135)</div><div>[2017-10-27 18:38:08.986665] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/106056906.jpg</div><div>[2017-10-27 18:38:08.986871] E [MSGID: 109023] [dht-rebalance.c:757:__dht_rebalance_create_dst_file] 0-<vol>-tier-dht: failed to create /path/to/106056906.jpg on <vol>-hot-dht [Input/output error]</div><div>[2017-10-27 18:38:08.986904] E [MSGID: 0] [dht-rebalance.c:1696:dht_migrate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - /path/to/106056906.jpg</div><div>[2017-10-27 18:38:08.991468] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate /path/to/106056906.jpg [No space left on device]</div><div>[2017-10-27 18:38:08.991505] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for 106056906.jpg(gfid:07f5e5d4-315f-4299-a62f-6bd8f159c89d)</div><div>[2017-10-27 18:38:09.025433] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/114649988.jpg</div></div><div><br></div><div>I wanted to add a couple data points here:</div><div><br></div><div>- Most (95%) of the logging is logged to node1 of the 2 node cluster. </div><div><br></div><div> The tierd.log file on node1 is 588M in size due to all of the failure errors. The tierd.log file on node2 is only ~205K in size.</div><div> I believe I posted earlier that all promoted files are listed on node1:</div><div><br></div><div> # gluster vol tier <vol> status</div><div> Node Promoted files Demoted files Status run time in h:m:s </div><div> ------ --------- --------- --------- --------- </div><div> <node2> 0 0 in progress 601:37:43</div><div> <node1> 271966 0 in progress 601:37:42</div><div><br></div><div> Is this expected behavior?</div><div><br></div><div>- We are sharing the data (the same share) via SMB and AFP to be accessed by PC's and Mac's. The Mac's are using AFP since they have so much difficultly with SMB and network file shares.</div><div><br></div><div> I know the Mac's create all kinds of 'special' files when working on the share, could there be a problem with certain files and tiering? For example (from node2 tierd.log):</div><div> </div><div> [2017-10-26 19:30:08.147159] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for .DS_Store(gfid:db430070-b9c5-4bd2-b4c6-a347b838a97e)</div><div> [2017-10-26 22:28:08.218565] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for .DS_Store(gfid:f745bea6-04bd-4904-8237-1bd7c9c92f5b)</div><div> [2017-10-26 22:28:08.221909] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for .DS_Store(gfid:bed73314-8740-4822-9fb7-95257434e283)</div><div> [2017-10-26 22:28:08.223767] I [MSGID: 109038] [tier.c:1169:tier_migrate_using_query_file] 0-<vol>-tier-dht: Promotion failed for .DS_Store(gfid:bf1df49b-c264-449d-9bc6-65bcfd48fa4e)</div><div><br></div><div> The .DS_Store files are Mac specific files..</div><div><br></div><div> Since users work directly off of the share, are there potential problems with tiering and locks? I do see warnings (on node1 tierd.log):</div><div><br></div><div> [2017-10-27 18:30:08.719976] W [MSGID: 109023] [dht-rebalance.c:639:__is_file_migratable] 0-<vol>-tier-dht: Migrate file failed: /path/to/<a href="http://file.ai">file.ai</a>: File has locks. Skipping file migration</div><div> [2017-10-27 18:32:08.483971] W [MSGID: 109023] [dht-rebalance.c:639:__is_file_migratable] 0-<vol>-tier-dht: Migrate file failed: /path/to/<a href="http://file-v1.ai">file-v1.ai</a>: File has locks. Skipping file migration</div><div><br></div><div><br></div><div>- The directory structure (over the many years) has spaces in the names of files and folders, sometimes I'm finding, even at the end of a file.</div><div><br></div><div> Could spaces in names of files and folders be causing issues with tiering?</div><div><br></div><div><br></div><div>I'm still not sure what the [No space left on device] messages are coming from as it does not appear that there are any space issues. Even before I turned off quota on the volume the sizing appeared to be fine:</div><div><br></div><div><br></div><div><div># gluster vol quota <vol> list</div><div> Path Hard-limit Soft-limit Used Available Soft-limit exceeded? Hard-limit exceeded?</div><div>-------------------------------------------------------------------------------------------------------------------------------</div><div>/path1 500.0GB 80%(400.0GB) 1.9MB 500.0GB No No</div><div>/path2 25.0TB 80%(20.0TB) 19.2TB 5.8TB No No</div></div><div><br></div><div><br></div><div>I will have some time this weekend to take the shares offline. Are there any steps I can take to clean up the hot tier, resync, or other, to ensure all is in a good state?</div><div><br></div><div>Thanks in advance..</div><div><br></div><div>HB</div><div><br></div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 26, 2017 at 9:17 PM, Milind Changire <span dir="ltr"><<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Herb,<br></div>I'm trying to weed out issues here.<br><br></div>So, I can see quota turned <b>on</b> and would like you to check the quota settings and test to see system behavior <b>if quota is turned off</b>.<br></div><div><br></div><div>Although the file size that failed migration was 29K, I'm being a bit paranoid while weeding out issues.</div><div><br></div><div>Are you still facing tiering errors ?</div><div>I can see your response to Alex with the disk space consumption and found it a bit ambiguous w.r.t. state of affairs.<br></div><div><br></div><div>--<br></div>Milind<br><br><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 24, 2017 at 11:34 PM, Herb Burnswell <span dir="ltr"><<a href="mailto:herbert.burnswell@gmail.com" target="_blank">herbert.burnswell@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div style="font-size:12.8px"><div>Milind - Thank you for the response..</div><span><div><br></div><div>>> What are the high and low watermarks for the tier set at ?<br><br></div><div style="font-size:small"># gluster volume get <vol> cluster.watermark-hi</div></span><div style="font-size:small">Option Value </div><div style="font-size:small">------ ----- </div><div style="font-size:small">cluster.watermark-hi 90 </div><span><div style="font-size:small"><br></div><div style="font-size:small"># gluster volume get <vol> cluster.watermark-low</div></span><div style="font-size:small">Option Value </div><div style="font-size:small">------ ----- </div><div style="font-size:small">cluster.watermark-low 75 </div><div><br></div></div><span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">>> What is the size of the file that failed to migrate as per the following tierd log:</div></span><span class="m_-5716082969068876722m_5508430874479203959gmail-im" style="font-size:12.8px"><span><div><div style="font-size:12.8px"><br></div><div>>> [2017-10-19 17:52:07.519614] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:edaf97e1-02e0-4838<wbr>-9d26-71ea3aab22fb)</div></div><div><br></div></span><div>The file was a word doc @ 29K in size.</div><div><br></div></span><span><div style="font-size:12.8px">>>If possible, a <b>gluster volume info</b> would also help, instead of going to and fro with questions.<br></div><div style="font-size:12.8px"><br></div></span><div style="font-size:12.8px"><div style="font-size:small"># gluster vol info</div><div style="font-size:small"> </div><div style="font-size:small">Volume Name: ctdb</div><div style="font-size:small">Type: Replicate</div><div style="font-size:small">Volume ID: f679c476-e0dd-4f3a-9813-1b2601<wbr>6b5384</div><div style="font-size:small">Status: Started</div><div style="font-size:small">Snapshot Count: 0</div><div style="font-size:small">Number of Bricks: 1 x 2 = 2</div><div style="font-size:small">Transport-type: tcp</div><div style="font-size:small">Bricks:</div><div style="font-size:small">Brick1: <node1>:/mnt/ctdb_local/brick</div><div style="font-size:small">Brick2: <node2>:/mnt/ctdb_local/brick</div><div style="font-size:small">Options Reconfigured:</div><div style="font-size:small">nfs.disable: on</div><div style="font-size:small">transport.address-family: inet</div><div style="font-size:small"> </div><div style="font-size:small">Volume Name: <vol></div><div style="font-size:small">Type: Tier</div><div style="font-size:small">Volume ID: 7710ed2f-775e-4dd9-92ad-66407c<wbr>72b0ad</div><div style="font-size:small">Status: Started</div><div style="font-size:small">Snapshot Count: 0</div><div style="font-size:small">Number of Bricks: 8</div><div style="font-size:small">Transport-type: tcp</div><div style="font-size:small">Hot Tier :</div><div style="font-size:small">Hot Tier Type : Distributed-Replicate</div><div style="font-size:small">Number of Bricks: 2 x 2 = 4</div><div style="font-size:small">Brick1: <node2>:/mnt/brick_nvme1/brick</div><div style="font-size:small">Brick2: <node1>:/mnt/brick_nvme2/brick</div><div style="font-size:small">Brick3: <node2>:/mnt/brick_nvme2/brick</div><div style="font-size:small">Brick4: <node1>:/mnt/brick_nvme1/brick</div><div style="font-size:small">Cold Tier:</div><div style="font-size:small">Cold Tier Type : Distributed-Replicate</div><div style="font-size:small">Number of Bricks: 2 x 2 = 4</div><div style="font-size:small">Brick5: <node1>:/mnt/brick1/brick</div><div style="font-size:small">Brick6: <node2>:/mnt/brick2/brick</div><div style="font-size:small">Brick7: <node1>:/mnt/brick2/brick</div><div style="font-size:small">Brick8: <node2>:/mnt/brick1/brick</div><div style="font-size:small">Options Reconfigured:</div><div style="font-size:small">cluster.lookup-optimize: on</div><div style="font-size:small">client.event-threads: 4</div><div style="font-size:small">server.event-threads: 4</div><div style="font-size:small">performance.write-behind-windo<wbr>w-size: 4MB</div><div style="font-size:small">performance.cache-size: 16GB</div><div style="font-size:small">features.quota-deem-statfs: on</div><div style="font-size:small">features.inode-quota: on</div><div style="font-size:small">features.quota: on</div><div style="font-size:small">nfs.disable: on</div><div style="font-size:small">transport.address-family: inet</div><div style="font-size:small">features.ctr-enabled: on</div><div style="font-size:small">cluster.tier-mode: cache</div><div style="font-size:small">performance.io-cache: off</div><div style="font-size:small">performance.quick-read: off</div><div style="font-size:small">cluster.tier-max-files: 1000000</div></div></div><span class="m_-5716082969068876722HOEnZb"><font color="#888888"><div><br></div><div><br></div><div>HB</div><div> <br></div><div><br></div><div><br></div></font></span></div><div class="m_-5716082969068876722HOEnZb"><div class="m_-5716082969068876722h5"><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Oct 22, 2017 at 8:41 AM, Milind Changire <span dir="ltr"><<a href="mailto:mchangir@redhat.com" target="_blank">mchangir@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Herb,</div><div>What are the high and low watermarks for the tier set at ?<br><br></div># gluster volume get <vol> cluster.watermark-hi<br><br></div># gluster volume get <vol> cluster.watermark-low</div><div><br></div><div>What is the size of the file that failed to migrate as per the following tierd log:</div><span><div><div style="font-size:12.8px"><br></div><div>[2017-10-19 17:52:07.519614] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:edaf97e1-02e0-4838<wbr>-9d26-71ea3aab22fb)</div></div><div><br></div></span><div>If possible, a <b>gluster volume info</b> would also help, instead of going to and fro with questions.<br></div><div><br></div><div>--</div><div>Milind</div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="m_-5716082969068876722m_5508430874479203959h5">On Fri, Oct 20, 2017 at 12:42 AM, Herb Burnswell <span dir="ltr"><<a href="mailto:herbert.burnswell@gmail.com" target="_blank">herbert.burnswell@gmail.com</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_-5716082969068876722m_5508430874479203959h5"><div dir="ltr"><span style="font-size:12.8px">All,</span><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I am new to gluster and have some questions/concerns about some tiering errors that I see in the log files.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">OS: CentOs 7.3.1611</div><div style="font-size:12.8px">Gluster version: 3.10.5</div><div style="font-size:12.8px">Samba version: 4.6.2</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">I see the following (scrubbed):</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Node 1 /var/log/glusterfs/tier/<vol>/<wbr>tierd.log:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>[2017-10-19 17:52:07.519614] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:edaf97e1-02e0-4838<wbr>-9d26-71ea3aab22fb)</div><div>[2017-10-19 17:52:07.525110] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/<file></div><div>[2017-10-19 17:52:07.526088] E [MSGID: 109023] [dht-rebalance.c:757:__dht_reb<wbr>alance_create_dst_file] 0-<vol>-tier-dht: failed to create <file> on <vol>-hot-dht [Input/output error]</div><div>[2017-10-19 17:52:07.526111] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - <file></div><div>[2017-10-19 17:52:07.527214] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate <file> [No space left on device]</div><div>[2017-10-19 17:52:07.527244] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:fb4411c4-a387-4e5f<wbr>-a2b7-897633ef4aa8)</div><div>[2017-10-19 17:52:07.533510] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/<file></div><div>[2017-10-19 17:52:07.534434] E [MSGID: 109023] [dht-rebalance.c:757:__dht_reb<wbr>alance_create_dst_file] 0-<vol>-tier-dht: failed to create <file> on <vol>-hot-dht [Input/output error]</div><div>[2017-10-19 17:52:07.534453] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - <file></div><div>[2017-10-19 17:52:07.535570] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate <file> [No space left on device]</div><div>[2017-10-19 17:52:07.535594] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:fba421e7-0500-47c4<wbr>-bf67-10a40690e13d)</div><div>[2017-10-19 17:52:07.541363] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/<file></div><div>[2017-10-19 17:52:07.542296] E [MSGID: 109023] [dht-rebalance.c:757:__dht_reb<wbr>alance_create_dst_file] 0-<vol>-tier-dht: failed to create <file> on <vol>-hot-dht [Input/output error]</div><div>[2017-10-19 17:52:07.542357] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - <file></div><div>[2017-10-19 17:52:07.543480] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate <file> [No space left on device]</div><div>[2017-10-19 17:52:07.543521] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:fe6799e1-42e6-43e5<wbr>-a7eb-ac8facfcbc9f)</div><div>[2017-10-19 17:52:07.549959] E [MSGID: 109011] [dht-common.c:7188:dht_create] 0-<vol>-hot-dht: no subvolume in layout for path=/path/to/<file></div><div>[2017-10-19 17:52:07.550901] E [MSGID: 109023] [dht-rebalance.c:757:__dht_reb<wbr>alance_create_dst_file] 0-<vol>-tier-dht: failed to create <file> on <vol>-hot-dht [Input/output error]</div><div>[2017-10-19 17:52:07.550922] E [MSGID: 0] [dht-rebalance.c:1696:dht_migr<wbr>ate_file] 0-<vol>-tier-dht: Create dst failed on - <vol>-hot-dht for file - <file></div><div>[2017-10-19 17:52:07.551896] E [MSGID: 109037] [tier.c:969:tier_migrate_link] 0-<vol>-tier-dht: Failed to migrate <file> [No space left on device]</div><div>[2017-10-19 17:52:07.551917] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:ffe3a3f2-b170-43f0<wbr>-a9fb-97c78e3173eb)</div><div>[2017-10-19 17:52:07.551945] E [MSGID: 109037] [tier.c:2565:tier_run] 0-<vol>-tier-dht: Promotion failed</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Node 1 /var/log/samba/glusterfs-<vol><wbr>-pool.log:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>[2017-10-18 17:13:41.481860] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-0: remote operation failed. Path: /pool/testing (7d89b9a8-3e5d-4f28-9e57-039fe<wbr>4416994) [Invalid argument]</div><div>[2017-10-18 17:13:41.481860] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-1: remote operation failed. Path: /pool/testing (7d89b9a8-3e5d-4f28-9e57-039fe<wbr>4416994) [Invalid argument]</div><div>[2017-10-18 17:13:41.485916] E [MSGID: 109089] [dht-helper.c:517:dht_check_an<wbr>d_open_fd_on_subvol_task] 0-<vol>-tier-dht: Failed to open the fd (0x7f02bf1ff570, flags=00) on file 7d89b9a8-3e5d-4f28-9e57-039fe4<wbr>416994 @ <vol>-cold-dht [Invalid argument]</div><div>[2017-10-18 17:13:41.488223] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-0: remote operation failed. Path: /pool/testing (7d89b9a8-3e5d-4f28-9e57-039fe<wbr>4416994) [Invalid argument]</div><div>[2017-10-18 17:13:41.488235] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-1: remote operation failed. Path: /pool/testing (7d89b9a8-3e5d-4f28-9e57-039fe<wbr>4416994) [Invalid argument]</div><div>[2017-10-18 17:13:41.489060] E [MSGID: 109089] [dht-helper.c:517:dht_check_an<wbr>d_open_fd_on_subvol_task] 0-<vol>-tier-dht: Failed to open the fd (0x7f02bf1feb50, flags=00) on file 7d89b9a8-3e5d-4f28-9e57-039fe4<wbr>416994 @ <vol>-cold-dht [Invalid argument]</div><div>[2017-10-18 17:13:42.339936] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-4: remote operation failed. Path: /pool (34d76e11-412f-4bc6-9a3e-b1f89<wbr>658f13b) [Invalid argument]</div><div>[2017-10-18 17:13:42.339988] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-5: remote operation failed. Path: /pool (34d76e11-412f-4bc6-9a3e-b1f89<wbr>658f13b) [Invalid argument]</div><div>[2017-10-18 17:13:42.343769] E [MSGID: 109089] [dht-helper.c:517:dht_check_an<wbr>d_open_fd_on_subvol_task] 0-<vol>-tier-dht: Failed to open the fd (0x7f02bf2012c0, flags=00) on file 34d76e11-412f-4bc6-9a3e-b1f896<wbr>58f13b @ <vol>-hot-dht [Invalid argument]</div><div>[2017-10-18 17:13:42.345374] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-4: remote operation failed. Path: /pool (34d76e11-412f-4bc6-9a3e-b1f89<wbr>658f13b) [Invalid argument]</div><div>[2017-10-18 17:13:42.345401] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-5: remote operation failed. Path: /pool (34d76e11-412f-4bc6-9a3e-b1f89<wbr>658f13b) [Invalid argument]</div><div>[2017-10-18 17:13:42.346259] E [MSGID: 109089] [dht-helper.c:517:dht_check_an<wbr>d_open_fd_on_subvol_task] 0-<vol>-tier-dht: Failed to open the fd (0x7f02bf201130, flags=00) on file 34d76e11-412f-4bc6-9a3e-b1f896<wbr>58f13b @ <vol>-hot-dht [Invalid argument]</div><div>[2017-10-18 17:13:59.541591] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.</div><div>[2017-10-18 17:13:59.541748] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.</div><div>[2017-10-18 17:13:59.541887] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-2: All subvolumes are down. Going offline until atleast one of them comes back up.</div><div>[2017-10-18 17:13:59.541977] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-3: All subvolumes are down. Going offline until atleast one of them comes back up.</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Node 2 /var/log/gluster/tier/<vol>/ti<wbr>erd.log:<br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>[2017-10-16 15:54:08.662873] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:fffd714e-b2d2-42d3<wbr>-a31f-72673276e3d0)</div><div>[2017-10-16 16:00:07.201584] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:f10365e1-747b-4985<wbr>-97b9-8b5dc61ac464)</div><div>[2017-10-16 16:00:07.372559] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:f95f17bf-b696-44cd<wbr>-aae0-d8ac38149aa5)</div><div>[2017-10-16 16:06:06.880522] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:ec451f6c-8971-4f9b<wbr>-a04f-00f96db9b46a)</div><div>[2017-10-16 16:06:08.062080] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:e658cd70-3f6d-4b25<wbr>-8d9f-0d4c24d3ec5d)</div><div>[2017-10-16 16:06:08.288298] I [MSGID: 109038] [tier.c:1169:tier_migrate_usin<wbr>g_query_file] 0-<vol>-tier-dht: Promotion failed for <file>(gfid:f22df67a-88e5-4fae<wbr>-aab0-b00e04f9a6e1)</div><div>[2017-10-18 15:55:06.446416] I [MSGID: 109028] [dht-rebalance.c:4792:gf_defra<wbr>g_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1376671.00 secs</div><div>[2017-10-18 15:55:06.446433] I [MSGID: 109028] [dht-rebalance.c:4796:gf_defra<wbr>g_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 47887089, failures: 3594, skipped: 0</div><div>[2017-10-19 00:00:00.501576] I [MSGID: 109038] [tier.c:2391:tier_prepare_comp<wbr>act] 0-<vol>-tier-dht: Start compaction on cold tier</div><div>[2017-10-19 00:00:00.502016] I [MSGID: 109038] [tier.c:2403:tier_prepare_comp<wbr>act] 0-<vol>-tier-dht: End compaction on cold tier</div><div>[2017-10-19 00:00:00.501608] I [MSGID: 109038] [tier.c:2391:tier_prepare_comp<wbr>act] 0-<vol>-tier-dht: Start compaction on cold tier</div><div>[2017-10-19 00:00:00.502076] I [MSGID: 109038] [tier.c:2403:tier_prepare_comp<wbr>act] 0-<vol>-tier-dht: End compaction on cold tier</div><div>[2017-10-19 16:03:49.522991] I [MSGID: 109028] [dht-rebalance.c:4792:gf_defra<wbr>g_status_get] 0-glusterfs: Rebalance is in progress. Time taken is 1463594.00 secs</div><div>[2017-10-19 16:03:49.523017] I [MSGID: 109028] [dht-rebalance.c:4796:gf_defra<wbr>g_status_get] 0-glusterfs: Files migrated: 0, size: 0, lookups: 52790654, failures: 3594, skipped: 0</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Node 2 /var/log/samba/glusterfs-<vol><wbr>-pool.log:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>[2017-10-18 16:49:09.218062] E [MSGID: 114031] [client-rpc-fops.c:443:client3<wbr>_3_open_cbk] 0-<vol>-client-4: remote operation failed. Path: /pool (34d76e11-412f-4bc6-9a3e-b1f89<wbr>658f13b) [Invalid argument]</div><div>[2017-10-18 16:49:09.218254] E [MSGID: 109089] [dht-helper.c:517:dht_check_an<wbr>d_open_fd_on_subvol_task] 0-<vol>-tier-dht: Failed to open the fd (0x7f009b36bac0, flags=00) on file 34d76e11-412f-4bc6-9a3e-b1f896<wbr>58f13b @ <vol>-hot-dht [Invalid argument]</div><div>[2017-10-18 16:49:09.222783] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-0: All subvolumes are down. Going offline until atleast one of them comes back up.</div><div>[2017-10-18 16:49:09.222912] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-1: All subvolumes are down. Going offline until atleast one of them comes back up.</div><div>[2017-10-18 16:49:09.223079] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-2: All subvolumes are down. Going offline until atleast one of them comes back up.</div><div>[2017-10-18 16:49:09.223200] E [MSGID: 108006] [afr-common.c:4808:afr_notify] 0-<vol>-replicate-3: All subvolumes are down. Going offline until atleast one of them comes back up.</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Status:<br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div># gluster vol tier <vol> status</div><div><br></div><div>Node Promoted files Demoted files Status run time in h:m:s </div><div>--------- --------- --------- --------- --------- </div><div>Node1 190861 0 in progress 408:34:13</div><div>Node2 0 0 in progress 408:34:14</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Hot tier bricks:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"># df -h</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div>/dev/mapper/vg_bricks-brick_nv<wbr>me1 1.4T 551G 883G 39% /mnt/brick_nvme1</div><div>/dev/mapper/vg_bricks-brick_nv<wbr>me2 1.4T 512G 922G 36% /mnt/brick_nvme2</div></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Can anyone point me in the right direction as to what may be going on? Any guidance is greatly appreciated.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Thanks in advance,</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">HB</div></div>
<br></div></div>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><span class="m_-5716082969068876722m_5508430874479203959HOEnZb"><font color="#888888"><br></font></span></blockquote></div><span class="m_-5716082969068876722m_5508430874479203959HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-5716082969068876722m_5508430874479203959m_-819107161572781637gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Milind<br><br></div></div></div></div>
</font></span></div>
</blockquote></div><br></div>
</div></div><br>______________________________<wbr>_________________<br>
Gluster-users mailing list<br>
<a href="mailto:Gluster-users@gluster.org" target="_blank">Gluster-users@gluster.org</a><br>
<a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/mailm<wbr>an/listinfo/gluster-users</a><br></blockquote></div><br><br clear="all"><br>-- <br><div class="m_-5716082969068876722gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr">Milind<br><br></div></div></div></div>
</div>
</div></div></blockquote></div><br></div>