<div dir="ltr"><div style="font-size:12.8px">The sizes of the files are extremely varied, there are millions of small (<1 MB) files and thousands of files larger than 1 GB.</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Attached is the tier log for gluster1 and gluster2. These are full of "demotion failed" messages, which is also shown in the status:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px"><div style="font-size:12.8px">[root@pod-sjc1-gluster1 gv0]# gluster volume tier gv0 status</div><div style="font-size:12.8px">Node Promoted files Demoted files Status run time in h:m:s </div><div style="font-size:12.8px">--------- --------- --------- --------- --------- </div><div style="font-size:12.8px">localhost 25940 0 in progress 112:21:49</div><div style="font-size:12.8px">pod-sjc1-gluster2 0 2917154 in progress 112:21:49</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">Is it normal to have promotions and demotions only happen on each server but not both?</div><div style="font-size:12.8px"><br></div></div><div style="font-size:12.8px">Volume info:</div><div style="font-size:12.8px"><br></div><div style="font-size:12.8px">[root@pod-sjc1-gluster1 ~]# gluster volume info</div><div style="font-size:12.8px"> </div><div style="font-size:12.8px">Volume Name: gv0</div><div style="font-size:12.8px">Type: Distributed-Replicate</div><div style="font-size:12.8px">Volume ID: d490a9ec-f9c8-4f10-a7f3-e1b6d3<wbr>ced196</div><div style="font-size:12.8px">Status: Started</div><div style="font-size:12.8px">Snapshot Count: 13</div><div style="font-size:12.8px">Number of Bricks: 3 x 2 = 6</div><div style="font-size:12.8px">Transport-type: tcp</div><div style="font-size:12.8px">Bricks:</div><div style="font-size:12.8px">Brick1: pod-sjc1-gluster1:/data/brick1<wbr>/gv0</div><div style="font-size:12.8px">Brick2: pod-sjc1-gluster2:/data/brick1<wbr>/gv0</div><div style="font-size:12.8px">Brick3: pod-sjc1-gluster1:/data/brick2<wbr>/gv0</div><div style="font-size:12.8px">Brick4: pod-sjc1-gluster2:/data/brick2<wbr>/gv0</div><div style="font-size:12.8px">Brick5: pod-sjc1-gluster1:/data/brick3<wbr>/gv0</div><div style="font-size:12.8px">Brick6: pod-sjc1-gluster2:/data/brick3<wbr>/gv0</div><div style="font-size:12.8px">Options Reconfigured:</div><div style="font-size:12.8px">performance.cache-refresh-time<wbr>out: 60</div><div style="font-size:12.8px">performance.stat-prefetch: on</div><div style="font-size:12.8px">server.allow-insecure: on</div><div style="font-size:12.8px">performance.flush-behind: on</div><div style="font-size:12.8px">performance.rda-cache-limit: 32MB</div><div style="font-size:12.8px">network.tcp-window-size: 1048576</div><div style="font-size:12.8px">performance.nfs.io-threads: on</div><div style="font-size:12.8px">performance.write-behind-windo<wbr>w-size: 4MB</div><div style="font-size:12.8px">performance.nfs.write-behind-w<wbr>indow-size: 512MB</div><div style="font-size:12.8px">performance.io-cache: on</div><div style="font-size:12.8px">performance.quick-read: on</div><div style="font-size:12.8px">features.cache-invalidation: on</div><div style="font-size:12.8px">features.cache-invalidation-ti<wbr>meout: 600</div><div style="font-size:12.8px">performance.cache-invalidation<wbr>: on</div><div style="font-size:12.8px">performance.md-cache-timeout: 600</div><div style="font-size:12.8px">network.inode-lru-limit: 90000</div><div style="font-size:12.8px">performance.cache-size: 4GB</div><div style="font-size:12.8px">server.event-threads: 16</div><div style="font-size:12.8px">client.event-threads: 16</div><div style="font-size:12.8px">features.barrier: disable</div><div style="font-size:12.8px">transport.address-family: inet</div><div style="font-size:12.8px">nfs.disable: on</div><div style="font-size:12.8px">performance.client-io-threads: on</div><div style="font-size:12.8px">cluster.lookup-optimize: on</div><div style="font-size:12.8px">server.outstanding-rpc-limit: 1024</div><div style="font-size:12.8px">auto-delete: enable</div><div style="font-size:12.8px"><br></div><div><div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px"># gluster volume status</span></div><div><span style="font-size:12.8px">Status of volume: gv0</span></div><div><span style="font-size:12.8px">Gluster process TCP Port RDMA Port Online Pid</span></div><div><span style="font-size:12.8px">------------------------------<wbr>------------------------------<wbr>------------------</span></div><div><span style="font-size:12.8px">Hot Bricks:</span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster2:/data/</span></div><div><span style="font-size:12.8px">hot_tier/gv0 49219 0 Y 26714</span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster1:/data/</span></div><div><span style="font-size:12.8px">hot_tier/gv0 49199 0 Y 21325</span></div><div><span style="font-size:12.8px">Cold Bricks:</span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster1:/data/</span></div><div><span style="font-size:12.8px">brick1/gv0 49152 0 Y 3178 </span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster2:/data/</span></div><div><span style="font-size:12.8px">brick1/gv0 49152 0 Y 4818 </span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster1:/data/</span></div><div><span style="font-size:12.8px">brick2/gv0 49153 0 Y 3186 </span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster2:/data/</span></div><div><span style="font-size:12.8px">brick2/gv0 49153 0 Y 4829 </span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster1:/data/</span></div><div><span style="font-size:12.8px">brick3/gv0 49154 0 Y 3194 </span></div><div><span style="font-size:12.8px">Brick pod-sjc1-gluster2:/data/</span></div><div><span style="font-size:12.8px">brick3/gv0 49154 0 Y 4840 </span></div><div><span style="font-size:12.8px">Tier Daemon on localhost N/A N/A Y 20313</span></div><div><span style="font-size:12.8px">Self-heal Daemon on localhost N/A N/A Y 32023</span></div><div><span style="font-size:12.8px">Tier Daemon on pod-sjc1-gluster1 N/A N/A Y 24758</span></div><div><span style="font-size:12.8px">Self-heal Daemon on pod-sjc1-gluster2 N/A N/A Y 12349</span></div><div><span style="font-size:12.8px"> </span></div><div><span style="font-size:12.8px">Task Status of Volume gv0</span></div><div><span style="font-size:12.8px">------------------------------<wbr>------------------------------<wbr>------------------</span></div><div><span style="font-size:12.8px">There are no active volume tasks</span></div><div><span style="font-size:12.8px"> </span></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Jan 9, 2018 at 10:33 PM, Hari Gowtham <span dir="ltr"><<a href="mailto:hgowtham@redhat.com" target="_blank">hgowtham@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
Can you send the volume info, and volume status output and the tier logs.<br>
And I need to know the size of the files that are being stored.<br>
<div><div class="h5"><br>
On Tue, Jan 9, 2018 at 9:51 PM, Tom Fite <<a href="mailto:tomfite@gmail.com">tomfite@gmail.com</a>> wrote:<br>
> I've recently enabled an SSD backed 2 TB hot tier on my 150 TB 2 server / 3<br>
> bricks per server distributed replicated volume.<br>
><br>
> I'm seeing IO get blocked across all client FUSE threads for 10 to 15<br>
> seconds while the promotion daemon runs. I see the 'glustertierpro' thread<br>
> jump to 99% CPU usage on both boxes when these delays occur and they happen<br>
> every 25 minutes (my tier-promote-frequency setting).<br>
><br>
> I suspect this has something to do with the heat database in sqlite, maybe<br>
> something is getting locked while it runs the query to determine files to<br>
> promote. My volume contains approximately 18 million files.<br>
><br>
> Has anybody else seen this? I suspect that these delays will get worse as I<br>
> add more files to my volume which will cause significant problems.<br>
><br>
> Here are my hot tier settings:<br>
><br>
> # gluster volume get gv0 all | grep tier<br>
> cluster.tier-pause off<br>
> cluster.tier-promote-frequency 1500<br>
> cluster.tier-demote-frequency 3600<br>
> cluster.tier-mode cache<br>
> cluster.tier-max-promote-file-<wbr>size 10485760<br>
> cluster.tier-max-mb 64000<br>
> cluster.tier-max-files 100000<br>
> cluster.tier-query-limit 100<br>
> cluster.tier-compact on<br>
> cluster.tier-hot-compact-<wbr>frequency 86400<br>
> cluster.tier-cold-compact-<wbr>frequency 86400<br>
><br>
> # gluster volume get gv0 all | grep threshold<br>
> cluster.write-freq-threshold 2<br>
> cluster.read-freq-threshold 5<br>
><br>
> # gluster volume get gv0 all | grep watermark<br>
> cluster.watermark-hi 92<br>
> cluster.watermark-low 75<br>
><br>
</div></div>> ______________________________<wbr>_________________<br>
> Gluster-users mailing list<br>
> <a href="mailto:Gluster-users@gluster.org">Gluster-users@gluster.org</a><br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users" rel="noreferrer" target="_blank">http://lists.gluster.org/<wbr>mailman/listinfo/gluster-users</a><br>
<span class="HOEnZb"><font color="#888888"><br>
<br>
<br>
--<br>
Regards,<br>
Hari Gowtham.<br>
</font></span></blockquote></div><br></div>