[Gluster-users] Testing sharding on tiered volume

[Viktor Nosov]

Ben,

For this set of tests we are using bricks provisioned on RAID storage. We are not trying to test performance of tiered volume right now. The goal is to find solution to handle large files that do not fit into hot tier.
You are correct that there is a lot of promotions and demotions of shards  when we are reading or writing large file. But at least sharding  let us do what pure tiered volume rejects.
Regarding to our experiments, tiered volume puts into hot  tier only file's shards that are accessed. All other file shards are staying on cold tier. Access to shards is managed by the shard translator.
After hitting problem while deleting files we stopped testing. We hope that somebody with knowledge of sharded and tiered volume implementations tells us how difficult could be to fix this issue. 

Best regards,
Viktor Nosov 

-----Original Message-----
From: Ben Turner [mailto:bturner at redhat.com] 
Sent: Sunday, December 17, 2017 5:22 PM
To: Viktor Nosov
Cc: gluster-users at gluster.org
Subject: Re: [Gluster-users] Testing sharding on tiered volume

----- Original Message -----
> From: "Viktor Nosov" <vnosov at stonefly.com>
> To: gluster-users at gluster.org
> Cc: vnosov at stonefly.com
> Sent: Friday, December 8, 2017 5:45:25 PM
> Subject: [Gluster-users] Testing sharding on tiered volume
> 
> Hi,
> 
> I'm looking to use sharding on tiered volume. This is very attractive 
> feature that could benefit tiered volume to let it handle larger files 
> without hitting the "out of (hot)space problem".
> I decided to set  test configuration on GlusterFS 3.12.3  when tiered 
> volume has 2TB cold and 1GB hot segments. Shard size is set to be 16MB.
> For testing 100GB files are used. It seems writes and reads are going well.
> But I hit problem trying to delete files from the volume. One of 
> GlusterFS processes hit segmentation fault.
> The problem is reproducible each time.  It was submitted to Red Hat 
> Bugzilla bug list and has ID 1521119.
> You can find details at the attachments to the bug.
> 
> I'm wondering are there other users who are interested to apply 
> sharding to tiered volumes and are experienced similar problems?
>  How  this problem can be resolved or could it be avoided?

This isn't a config I have tried before, from the BZ it mentions:

-The VOL is shared out over SMB to a windows client -You have a 1GB hot tier, 2099GB cold tier -You have features.shard-block-size: 16MB and cluster.tier-demote-frequency: 150

What are you using for the hot tier that has only 1GB, some sort of RAM disk or battery back flash or something?  

With that small of a hot tier you may run into some strange performance characteristics, AFAIK the current tiering implementation uses rebalance to move files between tiers when the tier demote freq times out.  You may end up spending alot of time waiting for your hot files to rebalance to the cold tier since its out of space, you will also probably have other files being written to the cold tier with the hot tier full, further using up your IOPs.  

I don't know how tiering would treat sharded files, would it only promote the shards of the file that are in use or would it try to put the whole file / all the shards on the hot tier?  

If you get a free min update me on what you are trying todo, happy to help however I can.

-b


> 
> Best regards,
> 
>  Viktor Nosov
> 
> 
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>