[Gluster-users] On sharded tiered volume, only first shard of new file goes on hot tier.

[Jeff Byers]

Does anyone have any ideas about how to fix, or to work-around the
following issue?
Thanks!

Bug 1549714 - On sharded tiered volume, only first shard of new file
goes on hot tier.
https://bugzilla.redhat.com/show_bug.cgi?id=1549714

On sharded tiered volume, only first shard of new file goes on hot tier.

On a sharded tiered volume, only the first shard of a new file
goes on the hot tier, the rest are written to the cold tier.

This is unfortunate for archival applications where the hot
tier is fast, but the cold tier is very slow. After the tier-
promote-frequency (default 120 seconds), all of the shards do
migrate to hot tier, but for archival applications, this
migration is not helpful since the file is likely to not be
accessed again just after being copied on, and it will later
just migrate back to the cold tier.

Sharding should be, and needs to be used with very large
archive files because of bug:

    Bug 1277112 - Data Tiering:File create and new writes to
    existing file fails when the hot tier is full instead of
    redirecting/flushing the data to cold tier

which sharding with tiering helps mitigate.

This problem occurs in GlusterFS 3.7.18 and 3.12.3, at least.
I/O size doesn't make any difference, I tried multiple sizes.
I didn't find any volume configuration options that helped
in getting all of the new shards to go directly to the hot tier.

# dd if=/dev/root bs=64M of=/samba/tiered-sharded-vol/file-1 count=6
402653184 bytes (403 MB) copied, 1.31154 s, 307 MB/s
# ls -lrtdh /samba/tiered-sharded-vol/*
/exports/brick-*/tiered-sharded-vol/*
/exports/brick-*/tiered-sharded-vol/.shard/* 2>/dev/null
---------T    0 Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4
-rw-r--r-- 384M Feb 27 08:58 /samba/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5
# sleep 120
# ls -lrtdh /samba/tiered-sharded-vol/*
/exports/brick-*/tiered-sharded-vol/*
/exports/brick-*/tiered-sharded-vol/.shard/* 2>/dev/null
---------T    0 Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5
-rw-r--r-- 384M Feb 27 08:58 /samba/tiered-sharded-vol/file-1
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4

Volume Name: tiered-sharded-vol
Type: Tier
Volume ID: 8c09077a-371e-4d30-9faa-c9c76d7b1b57
Status: Started
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.10.60.169:/exports/brick-hot/tiered-sharded-vol
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: 10.10.60.169:/exports/brick-cold/tiered-sharded-vol
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
features.shard: on
features.shard-block-size: 64MB
server.allow-insecure: on
performance.quick-read: off
performance.stat-prefetch: off
nfs.disable: on
nfs.addr-namelookup: off
performance.readdir-ahead: on
snap-activate-on-create: enable
cluster.enable-shared-storage: disable


~ Jeff Byers ~