[Bugs] [Bug 1549714] New: On sharded tiered volume, only first shard of new file goes on hot tier.

[bugzilla at redhat.com]

https://bugzilla.redhat.com/show_bug.cgi?id=1549714

            Bug ID: 1549714
           Summary: On sharded tiered volume, only first shard of new file
                    goes on hot tier.
           Product: GlusterFS
           Version: 3.12
         Component: sharding
          Severity: medium
          Assignee: bugs at gluster.org
          Reporter: jbyers at stonefly.com
        QA Contact: bugs at gluster.org
                CC: bugs at gluster.org



On sharded tiered volume, only first shard of new file goes on hot tier.

On a sharded tiered volume, only the first shard of a new file
goes on the hot tier, the rest are written to the cold tier.

This is unfortunate for archival applications where the hot
tier is fast, but the cold tier is very slow. After the tier-
promote-frequency (default 120 seconds), all of the shards do
migrate to hot tier, but for archival applications, this
migration is not helpful since the file is likely not be
accessed again just after being copied on, and it will later
just migrate back to the cold tier.

Sharding should be, and really needs to be used with very large
archive files because of bug:

    Bug 1277112 - Data Tiering:File create and new writes to
    existing file fails when the hot tier is full instead of
    redirecting/flushing the data to cold tier

which sharding with tiering helps mitigate.

This problem occurs in GlusterFS 3.7.18 and 3.12.3, at least.
I/O size doesn't make any difference, I tried multiple sizes.
I didn't find any volume configuration options that helped
in getting all of the new shards to go directly to the hot tier.

# dd if=/dev/root bs=64M of=/samba/tiered-sharded-vol/file-1 count=6
402653184 bytes (403 MB) copied, 1.31154 s, 307 MB/s
# ls -lrtdh /samba/tiered-sharded-vol/* /exports/brick-*/tiered-sharded-vol/*
/exports/brick-*/tiered-sharded-vol/.shard/* 2>/dev/null
---------T    0 Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4
-rw-r--r-- 384M Feb 27 08:58 /samba/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5
# sleep 120
# ls -lrtdh /samba/tiered-sharded-vol/* /exports/brick-*/tiered-sharded-vol/*
/exports/brick-*/tiered-sharded-vol/.shard/* 2>/dev/null
---------T    0 Feb 27 08:58 /exports/brick-cold/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58 /exports/brick-hot/tiered-sharded-vol/file-1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4
-rw-r--r--  64M Feb 27 08:58
/exports/brick-hot/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5
-rw-r--r-- 384M Feb 27 08:58 /samba/tiered-sharded-vol/file-1
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.5
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.1
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.2
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.3
---------T    0 Feb 27 09:00
/exports/brick-cold/tiered-sharded-vol/.shard/85c2cb2e-65ec-4d82-9500-ec6ba361cbbf.4

Volume Name: tiered-sharded-vol
Type: Tier
Volume ID: 8c09077a-371e-4d30-9faa-c9c76d7b1b57
Status: Started
Number of Bricks: 2
Transport-type: tcp
Hot Tier :
Hot Tier Type : Distribute
Number of Bricks: 1
Brick1: 10.10.60.169:/exports/brick-hot/tiered-sharded-vol
Cold Tier:
Cold Tier Type : Distribute
Number of Bricks: 1
Brick2: 10.10.60.169:/exports/brick-cold/tiered-sharded-vol
Options Reconfigured:
cluster.tier-mode: cache
features.ctr-enabled: on
features.shard: on
features.shard-block-size: 64MB
server.allow-insecure: on
performance.quick-read: off
performance.stat-prefetch: off
nfs.disable: on
nfs.addr-namelookup: off
performance.readdir-ahead: on
snap-activate-on-create: enable
cluster.enable-shared-storage: disable

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are on the CC list for the bug.
You are the assignee for the bug.