[Gluster-users] Gluster tier in progress after attach

[Jeevan Patnaik]

Hi,

What do you mean by expanding volume?

I mean adding new bricks to extend storage. It is mentioned that detaching
tier is must for adding or removing bricks.

What command did you use to detach the tier? And how did you check that the
files still exist on the hot tier?

gluster volume tier volume_name detach start

I check the files directly from backend filesystem on disk and there are
still left over files on all hot tier disks.

Anyhow I proceeded with detach commit to see what would happen..so after
commit, I can't access those files from client filesystems..so they're gone
with the hot tier and were not moved to cold tier.

And even if I try to attach the hot tier bricks again to bring back the
files, attach won't work with existing files.

So, it seems we shouldn't detach hot tier without making sure all files are
moved to cold tier..but in my case, some files just won't move, even though
they aren't being accessed anywhere.

There's a statement saying POSIX locks can cause files to be still left in
hot tier and in that case, either the application causing posix lock has to
be closed or files should be moved manually (but where?)... however, I
don't think in my case there's any posix locks as the files are not
accessed by any application except gluster itself.

Regards,
Jeevan.


On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, <nbalacha at redhat.com>
wrote:

> It has been a while since I worked on tier but here is what I remember:
>
>
> On 24 September 2018 at 19:15, Jeevan Patnaik <g1patnaik at gmail.com> wrote:
>
>> Hi Nithya,
>>
>> No..the files are not being accessed. And tiering mode is in cache mode
>> and what I understood is in cache mode every file is moved to hot tier
>> until it reaches low watermark, then it only promotes highly accessed files.
>>
>> Not quite . All files will initially exist on the cold tier. Any new
> files will be created on the hot tier. Any old files that are accessed will
> be promoted ot the hot tier.
>
>
> And do you mean everytime we expand volume,  we are virtually flushing the
>> tier and it regenerates only when a files are accessed?
>>
>> What do you mean by expanding the volume?
>
>
>> Also I see one more issue now, I hav generated another 1000 files after
>> tiering is started and I see all 1000 files are now created on hot tier.
>> Now, I detached again..in detach status it shows completed..but when I
>> checked, around 260 files are still located on hot tier..those files are
>> not being used anywhere..ca you tell why are they still not moved to cold
>> tier and why the detach status is showing a wrong status.
>>
>
> What command did you use to detach the tier? And how did you check that
> the files still exist on the hot tier?
>
> Regards,
> Nithya
>
>>
>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 hosts
>> also.
>>
>> Regards,
>> Jeevan.
>>
>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, <nbalacha at redhat.com>
>> wrote:
>>
>>> Are those files being accessed? Tiering will only promote those that
>>> have been accessed recently.
>>>
>>> Regards,
>>> Nithya
>>>
>>> On 24 September 2018 at 18:32, Jeevan Patnaik <g1patnaik at gmail.com>
>>> wrote:
>>>
>>>> I see it still not promoting any files. Do we need to run any command
>>>> to force the movement?
>>>>
>>>> This would be an issue if we need to expand disk as then we also need
>>>> to detach existing tier and attach again and expect the data to be promoted
>>>> to the hot tier.
>>>>
>>>> Regards,
>>>> Jeevan.
>>>>
>>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, <g1patnaik at gmail.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have created a 18 disk replica 3 volume and created 1000 random
>>>>> files each of 10M.
>>>>>
>>>>> Later I have attached 15 disk replica 3 tier with each hot tier disk
>>>>> coming from cold tier host, except on 3 (as we don't have hot tier disk)
>>>>>
>>>>> After attach, I see no files are promoted at even after 15 minutes.
>>>>> Status is showing as In progress.
>>>>>
>>>>> Is it always this much slow? For 10G data, 15 minutes is too much
>>>>> Also will the data try to be promoted to a hot tier coming from the
>>>>> same host to reduce the time taken to move the files?
>>>>>
>>>>> Cold tier is made of RAID supported SSD and hot tier is made of NVMe
>>>>> SSDs.
>>>>>
>>>>>
>>>>> Regards,
>>>>> Jeevan.
>>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180924/75839936/attachment.html>