[Gluster-users] Gluster tier in progress after attach

[Nithya Balachandran]

On 24 September 2018 at 21:01, Jeevan Patnaik <g1patnaik at gmail.com> wrote:

> Hi,
>
> What do you mean by expanding volume?
>
> I mean adding new bricks to extend storage. It is mentioned that detaching
> tier is must for adding or removing bricks.
>
> What command did you use to detach the tier? And how did you check that
> the files still exist on the hot tier?
>
> gluster volume tier volume_name detach start
>
> I check the files directly from backend filesystem on disk and there are
> still left over files on all hot tier disks.
>
> Anyhow I proceeded with detach commit to see what would happen..so after
> commit, I can't access those files from client filesystems..so they're gone
> with the hot tier and were not moved to cold tier.
>

Did you check that the detach tier operation had completed before running
detach commit?

>
> And even if I try to attach the hot tier bricks again to bring back the
> files, attach won't work with existing files.
>
> So, it seems we shouldn't detach hot tier without making sure all files
> are moved to cold tier..but in my case, some files just won't move, even
> though they aren't being accessed anywhere.
>



>
> There's a statement saying POSIX locks can cause files to be still left in
> hot tier and in that case, either the application causing posix lock has to
> be closed or files should be moved manually (but where?)... however, I
> don't think in my case there's any posix locks as the files are not
> accessed by any application except gluster itself.
>
> Regards,
> Jeevan.
>
>
> On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, <nbalacha at redhat.com>
> wrote:
>
>> It has been a while since I worked on tier but here is what I remember:
>>
>>
>> On 24 September 2018 at 19:15, Jeevan Patnaik <g1patnaik at gmail.com>
>> wrote:
>>
>>> Hi Nithya,
>>>
>>> No..the files are not being accessed. And tiering mode is in cache mode
>>> and what I understood is in cache mode every file is moved to hot tier
>>> until it reaches low watermark, then it only promotes highly accessed files.
>>>
>>> Not quite . All files will initially exist on the cold tier. Any new
>> files will be created on the hot tier. Any old files that are accessed will
>> be promoted ot the hot tier.
>>
>>
>> And do you mean everytime we expand volume,  we are virtually flushing
>>> the tier and it regenerates only when a files are accessed?
>>>
>>> What do you mean by expanding the volume?
>>
>>
>>> Also I see one more issue now, I hav generated another 1000 files after
>>> tiering is started and I see all 1000 files are now created on hot tier.
>>> Now, I detached again..in detach status it shows completed..but when I
>>> checked, around 260 files are still located on hot tier..those files are
>>> not being used anywhere..ca you tell why are they still not moved to cold
>>> tier and why the detach status is showing a wrong status.
>>>
>>
>> What command did you use to detach the tier? And how did you check that
>> the files still exist on the hot tier?
>>
>> Regards,
>> Nithya
>>
>>>
>>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
>>> hosts also.
>>>
>>> Regards,
>>> Jeevan.
>>>
>>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, <nbalacha at redhat.com>
>>> wrote:
>>>
>>>> Are those files being accessed? Tiering will only promote those that
>>>> have been accessed recently.
>>>>
>>>> Regards,
>>>> Nithya
>>>>
>>>> On 24 September 2018 at 18:32, Jeevan Patnaik <g1patnaik at gmail.com>
>>>> wrote:
>>>>
>>>>> I see it still not promoting any files. Do we need to run any command
>>>>> to force the movement?
>>>>>
>>>>> This would be an issue if we need to expand disk as then we also need
>>>>> to detach existing tier and attach again and expect the data to be promoted
>>>>> to the hot tier.
>>>>>
>>>>> Regards,
>>>>> Jeevan.
>>>>>
>>>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, <g1patnaik at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have created a 18 disk replica 3 volume and created 1000 random
>>>>>> files each of 10M.
>>>>>>
>>>>>> Later I have attached 15 disk replica 3 tier with each hot tier disk
>>>>>> coming from cold tier host, except on 3 (as we don't have hot tier disk)
>>>>>>
>>>>>> After attach, I see no files are promoted at even after 15 minutes.
>>>>>> Status is showing as In progress.
>>>>>>
>>>>>> Is it always this much slow? For 10G data, 15 minutes is too much
>>>>>> Also will the data try to be promoted to a hot tier coming from the
>>>>>> same host to reduce the time taken to move the files?
>>>>>>
>>>>>> Cold tier is made of RAID supported SSD and hot tier is made of NVMe
>>>>>> SSDs.
>>>>>>
>>>>>>
>>>>>> Regards,
>>>>>> Jeevan.
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>
>>>>
>>>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180924/4163a66d/attachment.html>