[Gluster-users] Gluster tier in progress after attach

[Jeevan Patnaik]

Ok, I will check.

Thank you.

On Mon 24 Sep, 2018, 9:37 PM Nithya Balachandran, <nbalacha at redhat.com>
wrote:

> Please check the rebalance log to see why those files were not migrated.
>
> Regards,
> Nithya
>
> On 24 September 2018 at 21:21, Jeevan Patnaik <g1patnaik at gmail.com> wrote:
>
>> Hi,
>>
>> Yes, I did. The status shows all as completed.
>>
>> Regards,
>> Jeevan.
>>
>> On Mon 24 Sep, 2018, 9:20 PM Nithya Balachandran, <nbalacha at redhat.com>
>> wrote:
>>
>>>
>>>
>>> On 24 September 2018 at 21:01, Jeevan Patnaik <g1patnaik at gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> What do you mean by expanding volume?
>>>>
>>>> I mean adding new bricks to extend storage. It is mentioned that
>>>> detaching tier is must for adding or removing bricks.
>>>>
>>>> What command did you use to detach the tier? And how did you check that
>>>> the files still exist on the hot tier?
>>>>
>>>> gluster volume tier volume_name detach start
>>>>
>>>> I check the files directly from backend filesystem on disk and there
>>>> are still left over files on all hot tier disks.
>>>>
>>>> Anyhow I proceeded with detach commit to see what would happen..so
>>>> after commit, I can't access those files from client filesystems..so
>>>> they're gone with the hot tier and were not moved to cold tier.
>>>>
>>>
>>> Did you check that the detach tier operation had completed before
>>> running detach commit?
>>>
>>>>
>>>> And even if I try to attach the hot tier bricks again to bring back the
>>>> files, attach won't work with existing files.
>>>>
>>>> So, it seems we shouldn't detach hot tier without making sure all files
>>>> are moved to cold tier..but in my case, some files just won't move, even
>>>> though they aren't being accessed anywhere.
>>>>
>>>
>>>
>>>
>>>>
>>>> There's a statement saying POSIX locks can cause files to be still left
>>>> in hot tier and in that case, either the application causing posix lock has
>>>> to be closed or files should be moved manually (but where?)... however, I
>>>> don't think in my case there's any posix locks as the files are not
>>>> accessed by any application except gluster itself.
>>>>
>>>> Regards,
>>>> Jeevan.
>>>>
>>>>
>>>> On Mon 24 Sep, 2018, 8:45 PM Nithya Balachandran, <nbalacha at redhat.com>
>>>> wrote:
>>>>
>>>>> It has been a while since I worked on tier but here is what I remember:
>>>>>
>>>>>
>>>>> On 24 September 2018 at 19:15, Jeevan Patnaik <g1patnaik at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Nithya,
>>>>>>
>>>>>> No..the files are not being accessed. And tiering mode is in cache
>>>>>> mode and what I understood is in cache mode every file is moved to hot tier
>>>>>> until it reaches low watermark, then it only promotes highly accessed files.
>>>>>>
>>>>>> Not quite . All files will initially exist on the cold tier. Any new
>>>>> files will be created on the hot tier. Any old files that are accessed will
>>>>> be promoted ot the hot tier.
>>>>>
>>>>>
>>>>> And do you mean everytime we expand volume,  we are virtually flushing
>>>>>> the tier and it regenerates only when a files are accessed?
>>>>>>
>>>>>> What do you mean by expanding the volume?
>>>>>
>>>>>
>>>>>> Also I see one more issue now, I hav generated another 1000 files
>>>>>> after tiering is started and I see all 1000 files are now created on hot
>>>>>> tier. Now, I detached again..in detach status it shows completed..but when
>>>>>> I checked, around 260 files are still located on hot tier..those files are
>>>>>> not being used anywhere..ca you tell why are they still not moved to cold
>>>>>> tier and why the detach status is showing a wrong status.
>>>>>>
>>>>>
>>>>> What command did you use to detach the tier? And how did you check
>>>>> that the files still exist on the hot tier?
>>>>>
>>>>> Regards,
>>>>> Nithya
>>>>>
>>>>>>
>>>>>> I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
>>>>>> hosts also.
>>>>>>
>>>>>> Regards,
>>>>>> Jeevan.
>>>>>>
>>>>>> On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, <
>>>>>> nbalacha at redhat.com> wrote:
>>>>>>
>>>>>>> Are those files being accessed? Tiering will only promote those that
>>>>>>> have been accessed recently.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Nithya
>>>>>>>
>>>>>>> On 24 September 2018 at 18:32, Jeevan Patnaik <g1patnaik at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> I see it still not promoting any files. Do we need to run any
>>>>>>>> command to force the movement?
>>>>>>>>
>>>>>>>> This would be an issue if we need to expand disk as then we also
>>>>>>>> need to detach existing tier and attach again and expect the data to be
>>>>>>>> promoted to the hot tier.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Jeevan.
>>>>>>>>
>>>>>>>> On Mon 24 Sep, 2018, 6:17 PM Jeevan Patnaik, <g1patnaik at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have created a 18 disk replica 3 volume and created 1000 random
>>>>>>>>> files each of 10M.
>>>>>>>>>
>>>>>>>>> Later I have attached 15 disk replica 3 tier with each hot tier
>>>>>>>>> disk coming from cold tier host, except on 3 (as we don't have hot tier
>>>>>>>>> disk)
>>>>>>>>>
>>>>>>>>> After attach, I see no files are promoted at even after 15
>>>>>>>>> minutes. Status is showing as In progress.
>>>>>>>>>
>>>>>>>>> Is it always this much slow? For 10G data, 15 minutes is too much
>>>>>>>>> Also will the data try to be promoted to a hot tier coming from
>>>>>>>>> the same host to reduce the time taken to move the files?
>>>>>>>>>
>>>>>>>>> Cold tier is made of RAID supported SSD and hot tier is made of
>>>>>>>>> NVMe SSDs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Jeevan.
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Gluster-users mailing list
>>>>>>>> Gluster-users at gluster.org
>>>>>>>> https://lists.gluster.org/mailman/listinfo/gluster-users
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.gluster.org/pipermail/gluster-users/attachments/20180924/63e8c991/attachment.html>