[Gluster-users] [Gluster-devel] Fwd: dht_is_subvol_filled messages on client

Thu May 5 12:00:13 UTC 2016

On Thu, May 5, 2016 at 4:59 PM, Serkan Çoban <cobanserkan at gmail.com> wrote:
> Ah I see how can I overlook this, my bad sorry everyone for taking
> time to help me...
>>BTW, how large is the volume you have?
> 9PB usable :)

Happy to help!

>
> Serkan
>
> On Thu, May 5, 2016 at 2:07 PM, Xavier Hernandez <xhernandez at datalab.es> wrote:
>> On 05/05/16 11:31, Kaushal M wrote:
>>>
>>> On Thu, May 5, 2016 at 2:36 PM, David Gossage
>>> <dgossage at carouselchecks.com> wrote:
>>>>
>>>>
>>>>
>>>>
>>>> On Thu, May 5, 2016 at 3:28 AM, Serkan Çoban <cobanserkan at gmail.com>
>>>> wrote:
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> You can find the output below link:
>>>>> https://www.dropbox.com/s/wzrh5yp494ogksc/status_detail.txt?dl=0
>>>>>
>>>>> Thanks,
>>>>> Serkan
>>>>
>>>>
>>>>
>>>> Maybe not issue, but playing one of these things is not like the other I
>>>> notice of all the bricks only one seems to be different at a quick glance
>>>>
>>>> Brick                : Brick 1.1.1.235:/bricks/20
>>>> TCP Port             : 49170
>>>> RDMA Port            : 0
>>>> Online               : Y
>>>> Pid                  : 26736
>>>> File System          : ext4
>>>> Device               : /dev/mapper/vol0-vol_root
>>>> Mount Options        : rw,relatime,data=ordered
>>>> Inode Size           : 256
>>>> Disk Space Free      : 86.1GB
>>>> Total Disk Space     : 96.0GB
>>>> Inode Count          : 6406144
>>>> Free Inodes          : 6381374
>>>>
>>>> Every other brick seems to be 7TB and xfs but this one.
>>>
>>>
>>> Looks like the brick fs isn't mounted, and the root-fs is being used
>>> instead. But that still leaves enough inodes free.
>>>
>>> What I suspect is that one of the cluster translators is mixing up
>>> stats when aggregating from multiple bricks.
>>> From the log snippet you gave in the first mail, it seems like the
>>> disperse translator is possibly involved.
>>
>>
>> Currently ec takes the number of potential files in the subvolume (f_files)
>> as the maximum of all its subvolumes, but it takes the available count
>> (f_ffree) as the minumum of all its volumes.
>>
>> This causes max to be ~781.000.000, but free will be ~6.300.000. This gives
>> a ~0.8% available, i.e. almost 100% full.
>>
>> Given the circumstances I think it's the correct thing to do.
>>
>> Xavi
>>
>>
>>>
>>> BTW, how large is the volume you have? Those are a lot of bricks!
>>>
>>> ~kaushal
>>>
>>>
>>>>
>>>>
>>>>
>>>>>
>>>>>
>>>>> On Thu, May 5, 2016 at 9:33 AM, Xavier Hernandez <xhernandez at datalab.es>
>>>>> wrote:
>>>>>>
>>>>>> Can you post the result of 'gluster volume status v0 detail' ?
>>>>>>
>>>>>>
>>>>>> On 05/05/16 06:49, Serkan Çoban wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi, Can anyone suggest something for this issue? df, du has no issue
>>>>>>> for the bricks yet one subvolume not being used by gluster..
>>>>>>>
>>>>>>> On Wed, May 4, 2016 at 4:40 PM, Serkan Çoban <cobanserkan at gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I changed cluster.min-free-inodes to "0". Remount the volume on
>>>>>>>> clients. inode full messages not coming to syslog anymore but I see
>>>>>>>> disperse-56 subvolume still not being used.
>>>>>>>> Anything I can do to resolve this issue? Maybe I can destroy and
>>>>>>>> recreate the volume but I am not sure It will fix this issue...
>>>>>>>> Maybe the disperse size 16+4 is too big should I change it to 8+2?
>>>>>>>>
>>>>>>>> On Tue, May 3, 2016 at 2:36 PM, Serkan Çoban <cobanserkan at gmail.com>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> I also checked the df output all 20 bricks are same like below:
>>>>>>>>> /dev/sdu1 7.3T 34M 7.3T 1% /bricks/20
>>>>>>>>>
>>>>>>>>> On Tue, May 3, 2016 at 1:40 PM, Raghavendra G
>>>>>>>>> <raghavendra at gluster.com>
>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Mon, May 2, 2016 at 11:41 AM, Serkan Çoban
>>>>>>>>>> <cobanserkan at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 1. What is the out put of du -hs <back-end-export>? Please get
>>>>>>>>>>>> this
>>>>>>>>>>>> information for each of the brick that are part of disperse.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sorry. I needed df output of the filesystem containing brick. Not
>>>>>>>>>> du.
>>>>>>>>>> Sorry
>>>>>>>>>> about that.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> There are 20 bricks in disperse-56 and the du -hs output is like:
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 1.8M /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>> 80K /bricks/20
>>>>>>>>>>>
>>>>>>>>>>> I see that gluster is not writing to this disperse set. All other
>>>>>>>>>>> disperse sets are filled 13GB but this one is empty. I see
>>>>>>>>>>> directory
>>>>>>>>>>> structure created but no files in directories.
>>>>>>>>>>> How can I fix the issue? I will try to rebalance but I don't think
>>>>>>>>>>> it
>>>>>>>>>>> will write to this disperse set...
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Sat, Apr 30, 2016 at 9:22 AM, Raghavendra G
>>>>>>>>>>> <raghavendra at gluster.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Fri, Apr 29, 2016 at 12:32 AM, Serkan Çoban
>>>>>>>>>>>> <cobanserkan at gmail.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi, I cannot get an answer from user list, so asking to devel
>>>>>>>>>>>>> list.
>>>>>>>>>>>>>
>>>>>>>>>>>>> I am getting [dht-diskusage.c:277:dht_is_subvol_filled]
>>>>>>>>>>>>> 0-v0-dht:
>>>>>>>>>>>>> inodes on subvolume 'v0-disperse-56' are at (100.00 %), consider
>>>>>>>>>>>>> adding more bricks.
>>>>>>>>>>>>>
>>>>>>>>>>>>> message on client logs.My cluster is empty there are only a
>>>>>>>>>>>>> couple
>>>>>>>>>>>>> of
>>>>>>>>>>>>> GB files for testing. Why this message appear in syslog?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> dht uses disk usage information from backend export.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. What is the out put of du -hs <back-end-export>? Please get
>>>>>>>>>>>> this
>>>>>>>>>>>> information for each of the brick that are part of disperse.
>>>>>>>>>>>> 2. Once you get du information from each brick, the value seen by
>>>>>>>>>>>> dht
>>>>>>>>>>>> will
>>>>>>>>>>>> be based on how cluster/disperse aggregates du info (basically
>>>>>>>>>>>> statfs
>>>>>>>>>>>> fop).
>>>>>>>>>>>>
>>>>>>>>>>>> The reason for 100% disk usage may be,
>>>>>>>>>>>> In case of 1, backend fs might be shared by data other than
>>>>>>>>>>>> brick.
>>>>>>>>>>>> In case of 2, some issues with aggregation.
>>>>>>>>>>>>
>>>>>>>>>>>>> Is is safe to
>>>>>>>>>>>>> ignore it?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> dht will try not to have data files on the subvol in question
>>>>>>>>>>>> (v0-disperse-56). Hence lookup cost will be two hops for files
>>>>>>>>>>>> hashing
>>>>>>>>>>>> to
>>>>>>>>>>>> disperse-56 (note that other fops like read/write/open still have
>>>>>>>>>>>> the
>>>>>>>>>>>> cost
>>>>>>>>>>>> of single hop and dont suffer from this penalty). Other than that
>>>>>>>>>>>> there
>>>>>>>>>>>> is
>>>>>>>>>>>> no significant harm unless disperse-56 is really running out of
>>>>>>>>>>>> space.
>>>>>>>>>>>>
>>>>>>>>>>>> regards,
>>>>>>>>>>>> Raghavendra
>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Gluster-devel mailing list
>>>>>>>>>>>>> Gluster-devel at gluster.org
>>>>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> Raghavendra G
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Gluster-devel mailing list
>>>>>>>>>>> Gluster-devel at gluster.org
>>>>>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-devel
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Raghavendra G
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Gluster-users mailing list
>>>>>>> Gluster-users at gluster.org
>>>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>>>>
>>>>>>
>>>>> _______________________________________________
>>>>> Gluster-users mailing list
>>>>> Gluster-users at gluster.org
>>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Gluster-users mailing list
>>>> Gluster-users at gluster.org
>>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>
>>