[Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

[Prashanth Pai]

Hi Luis,

Currently, even with storage policies, there is no mechanism to update the container DBs with details of files created/modified/deleted over filesystem interface. Hence a GET on a container would list objects that may or may not exist on disk. Also the metadata of container (bytes and object count) could be outdated. We need a mechanism to detect change in GlusterFS (have explored changelog and inotify, both not feasible) and then lazily update Swift DBs so that the container listing and metadata would eventually look right.

One area where this feature "might" come handy is we can selectively crawl only those directories where the object counts and bytes do not match with those values stored in respective DBs. There are some special corner cases such as marker dirs. However, if the object count feature in glusterfs does not distinguish between files and directories, it's not of much use to swiftonfile.

Regards,
 -Prashanth Pai

----- Original Message -----
From: "Luis Pabon" <lpabon at redhat.com>
To: "Prashanth Pai" <ppai at redhat.com>
Cc: "Vijaikumar M" <vmallika at redhat.com>, "gluster-devel at gluster.org >> Gluster Devel" <gluster-devel at gluster.org>
Sent: Monday, February 23, 2015 2:21:32 AM
Subject: Re: [Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

Quick question, 
  Is this feature still necessary since gluster-swift is now SwiftOnFile and is deployed as a storage policy of Swift which uses the Swift container database?

- Luis

----- Original Message -----
From: "Prashanth Pai" <ppai at redhat.com>
To: "Vijaikumar M" <vmallika at redhat.com>
Cc: "gluster-devel at gluster.org >> Gluster Devel" <gluster-devel at gluster.org>
Sent: Tuesday, February 17, 2015 9:47:20 PM
Subject: Re: [Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

This might be helpful for  better integration with Swift. Swift maintains accounting metadata per account and per container. From glusterfs perspective, accounts are (represented as) first level directories in root and containers are second level directories. So accounts contain containers.

Each account has following metadata:
X-Account-Object-Count
X-Account-Bytes-Used
X-Account-Container-Count

Each container has following metadata:
X-Container-Object-Count
X-Container-Bytes-Used

I've got a few questions:
Will there be separate count stored for files vs directories ?
Does the count represent entire sub tree or just the immediate directory contents ?
Would a getxattr on a dir for count aggregate counts from all distributed bricks ?

Thanks.

Regards,
 -Prashanth Pai

----- Original Message -----
From: "Vijaikumar M" <vmallika at redhat.com>
To: "gluster-devel at gluster.org >> Gluster Devel" <gluster-devel at gluster.org>
Sent: Tuesday, February 17, 2015 5:22:34 PM
Subject: [Gluster-devel] Object Quota feature proposal for GlusterFS-3.7

Hi All, 

We are proposing files/objects quota feature for GlusterFS-3.7. 

Here is the feature page: http://www.gluster.org/community/documentation/index.php/Features/Object_Count 


'Object Quotas' is an enhancement to the existing 'File Usage Quotas' and has the following benefits: 


    * Easy to query number of objects present in a volume. 
    * Can serve as an accounting mechanism for quota enforcement based on number of Inodes. 
    * This interface will be useful for integration with OpenStack Swift and Ceilometer. 

We can set the following Quota object limits, similar to file usage: 


    * 1) Directory level - limit the number of files at the directory level 
    * 2) Volume level - limit the number of files at the volume level 



Looking forward for Question/Feedback. 

Thanks, 
Vijay 


_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel
_______________________________________________
Gluster-devel mailing list
Gluster-devel at gluster.org
http://www.gluster.org/mailman/listinfo/gluster-devel