[Gluster-devel] [Gluster-users] Quota list not reflecting disk usage

Steve Dainard sdainard at spd1.com
Thu Feb 11 19:58:29 UTC 2016


What would happen if I:
- Did not disable quotas
- Did not stop the volume (140T volume takes at least 3-4 days to do
any find operations, which is too much downtime)
- Find and remove all xattrs:
trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri on
the /brick/volumename/modules
- set the dirty bit on /brick/volumename/modules

As far as an upgrade to 3.7, I'm not comfortable with running the
newest release - which version is RHGS based on? I typically like to
follow supported product version if I can, so I know most of the kinks
are worked out :)

On Wed, Feb 10, 2016 at 11:02 PM, Manikandan Selvaganesh
<mselvaga at redhat.com> wrote:
> Hi Steve,
>
> We suspect the mismatching in accounting is probably because of the
> xattr's being not cleaned up properly. Please ensure you do the following
> steps and make sure the xattr's are cleaned up properly before quota
> is enabled for the next time.
>
> 1) stop the volume
> 2) on each brick in the backend do
>    Find and remove all the xattrs and make sure they are not present
>      # find <brickpath>/module | xargs getfattr -d -m . -e hex | grep quota | grep -E 'contri|size'
>      # setxattr -x xattrname <path>
>
> 3) set dirty on <brickpath>/
>      # setxattr -n trusted.glusterfs.quota.dirty -v 0x3100
>    By setting dirty value on root as 1(0x3100), the contri will be calculated again
>    and the proper contri will be crawled and updated again.
>
> 4) Start volume and from a fuse mount
>        # stat /mountpath
>
> If you have ever performed a rename, then there is a possibility of two contributions
> getting created for a single entry.
>
> We have fixed quite some rename issues and have refactored the marker approach. Also
> as I have mentioned already we have also done Versioning of xattr's which solves the
> issue you are facing in 3.7. It would be really helpful in a production environment if
> you could upgrade to 3.7
>
> --
> Thanks & Regards,
> Manikandan Selvaganesh.
>
> ----- Original Message -----
> From: "Steve Dainard" <sdainard at spd1.com>
> To: "Manikandan Selvaganesh" <mselvaga at redhat.com>
> Cc: "Vijaikumar Mallikarjuna" <vmallika at redhat.com>, "Gluster Devel" <gluster-devel at gluster.org>, "gluster-users at gluster.org List" <gluster-users at gluster.org>
> Sent: Thursday, February 11, 2016 1:48:19 AM
> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>
> So after waiting out the process of disabling quotas, waiting for the
> xattrs to be cleaned up, re-enabling quotas and waiting for the
> xattr's to be created, then applying quotas I'm running into the same
> issue.
>
> Yesterday at ~2pm one of the quotas was listed as:
> /modules|100.0GB|18.3GB|81.7GB
>
> I initiated a copy from that glusterfs fuse mount to another fuse
> mount for a different volume, and now I'm seeing:
> /modules|100.0GB|27.4GB|72.6GB
>
> So an increase of 9GB usage.
>
> There were no writes at all to this directory during or after the cp.
>
> I did a bit of digging through the /modules directory on one of the
> gluster nodes and created this spreadsheet:
> https://docs.google.com/spreadsheets/d/1l_6ze68TCOcx6LEh9MFwmqPZ9bM-70CUlSM_8tpQ654/edit?usp=sharing
>
> The /modules/R/3.2.2 directory quota value doesn't come close to
> matching the du value.
>
> Funny bit, there are TWO quota contribution attributes:
> # getfattr -d -m quota -e hex 3.2.2
> # file: 3.2.2
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x0000000009af6000
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x0000000013fda800
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.size=0x0000000013fda800
>
> For reference, another directory /modules/R/2.14.2 has only one
> contribution attribute:
> # getfattr -d -m quota -e hex 2.14.2
> # file: 2.14.2
> trusted.glusterfs.quota.c890be20-1bb9-4aec-a8d0-eacab0446f16.contri=0x0000000000692800
> trusted.glusterfs.quota.dirty=0x3000
> trusted.glusterfs.quota.size=0x0000000000692800
>
> Questions:
> 1. Why wasn't the
> trusted.glusterfs.quota.242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3.contri=0x0000000009af6000
> cleaned up?
> 2A. How can I remove old attributes from the fs, and then force a
> re-calculation of contributions for the quota path /modules once I've
> done this on all gluster nodes?
> 2B. Or am I stuck yet again removing quotas completely, waiting for
> the automated setfattr to remove the quotas for
> c890be20-1bb9-4aec-a8d0-eacab0446f16 ID, manually removing attrs for
> 242dcfd9-6aea-4cb8-beb2-c0ed91ad70d3, re-enabling quotas, waiting for
> xattrs to be generated, then enabling limits?
> 3. Shouldn't there be a command to re-trigger quota accounting on a
> directory that confirms the attrs are set correctly and checks that
> the contribution attr actually match disk usage?
>
> On Tue, Feb 2, 2016 at 3:00 AM, Manikandan Selvaganesh
> <mselvaga at redhat.com> wrote:
>> Hi Steve,
>>
>> As you have mentioned, if you are using a glusterfs version lesser than 3.7,
>> then you are doing it right. We are sorry to say but unfortunately that's the only
>> way(manually going and cleaning up the xattr's before enabling quota or wait for
>> the process to complete itself, which would take quite some time depending upon the
>> files) that can be done so as not to mess up quota enforcing/accounting. Also, we could
>> not find anything that could help us with the logs too. Thanks for the
>> point. We are in the process of writing blogs and documenting clearly about quota and
>> it's internal working. There is an initial blog[1] which we have written. More blogs will
>> follow.
>>
>> With glusterfs-3.7, we have introduced something called "Quota versioning".
>> So whenever you enable quota, we are suffixing a number(1..N) with the quota xattr's,
>> say you enable quota for the first time and the xattr will be like,
>> "trusted.glusterfs.quota.size.<suffix number from 1..N>". So all the quota related xattr's
>> will have the number suffixed to the xattr. With the versioning patch[2], when you disable and
>> enable quota again for the next time, it will be "trusted.glusterfs.quota.size.2"(Similarly
>> for other quota related xattr's). So quota accounting can happen independently depending on
>> the suffix and the cleanup process can go on independently which solves the issue that you
>> have.
>>
>> [1] https://manikandanselvaganesh.wordpress.com/
>>
>> [2] http://review.gluster.org/12386
>>
>> --
>> Thanks & Regards,
>> Manikandan Selvaganesh.
>>
>> ----- Original Message -----
>> From: "Vijaikumar Mallikarjuna" <vmallika at redhat.com>
>> To: "Steve Dainard" <sdainard at spd1.com>
>> Cc: "Manikandan Selvaganesh" <mselvaga at redhat.com>
>> Sent: Tuesday, February 2, 2016 10:12:51 AM
>> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>
>> Hi Steve,
>>
>> Sorry for the delay. Mani and myself was busy with something else at work,
>> we will update you on this by eod.
>>
>> Many quota issues has been fixed in 3.7, also version numbers are added to
>> quota xattrs, so when quota is disabled we don't need to cleanup the xattrs.
>>
>> Thanks,
>> Vijay
>>
>>
>>
>>
>>
>> On Tue, Feb 2, 2016 at 12:26 AM, Steve Dainard <sdainard at spd1.com> wrote:
>>
>>> I haven't heard anything back on this thread so here's where I've landed:
>>>
>>> It appears that the quota xattr's are not being cleared when quota's
>>> are disabled, so when they are disabled and re-enabled the value for
>>> size is added to the previous size, making it appear that the 'Used'
>>> space is significantly greater than it should be. This seems like a
>>> bug, but I don't know what to file it against, or if the logs I
>>> attached prove this.
>>>
>>> Also; the documentation doesn't make mention of how the quota system
>>> works, and what happens when quotas are enabled/disabled. There seems
>>> to be a background task for both settings:
>>> On enable: "/usr/bin/find . -exec /usr/bin/stat {} \ ;"
>>> On disable: setfattr is removing quota xattrs
>>>
>>> The thing is neither of these tasks are listed in 'gluster volume
>>> status <volume>' ie:
>>>
>>> Status of volume: storage
>>> Gluster process Port Online Pid
>>>
>>> ------------------------------------------------------------------------------
>>> Brick 10.0.231.50:/mnt/raid6-storage/storage 49156 Y 24899
>>> Brick 10.0.231.51:/mnt/raid6-storage/storage 49156 Y 2991
>>> Brick 10.0.231.52:/mnt/raid6-storage/storage 49156 Y 28853
>>> Brick 10.0.231.53:/mnt/raid6-storage/storage 49153 Y 2705
>>> NFS Server on localhost N/A N N/A
>>> Quota Daemon on localhost N/A Y 30066
>>> NFS Server on 10.0.231.52 N/A N N/A
>>> Quota Daemon on 10.0.231.52 N/A Y 24976
>>> NFS Server on 10.0.231.53 N/A N N/A
>>> Quota Daemon on 10.0.231.53 N/A Y 30334
>>> NFS Server on 10.0.231.51 N/A N N/A
>>> Quota Daemon on 10.0.231.51 N/A Y 15781
>>>
>>> Task Status of Volume storage
>>>
>>> ------------------------------------------------------------------------------
>>> ******There are no active volume tasks*******
>>>
>>> (I added the asterisks above)
>>> So without any visibility into these running tasks, or knowing of
>>> their existence (not documented) it becomes very difficult to know
>>> what's going on. On any reasonably large storage system these tasks
>>> take days to complete and there should be some indication of this.
>>>
>>> Where I'm at right now:
>>> - I disabled the quota's on volume 'storage'
>>> - I started to manually remove xattrs until I realized there is an
>>> automated task to do this.
>>> - After waiting for 'ps aux | grep setfattr' to return nothing, I
>>> re-enabled quotas
>>> - I'm currently waiting for the stat tasks to complete
>>> - Once the entire filesystem has been stat'ed, I'm going to set limits
>>> again.
>>>
>>> As a note, this is a pretty brutal process on a system with 140T of
>>> storage, and I can't imagine how much worse this would be if my nodes
>>> had more than 12 disks per, or if I was at PB scale.
>>>
>>> On Mon, Jan 25, 2016 at 12:31 PM, Steve Dainard <sdainard at spd1.com> wrote:
>>> > Here's a l link to a tarball of one of the gluster hosts logs:
>>> > https://dl.dropboxusercontent.com/u/21916057/gluster01.tar.gz
>>> >
>>> > I wanted to include past logs in case they were useful.
>>> >
>>> > Also, the volume I'm trying to get quota's working on is 'storage'
>>> > you'll notice I have a brick issue on a different volume 'vm-storage'.
>>> >
>>> > In regards to the 3.7 upgrade. I'm a bit hesitant to move to the
>>> > current release, I prefer to stay on a stable release with maintenance
>>> > updates if possible.
>>> >
>>> > On Mon, Jan 25, 2016 at 12:09 PM, Manikandan Selvaganesh
>>> > <mselvaga at redhat.com> wrote:
>>> >> Hi Steve,
>>> >>
>>> >> Also, do you have any plans to upgrade to the latest version. With 3.7,
>>> >> we have re factored some approaches used in quota and marker and that
>>> have
>>> >> fixed quite some issues.
>>> >>
>>> >> --
>>> >> Thanks & Regards,
>>> >> Manikandan Selvaganesh.
>>> >>
>>> >> ----- Original Message -----
>>> >> From: "Manikandan Selvaganesh" <mselvaga at redhat.com>
>>> >> To: "Steve Dainard" <sdainard at spd1.com>
>>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >> Sent: Tuesday, January 26, 2016 1:31:10 AM
>>> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>> >>
>>> >> Hi Steve,
>>> >>
>>> >> Could you send us the glusterfs logs, it could help us debug the issue!!
>>> >>
>>> >> --
>>> >> Thanks & Regards,
>>> >> Manikandan Selvaganesh.
>>> >>
>>> >> ----- Original Message -----
>>> >> From: "Steve Dainard" <sdainard at spd1.com>
>>> >> To: "Manikandan Selvaganesh" <mselvaga at redhat.com>
>>> >> Cc: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >> Sent: Tuesday, January 26, 2016 12:56:22 AM
>>> >> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>> >>
>>> >> Something is seriously wrong with the quota output:
>>> >>
>>> >> # gluster volume quota storage list
>>> >>                   Path                   Hard-limit Soft-limit   Used
>>> >> Available  Soft-limit exceeded? Hard-limit exceeded?
>>> >>
>>> ---------------------------------------------------------------------------------------------------------------------------
>>> >> /projects-CanSISE                         10.0TB       80%      27.8TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/climate                           105.0TB       80%     307.1TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/forestry                           50.0GB       80%      61.9GB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/projects                          800.0GB       80%       2.0TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/strays                             85.0GB       80%     230.5GB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/gis                                 2.2TB       80%       6.3TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/modperl                             1.0TB       80%     953.2GB
>>> >>  70.8GB             Yes                   No
>>> >> /data4/dem                                 1.0GB       80%      0Bytes
>>> >>   1.0GB              No                   No
>>> >> /projects-hydrology-archive0               5.0TB       80%      14.4TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /climate-downscale-idf-ec                  7.5TB       80%       5.1TB
>>> >>   2.4TB              No                   No
>>> >> /climate-downscale-idf                     5.0TB       80%       6.1TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /home                                      5.0TB       80%      11.8TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /projects-hydrology-scratch0               7.0TB       80%     169.1GB
>>> >>   6.8TB              No                   No
>>> >> /projects-rci-scratch                     10.0TB       80%       1.9TB
>>> >>   8.1TB              No                   No
>>> >> /projects-dataportal                       1.0TB       80%     775.4GB
>>> >> 248.6GB              No                   No
>>> >> /modules                                   1.0TB       80%      36.1GB
>>> >> 987.9GB              No                   No
>>> >> /data4/climate/downscale/CMIP5            65.0TB       80%      56.4TB
>>> >>   8.6TB             Yes                   No
>>> >>
>>> >> Gluster is listing 'Used' space of over 307TB on /data4/climate, but
>>> >> the volume capacity is only 146T.
>>> >>
>>> >> This has happened after disabling quotas on the volume, re-enabling
>>> >> quotas, and then setting quotas again. There was a lot of glusterfsd
>>> >> CPU usage afterwards, and now 3 days later the quota's I set were all
>>> >> missing except
>>> >>
>>> >> /data4/projects|800.0GB|2.0TB|0Bytes
>>> >>
>>> >> So I re-set the quotas and the output above is what I have.
>>> >>
>>> >> Previous to disabling quota's this was the output:
>>> >> # gluster volume quota storage list
>>> >>                   Path                   Hard-limit Soft-limit   Used
>>> >> Available  Soft-limit exceeded? Hard-limit exceeded?
>>> >>
>>> ---------------------------------------------------------------------------------------------------------------------------
>>> >> /data4/climate                           105.0TB       80%     151.6TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /data4/forestry                           50.0GB       80%      45.4GB
>>> >>   4.6GB             Yes                   No
>>> >> /data4/projects                          800.0GB       80%     753.1GB
>>> >>  46.9GB             Yes                   No
>>> >> /data4/strays                             85.0GB       80%      80.8GB
>>> >>   4.2GB             Yes                   No
>>> >> /data4/gis                                 2.2TB       80%       2.1TB
>>> >>  91.8GB             Yes                   No
>>> >> /data4/modperl                             1.0TB       80%     948.1GB
>>> >>  75.9GB             Yes                   No
>>> >> /data4/dem                                 1.0GB       80%      0Bytes
>>> >>   1.0GB              No                   No
>>> >> /projects-CanSISE                         10.0TB       80%      11.9TB
>>> >>  0Bytes             Yes                  Yes
>>> >> /projects-hydrology-archive0               5.0TB       80%       4.8TB
>>> >> 174.0GB             Yes                   No
>>> >> /climate-downscale-idf-ec                  7.5TB       80%       5.0TB
>>> >>   2.5TB              No                   No
>>> >> /climate-downscale-idf                     5.0TB       80%       3.8TB
>>> >>   1.2TB              No                   No
>>> >> /home                                      5.0TB       80%       4.7TB
>>> >> 283.8GB             Yes                   No
>>> >> /projects-hydrology-scratch0               7.0TB       80%      95.9GB
>>> >>   6.9TB              No                   No
>>> >> /projects-rci-scratch                     10.0TB       80%       1.7TB
>>> >>   8.3TB              No                   No
>>> >> /projects-dataportal                       1.0TB       80%     775.4GB
>>> >> 248.6GB              No                   No
>>> >> /modules                                   1.0TB       80%      14.6GB
>>> >> 1009.4GB              No                   No
>>> >> /data4/climate/downscale/CMIP5            65.0TB       80%      56.4TB
>>> >>   8.6TB             Yes                   No
>>> >>
>>> >> I was so focused on the /projects-CanSISE quota not being accurate
>>> >> that I missed that the 'Used' space on /data4/climate is listed higher
>>> >> then the total gluster volume capacity.
>>> >>
>>> >> On Mon, Jan 25, 2016 at 10:52 AM, Steve Dainard <sdainard at spd1.com>
>>> wrote:
>>> >>> Hi Manikandan
>>> >>>
>>> >>> I'm using 'du' not df in this case.
>>> >>>
>>> >>> On Thu, Jan 21, 2016 at 9:20 PM, Manikandan Selvaganesh
>>> >>> <mselvaga at redhat.com> wrote:
>>> >>>> Hi Steve,
>>> >>>>
>>> >>>> If you would like disk usage using df utility by taking quota limits
>>> into
>>> >>>> consideration, then you are expected to run the following command.
>>> >>>>
>>> >>>>    'gluster volume set VOLNAME quota-deem-statfs on'
>>> >>>>
>>> >>>> with older versions where quota-deem-statfs is OFF by default.
>>> However with
>>> >>>> the latest versions, quota-deem-statfs is by default ON. In this
>>> case, the total
>>> >>>> disk space of the directory is taken as the quota hard limit set on
>>> the directory
>>> >>>> of the volume and disk utility would display accordingly. This
>>> answers why there is
>>> >>>> a mismatch in disk utility.
>>> >>>>
>>> >>>> Next, answering to quota mechanism and accuracy: There is something
>>> called timeouts
>>> >>>> in quota. For performance reasons, quota caches the directory size on
>>> client. You can
>>> >>>> set timeout indicating the maximum valid duration of directory sizes
>>> in cache,
>>> >>>> from the time they are populated. By default the hard-timeout is 5s
>>> and soft timeout
>>> >>>> is 60s. Setting a timeout of zero will do a force fetching of
>>> directory sizes from server
>>> >>>> for every operation that modifies file data and will effectively
>>> disables directory size
>>> >>>> caching on client side. If you do not have a timeout of 0(which we do
>>> not encourage due to
>>> >>>> performance reasons), then till you reach soft-limit, soft timeout
>>> will be taken into
>>> >>>> consideration, and only for every 60s operations will be synced and
>>> that could cause the
>>> >>>> usage to exceed more than the hard-limit specified. If you would like
>>> quota to
>>> >>>> strictly enforce then please run the following commands,
>>> >>>>
>>> >>>>     'gluster v quota VOLNAME hard-timeout 0s'
>>> >>>>     'gluster v quota VOLNAME soft-timeout 0s'
>>> >>>>
>>> >>>> Appreciate your curiosity in exploring and if you would like to know
>>> more about quota
>>> >>>> please refer[1]
>>> >>>>
>>> >>>> [1]
>>> http://gluster.readthedocs.org/en/release-3.7.0-1/Administrator%20Guide/Directory%20Quota/
>>> >>>>
>>> >>>> --
>>> >>>> Thanks & Regards,
>>> >>>> Manikandan Selvaganesh.
>>> >>>>
>>> >>>> ----- Original Message -----
>>> >>>> From: "Steve Dainard" <sdainard at spd1.com>
>>> >>>> To: "gluster-users at gluster.org List" <gluster-users at gluster.org>
>>> >>>> Sent: Friday, January 22, 2016 1:40:07 AM
>>> >>>> Subject: Re: [Gluster-users] Quota list not reflecting disk usage
>>> >>>>
>>> >>>> This is gluster 3.6.6.
>>> >>>>
>>> >>>> I've attempted to disable and re-enable quota's on the volume, but
>>> >>>> when I re-apply the quotas on each directory the same 'Used' value is
>>> >>>> present as before.
>>> >>>>
>>> >>>> Where is quotad getting its information from, and how can I clean
>>> >>>> up/regenerate that info?
>>> >>>>
>>> >>>> On Thu, Jan 21, 2016 at 10:07 AM, Steve Dainard <sdainard at spd1.com>
>>> wrote:
>>> >>>>> I have a distributed volume with quota's enabled:
>>> >>>>>
>>> >>>>> Volume Name: storage
>>> >>>>> Type: Distribute
>>> >>>>> Volume ID: 26d355cb-c486-481f-ac16-e25390e73775
>>> >>>>> Status: Started
>>> >>>>> Number of Bricks: 4
>>> >>>>> Transport-type: tcp
>>> >>>>> Bricks:
>>> >>>>> Brick1: 10.0.231.50:/mnt/raid6-storage/storage
>>> >>>>> Brick2: 10.0.231.51:/mnt/raid6-storage/storage
>>> >>>>> Brick3: 10.0.231.52:/mnt/raid6-storage/storage
>>> >>>>> Brick4: 10.0.231.53:/mnt/raid6-storage/storage
>>> >>>>> Options Reconfigured:
>>> >>>>> performance.cache-size: 1GB
>>> >>>>> performance.readdir-ahead: on
>>> >>>>> features.quota: on
>>> >>>>> diagnostics.brick-log-level: WARNING
>>> >>>>>
>>> >>>>> Here is a partial list of quotas:
>>> >>>>> # /usr/sbin/gluster volume quota storage list
>>> >>>>>                   Path                   Hard-limit Soft-limit   Used
>>> >>>>> Available  Soft-limit exceeded? Hard-limit exceeded?
>>> >>>>>
>>> ---------------------------------------------------------------------------------------------------------------------------
>>> >>>>> ...
>>> >>>>> /projects-CanSISE                         10.0TB       80%
>>> 11.9TB
>>> >>>>>  0Bytes             Yes                  Yes
>>> >>>>> ...
>>> >>>>>
>>> >>>>> If I du on that location I do not get 11.9TB of space used (fuse
>>> mount point):
>>> >>>>> [root at storage projects-CanSISE]# du -hs
>>> >>>>> 9.5T .
>>> >>>>>
>>> >>>>> Can someone provide an explanation for how the quota mechanism tracks
>>> >>>>> disk usage? How often does the quota mechanism check its accuracy?
>>> And
>>> >>>>> how could it get so far off?
>>> >>>>>
>>> >>>>> Can I get gluster to rescan that location and update the quota usage?
>>> >>>>>
>>> >>>>> Thanks,
>>> >>>>> Steve
>>> >>>> _______________________________________________
>>> >>>> Gluster-users mailing list
>>> >>>> Gluster-users at gluster.org
>>> >>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>> >> _______________________________________________
>>> >> Gluster-users mailing list
>>> >> Gluster-users at gluster.org
>>> >> http://www.gluster.org/mailman/listinfo/gluster-users
>>> _______________________________________________
>>> Gluster-users mailing list
>>> Gluster-users at gluster.org
>>> http://www.gluster.org/mailman/listinfo/gluster-users
>>>


More information about the Gluster-devel mailing list