[Gluster-users] Replicated and Non Replicated Bricks on Same Partition

[Robert Hajime Lanning]

On 04/29/13 20:28, Anand Avati wrote:
>
> On Mon, Apr 29, 2013 at 9:19 AM, Heath Skarlupka 
> <heath.skarlupka at ssec.wisc.edu <mailto:heath.skarlupka at ssec.wisc.edu>> 
> wrote:
>
>     Gluster-Users,
>
>     We currently have a 30 node Gluster Distributed-Replicate 15 x 2
>     filesystem.  Each node has a ~20TB xfs filesystem mounted to /data
>     and the bricks live on /data/brick.  We have been very happy with
>     this setup, but are now collecting more data that doesn't need to
>     be replicated because it can be easily regenerated.  Most of the
>     data lives on our replicated volume and is starting to waste
>     space.  My plan was to create a second directory under the /data
>     partition called /data/non_replicated_brick on each of the 30
>     nodes and start up a second Gluster filesystem.  This would allow
>     me to dynamically size the replicated and non_replicated space
>     based on our current needs.
>
>     I'm a bit worried about going forward with this because I haven't
>     seen many users talk about putting two gluster bricks on the same
>     underlying filesystem.  I've gotten passed the technical hurdle
>     and know that it is technically possible, but I'm worried about
>     corner cases and issues that might crop up when we add more bricks
>     and need to rebalance both gluster volumes at once.  Does anybody
>     have any insight in what the caveats of doing this are or are
>     there any users putting multiple bricks on a single filesystem in
>     the 50-100 node size range.  Thank you all for your insights and help!
>
>
> This is a very common use case and should work fine. In the future we 
> are exploring better integration with dm-thinp so that each brick has 
> its own XFS filesystem on a thin provisioned logical volume. But for 
> now you can create a second volume on the same XFS filesystems.
>
> Avati
>

There is an issue when replicated bricks fill unevenly.  The 
non-replicated volume will cause uneven filling of bricks as seen in the 
replicated volume.

I am not sure how ENOSPC is handled asymmetrically, but if the fuller 
brick happens to be down during a write that would be causing ENOSPC, 
you won't get the error and replication will fail, when the self-heal 
kicks in.

-- 
Mr. Flibble
King of the Potato People