[Gluster-users] Docker swarm on top of Replica 3?

[Strahil Nikolov]

On February 23, 2020 12:06:03 PM GMT+02:00, Shareef Jalloq <shareef at jalloq.co.uk> wrote:
>Thanks for this, I missed it when it came through originally.
>
>My next question is where is the detailed documentation for storage
>options?  The 'Formatting and Mounting Bricks' section of the
>documentation
>seems to give an example of a single drive mounted as a thin volvolume 


You can use either https://docs.gluster.org/en/latest/ or the documents at:
https://access.redhat.com/documentation/en-US/red_hat_gluster_storage/3.5/

>Where
>is the documentation for the benefits/drawbacks of different options,
>RAID
>configs, requirements vs recommendations?

I haven't seen such, but there  could be any.
If you care  for your data use volume of type 'replicate' . If you need to save bandwidth you can use arbiter as the third brick - so your client will write the data only at 2  gluster nodes, while the third get's only metadata information.
The replicate volume can easily be extended and converted to 'replicated distributed' volume.
My personal opinion is  to use  each disk as a separate brick.This works quite well for small amount of disks,  but if you have alot of disks - that could cause a problem. Most of the gluster  users here  setup Raid5 or Raid6 of spinning disks which is reducing the recover time when  a disk in a raid fails. Keep in mind that 'replicate' type of volumes  are excellent in read performance, as  your client  is reading from 3  bricks at the same time.

>For example, in my case of having two SSDs, valid questions might be:
> - what's the best configuration for performance
> - can I create a striped LV to use as a brick or should I just be
>creating two bricks from the two SSDs?
>- what are the considerations for being able to expand the pool at a
>later
>stage?
As I mentioned  , you have the freedom to take both of the approaches.

With the stripped LV - you can put bigger files inside the brick (when sharding is not used). More data has to be replicated after a single disk failiure  (as the whole brick is lost). As those are SSDs, there is a high chance that both fail in a very short time. We had a raid1 of SSDs where both disks died in less than a week from each other.
As the writes are replicated  accross  the cluster - you could expect the same on the other gluster peer.
So I would recommend you to add 1 SSD per gluster peer (arbiter should be OK, as it writes less data), fill in some data and start testing. Two weeks later you can add  a second group of bricks and you will know that you have a week after  the first group fails.

In my case, I have completely reorganized my storage while the volumes were running. I've moved to a thin LVM (gluster needs it for snapshits) one brick at a time,  so you can reorganize  your  volumes' bricks layout when needed.

>I can't find any sort of discussion on these sort of questions that
>would
>be asked in a planning phase.
>
It will be better to share your current  hardware that is available (Servers, disks, NICs) , and performance requirements - and only then you can get recommendations.

You have a plethora of options:
Use each disk as a replica brick.
Use Raid (maybe raid0 for 2 disks or raid5 for more disks) and ontop of that you can create your bricks.
You can create create a raid of spinning disks with LVM cache from those  2 SSDs (either write-through or write-back mode).
Linux gives you a lots of opportunities - even to use ZFS on linux (some gluster members can share their experience on that).



>Thanks.
>
>On Mon, Feb 17, 2020 at 10:18 AM Strahil Nikolov
><hunter86_bg at yahoo.com>
>wrote:
>
>> On February 17, 2020 9:10:42 AM GMT+02:00, Shareef Jalloq <
>> shareef at jalloq.co.uk> wrote:
>> >Hi there,
>> >
>> >New user here and I'm still reading documentation but had a question
>> >regarding suitability of Gluster for different applications. I've
>just
>> >read
>> >the note that states Gluster isn't suitable for things like a NoSQL
>DB
>> >and
>> >wanted to know more.
>> >
>> >So on the DB front, what's the technical reason for this? High iops
>to
>> >small files? Have I missed the documentation for this?
>> >
>> >What I'm trying to do is build a HA Docker Swarm on top of Gluster.
>I
>> >was
>> >assuming I could just mount the Gluster volume to
>/mnt/persistent_data,
>> >or
>> >whatever, and use Docker Volumes to map into the containers? Are
>there
>> >reasons for not doing this?
>> >
>> >One of the services I need to run is Git LFS which uses a DB to
>store
>> >large/binary files and uses file locking. Is this an issue?
>> >
>> >Thanks, Shareef.
>>
>> Hi Shareef,
>>
>> Actually there is no problem to run a DB ontop of gluster.
>> There is a set of predefined settings for db workload:
>>
>>  '[root at host groups]# ll /var/lib/glusterd/groups/db-workload
>> -rw-r--r--. 1 root root 337 Oct 16 13:57
>> /var/lib/glusterd/groups/db-workload
>>
>> The only limit is the IOPS of your disks and the bandwidth between
>the
>> clients and nodes. Gluster supports RDMA, so lattency can be kept to
>the
>> minimum and with NVMEs , you can reach (and even exceed)  performance
>of
>> most storages while having the ability to scale-out as per your
>needs.
>> For high performance , you should consider  using libgfapi or NFS
>Ganesha.
>>
>>
>> Best Regards,
>> Strahil Nikolov
>>
Best Regards,
Strahil Nikolov