[Gluster-users] Looking for use cases / opinions

Wed Nov 9 17:28:02 UTC 2016

Serkan

I'd be interested to know how your disks are attached (SAS?)?  Do you use
any hardware RAID, or zfs and do you have and SSDs in there?

On 9 November 2016 at 06:17, Serkan Çoban <cobanserkan at gmail.com> wrote:

> Hi, I am using 26x8TB disks per server. There are 60 servers in gluster
> cluster.
> Each disk is a brick and configuration is 16+4 EC, 9PB single volume.
> Clients are using fuse mounts.
> Even with 1-2K files in a directory, ls from clients takes ~60 secs.
> So If you are sensitive to metadata operations, I suggest another
> approach...
>
>
> On Wed, Nov 9, 2016 at 1:05 PM, Frank Rothenstein
> <f.rothenstein at bodden-kliniken.de> wrote:
> > As you said you want to have 3 or 4 replicas, so i would use the zfs
> > knowledge and build 1 zpool per node with whatever config you know is
> > fastest on this kind of hardware and as safe as you need (stripe,
> > mirror, raidz1..3 - resilvering zfs is faster than healing gluster, I
> > think) . 1 node -> 1 brick (per gluster volume).
> >
> > Frank
> > Am Dienstag, den 08.11.2016, 19:19 +0000 schrieb Thomas Wakefield:
> >> We haven’t decided how the JBODS would be configured.  They would
> >> likely be SAS attached without a raid controller for improved
> >> performance.  I run large ZFS arrays this way, but only in single
> >> server NFS setups right now.
> >> Mounting each hard drive as it’s own brick would probably give the
> >> most usable space, but would need scripting to manage building all
> >> the bricks.  But does Gluster handle 1000’s of small bricks?
> >>
> >>
> >>
> >> > On Nov 8, 2016, at 9:18 AM, Frank Rothenstein <f.rothenstein at bodden
> >> > -kliniken.de> wrote:
> >> >
> >> > Hi Thomas,
> >> >
> >> > thats a huge storage.
> >> > What I can say from my usecase - dont use Gluster directly if the
> >> > files
> >> > are small. I dont know, if the file count matters, but if the files
> >> > are
> >> > small (few KiB), Gluster takes ages to remove for example. Doing
> >> > the
> >> > same in a VM with e.g. ext4 disk on the very same Gluster gives a
> >> > big
> >> > speedup.
> >> > There are many options for a new Gluster volume, like Lindsay
> >> > mentioned.
> >> > And there are other options, like Ceph, OrangeFS.
> >> > How do you want to use the JBODs? I dont think you would use every
> >> > single drive as a brick... How are these connected to the servers?
> >> >
> >> > Im only dealing with about 10TiB Gluster volumes, so by far not at
> >> > your
> >> > planned level, but I really would like to see some results, if you
> >> > go
> >> > for Gluster!
> >> >
> >> > Frank
> >> >
> >> >
> >> > Am Dienstag, den 08.11.2016, 13:49 +0000 schrieb Thomas Wakefield:
> >> > > I think we are leaning towards erasure coding with 3 or 4
> >> > > copies.  But open to suggestions.
> >> > >
> >> > >
> >> > > > On Nov 8, 2016, at 8:43 AM, Lindsay Mathieson <lindsay.mathieso
> >> > > > n at gm
> >> > > > ail.com> wrote:
> >> > > >
> >> > > > On 8/11/2016 11:38 PM, Thomas Wakefield wrote:
> >> > > > > High Performance Computing, we have a small cluster on campus
> >> > > > > of
> >> > > > > about 50 linux compute servers.
> >> > > > >
> >> > > >
> >> > > > D'oh! I should have thought of that.
> >> > > >
> >> > > >
> >> > > > Are you looking at replication (2 or 3)/disperse or pure
> >> > > > disperse?
> >> > > >
> >> > > > --
> >> > > > Lindsay Mathieson
> >> > > >
> >> > >
> >> > > _______________________________________________
> >> > > Gluster-users mailing list
> >> > > Gluster-users at gluster.org
> >> > > http://www.gluster.org/mailman/listinfo/gluster-users
> >> >
> >> >
> >> >
> >> >
> >> >
> >> > ___________________________________________________________________
> >> > ___________
> >> > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
> >> > Sandhufe 2
> >> > 18311 Ribnitz-Damgarten
> >> >
> >> > Telefon: 03821-700-0
> >> > Fax:       03821-700-240
> >> >
> >> > E-Mail: info at bodden-kliniken.de   Internet: http://www.bodden-klini
> >> > ken.de
> >> >
> >> > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-
> >> > Nr.: 079/133/40188
> >> > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr.
> >> > Falko Milski
> >> >
> >> > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten
> >> > Adressaten bestimmt. Wenn Sie nicht der vorge-
> >> > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten,
> >> > beachten Sie bitte, dass jede Form der Veröf-
> >> > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-
> >> > Mail unzulässig ist. Wir bitten Sie, sofort den
> >> > Absender zu informieren und die E-Mail zu löschen.
> >> >
> >> >
> >> >             Bodden-Kliniken Ribnitz-Damgarten GmbH 2016
> >> > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
> >> >
> >>
> >>
> >
> >
> >
> >
> >
> > ____________________________________________________________
> __________________
> > BODDEN-KLINIKEN Ribnitz-Damgarten GmbH
> > Sandhufe 2
> > 18311 Ribnitz-Damgarten
> >
> > Telefon: 03821-700-0
> > Fax:       03821-700-240
> >
> > E-Mail: info at bodden-kliniken.de   Internet:
> http://www.bodden-kliniken.de
> >
> > Sitz: Ribnitz-Damgarten, Amtsgericht: Stralsund, HRB 2919, Steuer-Nr.:
> 079/133/40188
> > Aufsichtsratsvorsitzende: Carmen Schröter, Geschäftsführer: Dr. Falko
> Milski
> >
> > Der Inhalt dieser E-Mail ist ausschließlich für den bezeichneten
> Adressaten bestimmt. Wenn Sie nicht der vorge-
> > sehene Adressat dieser E-Mail oder dessen Vertreter sein sollten,
> beachten Sie bitte, dass jede Form der Veröf-
> > fentlichung, Vervielfältigung oder Weitergabe des Inhalts dieser E-Mail
> unzulässig ist. Wir bitten Sie, sofort den
> > Absender zu informieren und die E-Mail zu löschen.
> >
> >
> >              Bodden-Kliniken Ribnitz-Damgarten GmbH 2016
> > *** Virenfrei durch Kerio Mail Server und Sophos Antivirus ***
> >
> > _______________________________________________
> > Gluster-users mailing list
> > Gluster-users at gluster.org
> > http://www.gluster.org/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://www.gluster.org/mailman/listinfo/gluster-users
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20161109/f18255bb/attachment.html>