<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1926525053;
mso-list-type:hybrid;
mso-list-template-ids:-842622344 470351889 470351897 470351899 470351887 470351897 470351899 470351887 470351897 470351899;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-ZA link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal>Hi All,<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have a couple of questions which I hope that there’s someone to just shed some light on for me please. I don’t think it’s anything serious, other than really just trying to better understand how the underlying GlusterFS works.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Firstly, I plan to build a GluserFS using SM 5018D8-* boxes. Essentially, 12 x 10TB disks, 128GB Ram, and a Xeon D-1537 CPU. It does have an on-board LSI RAID controller, but there’s not a lot of detail forthcoming in terms of RAID configurations (caches for example).<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Firstly, in terms of the nodes, I don’t care TOO much for data integrity (i.e. it is OK to lose SOME of data, but availability in terms of underlying hardware is more important). Secondly, it may not be the perfect scenario for GlusterFS (although it works perfectly fine currently through NFS on standard servers), but we are talking about millions of > 500K < 1M files. Files are stored in a very specific structure, so each file is read/written precisely to a unique directory. There’s no expensive scanning of directories (i.e. ls) happening or anything like that. It’s a simple and very static read/write operation for each file on the system.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Currently we store articles using a MD5 hash algorithm for a file name, and use 5 directory levels, so, /a/a0/a02/a02b/a02ba/a02ba1234567813dfa23bd2348901d33 Again everything works fine using multiple servers and standard ext4 / nfs exports. We host /a on one server, /b on another server, etc. So whilst the directories (and IO load) is split to address load issues, we are a bit limited in terms of how and how much we can expand. I’m hoping to move all of this to GlusterFS. The applications are very random IO intensive, and whilst we are nowhere CLOSE to the capabilities of the hardware, it is actually the seek times that are our limiting factor and the biggest bottleneck. Therefore, I am fairly certain that growing through NFS, or, GlusterFS should be suitable and workable for us.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>My main reason for wanting to go GlusterFS is mostly related to better and easier expansion of storage. It seems that it is easier to manage, whilst also providing some degree of redundancy (even if only partially in the case of a Distributed volume, which I believe would be adequate for us). All drives are hot swappable, and we will more than likely either look at a Distributed, or Stripped volume. In the case of a Distributed system, we can still live with the fact that the majority of files remain available, whilst a certain amount of files becomes unavailable should a node or brick fail, so Distributed will more than likely be adequate for our needs. Stripped would be nice to have, but I think it would have some complexities given our specific use case. We are also talking high concurrently (we do about 6K read/writes per second over NFS currently, per NFS server)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>1 On the client(s), mounting the GlusterFS the documentation is clear in that it will only fetch the GlusterFS configuration, whilst there after reading/writing directly to the GlusterFS nodes. How non-stop is this? If there is already a mount made and additional nodes are added / removed from the GlusterFS, does the client(s) get informed of this without the need to re-mount the file system? What about the capacity on the mount (at the client) when a node is added? Basically, how non-stop is this operation? Can I assume that (in a perfect world) the client would never need to re-mount the file system? Are there any operations in GlusterFS that would require a client to re-mount?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>2 Given the Distributed nature of GlusterFS and how files are written to nodes, would it be safe to assume that how more nodes there are in the GlusterFS, how better the IO performance would be? Surely, the IO load is distributed between the nodes, together with the individual files, right? What kind of IO could (or should) reasonably be expected given the hardware mentioned above (yes, I know this is a how long is a piece of string question)?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>3 When bricks are added / removed / expanded / rebalanced / etc… What does GlusterFS actually do? What would happen for instance, if I have a 250TB volume, with 10M files on it, and I add another node with ~50TB? What is the impact on performance whilst these expensive operations are run? Again, how non-stop is this in terms of the clients reading/writing a few thousand files per second? If running a *<b>purely</b>* Distributed volume, would a rebalance still be required when adding a new node? What impact does add/remove/rebalance have on large GlusterFS systems? Especially a rebalance, I would expect the operation to become more and more expensive as more and more bricks are added? Given the large amount of files I intend to have on GlusterFS, I am concerned about directory scans (for example) happening internally in GlusterFS…<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Hopefully I wasn’t too vague in my questions, but let’s see of some questions at least could be dealt with :-)<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks,<o:p></o:p></p><p class=MsoNormal>Chris.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>