<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<div>
<div id="x_compose-container" itemscope="" itemtype="https://schema.org/EmailMessage" style="direction:ltr">
<span itemprop="creator" itemscope="" itemtype="https://schema.org/Organization"><span itemprop="name"></span></span>
<div>
<div>
<div style="direction:ltr">Thanks Diego. This is invaluable information, appreciate it immensely. I had heard previously that you can always go back to previous Gluster binaries, but without understanding the data structures behind Gluster, I had no idea how
safe that was. Backing up the lib folder makes perfect sense.</div>
<div><br>
</div>
<div style="direction:ltr">The performance issues we're specifically keen to address are the small-file performance improvements introduced in 3.7. I feel that a lot of the complaints we get are from people using apps that are {slowly} crawling massively deep
folders via SMB. I'm hoping that the improvements made in 3.7 have stayed intact in 3.10! Otherwise, is there a generally accepted "fast and stable" version earlier than 3.10 that we should be looking at as an interim step?</div>
<div><br>
</div>
<div style="direction:ltr">Brett</div>
</div>
<div><br>
</div>
<div class="x_acompli_signature"></div>
</div>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Diego Remolina <dijuremo@gmail.com><br>
<b>Sent:</b> Tuesday, August 8, 2017 10:39:27 PM<br>
<b>To:</b> Brett Randall<br>
<b>Cc:</b> gluster-users@gluster.org List<br>
<b>Subject:</b> Re: [Gluster-users] Upgrading from 3.6.3 to 3.10/3.11</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">I had a mixed experience going from 3.6.6 to 3.10.2 on a two server<br>
setup. I have since upgraded to 3.10.3 but I still have a bad problem<br>
with specific files (see CONS below).<br>
<br>
PROS<br>
- Back on a "supported" version.<br>
- Windows roaming profiles (small file performance) improved<br>
significantly via samba. This may be due to new tuning options added<br>
(see my tuning options for the volume below):<br>
Volume Name: export<br>
Type: Replicate<br>
Volume ID: ---snip---<br>
Status: Started<br>
Snapshot Count: 0<br>
Number of Bricks: 1 x 2 = 2<br>
Transport-type: tcp<br>
Bricks:<br>
Brick1: 10.0.1.7:/bricks/hdds/brick<br>
Brick2: 10.0.1.6:/bricks/hdds/brick<br>
Options Reconfigured:<br>
performance.stat-prefetch: on<br>
performance.cache-min-file-size: 0<br>
network.inode-lru-limit: 65536<br>
performance.cache-invalidation: on<br>
features.cache-invalidation: on<br>
performance.md-cache-timeout: 600<br>
features.cache-invalidation-timeout: 600<br>
performance.cache-samba-metadata: on<br>
transport.address-family: inet<br>
server.allow-insecure: on<br>
performance.cache-size: 10GB<br>
cluster.server-quorum-type: server<br>
nfs.disable: on<br>
performance.io-thread-count: 64<br>
performance.io-cache: on<br>
cluster.lookup-optimize: on<br>
cluster.readdir-optimize: on<br>
server.event-threads: 5<br>
client.event-threads: 5<br>
performance.cache-max-file-size: 256MB<br>
diagnostics.client-log-level: INFO<br>
diagnostics.brick-log-level: INFO<br>
cluster.server-quorum-ratio: 51%<br>
<br>
CONS<br>
- New problems came up with specific files (Autodesk Revit files) for<br>
which no solution has been found, other than stop using samba vfs<br>
gluster plugin and also doing some stupid file renaming game. See:<br>
<a href="http://lists.gluster.org/pipermail/gluster-users/2017-June/031377.html">http://lists.gluster.org/pipermail/gluster-users/2017-June/031377.html</a><br>
- With 3.6.6 I had a nightly rsync process that would copy all the<br>
data from the gluster server pair to another server (nightly backup).<br>
This operation used to finish between 1-2AM every day. After upgrade,<br>
this operation is much slower with rsync finishing up between 3-5AM.<br>
- I have not looked a lot into it, but after 40-ish days after the<br>
upgrade, the gluster mount in one server became stuck and I had to<br>
reboot the servers.<br>
<br>
As for recommendations, definitively do *not* go with 3.11 as that is<br>
*not* a long term release. Stay with 3.10.<br>
<a href="https://www.gluster.org/community/release-schedule/">https://www.gluster.org/community/release-schedule/</a><br>
<br>
Make sure you have the 3.6.3 rpms available to downgrade if needed.<br>
You can always go back to the previous rpms if you have them available<br>
(this is not easy if you have a mix with other distros, i.e ubuntu,<br>
where the ppa only have the latest .deb file for each minor version).<br>
<br>
You must schedule downtime and bring the whole gluster down for the<br>
upgrade. Upgrade all servers, then clients then test, test, test and<br>
test more (I did not notice my Revit file problem until users brought<br>
it to my attention). If things are going well in your testing, then<br>
you should do the op version upgrade, but not before committing to<br>
staying with 3.10. It is truth you can lower the op version later<br>
manually, but then you have to manually edit several files on each<br>
server, so I say, stay with the *older* op version until you are sure<br>
you want to stay on 3.10 then upgrade the op version.<br>
<br>
<a href="https://gluster.readthedocs.io/en/latest/Upgrade-Guide/op_version/">https://gluster.readthedocs.io/en/latest/Upgrade-Guide/op_version/</a><br>
<br>
Prior to any changes, backup all your gluster server configuration<br>
folders ( /var/lib/glusterd/ ) in every single server. That will allow<br>
you to go back to the moment before upgrade if really needed.<br>
<br>
HTH,<br>
<br>
Diego<br>
<br>
<br>
<br>
On Tue, Aug 8, 2017 at 6:51 AM, Brett Randall <brett.randall@gmail.com> wrote:<br>
> Hi all<br>
><br>
> We have a 20-node, 1pb Gluster deployment that is running 3.6.3 - the same<br>
> version we installed on day 1. There are obviously numerous performance and<br>
> feature improvements that we'd like to take advantage of. However, this is a<br>
> production system and we don't have a replica of it that we can test the<br>
> upgrade on.<br>
><br>
> We're running CentOS 6.6 with official Gluster binaries. We rely on<br>
> Gluster's NFS daemon, and also use samba-glusterfs with samba for SMB access<br>
> to our Gluster volume.<br>
><br>
> What risks might we face with an upgrade from 3.6 to 3.10/3.11? And what<br>
> rollback options do we have?<br>
><br>
> More importantly, is there anyone who would be willing to work for a<br>
> retainer plus worked hours to be "on call" in case we have problems during<br>
> the upgrade? Someone with plenty of experience in Gluster over the years and<br>
> could diagnose any issues we may experience in an upgrade. If you're<br>
> interested, please e-mail me off-list. I'm, of course, interested in advice<br>
> on-list as well.<br>
><br>
> Thanks<br>
><br>
> Brett.<br>
><br>
> _______________________________________________<br>
> Gluster-users mailing list<br>
> Gluster-users@gluster.org<br>
> <a href="http://lists.gluster.org/mailman/listinfo/gluster-users">http://lists.gluster.org/mailman/listinfo/gluster-users</a><br>
</div>
</span></font>
</body>
</html>