[Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Anoop C S]

On Tue, 2019-10-29 at 10:59 +0100, Riccardo Murri wrote:
> Hello,
> 
> I recently upgraded[2] our servers from GlusterFS 3.8 (old GlusterFS
> repo for Ubuntu 16.04) to 6.0 (gotten from the GlusterFS PPA for
> Ubuntu 16.04 "xenial").
> 
> The sustained write performance nearly dropped to half it was before.
> We copy a large (a few 10'000s) number of image files (each 2 to 10
> MB
> size) from the microscope where they were acquired to a SAMBA server
> which mounts[1] the GlusterFS volume; before the upgrade, we could
> write at about 40MB/s, after the upgrade, this dropped to 20MB/s.
> 
> This is the version of server and client software installed:
> ```
> $ dpkg -l '*gluster*'
> Desired=Unknown/Install/Remove/Purge/Hold
> > Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-
> > aWait/Trig-pend
> > / Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
> > > /
> > > Name                           Version              Architecture
>       Description
> +++-==============================-====================-
> ====================-
> ==================================================================
> ii  glusterfs-client               6.5-ubuntu1~xenial1  amd64
>       clustered file-system (client package)
> ii  glusterfs-common               6.5-ubuntu1~xenial1  amd64
>       GlusterFS common libraries and translator modules
> ii  glusterfs-server               6.5-ubuntu1~xenial1  amd64
>       clustered file-system (server package)
> ```
> Op version has been upped to 60000:
> ```
> $ sudo gluster volume get all cluster.op-version
> Option                                  Value
> ------                                  -----
> cluster.op-version                      60000
> 
> $ sudo gluster volume get all cluster.max-op-version
> Option                                  Value
> ------                                  -----
> cluster.max-op-version                  60000
> ```
> 
> Running `sudo gluster volume status all clients` reports that all
> clients are on op-version 60000, too.
> 
> Any suggestions on what to look for or changes to try out?
> 
> Thanks,
> Riccardo
> 
> [1]: I have tried both the config where SAMBA 4.8 is using the
> vfs_glusterfs.so backend, and the one where `smbd` is just writing to
> a locally-mounted directory.  Doesn't seem to make a difference.

Samba v4.8 is an EOL ed version. Please consider updating Samba to at
least v4.9(rather v4.10) or higher.

Can you paste the output of `testparm -s` along with the output of
`gluster volume info <VOLNAME>` ?

> [2]: Actually, since the servers are VMs on an OpenStack cloud, I
> created new virtual machines, installed GlusterFS 6 fresh, mounted
> the
> old bricks in the same brick locations,

How did you mount old bricks in the new location?

> and restarted the cluster.  I
> had to fiddle a bit with the files in `/var/lib/glusterfs` because
> the
> hostnames and IPs changed but did not do anything else than `sed -e
> s/old_hostname/new_hostname/` or similarly renaming files. In
> particular, I did not touch the extended attributes in the brick
> directory.
> ________
> 
> Community Meeting Calendar:
> 
> APAC Schedule -
> Every 2nd and 4th Tuesday at 11:30 AM IST
> Bridge: https://bluejeans.com/118564314
> 
> NA/EMEA Schedule -
> Every 1st and 3rd Tuesday at 01:00 PM EDT
> Bridge: https://bluejeans.com/118564314
> 
> Gluster-users mailing list
> Gluster-users at gluster.org
> https://lists.gluster.org/mailman/listinfo/gluster-users