[Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello,

I recently upgraded[2] our servers from GlusterFS 3.8 (old GlusterFS
repo for Ubuntu 16.04) to 6.0 (gotten from the GlusterFS PPA for
Ubuntu 16.04 "xenial").

The sustained write performance nearly dropped to half it was before.
We copy a large (a few 10'000s) number of image files (each 2 to 10 MB
size) from the microscope where they were acquired to a SAMBA server
which mounts[1] the GlusterFS volume; before the upgrade, we could
write at about 40MB/s, after the upgrade, this dropped to 20MB/s.

This is the version of server and client software installed:
```
$ dpkg -l '*gluster*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                           Version              Architecture
      Description
+++-==============================-====================-====================-==================================================================
ii  glusterfs-client               6.5-ubuntu1~xenial1  amd64
      clustered file-system (client package)
ii  glusterfs-common               6.5-ubuntu1~xenial1  amd64
      GlusterFS common libraries and translator modules
ii  glusterfs-server               6.5-ubuntu1~xenial1  amd64
      clustered file-system (server package)
```
Op version has been upped to 60000:
```
$ sudo gluster volume get all cluster.op-version
Option                                  Value
------                                  -----
cluster.op-version                      60000

$ sudo gluster volume get all cluster.max-op-version
Option                                  Value
------                                  -----
cluster.max-op-version                  60000
```

Running `sudo gluster volume status all clients` reports that all
clients are on op-version 60000, too.

Any suggestions on what to look for or changes to try out?

Thanks,
Riccardo

[1]: I have tried both the config where SAMBA 4.8 is using the
vfs_glusterfs.so backend, and the one where `smbd` is just writing to
a locally-mounted directory.  Doesn't seem to make a difference.

[2]: Actually, since the servers are VMs on an OpenStack cloud, I
created new virtual machines, installed GlusterFS 6 fresh, mounted the
old bricks in the same brick locations, and restarted the cluster.  I
had to fiddle a bit with the files in `/var/lib/glusterfs` because the
hostnames and IPs changed but did not do anything else than `sed -e
s/old_hostname/new_hostname/` or similarly renaming files. In
particular, I did not touch the extended attributes in the brick
directory.