[Gluster-users] Performance drop when upgrading from 3.8 to 6.5

[Riccardo Murri]

Hello Anoop,

many thanks for your fast reply!  My comments inline below:


> > [1]: I have tried both the config where SAMBA 4.8 is using the
> > vfs_glusterfs.so backend, and the one where `smbd` is just writing to
> > a locally-mounted directory.  Doesn't seem to make a difference.
>
> Samba v4.8 is an EOL ed version. Please consider updating Samba to at
> least v4.9(rather v4.10) or higher.

This is going to be tricky: I could find no backport package of recent
SAMBA to Ubuntu 16.04; I am using this one which has SAMBA 4.8
https://launchpad.net/~mumblepins

More recent packages from either the Ubuntu or Debian repositories do
not build on Ubuntu 16.04 because of changes in the packaging
infrastructure.

Anyway, I was running SAMBA 4.8 before the upgrade and still getting
40MB/s, so I don't think SAMBA is the core of the issue...

> Can you paste the output of `testparm -s` along with the output of
> `gluster volume info <VOLNAME>` ?

Here's `testparm -s` on the server using `vfs_glusterfs` (the "active"
share is the one with the perf problems)::

```
$ testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
WARNING: The "syslog only" option is deprecated
Processing section "[homes]"
Processing section "[active]"
Loaded services file OK.
WARNING: some services use vfs_fruit, others don't. Mounting them in
conjunction on OS X clients results in undefined behaviour.

Server role: ROLE_STANDALONE

# Global parameters
[global]
    dns proxy = No
    load printers = No
    map to guest = Bad User
    name resolve order = lmhosts
    netbios name = REDACTED1
    obey pam restrictions = Yes
    pam password change = Yes
    passwd chat = *Enter\snew\s*\spassword:* %n\n
*Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
    passwd program = /usr/bin/passwd %u
    printcap cache time = 0
    printcap name = /dev/null
    security = USER
    server role = standalone server
    server string = SAMBA Server %v
    syslog only = Yes
    unix password sync = Yes
    workgroup = REDACTED
    idmap config * : backend = tdb


[homes]
    browseable = No
    comment = Work Directories
    create mask = 0700
    directory mask = 0700
    read only = No
    valid users = %S
    vfs objects = fruit streams_xattr


[active]
    create mask = 0775
    directory mask = 0775
    kernel share modes = No
    path = /active
    read only = No
    vfs objects = glusterfs
    glusterfs:volume = glusterfs
    glusterfs:volfile_server = glusterfs5 glusterfs4 glusterfs3
glusterfs2 glusterfs1
    glusterfs:logfile = /var/log/samba/glusterfs-vol-active.log
    glusterfs:loglevel = 1
```


Here's `testparm -s` on the server writing directly to the GlusterFS
mount point::

```
$ testparm -s
Load smb config files from /etc/samba/smb.conf
rlimit_max: increasing rlimit_max (1024) to minimum Windows limit (16384)
WARNING: The "syslog only" option is deprecated
Processing section "[homes]"
Processing section "[active]"
Loaded services file OK.
WARNING: some services use vfs_fruit, others don't. Mounting them in
conjunction on OS X clients results in undefined behaviour.

Server role: ROLE_STANDALONE

# Global parameters
[global]
    allow insecure wide links = Yes
    dns proxy = No
    load printers = No
    map to guest = Bad User
    name resolve order = lmhosts
    netbios name = REDACTED2
    obey pam restrictions = Yes
    pam password change = Yes
    passwd chat = *Enter\snew\s*\spassword:* %n\n
*Retype\snew\s*\spassword:* %n\n *password\supdated\ssuccessfully* .
    passwd program = /usr/bin/passwd %u
    printcap cache time = 0
    printcap name = /dev/null
    security = USER
    server role = standalone server
    server string = SAMBA Server %v
    syslog only = Yes
    unix password sync = Yes
    workgroup = REDACTED
    idmap config * : backend = tdb


[homes]
    browseable = No
    comment = Work Directories
    create mask = 0700
    directory mask = 0700
    read only = No
    valid users = %S
    vfs objects = fruit streams_xattr


[active]
    create mask = 0775
    directory mask = 0775
    path = /data/active
    read only = No
    wide links = Yes
```

Here's the volume info:
```
$ sudo gluster volume info glusterfs

Volume Name: glusterfs
Type: Distribute
Volume ID: a3358ff6-5cec-4a65-9ecf-a63bbe56dfd9
Status: Started
Snapshot Count: 0
Number of Bricks: 5
Transport-type: tcp
Bricks:
Brick1: glusterfs5:/srv/glusterfs
Brick2: glusterfs4:/srv/glusterfs
Brick3: glusterfs3:/srv/glusterfs
Brick4: glusterfs1:/srv/glusterfs
Brick5: glusterfs2:/srv/glusterfs
Options Reconfigured:
diagnostics.client-log-level: WARNING
diagnostics.brick-log-level: INFO
features.uss: disable
features.barrier: disable
performance.client-io-threads: on
transport.address-family: inet
nfs.disable: on
snap-activate-on-create: enable
```


> > [2]: Actually, since the servers are VMs on an OpenStack cloud, I
> > created new virtual machines, installed GlusterFS 6 fresh, mounted
> > the old bricks in the same brick locations,
>
> How did you mount old bricks in the new location?

They're `ext4` disks:
```
$ fgrep glusterfs /etc/fstab
LABEL=/srv/glusterfs /srv/glusterfs ext4 rw 0 0
```

Actually, the GlusterFS volume is 95% full -- could this have an impact?

Thanks,
Riccardo