[Gluster-users] Terrible performance when a gluster node is down/rebooting

[David Cowe]

Hello,

We have a 5 node gluster setup with the following configuration;


[root at gb0015nasslow01 Test]# gluster vol info all

Volume Name: data
Type: Tier
Volume ID: 702daa3d-b3fa-4e66-bcec-a13f7ec1d47d
Status: Started
Number of Bricks: 6
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: glusternasbackup01:/bricks/hotbrick/brick
Brick2: glusternasfast02:/bricks/brick4/brick
Brick3: glusternasfast01:/bricks/brick3/brick
Cold Tier:
Cold Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick4: glusternasslow01:/bricks/brick1/brick
Brick5: glusternasslow02:/bricks/brick2/brick
Brick6: glusternasbackup01:/bricks/coldbrick/brick
Options Reconfigured:
performance.write-behind-window-size: 100mb
performance.cache-size: 20GB
storage.batch-fsync-delay-usec: 0
server.allow-insecure: on
performance.stat-prefetch: off
performance.readdir-ahead: on
features.ctr-enabled: on
cluster.tier-mode: cache


When we shutdown/reboot/pull the power from 1 of the nodes, the other
servers are very laggy when even doing an ls within the glusterfs mounted
filesystem (/glusterdata).

df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  4.7G   46G  10% /
devtmpfs                  48G     0   48G   0% /dev
tmpfs                     48G     0   48G   0% /dev/shm
tmpfs                     48G   66M   48G   1% /run
tmpfs                     48G     0   48G   0% /sys/fs/cgroup
/dev/sdb1                110T  221M  110T   1% /bricks/brick1
/dev/sda2                497M  201M  297M  41% /boot
/dev/sda1                200M  9.5M  191M   5% /boot/efi
/dev/mapper/centos-home  225G   33M  224G   1% /home
localhost:data           109T  464G  108T   1% /glusterdata
tmpfs                    9.5G     0  9.5G   0% /run/user/0

During the time the node is down or coming back up, we not only get the lag
described above on the server but file reads and writes to the gluster
volume pretty much stall.




Can anyone give me some much needed advice as surely this should not be the
case?




*packages installed;*

yum list gluster*

Installed Packages
glusterfs.x86_64                        3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-api.x86_64                    3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-cli.x86_64                    3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-client-xlators.x86_64         3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-fuse.x86_64                   3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-libs.x86_64                   3.7.14-1.el7
@glusterfs-3.7-x86_64
glusterfs-server.x86_64                 3.7.14-1.el7
@glusterfs-3.7-x86_64


yum list samba*

Installed Packages
samba.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client.x86_64
        4.2.10-7.el7_2
@centos7-updates-x86_64
samba-client-libs.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common.noarch
        4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-libs.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64
samba-common-tools.x86_64
        4.2.10-7.el7_2
@centos7-updates-x86_64
samba-libs.x86_64
        4.2.10-7.el7_2
@centos7-updates-x86_64
samba-vfs-glusterfs.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-clients.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64
samba-winbind-modules.x86_64
         4.2.10-7.el7_2
@centos7-updates-x86_64


yum list ctdb*

Installed Packages
ctdb.x86_64
4.2.10-7.el7_2
@centos7-updates-x86_64


Background - We use CTDB for VIP's and heartbeat and we present our gluster
volume via samba to our Windows clients.

Regards,
David
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160830/bd3d63f0/attachment.html>