[Gluster-users] Upgrade 3.7.6 -> 3.7.13 one gluster server disconnected 1 of 3 volumes

[Steve Dainard]

About 5 hours after upgrading gluster 3.7.6 -> 3.7.13 on Centos 7, one of
my gluster servers disconnected its volume. The other two volumes this host
serves were not affected.

# gluster volume status storage
Status of volume: storage
Gluster process                             TCP Port  RDMA Port  Online  Pid
------------------------------------------------------------------------------
Brick 10.0.231.50:/mnt/raid6-storage/storage
            49159     0          Y       30743
Brick 10.0.231.51:/mnt/raid6-storage/storage
            49159     0          Y       676
*Brick 10.0.231.52:/mnt/raid6-storage/storag**e
              N/A       N/A        N       N/A  *
Brick 10.0.231.53:/mnt/raid6-storage/storage
            49154     0          Y       10253
Brick 10.0.231.54:/mnt/raid6-storage/storage
            49153     0          Y       2792
Brick 10.0.231.55:/mnt/raid6-storage/storage
            49153     0          Y       13590
Brick 10.0.231.56:/mnt/raid6-storage/storage
            49152     0          Y       9281
NFS Server on localhost                     2049      0          Y
30775
Quota Daemon on localhost                   N/A       N/A        Y
30781
NFS Server on 10.0.231.54                   2049      0          Y
2817
Quota Daemon on 10.0.231.54                 N/A       N/A        Y
2824
NFS Server on 10.0.231.51                   2049      0          Y
710
Quota Daemon on 10.0.231.51                 N/A       N/A        Y
719
NFS Server on 10.0.231.52                   2049      0          Y
9090
Quota Daemon on 10.0.231.52                 N/A       N/A        Y
9098
NFS Server on 10.0.231.55                   2049      0          Y
13611
Quota Daemon on 10.0.231.55                 N/A       N/A        Y
13619
NFS Server on 10.0.231.56                   2049      0          Y
9303
Quota Daemon on 10.0.231.56                 N/A       N/A        Y
9310
NFS Server on 10.0.231.53                   2049      0          Y
26304
Quota Daemon on 10.0.231.53                 N/A       N/A        Y
26320

Task Status of Volume storage
------------------------------------------------------------------------------
There are no active volume tasks

I see lots of logs related to trashcan (failed [file exists]), set xattrs
(failed [no such file or directory]), quota (invalid arguments) in the
brick logs, which I enabled as a feature after the upgrade this morning.

After restarting glusterd on that host, the volume came back online.

I've attached logs from that host if someone can take a look.

# gluster volume info storage

Volume Name: storage
Type: Distribute
Volume ID: 6f95525a-94d7-4174-bac4-e1a18fe010a2
Status: Started
Number of Bricks: 7
Transport-type: tcp
Bricks:
Brick1: 10.0.231.50:/mnt/raid6-storage/storage
Brick2: 10.0.231.51:/mnt/raid6-storage/storage
Brick3: 10.0.231.52:/mnt/raid6-storage/storage
Brick4: 10.0.231.53:/mnt/raid6-storage/storage
Brick5: 10.0.231.54:/mnt/raid6-storage/storage
Brick6: 10.0.231.55:/mnt/raid6-storage/storage
Brick7: 10.0.231.56:/mnt/raid6-storage/storage
Options Reconfigured:
nfs.disable: no
features.trash-max-filesize: 1GB
features.trash: on
features.quota-deem-statfs: on
features.inode-quota: on
features.quota: on
performance.readdir-ahead: on

# rpm -qa  |grep glusterfs
glusterfs-fuse-3.7.13-1.el7.x86_64
glusterfs-cli-3.7.13-1.el7.x86_64
glusterfs-3.7.13-1.el7.x86_64
glusterfs-server-3.7.13-1.el7.x86_64
glusterfs-api-3.7.13-1.el7.x86_64
glusterfs-libs-3.7.13-1.el7.x86_64
glusterfs-client-xlators-3.7.13-1.el7.x86_64


Thanks,
Steve
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160822/bfb24427/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: etc-glusterfs-glusterd.vol.log
Type: text/x-log
Size: 161664 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160822/bfb24427/attachment-0002.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mnt-raid6-storage-storage.log
Type: text/x-log
Size: 644099 bytes
Desc: not available
URL: <http://www.gluster.org/pipermail/gluster-users/attachments/20160822/bfb24427/attachment-0003.bin>