[Gluster-users] peer not online again after reboot

Muslim Adel Abu Taha muslim.adel at gmail.com
Sun Nov 18 21:08:14 UTC 2012


Hello there,

I've 6 servers creating a 50TB storage. My setup is Distribution +
Replication. My servers are named: storage01 - storage06, and they are
running Ubuntu 12.04 64-bit.
Today I decided to install available packages updates on the servers. So I
started with storage06. After the installation of new packages and a new
kernel, I decided to reboot it to use the newly installed kernel. I issued
the reboot and withing minutes, the server was back again. But gluster on
it seems to be offline:

When I run the below command on storage01 (The server where I created all
the volumes) I get the below:

root at storage01:~# gluster peer status
Number of Peers: 5

Hostname: storage05
Uuid: 9274b525-8089-4cc7-afd8-5031b7e2774e
State: Peer in Cluster (Connected)

Hostname: storage02
Uuid: a0d68c85-b34a-4052-9d45-cbdd9b08ad18
State: Peer in Cluster (Connected)

Hostname: storage06
Uuid: 5aadb3d0-d7bc-447f-83a7-368104b4ddc4
State: Peer in Cluster (Connected)

Hostname: storage03
Uuid: 94d0375a-31f2-4bb1-9a5d-647c20eedbcf
State: Peer in Cluster (Connected)

Hostname: storage04
Uuid: 888cb603-ffbf-4f09-a0a4-890170519eb9
State: Peer in Cluster (Connected)


But when I run volume info, I get:

gluster volume status all detail
Status of volume: orig
------------------------------------------------------------------------------
Brick                : Brick storage01:/mnt/storage/orig
Port                 : 24012
Online               : Y
Pid                  : 775
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.5TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864568149
------------------------------------------------------------------------------
Brick                : Brick storage02:/mnt/storage/orig
Port                 : 24012
Online               : Y
Pid                  : 22838
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 8.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864445198
------------------------------------------------------------------------------
Brick                : Brick storage03:/mnt/storage/orig
Port                 : 24012
Online               : Y
Pid                  : 11342
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561821
------------------------------------------------------------------------------
Brick                : Brick storage04:/mnt/storage/orig
Port                 : 24012
Online               : Y
Pid                  : 10887
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864557582
------------------------------------------------------------------------------
Brick                : Brick storage05:/mnt/storage/orig
Port                 : 24012
Online               : Y
Pid                  : 32098
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561308
------------------------------------------------------------------------------
Brick                : Brick storage06:/mnt/storage/orig
Port                 : 24012
Online               : N
Pid                  : N/A
File System          : N/A
Device               : N/A
Mount Options        : N/A
Inode Size           : N/A
Disk Space Free      : 0Bytes
Total Disk Space     : 0Bytes
Inode Count          : N/A
Free Inodes          : N/A

Status of volume: mp3
------------------------------------------------------------------------------
Brick                : Brick storage01:/mnt/storage/mp3
Port                 : 24009
Online               : Y
Pid                  : 737
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.5TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864568149
------------------------------------------------------------------------------
Brick                : Brick storage02:/mnt/storage/mp3
Port                 : 24009
Online               : Y
Pid                  : 22802
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 8.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864445198
------------------------------------------------------------------------------
Brick                : Brick storage03:/mnt/storage/mp3
Port                 : 24009
Online               : Y
Pid                  : 11306
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561821
------------------------------------------------------------------------------
Brick                : Brick storage04:/mnt/storage/mp3
Port                 : 24009
Online               : Y
Pid                  : 10853
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864557582
------------------------------------------------------------------------------
Brick                : Brick storage05:/mnt/storage/mp3
Port                 : 24009
Online               : Y
Pid                  : 32062
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561308
------------------------------------------------------------------------------
Brick                : Brick storage06:/mnt/storage/mp3
Port                 : 24009
Online               : N
Pid                  : N/A
File System          : N/A
Device               : N/A
Mount Options        : N/A
Inode Size           : N/A
Disk Space Free      : 0Bytes
Total Disk Space     : 0Bytes
Inode Count          : N/A
Free Inodes          : N/A

Status of volume: flv
------------------------------------------------------------------------------
Brick                : Brick storage01:/mnt/storage/flv
Port                 : 24011
Online               : Y
Pid                  : 435
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.5TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864568149
------------------------------------------------------------------------------
Brick                : Brick storage02:/mnt/storage/flv
Port                 : 24011
Online               : Y
Pid                  : 22544
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 8.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864445198
------------------------------------------------------------------------------
Brick                : Brick storage03:/mnt/storage/flv
Port                 : 24011
Online               : Y
Pid                  : 11051
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561821
------------------------------------------------------------------------------
Brick                : Brick storage04:/mnt/storage/flv
Port                 : 24011
Online               : Y
Pid                  : 10595
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864557582
------------------------------------------------------------------------------
Brick                : Brick storage05:/mnt/storage/flv
Port                 : 24011
Online               : Y
Pid                  : 31798
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561308
------------------------------------------------------------------------------
Brick                : Brick storage06:/mnt/storage/flv
Port                 : 24011
Online               : N
Pid                  : N/A
File System          : N/A
Device               : N/A
Mount Options        : N/A
Inode Size           : N/A
Disk Space Free      : 0Bytes
Total Disk Space     : 0Bytes
Inode Count          : N/A
Free Inodes          : N/A

Status of volume: mp4
------------------------------------------------------------------------------
Brick                : Brick storage01:/mnt/storage/mp4
Port                 : 24010
Online               : Y
Pid                  : 698
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.5TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864568149
------------------------------------------------------------------------------
Brick                : Brick storage02:/mnt/storage/mp4
Port                 : 24010
Online               : Y
Pid                  : 22756
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 8.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864445198
------------------------------------------------------------------------------
Brick                : Brick storage03:/mnt/storage/mp4
Port                 : 24010
Online               : Y
Pid                  : 11272
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561821
------------------------------------------------------------------------------
Brick                : Brick storage04:/mnt/storage/mp4
Port                 : 24010
Online               : Y
Pid                  : 10818
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.3TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864557582
------------------------------------------------------------------------------
Brick                : Brick storage05:/mnt/storage/mp4
Port                 : 24010
Online               : Y
Pid                  : 32027
File System          : xfs
Device               : /dev/mapper/storage_vg-videos
Mount Options        : rw
Inode Size           : 256
Disk Space Free      : 11.4TB
Total Disk Space     : 18.0TB
Inode Count          : 3865470528
Free Inodes          : 3864561308
------------------------------------------------------------------------------
Brick                : Brick storage06:/mnt/storage/mp4
Port                 : 24010
Online               : N
Pid                  : N/A
File System          : N/A
Device               : N/A
Mount Options        : N/A
Inode Size           : N/A
Disk Space Free      : 0Bytes
Total Disk Space     : 0Bytes
Inode Count          : N/A
Free Inodes          : N/A


As you can see, all bricks from storage06 are offiline. I tried to restart
gluster daemon many times, I rebooted, and I even stopped all volumes and
started them again but the same result. My main issue is that this is a
production environment, and the whole load is going to storage05 (load 35
now and all CPUs 100% full) and I can't do a lot of trials on it.

Your advice please?

Thanks & Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121118/ad9887e4/attachment.html>


More information about the Gluster-users mailing list