[Gluster-users] peer not online again after reboot
Muslim Adel Abu Taha
muslim.adel at gmail.com
Sun Nov 18 21:08:14 UTC 2012
Hello there,
I've 6 servers creating a 50TB storage. My setup is Distribution +
Replication. My servers are named: storage01 - storage06, and they are
running Ubuntu 12.04 64-bit.
Today I decided to install available packages updates on the servers. So I
started with storage06. After the installation of new packages and a new
kernel, I decided to reboot it to use the newly installed kernel. I issued
the reboot and withing minutes, the server was back again. But gluster on
it seems to be offline:
When I run the below command on storage01 (The server where I created all
the volumes) I get the below:
root at storage01:~# gluster peer status
Number of Peers: 5
Hostname: storage05
Uuid: 9274b525-8089-4cc7-afd8-5031b7e2774e
State: Peer in Cluster (Connected)
Hostname: storage02
Uuid: a0d68c85-b34a-4052-9d45-cbdd9b08ad18
State: Peer in Cluster (Connected)
Hostname: storage06
Uuid: 5aadb3d0-d7bc-447f-83a7-368104b4ddc4
State: Peer in Cluster (Connected)
Hostname: storage03
Uuid: 94d0375a-31f2-4bb1-9a5d-647c20eedbcf
State: Peer in Cluster (Connected)
Hostname: storage04
Uuid: 888cb603-ffbf-4f09-a0a4-890170519eb9
State: Peer in Cluster (Connected)
But when I run volume info, I get:
gluster volume status all detail
Status of volume: orig
------------------------------------------------------------------------------
Brick : Brick storage01:/mnt/storage/orig
Port : 24012
Online : Y
Pid : 775
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.5TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864568149
------------------------------------------------------------------------------
Brick : Brick storage02:/mnt/storage/orig
Port : 24012
Online : Y
Pid : 22838
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 8.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864445198
------------------------------------------------------------------------------
Brick : Brick storage03:/mnt/storage/orig
Port : 24012
Online : Y
Pid : 11342
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561821
------------------------------------------------------------------------------
Brick : Brick storage04:/mnt/storage/orig
Port : 24012
Online : Y
Pid : 10887
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864557582
------------------------------------------------------------------------------
Brick : Brick storage05:/mnt/storage/orig
Port : 24012
Online : Y
Pid : 32098
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561308
------------------------------------------------------------------------------
Brick : Brick storage06:/mnt/storage/orig
Port : 24012
Online : N
Pid : N/A
File System : N/A
Device : N/A
Mount Options : N/A
Inode Size : N/A
Disk Space Free : 0Bytes
Total Disk Space : 0Bytes
Inode Count : N/A
Free Inodes : N/A
Status of volume: mp3
------------------------------------------------------------------------------
Brick : Brick storage01:/mnt/storage/mp3
Port : 24009
Online : Y
Pid : 737
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.5TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864568149
------------------------------------------------------------------------------
Brick : Brick storage02:/mnt/storage/mp3
Port : 24009
Online : Y
Pid : 22802
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 8.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864445198
------------------------------------------------------------------------------
Brick : Brick storage03:/mnt/storage/mp3
Port : 24009
Online : Y
Pid : 11306
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561821
------------------------------------------------------------------------------
Brick : Brick storage04:/mnt/storage/mp3
Port : 24009
Online : Y
Pid : 10853
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864557582
------------------------------------------------------------------------------
Brick : Brick storage05:/mnt/storage/mp3
Port : 24009
Online : Y
Pid : 32062
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561308
------------------------------------------------------------------------------
Brick : Brick storage06:/mnt/storage/mp3
Port : 24009
Online : N
Pid : N/A
File System : N/A
Device : N/A
Mount Options : N/A
Inode Size : N/A
Disk Space Free : 0Bytes
Total Disk Space : 0Bytes
Inode Count : N/A
Free Inodes : N/A
Status of volume: flv
------------------------------------------------------------------------------
Brick : Brick storage01:/mnt/storage/flv
Port : 24011
Online : Y
Pid : 435
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.5TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864568149
------------------------------------------------------------------------------
Brick : Brick storage02:/mnt/storage/flv
Port : 24011
Online : Y
Pid : 22544
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 8.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864445198
------------------------------------------------------------------------------
Brick : Brick storage03:/mnt/storage/flv
Port : 24011
Online : Y
Pid : 11051
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561821
------------------------------------------------------------------------------
Brick : Brick storage04:/mnt/storage/flv
Port : 24011
Online : Y
Pid : 10595
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864557582
------------------------------------------------------------------------------
Brick : Brick storage05:/mnt/storage/flv
Port : 24011
Online : Y
Pid : 31798
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561308
------------------------------------------------------------------------------
Brick : Brick storage06:/mnt/storage/flv
Port : 24011
Online : N
Pid : N/A
File System : N/A
Device : N/A
Mount Options : N/A
Inode Size : N/A
Disk Space Free : 0Bytes
Total Disk Space : 0Bytes
Inode Count : N/A
Free Inodes : N/A
Status of volume: mp4
------------------------------------------------------------------------------
Brick : Brick storage01:/mnt/storage/mp4
Port : 24010
Online : Y
Pid : 698
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.5TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864568149
------------------------------------------------------------------------------
Brick : Brick storage02:/mnt/storage/mp4
Port : 24010
Online : Y
Pid : 22756
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 8.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864445198
------------------------------------------------------------------------------
Brick : Brick storage03:/mnt/storage/mp4
Port : 24010
Online : Y
Pid : 11272
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561821
------------------------------------------------------------------------------
Brick : Brick storage04:/mnt/storage/mp4
Port : 24010
Online : Y
Pid : 10818
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.3TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864557582
------------------------------------------------------------------------------
Brick : Brick storage05:/mnt/storage/mp4
Port : 24010
Online : Y
Pid : 32027
File System : xfs
Device : /dev/mapper/storage_vg-videos
Mount Options : rw
Inode Size : 256
Disk Space Free : 11.4TB
Total Disk Space : 18.0TB
Inode Count : 3865470528
Free Inodes : 3864561308
------------------------------------------------------------------------------
Brick : Brick storage06:/mnt/storage/mp4
Port : 24010
Online : N
Pid : N/A
File System : N/A
Device : N/A
Mount Options : N/A
Inode Size : N/A
Disk Space Free : 0Bytes
Total Disk Space : 0Bytes
Inode Count : N/A
Free Inodes : N/A
As you can see, all bricks from storage06 are offiline. I tried to restart
gluster daemon many times, I rebooted, and I even stopped all volumes and
started them again but the same result. My main issue is that this is a
production environment, and the whole load is going to storage05 (load 35
now and all CPUs 100% full) and I can't do a lot of trials on it.
Your advice please?
Thanks & Regards,
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://supercolony.gluster.org/pipermail/gluster-users/attachments/20121118/ad9887e4/attachment.html>
More information about the Gluster-users
mailing list