[Gluster-users] One node goes offline, the other node can't see the replicated volume anymore
Greg Scott
GregScott at infrasupport.com
Tue Jul 16 05:38:33 UTC 2013
Hmmm - I mount my firewall-scripts volume like this now in fstab:
localhost:/firewall-scripts /firewall-scripts glusterfs defaults,_netdev 0 0
But gluster volume info still looks like this on both nodes:
[root at chicago-fw1 system]# gluster volume info
Volume Name: firewall-scripts
Type: Replicate
Volume ID: 239b6401-e873-449d-a2d3-1eb2f65a1d4c
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 192.168.253.1:/gluster-fw1
Brick2: 192.168.253.2:/gluster-fw2
[root at chicago-fw1 system]#
Seems to me, my Gluster volume would still have some dependency on networking, even though I mount it with localhost in fstab, right?
In all our trial and error, the only thing that's worked so far is putting in that 30 second delay in my custom rc.local file. So instead of fighting this, why not try my hand at a systemd service file named, say, glustermount.service.
So here is the glustermout.service file I made. It runs a script named /etc/rc.d/glustermount.sh pasted in below.
[root at chicago-fw1 rc.d]# cd /usr/lib/systemd/system
[root at chicago-fw1 system]# more glustermount.service
# This unit is Greg's attempt to mount my Gluster filesystem.
[Unit]
Description=Set up Gluster mounts
ConditionFileIsExecutable=/etc/rc.d/glustermount.sh
After=network.target glusterd.service
[Service]
Type=oneshot
ExecStart=/etc/rc.d/glustermount.sh
TimeoutSec=0
RemainAfterExit=no
SysVStartPriority=99
[Install]
WantedBy=multi-user.target
[root at chicago-fw1 system]#
Here is glustermount.sh as it exists right now. Really just a renamed rc.local I've been using the whole time, now with some debug echo lines.
[root at chicago-fw1 rc.d]# more glustermount.sh
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
#
# Note removed by default starting in Fedora 16.
touch /var/lock/subsys/local
#***********************************
# Local stuff below
date
echo "Sleeping 30 seconds."
### sleep 30
date
echo "Making sure the Gluster stuff is mounted"
echo "Mounted before mount -av"
df -h
mount -av
echo "Mounted after mount -av"
df -h
# The fstab mounts happen early in startup, then Gluster starts up later.
# By now, Gluster should be up and running and the mounts should work.
# That _netdev option is supposed to account for the delay but doesn't seem
# to work right.
echo "Starting up firewall common items"
## /firewall-scripts/etc/rc.d/common-rc.local
[root at chicago-fw1 rc.d]#
I should see mounts before and after when I look at a systemctl status. But I don't. And systemctl status shows my /firewall-scripts filesystem successfully mounted, but when I go to look with df -h, it's not mounted. Yet another midnight mystery - take a look.
[root at chicago-fw1 system]# systemctl start glustermount.service
[root at chicago-fw1 system]# systemctl status glustermount.service -n 50
glustermount.service - Set up Gluster mounts
Loaded: loaded (/usr/lib/systemd/system/glustermount.service; enabled)
Active: inactive (dead) since Tue 2013-07-16 00:27:36 CDT; 5s ago
Process: 2464 ExecStart=/etc/rc.d/glustermount.sh (code=exited, status=0/SUCCESS)
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Tue Jul 16 00:27:35 CDT 2013
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Sleeping 30 seconds.
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Tue Jul 16 00:27:35 CDT 2013
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Making sure the Gluster stuff is mounted
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Mounted before mount -av
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: Filesystem Size Used Avail Use% Mounted on
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/mapper/fedora-root 14G 3.9G 8.7G 31% /
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: devtmpfs 990M 0 990M 0% /dev
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs 996M 0 996M 0% /dev/shm
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs 996M 888K 996M 1% /run
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs 996M 0 996M 0% /sys/...roup
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: tmpfs 996M 0 996M 0% /tmp
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/sda2 477M 87M 365M 20% /boot
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/sda1 200M 9.4M 191M 5% /boot/efi
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: /dev/mapper/fedora-gluster--fw1 7.9G 33M 7.8G 1% /gluster-fw1
Jul 16 00:27:35 chicago-fw1 glustermount.sh[2464]: extra arguments at end (ignored)
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: / : ignored
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /boot : already mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /boot/efi : already mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /gluster-fw1 : already mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: swap : ignored
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: /firewall-scripts : successfully mounted
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: Mounted after mount -av
Jul 16 00:27:36 chicago-fw1 glustermount.sh[2464]: Filesystem Size Used Avail Use% Mounted on
Jul 16 00:27:36 chicago-fw1 systemd[1]: Started Set up Gluster mounts.
[root at chicago-fw1 system]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 14G 3.9G 8.7G 31% /
devtmpfs 990M 0 990M 0% /dev
tmpfs 996M 0 996M 0% /dev/shm
tmpfs 996M 888K 996M 1% /run
tmpfs 996M 0 996M 0% /sys/fs/cgroup
tmpfs 996M 0 996M 0% /tmp
/dev/sda2 477M 87M 365M 20% /boot
/dev/sda1 200M 9.4M 191M 5% /boot/efi
/dev/mapper/fedora-gluster--fw1 7.9G 33M 7.8G 1% /gluster-fw1
[root at chicago-fw1 system]#
Finally, running the same glustermount.sh script by hand works just fine. It **only** breaks when run from systemd. Here is what happens when I run it by hand. Compare below to what happens above when I tell systemd to run it.
[root at chicago-fw1 system]#
[root at chicago-fw1 system]# /etc/rc.d/glustermount.sh
Tue Jul 16 00:23:15 CDT 2013
Sleeping 30 seconds.
Tue Jul 16 00:23:15 CDT 2013
Making sure the Gluster stuff is mounted
Mounted before mount -av
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 14G 3.9G 8.7G 31% /
devtmpfs 990M 0 990M 0% /dev
tmpfs 996M 0 996M 0% /dev/shm
tmpfs 996M 888K 996M 1% /run
tmpfs 996M 0 996M 0% /sys/fs/cgroup
tmpfs 996M 0 996M 0% /tmp
/dev/sda2 477M 87M 365M 20% /boot
/dev/sda1 200M 9.4M 191M 5% /boot/efi
/dev/mapper/fedora-gluster--fw1 7.9G 33M 7.8G 1% /gluster-fw1
/ : ignored
/boot : already mounted
/boot/efi : already mounted
/gluster-fw1 : already mounted
swap : ignored
extra arguments at end (ignored)
/firewall-scripts : successfully mounted
Mounted after mount -av
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 14G 3.9G 8.7G 31% /
devtmpfs 990M 0 990M 0% /dev
tmpfs 996M 0 996M 0% /dev/shm
tmpfs 996M 888K 996M 1% /run
tmpfs 996M 0 996M 0% /sys/fs/cgroup
tmpfs 996M 0 996M 0% /tmp
/dev/sda2 477M 87M 365M 20% /boot
/dev/sda1 200M 9.4M 191M 5% /boot/efi
/dev/mapper/fedora-gluster--fw1 7.9G 33M 7.8G 1% /gluster-fw1
localhost:/firewall-scripts 7.6G 19M 7.2G 1% /firewall-scripts
Starting up firewall common items
[root at chicago-fw1 system]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/fedora-root 14G 3.9G 8.7G 31% /
devtmpfs 990M 0 990M 0% /dev
tmpfs 996M 0 996M 0% /dev/shm
tmpfs 996M 888K 996M 1% /run
tmpfs 996M 0 996M 0% /sys/fs/cgroup
tmpfs 996M 0 996M 0% /tmp
/dev/sda2 477M 87M 365M 20% /boot
/dev/sda1 200M 9.4M 191M 5% /boot/efi
/dev/mapper/fedora-gluster--fw1 7.9G 33M 7.8G 1% /gluster-fw1
localhost:/firewall-scripts 7.6G 19M 7.2G 1% /firewall-scripts
[root at chicago-fw1 system]#
- Greg
More information about the Gluster-users
mailing list