[Gluster-users] One node goes offline, the other node can't see the replicated volume anymore

Greg Scott GregScott at infrasupport.com
Tue Jul 16 14:31:21 UTC 2013

I'm the first to admit, I don't understand systemd yet.  I wanted to know - how do you know what services you're starting up?  What's the systemd equivalent of chkconfig --list?  Well, maybe I found it - just type "systemctl" with no switches.  So check out the differences between fw1 and fw2.  I have no idea what this means but I think it might be relevant.  

First, systemctl on fw1.  Notice everything is loaded and active except a mystery service named rng.service I don't think we care about.  I'll paste in all 118 lines from fw1.  Now scroll down and look at the extract from fw2.  So nobody is forced to wade through 200+ lines of output, I'll only paste in the relevant service that failed on fw2, named firewall\x2dscripts.mount.  

This same service works on fw1, fails on fw2 with a status of "No such file or directory" - status posted below.  I wonder what it does?  That backslash character in a filename seems fishy.  The only reference I can find anywhere is in a directory, /run/systemd/generator.  I'll paste in what it looks like at the very bottom.

[root at chicago-fw1 rc.d]# systemctl
UNIT                                                LOAD   ACTIVE SUB       DESCRIPTION
proc-sys-fs-binfmt_misc.automount                   loaded active waiting   Arbitrary Executable File Formats File System Automo
sys-devices-pci0...-0000:00:1b.0-sound-card0.device loaded active plugged   NM10/ICH7 Family High Definition Audio Controller
sys-devices-pci0...1-0000:02:00.0-net-enp2s0.device loaded active plugged   RTL8111/8168 PCI Express Gigabit Ethernet controller
sys-devices-pci0...2-0000:03:00.0-net-enp3s0.device loaded active plugged   RTL8111/8168 PCI Express Gigabit Ethernet controller
sys-devices-pci0...0-0000:05:04.0-net-enp5s4.device loaded active plugged   RTL-8110SC/8169SC Gigabit Ethernet
sys-devices-pci0...0-0000:05:06.0-net-enp5s6.device loaded active plugged   RTL-8110SC/8169SC Gigabit Ethernet
sys-devices-pci0...0-0000:05:07.0-net-enp5s7.device loaded active plugged   RTL-8110SC/8169SC Gigabit Ethernet
sys-devices-pci0...:0-2:0:0:0-block-sda-sda1.device loaded active plugged   SanDisk_SDSSDRC032G
sys-devices-pci0...:0-2:0:0:0-block-sda-sda2.device loaded active plugged   SanDisk_SDSSDRC032G
sys-devices-pci0...:0-2:0:0:0-block-sda-sda3.device loaded active plugged   SanDisk_SDSSDRC032G
sys-devices-pci0...et2:0:0-2:0:0:0-block-sda.device loaded active plugged   SanDisk_SDSSDRC032G
sys-devices-platform-serial8250-tty-ttyS2.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS2
sys-devices-platform-serial8250-tty-ttyS3.device    loaded active plugged   /sys/devices/platform/serial8250/tty/ttyS3
sys-devices-pnp0-00:0a-tty-ttyS0.device             loaded active plugged   /sys/devices/pnp0/00:0a/tty/ttyS0
sys-devices-pnp0-00:0b-tty-ttyS1.device             loaded active plugged   /sys/devices/pnp0/00:0b/tty/ttyS1
sys-devices-virtual-block-dm\x2d0.device            loaded active plugged   /sys/devices/virtual/block/dm-0
sys-devices-virtual-block-dm\x2d1.device            loaded active plugged   /sys/devices/virtual/block/dm-1
sys-devices-virtual-block-dm\x2d2.device            loaded active plugged   /sys/devices/virtual/block/dm-2
sys-module-configfs.device                          loaded active plugged   /sys/module/configfs
sys-module-fuse.device                              loaded active plugged   /sys/module/fuse
sys-subsystem-net-devices-enp2s0.device             loaded active plugged   RTL8111/8168 PCI Express Gigabit Ethernet controller
sys-subsystem-net-devices-enp3s0.device             loaded active plugged   RTL8111/8168 PCI Express Gigabit Ethernet controller
sys-subsystem-net-devices-enp5s4.device             loaded active plugged   RTL-8110SC/8169SC Gigabit Ethernet
sys-subsystem-net-devices-enp5s6.device             loaded active plugged   RTL-8110SC/8169SC Gigabit Ethernet
sys-subsystem-net-devices-enp5s7.device             loaded active plugged   RTL-8110SC/8169SC Gigabit Ethernet
-.mount                                             loaded active mounted   /
boot-efi.mount                                      loaded active mounted   /boot/efi
boot.mount                                          loaded active mounted   /boot
dev-hugepages.mount                                 loaded active mounted   Huge Pages File System
dev-mqueue.mount                                    loaded active mounted   POSIX Message Queue File System
firewall\x2dscripts.mount                           loaded active mounted   /firewall-scripts
gluster\x2dfw1.mount                                loaded active mounted   /gluster-fw1
sys-fs-fuse-connections.mount                       loaded active mounted   FUSE Control File System
sys-kernel-config.mount                             loaded active mounted   Configuration File System
sys-kernel-debug.mount                              loaded active mounted   Debug File System
tmp.mount                                           loaded active mounted   Temporary Directory
cups.path                                           loaded active waiting   CUPS Printer Service Spool
systemd-ask-password-plymouth.path                  loaded active waiting   Forward Password Requests to Plymouth Directory Watc
systemd-ask-password-wall.path                      loaded active waiting   Forward Password Requests to Wall Directory Watch
abrt-ccpp.service                                   loaded active exited    Install ABRT coredump hook
abrt-oops.service                                   loaded active running   ABRT kernel log watcher
abrt-xorg.service                                   loaded active running   ABRT Xorg log watcher
abrtd.service                                       loaded active running   ABRT Automated Bug Reporting Tool
alsa-state.service                                  loaded active running   Manage Sound Card State (restore and store)
atd.service                                         loaded active running   Job spooling tools
auditd.service                                      loaded active running   Security Auditing Service
avahi-daemon.service                                loaded active running   Avahi mDNS/DNS-SD Stack
chronyd.service                                     loaded active running   NTP client/server
crond.service                                       loaded active running   Command Scheduler
dbus.service                                        loaded active running   D-Bus System Message Bus
fedora-loadmodules.service                          loaded active exited    Load legacy module configuration
fedora-readonly.service                             loaded active exited    Configure read-only root support
getty at tty1.service                                  loaded active running   Getty on tty1
glusterd.service                                    loaded active running   GlusterFS an clustered file-system server
irqbalance.service                                  loaded active running   irqbalance daemon
lvm2-lvmetad.service                                loaded active running   LVM2 metadata daemon
lvm2-monitor.service                                loaded active exited    Monitoring of LVM2 mirrors, snapshots etc. using dme
mcelog.service                                      loaded active running   Machine Check Exception Logging Daemon
network.service                                     loaded active exited    LSB: Bring up/down networking
rc-local.service                                    loaded active running   /etc/rc.d/rc.local Compatibility
rngd.service                                        loaded failed failed    Hardware RNG Entropy Gatherer Daemon
rpcbind.service                                     loaded active running   RPC bind service
rsyslog.service                                     loaded active running   System Logging Service
smartd.service                                      loaded active running   Self Monitoring and Reporting Technology (SMART) Dae
sshd.service                                        loaded active running   OpenSSH server daemon
systemd-journald.service                            loaded active running   Journal Service
systemd-logind.service                              loaded active running   Login Service
systemd-readahead-collect.service                   loaded active exited    Collect Read-Ahead Data
systemd-readahead-replay.service                    loaded active exited    Replay Read-Ahead Data
systemd-remount-fs.service                          loaded active exited    Remount Root and Kernel File Systems
systemd-sysctl.service                              loaded active exited    Apply Kernel Variables
systemd-tmpfiles-setup.service                      loaded active exited    Recreate Volatile Files and Directories
systemd-udev-trigger.service                        loaded active exited    udev Coldplug all Devices
systemd-udevd.service                               loaded active running   udev Kernel Device Manager
systemd-user-sessions.service                       loaded active exited    Permit User Sessions
systemd-vconsole-setup.service                      loaded active exited    Setup Virtual Console
vsftpd.service                                      loaded active running   Vsftpd ftp daemon
avahi-daemon.socket                                 loaded active listening Avahi mDNS/DNS-SD Stack Activation Socket
cups.socket                                         loaded active listening CUPS Printing Service Sockets
dbus.socket                                         loaded active running   D-Bus System Message Bus Socket
dm-event.socket                                     loaded active listening Device-mapper event daemon FIFOs
iscsid.socket                                       loaded active listening Open-iSCSI iscsid Socket
iscsiuio.socket                                     loaded active listening Open-iSCSI iscsiuio Socket
lvm2-lvmetad.socket                                 loaded active running   LVM2 metadata daemon socket
pcscd.socket                                        loaded active listening PC/SC Smart Card Daemon Activation Socket
rpcbind.socket                                      loaded active listening RPCbind Server Activation Socket
syslog.socket                                       loaded active running   Syslog Socket
systemd-initctl.socket                              loaded active listening /dev/initctl Compatibility Named Pipe
systemd-journald.socket                             loaded active running   Journal Socket
systemd-shutdownd.socket                            loaded active listening Delayed Shutdown Socket
systemd-udevd-control.socket                        loaded active listening udev Control Socket
systemd-udevd-kernel.socket                         loaded active running   udev Kernel Socket
dev-dm\x2d1.swap                                    loaded active active    /dev/dm-1
basic.target                                        loaded active active    Basic System
cryptsetup.target                                   loaded active active    Encrypted Volumes
getty.target                                        loaded active active    Login Prompts
local-fs-pre.target                                 loaded active active    Local File Systems (Pre)
local-fs.target                                     loaded active active    Local File Systems
multi-user.target                                   loaded active active    Multi-User System
network-online.target                               loaded active active    Network is Online
network.target                                      loaded active active    Network
paths.target                                        loaded active active    Paths
remote-fs.target                                    loaded active active    Remote File Systems
sockets.target                                      loaded active active    Sockets
sound.target                                        loaded active active    Sound Card
swap.target                                         loaded active active    Swap
sysinit.target                                      loaded active active    System Initialization
timers.target                                       loaded active active    Timers
systemd-readahead-done.timer                        loaded active elapsed   Stop Read-Ahead Data Collection 10s After Completed
systemd-tmpfiles-clean.timer                        loaded active waiting   Daily Cleanup of Temporary Directories

LOAD   = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB    = The low-level unit activation state, values depend on unit type.

110 loaded units listed. Pass --all to see loaded but inactive units, too.
To show all installed unit files use 'systemctl list-unit-files'.
lines 94-118/118 (END)

To save a headache, now just the extract from fw2

[root at chicago-fw2 ~]# systemctl
UNIT                                          LOAD   ACTIVE SUB       DESCRIPTION
firewall\x2dscripts.mount                     loaded failed failed    /firewall-scripts
[root at chicago-fw2 rc.d]# systemctl status firewall\x2dscripts.mount -n 50
   Loaded: error (Reason: No such file or directory)
   Active: inactive (dead)

And here is what that mystery service file looks like.  Is my dash, "-", some kind of illegal character in fstab??

# Automatically generated by systemd-fstab-generator



More information about the Gluster-users mailing list