[Gluster-users] One node goes offline, the other node can't see the replicated volume anymore

[Joe Julian]

Ben may be on to something. If you're using NetworkManager, there may be 
a chicken and egg problem due to the lack of a hardware link being 
established (no switch). What if you mount from localhost?

On 07/15/2013 02:32 PM, Ben Turner wrote:
> Hi Greg.  I don't know if this is the thread I replied to before but it still sound to me like your NICs aren't fully up when the gluster mount is getting mounted.  The _netdev(at least the version in RHEL 6, I haven't looked at others) doesn't check if the NIC is fully up, it only looks to see if the NW manager lock file exists.  When I saw this happen in my tests the lockfile existed but the NIC was still initializing and unable to send/receive traffic to mount the FS.  I was able to put a sleep in the initscript to work around this:
>
> # diff -pruN /etc/rc.d/init.d/netfs /tmp/initrd/netfs
> --- /etc/rc.d/init.d/netfs        2013-04-26 14:32:28.759283055 -0400
> +++ /tmp/initrd/netfs        2013-04-26 14:31:38.320059175 -0400
> @@ -32,8 +32,6 @@ NETDEVMTAB=$(LC_ALL=C awk '$4 ~ /_netdev
>   # See how we were called.
>   case "$1" in
>     start)
> -        echo "Sleeping 30 seconds for NW init workaround -benT"
> -        sleep 30
>           [ ! -f /var/lock/subsys/network ] && ! nm-online -x >/dev/null 2>&1 && exit 0
>           [ "$EUID" != "0" ] && exit 4
>           [ -n "$NFSFSTAB" ] &&
>
> I just used the sleep for testing, the preferred way of dealing with this is probably using the LINKDELAY option in your /etc/sysconfig/network-scripts/ifcfg-* script.  This variable will cause the network scripts to delay $LINKDELAY number of seconds.  Can you try testing with either of those options to see if you are able to mount at boot?
>
> -b
>
> ----- Original Message -----
>> From: "Greg Scott" <GregScott at infrasupport.com>
>> To: "Joe Julian" <joe at julianfamily.org>
>> Cc: gluster-users at gluster.org
>> Sent: Monday, July 15, 2013 4:44:10 PM
>> Subject: Re: [Gluster-users] One node goes offline, the other node can't see the replicated volume anymore
>>
>> This time after rebooting both nodes, neither one shows /firewall-scripts
>> mounted after a login.   But mount -av by hand is successful on both nodes.
>> Fw1 and fw2 both behave identically.  Here is what fw1 looks like.  Fw2 is
>> identical.   This aspect of the problem is screaming timing glitch.
>>
>> login as: root
>> root at 10.10.10.71's password:
>> Last login: Mon Jul 15 15:19:41 2013 from tinahp100b.infrasupport.local
>> [root at chicago-fw1 ~]# df -h
>> Filesystem                       Size  Used Avail Use% Mounted on
>> /dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
>> devtmpfs                         990M     0  990M   0% /dev
>> tmpfs                            996M     0  996M   0% /dev/shm
>> tmpfs                            996M  888K  996M   1% /run
>> tmpfs                            996M     0  996M   0% /sys/fs/cgroup
>> tmpfs                            996M     0  996M   0% /tmp
>> /dev/sda2                        477M   87M  365M  20% /boot
>> /dev/sda1                        200M  9.4M  191M   5% /boot/efi
>> /dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
>> [root at chicago-fw1 ~]# mount -av
>> /                        : ignored
>> /boot                    : already mounted
>> /boot/efi                : already mounted
>> /gluster-fw1             : already mounted
>> swap                     : ignored
>> extra arguments at end (ignored)
>> /firewall-scripts        : successfully mounted
>> [root at chicago-fw1 ~]# df -h
>> Filesystem                       Size  Used Avail Use% Mounted on
>> /dev/mapper/fedora-root           14G  3.9G  8.7G  31% /
>> devtmpfs                         990M     0  990M   0% /dev
>> tmpfs                            996M     0  996M   0% /dev/shm
>> tmpfs                            996M  888K  996M   1% /run
>> tmpfs                            996M     0  996M   0% /sys/fs/cgroup
>> tmpfs                            996M     0  996M   0% /tmp
>> /dev/sda2                        477M   87M  365M  20% /boot
>> /dev/sda1                        200M  9.4M  191M   5% /boot/efi
>> /dev/mapper/fedora-gluster--fw1  7.9G   33M  7.8G   1% /gluster-fw1
>> 192.168.253.1:/firewall-scripts  7.6G   19M  7.2G   1% /firewall-scripts
>> [root at chicago-fw1 ~]#
>>
>> - Greg
>>
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://supercolony.gluster.org/mailman/listinfo/gluster-users
>>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://supercolony.gluster.org/mailman/listinfo/gluster-users