[Gluster-users] Netboot / PXE-Boot from glusterfs? WORKS!

[Jan]

It took me a few hours (mainly because it has been some years that I 
played around with netboot and such), but I got it:

It needs some cleanup and still gives some errors during the boot 
process, but I can boot a clean, fresh debian lenny via PXE directly 
from a glusterfs Storage as a root-filesystem.

Of course, you need a dhcpd and tftpd running somewhere so the 
netboot-server can get its ip, kernel and initrd.


this is a very short explanation, as I said it still needs some clean-up 
and I will hopefully end up with a simple script or debian-package that 
doeas all the work for you:

you need a debian installation to boot:
debootstrap lenny $INSTALLDIR http://ftp.de.debian.org/debian
chroot $INSTALLDIR
echo "deb http://www.backports.org/debian lenny-backports contrib main 
non-free" >>/etc/apt/sources.list
apt-get update
apt-get install glusterfs-client debian-backports-keyring vim rsync 
linux-image-amd64 dnsutils

Add a hook to have initramfs copy all necessary stuff to the initrd:
/etc/initramfs-tools/hooks/glusterfs
---cut---
#!/bin/sh
               PREREQ=""
               prereqs()
               {
                    echo "$PREREQ"
               }

               case $1 in
               prereqs)
                    prereqs
                    exit 0
                    ;;
               esac

. /usr/share/initramfs-tools/hook-functions
# Begin real processing below this line

copy_exec /sbin/mount.glusterfs
copy_exec /usr/sbin/glusterfs
copy_exec /sbin/ifconfig
copy_exec /usr/bin/host

test -d  ${DESTDIR}/usr/lib/glusterfs/ || mkdir -p 
${DESTDIR}/usr/lib/glusterfs/
cp -ar /usr/lib/gluster* ${DESTDIR}/usr/lib/

cp /lib/libnss_* /lib/libresolv* /lib/ld-linux-* /lib/libc.* ${DESTDIR}/lib
---cut---

Now, add a short script to the initrd that will be called automatically 
when the boot=glusterfs kernel parameter is set (via pxe)
/etc/initramfs-tools/scripts/glusterfs
---cut---
mountroot()
{
echo 192.168.200.206 testserv1 >>/etc/hosts
echo 192.168.200.207 linux-netboot >>/etc/hosts
echo 192.168.200.206 >>/etc/resolv.conf
hostname linux-netboot
configure_networking
mkdir -p /var/log/glusterfs
sh /sbin/mount.glusterfs  ${NFSROOT} ${rootmnt}
}

---cut---

run (in the chroot):
update-initramfs -u
to create a new initrd.


PXE-config on the server:

prompt 1
default lenny
timeout 10
label lenny
kernel vmlinuz
append vga=794 initrd=initrd boot=glusterfs nfsroot=192.168.200.206 ip=dhcp

(the kernel and initrd are symlinks to the glusterfs share that holds 
the root-filesystem:)
lrwxrwxrwx 1 root root 24 18. Jun 20:29 /var/lib/tftpboot/initrd -> 
/data/netboot/initrd.img
lrwxrwxrwx 1 root root 21 18. Jun 20:29 /var/lib/tftpboot/vmlinuz -> 
/data/netboot/vmlinuz

Setting it up this way makes it even possible to do even kernel-updates 
via apt-get on the netboot-client without having to touch anything on 
the dhcp/pxe-server.

more to come... I need to finish some other important work first, but I 
will hopefully be able to clean everything up by the end of next week 
and have a howto and some script or package ready for others that are 
interested.

Jan


Am 18.06.2010 09:57, schrieb Jan:
> Hello Benjamin,
>
> I think there should be more elegant ways to build the initrd.
>
> I set up diskless servers that got their root-directory via nfs a few 
> years ago.
> They shared the same installation (rootdir) and we had scripts in the 
> initrd to mount special directories under /var based on the hostname.
> It was a little messy doing kernel-updates and updating everything.
>
> Debian builds its initrd automatically after the installation of a new 
> kernel, based on your hardware and configuration. The tool used for 
> that (initramfs) has a lot of hooks to add your own modules or scripts.
> I am sure it is possible to build a simple debian-package that adds 
> those scripts to your installation. Afterwards, you should even be 
> able to make a kernel-upgrade on the server although it is a net install.
> Setting up a new installation should be as simple as starting 
> debootstrap (sets up a blank debian-installation (any architecture) in 
> a subfolder, can also install custom packages listed on the command 
> line), and then configure the tftp-server.
> On the diskless server, you should have all possibilities (including 
> installation of packages, kernel-updates) that you have on any other 
> server that has its own disk.
>
> I hope I'll have the time to have a closer look at this next week.
>
> Jan
>
>
> Am 17.06.2010 17:16, schrieb Benjamin Hudgens:
>> Here were the notes for us in Debian.  Since these notes are mildly out
>> of context hopefully they simply give you a starting point.  Rsync is
>> the tool we use to slurp the debian image.  You manage gluster as you
>> would on a normal install.  However, you manage the configs and such on
>> a copy at (X) location which is then slurped into ram on a boot.  For
>> our dumb storage boxes we dynamically detect drives that need
>> filesystems, format, then add to gluster configs.  Thus our server
>> configs get generated on-the-fly.  This allows us to maintain a single
>> master instance of the OS image and deploy storage boxes quickly.
>>
>> ##############################
>>
>> Summary:
>>
>> We want to run a slim install of Debian in ram.  Controlling how we
>> mount Gluster is simple.  You manage this as you would a regular
>> install.  Reboot any machine and the changes take effect.  You can blow
>> away a machine at any time since the OS on our storage machines are
>> irrelevant and pull from our master copy.
>>
>> Below remember that ${rootmnt} is pooled from kernel args for the PXE
>> boot.
>>
>> Unpackage the initrd:
>> gzip -dc<  ../initrd.img-2.6.30-2-686.netboot | cpio -i
>>
>> Edit ./init
>> Modifications start at line 205:
>>
>> ------------
>> ## Make a ramdisk
>> mkdir /ram
>> mount -t tmpfs -o size=2G tmpfs /ram
>>
>> ##Move the libraries over since rsync was failing initially.  This just
>> grabs all the libraries from root
>> mount ${rootmnt}/lib /lib
>> mount ${rootmnt}/usr/lib /usr/lib
>>
>> ##Copy the filesystem into ram
>> ${rootmnt}/usr/bin/rsync -rav ${rootmnt}/ /ram/
>>
>> ##Change the rootmnt point to our new location - the rest of the script
>> will take care of booting us up
>> rootmnt=/ram
>> ------
>>
>> Now re-create the initrd image:
>>
>> find . -print0 | cpio -0 -H newc -ov | gzip -c>
>> ../initrd.img-2.6.30-2-686.netboot
>>
>> ################################
>>
>>
>>
>>
>> -----Original Message-----
>> From: gluster-users-bounces at gluster.org
>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Jan
>> Sent: Thursday, June 17, 2010 6:30 AM
>> To: gluster-users at gluster.org
>> Subject: Re: [Gluster-users] Netboot / PXE-Boot from glusterfs?
>>
>> Am 17.06.2010 13:15, schrieb Benjamin Hudgens:
>>> Hello Jan,
>>>
>>> Our company took the approach of slurping our OS into a ram drive and
>>> then mounting file system points from Gluster.  The OS becomes
>>> expendable.  In our case (large amounts of dumb storage machines) this
>>> is okay.  We were itching to get away from NFS.  Boot time is slow
>> while
>>> it reads directly from network ->   ram.  However, the final result is
>> an
>>> OS that is extremely fast and no NFS dependency.
>>>
>>> Obviously this approach is only applicable in certain situations.
>>>
>>>
>> Hello Benjamin,
>>
>> Thanks, that's an interesting idea - as long as the root filesystem is
>> not too big.
>> But even then, one might be able to split it up in the most important
>> files and directories that are needed to boot and mount the
>> glusterfs-directories.
>> This needs some time to figure out what are the minimum files needed to
>> boot until the glusterfs-directories are available - and a minor change
>> in the initrd (mount NFS, create ramdisk, copy files). But definitely
>> worth a try.
>>
>> thanks
>> Jan
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users