[Gluster-users] Netboot / PXE-Boot from glusterfs? WORKS!

Jan kirchy at gmx.de
Mon Jun 21 15:09:43 UTC 2010

It took me a few hours (mainly because it has been some years that I 
played around with netboot and such), but I got it:

It needs some cleanup and still gives some errors during the boot 
process, but I can boot a clean, fresh debian lenny via PXE directly 
from a glusterfs Storage as a root-filesystem.

Of course, you need a dhcpd and tftpd running somewhere so the 
netboot-server can get its ip, kernel and initrd.

this is a very short explanation, as I said it still needs some clean-up 
and I will hopefully end up with a simple script or debian-package that 
doeas all the work for you:

you need a debian installation to boot:
debootstrap lenny $INSTALLDIR http://ftp.de.debian.org/debian
echo "deb http://www.backports.org/debian lenny-backports contrib main 
non-free" >>/etc/apt/sources.list
apt-get update
apt-get install glusterfs-client debian-backports-keyring vim rsync 
linux-image-amd64 dnsutils

Add a hook to have initramfs copy all necessary stuff to the initrd:
                    echo "$PREREQ"

               case $1 in
                    exit 0

. /usr/share/initramfs-tools/hook-functions
# Begin real processing below this line

copy_exec /sbin/mount.glusterfs
copy_exec /usr/sbin/glusterfs
copy_exec /sbin/ifconfig
copy_exec /usr/bin/host

test -d  ${DESTDIR}/usr/lib/glusterfs/ || mkdir -p 
cp -ar /usr/lib/gluster* ${DESTDIR}/usr/lib/

cp /lib/libnss_* /lib/libresolv* /lib/ld-linux-* /lib/libc.* ${DESTDIR}/lib

Now, add a short script to the initrd that will be called automatically 
when the boot=glusterfs kernel parameter is set (via pxe)
echo testserv1 >>/etc/hosts
echo linux-netboot >>/etc/hosts
echo >>/etc/resolv.conf
hostname linux-netboot
mkdir -p /var/log/glusterfs
sh /sbin/mount.glusterfs  ${NFSROOT} ${rootmnt}


run (in the chroot):
update-initramfs -u
to create a new initrd.

PXE-config on the server:

prompt 1
default lenny
timeout 10
label lenny
kernel vmlinuz
append vga=794 initrd=initrd boot=glusterfs nfsroot= ip=dhcp

(the kernel and initrd are symlinks to the glusterfs share that holds 
the root-filesystem:)
lrwxrwxrwx 1 root root 24 18. Jun 20:29 /var/lib/tftpboot/initrd -> 
lrwxrwxrwx 1 root root 21 18. Jun 20:29 /var/lib/tftpboot/vmlinuz -> 

Setting it up this way makes it even possible to do even kernel-updates 
via apt-get on the netboot-client without having to touch anything on 
the dhcp/pxe-server.

more to come... I need to finish some other important work first, but I 
will hopefully be able to clean everything up by the end of next week 
and have a howto and some script or package ready for others that are 


Am 18.06.2010 09:57, schrieb Jan:
> Hello Benjamin,
> I think there should be more elegant ways to build the initrd.
> I set up diskless servers that got their root-directory via nfs a few 
> years ago.
> They shared the same installation (rootdir) and we had scripts in the 
> initrd to mount special directories under /var based on the hostname.
> It was a little messy doing kernel-updates and updating everything.
> Debian builds its initrd automatically after the installation of a new 
> kernel, based on your hardware and configuration. The tool used for 
> that (initramfs) has a lot of hooks to add your own modules or scripts.
> I am sure it is possible to build a simple debian-package that adds 
> those scripts to your installation. Afterwards, you should even be 
> able to make a kernel-upgrade on the server although it is a net install.
> Setting up a new installation should be as simple as starting 
> debootstrap (sets up a blank debian-installation (any architecture) in 
> a subfolder, can also install custom packages listed on the command 
> line), and then configure the tftp-server.
> On the diskless server, you should have all possibilities (including 
> installation of packages, kernel-updates) that you have on any other 
> server that has its own disk.
> I hope I'll have the time to have a closer look at this next week.
> Jan
> Am 17.06.2010 17:16, schrieb Benjamin Hudgens:
>> Here were the notes for us in Debian.  Since these notes are mildly out
>> of context hopefully they simply give you a starting point.  Rsync is
>> the tool we use to slurp the debian image.  You manage gluster as you
>> would on a normal install.  However, you manage the configs and such on
>> a copy at (X) location which is then slurped into ram on a boot.  For
>> our dumb storage boxes we dynamically detect drives that need
>> filesystems, format, then add to gluster configs.  Thus our server
>> configs get generated on-the-fly.  This allows us to maintain a single
>> master instance of the OS image and deploy storage boxes quickly.
>> ##############################
>> Summary:
>> We want to run a slim install of Debian in ram.  Controlling how we
>> mount Gluster is simple.  You manage this as you would a regular
>> install.  Reboot any machine and the changes take effect.  You can blow
>> away a machine at any time since the OS on our storage machines are
>> irrelevant and pull from our master copy.
>> Below remember that ${rootmnt} is pooled from kernel args for the PXE
>> boot.
>> Unpackage the initrd:
>> gzip -dc<  ../initrd.img-2.6.30-2-686.netboot | cpio -i
>> Edit ./init
>> Modifications start at line 205:
>> ------------
>> ## Make a ramdisk
>> mkdir /ram
>> mount -t tmpfs -o size=2G tmpfs /ram
>> ##Move the libraries over since rsync was failing initially.  This just
>> grabs all the libraries from root
>> mount ${rootmnt}/lib /lib
>> mount ${rootmnt}/usr/lib /usr/lib
>> ##Copy the filesystem into ram
>> ${rootmnt}/usr/bin/rsync -rav ${rootmnt}/ /ram/
>> ##Change the rootmnt point to our new location - the rest of the script
>> will take care of booting us up
>> rootmnt=/ram
>> ------
>> Now re-create the initrd image:
>> find . -print0 | cpio -0 -H newc -ov | gzip -c>
>> ../initrd.img-2.6.30-2-686.netboot
>> ################################
>> -----Original Message-----
>> From: gluster-users-bounces at gluster.org
>> [mailto:gluster-users-bounces at gluster.org] On Behalf Of Jan
>> Sent: Thursday, June 17, 2010 6:30 AM
>> To: gluster-users at gluster.org
>> Subject: Re: [Gluster-users] Netboot / PXE-Boot from glusterfs?
>> Am 17.06.2010 13:15, schrieb Benjamin Hudgens:
>>> Hello Jan,
>>> Our company took the approach of slurping our OS into a ram drive and
>>> then mounting file system points from Gluster.  The OS becomes
>>> expendable.  In our case (large amounts of dumb storage machines) this
>>> is okay.  We were itching to get away from NFS.  Boot time is slow
>> while
>>> it reads directly from network ->   ram.  However, the final result is
>> an
>>> OS that is extremely fast and no NFS dependency.
>>> Obviously this approach is only applicable in certain situations.
>> Hello Benjamin,
>> Thanks, that's an interesting idea - as long as the root filesystem is
>> not too big.
>> But even then, one might be able to split it up in the most important
>> files and directories that are needed to boot and mount the
>> glusterfs-directories.
>> This needs some time to figure out what are the minimum files needed to
>> boot until the glusterfs-directories are available - and a minor change
>> in the initrd (mount NFS, create ramdisk, copy files). But definitely
>> worth a try.
>> thanks
>> Jan
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-users at gluster.org
>> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
> _______________________________________________
> Gluster-users mailing list
> Gluster-users at gluster.org
> http://gluster.org/cgi-bin/mailman/listinfo/gluster-users

More information about the Gluster-users mailing list