Linux NFS Root and PXE-Boot

Monday, November 6, 2006 by digitalpeer

Linux kernel hacking and test running on the same machine becomes a major pain. This tutorial explains how to separate the two easily for a quick code-and-test loop. This tutorial explains how to setup a Linux thin client that boots using PXE by pulling a remote Linux kernel image and mounting a remote root file system over NFS. This is only possible if your client machine has a network card that supports this (do you remember seeing some type of option like press N to boot from network just after posting?). I am using Fedora Core 5 as my server, so some of the details may be specific to FC.

Most of the details of setting up the PXE boot server were found at Setting up a PXE-Boot Server.

1) yum install tftp-server dhcp
Make sure you have an NFS server.

2) create /etc/dhcpd.conf

allow bootp; allow booting;  ddns-update-style interim; subnet 192.168.0.0 netmask 255.255.255.0 { range 192.168.0.10 192.168.0.254; default-lease-time 3600; max-lease-time 4800; option routers 192.168.0.1; option domain-name-servers 192.168.0.1; option subnet-mask 255.255.255.0; option domain-name "llama.net"; option time-offset -8; }  host llama0 { hardware ethernet 04:4B:80:80:80:03; fixed-address 192.168.0.254; option host-name "llama0"; filename "pxelinux.0"; }

Change MAC address to match that of the client machine.

3) Create the directory and add the following line to /etc/exports for the nfs share.

/nfsroot *(rw,no_all_squash,no_root_squash)

4) cp -a /bin /dev /etc /home /lib /mnt /sbin /tmp /usr /var /nfsroot
from a good Fedora Core install. You'll have to cleanup and change some of these files for your client at some point.

5) Activate tftp within xinetd by changing disable=yes to disable=no in /etc/xinetd.d/tftp. Restart xinetd. The tftp root is at /tftpboot.

6) Compile your kernel with the following options and move bzImage to /tftpboot/:
- NFS filesystem support (CONFIG_NFS_FS).
- Root file system on NFS (CONFIG_ROOT_NFS).
- Ethernet (10 or 100Mbit) (CONFIG_NET_ETHERNET).
- The ethernet driver for the clent's network card.
- IP: kernel level autoconfiguration (CONFIG_IP_PNP)
- BOOTP support (CONFIG_IP_PNP_BOOTP)

7) Configure pxelinux. First create the directory /tftpboot/pxelinux.cfg (and make it world readable). Inside that directory you need to create a number of zero size files (use touch):

01-04-4B-80-80-80-03
C
C0
C0A
C0A8
C0A80
C0A800
C0A800F
C0A800FE
01-04-4B-80-80-80-03

The first 8 are the hex representation of the 192.168.0.254 IP address that your PXE boot client will be assigned. The permutations allow a broader IP subnet to be searched first for matches. The last entry is the MAC address of your PXE boot client's NIC (with dashes substituted for the colons), with '01' pre-pended. The "01" at the front represents a hardware type of Ethernet, so pxelinux.0 see's the configuration string as an IP address.

Copy /usr/lib/syslinux/pxelinux.0 from your Fedora install to /tftpboot/

8) Now create the default pxelinux configuration inside the new file
/tftpboot/pxelinux.cfg/default:

prompt 1 default linux timeout 100  label linux kernel bzImage append init=/sbin/init root=/dev/nfs rw nfsroot=192.168.0.1:/nfsroot ip=192.168.0.254:192.168.0.1:192.168.0.1:255.255.255.0:llama0::off noapic acpi=off

The last line is where you can stick any kernel parameters needed. You'll notice this menu is somewhat similar to a grub config file.

Check out the following articles for more information:
PXELINUX - SYSLINUX for network boot
NFS-Root-Client Mini-HOWTO
NFS-Root mini-HOWTO
NFS-Root

External Links SummaryPXE Boot server [www.linux-sxs.org]
http://www.ibiblio.org/pub/Linux/docs/HOWTO/other-formats/html_single/NFS-Root-Client-mini-HOWTO.... [www.ibiblio.org]
faqs.org [www.faqs.org]
http://www.tldp.org/HOWTO/text/NFS-Root [www.tldp.org]
PXELINUX - SYSLINUX for network boot [syslinux.zytor.com

Fedora - How To Running ARM Build under QEMU

http://fedoraproject.org/wiki/ARM/HowToQemu

Introduction

Qemu is a well-known emulator that supports ARM platforms, and can be used to run the Fedora-ARM distribution. This provides a convenient platform to try out the distribution as well as to development and customization.

The howto describes a process to get the Fedora-ARM distribution running under QEMU. The host system is assumed to be Fedora, but most of the process should work on any other Linux system as well.

The QEMU system is set up to get its IP address using DHCP and its root file system over NFS from the host system. This requires networking to be setup between the host system and the QEMU guest.

Install and Setup QEMU Networking

Install QEMU

This How To has been tested with QEMU 0.9 on Fedora 7 system. If you are running Fedora 7, you can just install qemu using yum.

$yum install qemu 

Host Networking Setup

The QEMU system is set up to get its IP address using DHCP and its root file system over NFS from the host system. This requires networking to be setup between the host system and the QEMU guest.

The networking setup uses host TAP devices to connect to QEMU. In recent kernels, this requires CAP_NET_ADMIN capability. This How To assumes that you can run commands as root (or using sudo) whenever necessary.

The host system needs to TUN/TAP networking enabled (CONFIG_TUN=m or CONFIG_TUN=y). You can verify this using:

$grep CONFIG_TUN= /boot/config-`uname -r` 

Also make sure that /dev/net/tun exists. If not, you can make it as follows:

$mknod /dev/net/tun c 10 200 

Now, we need to set up a network bridge interface.

$/usr/sbin/brctl addbr br0

$/sbin/ifconfig eth0 0.0.0.0 promisc up

$/usr/sbin/brctl addif br0 eth0

$/sbin/dhclient br0

$/sbin/iptables -F FORWARD

Also, create a script qemu-ifup as follows. This will be needed when we boot into QEMU.

#!/bin/sh

/sbin/ifconfig $1 0.0.0.0 promisc up

/usr/sbin/brctl addif br0 $1

Build a kernel image to run under QEMU

First you will need to have an ARM cross-compiler. If you dont have one, download one from CodeSourcery's web-site, install it, and ensure that is it in your path.

$export arch=ARM

$export CROSS_COMPILE=arm-none-linux-gnueabi-

Download Linux kernel (I have tested it with 2.6.21 and 2.6.22) and build it for ARM Versatile board. But, first you will have to customize the defconfig for it to work correctly.

$cp arch/arm/configs/versatile_defconfig .config

$make menuconfig

Enable DHCP Support (CONFIG_IP_PNP_DHCP). It is under Networking -> Networking Support -> Networking Options -> TCP/IP Networking -> IP: Kernel Level autoconfiguration.

Enable Universal Tun/Tap Driver Support (CONFIG_TUN). It is under Device Drivers -> Network Device Support -> Network Device Support.

Enable ARM EABI Support (CONFIG_AEABI). It is under Kernel Features.

Enable tmpfs support (CONFIG_TMPFS). It is under File Systems -> Pseudo File Systems.

Build the kernel.

make

Root File System Setup

Download and Set up the basic root file system

Download the root file system tarball and untar it. We will now customize it to make it bootable.

Creating /etc/fstab

The /etc/fstab file is used to mount various file systems that are needed for proper functioning of the system. Create a new /etc/fstab file with the following contents. Note that root (/) is set up for NFS.

/dev/root / nfs defaults 1 1

devpts /dev/pts devpts gid=5,mode=620 0 0

tmpfs /dev/shm tmpfs defaults 0 0

proc /proc proc defaults 0 0

sysfs /sys sysfs defaults 0 0

Creating /etc/resolv.conf

The /etc/resolv.conf needs to be configured with the DNS server that you are using. You can create this by simply copying the file from the host system.

Creating /etc/hosts

Create /etc/hosts with the following.

127.0.0.1       localhost 

Creating /etc/sysconfig/network

NETWORKING=yes

HOSTNAME=qemu-arm-guest

Create the basic device nodes

We are not using initrd or initramfs for initial root file system -- so we need to set up a few device nodes before udev kicks in.

mknod /dev/console c 5 1

mknod /dev/null c 1 3

mknod /dev/zero c 1 5

Set the root password

We need to set the root password so that we can log in on boot. This can be done by booting with init=/bin/sh and then using chpasswd to change the password. Here, we just make the change in /etc/passwd.

The /etc/passwd file has an entry for root:

root:*:0:0:root:/root:/bin/bash

Edit the file, and remove the * (which prevents login), so that the entry looks like:

root::0:0:root:/root:/bin/bash

Export the Root File System over NFS

This assumes that the root file system is in /mnt/ARM_FS. We need to export it through NFS. Add the following in your /etc/exports.

/mnt/ARM_FS/ *(rw,sync,no_root_squash) 

Now, restart the NFS service.

#/sbin/service nfs restart 

Boot into QEMU

Now you are ready to boot into QEMU. Replace <host-ip> with the IP address of the host machine.

$qemu-system-arm -M versatilepb -kernel zImage-versatile -append root="/dev/nfs nfsroot=<host-ip>:/mnt/ARM_FS rw ip=dhcp" -net nic,vlan=0 -net tap,vlan=0,ifname=tap0,script=./qemu-ifup

MicroSUSE System Builder's Guide

http://en.opensuse.org/MicroSUSE_System_Builder's_Guide

Building A Minimal i386 System

These are the steps required to build a minimal system for an i386 PC-like target.

1. Become the root user.
2. Create a temporary directory.
3. Unpack the following tarballs in your temporary directory: (Depending on whether you have built the MicroSUSE distribution manually or are using the RPMs, the target system packages can be found in tarballs/i386 or /usr/lib/microsuse/i386/target.)
1. root-i386.tar
2. busybox-i386.tar
3. kernel-qemu-i386.tar
4. Move the kernel image (bzImage-<kernel version>-qemu-i386) out of your temporary directory.
5. Create an ext2 filesystem image by running these commands:
1. dd if=/dev/zero of=ususe.img bs=1M count=2
2. mkfs.ext2 -N 512 ususe.img
6. Mount your filesystem image and copy all files from your temporary directory onto it, then unmount it.

Et voila, you're done. If you have QEMU installed, you can run your system like so:

qemu-system-i386 -hda ususe.img -kernel <path to the kernel image>/bzImage-<kernel version>-qemu-i386 -append root=/dev/hda

[edit]

Building A Minimal ARM System

Building a system for an Integrator/CP board with an ARM 926 CPU is a little bit more tricky than building for a plain old PC, depending on whether you want your console on the UART or the LCD. Here's how it works:

1. Become the root user.
2. Create a temporary directory.
3. Unpack the following tarballs in your temporary directory: (Depending on whether you have built the MicroSUSE distribution manually or are using the RPMs, the target system packages can be found in tarballs/arm or /usr/lib/microsuse/arm/target.)
1. root-arm.tar
2. busybox-arm.tar
3. kernel-qemu-arm.tar
4. Move the kernel image (zImage-<kernel version>-qemu-arm) out of your temporary directory.
5. Now, depending on whether you want your console on the serial line or on the LCD panel, you have to do these steps:

Serial console:

1. Create a device node for the serial console:
   mknod <your directory>/dev/ttyAMA0 c 204 64
2. Make sure Busybox init starts a getty on that console by editing etc/inittab in your temporary directory and replacing the line that says
   #ttyS0::respawn:/sbin/getty -L ttyS0 115200 vt100
   with one that says
   ttyAMA0::respawn:/sbin/getty -L ttyAMA0 115200 vt100
3. To be able to log on as root (which is what you will want to do as the system does not have any other users), you also have to edit etc/securetty, adding a line that says
   ttyAMA0
   at the bottom.

Framebuffer console:

4. Create a device node for the FB console:
   mknod <your directory>/dev/tty1 c 4 1
6. Create an ext2 filesystem image by running these commands:
1. dd if=/dev/zero of=ususe.img bs=1M count=2
2. mkfs.ext2 -N 512 ususe.img
7. Mount your filesystem image and copy all files from your temporary directory onto it. Unmount your filesystem image.

Phew, that's it. Again, if you have QEMU installed, you can test your system like this:

Serial console:

qemu-system-arm -kernel <path to the kernel image>/zImage-<kernel version>-qemu-arm -initrd ususe.img -nographic -append console=ttyAMA0

Framebuffer console:

qemu-system-arm -kernel <path to the kernel image>/zImage-<kernel version>-qemu-arm -initrd ususe.img -append console=tty1

Note that you need a QEMU build newer than version 0.8.0 for this to work!

We have built a "hardfloat" system here. If you want to build a system that does not require an FPA or an FP emulator, you can simply use the arm_nofpu instead of the arm tarballs.

Running Linux for ARM processors under QEMU

Paul Dwerryhouse <paul@dwerryhouse.com.au>
June 28, 2006
http://nepotismia.com/linux/qemu/arm/
Introduction
QEMU is well-known as a free replacement for VMware, allowing users to run a PC within a PC. What isn't so well known about QEMU is that, in addition to emulating x86 architectures, it can emulate AMD64, Sparc, MIPS, PPC and ARM CPUs.
In the case of the ARM architecture, QEMU provides a convenient, if slow, environment in which development can be done for embedded systems.
This article describes the process involved in building a Debian/ARM server running under QEMU. It assumes that Debian is also being used as the host server.
Since QEMU's arm emulator has no ability to emulate either IDE or SCSI disks, it will be necessary to install the server on an NFS exported partition.
Preparing the host server
Firstly, install the required packages on your host server:

host# apt-get qemu vde nfs-kernel-server debootstrap

debootstrap allows the creation of a base Debian or Ubuntu system, and is a very simple method for building chroots and virtual machines quickly.
VDE is a virtual network emulator, much like user mode linux's uml_net, allowing a QEMU-emulated machine to connect to the host server's network through a tap interface.
Make sure that your host kernel has support for the "Universal TUN/TAP device driver"; the kernel configuration option for this is named "TUN". If you're running a modularised kernel, there's a good chance that it will be named 'tun.o'.
Once you have a kernel with tun support, set up a tap interface for the emulated machine to communicate through. In /etc/network/interfaces on your host machine, add the following stanza:

auto tap0
iface tap0 inet static
address 10.1.6.1
netmask 255.255.255.0
network 10.1.6.0
broadcast 10.1.6.255
vde-switch vde-net

Ensure that /dev/net/tun is owned by group vde-net and that you are also a member of that group; add this to your host server's /etc/group file and log out and back in again, if necessary.
Then, bring up the interface:

host# ifup tap0

You'll need a partition with enough spare space on it to hold a basic Debian installation; make sure you have enough room on it to grow in the future. You can probably get away with 300Mb at the minimum. I placed my tree under /nfs/share/arm/.
Now, run debootstrap to download and install the base Debian/arm distribution, and copy a few of the files from your host system into it:

host# mkdir /nfs/share/arm
host# debootstrap --foreign --arch arm etch /nfs/share/arm
host# cd /nfs/share/arm
host# cp /etc/passwd etc/passwd
host# cp /etc/shadow etc/shadow
host# cp /etc/group etc/group
host# echo "proc /proc proc defaults 0 0" > etc/fstab
host# echo "10.1.6.1:/nfs/share/arm / nfs defaults 0
1" >> etc/fstab

debootstrap's --foreign option tells it to only perform the first stage of the bootstrap process, downloading and unpacking the packages. Since we're downloading Debian for a different architecture from that on which we're running on the host server, our host system would not be able to use this new system in a chroot to run the pre- and post-installation scripts.

We're going to use the same server that is being used as the host server for running QEMU as the NFS server for the guest server's root filesystem; an emulated system is slow enough as it is, we don't need to make things worse by having a real ethernet network in between the guest OS and its filesystem (as opposed to a virtual ethernet, in this case).
Share the new directory out via NFS, by adding the following entry to /etc/exports:

/nfs/share/arm 10.1.6.0/24(rw,no_root_squash,sync)

Then export the filesystem:

host# exportfs -a

Download a kernel for arm-linux; there's one provided in the arm-test files on the qemu website (and a copy here.)
Place the arm-linux zimage in /usr/local/etc/images/zImage.arm, and then create the following script as /usr/local/bin/start-qemu-arm:

#!/bin/sh
console="ttyAMA0" # serial console
nfsserver="10.1.6.1" # address of NFS server
nfsdir="/nfs/share/arm" # exported share where debian/arm is installed
address="10.1.6.50" # address for guest server
gateway="10.1.6.1" # default gateway
netmask="255.255.255.0" # subnet mask
hostname="arm.home" # hostname for guest server
device="eth0" # interface that guest server will use
mem=256 # memory for guest server in Mb
tap="/var/run/vde/tap0.ctl" # vde tap socket
kernel="/usr/local/etc/images/zImage.arm" # arm kernel
nfsopts="rsize=8192,wsize=8192,hard,intr,tcp,nolock" # nfs options
consoleopt="console=$console"
nfsrootopt="nfsroot=$nfsserver:$nfsdir,$nfsopts"
ipopt="ip=$address::$gateway:$netmask:$hostname:$device"

init=""

if [ "x$1" == "xsingle" ]
then
init="init=/bin/bash"
fi
vdeq qemu-system-arm -sock $tap -m $mem \
-kernel $kernel -nographic \
-append "$consoleopt root=/dev/nfs $nfsrootopt
$ipopt $init"

Adjust the parameters at the top of the script to suit your requirements.

Configuring the guest server
Now, start the new arm server in single user mode - from your user account, since it does not need to be run as root:

host$ /usr/local/bin/start-qemu-arm single

The emulated guest server should start up and boot into a bash prompt. The filesystem will be mounted read-only, and it will be necessary to remount it read-write before any further work can be done on it:

guest# mount -n -o remount,rw /
guest# mount /proc

Now run the second stage of debootstrap, within the guest system, to finalise the installation:

guest# cd /
guest# ./debootstrap/debootstrap --second-stage

This will probably take a while to run; the emulator isn't particularly fast. When it's finished, adjust a few remaining files, such as /etc/hostname and /etc/resolv.conf; also install ssh:

guest# apt-get install ssh

Once this is done, you can shut the server down. It's safe to just kill the qemu process from the host machine - since its filesystem is mounted from an NFS server, there's no need to shut it down cleanly.

Now you can start it up completely:

host$ /usr/local/bin/start-qemu-arm

When it finishes booting, you should be able to ssh into it on 10.1.6.50, and you'll have a working Debian installation running on an emulated ARM processor.