Net boot (PXE + iSCSI) with a RaspberryPi 3
By darknao on Wednesday 14 March 2018, 13:32 - linux - Permalink
I'll show you here how to boot and run your Raspberry Pi 3 without SD card.
I'm using a Synology NAS to host an iSCSI LUN which will be used as the root filesystem for the pi.
In order to mount this LUN, the pi will need a starting kernel & initrd image to get the network up, connect to the iSCSI target (the NAS) and mount the root filesystem.
This kernel & module will be provided by a PXE server (also running on that NAS).
Before going further, I'll assume you already have a PXE server up and running. In this tutorial, I use ISC-DHCPd server and tftp-hpa.
You'll also need a running install of Raspbian on your pi to do all the preparation.
Enable PXE Boot capabilities
Check if the usb_boot_mode is set in your OTP memory :
$ vcgencmd otp_dump | grep 17: 17:1020000a
If the usb_boot_mode bit is set, the value should be 3020000a
If this is not the case, install the last firmware update (important, this will not work with the current stable firmware):
$ sudo rpi-update
Add this line to your config.txt to set the usb_boot_mode bit in the OTP memory:
program_usb_boot_mode=1
Then reboot
Now, if you check the OTP again, you should get the wanted value
$ vcgencmd otp_dump | grep 17: 17:3020000a
If everything is ok, remove the "program_usb_boot_mode" line from your config.txt
You can now check that the PXE boot mode is working by power cycling your pi, without SDcard and with a ethernet cable plugged in.
The ethernet leds should light up after a few seconds.
Put the SDcard back, and reboot.
LUN creation and configuration
In my case, the LUN will be created on a Synology NAS (DS413).
I'll not describe how to create a LUN, there is already plenty of tutorial on Google.
Just the basics:
You'll need a LUN (obviously) and a iSCSI Target with this new LUN mapped to it
To connect to this target, you'll need to provide the IQN of this target, the IP of the iSCSI server (the NAS in my case), and your authentication credentials, if any.
Pi iSCSI configuration
Install the open-iscsi service & utilities
$ sudo apt-get install open-iscsi
Start the iscsi service
$ sudo systemctl start open-iscsi
You can define your own initiator name in /etc/iscsi/initiatorname.iscsi or keep the generated one.
ex:
InitiatorName=iqn.2016-11.org.raspberrypi:superPI
Discover all target available on your iSCSI server
$ sudo iscsiadm -m discovery -t sendtargets -p <IP_OF_YOUR_ISCSI_SERVER>
ex:
$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.0.6 192.168.0.6:3260,1 iqn.2000-01.com.synology:BlackSyno.PiTarget
You should find here your newly created target. Log into it
$ sudo iscsiadm -m node -l -T <TARGET_IQN> -p <IP_OF_YOUR_ISCSI_SERVER>
ex:
$ sudo iscsiadm -m node -l -T iqn.2000-01.com.synology:BlackSyno.PiTarget -p 192.168.0.6 Logging in to [iface: default, target: iqn.2000-01.com.synology:BlackSyno.PiTarget, portal: 192.168.0.6,3260] (multiple) Login to [iface: default, target: iqn.2000-01.com.synology:BlackSyno.PiTarget, portal: 192.168.0.6,3260] successful.
Check if your pi has detected your new drive
$ tail /var/log/messages or fdisk -l
In most of the case, your new disk should be named /dev/sda ex:
Nov 5 17:04:12 raspberrypi kernel: [ 2441.542921] sd 4:0:0:1: Attached scsi generic sg1 type 0 Nov 5 17:04:12 raspberrypi kernel: [ 2441.545681] sd 4:0:0:1: [sda] 25165824 512-byte logical blocks: (12.9 GB/12.0 GiB) Nov 5 17:04:12 raspberrypi kernel: [ 2441.550756] sd 4:0:0:1: [sda] Write Protect is off Nov 5 17:04:12 raspberrypi kernel: [ 2441.551927] sd 4:0:0:1: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA Nov 5 17:04:12 raspberrypi kernel: [ 2441.587028] sd 4:0:0:1: [sda] Attached SCSI disk
In order to be able to connect to your iSCSI drive during the boot, you'll need to load an initrd image with the required module.
Creating an initrd image is quite simple on Raspbian
First, tell the initramfs tool to include the iscsi module by creating the required flag file
$ sudo touch /etc/iscsi/iscsi.initramfs
And create the initramfs image for you current kernel
$ sudo update-initramfs -v -k `uname -r` -c
Your new initrd can be found in /boot
$ ls -lrt /boot/init* -rwxr-xr-x 1 root root 9937953 Nov 5 17:27 /boot/initrd.img-4.4.27-v7+
Before continuing on your PI, it's time to set your PXE server ready for the PI
PXE Configuration
I assume here you already have a PXE server running at your disposable.
On my NAS, I use ISC-DHCPd server and tftp-hpa
You can use whatever you want, but you must be able to set the Vendor-Option Option 43 in your DHCP server or the PI will never initiate the boot
You will need the MAC address of your pi (ip a), and its serial number (grep Serial /proc/cpuinfo)
On your PXE server, in /opt/etc/dhcpd.conf (for isc-dhcp), add the following paragraph
host pi {
hardware ethernet b8:27:eb:1e:57:f7;
next-server 192.168.0.6;
filename "";
# Very important (Vendor-Option Option 43)
option vendor-encapsulated-options "Raspberry Pi Boot ";
}
If you have more than one raspberry (like, for a cluster) you can use a group instead :
group {
next-server 192.168.0.6;
filename "";
option vendor-encapsulated-options "Raspberry Pi Boot ";
host redpi {
hardware ethernet b8:27:eb:45:53:f8;
}
host bluepi {
hardware ethernet b8:27:eb:ed:73:b9;
}
host greenpi {
hardware ethernet b8:27:eb:0b:56:29;
}
host whitepi {
hardware ethernet b8:27:eb:a5:b4:83;
}
}
On your TFTP server root, in /opt/tftpboot in my case, create a directory with your PI serial (strip all leading 0)
$ mkdir /opt/tftpboot/511e57f7
Retrieve all boot files from your PI to this new folder
$ scp -r pi@yourPi:/boot/* /tftpboot/511e57f7/
Put the bootcode.bin file at the root of your tftp server
$ cp /tftpboot/511e57f7/bootcode.bin /tftpboot/
Modify your config.txt (in your tftp server), and add the following line
initramfs <YOUR_INITRD_IMAGE> followkernel
ex:
initramfs initrd.img-4.4.27-v7+ followkernel
New system installation
You now need to install a raspbian (or any other distribution) on your new iSCSI drive
Two possibilities:
- Copy your current installation
- Deploy a new installation image
Copy your current installation (easier)
Format your new disk
$ sudo mkfs.ext4 /dev/sda
Get your new drive UUID, will be used in your cmdline.txt later
$ sudo blkid /dev/sda /dev/sda: UUID="afb74bd9-1934-42d7-a7d6-289d2f3ba0d7" TYPE="ext4"
Now, mount your new disk
$ mount /dev/sda /mnt
And copy your current rasbian install (apt-get install rsync may be needed here)
$ sudo rsync -avhP --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
Create all system directories
$ sudo mkdir /mnt/{dev,proc,sys,boot,mnt}
Deploy a new installation image
Download your distro image on your PI
In my case, i'll use the raspbian lite image
You can use dd to copy the image in your new iSCSI drive:
$ sudo dd bs=4M if=/tmp/2016-09-23-raspbian-jessie-lite.img of=/dev/sdb
You'll need your new root partition UUID from your new drive, will be used in your cmdline.txt later
$ sudo blkid /dev/sdb2 /dev/sdb2: UUID="3598ef8e-09be-47ef-9d01-f24cf61dff1d" TYPE="ext4" PARTUUID="5a7089a1-02"
Mount your new installation
$ sudo mount /dev/sdb2 /mnt
Prepare the new system & chroot in it
sudo mount --bind /proc /mnt/proc sudo mount --bind /dev /mnt/dev sudo mount --bind /sys /mnt/sys sudo mount --bind /run /mnt/run sudo chroot /mnt
Refresh the APT cache, and install rpi-update
apt-get update apt-get install rpi-update
Install the last firmware update on this new system (this is to get the same kernel modules level you have on your sd card)
rpi-update
Install the iscsi module
apt-get install open-iscsi
Create the initramfs iscsi flag file, for future initrd generation
touch /etc/iscsi/iscsi.initramfs
New system installation is complete
Final tuning
Since your root directory will be mounted during the initrd phase, you'll not need it in fstab, same for /boot
So, basically, remove everything sdcard related (/dev/mmcblk0p*) from your /mnt/etc/fstab
But, the update-initramfs command look into this file to get the FS type of your /, and include the related fsck command into the initrd (like fsck.ext4)
It's not mandatory, but you can add your newly created root FS at the begginning :
UUID=3598ef8e-09be-47ef-9d01-f24cf61dff1d / ext4 defaults 1 1
Your network interface will also be configured at boot time during the initrd phase, so you may want to disable the dhcp client for this card or you will get 2 addresses for a single interface
in /mnt/etc/dhcpcd.conf, add
denyinterfaces eth0
If you do that, you will also need to manually set your DNS, in /mnt/etc/resolvconf.conf
name_servers=192.168.0.6
Now, you need to set all required parameters to connect to your iSCSI drive during the boot
For that, modify your cmdline.txt (on your tftp server), and add this ( keep everything on a single line )
ip=::::<hostname>:eth0:dhcp root=UUID=afb74bd9-1934-42d7-a7d6-289d2f3ba0d7 ISCSI_INITIATOR=<YOUR_PI_IQN> ISCSI_TARGET_NAME=<TARGET_IQN> ISCSI_TARGET_IP=<IP_OF_YOUR_ISCSI_SERVER> ISCSI_TARGET_PORT=3260 rw
If you use CHAP authentication on your iSCSI target, use ISCSI_USERNAME and ISCSI_PASSWORD variables
so your file should look like this
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=::::superpi:eth0:dhcp root=UUID=3598ef8e-09be-47ef-9d01-f24cf61dff1d rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait qmap=fr ISCSI_INITIATOR=iqn.2016-11.org.raspberrypi:superpi ISCSI_TARGET_NAME=iqn.2000-01.com.synology:BlackSyno.PiTarget ISCSI_TARGET_IP=192.168.0.6 ISCSI_TARGET_PORT=3260 rw
Now, you can shutdown, remove the SDcard, and power your pi back on.
Additional stuff
Green led
The green led usually flash on SD card activity. Since we don't use it anymore, you can stop it from blinking.
Add this line to your config.txt:
dtparam=act_led_trigger=none
Other triggers are available if you want (heartbeat is a nice alternative)
Disable ipv6
You can disable ipv6 if you don't need it.
in /etc/sysctl.conf :
# Disable IPv6 net.ipv6.conf.all.disable_ipv6 = 1
Then apply :
$ sudo sysctl -p
Resize your root partition
In case of a new installation, you may want to increase you root FS to take all your disk.
Use fdisk to do that (delete then recreate the partition using the same start sector)
Reboot, then resize2fs /
Manage kernel update
To keep the kernel up to date on your PXE server, you can mount it at boot on your pi.
in your /etc/fstab (replace with your PXE IP and Pi serial) :
192.168.0.6:/opt/tftpboot/5d4553f8 /boot nfs nfsvers=3,nolock 0 0
The bootcode.bin and start.elf are the first two files to be loaded by your PI. Since they are located in the root directory of your Tftp server, they'll be shared with all your Pi (if you have more than one).
Make sure to do some backup, you'll need it if the upgrade go south and your Pi can't boot anymore.
After that, you can update your distro with apt-get dist-upgrade
The kernel update **SHOULD** recreate your inirtd, but if not, you'll have to do it yourself :
sudo update-initramfs -v -k 4.4.26-v7+ -c
Don't forget to edit your config.txt with this new initrd img
reboot, and pray
More information:
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md
http://backreference.org/2013/12/23/diskless-iscsi-boot-with-pxe-howto/