darknao's stuff - Tag - raspberrypi2018-03-21T21:27:44+01:00urn:md5:fe81f8e34a9707902b13938791dc2b60DotclearNet boot (PXE + iSCSI) with a RaspberryPi 3urn:md5:36a1b3c323523ddde342ad1fe88b5d9d2018-03-14T13:32:00+01:002018-03-21T22:27:44+01:00darknaolinuxiscsipipxeraspberrypi <p>I'll show you here how to boot and run your Raspberry Pi 3 without SD card.<br />
I'm using a Synology NAS to host an iSCSI LUN which will be used as the root filesystem for the pi.<br />
In order to mount this LUN, the pi will need a starting kernel & initrd image to get the network up, connect to the iSCSI target (the NAS) and mount the root filesystem.<br />
This kernel & module will be provided by a PXE server (also running on that NAS).<br /></p>
<p><a href="https://stuff.drkn.ninja/public/rpi/IMG_6210-2.jpg" title="IMG_6210-2.jpg"><img src="https://stuff.drkn.ninja/public/rpi/.IMG_6210-2_m.jpg" alt="IMG_6210-2.jpg" style="display:table; margin:0 auto;" title="IMG_6210-2.jpg, nov. 2016" /></a></p>
<p>Before going further, I'll assume you already have a PXE server up and running. In this tutorial, I use ISC-DHCPd server and tftp-hpa.<br />
You'll also need a running install of Raspbian on your pi to do all the preparation.</p>
<h3>Enable PXE Boot capabilities</h3>
<p>Check if the usb_boot_mode is set in your OTP memory :</p>
<pre>$ vcgencmd otp_dump | grep 17:
17:1020000a</pre>
<p>If the usb_boot_mode bit is set, the value should be <strong>3020000a</strong><br />
If this is not the case, install the last firmware update (important, this will not work with the current stable firmware):</p>
<pre>$ sudo rpi-update</pre>
<p>Add this line to your <strong>config.txt</strong> to set the usb_boot_mode bit in the OTP memory:</p>
<pre>program_usb_boot_mode=1</pre>
<p>Then reboot</p>
<p>Now, if you check the OTP again, you should get the wanted value</p>
<pre>$ vcgencmd otp_dump | grep 17:
17:3020000a</pre>
<p>If everything is ok, remove the "program_usb_boot_mode" line from your <strong>config.txt</strong></p>
<p>You can now check that the PXE boot mode is working by power cycling your pi, without SDcard and with a ethernet cable plugged in.<br />
The ethernet leds should light up after a few seconds.</p>
<p>Put the SDcard back, and reboot.</p>
<h3>LUN creation and configuration</h3>
<p>In my case, the LUN will be created on a Synology NAS (DS413).<br />
I'll not describe how to create a LUN, there is already plenty of tutorial on Google.<br />
Just the basics:<br />
You'll need a LUN (obviously) and a iSCSI Target with this new LUN mapped to it<br />
To connect to this target, you'll need to provide the IQN of this target, the IP of the iSCSI server (the NAS in my case), and your authentication credentials, if any.</p>
<h3>Pi iSCSI configuration</h3>
<p>Install the open-iscsi service & utilities</p>
<pre>$ sudo apt-get install open-iscsi</pre>
<p>Start the iscsi service</p>
<pre>$ sudo systemctl start open-iscsi</pre>
<p>You can define your own initiator name in <strong>/etc/iscsi/initiatorname.iscsi</strong> or keep the generated one.<br />
ex:</p>
<pre>InitiatorName=iqn.2016-11.org.raspberrypi:superPI</pre>
<p>Discover all target available on your iSCSI server</p>
<pre>$ sudo iscsiadm -m discovery -t sendtargets -p <IP_OF_YOUR_ISCSI_SERVER></pre>
<p>ex:</p>
<pre>$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.0.6
192.168.0.6:3260,1 iqn.2000-01.com.synology:BlackSyno.PiTarget</pre>
<p>You should find here your newly created target. Log into it</p>
<pre>$ sudo iscsiadm -m node -l -T <TARGET_IQN> -p <IP_OF_YOUR_ISCSI_SERVER></pre>
<p>ex:</p>
<pre>
$ sudo iscsiadm -m node -l -T iqn.2000-01.com.synology:BlackSyno.PiTarget -p 192.168.0.6
Logging in to [iface: default, target: iqn.2000-01.com.synology:BlackSyno.PiTarget, portal: 192.168.0.6,3260] (multiple)
Login to [iface: default, target: iqn.2000-01.com.synology:BlackSyno.PiTarget, portal: 192.168.0.6,3260] successful.
</pre>
<p>Check if your pi has detected your new drive</p>
<pre>$ tail /var/log/messages or fdisk -l</pre>
<p>In most of the case, your new disk should be named /dev/sda
ex:</p>
<pre>
Nov 5 17:04:12 raspberrypi kernel: [ 2441.542921] sd 4:0:0:1: Attached scsi generic sg1 type 0
Nov 5 17:04:12 raspberrypi kernel: [ 2441.545681] sd 4:0:0:1: [sda] 25165824 512-byte logical blocks: (12.9 GB/12.0 GiB)
Nov 5 17:04:12 raspberrypi kernel: [ 2441.550756] sd 4:0:0:1: [sda] Write Protect is off
Nov 5 17:04:12 raspberrypi kernel: [ 2441.551927] sd 4:0:0:1: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Nov 5 17:04:12 raspberrypi kernel: [ 2441.587028] sd 4:0:0:1: [sda] Attached SCSI disk
</pre>
<p>In order to be able to connect to your iSCSI drive during the boot, you'll need to load an initrd image with the required module.<br />
Creating an initrd image is quite simple on Raspbian</p>
<p>First, tell the initramfs tool to include the iscsi module by creating the required flag file</p>
<pre>$ sudo touch /etc/iscsi/iscsi.initramfs</pre>
<p>And create the initramfs image for you current kernel</p>
<pre>$ sudo update-initramfs -v -k `uname -r` -c</pre>
<p>Your new initrd can be found in /boot</p>
<pre>$ ls -lrt /boot/init*
-rwxr-xr-x 1 root root 9937953 Nov 5 17:27 /boot/initrd.img-4.4.27-v7+</pre>
<p>Before continuing on your PI, it's time to set your PXE server ready for the PI</p>
<h3>PXE Configuration</h3>
<p>I assume here you already have a PXE server running at your disposable.<br />
On my NAS, I use ISC-DHCPd server and tftp-hpa<br />
You can use whatever you want, but you must be able to set the <strong>Vendor-Option Option 43</strong> in your DHCP server or the PI will never initiate the boot<br />
You will need the MAC address of your pi (<strong>ip a</strong>), and its serial number (<strong>grep Serial /proc/cpuinfo</strong>)</p>
<p>On your PXE server, in /opt/etc/dhcpd.conf (for isc-dhcp), add the following paragraph</p>
<pre>
host pi {
hardware ethernet b8:27:eb:1e:57:f7;
next-server 192.168.0.6;
filename "";
# Very important (Vendor-Option Option 43)
option vendor-encapsulated-options "Raspberry Pi Boot ";
}
</pre>
<p>If you have more than one raspberry (like, for a cluster) you can use a group instead :</p>
<pre>
group {
next-server 192.168.0.6;
filename "";
option vendor-encapsulated-options "Raspberry Pi Boot ";
host redpi {
hardware ethernet b8:27:eb:45:53:f8;
}
host bluepi {
hardware ethernet b8:27:eb:ed:73:b9;
}
host greenpi {
hardware ethernet b8:27:eb:0b:56:29;
}
host whitepi {
hardware ethernet b8:27:eb:a5:b4:83;
}
}
</pre>
<p>On your TFTP server root, in /opt/tftpboot in my case, create a directory with your PI serial (strip all leading 0)</p>
<pre>$ mkdir /opt/tftpboot/511e57f7</pre>
<p>Retrieve all boot files from your PI to this new folder</p>
<pre>$ scp -r pi@yourPi:/boot/* /tftpboot/511e57f7/</pre>
<p>Put the <strong>bootcode.bin</strong> file at the root of your tftp server</p>
<pre>$ cp /tftpboot/511e57f7/bootcode.bin /tftpboot/</pre>
<p>Modify your config.txt (in your tftp server), and add the following line</p>
<pre>initramfs <YOUR_INITRD_IMAGE> followkernel</pre>
<p>ex:</p>
<pre>initramfs initrd.img-4.4.27-v7+ followkernel</pre>
<h3>New system installation</h3>
<p>You now need to install a raspbian (or any other distribution) on your new iSCSI drive<br />
Two possibilities:<br />
- Copy your current installation<br />
- Deploy a new installation image<br /></p>
<h4>Copy your current installation (easier)</h4>
<p>Format your new disk</p>
<pre>$ sudo mkfs.ext4 /dev/sda</pre>
<p>Get your new drive UUID, will be used in your cmdline.txt later</p>
<pre>$ sudo blkid /dev/sda
/dev/sda: UUID="afb74bd9-1934-42d7-a7d6-289d2f3ba0d7" TYPE="ext4"</pre>
<p>Now, mount your new disk</p>
<pre>$ mount /dev/sda /mnt</pre>
<p>And copy your current rasbian install (apt-get install rsync may be needed here)</p>
<pre>
$ sudo rsync -avhP --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/
</pre>
<p>Create all system directories</p>
<pre>$ sudo mkdir /mnt/{dev,proc,sys,boot,mnt}</pre>
<h4>Deploy a new installation image</h4>
<p>Download your distro image on your PI<br />
In my case, i'll use the raspbian lite image<br />
You can use dd to copy the image in your new iSCSI drive:</p>
<pre>$ sudo dd bs=4M if=/tmp/2016-09-23-raspbian-jessie-lite.img of=/dev/sdb</pre>
<p>You'll need your new root partition UUID from your new drive, will be used in your cmdline.txt later</p>
<pre>$ sudo blkid /dev/sdb2
/dev/sdb2: UUID="3598ef8e-09be-47ef-9d01-f24cf61dff1d" TYPE="ext4" PARTUUID="5a7089a1-02"</pre>
<p>Mount your new installation</p>
<pre>$ sudo mount /dev/sdb2 /mnt</pre>
<p>Prepare the new system & chroot in it</p>
<pre>
sudo mount --bind /proc /mnt/proc
sudo mount --bind /dev /mnt/dev
sudo mount --bind /sys /mnt/sys
sudo mount --bind /run /mnt/run
sudo chroot /mnt
</pre>
<p>Refresh the APT cache, and install rpi-update</p>
<pre>
apt-get update
apt-get install rpi-update
</pre>
<p>Install the last firmware update on this new system (this is to get the same kernel modules level you have on your sd card)</p>
<pre>
rpi-update
</pre>
<p>Install the iscsi module</p>
<pre>
apt-get install open-iscsi
</pre>
<p>Create the initramfs iscsi flag file, for future initrd generation</p>
<pre>
touch /etc/iscsi/iscsi.initramfs
</pre>
<p>New system installation is complete</p>
<h3>Final tuning</h3>
<p>Since your root directory will be mounted during the initrd phase, you'll not need it in fstab, same for /boot<br />
So, basically, remove everything sdcard related (/dev/mmcblk0p*) from your /mnt/etc/fstab<br />
But, the update-initramfs command look into this file to get the FS type of your /, and include the related fsck command into the initrd (like fsck.ext4)<br />
It's not mandatory, but you can add your newly created root FS at the begginning :<br /></p>
<pre>
UUID=3598ef8e-09be-47ef-9d01-f24cf61dff1d / ext4 defaults 1 1
</pre>
<p>Your network interface will also be configured at boot time during the initrd phase, so you may want to disable the dhcp client for this card or you will get 2 addresses for a single interface<br />
in /mnt/etc/dhcpcd.conf, add</p>
<pre>denyinterfaces eth0</pre>
<p>If you do that, you will also need to manually set your DNS, in /mnt/etc/resolvconf.conf</p>
<pre>name_servers=192.168.0.6</pre>
<p>Now, you need to set all required parameters to connect to your iSCSI drive during the boot<br />
For that, modify your cmdline.txt (on your tftp server), and add this ( keep everything on a single line )</p>
<pre>
ip=::::<hostname>:eth0:dhcp
root=UUID=afb74bd9-1934-42d7-a7d6-289d2f3ba0d7
ISCSI_INITIATOR=<YOUR_PI_IQN>
ISCSI_TARGET_NAME=<TARGET_IQN>
ISCSI_TARGET_IP=<IP_OF_YOUR_ISCSI_SERVER>
ISCSI_TARGET_PORT=3260
rw
</pre>
<p>If you use CHAP authentication on your iSCSI target, use ISCSI_USERNAME and ISCSI_PASSWORD variables</p>
<p>so your file should look like this</p>
<pre>
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=::::superpi:eth0:dhcp root=UUID=3598ef8e-09be-47ef-9d01-f24cf61dff1d rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait qmap=fr ISCSI_INITIATOR=iqn.2016-11.org.raspberrypi:superpi ISCSI_TARGET_NAME=iqn.2000-01.com.synology:BlackSyno.PiTarget ISCSI_TARGET_IP=192.168.0.6 ISCSI_TARGET_PORT=3260 rw
</pre>
<p>Now, you can shutdown, remove the SDcard, and power your pi back on.</p>
<h3>Additional stuff</h3>
<h4>Green led</h4>
<p>The green led usually flash on SD card activity. Since we don't use it anymore, you can stop it from blinking.<br /></p>
<p>Add this line to your config.txt:</p>
<pre>
dtparam=act_led_trigger=none
</pre>
<p>Other <a href="https://raspberrypi.stackexchange.com/a/69759">triggers</a> are available if you want (<strong>heartbeat</strong> is a nice alternative)</p>
<h4>Disable ipv6</h4>
<p>You can disable ipv6 if you don't need it.<br /></p>
<p>in /etc/sysctl.conf :</p>
<pre>
# Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
</pre>
<p>Then apply :</p>
<pre>$ sudo sysctl -p</pre>
<h4>Resize your root partition</h4>
<p>In case of a new installation, you may want to increase you root FS to take all your disk.<br />
Use fdisk to do that (delete then recreate the partition using the <strong>same start sector</strong>)<br />
Reboot, then resize2fs /</p>
<h4>Manage kernel update</h4>
<p>To keep the kernel up to date on your PXE server, you can mount it at boot on your pi.<br />
in your /etc/fstab (replace with your PXE IP and Pi serial) :</p>
<pre>
192.168.0.6:/opt/tftpboot/5d4553f8 /boot nfs nfsvers=3,nolock 0 0
</pre>
<p>The <strong>bootcode.bin</strong> and <strong>start.elf</strong> are the first two files to be loaded by your PI. Since they are located in the root directory of your Tftp server, they'll be shared with all your Pi (if you have more than one).<br />
Make sure to do some backup, you'll need it if the upgrade go south and your Pi can't boot anymore.<br /></p>
<p>After that, you can update your distro with apt-get dist-upgrade</p>
<p>The kernel update **SHOULD** recreate your inirtd, but if not, you'll have to do it yourself :</p>
<pre>
sudo update-initramfs -v -k 4.4.26-v7+ -c
</pre>
<p>Don't forget to edit your <strong>config.txt</strong> with this new initrd img</p>
<p>reboot, and pray</p>
<p>More information:<br />
<a href="https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md">https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md</a><br />
<a href="http://backreference.org/2013/12/23/diskless-iscsi-boot-with-pxe-howto/">http://backreference.org/2013/12/23/diskless-iscsi-boot-with-pxe-howto/</a></p>