This tutorial is a work in progress. For now, it's in a raw state, but it should help you to set you up (or not, idk).

I'll show you here how to boot and run your Raspberry Pi 3 without SD card.
I'm using a Synology NAS to host an iSCSI LUN which will be used as the root filesystem for the pi.
In order to mount this LUN, the pi will need a starting kernel & initrd image to get the network up, connect to the iSCSI target (the NAS) and mount the root filesystem.
This kernel & module will be provided by a PXE server (also running on that NAS).

IMG_6210-2.jpg

Before going further, I'll assume you already have a PXE server up and running. In this tutorial, I use ISC-DHCPd server and tftp-hpa.
You'll also need a running install of Raspbian on your pi to do all the preparation.

Enable PXE Boot capabilities

Check if the usb_boot_mode is set in your OTP memory :

$ vcgencmd otp_dump | grep 17:
17:1020000a

If the usb_boot_mode bit is set, the value should be 3020000a
If this is not the case, install the last firmware update (important, this will not work with the current stable firmware):

$ sudo BRANCH=next rpi-update

Add this line to your config.txt to set the usb_boot_mode bit in the OTP memory:

program_usb_boot_mode=1

Then reboot

Now, if you check the OTP again, you should get the wanted value

$ vcgencmd otp_dump | grep 17:
17:3020000a

If everything is ok, remove the "program_usb_boot_mode" line from your config.txt

You can now check that the PXE boot mode is working by power cycling your pi, without SDcard and with a ethernet cable plugged in.
The ethernet leds should light up after a few seconds.

Put the SDcard back, and reboot.

LUN creation and configuration

In my case, the LUN will be created on a Synology NAS (DS413).
I'll not describe how to create a LUN, there is already plenty of tutorial on Google.
Just the basics:
You'll need a LUN (obviously) and a iSCSI Target with this new LUN mapped to it
To connect to this target, you'll need to provide the IQN of this target, the IP of the iSCSI server (the NAS in my case), and your authentication credentials, if any.

Pi iSCSI configuration

Install the open-iscsi service & utilities

$ sudo apt-get install open-iscsi

Start the iscsi service

$ sudo systemctl start open-iscsi

You can define your own initiator name in /etc/iscsi/initiatorname.iscsi or keep the generated one.
ex:

InitiatorName=iqn.2016-11.org.raspberrypi:superPI

Discover all target available on your iSCSI server

$ sudo iscsiadm -m discovery -t sendtargets -p <IP_OF_YOUR_ISCSI_SERVER>

ex:

$ sudo iscsiadm -m discovery -t sendtargets -p 192.168.0.6
192.168.0.6:3260,1 iqn.2000-01.com.synology:BlackSyno.PiTarget

You should find here your newly created target. Log into it

$ sudo iscsiadm -m node -l -T <TARGET_IQN> -p <IP_OF_YOUR_ISCSI_SERVER>

ex:

$ sudo iscsiadm -m node -l -T iqn.2000-01.com.synology:BlackSyno.PiTarget -p 192.168.0.6
Logging in to iface: default, target: iqn.2000-01... (multiple)
Login to iface: default, target: iqn.2000-01... successful.

Check if your pi has detected your new drive

$ tail /var/log/messages or fdisk -l

In most of the case, your new disk should be named /dev/sda ex:

Nov  5 17:04:12 raspberrypi kernel: [ 2441.542921] sd 4:0:0:1: Attached scsi generic sg1 type 0
Nov  5 17:04:12 raspberrypi kernel: [ 2441.545681] sd 4:0:0:1: [sda] 25165824 512-byte logical blocks: (12.9 GB/12.0 GiB)
Nov  5 17:04:12 raspberrypi kernel: [ 2441.550756] sd 4:0:0:1: [sda] Write Protect is off
Nov  5 17:04:12 raspberrypi kernel: [ 2441.551927] sd 4:0:0:1: [sda] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
Nov  5 17:04:12 raspberrypi kernel: [ 2441.587028] sd 4:0:0:1: [sda] Attached SCSI disk

In order to be able to connect to your iSCSI drive during the boot, you'll need to load an initrd image with the required module.
Creating an initrd image is quite simple on Raspbian

First, tell the initramfs tool to include the iscsi module by creating the required flag file

$ sudo touch /etc/iscsi/iscsi.initramfs

And create the initramfs image for you current kernel

$ sudo update-initramfs -v -k `uname -r` -c

Your new initrd can be found in /boot

$ ls -lrt /boot/init*
-rwxr-xr-x 1 root root 9937953 Nov  5 17:27 /boot/initrd.img-4.4.27-v7+

Before continuing on your PI, it's time to set your PXE server ready for the PI

PXE Configuration

I assume here you already have a PXE server running at your disposable.
On my NAS, I use ISC-DHCPd server and tftp-hpa
You can use whatever you want, but you must be able to set the Vendor-Option Option 43 in your DHCP server or the PI will never initiate the boot
You will need the MAC address of your pi (ip a), and its serial number (grep Serial /proc/cpuinfo)

On your PXE server, in /opt/etc/dhcpd.conf (for isc-dhcp), add the following paragraph

  host pi {
    hardware ethernet b8:27:eb:1e:57:f7;
    next-server 192.168.0.6;
    filename "";
    # Very important (Vendor-Option Option 43)
    option vendor-encapsulated-options "Raspberry Pi Boot   ";
  }

On your TFTP server root, in /opt/tftpboot in my case, create a directory with your PI serial (strip all leading 0)

$ mkdir /opt/tftpboot/511e57f7

Retrieve all boot files from your PI to this new folder

$ scp -r pi@yourPi:/boot/* /tftpboot/511e57f7/

Put the bootcode.bin file at the root of your tftp server

$ cp /tftpboot/511e57f7/bootcode.bin /tftpboot/

Modify your config.txt (in your tftp server), and add the following line

initramfs <YOUR_INITRD_IMAGE> followkernel

ex:

initramfs initrd.img-4.4.27-v7+ followkernel

New system installation

You now need to install a raspbian (or any other distribution) on your new iSCSI drive
Two possibilities:
- Copy your current installation
- Deploy a new installation image

Copy your current installation (easier)

Format your new disk

$ sudo mkfs.ext4 /dev/sda

Get your new drive UUID, will be used in your cmdline.txt later

$ sudo blkid /dev/sda
/dev/sda: UUID="afb74bd9-1934-42d7-a7d6-289d2f3ba0d7" TYPE="ext4"

Now, mount your new disk

$ mount /dev/sda /mnt

And copy your current rasbian install (apt-get install rsync may be needed here)

$ sudo rsync -avhP --exclude /boot --exclude /proc --exclude /sys --exclude /dev --exclude /mnt / /mnt/

Create all system directories

$ sudo mkdir /mnt/{dev,proc,sys,boot,mnt}

Deploy a new installation image

Download your distro image on your PI
In my case, i'll use the raspbian lite image
You can use dd to copy the image in your new iSCSI drive:

$ sudo dd bs=4M if=/tmp/2016-09-23-raspbian-jessie-lite.img of=/dev/sdb

You'll need your new root partition UUID from your new drive, will be used in your cmdline.txt later

$ sudo blkid /dev/sdb2
/dev/sdb2: UUID="3598ef8e-09be-47ef-9d01-f24cf61dff1d" TYPE="ext4" PARTUUID="5a7089a1-02"

Mount your new installation

$ sudo mount /dev/sdb2 /mnt

Prepare the new system & chroot in it

sudo mount --bind /proc /mnt/proc
sudo mount --bind /dev /mnt/dev
sudo mount --bind /sys /mnt/sys
sudo mount --bind /run /mnt/run
sudo chroot /mnt

Refresh the APT cache, and install rpi-update

apt-get update
apt-get install rpi-update

Install the last firmware update on this new system (this is to get the same kernel modules level you have on your sd card)

BRANCH=next rpi-update

Install the iscsi module

sudo apt-get install open-iscsi

Create the initramfs iscsi flag file, for future initrd generation

sudo touch /etc/iscsi/iscsi.initramfs

New system installation is complete

Final tuning

Since your root directory will be mounted during the initrd phase, you'll not need it in fstab, same for /boot
So, basically, remove everything sdcard related (/dev/mmcblk0p*) from your /mnt/etc/fstab
But, the update-initramfs command look into this file to get the FS type of your /, and include the related fsck command into the initrd (like fsck.ext4)
It's not mandatory, but you can add your newly created root FS at the begginning :

UUID=3598ef8e-09be-47ef-9d01-f24cf61dff1d       /       ext4    defaults        1       1

Your network interface will also be configured at boot time during the initrd phase, so you may want to disable the dhcp client for this card or you will get 2 addresses for a single interface
in /mnt/etc/dhcpcd.conf, add

denyinterfaces eth0

If you do that, you will also need to manually set your DNS, in /mnt/etc/resolvconf.conf

name_servers=192.168.0.6

Now, you need to set all required parameters to connect to your iSCSI drive during the boot
For that, modify your cmdline.txt (on your tftp server), and add this ( keep everything on a single line )

ip=::::<hostname>:eth0:dhcp
root=UUID=afb74bd9-1934-42d7-a7d6-289d2f3ba0d7
ISCSI_INITIATOR=<YOUR_PI_IQN>
ISCSI_TARGET_NAME=<TARGET_IQN>
ISCSI_TARGET_IP=<IP_OF_YOUR_ISCSI_SERVER>
ISCSI_TARGET_PORT=3260
rw

If you use CHAP authentication on your iSCSI target, use ISCSI_USERNAME and ISCSI_PASSWORD variables

so your file should look like this

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 ip=::::superpi:eth0:dhcp root=UUID=3598ef8e-09be-47ef-9d01-f24cf61dff1d rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait qmap=fr ISCSI_INITIATOR=iqn.2016-11.org.raspberrypi:superpi ISCSI_TARGET_NAME=iqn.2000-01.com.synology:BlackSyno.PiTarget ISCSI_TARGET_IP=192.168.0.6 ISCSI_TARGET_PORT=3260 rw

Now, you can shutdown, remove the SDcard, and power your pi back on.


## Tuning ##

  1. stop green led from blinking
  2. add this to config.txt:
  3. Disable the ACT LED.
dtparam=act_led_trigger=none
  1. Disable ipv6
  2. in /etc/sysctl.conf
  3. Disable IPv6
net.ipv6.conf.all.disable_ipv6 = 1
  1. then apply
sudo sysctl -p
  1. In the case of a new installation, you may want to increase you root FS to take all your disk
  2. Use fdisk to do that (delete then recreate the partition using the same start sector)
  3. reboot, then resize2fs
  1. Mount /boot from your PXE server
sudo mount -o r 192.168.0.6:/opt/tftpboot/511e57f7 /boot
  1. save your bootcode.bin and start.elf in some safe place, you'll need it if the upgrade go south
  1. then, you can update your distro with apt-get dist-upgrade
  1. firmware and bootloader in stable repository doesnt support boot over PXE very well
  2. If the package raspberry-bootloader or raspberry-firmware get updated, they will override your working bootloader in /boot
  3. mso ake sure to get the latest version with a rpi-update afterward
sudo BRANCH=next rpi-update
  1. if rpi-update say you're using the latest version, remove the flag and try again
sudo rm /boot/.firmware_revision
  1. generate the new initramfs
sudo update-initramfs -v -k 4.4.26-v7+ -c
  1. and edit your config.txt with this new initrd img
  1. reboot, and pray

More information:
https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/net.md http://backreference.org/2013/12/23/diskless-iscsi-boot-with-pxe-howto/