Foreman in IPv6-mostly and IPv6-only environments

Foreman is clearly rooted in an IPv4 world. You can see it everywhere. Like it does not want to give you a host without IPv4, but without IPv6 is fine. But it does not have to be this way, so let’s go together on a journey to bring Foreman and the things around it into the present.

Background

We start with a typical Foreman installation, used to bootstrap new system installations. This Foreman uses a ISC DHCP server and a TFTP server to provide systems with boot information. It uses a Foreman-Proxy to configure those DHCP and TFTP servers.

The DHCP server stores leases for every system. For each mac address the DHCP server stores the host name, IPv4 address and the boot file, pointing to a standard PXELinux. The leases are setup by Foreman on every change to its internal state.

The TFTP server stores all files required for booting. The TFTP server serves the static PXELinux binaries. It also stores PXELinux config files for each mac address separately. Those configs tell each system if it should boot an installer or the system on the local storage. Foreman modifies those config files all the time, depending on the internal state.

This means that we have a lot of locations to store data. Not only does Foreman store all information in its own database. But also the DHCP and TFTP server need to store information for normal operation of the overall system. We can restore this information if lost, but this process is manual.

Booting via iPXE and IPv6

qemu uses iPXE as the standard PXE loader. libvirt for example uses qemu. iPXE is powerful, it includes its own scripting language. Also it implements IPv6 an a useful state.

iPXE supports scripts. Foreman uses that in the iPXE intermediate script, which shows up as “PXE Loader” on the “Operation System” tab. This script tries the dhcp iPXE command on many network devices, which then tries both IPv4 and IPv6 configuration. If iPXE finds some configuration, it asks Foreman for the next boot loader stage, which is also a script that either run an installer or the installed system.

iPXE supports IPv6 in a useful state. iPXE can receive IPv6 Router Advertisements and uses them to set its own host address and gateway. Also it uses the “other information” flag to enable the DHCPv6 client on request. It then asks via DHCPv6 for additional config, like the boot file. The boot file points to the next stage loader, so is a critical piece of information.

dnsmasq can provide both Router Advertisements and stateless DHCPv6.

dhcp-range=2001:db8::,ra-stateless
dhcp-option=option6:bootfile-url,http://foreman.example.com/unattended/iPXE?bootstrap=1

Booting via UEFI and IPv6

If you have a current iPXE, the whole thing is easy. But now UEFI comes into play. UEFI also needs a network stack and qemu provides iPXE for that. But iPXE provides only network access, UEFI itself does the communication. But UEFI would not be UEFI if it would do things sensibly. This means at least the reference implementation EDK2 does not support Router Advertisements, just stateful DHCPv6.

But the lack of the iPXE core functionality also means we loose an important feature we leveraged above, the scripting language. So even with DHCPv6 providing addresses, the only thing this setup can now do is loading UEFI binaries.

But we still have a full iPXE for EFI, just as standalone binary. We can load this binary via TFTP and then reuse the same boot chain as without UEFI. Just be aware it is much slower, as UEFI takes minutes trying to boot via IPv4.

dhcp-range=2001:db8::1,2001:db8::ffff
dhcp-option=option6:dns-server,[2001:db8::]
dhcp-option=tag:pxe-amd64,option6:bootfile-url,tftp://2001:db8::/ipxe-amd64.efi
dhcp-vendorclass=set:pxe-amd64,enterprise:343,PXEClient:Arch:00007:UNDI:003001
enable-tftp
tftp-root=/usr/lib/ipxe/

Bugs

Of cause we found bugs, but for all of them we have workarounds.

Foreman and IPv6-only

Foreman does not really like the IPv4 network to be empty. If you try to unset the IPv4 network in the interface config, it just adds one back. I did not write a bug report for this yet.

A workaround I found is:

• Create a subnet using the IPv4 link-local range 169.254.0.0/16.
• Assign this network to the interface.

The network information vanishes then and only the IPv6 network remains.

iPXE provides bogus parameter to preseed

The “preseed_kernel_options” template snippet adds to the command line for preseed (aka Debian) netcfg/get_ipaddress=${netX/ip}. This uses internal iPXE variables, which are empty if no DHCPv4 exists. Filed as issue #38536.

--- a/app/views/unattended/provisioning_templates/iPXE/preseed_default_ipxe.erb
+++ b/app/views/unattended/provisioning_templates/iPXE/preseed_default_ipxe.erb
@@ -36,7 +36,7 @@ ping --count 1 ${netX/dns} || echo Ping to DNS failed or ping command not availa
 <% kernel = boot_files_uris[0] -%>
 <% initrd = boot_files_uris[1] -%>
 
-kernel <%= kernel %> initrd=initrd.img interface=auto url=<%= url %> ramdisk_size=10800 root=/dev/rd/0 rw auto <%= snippet("preseed_kernel_options", variables: {ipxe_net: true}).strip %>
+kernel <%= kernel %> initrd=initrd.img interface=auto url=<%= url %> ramdisk_size=10800 root=/dev/rd/0 rw auto <%= snippet("preseed_kernel_options").strip %>
 
 initrd <%= initrd %>
 

Conclusion

It needs a bit of work, but Foreman can work both in IPv6-mostly and IPv6-only environments. All communication between managed systems and Foreman including support services uses IPv6 if configured this way.

We also removed multiple services that store data. Like the ISC DHCP server and the TFTP server. The replacement is mostly stateless.