PXE and GRUB - how to troubleshoot?

Question

I'm working on setting up netboot, and got it working with one motherboard, but it doesn't quite seem to work with another; and I don't see enough output to figure out why.

So, the setup on the 'PXE server' is that DHCP and TFTP reside on the same server, and DHCP is configured with:

root@vogon:/etc/dhcp# cat dhcpd.conf
option domain-name "somewhere.com";
option domain-name-servers 192.168.50.9, 8.8.8.8;
allow booting;
allow bootp;

default-lease-time 600;
max-lease-time 7200;
ddns-update-style none;

# If this DHCP server is the official DHCP server for the local
# network, the authoritative directive should be uncommented.
authoritative;

subnet 192.168.50.0 netmask 255.255.255.0 {
  range 192.168.50.10 192.168.50.190;
  option routers 192.168.50.1;
  option broadcast-address 192.168.50.255;
  next-server 192.168.50.9;
  #filename "debian-installer/amd64/bootnetx64.efi";
  filename "grubx64.efi";
}

TFTP uses /srv/tftp as root dir, and grubx64.efi will read /debian-installer/amd64/grub/grub.cfg because I'm using the bootloader from the debian 11 distro:

root@vogon:/srv/tftp# cat debian-installer/amd64/grub/grub.cfg
set menu_color_normal=cyan/blue
set menu_color_highlight=white/blue
set gfxpayload=text
set timeout=-1

menuentry 'Debian 11'{
    set background_color=black
    linux    /debian/11/amd64/linux priority=low vga=788 ---
    initrd   /debian/11/amd64/initrd.gz
}

menuentry "Ubuntu 20.04" {
        linux /ubuntu/20.04/amd64/linux only-ubiquity ip=dhcp ---
        initrd /ubuntu/20.04/amd64/initrd.gz
}

I have used this setup to successfully install both Debian 11 and Ubuntu 20.04 on a ROG Strix X570-F Gaming motherboard with an AMD Ryzen 9 5950X CPU. In /var/log/syslog I see the following from the tftp server:

Jul  8 13:33:33 vogon in.tftpd[45602]: RRQ from 192.168.50.161 filename grubx64.efi
Jul  8 13:33:33 vogon in.tftpd[45602]: tftp: client does not accept options
Jul  8 13:33:33 vogon in.tftpd[45603]: RRQ from 192.168.50.161 filename grubx64.efi
Jul  8 13:33:34 vogon in.tftpd[45604]: RRQ from 192.168.50.161 filename /debian-installer/amd64/grub/x86_64-efi/command.lst
Jul  8 13:33:34 vogon in.tftpd[45605]: RRQ from 192.168.50.161 filename /debian-installer/amd64/grub/x86_64-efi/fs.lst
Jul  8 13:33:34 vogon in.tftpd[45606]: RRQ from 192.168.50.161 filename /debian-installer/amd64/grub/x86_64-efi/crypto.lst
Jul  8 13:33:34 vogon in.tftpd[45607]: RRQ from 192.168.50.161 filename /debian-installer/amd64/grub/x86_64-efi/terminal.lst
Jul  8 13:33:34 vogon in.tftpd[45608]: RRQ from 192.168.50.161 filename /debian-installer/amd64/grub/grub.cfg
Jul  8 13:33:37 vogon in.tftpd[45609]: RRQ from 192.168.50.161 filename /ubuntu/20.04/amd64/linux
Jul  8 13:33:40 vogon in.tftpd[45610]: RRQ from 192.168.50.161 filename /ubuntu/20.04/amd64/initrd.gz

However, with another motherboard, ASUS ProArt Z690-CREATOR WIFI Intel Z690 PCIe 5.0 ATX, and Intel Core i9 12900KS Special Edition 16 Core Alder Lake Unlocked CPU, I only see the following in the log:

Jul 12 10:49:00 vogon in.tftpd[58652]: RRQ from 192.168.50.136 filename grubx64.efi
Jul 12 10:49:00 vogon in.tftpd[58652]: tftp: client does not accept options
Jul 12 10:49:00 vogon in.tftpd[58653]: RRQ from 192.168.50.136 filename grubx64.efi

and the text 'Welcome to GRUB!' on the client screen. This text is only found in the bootloader, grubx64.efi, so it seems it actually starts running, but never goes on to looking for the other files.

Any suggestions for what I can do to troubleshoot this further?

Answer 1

You don't seem to have the option boot-size line anywhere in your dhcpd configuration. Some UEFI implementations require that this option is used to indicate the size of the UEFI PXE bootloader file, so the UEFI firmware will be able to allocate the right amount of memory for it in advance.

Adding the option (with the correct size specified) should never hurt and might help some UEFI implementations. So please run du -B 512 /srv/tftp/grubx64.efi to determine the size of the grub64.efi file in 512-byte blocks, and then add

option boot-size <size in blocks>;

within the subnet declaration in your dhcpd.conf file.

Having the boot-size option defined was apparently a requirement for PXE boot with earlier versions of the UEFI specification, but I would not be surprised if later versions have made that optional, or if some firmware authors have just chosen to write a PXE boot file download routine that uses the TFTP tsize option to request the TFTP server to tell the total size of the incoming file transfer before it begins, if that option is available.

If adding the option boot-size line doesn't help, and the firmware doesn't provide you any useful diagnostics, then you might want to dump the TFTP traffic and analyze it (e.g. with wireshark) to verify that the entire file gets transferred. On the PXE boot server, you might do something like this:

tcpdump -i eno1 -Knpvv -s0 -w pxe-tftp.cap udp

then make a PXE boot attempt while the tcpdump is running, then press Ctrl+C to stop the tcpdump. You might not be able to filter just the TFTP packets at tcpdump, because the TFTP server may allocate a different port for sending data to the client, and you won't be able to know in advance which port that will be.

Once you get the dump opened in Wireshark, just type in tftp into the filter expression bar and you'll see only the TFTP packets you're interested in. Then right-click on any TFTP packet displayed in the packet list window, and make sure Protocol Preferences -> Trivial File Transfer Protocol -> Reassemble fragmented TFTP files is checked. Now you can find the last packet of each TFTP file transfer (should be helpfully marked with (last)), open the Trivial File Transfer Protocol branch in the packet analysis window, and see a line like:

[nn TFTP Fragments (nnnnn bytes): ...]

The (nnnnn bytes) part will tell you the total transferred size for the file. If it does not match the actual size of the transferred file, you've probably found the cause. Fixing it might require a firmware update, though.