Rebase Xen onto newest upstream version and then backport patches to 4.19

[m-iwanicki]

Affected component(s) or functionality (if applicable)
Xen

Brief summary
Rebase Xen to the newest upstream version, and then backport these patches to 4.19, so that we can build our packages for Qubes OS.

Version
Rebase aem-staging-2025-09-13-v4-wip onto newest upstream/staging

Additional context

• Based on TrenchBoot Community call on Feb 26, 2026: we should make the changes work on master before considering rebasing onto 4.19
• Make changes work in meta-trenchboot first before testing with Qubes (simpler, if Xen doesn't work with meta-trenchboot then it likely won't work with Qubes, and we also want to upstream those changes to Xen)

[m-iwanicki]

Current state:

• Rebased aem-staging-2025-09-13-v4-wip: aem-staging-2026-03-13
• Applied fix: TrenchBoot/xen@2be6e38 for Upstream TrenchBoot as Anti Evil Maid - UEFI boot mode support to Xen #46 (comment)
• meta-trenchboot test PR/branch: Update Xen to current aem-staging. zarhus/meta-trenchboot#59 (aem-staging-rebase)
• Tests done on meta-trenchboot built from c03b01

Success == Xen + Trenchboot booted and PCRs 17 & 18 were extended.

Platform	TPM	Firmware	Result	Notes
HP t630 Thin Client/8158	1.2	BIOS M40 v01.05, UEFI boot	Success	Xen Panic
HP t630 Thin Client/8158	1.2	BIOS M40 v01.05, Legacy boot	Success	Xen Panic
HP t630 Thin Client/8158	2.0	BIOS M40 v01.07, UEFI boot	Success	TPM errors
HP t630 Thin Client/8158	2.0	BIOS M40 v01.07, Legacy boot	Success	TPM errors
Protectli VP4670	2.0	Dasharo (coreboot + SeaBIOS) v0.9.0-rc1 (legacy)	Failure	Also fails on meta-trenchboot release version. Protectli TXT.ERRORCODE
Protectli VP4670	2.0	Dasharo (coreboot + UEFI) v1.2.0-rc2	Failure	Also fails on meta-trenchboot release version. Protectli TXT.ERRORCODE
DELL Optiplex 7010	1.2	Dasharo (coreboot + SeaBIOS) v0.1.0-rc3 (legacy)	Failure	Screen is cleared (on serial only) but after that no output. Frozen indefinitely.
DELL Optiplex 7010	1.2	Dasharo (coreboot + UEFI) v0.1.2-rc1	Failure	TXT.ERRORCODE = 0x80000000

Notes

Xen Panic

Had this also on older Xen (meta-trenchboot v0.5.2 release)

Sometimes Xen will fail to boot due to FATAL PAGE FAULT

(XEN) HVM: SVM enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB, 1GB
(XEN) AMD-Vi: Warning: Log Run bit 8 is not cleared before reset
(XEN) AMD-Vi: Warning: Log Run bit 8 is not cleared before reset
(XEN) APIC ID 0x11 (CPU1) didn't finish start sequence
(XEN) ----[ Xen-4.22-unstable  x86_64  debug=y  Not tainted ]----
(XEN) CPU:    0
(XEN) RIP:    e008:[<ffff82d0403626af>] arch/x86/smpbo#cpu_smpboot_free+0x223/0x5c5
(XEN) RFLAGS: 0000000000010216   CONTEXT: hypervisor
(XEN) rax: ffff8302d50d4db0   rbx: ffff82d040808800   rcx: 0000000000000010
(XEN) rdx: ffff82d040808700   rsi: 0000000000000001   rdi: 0000000000000001
(XEN) rbp: ffff82d04065fd00   rsp: ffff82d04065fc90   r8:  ffff8302d50bc9f0
(XEN) r9:  ffff8302d50bc010   r10: 0000000000000001   r11: 0000000000000000
(XEN) r12: 00000000ffffffff   r13: 0000000000000001   r14: 0000000000008002
(XEN) r15: 0000000000008000   cr0: 000000008005003b   cr4: 00000000001506e0
(XEN) cr3: 00000000d3822000   cr2: ffff830ad50d4da8
(XEN) fsb: 0000000000000000   gsb: 0000000000000000   gss: 0000000000000000
(XEN) ds: 0000   es: 0000   fs: 0000   gs: 0000   ss: 0000   cs: e008
(XEN) Xen code around <ffff82d0403626af> (arch/x86/smpbo#cpu_smpboot_free+0x223/0x5c5):
(XEN)  00 48 8b 05 51 60 4b 00 <4a> 8b 3c e0 8b 35 47 d9 49 00 e8 a9 27 ea ff 85
(XEN) Xen stack trace from rsp=ffff82d04065fc90:
(XEN)    ffff82d040237ef0 ffff8302d50bc9d0 ffff82d04065fce0 ffff82d04024136a
(XEN)    0000000000000000 0000000000000000 ffff8302d50bc9e0 ffff82d040825028
(XEN)    0000000100000001 0000000000000001 0000000000000000 0000000000000001
(XEN)    0000000000008002 0000000000008000 ffff82d04065fd40 ffff82d04036380f
(XEN)    0000000000008000 ffff82d040826320 ffff82d040826318 ffff82d040825028
(XEN)    0000000000008002 0000000000008000 ffff82d04065fd88 ffff82d04022f844
(XEN)    ffff82d04065fdb0 0000000000000001 0000000000000001 00000000fffffffb
(XEN)    0000000000000001 ffff82d04045d378 ffff82d04092f1c0 ffff82d04065fda0
(XEN)    ffff82d0402061c0 0000000000000001 ffff82d04065fdc8 ffff82d040206891
(XEN)    ffff82d0408252f0 0000000000000001 0000000000000000 ffff82d04065ff00
(XEN)    ffff82d040643ceb ffff82d04069c148 00000002da8db000 ffff82d04093ff08
(XEN)    00000002defff400 00000000011e2000 ffff82d04069713c 0000000000100000
(XEN)    ffff82d04069c148 ffff82d040600000 ffff82d040400000 ffff82d000800163
(XEN)    ffff82d04069713c 00000000002df000 00000000d2e00001 0000001300000000
(XEN)    ffff8302d50f0000 ffff82d04065ff08 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 0000000000000000 0000000800000000
(XEN)    000000010000006e 0000000000000003 00000000000002f8 ffff82d040927000
(XEN)    ffff82d040927000 0000000000000000 0000000000000000 0000000000000000
(XEN)    0000000000000000 0000000000000000 00000000d365ff00 0000000000000000
(XEN) Xen call trace:
(XEN)    [<ffff82d0403626af>] R arch/x86/smpbo#cpu_smpboot_free+0x223/0x5c5
(XEN)    [<ffff82d040237ef0>] S _spin_unlock+0x11/0x37
(XEN)    [<ffff82d04036380f>] F arch/x86/smpbo#cpu_smpboot_callback+0x51/0x61e
(XEN)    [<ffff82d04022f844>] F notifier_call_chain+0x6c/0x9a
(XEN)    [<ffff82d0402061c0>] F common/cpu.c#cpu_notifier_call_chain+0x1b/0x3a
(XEN)    [<ffff82d040206891>] F cpu_up+0x8e/0xcc
(XEN)    [<ffff82d040643ceb>] F __start_xen+0x1fa1/0x2588
(XEN)
(XEN) Pagetable walk from ffff830ad50d4da8:
(XEN)  L4[0x106] = 80000000d3820063 ffffffffffffffff
(XEN)  L3[0x02b] = 0000000000000000 ffffffffffffffff
(XEN)
(XEN) ****************************************
(XEN) Panic on CPU 0:
(XEN) FATAL PAGE FAULT
(XEN) [error_code=0000]
(XEN) Faulting linear address: ffff830ad50d4da8
(XEN) ****************************************
(XEN)
(XEN) Reboot in five seconds...
(XEN) Resetting with ACPI MEMORY or I/O RESET_REG.

TPM errors

anti-evil-maid-tpm-setup output. Errors are from kernel

Choose SRK password:
Confirm SRK password:
[   28.010034] tpm2_nvreadpubl[592]: segfault at 557dde88e62f ip 00007f91e115c367 sp 00007ffc89f77e00 error 4 in libc.so.6[96367,7f91e10ea000+151000] likely on CPU 1 (core 1, socket 1)
[   28.011577] Code: 09 10 00 e8 6b d4 f9 ff 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 <48> 8b 47 f8 48 8b 1d 86 8a 13 00 a8 02 64 8b 2b 75 57 48 8b 15 18
[   28.192459] tpm2_nvreadpubl[607]: segfault at 55f2488f29cd ip 00007fa21eb1c367 sp 00007ffece1484e0 error 4 in libc.so.6[96367,7fa21eaaa000+151000] likely on CPU 1 (core 1, socket 1)
[   28.193192] Code: 09 10 00 e8 6b d4 f9 ff 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 <48> 8b 47 f8 48 8b 1d 86 8a 13 00 a8 02 64 8b 2b 75 57 48 8b 15 18
anti-evil-maid-tpm-setup: Taking ownership of the TPM...
[   53.443597] tpm2_nvreadpubl[630]: segfault at 559bdee89636 ip 00007fe2aba04367 sp 00007ffc71aa98b0 error 4 in libc.so.6[96367,7fe2ab992000+151000] likely on CPU 0 (core 0, socket 1)
[   53.445216] Code: 09 10 00 e8 6b d4 f9 ff 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 48 85 ff 0f 84 bb 00 00 00 55 48 8d 77 f0 53 48 83 ec 18 <48> 8b 47 f8 48 8b 1d 86 8a 13 00 a8 02 64 8b 2b 75 57 48 8b 15 18
anti-evil-maid-tpm-setup: Creating TPM ID...
anti-evil-maid-tpm-setup: Creating freshness token NVRAM area...
anti-evil-maid-tpm-setup: Generating TPM NVRAM area AUTHWRITE password
WARN: Reading full size of the NV index

Protectli Error Code

Platform reboots without any output from Xen.
Tried on multiple firmware versions (built by me or older ones known to work) and on 2 different platforms. Same results on both legacy and UEFI, slaunch_state returns: TXT.ERRORCODE: 0xc18b0441.
If I decoded this this code correctly it is:

bit	name	value
31	valid	1 = valid
30	External	1 = external
27:16	Minor Error Code	0x18b = TPM2.0 error code, tpm2_rc_decode 0x18b returns: tpm:handle(1):the handle is not correct for the use
15	SW source	0 = ACM
14:10	Major Error Code	1=ERR_TPM_ERROR
9:4	Class Code	4=CLASS_TPM_ACCESS
3:0	Module Type	1=SINIT

@miczyg1 said it might be due to unprovisioned TPM.

Optiplex UEFI

Xen from meta-trenchboot v0.5.2 works. Rebased one reboots after printing (with sync_console):

(XEN)  - Dom0 mode: Relaxed
(XEN) Interrupt remapping enabled
(XEN) nr_sockets: 1
(XEN) Enabled directed EOI with ioapic_ack_old on!
(XEN) Enabling APIC mode.  Using 1 I/O APICs
(XEN) ENABLING IO-APIC IRQs
(XEN)  -> Using old ACK method
(XEN) ..TIMER: vector=0xF0 apic1=0 pin1=2 apic2=0 pin2=0
(XEN) TSC deadline timer enabled
(XEN) Wallclock source: CMOS RTC
(XEN) Allocated console ring of 32 KiB.
(XEN) mwait-idle: MWAIT substates: 0x1120
(XEN) mwait-idle: v0.4.1 model 0x3a
(XEN) mwait-idle: lapic_timer_reliable_states 0x2
(XEN) HPET: 4 timers usable for broadcast (8 total)
(XEN) VMX: Supported advanced features:
(XEN)  - APIC MMIO access virtualisation
(XEN)  - APIC TPR shadow
(XEN)  - Extended Page Tables (EPT)
(XEN)  - Virtual-Processor Identifiers (VPID)
(XEN)  - Virtual NMI
(XEN)  - MSR direct-access bitmap
(XEN)  - Unrestricted Guest
(XEN) HVM: ASIDs enabled
(XEN) VMX: Disabling executable EPT superpages due to CVE-2018-12207
(XEN) HVM: VMX enabled
(XEN) HVM: Hardware Assisted Paging (HAP) detected
(XEN) HVM: HAP page sizes: 4kB, 2MB

It reboots sometime after waking APs in wake_aps_in_txt (after setting wakeup_addr) likely in start_secondary.

Working branches:

• v0.5.0
• aem-staging-2025-05-12-v2

Failing branches:

• aem-staging-2025-05-30-v3 - no output and reboots with 0xc00020c1 txt.errorcode
• aem-staging-2025-09-24-v4 - same as aem-staging-2026-03-13

I also rebased working aem-staging-2025-05-12-v2 onto the same staging commit as v3 (no merge conflicts), v4 and aem-staging-2026-03-13 , which resulted in the same outputs as mentioned above, so it's not due to Trenchboot changes between versions.

[m-iwanicki]

Something I noticed when trying to find why/where v2 started failing is that booting Xen + Trenchboot works if you boot Linux before that and then reboot. That way v2 rebased onto 1f5e64755d0e works (about 18 additional upstream commits after v2). Trying to boot Trenchboot second time results in v4 issues with the same TXT.errorcode. When rebased onto 1f75bd375d07 (halfway to v3) it fails the same as v3.
So somewhere between v2 and v3 trenchboot boot became unstable and then stopped working, at least on Optiplex.
You can check list of additional upstream commits between v2 and v3 via: git log f6042f38e62..d7d55c27cc32 --oneline (common commits can be found via git merge-base aem-staging-2025-05-12-v2 aem-staging-2025-05-30-v3 for f604... and git merge-base aem-staging-2025-05-30-v3 aem-staging-2025-09-24-v4 for d7d5...)

[accek-itl]

RE: Protectli Error Code

TLDR: this does not seem related to the rebase.

@miczyg1 said it might be due to unprovisioned TPM.

I agree with this.

This is how to check if the TPM contains the Platform Supplier Policy required by SINIT:

tpm2_nvreadpublic    # look for 0x01C10102
tpm2_nvread 0x01C10102

If this does not exist, it is the reason for TXT.ERRORCODE 0xc18b0441. But this should be configured by BIOS/manufacturer.
Theoretically it can be copied from a similar machine, but if vendor did not do it, more pieces may be missing. Maybe clearing TPM in BIOS could provision it.

You can also check the Platform Owner Policy:

tpm2_nvread 0x01C10106

It's ok if it does not exist, but if it's in some broken state, you can clear it (it's optional):

tpm2_nvundefine 0x01C10106

RE: Optiplex UEFI

TXT.ERRORCODE = 0x80000000 indicates successful launch from SINIT perspective. Something later is broken.

[accek-itl]

Apart from the Xen Panics, did all the platforms from the table work before the rebase?
Is there any way I can help debugging?

[m-iwanicki]

did all the platforms from the table work before the rebase

The newest rebase that works on Optiplex is aem-staging-2025-05-12-v2 (mentioned at the end of #82 (comment)). Couldn't get v3 or v4 to work.

Not sure about Protectli, @SergiiDmytruk was doing previous rebases, not sure on which platforms have you tested it previously?

[SergiiDmytruk]

Before the rebases the release was tested on whatever is listed as tested with v0.5 here. Various versions sent upsteam weren't extensively tested (can't use openQA for that) other than when CI had some issue and I believe that I used Protectli in those cases.

@m-iwanicki, I thought you've determined that 18 upstream commits may be responsible for the issue and were going to bisect/selectively revert them to find when exactly boot stops working. That didn't work?