Issue on the 'qemu-tdx' attestation variant

[Ruoyu-y]

Issue description

There's already QEMU TDX code within Constellation to support the qemu-tdx attestation variant. After enable them in the source code and leverage the local Libvirt/QEMU(already have TDX support) to bootstrap constellation, i met an error like this:

Error: error creating libvirt domain: internal error: qemu unexpectedly closed the monitor: 2024-02-26T01:39:08.566024Z qemu-s
2024-02-26T01:39:08.566075Z qemu-system-x86_64: warning: ==============================================================
2024-02-26T01:39:08.566083Z qemu-system-x86_64: warning: !!!    Warning: Please upgrade to upstream version TDVF    !!!
2024-02-26T01:39:08.566090Z qemu-system-x86_64: warning: !!!             Old version will be deprecated soon        !!!
2024-02-26T01:39:08.566096Z qemu-system-x86_64: warning: ==============================================================
2024-02-26T01:39:08.566102Z qemu-system-x86_64: failed to parse TDVF for TDX VM

However, this OVMF could boot up TDs successfully in my local environment. Would there be cases that i misconfigure the constellation-conf.yaml or something within terraform files? Any hint?

Steps to reproduce the behavior

No response

Version

No response

Constellation Config

Configurations used:

version: v4 # Schema version of this configuration file.
image: v2.14.3 # Machine image version used to create Constellation nodes.
name: constell # Name of the cluster.
kubernetesVersion: v1.28.5 # Kubernetes version to be installed into the cluster.
microserviceVersion: v2.16.0-pre.0.20240205105659-a97569b111a7 # Microservice version to be installed into the cluster. Defaults to the version of the CLI.
debugCluster: false # DON'T USE IN PRODUCTION: enable debug mode and use debug images.
customEndpoint: "" # Optional custom endpoint (DNS name) for the Constellation API server.
internalLoadBalancer: false # Flag to enable/disable the internal load balancer. If enabled, the Constellation is only accessible from within the VPC.
serviceCIDR: 10.96.0.0/12 # The Kubernetes Service CIDR to be used for the cluster. This value will only be used during the first initialization of the Constellation.
# Supported cloud providers and their specific configurations.
provider:
  # Configuration for QEMU as provider.
  qemu:
    imageFormat: raw # Format of the image to use for the VMs. Should be either qcow2 or raw.
    vcpus: 2 # vCPU count for the VMs.
    memory: 2048 # Amount of memory per instance (MiB).
    metadataAPIServer: docker.io/rry1/qemu-metadata-api:v2.15.0-pre.0.20240131153006-08491f2d8f81@sha256:fbdb3429f7f248141d087f076581997e62072ab571a75c828025a3ace1699caa # Container image to use for the QEMU metadata server.
    libvirtSocket: "qemu:///system" # Libvirt connection URI. Leave empty to start a libvirt instance in Docker.
    libvirtContainerImage: docker.io/rry1/libvirt:v2.15.0-pre.0.20240131153006-08491f2d8f81@sha256:231c09d1574fddb6a681b787d0d40edfd08dae15411dfe6ebab38a7fa57bf1b5 # Container image to use for launching a containerized libvirt daemon. Only relevant if `libvirtSocket = ""`.
    nvram: production # NVRAM template to be used for secure boot. Can be sentinel value "production", "testing" or a path to a custom NVRAM template
    firmware: "/usr/share/qemu/OVMF_CODE.fd" # Path to the OVMF firmware. Leave empty for auto selection.
# Node groups to be created in the cluster.
...

[malt3]

I would expect that this is an issue with the command line parameters that qemu is started with. Maybe you can detect a misconfiguration by looking at the libvirt logs in /var/log/libvirt/qemu/$vmname.log?
You might need to set a higher verbosity for libvirt: https://libvirt.org/kbase/debuglogs.html

The error message makes it sound like an issue with loading the TDVF, so the issue is likely somewhere between libvirt and qemu.
Maybe inspecting the domxml file that libvirt uses for VMs is also helpful.

[Ruoyu-y]

I would expect that this is an issue with the command line parameters that qemu is started with. Maybe you can detect a misconfiguration by looking at the libvirt logs in /var/log/libvirt/qemu/$vmname.log? You might need to set a higher verbosity for libvirt: https://libvirt.org/kbase/debuglogs.html

The error message makes it sound like an issue with loading the TDVF, so the issue is likely somewhere between libvirt and qemu. Maybe inspecting the domxml file that libvirt uses for VMs is also helpful.

The log looks like this after changing the logfilter of libvirt to 'debug', but i don't see any hint related to that:

PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin \
HOME=/var/lib/libvirt/qemu/domain-1-constell-control-pla \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-1-constell-control-pla/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-1-constell-control-pla/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-1-constell-control-pla/.config \
/usr/bin/qemu-system-x86_64 \
-name guest=constell-control-plane-bee79a15-0,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-1-constell-control-pla/master-key.aes"}' \
-machine pc-q35-7.2,usb=off,dump-guest-core=off,kernel_irqchip=split,confidential-guest-support=lsec0,memory-backend=pc.ram \
-accel kvm \
-cpu host,migratable=on \
-m 2048 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":2147483648}' \
-overcommit mem-lock=off \
-smp 2,sockets=2,cores=1,threads=1 \
-uuid 83a1fb49-af14-421f-bec2-8e52a5e28007 \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc \
-no-hpet \
-no-shutdown \
-boot strict=on \
-kernel /var/lib/libvirt/images/constell-kernel \
-initrd /var/lib/libvirt/images/constell-initrd \
-append 'ima_hash=sha384 ima_policy=critical_data' \
-device '{"driver":"pcie-root-port","port":8,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x1"}' \
-device '{"driver":"pcie-root-port","port":9,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x1.0x1"}' \
-device '{"driver":"pcie-root-port","port":10,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x1.0x2"}' \
-device '{"driver":"pcie-root-port","port":11,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x1.0x3"}' \
-device '{"driver":"pcie-root-port","port":12,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x1.0x4"}' \
-device '{"driver":"pcie-root-port","port":13,"chassis":6,"id":"pci.6","bus":"pcie.0","addr":"0x1.0x5"}' \
-device '{"driver":"pcie-root-port","port":14,"chassis":7,"id":"pci.7","bus":"pcie.0","addr":"0x1.0x6"}' \
-device '{"driver":"qemu-xhci","id":"usb","bus":"pci.2","addr":"0x0"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.3","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/constell-node-image","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/constellation-control-plane-bee79a15-0-boot","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":"libvirt-3-format"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.4","addr":"0x0","drive":"libvirt-2-format","id":"virtio-disk0","bootindex":1}' \
-blockdev '{"driver":"file","filename":"/var/lib/libvirt/images/constellation-control-plane-bee79a15-0-state","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"driver":"qcow2","file":"libvirt-1-storage","backing":null}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.5","addr":"0x0","drive":"libvirt-1-format","id":"virtio-disk1"}' \
-netdev tap,fd=32,vhost=on,vhostfd=37,id=hostnet0 \
-device '{"driver":"virtio-net-pci","netdev":"hostnet0","id":"net0","mac":"52:54:00:6e:23:69","bus":"pci.1","addr":"0x0"}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev pty,id=charconsole0 \
-device '{"driver":"virtconsole","chardev":"charconsole0","id":"console0"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"virtio-balloon-pci","id":"balloon0","bus":"pci.6","addr":"0x0"}' \
-object '{"qom-type":"tdx-guest","id":"lsec0","debug":true,"sept-ve-disable":true,"quote-generation-service":"vsock:2:4050"}' \
-cpu host,-kvm-steal-time \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/4 (label charconsole0)
2024-03-11T01:26:41.530043Z qemu-system-x86_64: Cannot find TDX_METADATA_OFFSET_GUID
2024-03-11T01:26:41.530095Z qemu-system-x86_64: warning: ==============================================================
2024-03-11T01:26:41.530101Z qemu-system-x86_64: warning: !!!    Warning: Please upgrade to upstream version TDVF    !!!
2024-03-11T01:26:41.530106Z qemu-system-x86_64: warning: !!!             Old version will be deprecated soon        !!!
2024-03-11T01:26:41.530111Z qemu-system-x86_64: warning: ==============================================================
2024-03-11T01:26:41.530115Z qemu-system-x86_64: failed to parse TDVF for TDX VM
2024-03-11 01:26:41.985+0000: shutting down, reason=failed