Design and Hardening Decisions in the dstack Yocto Layer

Overview

The dstack-owned Yocto layer under os/yocto/layers/meta-dstack/ is designed to create a minimally secure image for booting Confidential Virtual Machines (CVMs). Our design philosophy prioritizes attack surface reduction while maintaining TDX-aware functionality. This document outlines the architectural decisions and trade-offs made during development.

Key Design Decisions

1. Yocto Kernel Recipe Selection

Decision: We use linux-yocto-dev (development recipes) instead of linux-yocto (stable recipes).

Rationale: We use Scarthgap version of Yocto, which is the latest version when we started the dstack project. The stable linux-yocto recipe in Scarthgap is based on kernel 6.6 which does not support RTMR[1-2]. So we switch to linux-yocto-dev to get the kernel 6.9 support. The latest released Yocto (Walnascar) updated kernel to 6.12 which meets our requirements. We plan to upgrade the Yocto version later, but this is not trivial as all downstream Yocto recipes need to be updated to adapt to the major Yocto version change.

2. TDVF vs td-shim Boot Firmware

Decision: We use Intel's TDVF implementation integrated in OVMF rather than td-shim.

Current State: TDVF is a mature, proven solution currently in use. td-shim is a Rust-based implementation aimed at minimizing attack surface.

Rationale: td-shim cannot currently boot our dstack system successfully. td-shim is considered too new for production use in our current requirements. TDVF provides stable, tested functionality for our TDX requirements.

3. TDX Guest Driver Implementation

Decision: We use the in-tree confidential guest drivers and the Linux TSM report interface where they are available. For Intel TDX, the dstack kernel configuration enables CONFIG_TDX_GUEST_DRIVER=y and CONFIG_TSM_REPORTS=y.

Rationale: The in-tree TDX guest driver exposes the standard /dev/tdx_guest device and configfs-tsm report interface. Newer kernels also expose RTMR extension through the TSM measurement sysfs path, which dstack uses for RTMR3 runtime events. This keeps dstack aligned with the standard Linux kernel ABI while preserving measured runtime events.

Implementation Notes: The unified dstack confidential-guest image also includes AMD SEV-SNP kernel features. Platform-specific native TEE interfaces are advanced compatibility surfaces for applications that need the kernel ABI directly. The dstack socket remains the normal application API for quotes, keys, application information, and runtime events.

4. Randomness Generation and Seeding

Security Requirement: Ensure cryptographically secure randomness without trusting the host system.

We configure the kernel with specific command-line arguments:

random.trust_cpu=y
random.trust_bootloader=n

random.trust_cpu=y enables trust in CPU-provided randomness (Intel RDRAND). random.trust_bootloader=n prevents untrusted host-provided entropy.

Intel RDRAND hardware RNG instruction. System Integration: Proper seeding of /dev/random and /dev/urandom. Application Support: Ensures container and application randomness needs.

See here for more details.

5. Secure System Time

Implementation: dstack OS enforces the guest kernel uses TSC as the only timer source by appending tsc=reliable no-kvmclock to the kernel cmdline. It also enforces the use of NTS with built-in trusted servers to synchronize system time.

Behavior: When secure_time is enabled in the app-compose.json configuration, the system ensures time synchronization is completed before requesting application keys. If secure_time is disabled, time synchronization is not enforced before application launch.

Rationale: Time synchronization is provided as an optional feature because the process typically requires tens of seconds to complete. Applications can function without the secure_time option enabled and may implement their own time synchronization mechanisms if required.

See here for more details.