---

name: billet-infra-terraform-ansible description: "The infrastructure layers around billet: the Terraform modules under terraform/modules (root, control-plane-ec2-sqlite, fleet-ec2, fleet-codebuild, state-rds-postgres, control-plane-postgres, converge-aws-ssm, converge-cloudflare, converge-cloudflare-warp), the tf-* gates and what each catches, internal/tfclass and internal/tfpolicy, and the Ansible collection (the host and development_host roles, their variables, the upgrade transaction and the scenario tests). Load when touching any .tf, .tfvars or classification.json, any file under ansible_collections/, the IAM billet generates, or when a converge or an apply touches a host that is running work."

Terraform and Ansible

What this area is

Three layers own three things. Terraform creates cloud resources and returns narrow outputs; Ansible converges existing machines over SSH (packages, users, units, Firecracker, Ceph, bridges, the config, the upgrade transaction); billet owns live jobs, leases, identity, custody and drains. docs/reference/decisions/adr-004-terraform-provider.md defers a Terraform provider until billet has a configuration API, so terraform/modules/billet is an infrastructure module, and enrollment stays a human fingerprint comparison outside terraform apply. ansible_collections/junioryono/billet holds the host role (a Linux control plane and Firecracker compute host), ssh_access (the keys a converge connects with and sshd hardened only once they leave a way in), cloudflared_connector and warp_connector (the host halves of Routes C and D below) and development_host (a developer machine beside the runners), with examples under examples/single-host-docker and examples/firecracker-host.

Rules

The terraform gates are outside make check and CI runs them on every PR. tf-fmt-check, tf-validate, tf-test, tf-lint (tflint with the aws ruleset pinned in .tflint.hcl; --config takes an absolute path, because a relative one resolves against --chdir and is silently not found), tf-scan (trivy with --skip-check-update pinned to the embedded rego bundle, a second pass with tests/scan.tfvars so feature-flagged resources are seen, and a per-module .trivyignore carrying its own justifications). TF_MODULES is discovered by find, because a listed module let two siblings go green through the whole suite without ever being validated; tf-modules-check refuses an empty or incomplete list (a .tf.json-only module would otherwise be skipped in silence). terraform-consumer-floor validates as a Terraform 1.9 consumer. make tools installs the pins; make module-sources is inside check because it needs only git and shell.

The root billet init hybrid writes is validated by terraform, not by Go. scripts/hybrid-root-check.sh (make hybrid-root-check, in CI's terraform job) generates the untrusted, trusted, builder and cache shapes, rewrites the module source to this checkout, and runs terraform fmt -check, init and validate, because HCL rendered as text can name a module input that no longer exists and every Go test would stay green.

tfclass says what a change costs a running deployment; terraform plan cannot. ADR-004 keeps live billet nodes outside Terraform, so a plan does not know these hosts are running somebody's build and "1 to change" reads the same for a tag and for the instance holding the ledger. classification.json beside the module records the cost of every resource change; make tf-classify (in the Go job as well) is the vacuity gate proving every module resource is classified, and scripts/tfclassify classifies a real plan for an operator.

tfpolicy keeps the module's IAM equal to the generator's. The fleet-ec2 module once hand-wrote its grant, a second source of truth beside internal/awspolicy; the drift test reads the generator's committed rendering (UPDATE_TF_POLICY=1 regenerates) so the module can never grant a permission billet init iam would not.

Modules pin to release tags, and every layer names the same version. A release carries the binary, the collection and the module, so billet_version, requirements.yml and every ?ref= should read one vX.Y.Z; cutting a release rewrites the READMEs from main to that tag. A moving target makes a converge non-deterministic and drives a real drain on a day nobody chose, which is why the host role refuses billet_version: latest. Releases up to v0.3.26 carry no module.

What each module creates. The root adopts or creates VPC, subnet, IGW and route table and composes two children; moved.tf maps every relocated resource. control-plane-ec2-sqlite: the controller with EC2 auto-recovery (not an ASG, which launches a fresh instance that does not reattach the data volume), its security group and alarm, and a retained encrypted gp3 ledger volume with prevent_destroy, output as ledger_volume_id. fleet-ec2: the node role and instance profile from awspolicy, the trusted-runner security group, the cache bucket and optional per-deployment KMS key, the spot queues and the EventBridge→Lambda router (spot_router.py, unit-tested by make lambda-test because its drop-versus-re-raise classification decides whether a two-minute warning is lost — the module passes every created queue's name in as BILLET_INTERRUPTION_QUEUE_NAMES, comma-separated, because a tag naming a queue outside that set is the only positive proof a warning is another deployment's; AccessDenied alone is not, being equally what an absent or unpropagated grant on a queue the router SERVES answers, and everything the router cannot prove foreign is re-raised onto the spot_router_errors alarm). Several spot nodes are several queues, and spot_node_names is the one input that makes them: each entry creates a queue named exactly it, and the node role's consumer grant, the router's forwarding grant and the served set all derive from the one list local.spot_queues, so the three converge to the same set from one input — a queue granted by hand but never named to the router was exactly the shape that dropped a warning while the grant propagated (#66), and the plan test asserts the three sets equal the declared queues rather than a literal. Convergence is not simultaneity: the router policy carries depends_on on the function so the served set lands BEFORE the grant widens, because a function that already serves a name re-raises the new queue's AccessDenied while the grant propagates and one that does not drops it; the Lambda suite asserts that edge positionally, since terraform test exposes no dependency graph. The cap of 16 bounds this input's share of IAM's 10,240-character combined inline-policy quota, not an override's. fleet-codebuild: the project, an optional reserved fleet (the only route to managed macOS), both roles, the log group and the JIT parameter path. state-rds-postgres and control-plane-postgres: the PostgreSQL ledger (or a grant for one you run; it never learns the password) and a controller with no ledger volume, so it can be rebuilt rather than recovered. Outputs are the non-secret facts Ansible's billet_config needs: node_wire_address (the private address; a public posture supplies its own name), bootstrap_wire_address, cache_bucket, interruption_queue_url, spot_node_name, interruption_queue_urls (every spot node's queue keyed by its node.name), ledger_volume_id.

terraform destroy refuses under a running CodeBuild build, and it is the module that refuses. AWS's DeleteProject succeeds while a build is in its BUILD phase (measured 2026-09-02). terraform_data.active_build_guard depends on everything a build needs and runs refuse-active-builds.sh as a destroy-time provisioner; its triggers_replace is the project name, the region and the module's other build-facing identities, because a terraform_data's input updates in place and a rename would have replaced the project while merely updating the guard. The escape is an environment variable, BILLET_SKIP_ACTIVE_BUILD_GUARD=1, because a destroy-time provisioner sees state, not -var. The script pins TZ=UTC (the CLI renders times in the local zone and a lexical comparison was seven hours wrong) and never calls StopBuild. The module also refuses VPC inputs beside an adopted fleet_arn, because a fleetOverride makes CodeBuild ignore the project's VPC and the apply would succeed while the inputs went nowhere.

The host role owns the fail-closed ledger mount and the upgrade transaction. With billet_ledger_volume_id set to the module's output, the role resolves the volume by its NVMe identity (never a filesystem UUID, which a snapshot clone duplicates), formats only when blank, mounts through a systemd unit with no nofail, adds Requires= and RequiresMountsFor= to billet-server.service, proves the state directory is served by the expected volume and refuses to shadow a root-disk ledger. Without that, a failed mount lets the controller start on the root disk with a new ledger, identity and CA. billet_server_prepare_only: true is the stopping point for moving a control plane: it mounts and proves the volume and leaves both services installed, disabled and refused by their own units, so nothing can mint an identity until the old host's state is copied in; both are held because whichever role starts first mints the identity. Do not systemctl mask here, because the role renders the unit at the path masking would write. The upgrade transaction is in billet-releases-and-upgrades.

The role gives way to a rollout, and renders the timers that carry one out. billet_automatic_updates (default true) installs billet-upgrade.{service,timer} on a host whose server should run and billet-images.{service,timer} on a firecracker host, and enables the timers (installed disabled when false); tests/unit-parity.yml compares their directives with deploy/. upgrade-inspect.yml refuses a converge when /var/lib/billet/upgrades/active is a symlink, which is billet host-upgrade's claim, naming --status and --resume. account.yml reads both binaries' version and refuses a proved downgrade before the drain unless billet_allow_downgrade, because a stale billet_version beside a fleet rollouts have moved would otherwise drain the host every converge to be refused by the ledger's watermark. The role's transaction still refuses a PostgreSQL host; billet's timer upgrades that host. The gate in host-upgrade-order.yml counts billet_enable_server occurrences in services.yml, so a new task gates on billet_server_should_run, never on billet_enable_server. tasks/backup.yml renders and enables billet-backup.{service,timer} under that predicate and billet_backup_timer, and refuses an effective drop-in on the backup unit the way account.yml refuses one on the services (asked of systemd through DropInPaths after a reload, status judged), because the ledger fence is rendered into the unit; a consumer that carried the fence as a drop-in removes it before converging with this role.

A converge restarts services, so never drive one from a runner billet manages. The drain destroys the jobs on that host, including the one running the playbook. The role refuses by default; billet_allow_converge_from_billet_runner exists for an operator who has established the runner is backed by another host.

Reaching hosts is configuration management, not scheduling. billet needs no inbound connectivity. Route A is a workstation on the same network (or a small always-on machine registered as an ordinary Actions runner billet does not manage). Route B is converge-aws-ssm: the SSM connection plugin ships every module through an S3 bucket even for modules that send no files, and the App key transits it, so the module creates the bucket with versioning suspended and a short expiry, a hybrid activation, a scoped policy and a GitHub OIDC role trusting an exact subject (a wildcard admits any pull-request job; an environment subject does not close that). Route C is converge-cloudflare: a tunnel, an Access application and a service token, no S3 and no plaintext-secret path, requiring a domain on Cloudflare; cloudflared on the host is the cloudflared_connector role. Route D is converge-cloudflare-warp: the CI runner enrols as a headless WARP client with a service token (mdm.xml), authenticates as the shared non_identity@<team>.cloudflareaccess.com principal, and one Gateway L4 rule keyed on that principal is its entire reach; the module creates the token, the Service Auth (non_identity) enrolment policy and the rule, and deliberately never adopts the account's WARP enrolment application (an omitted optional plans to null on an adopted resource in provider v5, and policies is one). Measured (2026-09): Route C's Access SSH proxy never engaged on a real hybrid deployment while this route worked first time; Gateway stores a /32 as the bare address, so a rule written with /32s never converges; precedence is required because it is one account-wide sequence.

The shipped playbook junioryono.billet.fleet is the consumer's site.yml, and it runs the converge guard first in every play. Three plays in upgrade order (control_plane, linux, macos), no play-level become (the roles escalate per task; development_host reads the login user from the facts and a play-level become would make Homebrew run as root), development_host gated on billet_development_enabled, and every role skipping cleanly on absent inputs so an inventory that says nothing converges exactly the host role. Because ssh_access runs before the host role, the host role's own first-task refusal of a billet-managed runner would fire after a key was installed; each play therefore opens with include_role: junioryono.billet.host, tasks_from: converge-guard, tagged always on the include and on what it includes so --tags cannot skip it, and fleet-playbook-check proves the shape and that the refusal leaves no key behind, under ANSIBLE_HOST_PATTERN_MISMATCH=error with an empty macos group declared, because a targeted group the inventory never declares fails a strict run after the earlier plays converged: the generated inventory and both examples declare all three groups, empty ones as hosts: {}. billet init hybrid and both examples render - import_playbook: junioryono.billet.fleet; a consumer's own plays go after it, and a play CI must never run belongs in another file.

The connector roles read a bearer token per host from the environment and refuse what a healthy-looking mistake would hide. cloudflared_connector and warp_connector read BILLET_CLOUDFLARED_TOKEN_<HOST> and BILLET_WARP_CONNECTOR_TOKEN_<HOST> (the inventory hostname upper-cased, everything outside [A-Z0-9] an underscore; billet_*_token_env overrides the name), never -e, because ansible-playbook's argv is world-readable for the run and its environment is not. Two hosts in one play mapping to one name are refused before either acts: the names are computed from every play host's inventory (ansible_play_hosts_all, failed hosts included) in a set_fact, never from a fact each host records first, because under serial the later batches have recorded nothing when the first batch asks (unique(case_sensitive=true), since variable names are); and because hostvars does not see the vars of a role invocation, each host also asserts that the name it reads is the one the list holds for it. EVERY host asserts both, never one host for all: a run_once task is skipped for every host when the first host's inherited when is false, and a consumer may invoke the role under a condition. With no token each role does nothing, so a play can run them everywhere. cloudflared's token is base64({"a","s","t"}), billet_cloudflared_expected_tunnel_id is REQUIRED whenever a token is supplied and checked against the t before anything is installed, because a token for the other host's tunnel installs cleanly and comes up healthy advertising routes the host cannot serve (measured on the reference deployment), and an empty id would skip that one check by default. cloudflared service install takes the token only in argv and writes it verbatim, no newline, 0600 root, beside a unit that runs tunnel run --token-file (its source, cmd/cloudflared/common_service.go and linux_service.go; measured on 2026.7.3), so the role writes the file and renders that unit byte for byte (the template's own comment is a Jinja comment, because a rendered # line would make the units differ) and never runs the installer. THE DANGEROUS ACT IS ANY RESTART OF A RUNNING CONNECTOR, whatever causes it: the role reads the credential and unit as they are (a stat loop for owner, group and mode, which is metadata and uncensored; both slurps as root under no_log, because an older installer wrote --token <TOKEN> into ExecStart and the unit being adopted may carry it; the template through lookup) against the bytes, owner and mode it would write, asks systemd for the unit's ActiveState, and refuses to rewrite either unless the state is exactly inactive or failed (positive evidence nothing serves traffic; activating under a long ExecStartPost is running) or billet_cloudflared_carries_ansible_transport is a definite false. Reading rather than predicting in check mode, because a check-mode task that would change counts as changed in the recap and a refusal must precede every write. The state and the enablement are two systemd_service tasks, both no_log because the module returns the unit's whole status, ExecStart included, and an adopted unit may still carry an inline token in a dry run that did not rewrite it; "the state task changed" is what means a process was started or restarted (from inactive, failed or deactivating alike), and the wait keys on exactly that. An explicit answer must be true or false (yes and 1 are refused, not read as empty); empty derives from billet_cloudflared_routed_address: ansible_host equal to it is true, a DIFFERENT IPv4 literal (strict dotted quad) is false, and anything else (no ansible_host, a name, an instance id, IPv6) is could-not-tell, because two spellings can name one address (a name; 2001:db8::1 beside 2001:0db8:0:0:0:0:0:1) and the role resolves and canonicalises nothing. A first install restarts no running connector and is never refused. The registration wait runs only when the state task reports changed and reads only that invocation's journal (systemctl show -p InvocationID; an empty id is a refusal, never a window read), because cloudflared logs the line at startup and reconnect and a window read could find the previous process's line; in a dry run the service task still predicts a start or enable where the unit exists. warp-cli connector new takes the token only as <TOKEN> (measured, 2026.6.880.0), runs exactly on the daemon's own Registration Missing read from the one Status update: line (a daemon that answered without one is refused, not read as any state), and never re-enrols, because that tears down the interface the play may be running over. Both roles install gnupg and python3-debian from the distribution first (billet_*_prerequisites, empty for a host that has them; deb822_repository needs the latter on the host), stop a dry run on a host with no client yet, and in a dry run verify only a key that is already staged and only where gpg and python3-debian are already installed (a dry run predicts the prerequisite install and does not do it), because get_url in check mode still fetches to compare and fails on the staging directory the dry run never created. The cloudflared apt suite is pinned noble (no suite for newer codenames, measured on Ubuntu 26.04); the WARP repository publishes the real codename; the WARP sysctls set the IPv6 keys only where /proc/sys/net/ipv6 exists and remove them from the file first where it does not, because sysctl -p of a file carrying them fails before any loop condition helps; the WARP legacy /tmp key path is an input, so the gate removes a file in its own root. cloudflared-connector-check and warp-connector-check drive every refusal against fakes that record argv, compare the rendered unit byte for byte with the installer's (the 2026.7.3 tag's template, blank line before [Install] included), grep every whole encoded token in argv, output and the unit (a marker inside the JSON is invisible once encoded; the first gate could not fire), assert the journal was read by exactly the invocation the fake systemd runs (the fake answers an old line to a window read and nothing to a wrong id, and its bare show carries the unit's ExecStart), judge a refusal by EVERY recap row being unchanged and by the named hosts each having refused (a run_once assertion that failed on one host ends the play for the others without their asserting anything), run the role skipped on the first host with two later hosts sharing a name, run the roles under serial: 1 and with the name set on the role invocation, and never run the apt path past the keyring, because deb822_repository writes under /etc/apt with no path seam and a case that reached it would touch the real machine wherever it happened to have python3-debian (it failed on the CI runner for lacking it, and would have written sources on one that had it). Twenty mutations fail them.

The role's variables that must be supplied, never guessed. billet_binary_src or billet_version (with billet_release_url_base and the channel); billet_config (what billet init --emit ansible prints); billet_github_private_key_src (never replaced implicitly; the role refuses a converge when the managed key path holds nothing) and billet_github_target_key_srcs (target name to a local key file for every further billet_config.targets entry, installed at /etc/billet/app-private-key-<name>.pem under the same owned-versus-foreign rule, computed as one billet_github_target_keys list in tasks/main.yml and looped in tasks/account.yml; tests/key-policy-check.sh runs the target cases too); capacity, network ranges, billet_networks and guest DNS; billet_ceph_* (bootstrap is destructive and never inferred; every disk it may consume is named); billet_firecracker_version and checksums. The role installs nothing it cannot verify and refuses to start either service before billet check passes.

ssh_access never hardens a host its own converge leaves with no way in, and that refusal is not gated. The reference deployment once had a conditional key install beside an unconditional hardening, and a missing variable produced a green run and a host reachable only from its console. A key is exactly a type, its material and an optional comment on one line, and its identity is the MATERIAL: ansible.posix.authorized_key removes by material alone, ignoring the type and the comment, and replaces the options of a key it already holds (2.2.2's source), so a line with options (whose quoted strings defeat any tokenizer but the module's), a value spanning lines, one material with two states or under two types, two present entries for one material and the break-glass key beside an absent or present entry with its material are each refused before anything is written; so are a comment where a key should be, an absent-only inventory, and keys for root under any policy but yes or prohibit-password (the policy is normalised before it is compared and rendered). The drop-in is validated as the whole configuration sshd reads, with the file in place and a backup taken; the rescue restores the backup or removes the file only when the copy itself succeeded, judged from a stat the role took beforehand and never from the copy's missing return metadata; and the restart handler is notified by the validation, not the copy, so a rollback never restarts sshd onto a configuration another file broke. The account is ansible_user or, when the inventory leaves it to SSH configuration, id -un asked with ansible_become: false as a task variable, which outranks an inventory's escalation. With nothing configured it does nothing, which is what lets the collection's fleet playbook run it on every host in its control_plane and linux groups. Every key it installs is proved readable by ssh-keygen -l -f - on the host first, because the module installs any material and sshd -t reads no authorized_keys file; present keys are installed before any absent key is removed, so a rotation never leaves a hardened host keyless between two tasks; and unique(case_sensitive=true) is what compares materials, because Jinja's unique folds case. Its paths and validator are inputs (billet_ssh_sshd_config_dir, billet_ssh_authorized_keys_path, billet_ssh_validate_command), which is how ssh-access-check drives it against a temporary tree and a fake sshd with -e ansible_become=false (a connection variable outranks the tasks' become keyword). A refusal fragment in that gate is from a message, never a task name: Ansible prints every task's name whether or not it fails, so a fragment shared with the name passed with the assertion neutered (measured).

The scenario tests are make targets, each one converging the role in a specific state. host-upgrade-order (drain, migration, image, restart and rollback ordering), unit-parity (packaged units and the role's templates agree), ledger-mount-render, postgres-profile, alert-lifecycle, development-check-mode, emitted-block-check (a block billet init --emit ansible generated converges), example-check, key-policy-check, converge-guard-check, ssh-access-check (the ssh_access refusals and install against a fake sshd), cloudflared-connector-check and warp-connector-check (the connector roles against fakes), fleet-playbook-check (the shipped playbook's shape and guard), release-fetch-check, firecracker-example-check, fetch-retry-check (every get_url/uri in both roles carries timeout, register, until, retries and delay, driven by the billet_fetch_* and billet_development_fetch_* defaults; one stalled connection once failed a converge behind an approval gate), hybrid-emission-check (the inventory and playbook billet init hybrid writes pass Ansible's syntax check and ansible-inventory --list proves the tier catalogue crossed the anchor onto both hosts), host-fresh-check (the documented first step against a fresh machine). CI's host-lifecycle job runs them all; host-upgrade-live.sh is the destructive one.

The collection's galaxy.yml carries the release version. A collection installed from a Git tag reports the version in that file, so the cut-release workflow rewrites it.

Measured facts

• DeleteProject succeeds during a live build; the build runs to completion and BatchGetBuilds keeps answering for it.
• A lexical time comparison in the destroy guard was seven hours wrong on a laptop in a non-UTC zone.
• Two terraform modules were added and went green through every gate without being validated, linted or scanned, because the module list was a constant.
• tflint --config with a relative path: silently linted with bundled rules only.
• trivy 0.74 needs GOEXPERIMENT=jsonv2 to install under Go 1.26.

Where the tests are

• internal/tfclass/*_test.go (make tf-classify), internal/tfpolicy (the drift test), scripts/codebuild_destroy_guard_test.go, terraform/modules/*/tests/*.tftest.hcl, terraform/modules/billet/modules/fleet-ec2/lambda/*_test.py.
• ansible_collections/junioryono/billet/tests/* (the scenario playbooks and shell checks), scripts/check-module-sources.sh.

Related skills

billet-providers-aws (the IAM and the CodeBuild facts the modules encode), billet-releases-and-upgrades (the host transaction and version pinning), billet-config (--emit ansible), billet-lifecycle (the units the role renders).