---

description: "Runtime telemetry regression audit — drives a headless engine on per-game cells, diffs against checked-in baselines, and reports the state of the smoke / golden-frame gate matrix" argument-hint: "--game <key|all> [--regen] [--cell <EDID>]"

Runtime Telemetry Audit

Read .claude/commands/_audit-common.md (dedup, severity, finding format) and _audit-severity.md for shared protocol.

The runtime arm of the audit suite: the per-game audit-* skills inspect code, this one inspects what renders. It drives the engine headless on a representative cell, harvests stats, tex.missing, mesh.cache failed, light.dump and the bench: summary line, and diffs them against .claude/audit-baselines/runtime/<game>-<cell>.tsv. Counts moving the wrong way become findings. It also audits the integrity of its own harness and baselines (below).

Gate matrix (what exists, what each needs)

| Gate | Run | Needs | |------|-----|-------| | Telemetry diff (this skill) | .claude/commands/audit-runtime/capture.sh --game <key> --cell <EDID> --out /tmp/audit/runtime | release build (cargo build --release -p byroredux -p byro-dbg), Vulkan device, xvfb-run, game data | | Harness self-test | capture.sh --self-test | nothing (parsers, ±2 % tolerance, PID resolution, survivor sweep) | | Baseline schema tests | cargo test -p byroredux bench:: (every_baseline_carries_the_full_gating_metric_set, every_baseline_records_the_harness_bench_mode, every_baseline_records_the_bench_camera_pose, draw_split_rows_are_internally_ordered) | nothing | | Golden frames (game-data-free, not per-game): cube_demo_golden_frame, combustion_lab_golden_frame | cargo test --release -p byroredux --test golden_frames -- --ignored; regen BYROREDUX_REGEN_GOLDEN=1 | Vulkan device. Default-lane guard committed_baseline_matches_the_current_invocation fails when a case's capture args/frames change without a recapture | | Closed-HIGH/CRITICAL real-data pins (the #[ignore]d regression tests behind fixed HIGH issues: the #5005 corpse rule, #5228 precombine rebake, #5238 actor-value curve, #5489 PKIN, ...) | scripts/run-real-data-pins.sh [filter] (list: --list; one cargo process per pin from .github/real-data-pins.txt; the hosted guard every_real_data_pin_names_an_ignored_test_in_its_lane keeps every title a real #[ignore]d fn in its lane) | game data on disk; BYROREDUX_REQUIRE_GAME_DATA defaults to 1 so an absent install is a failure. Nothing else runs these — the repo has no registered runner (#6182) — so run it when the diff touches a pinned area and report pass / fail / not-run (never fold not-run into a pass); the plugin pins load one master per process (up to ~20 GB, plugin_ignored_tests_oom) | | Playable slice p0-door-interaction, p1-character-traversal, p2-melee-core (Skyrim also gates the per-kind combat_sound_impact / combat_sound_death_voice counters >= 1, not just the shared one-shot total, #5148, 7208af67c), p3-hud, p3-player-body, p5-save-restart, p5-f5-f9-quicksave, p5-door-transition, p5-soak, w1-water-traversal (docs/smoke-tests/<gate>.sh [game]); fixed-fixture p4-quest-route + p5-quest-persistence (Skyrim MS01; its Story Manager leg is the boot-CLOC TG06 start surviving F5/F9, since WIGreeting declines per #5380, d9995cfc6), p6-loading-model (Skyrim + FO4) | game-parameterised, default skyrim_se (the two P3 gates are Skyrim-only today); fixtures docs/smoke-tests/fixtures/{skyrim_se,fnv,fo3,fo4,oblivion}.env | Vulkan + game data. Exit 77 = SKIP (data absent), 2 = config error (unknown game / gate the fixture does not declare, e.g. FIXTURE_GATES fo3: p0, p5-save-restart, p5-f5-f9-quicksave, p2; fo4: p0,p2; oblivion: p0; skyrim_se: p0, p1, p2, p6 (#5674, 19b6fa93d: the default game used to run an unmeasured route, so a bare p5-soak.sh now exits 2 instead of failing at cycle 2); w1 only where a W1_* route is declared, FNV today), never a pass | | Smoke contracts | scripts/check-playable-smoke-contracts.sh + scripts/check-byro-dbg-harness-contracts.sh (both in CI job playable-smoke-contracts) | nothing: the first runs each gate with data neutralised and pins SKIP/config-error semantics (loops: p0/p1/p2/p5-save-restart/w1 per fixture; m48-menu-load + all five HUD smokes; dt1/m42/sm1/dt2/m47-3-obscript/fo4-ragdoll); the second statically pins every byro-dbg harness to bare-filename screenshot and the release BYRO_DEBUG_SERVER=1 opt-in (#5142) | | Milestone smokes | m-exteriors.sh [game\|all] [static\|boundary\|cycle\|water] (water freezes a fixture for all five LAND-carrying titles), m41-equip.sh, m41-ragdoll.sh, m43-quest-runtime.sh, m47-triggers.sh, m48-menu-load.sh, m48-{4,5,6,7}-*-hud.sh + m48-5-fnv-hud.sh, m-trees.sh, m34-day-night.sh, sm1-story-manager.sh (Skyrim, Story Manager dispatch, #5366), dt1-dialogue-layers.sh (FNV; its fo3 arg is a declared SKIP 77, never a pass, #5429, c6e48cbc6) / dt2-skyrim-forcegreet.sh (Skyrim; authored-topic, PKCU-template instance (#5511, 72e863d80) and generic arms) (dialogue layers, #5367), m42-eat-sleep.sh (FNV M42 Eat/Sleep; selection leg only: the settler's Sleep package is In-Cell for his own house, so under #5391 he idles, and the seat leg asserts no Seated instead of waiting for [m42] sleep npc=, 5f205f0ca; the seat runtime is unit-gated), m47-2-bleedout.sh (Skyrim RiverwoodSleepingGiantInn, essential-actor bleedout, M47.2; exits 77 without data but is in no contract loop, so its SKIP semantics are unpinned), m47-3-obscript.sh (FNV ObScript object/item scripts live, M47.3; SKIP 77 contract #6057, c9d7ad838), fo4-ragdoll.sh (FO4 XRGD-posed corpses + death ragdolls under BYRO_VALIDATION=1, #6070; also SKIPs 77 without the Khronos validation layer) | Vulkan + game data. Every live FNV gate mounts Update.bsa last like the profile (#5257, a4ede5fa0); the FNV P0/P1 entity floors were re-pinned to the post-withholding population (#5256, cd7b02a1d). SKIP (77) without data: m-exteriors, m34-day-night, m48-menu-load and the five HUD smokes (161ab5ae8, #4724; m48-5-fnv-hud since c47b2d72d, #5276), sm1/dt1/dt2/m42 (6a72c6042, #5427), m47-3-obscript, fo4-ragdoll. m41-equip, m41-ragdoll, m43-quest-runtime, m47-triggers and m-trees print SKIP but exit 0 without data and are in no contract loop: report them NOT RUN, never pass. m-trees.sh reads byro-dbg's (no entities) as a real zero (#5360, 811499b65) | | Renderer-correctness gates | cargo test --release -p byroredux --test cornell_rt_oracle -- --ignored --test-threads=1 (Cornell L0-L5 oracle plus cornell_parked_penumbra_converges_at_one_shadow_ray, the parked-penumbra RMSE gate from #5369's residual, 48bed0504; sampler pinned by shadow_ray_disk_samples_are_dithered_low_discrepancy), docs/smoke-tests/interior-godrays.sh (game-data-free shaft A/B), scripts/check-bench-determinism.sh (renderer-static scene-state determinism, Cornell by default), scripts/material-provider-matrix.sh 3 30 (five-game provider matrix), scripts/fsr-bench-matrix.sh 3 300 (bench-of-record; per-scene CWD = the game's Data/) | Vulkan RT device (+ game data for the matrix and bench) | | CI lanes | .github/workflows/: playable-smoke.yml and rt-correctness.yml (manual dispatch; self-hosted byroredux-game-data / byroredux-rt runners; its jobs are cornell, golden (golden_frames + the upscaler_quality FSR preset gate fsr_presets_track_the_native_reference_on_every_camera_path, #5625, 6914878f0) and material-provider-matrix), real-data-gates.yml (manual-only since fde8feec2 (#5905) — no registered byroredux-game-data runner, open #6182; NIF corpus harnesses, parser-crate suites and the closed-HIGH pins job, BYROREDUX_REQUIRE_GAME_DATA=1, --no-fail-fast since #5660 so a red binary no longer hides the ones after it); ci.yml jobs playable-smoke-contracts, shader-artifacts and vulkan-validation (lavapipe bench + check-bench-determinism.sh; since 6d5d8fa5f (#4987) the bench step must also print Selected GPU: — a deviceless clean exit fails, guarded in byroredux/src/scheduler_access_tests.rs); the build job also compiles core without parallel-scheduler (#5624) and runs a continue-on-error clippy --all-targets lane that reports without gating (#5621): a red there is /audit-tech-debt material, not a broken gate. release.yml (tag push v0.* / manual dry run) packages the Linux alpha tarball and gates nothing | hosted / self-hosted runner |

To bless a build, run the playable-slice gates you have data for and scripts/run-real-data-pins.sh (the closed-HIGH real-data pins) and report pass / SKIP / fail beside the scalar telemetry; never fold a SKIP into a pass. docs/smoke-tests/README.md is the per-script authority (specs docs/engine/playable-vertical-slice.md, docs/engine/p2-combat-fixture.md).

Invocation surface

--game <key> expands through assets/debug_profiles.toml (expand_game_profile_args, byroredux/src/boot/cli.rs) into --esm/--bsa/--textures-bsa (+ --materials-ba2 for FO4+) under <--games-root | $BYROREDUX_GAMES_ROOT | /mnt/data/SteamLibrary/steamapps/common>/<subdir>. --cell <EDID> loads an interior; --bench-frames N prints the single bench: line; --bench-mode renderer-static fixes dt = 0 and holds the authored camera (capture.sh always passes it); --bench-hold keeps the engine alive for byro-dbg on port 9876. Release builds bind the server only behind the explicit BYRO_DEBUG_SERVER=1 opt-in (63c0aee3b, #4752) — capture.sh sets it and BYRO_DEBUG_PORT on the launch and fails fast if the engine log reports the server unavailable instead of waiting out the ping loop. capture.sh also points BYROREDUX_SETTINGS_PATH at a fresh per-run file so a persisted render.upscaler cannot move the draw split (#4947); the bench: line reports the effective upscaler= / render_extent= / output_extent= — the TSVs do not store them, so report them beside the diff and treat a non-default upscaler as a harness fault.

Profile keys ([profiles.<key>]): fnv, fo3, oblivion, skyrim_se, fo4, starfield, fo76. fo76 is an asset-level profile with no sample_cells ("not a supported cell-load target"): not a runtime-audit game. .claude/audit-baselines/sf-esm/ holds Starfield ESM resolve-rate baselines for --sf-smoke, not this skill.

Baselines and cells

The table carries no metric values (they went stale within weeks): read each TSV's # regenerated: headers. The whole set was re-captured in one renderer-static run on 2026-09-30 (#5125, 2e402317e, engine 60cd7028c), after b9e961eeb's interior spawn ladder moved the authored camera in all five cells; that regen added the bench_camera_pos / bench_camera_forward rows. Later blocks: the carried entities_total moves bisected and attributed in the headers (#5131/#5133, 6fcf313e2); fo4/skyrim_se draw rows re-captured for 3c197ed8c's early-fragment admission (#5235, 8418a1791); light_count_spot added to all five, light rows re-captured at f75ac66cb (#5132, aa075ed35, 2026-10-05). No known-unattributed row remains: a new unexplained move is a finding. A pose-row mismatch means the draw rows compare two different views: attribute the pose change before reading any draw-split movement.

| Game | Cell | Baseline | Why this cell | |------|------|----------|---------------| | fnv | FreesideAtomicWrangler | yes | Primary guard; densest NPC interior (the SkinSlotPool cap case). Fallback profile sample GSDocMitchellHouse. | | fo3 | MegatonPlayerHouse | yes | Exterior-style architecture in an interior shell. | | oblivion | ICMarketDistrictTheGildedCarafe | yes | Smallest, cleanest cell; the only one with directional emitters. | | skyrim_se | WhiterunDragonsreach | yes | Per-entity hot-path stress; carries 2 corrupted control-char texture paths (AUDIT_RUNTIME_2026-06-14 RT-3). | | fo4 | InstituteBioScience | yes | BGSM-heavy + precombine CSG (M49). | | starfield | citycydoniamainlevel | none | The 2026-08-30 CRITICAL stall was closed (#3540, 0c45e779), but the residual load-time BLAS churn (~95k built) keeps the run huge, and the last two attempts (2026-09-11) were safety-aborted under memory pressure: unverified, neither known-good nor known-stalled. Needs ample free RAM; use --sf-smoke for coverage meanwhile. |

Any other (game, cell) with no baseline establishes one ("BASELINE CREATED"), not a regression guard. --game all runs each baselined game whose data dir resolves.

Parameters

--game <key|all> (required) · --cell <EDID> overrides the default cell (re-run a reported symptom) · --regen overwrites the baseline with current values (same intent as BYROREDUX_REGEN_GOLDEN=1; only after an intentional, eyeballed change).

Phase 1-2: Setup and capture

mkdir -p /tmp/audit/runtime; dedup per _audit-common.md; build release binaries. Run capture.sh from the repo root, serially; never hand-roll the launch or teardown. It launches under xvfb-run with --bench-frames N --bench-mode renderer-static --bench-hold (default 240 frames), pings byro-dbg (90 s), waits for the engine log's bench: line and then a stats answer (READY_DEADLINE_S in the script, default 180 s: cell load runs on the render thread, so the debug server answers ping long before it can answer queries), runs stats / tex.missing / mesh.cache failed / light.dump / quit, and writes <out>/<game>-<cell>.engine.log + .telem.txt (with bench_frame_max_ms).

• The bench mode is part of the baseline. renderer-static (authored camera) and system-live (live camera) cull different frustums, so the whole draw split moves with the mode, not the code (Oblivion GildedCarafe, same build: 330/20b/2c raster 22 live vs 330/78b/5c raster 132 static). bench.rs names renderer-static the regression-gate mode. Every TSV has a bench_mode row; check it first and diff nothing else on a mismatch. The harness refuses to start with BYROREDUX_FIXED_DT set (it silently selects a mode) and fails a capture whose bench: line reports another mode=. Never set BYROREDUX_FIXED_DT for this audit.
• Teardown/attribution assertions (each fails the capture): pre-flight (no byroredux process, port free; pgrep -x, never -f), real-PID kill (xvfb-run runs its command as a child, so signalling the wrapper leaves the engine holding port 9876 and the next game's numbers land under the wrong filename), and an Entities: (stats) vs entities= (bench:) cross-check within ±2 %. Any past --game all sweep on the old teardown may carry shifted telemetry. Parallel runs need a distinct BYRO_DEBUG_PORT per game for both engine and byro-dbg; otherwise serial.
• Where metrics live: wall_fps, frame_p50/p95/max_ms, draws=N/Mb/Kc, entities= and the trailing skin=L/M+S are all on the single bench: line (byroredux/src/app_events.rs), i.e. in .engine.log, not the byro-dbg stream. Do not read skin numbers from the once-per-second engine::stats line (a frozen dt never advances TotalTime).

Phase 3: Extract metrics

Keys must match the committed TSV rows (byroredux/src/bench.rs REQUIRED_METRICS). Write scalars to /tmp/audit/runtime/<game>-<cell>.current.tsv.

| Metric | Source | Direction | |--------|--------|-----------| | bench_mode | bench: mode= | exact, checked first | | bench_camera_pos / bench_camera_forward | bench: camera_pos= / camera_forward= (capture.sh copies them into the telem as capture: bench_camera_pos= / _forward=) | exact within 0.5 BU / 0.01 per component, checked second (#5125): a moved pose moves the frustum and the whole draw split, so on mismatch do NOT diff the draw rows — report the stale baseline and attribute the pose change (spawn placement, --cell resolution) first | | entities_total | bench: entities= | within ±2 % (either direction) | | tex_missing_base_color | tex.missing: count of [slot=base_color] lines | ≤ baseline (strict) | | tex_missing_all_slots | tex.missing summary count | informational: report Δ, never a finding | | mesh_cache_failed_count | mesh.cache failed summary | ≤ baseline | | light_count_point / light_count_spot / light_count_directional | light.dump: count of kind=Point / kind=Spot / kind=Directional rows (not the emitters: N tally, which includes directional ones; not the mere presence of a CellLightingRes block). #5132 — the Point→Spot reclassification (b9e961eeb) makes the split load-bearing: without the spot row, a correct reclassification reads as a point-count regression and a wrong one goes unnoticed | exact | | skin_pool_max | bench: skin=L/M+S (M) | exact | | skin_pool_overflow_attempts | skin= (S) | == 0 | | skin_pool_live | skin= (L) | advisory Δ | | bench_draws_cmds / _batches / _gpu_calls | draws=N/Mb/Kc | ≤ baseline ×1.1 | | bench_draws_raster_cmds | bench_draws_raster_cmds=R | report which side of DRAW_SORT_PARALLEL_THRESHOLD (3000) it lands | | bench_fps_p50 / _avg | wall_fps (one aggregate serves both) | advisory | | frame_p50/p95/max_ms | bench: | advisory, not stored in the TSVs: report absolute values only |

• Advisory means never a finding (bench_fps_*, frame_*_ms, skin_pool_live, tex_missing_all_slots): headless xvfb-run wall-clock jitter dominates small cells (a 14 % phantom fps "regression" with every structural metric unchanged, RT-2 #1701). For a real fps question re-run 3× or read frame_p95_ms/frame_max_ms. entities_total counts non-rendering bodies (colliders, rigs, markers) that creep benignly, so it gates only beyond ±2 %; the exact render-load contract is the draw split. skin_pool_live tracks population like entities_total; only overflow off 0 or a changed M gates.
• Draw-split invariant, check before diffing: batches <= raster_cmds <= cmds and gpu_calls <= 2 × batches (the two-sided blend split, needs_two_sided_blend_split, can record two draws per batch). A row set breaking it did not come from one run (a partial --regen): report a stale baseline and re-capture all four rows together. draw_split_rows_are_internally_ordered enforces only cmds >= batches >= gpu_calls.
• tex_missing_base_color vs _all_slots: tex.missing walks the full 26-role MaterialTextureHandles set; only the base-color bucket is comparable to old baselines.

Phase 4: Diff and severity

Compare current vs baseline. Absent baseline → copy with a # regenerated: YYYY-MM-DD header, report "BASELINE CREATED". --regen → overwrite, "BASELINE UPDATED" (never a finding). Regressed (against its direction) → one finding per metric:

• HIGH: tex_missing_base_color or mesh_cache_failed_count grew; skin_pool_overflow_attempts off 0 (an entity renders in bind pose for lack of a slot; SkinSlotPool cap #1284).
• MEDIUM: any other count moved against direction; a stale/incoherent baseline (mode mismatch, camera-pose mismatch, draw-split invariant broken).
• LOW: drift within ±5 % on a tolerance metric. A ≤-direction metric that improved passes but must be reported ("improved: tighten via --regen"): a loose ≤ baseline gate silently lets the old value return (#4420 tightened mesh-cache rows for this reason).

Phase 5: Report

docs/audits/AUDIT_RUNTIME_<TODAY>.md (header per _audit-common.md Report finalization): a per-game table (Game | Cell | Status PASS/REGRESSION/BASELINE CREATED/NOT RUN | Δ vs baseline, including the advisory Δs), the gate-matrix rows you ran with pass/SKIP/fail, then findings as ### RT-<n>: <metric> <moved> on <game> <cell> in the base format (Location = the baseline TSV row; Status NEW / Existing / Regression) plus Game, Cell, Baseline, Current. For a tex_missing_base_color bump, direct the fix at tex.missing entities (responsible REFRs), then the resolution chain byroredux/src/asset_provider/texture.rs and the single NIFAL boundary translate_material in byroredux/src/material_translate.rs; cross-check crates/nif/tests/translation_completeness.rs and run /audit-nifal. Suggest /audit-publish docs/audits/AUDIT_RUNTIME_<TODAY>.md.

Phase 6: Cleanup

rm -rf /tmp/audit/runtime (baselines untouched). Confirm nothing is left running: pgrep -x byroredux; pgrep -x byro-dbg and pkill -x if either reports (-x, not -f: -f matches your own shell's command line).

Harness and baseline integrity (audit these every run)

Paths: .claude/commands/audit-runtime/capture.sh, .claude/audit-baselines/runtime/, byroredux/src/bench.rs, byroredux/tests/golden_frames.rs, docs/smoke-tests/, .github/real-data-pins.txt, scripts/run-real-data-pins.sh, scripts/check-playable-smoke-contracts.sh, scripts/check-byro-dbg-harness-contracts.sh First step: git log --since=<last AUDIT_RUNTIME date> -- <Paths>; run capture.sh --self-test and the schema tests above.

• Each TSV's newest # regenerated: block says why its rows moved and every row came from one capture (partial regens are how the draw-split invariant broke). A baseline moved without a code reason is a finding; committing a TSV diff in the same commit as the engine change is the rule (.claude/audit-baselines/runtime/README.md).
• The skill's metric list, REQUIRED_METRICS and the TSV keys agree; docs that quote baseline values or the pre-split .claude/commands/audit-runtime.md path (#4771 fixed the baselines README's) are doc rot.
• scripts/check-playable-smoke-contracts.sh builds its data neutralisation from each fixture's FIXTURE_DATA_ENV and self-checks that every data variable the harness references is neutralised (e04fa1ef6, #5118; the old hand list missed BYROREDUX_OBLIVION_DATA). It also pins fixture-specific values (e.g. the FNV frozen NPC ref/base pair, re-pinned in d1ec1c57f): a fixture retarget without a contract update turns the CI job red.
• Data-gated tests elsewhere (crates/nif corpus harnesses, crates/menuxml/tests, crates/ui/tests) return early without data; only real-data-gates.yml and its local twin scripts/run-real-data-pins.sh (both with BYROREDUX_REQUIRE_GAME_DATA=1) make an absent corpus a failure.
• Do not launch a windowed engine while the user's own instance runs (feedback_no_parallel_engine_launch); the plugin --ignored tests can spike 20+ GB (plugin_ignored_tests_oom): do not run them alongside a Starfield capture.

References

Symptom record docs/audits/FALLOUT_SYMPTOMS_2026-05-26.md; smoke pattern docs/smoke-tests/README.md; golden-frame precedent byroredux/tests/golden_frames.rs; import-side sibling crates/nif/tests/translation_completeness.rs; SkinSlotPool cap #1284; draw-split #1258.