description: "Audit test-suite health — coverage gaps, weak assertions, flaky/stale tests" argument-hint: "[--limit <N>]"

Regression / Test-Hygiene Audit

Audit the test suite itself: where coverage is missing on code that can break, where tests assert too little to catch a regression, and where tests are stale, flaky, or testing the wrong thing. The output is a prioritized list of tests to add or strengthen — the safety net for every other audit's fixes.

Shared protocol: .claude/commands/_audit-common.md. Severity: .claude/commands/_audit-severity.md — a coverage gap on working code is LOW, but a gap on a path that has had bugs or that emits NES register data is MEDIUM (the blast radius is a silently-broken ROM).

A recent sprint closed ~100 issues, several against the test suite itself (REG-01 through REG-11 below, from AUDIT_REGRESSION_2026-06-28.md / -06-29.md). All are now fixed — don't re-report them — but re-verify status on every run: a fix landing doesn't guarantee it stays landed, and the skip-masking that REG-10/REG-11 removed (CC65-gated suites reporting real compile failures as "CC65 may not be installed" skips) is exactly the failure shape this audit exists to catch.

Parameters

--limit <N> — cap findings (report the top N by risk). Default: no cap.

Step 1: Inventory

ls tests/                                   # 52 test_*.py + conftest.py
python -m pytest -q --collect-only | tail -5
python -m pytest --cov=. --cov-report=term-missing -q   # if pytest-cov present

# The CC65-gated suites now use a real gate: `conftest.py:pytest_runtest_setup`
# skips @requires_cc65 only when ca65/ld65 are genuinely absent (via `shutil.which`),
# so with the toolchain present they compile real ROMs and assert unconditionally.
# Run them directly to confirm they PASS (not skip), don't just trust a green summary:
python -m pytest tests/test_ca65_export.py::TestCA65CompilationIntegration -v
python -m pytest tests/test_rom_validation_integration.py -v

Cross-reference tests/ against _audit-common.md § Project Layout: every subsystem should have at least one test module. Map test file → subsystem it covers.

Step 2: Coverage-Gap Dimensions

Dimension 1: Untested subsystems / modules

A source module with no corresponding test, or only an import-smoke test. Weight by risk: the NES-register path (nes/emulator_core.py, nes/pitch_table.py, nes/envelope_processor.py), the exporters, the mappers, and the compiler are the high-blast-radius gaps. A module with --cov < ~50% on a pipeline path is a finding.

Four modules previously sat at 36-71% coverage with no dedicated test module, plus one front-end (arranger/) with zero test references at all — all five closed, keep verifying the coverage didn't regress rather than re-flagging from scratch:

tracker/track_mapper.py — the legacy multi-track branch of assign_tracks_to_nes_channels (melody→pulse1, harmony→pulse2 w/ arpeggio, bass→triangle, drum-named→noise) was wholly unverified (REG-08/#48). Fixed by tests/test_track_mapper.py (synthetic 4-track + real-MIDI regression case); currently 71% cover.
compiler/cc65_wrapper.py — missing-tool detection (shutil.which → None), nonzero version-probe exit, stderr surfacing, and the full build() pipeline had no direct coverage (REG-09/#49). Fixed by tests/test_cc65_wrapper.py; currently 86%.
tracker/pattern_detector_parallel.py — the ProcessPoolExecutor map/merge, the fallback to EnhancedPatternDetector, and worker-count invariance were untested (REG-06/#46). Fixed by tests/test_pattern_detector_parallel.py; currently 78% (up from 36%).
mappers/factory.py / mappers/base.py — MapperFactory auto-select and each mapper's header/linker/capacity output were only exercised indirectly via test_ca65_export.py (REG-07/#47). Fixed by tests/test_mappers.py; factory.py now 74%, base.py 59% (residual gap: base.py's abstract-method stubs and a couple of capacity edge cases — worth a look, but no longer a blank gap).
arranger/ (role_analyzer.py, voice_allocator.py, gm_instruments.py, pipeline_integration.py) — the entire --arranger front-end had zero test references (REG-04/#44). Fixed by tests/test_arranger.py (role tagging, arpeggiation cadence, triangle no-duty invariant, frames-contract parity with process_all_tracks), plus tests/test_arranger_drum_detection.py, tests/test_arranger_frame_contract.py, and tests/test_voice_allocator.py (DPCM routing, noise-period routing). --arranger now also gets a real ROM-compile round trip (tests/test_e2e_pipeline.py::TestEndToEndPipeline::test_full_pipeline_arranger_mode, #129) — see Dimension 3.

Dimension 2: Weak assertions

Tests that run code but assert almost nothing — assert result is not None, assert len(x) > 0 where the values matter. The NES-output tests especially must assert exact register bytes / timer values, not just shape. Flag tests that would pass even if the music came out wrong.

Closed example, keep as the bar to hold other exporter tests to: several test_ca65_export.py / test_midi_parser_integration.py tests asserted only that a section/substring was present (header magic, "PATTERNS" in content) — would pass even with every note/timer/volume byte wrong (REG-05/#45). Fixed by a golden-bytes test class that parses test_midi/simple_loop.mid through the real pipeline and asserts the exact pulse1_sequence macro-bytecode stream and the first ntsc_period_low table bytes. Still worth a periodic grep for assertIn('PATTERNS' / bare-existence checks elsewhere in tests/test_ca65_export.py, tests/test_exporter_integration.py, and tests/test_famistudio_export.py — the golden-bytes fix only covers the one path it targeted.

Dimension 3: Round-trip / end-to-end gaps

The properties that unit tests miss: parse→...→ROM produces a valid ROM (tests/test_e2e_pipeline.py is the anchor); pattern compress→decompress equals the original; a generated .nes passes debug/check_rom.py. Missing round-trip coverage on compression is MEDIUM (it guards a CRITICAL failure mode).

Fixed: the 7-method TestCA65CompilationIntegration class in tests/test_ca65_export.py was RED against real ca65/ld65 — the shipped nes/audio_engine.asm had a relative branch (bcc @is_note) whose target was 130 bytes away, outside the 6502 ±127 range (REG-01/#39). Fixed by restructuring the dispatch (nes/audio_engine.asm:204-213, bcs+jmp instead of the out-of-range bcc) — confirmed: all 9 tests in TestCA65CompilationIntegration pass with the toolchain present. Re-verify this stays green; a relative-branch regression here is silent until an assembler is actually invoked.
Fixed (REG-10/#128): tests/test_rom_validation_integration.py is the designated "compile a real ROM and validate its bytes" gate. It previously had 5 of 9 tests SKIP even with ca65/ld65 present, because the hand-written music.asm fixture defined init_music:/update_music: as bare labels with no .export/.global, so ld65 failed with Unresolved external 'init_music' and an except Exception → pytest.skip masked it as "CC65 may not be installed". Fixed: the fixture now lives in tests/conftest.py's minimal_music_asm and emits .export init_music, update_music (matching the real exporter); the compile tests are gated with @pytest.mark.requires_cc65 (a real shutil.which gate in tests/conftest.py:pytest_runtest_setup) and each asserts compile_rom(...) unconditionally, and test_shared_music_asm_fixture_is_linkable pins the .export. Re-verify these PASS (not skip) with the toolchain present — a re-introduced except → skip here is exactly what this dimension catches.
Fixed (REG-15/#299): the two compile-failure (negative-path) tests in the same file (test_compilation_with_invalid_assembly, test_compilation_failure_without_rom_output) kept the except → pytest.skip("CC65 not installed") masking after REG-10 removed it from the compile-success tests, compounded by pass-either-way bodies. Now both are @requires_cc65 and assert compile_rom(...) is False plus no partial ROM. compile_rom() catches all failures and returns False (never raises), so any surviving except → skip in this file is dead masking — re-flag it.
Fixed (REG-11/#129): the anchor test_full_pipeline_midi_to_validated_rom (tests/test_e2e_pipeline.py) can now fail on a broken pipeline. The old try/except → pytest.skip, args.skip_validation = True, and if rom_path.exists(): guards are gone: the shared _run_pipeline helper sets skip_validation = False (validation ON) and _assert_valid_rom asserts unconditionally, all under a @pytest.mark.requires_cc65 gate. It also now exercises --arranger (test_full_pipeline_arranger_mode) and --no-patterns (test_full_pipeline_no_patterns_direct_export) through a real ca65/ld65 compile — closing the Dimension 1 arranger note above. Re-verify these assertions stay unconditional.

Dimension 4: Stale / wrong-target tests

Tests referencing renamed symbols, skipped/xfail tests with no tracking issue, tests that assert old (now-incorrect) behavior, or tests pinned to checked-in artifact files that have since changed. Confirm against current code.

Two closed examples worth knowing so they aren't re-flagged:

Four test classes in test_audio_fixes.py were @unittest.skip'd with "Obsolete: Assembly generation changed to MMC3 Macro Bytecode" and no tracking issue (REG-03/#42). All four actually targeted export_direct_frames (the direct/table-based export path, never removed) and still matched every assertion — the skip reason was stale, not the tests. Fixed: un-skipped; the one genuine failure (test_triangle_is_octave_above_pulse_period, a harmless pulse/16 vs triangle/32 integer-quantization drift) had its delta loosened from 0.05 to 0.08 to cover the observed max (~0.071) without masking a real regression.
verify_ca65_assembly() in tests/test_midi_parser_integration.py and the e2e assembly check asserted the old standalone .segment "HEADER" format the exporter no longer emits in macro-bytecode mode (REG-02/#40). Fixed (commit 184d3c9): verify_ca65_assembly (tests/test_midi_parser_integration.py:41-73) now branches on which export mode produced the content ("MMC3 Macro Bytecode" marker → asserts .segment "CODE_8000"/"BANK_00"/pulse1_sequence:/etc.; "Pattern Compressed" or standalone direct-frame → asserts the legacy HEADER/CODE/VECTORS sections, which are still correct for those modes). Note tests/test_e2e_pipeline.py:235's .segment "HEADER" assertion is not stale — it checks main.asm from NESProjectBuilder, a file the project builder (not the CA65 exporter) still always emits with a HEADER segment; don't conflate the two files when re-auditing this.

Recently-fixed stale-artifact case worth knowing: the hand-written music.asm fixture behind the ROM-validation tests (REG-10 above) had drifted from what the real exporter emits (missing the .export init_music, update_music directives) — the same failure shape this dimension exists to catch. Now fixed by moving it to tests/conftest.py's minimal_music_asm with the .export directives; don't re-flag it.

Dimension 5: Determinism / flakiness

Tests depending on multiprocessing scheduling (ParallelPatternDetector), dict/set ordering, wall-clock timing, or filesystem temp paths without isolation. These pass locally and fail in CI. Flag any test whose outcome can vary run-to-run.

Closed example: ParallelPatternDetector's determinism guarantee was not actually held — _select_best_patterns sorted candidates by score alone, so equal-score candidates resolved by ProcessPoolExecutor.as_completed arrival order, making which non-overlapping pattern won depend on host core count (REG-06/#46, same fix as Dimension 1). Fixed: a deterministic (start, length) tie-break was added, and tests/test_pattern_detector_parallel.py now pins identical patterns/references/ compression_ratio across max_workers=1/2/4 plus the pool-failure fallback. Re-run this test file specifically after any change to tracker/pattern_detector_parallel.py's scoring/selection — it's the one place a reintroduced non-determinism would surface.

Dimension 6: Fixture & isolation hygiene

Tests that write into the repo root instead of a temp dir, leak state between tests, or depend on a prior test having run. Check tests/conftest.py for shared fixtures (temp_dir, project_dir, minimal_midi_file, minimal_music_asm, valid_rom_file, and synthetic ROM-corruption fixtures) and whether they're used consistently — most compiler/ROM tests do use them. The music.asm fixture that had drifted from reality (missing .export directives) is now the shared minimal_music_asm fixture in tests/conftest.py, updated to emit .export init_music, update_music (see Dimension 3/4, REG-10) — re-verify the shared fixture is used consistently rather than re-hand-rolled per test.

Step 3: For each gap, specify the test

Don't just say "needs a test" — name the module, the property to assert, and the concrete input (a test_midi/ sample or a crafted JSON). A finding the /fix-issue pipeline can act on directly.

Output

Write to: docs/audits/AUDIT_REGRESSION_<TODAY>.md (YYYY-MM-DD). Structure:

Coverage map — subsystem → test module(s) → rough coverage / gap.
Findings — base format + Dimension; each names the test to add/strengthen.
Prioritized backlog — the top tests to write first, by blast radius.

Then suggest:

/audit-publish docs/audits/AUDIT_REGRESSION_<TODAY>.md

Test-Suite Health Audit

Recommended for