---
description: "Audit pattern detection and compression — round-trip integrity, schema, parallelism"
argument-hint: "[--focus <dims>]"
---

# Pattern Detection & Compression Audit

Audit the pattern-detection and compression subsystem — the stage that finds repeating
note/volume runs, deduplicates them into a `patterns`/`references` table, and reports a
`compression_ratio`. The headline risk is **round-trip integrity**: the system claims
compression is lossless, so decompressing `patterns` + `references` MUST reproduce the exact
original frame/event stream. Any divergence is CRITICAL (claims lossless, isn't).

Shared protocol: `.claude/commands/_audit-common.md` — read it for the project layout, the
dedup procedure, and especially the **detect-patterns inter-stage contract** (the dict keys
`patterns` / `references` / `stats` / `variations`, `--no-patterns` stub semantics, and the
Multiprocessing rule for `tracker/pattern_detector_parallel.py`). Severity floors:
`.claude/commands/_audit-severity.md` — note the special rows: round-trip mismatch = CRITICAL,
multiprocessing crash without the documented fallback firing = HIGH, inaccurate compression
stats = MEDIUM. Do not restate either file; apply them.

## Parameters (from $ARGUMENTS)
- `--focus <dims>` — comma-separated dimension numbers (e.g. `--focus 1,3`). Default: all.

## Extra Per-Finding Field
- **Dimension**: one of the dimensions below.

## Dimensions

### Dimension 1: Round-Trip Integrity (compress → decompress)
THE headline check. Compression must be lossless. Do not take the docstrings' word for it —
**actually round-trip a sample**. Two distinct paths exist and both must be verified:

- The pattern-dedup path: `tracker/pattern_detector.py` `PatternCompressor.compress_patterns`
  (`:799-829`) produces `compressed_data` + `pattern_refs`. (The former `PatternExporter`/
  `exporter.py` FamiTracker reconstruction path was deleted as dead + frame-space-buggy, #101
  — the `references` are analysis-only and no exporter consumes them, #4, both closed.) Build
  a small synthetic event list with known repeats, run `EnhancedPatternDetector.detect_patterns`
  (`tracker/pattern_detector.py:396`), and confirm every detected pattern's occurrences are
  exact repeats of its events. PAT-01 (#168) is now **fixed & closed**: the sequential detector
  persists an **exact-only** `positions`/`references` list — `positions` is
  `sorted(set(exact_matches))` (`pattern_detector.py:255`, `:284`), while a separate
  `occupied_positions` (exact + variation positions) is used *only* to block a different
  candidate from claiming the same frames during non-overlap selection (`:303-304`, `:315`) and
  is never persisted. So every stored reference now points at a true exact repeat of the
  pattern's `events` (variation positions no longer leak in). Any note/volume mismatch found by
  your own round-trip is still CRITICAL; still flag any change that starts having an exporter
  reconstruct from `references` (#4).
- The RLE/delta path (exporter/compression.py's `CompressionEngine`) was **removed as dead code
  (#302/EXP-09)**: no exporter or `main.py` ever called it — the CA65 paths do their own inline
  compression (macro-bytecode serializer / direct frame tables). Do not audit it; if you see a
  reference to `compression.py`/`CompressionEngine` in a stale report, it no longer exists.
- Existing coverage to check (and distrust if thin): `tests/test_pattern_integration.py`. A
  round-trip with no asserted frame-by-frame equality is not coverage. **Fixed (#311/PAT-10)**:
  `test_pattern_positions_format` now asserts `sequence[pos:pos+length] == stored events` for
  every position, not just that positions are ints, and a sibling test
  (`test_pattern_positions_exclude_transposed_decoy`) adds a transposed-decoy fixture so the
  assertion has teeth against the #168/#170 defect class — mirrors
  `tests/test_patterns.py::TestPositionsAreExactOnly`, which already covered this via
  `EnhancedPatternDetector`'s `references`; the integration-test file now covers the bare
  `PatternDetector` class directly too.

### Dimension 2: `pattern_result` Schema Integrity
The detect-patterns contract is a dict with `patterns`, `references`, `stats`, `variations`
(see `_audit-common.md`). Verify every producer returns all four with consistent shapes:
- `ParallelPatternDetector.detect_patterns` (`tracker/pattern_detector_parallel.py:31`),
  its `_empty_result` (`pattern_detector_parallel.py:274-283`), and the no-events early return
  (`pattern_detector_parallel.py:35-43`).
- `EnhancedPatternDetector.detect_patterns` (`tracker/pattern_detector.py:396`) and its
  no-events return (`pattern_detector.py:398-406`).
  (The dead `ThreadedPatternDetector` was removed entirely along with #102's fix — the class and
  file no longer exist. `grep -rn ThreadedPatternDetector` now only matches a doc comment
  (`tracker/pattern_detector.py:12`) and a regression test asserting it's gone
  (`tests/test_patterns.py:1184-1190`). This resolved P-08/#105's race/shape concerns by
  construction; #105 and #102 are both **closed**.)
- The `--no-patterns` stub built inline in `run_full_pipeline` (`main.py:865-881`): confirmed
  **fixed** (#104) — its `stats` keys (`original_size`, `compressed_size`, `compression_ratio`,
  `unique_patterns`, plus `total_events`/`patterned_events`/`coverage_ratio`) now match both
  detectors' schema exactly, and it reports `0` (not `1.0`) since direct export applies no
  compression (#17). The former top-level-`variations`-omission drift is also **fixed & closed**
  (#258/PAT-09): the stub now emits `'variations': {}` (`main.py:880`), matching the 4-key
  envelope both detectors return. PAT-06 (#172) is likewise **fixed & closed**: the two
  `_get_variation_summary` implementations now emit the SAME inner shape
  (`pattern_detector.py:502-520` and `pattern_detector_parallel.py:256-272` both return
  `{variation_count, exact_match_count, transposition_range, volume_range}`); the parallel path
  reports `variation_count = 0` and neutral `(0, 0)` ranges by design (exact repeats only), not
  a divergent key set. Confirm both shapes stay unified before trusting any consumer of
  `variations`.

### Dimension 3: Reference Offsets & Length Correctness
A wrong offset corrupts playback, not just space. Trace position→frame mapping end to end:
- `PatternCompressor.compress_patterns` (`pattern_detector.py:799-829`) fills `pattern_refs`
  with pattern **start positions** (sequence indices) — for the sequential detector these are
  now **exact-only** (PAT-01, #168, closed — see Dimension 1; the non-exact variation positions
  live in the non-persisted `occupied_positions` and never reach `references`). These are
  analysis-only — no exporter consumes them (#4, closed), and the one path that did
  (`PatternExporter`) was deleted as frame-space-buggy (#101, closed). Still confirm
  `len(compressed_data[pattern_id]['events'])` equals the pattern `length` — a mismatch would
  misalign any future consumer.
- The old "occurrence-index passed as pattern offset" bug (#102-adjacent, formerly described
  here as an `enumerate(positions)` loop in `main.py`) **no longer exists** — that whole code
  path was deleted. `run_full_pipeline` now calls `CA65Exporter.export_tables_with_patterns`
  with a literal empty `{}` for `references` (`main.py:917-924`), and the step-by-step
  `run_export` path (`main.py:519-560`) forwards whatever `references` a `detect-patterns` JSON
  file contains — but `export_tables_with_patterns` itself documents that the `references`
  argument is **not consumed** (`exporter/exporter_ca65.py:962-971`, #4, closed): `patterns`
  truthiness is only a boolean switch between direct frame export and the MMC3
  macro-bytecode serializer. Confirm this contract still holds before trusting any future PR
  that claims to "wire up" pattern references into the exporter.
- Overlap accounting: `_select_best_patterns` (`pattern_detector_parallel.py:216-254`) and the
  selection loop in `pattern_detector.py:296-315` mark `range(pos, pos+length)` used. Confirm
  no two retained patterns can claim the same frame (double-write) and no frame between the
  last pattern end and the song end is silently dropped. (PAT-04, #170, is now closed — the
  sequential detector's exact-match scan no longer self-overlaps; see Dimension 8.)

### Dimension 4: Compression-Ratio & Stats Accuracy
Cosmetic but must not mislead (MEDIUM floor when wrong).
`PatternCompressor.calculate_compression_stats` (`tracker/pattern_detector.py:843-891`)
computes `original_size = Σ len(events)*len(positions)` and
`compressed_size = Σ len(events)`, over detected patterns only. The former "x multiplier vs %
reduction" inconsistency is **fixed** (#17, closed): every print site now uses the same "%
reduction" convention — `main.py:655` (subcommand), `main.py:1046` (ROM success banner), and the
`--no-patterns` stub's `0` (`main.py:871`) is coherent with that convention (0% reduction, not
a misleading `1.0`).
PAT-03 (#169) is now **fixed & closed**: `calculate_compression_stats` takes a `total_events`
arg (`pattern_detector.py:843-844`) and reports a separate `coverage_ratio =
patterned_events / total_events` (`:879-881`), and both banners now print the dedup ratio
explicitly labeled as within the patterned subset PLUS a distinct "Pattern coverage" line
(subcommand `main.py:655`/`:657`, ROM banner `main.py:1046`/`:1048`). The dedup ratio still has
no relationship to emitted ROM bytes (actual size reduction comes from macro/instrument dedup
in the bytecode serializer, #4) — confirm the banner keeps the two numbers distinct and that
callers keep passing `total_events` (the analyzed/sampled count, #257/PAT-08), since omitting it
silently reads `coverage_ratio` as 0. The old PAT-01/PAT-04 inflation of `original_size` is also
gone now that `positions` is exact-only and the match scan no longer self-overlaps.

### Dimension 5: Parallel vs Sequential Equivalence + Documented Fallback
The two detectors' scoring is now **shared** (#103, closed) — verify the fix is complete, not
just that it merged the formulas:
- **Scoring**: both `ParallelPatternDetector`'s `_collect_length_candidates`
  (`pattern_detector_parallel.py:354-355`) and `EnhancedPatternDetector.detect_patterns`
  (`pattern_detector.py:238`, `:272`) now call the same module-level `score_pattern`
  (`pattern_detector.py:41`), the parallel path always passing `variation_count=0`. This is
  documented as by-design, not a bug: the parallel path's O(n) hash-grouping (#114) finds exact
  repeats only and structurally cannot detect transposed/volume-scaled variations, so it stays
  variation-free. Confirm this is still true and pinned by a test
  (`tests/test_patterns.py`) rather than re-diverging silently.
- PAT-05 (#171) is now **fixed & closed** — but the fix was a *doc correction*, not a behavior
  change: the two paths can still select *structurally different* pattern sets on identical
  input (the parallel path emits one candidate per distinct window, anchored at its first
  occurrence, and `_select_best_patterns` rejects the candidate wholesale if any position
  overlaps an already-selected pattern, so a winning pattern overlapping only a window's first
  occurrence loses that window's later occurrences entirely, while the sequential per-start scan
  recovers them via a later-anchored candidate). The `_collect_length_candidates` docstring now
  *correctly documents this non-equivalence* (`pattern_detector_parallel.py:311-320`) instead of
  overclaiming equivalence. Confirm the docstring still owns this caveat and hasn't regressed to
  an equivalence claim.
- **Fallback firing**: `run_full_pipeline` wraps `ParallelPatternDetector` in `try/except` and
  falls back to `EnhancedPatternDetector` (`main.py:827-853`), and inside the parallel detector
  `_detect_patterns_parallel` catches a pool-wide failure and calls `_detect_patterns_serial`
  (`pattern_detector_parallel.py:182-185`). Verify the inner serial fallback returns the raw
  `patterns` dict via `_select_best_patterns` (`pattern_detector_parallel.py:214`) — NOT the
  full `{patterns,references,stats,variations}` envelope — and that its caller (`detect_patterns`,
  `pattern_detector_parallel.py:76-94`) still wraps it via the compressor. If the serial
  fallback's return shape ever bypasses compression/stats, the fallback "fires" but yields a
  malformed result (HIGH).
- P-09 (#106) is now **fixed & closed**: the per-chunk `except` inside the
  *whole-pool-succeeded* path (`pattern_detector_parallel.py:164-178`) no longer silently drops
  a failed chunk's candidates — it recovers that length in-process via `_collect_length_candidates`,
  and only a length that ALSO fails the serial retry is recorded in `failed_lengths` and surfaced
  by a durable end-of-run warning (`pattern_detector_parallel.py:190-192`), not just a transient
  `pbar.write`. Confirm both the in-process retry and the persistent warning are still present.
- The outer `main.py` fallback re-trims events to `max_events` (default `DETECTOR_MAX_EVENTS` =
  1000, a named constant — not an ad-hoc "2000" as in older notes) via
  `sample_events_for_detection` (`main.py:844`) — confirm this matches the sequential detector's
  own internal cap (`pattern_detector.py:204-207`) so the warning reports the count actually
  retained, not a larger figure the detector would silently re-cut (#100, closed).

### Dimension 6: Multiprocessing Safety (pickle-ability, shared state, pool hygiene)
This dimension changed substantially since the O(n) hash-grouping rewrite (#114, closed) — the
old "entire sequence embedded in every chunk" design is gone:
- `_detect_patterns_parallel` (`pattern_detector_parallel.py:106-197`) now builds one **tiny**
  work chunk per pattern length — just `{'pattern_length': length}`
  (`pattern_detector_parallel.py:117-121`) — and ships the (potentially large) `sequence` and
  `valid_events` to each worker **once** via the `ProcessPoolExecutor(initializer=
  _init_pattern_worker, initargs=(sequence, valid_events))` call (`:146-150`), stashed as
  module globals `_WORKER_SEQUENCE`/`_WORKER_EVENTS` (`:289-298`) instead of being re-pickled
  per chunk × per length. This directly fixes the old memory-blowup / pickle-cost concern —
  confirm it stays true (no code re-introduces per-chunk copies of the full sequence).
  Everything shipped through `initargs` here is a plain list of tuples/dicts — confirm nothing
  non-picklable (a `tempo_map`, a closure, a numpy array) is added to that call in future
  changes, since `EnhancedTempoMap` itself is deliberately kept out of the worker payload.
- Confirm `_detect_patterns_worker` (`pattern_detector_parallel.py:371-379`) and
  `_init_pattern_worker` (`:293-298`) are module-level (picklable/importable by the child
  process) and that `_collect_length_candidates` (`:301-368`), the actual per-length work, reads
  the globals but mutates no shared state across workers.
- The dead `ThreadedPatternDetector` (formerly a second, thread-based, shared-`patterns`-dict
  race) is confirmed **removed** (#102, closed) — `grep -rn ThreadedPatternDetector` matches
  only a doc comment in `pattern_detector.py:12` and a regression test
  (`tests/test_patterns.py:1184-1190`) asserting it stays gone. No shared mutable state remains
  to audit here.

### Dimension 7: Large-File Sampling Not Dropping Musical Content
Sampling must not silently change the song. The old "three inconsistent limits" defect is
**fixed** (#102, closed) — there are now exactly two, sharing one implementation:
- `LARGE_FILE_THRESHOLD = 10000` (`main.py:818`) only prints advice — confirm it does not
  itself drop events.
- Both detectors now call the same module-level `sample_events_for_detection`
  (`tracker/pattern_detector.py:26-38`, uniform `np.linspace` sampling — not a head cut).
  `ParallelPatternDetector.detect_patterns` (`pattern_detector_parallel.py:60`) samples to
  `MAX_PATTERN_EVENTS = 15000` (`pattern_detector.py:16`); the sequential
  `PatternDetector.detect_patterns` (`pattern_detector.py:204-207`) samples to
  `DETECTOR_MAX_EVENTS = 1000` (`pattern_detector.py:23`) because it is O(n²)-ish. Trace whether
  the sampled `valid_events` is what later feeds `references`/frame reconstruction: since
  `export_tables_with_patterns` derives all bytes from `frames` (not from the sampled detection
  sequence, #4), the exported song is NOT altered by this sampling — only pattern-detection
  *quality* is reduced. Confirm this remains true; if a future change makes export consume the
  sampled sequence instead of `frames`, that would become CRITICAL data loss.
- The old third limit (the removed `ThreadedPatternDetector`'s 2000-stride) is gone along with
  the class (#102, closed) — confirm no new ad-hoc cap is introduced without updating the
  `MAX_PATTERN_EVENTS`/`DETECTOR_MAX_EVENTS` doc comment (`pattern_detector.py:8-23`) that
  explains why exactly two caps exist and don't shadow each other.

### Dimension 8: Pattern-Length Bounds & Match Semantics
- `min_pattern_length`/`max_pattern_length` constructor defaults are `3..32` for both detectors
  (`PatternDetector.__init__`, `pattern_detector.py:83`; `ParallelPatternDetector.__init__`,
  `pattern_detector_parallel.py:17`), but all three real entry points now consistently override
  `max_pattern_length` to the same named constants — `PATTERN_MIN_LENGTH = 3` /
  `PATTERN_MAX_LENGTH = 12` (`main.py:36-37`) — used by `run_detect_patterns`
  (`main.py:622-623`), the default parallel path (`main.py:829`), and its sequential fallback
  (`main.py:837`). This resolves the old "`detect-patterns` alone uses a default max of 32"
  drift; confirm all three call sites still reference the shared constants rather than a
  hardcoded literal before trusting this. Bounds are honored via
  `range(min, min(max, len(sequence))+1)` in both detectors — confirm a `max < min` or
  `len(sequence) < min` still can't produce a garbage/negative-range loop.
- `_find_pattern_matches` (`pattern_detector.py:320-338`, sequential detector only — the
  parallel worker uses a different O(n) grouping; see below) is now **correct** (PAT-04, #170,
  **fixed & closed**): the scan for matches after the anchor starts at `start_pos + pattern_len`
  (`:330`), not `start_pos + 1`, so a self-similar run (period < pattern length) can no longer
  produce a first "match" that overlaps the anchor window. The old `exact_matches`-count
  inflation (which fed `score_pattern` and `calculate_compression_stats` too high) and the
  divergence from the parallel path are gone (e.g. 12 identical notes, length 4 → both
  `[0, 4, 8]`). The two detectors are still NOT a "mirror" of each other — different algorithms
  (O(n²) per-start scan vs O(n) hash-grouped windows, #114) — but they now agree on the
  non-overlap invariant. Compare against the parallel detector's greedy `next_free` logic in
  `_collect_length_candidates` (`pattern_detector_parallel.py:340-345`). PAT-05 (#171, closed,
  see Dimension 5) documents the residual case where the parallel path's whole-candidate
  rejection can still lose valid later occurrences the sequential scan would recover.
- **#365 (PAT-A) is CLOSED**: `score_pattern`'s `>=MIN_PATTERN_OCCURRENCES` gate counts exact +
  variation occurrences together, but only exact positions are persisted into `references`
  (`positions`, PAT-01 above) — so a candidate could clear the gate almost entirely on
  variations while storing a **single** exact position (0% compression for that window), and
  its `occupied_positions` (exact + variations) would still block a genuinely-repeating shorter
  exact pattern that overlapped it. Symptom: `detect-patterns` reporting `compression_ratio 0.0`
  on songs with obvious repeats. The sequential selection loop
  (`pattern_detector.py:305-323`) now also requires `len(candidate['positions']) >=
  MIN_PATTERN_OCCURRENCES` (`:322`, the same named constant `score_pattern` gates on) and
  `continue`s — skipping the candidate entirely, **not** marking its range used — when a
  variation-inflated candidate fails that check. This aligns the sequential detector with the
  parallel one, which already requires `>=3` exact repeats (it always passes
  `variation_count=0`). Verify-the-fix: confirm a skipped candidate's `occupied_positions` are
  never added to `used_positions`, so the shorter exact pattern it used to block is free to win;
  confirm round-trip is unaffected (only degenerate, non-compressing candidates are dropped —
  the exporter derives every byte from `frames`, not `references`, #4).
- PAT-07 (#173) is now **fixed & closed**: `PatternCompressor._hash_pattern`
  (`pattern_detector.py:831-841`) returns the exact `Tuple[Tuple[int, int], ...]` of `(note,
  volume)` events (matching an updated type hint), NOT `hash()` of it. Used as the sole dedup
  key in `pattern_hash_map` (`compress_patterns`, `pattern_detector.py:807-819`) with no
  equality check on hit, keying on the tuple itself is exact and collision-free — the old 64-bit
  `hash()` could silently merge two different event tuples' `references` and drop a definition.
  Confirm the key stays the raw tuple (not re-wrapped in `hash()`/`str()`) if this path is touched.

### Dimension 9: Loop Detection Correctness
`tracker/loop_manager.py` derives loop points from `pattern_info['positions']`/`length`.
Check `LoopManager.detect_loops` (`loop_manager.py:11-50`): `loop_start = positions[-2]` and
`loop_end = positions[-1] + length` — confirm this can't produce `end <= start` (the jump
table guards it at `loop_manager.py:98`, `149`, but a malformed loop dict elsewhere may not).
Verify `generate_jump_table` keys on `loop_info['end']` (`loop_manager.py:103`, `152`): two
loops sharing an end frame would clobber each other in the jump table. Confirm
`EnhancedLoopManager` (`loop_manager.py:115`) registers tempo state on a key
(`f"loop_{end}_{start}"`, `:130`) that matches what `generate_jump_table` reads back
(`loop_manager.py:156`) — a key-format drift yields `None` tempo_state silently. Since loop
detection consumes `pattern_info['positions']` directly, note that with PAT-01 (#168) and
PAT-04 (#170) now fixed & closed, `positions` is exact-only and non-self-overlapping, so a loop
can no longer anchor on a non-exact-repeat or a self-overlapping match — this class of
downstream corruption is resolved rather than latent. Cross-check against
`tests/test_enhanced_loop_patterns.py`.

## Skeptical Checklist
- [ ] Did you actually run a compress→decompress round-trip on a sample and diff frame-by-frame (Dim 1)? Asserting from the docstring does not count.
- [ ] Do both real producers (`ParallelPatternDetector`, `EnhancedPatternDetector`) and the `--no-patterns` stub emit the same `stats` keys AND the same top-level/inner `variations` shape — all confirmed fixed (#104, #258, #172) — with no regression?
- [ ] Does `references`/`patterns` stay a pure boolean switch into the exporter with zero consumption of the actual reference values (`exporter/exporter_ca65.py:962-971`, #4)? (PAT-01, #168, is fixed so `references` is exact-only, but the exporter still must not consume it.)
- [ ] Does the parallel detector's lack of variations + PAT-05's (#171, closed but documented) anchor-blocking loss change the song vs the sequential detector on the same input, beyond the documented by-design scoring equivalence (#103)?
- [ ] Does the inner serial fallback return the bare patterns dict or the full envelope — and does its caller re-wrap it (Dim 5)?
- [ ] Does large-file sampling feed the *sampled* stream to export (data loss) or only to detection (index alignment) — confirm `frames`, not the sampled sequence, still drives every exported byte?
- [ ] Have `_find_pattern_matches`'s self-overlap (PAT-04, #170) or `_hash_pattern`'s int-as-hash (PAT-07, #173) — both now fixed & closed — regressed since this file was last refreshed?
- [ ] For each finding: re-read the code path, then try to disprove it before including it.

## Output
Write the report to: **`docs/audits/AUDIT_PATTERNS_<TODAY>.md`** (YYYY-MM-DD). Structure:
1. **Summary** — finding counts by severity, the single most important round-trip result (lossless confirmed / mismatch found), and the 3 highest-leverage fixes.
2. **Findings** — base format from `_audit-common.md` plus the `Dimension` field. Lead with any CRITICAL round-trip or data-loss finding.

Then suggest:
```
/audit-publish docs/audits/AUDIT_PATTERNS_<TODAY>.md
```