Automated summary

Guides the authoring of `fixtures/*.json` files consumed by `craw test` and the setup of a `craw eval --baseline` gate for a Definition.

Strengths

Spells out the exact fixture shape (`inputs` keyed by declared Parameters, optional `expected`) with a concrete example.
Enforces mock-by-default execution with all randomness routed through `--seed`, keeping suites deterministic and free.
Explains the eval gate: per-metric comparison within `--tolerance`, delta reporting, and the cost band.
Calls out secrets hygiene: credentials referenced by name in `.env.example`, never inlined in fixtures.

Limitations

Tightly coupled to the `craw` CLI and tooling; not transferable outside that ecosystem.
Does not document the metric output format or the exact syntax of baseline references.
Assumes familiarity with craw Definitions, Parameters and Outputs, which are not defined here.

When to use it

Load when adding test input sets or an eval gate to an existing craw Definition.

When not to use it

Avoid when writing tests for another framework or when no craw project exists.

Security analysis

Safe

Quality score88/100

The skill uses Bash only to run legitimate craw test and eval commands for fixture authoring and baseline comparison. It explicitly avoids live model execution by default, forbids inline secrets, and notes consequential actions fire only in eval mode. No destructive, exfiltrating, or obfuscated operations are present.

No concerns found

How this analysis is made

Examples

Add fixtures for a new Definition

Create fixtures/ for the triage-ticket definition: one fixture with a valid project and ticket_body, one with an empty ticket_body, and one asserting an expected Output. Then run craw test against them.

Set up an eval baseline gate

Wire up a craw eval --baseline gate for definitions/classify-ticket with a 0.05 tolerance, and show me the per-metric deltas and total_usd cost band.

Debug non-deterministic runs

Our fixtures give different results on each run. Check whether they're hitting a live model and make them mock-by-default with all randomness carried in --seed.

name: crawfish-authoring-fixtures-evals description: > Author fixtures/ for craw test and gate on a baseline with craw eval — deterministic, mock-by-default, randomness carried in --seed. Load when adding test inputs or an eval gate to a Definition. user-invocable: false allowed-tools: Read, Grep, Bash

Authoring `fixtures/` & evals

Name: Authoring Fixtures & Evals
Author: Skills Guide Bot

Derived from docs/specs/craw-code/authoring/fixtures-evals.md. Golden: demo/craw-code-golden/fixtures/.

fixtures/ feeds craw test (eval-as-test): each fixtures/*.json is one input set the Definition runs against. Evals gate against a saved baseline (craw eval --baseline).

The fixture shape

{"inputs": {"project": "acme", "ticket_body": "the login button does nothing"}}

A fixture is {"inputs": {...}, "expected": <optional>}. inputs maps each declared Parameter name to a value. With no expected, a fixture passes when the run executes cleanly; with expected, the Output value must match.

craw test definitions/<name> --fixtures fixtures

Determinism: mock-by-default

Fixtures run on the MockRuntime / the command profile's recorded transport, never a live model, so the suite is deterministic and free. All randomness is carried in --seed; the same seed replays byte-identically. Promote to --live only deliberately and always under --budget.

Spine rule (determinism-mock-default): Fixtures run mock-by-default; randomness is carried in --seed.

The eval baseline gate

craw eval --baseline <ref> compares per-metric scores against a saved baseline within a --tolerance, and reports the deltas plus the cost band (total_usd / expected_usd / worst_case_usd). Sinks fire only in eval mode; an optimize/search run never fires a consequential action. Read retryable on a craw.error.v1 to decide retry-vs-stop — a security rejection is retryable:false.

Fixtures never carry secrets. Reference credentials by name in .env.example, never inline.

From the file to a first result

The download contains exactly the content displayed on this page. This copy may differ from the latest version in the source repository.

View the source and supporting files

This download contains SKILL.md only. Any scripts, references, templates and dependencies are not included. Get the required files from the source, preserving their directory structure, before continuing.

1. Download the file

Download SKILL.md

From the root of a test project, save the file as .skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md, or use the command below. This folder is for inspecting the skill before installation.

Command for macOS/Linux or WSL: requires a POSIX shell and curl. On Windows without WSL, use the download button and create the folder manually.

mkdir -p '.skill-downloads/authoring-fixtures-evals-mrwakbph' && curl --fail --show-error --location 'https://skills-guides.com/api/skills/authoring-fixtures-evals-mrwakbph/download' --output '.skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md.part' && mv '.skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md.part' '.skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md'

2. Check prerequisites

Read SKILL.md before using it. If it references scripts or external services, inspect them and prepare the required access in your test environment.

Tools declared by the author (check availability in your agent): Read, Grep, Bash

No recognized prerequisites section in this file. This does not mean it is self-contained: libraries, tools and access requirements may be mentioned elsewhere in the content or source repository.

The platforms on this page are catalog indications. The paths below follow tool documentation; they do not prove this skill has been run in those tools. No minimum tool version is confirmed for this listing.

3. Load the skill

After inspection, copy the complete folder into your project using one of the commands below, then open a new tool session. The command stops if a folder with the same name already exists: compare the files before replacing it.

Claude Code

test -f '.skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md' && test -s '.skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md' && mkdir -p '.claude/skills' && test ! -e '.claude/skills/crawfish-authoring-fixtures-evals' && cp -R '.skill-downloads/authoring-fixtures-evals-mrwakbph' '.claude/skills/crawfish-authoring-fixtures-evals'

OpenSkills

OpenSkills must already be installed. The local install/read procedure is checked with OpenSkills 1.5.0, which requires Node.js ≥ 20.6.0. Catalog slugs are not resolved automatically.

Official OpenSkills installation

openskills install './.skill-downloads/authoring-fixtures-evals-mrwakbph'

openskills read 'authoring-fixtures-evals-mrwakbph'

Reading should display the installed file content and its base directory. It does not run the skill or install its dependencies.

For an agent that reads AGENTS.md, then sync the index. This command changes AGENTS.md: review the diff before using it.

openskills sync

Manual method or another tool: attach SKILL.md to a conversation, or open it in an agent that can read files. Include any required supporting resources. No universal folder is recognized by every tool.

4. Try a first task

Use a short, synthetic input: an API endpoint, a small document or an interface brief. In a new conversation, attach the file or give the agent its path, then adapt this request:

Read .skill-downloads/authoring-fixtures-evals-mrwakbph/SKILL.md and apply its instructions to this example: [describe a test input]. Identify any missing files, dependencies or access. Produce the result requested by the skill without inventing data or running unreviewed scripts.

Expected result: output matching the skill instructions, or a precise list of missing prerequisites. A read confirmation alone does not validate a first use. If an external resource is unavailable, that path remains untested.