skill.md — Claude working guide for mlperf-automations

Compact task-oriented reference. Read AGENTS.md for full technical detail.

Mental model (read this first)

mlcflow CLI  →  finds script by tags  →  resolves variations & deps  →
  customise.py:preprocess()  →  run.sh  →  customise.py:postprocess()  →
  caches new_env to ~/MLC/repos/local/cache/{uid}/

Two repos, two roles:

mlperf-automations (this repo) = content: 377 script directories + automation/ engine
mlcflow = driver CLI: installs via pip, dynamically loads the engine above

Three key files per script:

meta.yaml — identity, tags, deps, variations, env mapping (schema in automation/script/meta_schema.py)
customize.py — Python hooks: preprocess() builds the shell command; postprocess() reads results
run.sh — executes eval "${MLC_MLPERF_ENDPOINT_CMD}" and exits non-zero on failure

Task playbooks

"I need to understand what a script does"

cat script/<alias>/meta.yaml        # tags, deps, variations, input_mapping
cat script/<alias>/customize.py     # what it validates and what command it builds
cat script/<alias>/run.sh           # what actually executes
cat script/<alias>/README.md        # human summary

Key fields to read in meta.yaml: tags, deps, variations, input_mapping, new_env_keys, default_env.

"I need to add a new script"

Scaffold with mlc add script:

# Basic skeleton (copies template,generic)
mlc add script mlcommons@mlperf-automations:<alias> --tags=<tags>

# Copy nearest existing script as the template instead
mlc add script mlcommons@mlperf-automations:<alias> --tags=<tags> \
    --template_tags=app,mlperf,inference,reference

This creates script/<alias>/ with meta.yaml, customize.py, and run.sh. If multiple scripts match --template_tags, it prompts you to pick one.

Edit meta.yaml — update at minimum:
- alias, uid (already generated), tags, category
- input_mapping (CLI args → env vars)
- new_env_keys (what this script promises to export)
- deps chain
Edit customize.py — implement preprocess(i):
- Guard required env vars (return {'return':1,'error':'...'} if missing)
- Build the shell command string in env['MY_CMD']
Edit run.sh — ensure it evals the command and exits non-zero on failure.
mlc lint script --tags=<alias> — fix key order
mlct <alias> — run built-in tests
PR → main branch

"I need to add a variation"

Add to meta.yaml variations::

variations:
  my-variant:
    group: my-group        # omit group if stackable (free-standing)
    default: true          # omit if not the default
    env:
      MY_ENV_VAR: value
    deps:
      - tags: get,extra,dep   # extra dep only when this variation is active

Invoke: mlcr <tags>,_my-variant (underscore prefix on CLI).

"I need to add a conditional dependency"

deps:
  - tags: get,cuda
    enable_if_env:
      MLC_MLPERF_DEVICE: [gpu, cuda]   # only run if device is gpu or cuda
  - tags: get,rocm
    skip_if_env:
      MLC_HOST_OS_TYPE: [windows]      # skip on Windows

"I need to debug a script that fails silently"

mlcr <tags> --verbose                    # full debug output
mlc find cache --tags=<failing-dep>      # check if dep is cached
mlc rm cache --tags=<failing-dep>        # clear dep cache and re-run
mlc show cache --tags=<script-tags>      # inspect cached new_env
cat ~/MLC/repos/local/cache/*/mlc-cached-state.json | python3 -m json.tool

Common root causes:

Dep cached but stale → mlc rm cache --tags=<dep-tags>
new_env_keys missing from meta.yaml → key is silently dropped at the boundary
preprocess returned success but didn't set the expected env var → add assertion
MLC_TMP_* var expected downstream but not in new_env_keys → rename it or declare it

"I need to understand why an env var is not reaching a script"

Env propagation rules:

CLI --key=val → input_mapping → env[MAPPED_VAR]
env passes to deps unless dep has clean_env_keys that matches
Dep's new_env_keys is the only way values come back from a dep
MLC_TMP_* never cached, never passed to deps automatically
If a parent caches, only declared new_env_keys are replayed on cache hit

"I need to run the inference benchmark locally for testing"

# Quickest smoke test — resnet50, onnxruntime, CPU, 500 samples
mlcr run-mlperf,inference,_submission,_short,_r6.0-dev \
     --model=resnet50 --implementation=mlcommons-python \
     --backend=onnxruntime --device=cpu \
     --scenario=Offline --test_query_count=500 --target_qps=1 \
     --hw_name=my_machine --quiet

# Find performance (tunes target QPS before a real run)
mlcr run-mlperf,inference,_find-performance,_short,_r6.0-dev \
     --model=resnet50 --implementation=mlcommons-python \
     --backend=onnxruntime --device=cpu \
     --scenario=Offline --hw_name=my_machine --quiet

# Accuracy-only run
mlcr run-mlperf,inference,_accuracy-only,_short,_r6.0-dev \
     --model=resnet50 --implementation=mlcommons-python \
     --backend=onnxruntime --device=cpu \
     --scenario=Offline --hw_name=my_machine --quiet

# Full submission run (both modes + compliance + checker + tar)
mlcr run-mlperf,inference,_submission,_full,_r6.0-dev \
     --model=resnet50 --implementation=mlcommons-python \
     --backend=onnxruntime --device=cpu \
     --scenario=Offline --execution_mode=valid \
     --submitter=MLCommons --hw_name=my_machine --quiet

"I need to inspect or clear cache"

mlc find cache --tags=get,mlperf,endpoints  # find cache folder
mlc show cache --tags=get,mlperf,endpoints  # print new_env snapshot
mlc rm cache --tags=get,mlperf,endpoints    # delete specific cache
mlc rm cache -f                             # delete ALL caches
mlc prune cache                             # delete expired (past cache_expiration)

"I need to run in Docker"

mlcd app,mlperf,inference,endpoints,_echo-server --num_samples=50
# Rebuilds image if meta.yaml docker: section changed:
mlcd app,mlperf,inference,endpoints --docker_rebuild

Quick reference: meta.yaml field cheat-sheet

| Field | What it does | |---|---| | alias | Script directory name and lookup key | | uid | 16-hex unique ID; never change after first commit | | automation_uid | UID of the script automation type (5b4e0237da074764); the only automation type currently in this repo | | tags | Comma-separated discovery tags; must be a superset of what callers request | | category | Grouping label in docs | | default_env | Env vars set before variation env (lowest priority) | | new_env_keys | Only these keys leave the script; use * and ? wildcards | | input_mapping | CLI --key → ENV_VAR translation | | deps | Scripts to run before preprocess() | | prehook_deps | Scripts to run after preprocess(), before run.sh | | posthook_deps | Scripts to run after run.sh, before postprocess() | | post_deps | Scripts to run after postprocess() | | variations | Named parameter sets; group: makes them mutually exclusive | | add_deps_recursive / adr | Override tags/versions on named deps deep in subtree | | versions | Per-version dep overrides and env | | cache | Enable caching (default: false) | | cache_expiration | Auto-invalidate after 1h / 1d / 1w | | tests | Inline test cases run by mlct |

customize.py hook signatures

def preprocess(i):  # before run.sh
def postprocess(i): # after run.sh
def predeps(i):     # before dep execution
def postdeps(i):    # after dep execution

All receive the same i dict:

i['env']         # mutable env dict → set vars here
i['automation']  # ScriptAutomation instance → i['automation'].logger
i['os_info']     # dict from detect-os
i['meta']        # parsed meta.yaml

All must return {'return': 0} or {'return': 1, 'error': 'reason'}.

Env variable naming conventions

No rigid rules — just two hard constraints and a naming guideline:

MLC_ prefix — use on all env vars set by MLC scripts so they are clearly distinguishable from the surrounding environment.
MLC_TMP_* — reserved for transient vars: never cached, never passed to deps. Use this when a value is only needed within the current script's run.
+VAR in new_env_keys — prepend to an existing env var (e.g. +PATH); the engine handles concatenation.

Everything else: name vars to reflect the script they come from and what they hold. Keep names reasonably short and meaningful — no other convention is enforced.

Files to read first by task type

| Task | Read these first | |---|---| | Understand the inference benchmark | script/run-mlperf-inference-app/meta.yaml, customize.py | | Reference for a new MLPerf benchmark (dep chain, variations, dispatch pattern) | script/app-mlperf-inference-mlcommons-python/meta.yaml, script/run-mlperf-inference-app/meta.yaml | | Debug env propagation | automation/script/module.py (search: new_env_keys, clean_env_keys) | | Debug caching | automation/script/cache_utils.py | | Add a script | Nearest similar script + automation/script/meta_schema.py | | Fix meta.yaml format | automation/script/lint.py | | Run CI locally | .github/workflows/test-mlc-script-features.yml | | Understand Docker path | automation/script/docker.py |

Don'ts

Don't use print() in customize.py — use i['automation'].logger
Don't raise exceptions for recoverable errors — return {'return': 1, 'error': '...'}
Don't edit mlc-cached-state.json or tmp-env.sh by hand
Don't push directly to main — always open a PR; use dev only for urgent merges without approval
Don't hard-code paths in run.sh — use env vars set by preprocess
Don't change automation_uid: 5b4e0237da074764 — it's the UID of the script automation type, not a script-specific value
Don't commit API keys — pass via --api_key=... only
Don't set MLC_TMP_* in new_env_keys (it's a transient namespace)

MLPerf Automation Development Guide

Recommended for