A Compile-Time Safety Contract for LLM-Authored Fitness Plans

Abstract

Consumer-grade AI fitness products increasingly let large language models (LLMs) emit free-form workout and nutrition plans directly to end users. We present a reproducible, version-controlled safety benchmark of this deployment pattern. The corpus release (v0.6.0) contains twenty trainer-voice scenarios: fifteen 12-week scenarios spanning medical contraindications, menstrual-cycle adaptation, and constraint adherence, plus five short-plan scenarios (1–4 weeks) that exercise structural failure modes — missing rest days, over-fast progression, skipped on-ramps — that exercise blacklists cannot express.

We evaluate seven models across two vendors — gpt-5, gpt-5-mini, gpt-5-nano, gpt-4.1 (OpenAI) and Claude Opus 4.7, Sonnet 4.6, Haiku 4.5 (Anthropic) — in single-turn and 8-turn multi-turn protocols: 560 trials at a total inference cost of $169.99. The WPL contract reduces unsafe-trial rate 3–5× on every corpus and both phases (Lane A 32–51% unsafe → Lane B 8–17%), eliminating the medical-contraindication class entirely (46/84 → 0/84 across vendors). Raw-LLM safety degrades as model capability grows: the two flagships are the two worst raw-safety performers, while the safest models sit in the cheap non-flagship tier. Multi-turn drift drops from 42% to 6% of conversations under governance (44/105 → 6/105).

We additionally disclose, in full, a measurement artifact in a pre-publication snapshot of this benchmark — an extraction defect that reported "0 violations" where the corrected figure is 5–6 unsafe trials per sub-corpus — together with the three further methodology bugs found while fixing it; the corrected re-score required zero additional inference because every raw model output is committed. We publish every prompt, every model output, every score, and the deterministic scorer; every quantitative claim is offline-reproducible without further API spend.

Key results

• → Unsafe-trial rate reduced 3–5× on every corpus: Lane A 32–51% → Lane B 8–17%
• → Medical-contraindication class eliminated entirely: 46/84 → 0/84 across both vendors
• → Raw-LLM safety degrades as capability grows: the two flagships (Opus 4.7, gpt-5) are the two worst raw-safety performers
• → Multi-turn drift: 44/105 (42%) raw → 6/105 (6%) governed
• → Full measurement-integrity disclosure: a pre-publication extraction defect and three methodology bugs, corrected with zero extra inference

About this version

Adds the full Anthropic lineup (Opus 4.7, Sonnet 4.6, Haiku 4.5), five short-plan structural scenarios, a revised multi-turn protocol, the served-rate analysis, and a full measurement-integrity disclosure. 240 → 560 trials, 15 → 20 scenarios, 1 → 2 vendors.

Read on this page