Motion Graphics
Benchmark.

This tests language models writing motion-graphics code, not native text-to-video models. Every request goes through OpenRouter. The generated code is rendered with the same Remotion environment.

What stays the same

Three fixed briefs, two independent runs, one component contract, the same local fonts and palette, and a 32,000-token completion limit that includes reasoning. No browsing, external assets, or existing Cliphouse compositions.

What gets another attempt

Only a failed compile or render gets one repair request, with the error log. Successful videos receive no creative revision. Original attempts and repair costs are retained. Transport errors are recorded separately.

Automated recovery / R1

Only original failures receive up to three additional automated repair requests. Keep the same model, pinned provider, reasoning effort, brief, fonts, palette, and renderer. Return complete TSX directly; a single code fence or valid source JSON wrapper is accepted without editing the code. Supply the previous response and complete compiler/render diagnostics. Stop at the first successful render. No human code edits or creative feedback. Preserve original outcomes and report additional attempts and costs separately. Recovery uses a 64,000-token limit and direct TSX output instead of the original JSON-only format. These are additional repair results, not replacement first attempts.

Repair results · Repair protocol · Original results

What we disclose

The exact model ID, provider, request, reasoning setting, source code, OpenRouter generation time, and API cost. Generation time comes from the successful request's generation_time field in OpenRouter's generation metadata, converted from milliseconds to seconds. Providers are pinned and fallback is disabled. Up to four API requests overlap; renders remain sequential. Reasoning labels are not equivalent compute budgets across models.