The runtime for
verified software work
ML.ai turns an issue into a tested pull request through an engineered pipeline: a code graph that knows your repo, a contract fixed before work starts, twelve gates around every agent step, a sandbox per attempt, and an evidence bundle behind every claim. This is how it works, in detail.
Why we’re building
Software should be something anyone can build, and AI-written code should be something anyone can trust.
We believe everyone should be able to build software.
A student, a freelancer, a two-person startup, a platform team in a large enterprise: anyone with an idea should be able to turn it into working, tested software. Today that ability is rationed. Coding agents are priced by the token, stop when the budget runs out, and hand back work you still have to check line by line. Cheap generation hasn’t made software cheap, because trust is still expensive. We don’t ask you to trust the agent. We show you the evidence: the agent proposes, contracts and checks decide, you approve.
const submit = async () => { await charge(cart) if (paid.current) return paid.current = true await charge(cart)}The agent writes a diff. Nothing it produces is trusted yet.
The hard problems are system problems
Look at why agents fail on real repositories. Almost none of it is about raw intelligence. It's about context, control and checking, and those can be engineered. Every row here is a component on this page. That's the product.
Changes code nobody asked it to touch
Every failure maps to a component you can inspect on this page.
Why this means unlimited
When the system eliminates wasted work, each verified task costs less, so there's no reason to make you count tokens. Fair-use and concurrency rules apply.
Budget spent. The task stops at 62%.
A meter stops the work the moment the budget runs out.
Howruns a task
Every task takes the same path: twelve gates, five planes that each own one job, and eight rules the system never bends.
One task, start to finish
This example is modelled on a real fix: a checkout bug where clicking Pay twice charged the customer twice. Running ml fix #381 starts the task. The chart follows it from the first safety check to the moment it waits for you to approve. Most bars are the system working. The striped one is the system waiting for you.
Checks and your approval take longer than generation. They run in parallel where they do not share state, and typecheck and lint start the moment the patch lands.
The twelve gates
Every agent step passes the same fixed path: six gates before it, six after. Same gates on every surface, whatever happens inside the step.
Core gates shipping- 6
- gates before the step
- 1
- agent step, untrusted
- 6
- gates after the step
Works out what kind of step this is: edit, plan, fix, review or question, plus its risk.
Eight invariants
Rules the runtime never breaks. Every component is designed around them.
Attempts hold path-level leases on their worktree. Two attempts never edit the same file.
Every external action carries an idempotency key, so a retry can never double-apply.
If a tool, recipe or cache can do the step, the agent doesn't.
The runtime opens PRs. It never merges or deploys on its own.
'Done' is a verdict computed from checks, never a summary written by the agent.
Scope and acceptance checks are fixed before work starts. The agent can't move the goalposts.
If a gate can't decide (missing evidence, unknown path), the answer is no.
Every diff, command and tool call is validated against schema and policy before it runs.
How it is built
Each part of ML.ai has one job. The control plane makes decisions, the context plane knows your repo, the execution plane runs the work, the verification plane checks it, and the evidence plane records what happened.
Auth, tenancy, rate and concurrency limits
Mission → Task → Attempt lifecycle, retries, timeouts
Evaluates mlai.yml and contracts into allow/deny/approve
Six pre-step and six post-step gates
Symbols, calls, imports, routes, tables, tests, owners
Lexical + semantic + graph, fused with RRF
MLAI.md, Skills, durable decisions
Ephemeral container, CPU/mem/time caps, egress off
Fresh checkout at baseline SHA, path-level write leases
Allow-listed commands, schema-validated tool calls
Tiered checks chosen by impact and risk
Pure function of contract + evidence + policy
PR proof card, shareable proof page
Append-only, hash-chained bundles
Span per gate, step and check
Accepted, rejected and edited diffs
Deep dives
The parts that do the work, with the data structures and code behind them.
Code graph and ranked retrieval
Most agent mistakes start with the wrong context. ML.ai keeps a map of your codebase, every function, call, import and test, and hands each task only the slice it needs, inside a fixed budget.
To find that slice, three searches run side by side: by keyword, by meaning, and by how code connects. Their scores can’t be compared directly, so we merge the rankings instead, using Reciprocal Rank Fusion. After a change, we walk the graph backwards to find what it could break, so the most affected tests run first.
- ShippingCode graph and ranked retrieval
- In developmentImpact map
- 01ParseSyntax trees
Per-language parsers turn files into symbols. Keyed by content hash.
- 02LinkTyped edges
calls · imports · routes · tested_by · owned_by
- 03ClusterCommunities
Hierarchical clustering groups related code into named modules.
- 04LabelBudgeted labels
One representative per cluster gets a summary; exact names remain.
- 05ServeQuery + impact
Ranked retrieval and reverse-edge impact for each task.
Incremental on every commit. Parsing is keyed by content hash, so unchanged files are skipped.
Task contracts and policy as code
A contract fixes what one task must achieve and may touch, before the agent starts. A policy sets what any task in your repo may ever do. Both are plain files, reviewed like any other change.
- ShippingMLAI.md and permissions
- In developmentTask contracts and mlai.yml
# Task contract: fixed before the agent starts. Immutable (I-3).
id: task_7f3a
objective: "Stop double charges when Pay is clicked twice"
source: github.com/acme/storefront/issues/381
baseline: 4e1a9c2
scope:
allow:
- src/checkout/**
- src/components/PayButton.tsx
deny:
- infra/**
- "**/*.env*"
acceptance:
- id: regression
run: pnpm test -- checkout/double-submit.spec.ts
expect: pass
- id: suite
run: pnpm test
expect: pass
limits:
wall_clock: 30m
repair_attempts: 3
risk: high
approval: requiredFixed before the agent starts, and it cannot be changed mid-task.
Sandboxes and isolation
Every attempt runs in its own throwaway environment. That is why parallel jobs can’t step on each other, and why a failed attempt leaves nothing behind.
- ShippingIsolated parallel jobs
- In developmentFull isolation spec
A git worktree at the contract's baseline SHA. A failed attempt is discarded.
Parallel tasks in one mission lease disjoint paths.
Hard caps per sandbox. Runaway processes are killed and recorded.
Package registries go through an allow-listed cache.
Credentials go to allowed commands, never into the agent's context.
Tool calls are validated by name and schema.
Verification engine
Checks are picked by risk and by what the change touches. The cheap ones that catch the most run first, and checks that don’t share state run at the same time.
- ShippingTests and pull requests
- In developmentCheck tiers and verdicts
Every task. Scope, typecheck, lint, impacted tests, new regression test.
Before any PR. Full test suite, secrets scan, build.
When risk is high. Static security analysis, dependency audit, migration dry-run.
On sensitive paths. Approval, with the proof card attached.
Checks that catch the most for the least time run first. Checks that share no state run together.
Evidence bundles and proof cards
Every claim in a pull request points to a record: the command that ran, its exit code, how long it took, and a digest of its output. Records are append-only and chained by hash, so nothing can be quietly edited.
- In developmentEvidence bundles and proof cards
Checks
scope, 2 of 4 paths · 0.2stsc --noEmit · 11.4sregression spec · 2.8s214 tests · 48.0sNot yet proven
gateway latency under loadSafari double-tap timingWhat a reviewer sees on the pull request. Shared proof pages show the evidence, never your conversation.
The repair loop
“Try again” isn’t a strategy. Each kind of failure has its own response, and the same error twice in a row is never retried a third time.
- In developmentFailure taxonomy and limited repair
Targeted fix at the reported locations
Reproduce, find the cause, make a scoped fix
Discard the worktree, retrieve again, re-plan
Re-plan from the contract with fresh context
One retry with backoff, then a human
Stop. New approach or a human, never a third try
Durable missions
Quick edits stay interactive. Long work, such as a migration or an overnight upgrade, runs as a mission: a set of tasks that survives crashes, CI waits and restarts.
At every checkpoint ML.ai saves a short packet: the contract, the baseline, decisions made, failures seen, links to evidence, the budget left and the next step. A resumed attempt starts from that packet instead of replaying a long transcript.
- ShippingParallel jobs
- In developmentDurable missions
Every transition is an event, so a task can always be replayed.
Caching and efficiency
Most waste in an agent run is repetition: re-reading files, re-ranking, re-installing, re-running. Every layer here is keyed by a hash of its content, so identical work is never done twice.
- ShippingContent-hash caches
- ResearchRecipes
Saves: re-parsing unchanged files
Saves: re-sending stable context
Saves: re-ranking identical context
Saves: reinstalling packages
Saves: re-running an identical check
Saves: open-ended generation for known fixes
cost per verified task = (steps + checks + sandboxes + retries) / verified tasksCode graph and ranked retrieval
Most agent mistakes start with the wrong context. ML.ai keeps a map of your codebase, every function, call, import and test, and hands each task only the slice it needs, inside a fixed budget.
To find that slice, three searches run side by side: by keyword, by meaning, and by how code connects. Their scores can’t be compared directly, so we merge the rankings instead, using Reciprocal Rank Fusion. After a change, we walk the graph backwards to find what it could break, so the most affected tests run first.
- ShippingCode graph and ranked retrieval
- In developmentImpact map
- 01ParseSyntax trees
Per-language parsers turn files into symbols. Keyed by content hash.
- 02LinkTyped edges
calls · imports · routes · tested_by · owned_by
- 03ClusterCommunities
Hierarchical clustering groups related code into named modules.
- 04LabelBudgeted labels
One representative per cluster gets a summary; exact names remain.
- 05ServeQuery + impact
Ranked retrieval and reverse-edge impact for each task.
Incremental on every commit. Parsing is keyed by content hash, so unchanged files are skipped.
Research: how coding agents actually fail
Pass rates hide most of what goes wrong in real codebases. We study how agents fail, and each finding becomes part of the product.
Measure the whole system
We test full setups first, then switch off retrieval, verification or repair one at a time to see what each is worth.
Verified is the metric
We count the cost and time to reach a correct, tested result, not tokens per second.
Label every number
Each figure is marked measured, reported or projected, and the three are never mixed.
BlastBench
In preparationHow much of the codebase does an agent touch beyond what the fix needed?
Every unneeded line costs review time and raises merge risk, and pass-rate benchmarks don't see it.
- Part 1
- One-LineBench: the correct fix is tiny.
- Part 2
- Zero-LineBench: the right answer is to change nothing.
- Becomes
- Scope gates, and files changed versus files allowed on every proof card.
RetryBench
Protocol designedAfter a failure, does the agent fix the cause, or repeat the mistake at higher cost?
On long jobs, how an agent recovers matters more than whether it gets the first try right.
- Compares
- Six recovery strategies, grouped by the type of first failure.
- Becomes
- A failure taxonomy, a stop on repeated errors, and limited repair.
StickyBench
In preparationSame final task, different history: how much does old context change the answer?
Long sessions pile up wrong guesses and abandoned plans, and they quietly bend later answers.
- Design
- One fixed task under six history conditions.
- Becomes
- Recovery packets instead of transcripts, and scoped attempts.
Changes beyond what the fix needed
Minimal fix, 1 file
Typical patch, 9 files
How much of the codebase does an agent touch beyond what the fix needed?
Every unneeded line costs review time and raises merge risk, and pass-rate benchmarks don't see it.
- Part 1
- One-LineBench: the correct fix is tiny.
- Part 2
- Zero-LineBench: the right answer is to change nothing.
- Becomes
- Scope gates, and files changed versus files allowed on every proof card.
Changes beyond what the fix needed
Minimal fix, 1 file
Typical patch, 9 files
After a failure, does the agent fix the cause, or repeat the mistake at higher cost?
On long jobs, how an agent recovers matters more than whether it gets the first try right.
- Compares
- Six recovery strategies, grouped by the type of first failure.
- Becomes
- A failure taxonomy, a stop on repeated errors, and limited repair.
Same failure, two recoveries
Naive retry
TypeError: undefined.id (try 1)same error (try 2)budget exhaustedClassify, diagnose, repair
TypeError: undefined.id (try 1)test_failure: race on session loadtests passSame final task, different history: how much does old context change the answer?
Long sessions pile up wrong guesses and abandoned plans, and they quietly bend later answers.
- Design
- One fixed task under six history conditions.
- Becomes
- Recovery packets instead of transcripts, and scoped attempts.
One task, six histories
The agent itself said something wrong earlier. Is it more trusting of its own mistake?
Open problems
Six questions we are still working on, and the rule each one has to respect.
Ordering verification by information gain
Given a patch and its impact set, which subset of checks catches the most real regressions for the least compute? We learn per-repo check yield from history and run high-yield, low-cost checks first.
tier_2 always runs before a PR. Optimization only reorders, never skips required checks.
Reverse edges when types don't help
Static call graphs miss dynamic dispatch and reflection. We combine static edges with runtime traces from test runs to close the gap.
Metric: recall of tests that fail on a seeded regression, at a fixed test budget.
Migrations and fixes that shouldn't need generation
A recipe is a parameterized plan of codemods and checks. The agent only handles the residue a codemod cannot.
Kept only if it lowers cost per verified task with no rise in regressions.
What's the smallest state that fully resumes a task?
Too little and the resumed attempt repeats work; too much and stale context leaks back in. We compare packet variants by resume success and contamination.
Accepted, rejected and edited diffs as signal
Signals tune retrieval weights, check ordering and planning for one repository at a time, evaluated on held-out tasks.
Long-term, and never a substitute for tests
Can we predict which tests will break, to order checks and catch bad plans early? Predictions only reorder work, never replace a real check.
From finding to component
Each result on this page ends up as a part of the product.
What's next
What ships today, what we are building, and what is still research.
Now
ShippingIn the product today
- Infinite across CLI, IDE, GitHub and web
- Code graph with RRF-ranked context
- Isolated parallel jobs and routines
- Core gates, permissions, MLAI.md and Skills
- Tests run and PRs opened with summaries
Next
In developmentBeing built now
- Task contracts and the verdict engine
- Evidence bundles and proof cards
- Impact map with affected-test ordering
- Durable missions with recovery packets
- One mlai.yml across every surface
Later
ResearchIn the lab, not a promise
- Check ordering by learned yield
- Recipes for known migrations
- Per-repository learning
- Effect prediction, only once it beats real tests
Now
ShippingIn the product today
- Infinite across CLI, IDE, GitHub and web
- Code graph with RRF-ranked context
- Isolated parallel jobs and routines
- Core gates, permissions, MLAI.md and Skills
- Tests run and PRs opened with summaries
Agents write code
makes it provable
You describe the outcome. ML.ai plans it, scopes it, does the work in isolation, checks it, records the evidence and hands you something you can merge with confidence, whether that takes four minutes or four days.