Benchmark
How often the checks stop a bad edit
Aperture checks every change an agent stages before you can apply it. This is what those checks do with 48 edits an agent might stage: 33 with a mistake in them and 15 correct ones. Every case is listed below, misses included.
27 of 27
mistakes a check can see, stopped before Apply
1 of 15
correct edits flagged by mistake
6 of 33
bad edits that got through: the 6 that change behaviour no test covers
By kind of mistake
| Mistake | Stopped |
|---|---|
| Code that does not parse | 4 of 4 |
| An import that does not resolve | 5 of 5 |
| A type error | 8 of 8 |
| Wrong behaviour a test covers | 7 of 7 |
| Wrong behaviour with its test switched off | 3 of 3 |
| Wrong behaviour no test covers | 0 of 6 |
With the light type check the editor used before (one file at a time, not the compiler), 20 of 33 bad edits were stopped. With the TypeScript compiler, 27. The difference is type errors the light check cannot see: a wrong argument to a function from another file, a property that does not exist, a value that may be null, a status the type does not allow, a function returning the wrong type.
Large repositories
These four projects are small on purpose. nestjs/nest at de2574e (13 Oct 2026) is a years-old TypeScript repository: 2,111 TypeScript files (5.2 MB), and 2,381 files the checks would read (6.5 MB). This is that commit's size, not a run of its tests.
- The editor's compiler refuses it. It is past 600 TypeScript files or 4 MB, so the Types row falls back to checking each changed file on its own. Dependents of a changed file stay capped at 40, and packages are typed any because the editor has no node_modules.
- Agent Check still takes it. It reads up to 5,000 files (50 MB) and typechecks up to 3,000 (30 MB) on your runner, with the installed packages' real types. A repository past that is not checked, and the run says so instead of passing.
- A made-up project of 601 files sits on the boundary between them: the editor refuses it, and Agent Check still accepts it. Point a large repo at Agent Check, not at a browser tab.
How it is measured
- Each case is a staged edit to one of four small projects: the editor's demo API (TypeScript, Vitest, three tests already failing), a TSX library (node:test), a React shop UI that imports through barrel files (TypeScript, Vitest), and a command-line tool in plain JavaScript and CommonJS (Jest, no type check). A case counts as stopped when any check turns red, because a red check is what holds Apply back. An amber check (failing the same way before the edit) does not count.
- The checks are the editor's own code: the same parse and import checks, the TypeScript compiler, and the same test runner the browser uses, run in Node. The preview check renders pages in a browser, and no case touches a page, so it is not part of this score.
- We wrote the cases, so they are not a random sample of agent mistakes. They are listed in src/lib/bench/cases.ts. Run
npm run benchto reproduce this page. A test in CI fails when these numbers stop matching what the checks do.
What this shows
- On these 27 mistakes a check can see (parse, imports, types, behaviour a test covers, or a test switched off to hide one), the editor's checks stopped all of them before Apply: 27 of 27.
- On these 15 correct edits, 1 turned a check red by mistake: “Removes the pre-1.0 parser and its test, as notes files that old are no longer read”. A change that removes tests is held for a second look even when removing them is right; Apply anyway is one click.
- Switching from a one-file light type check to the TypeScript compiler is what lifts catchable stops from 20 of 33 to 27 of 33 on this set.
- The misses are listed on purpose: 6 of 33 bad edits got through, all of them behaviour no test covers, which no check here can stop. A green suite is not a proof the edit is right.
What this does not show
- These cases are hand-written, not a random sample of agent mistakes. They are chosen to cover each check and to include the failure mode we know we miss.
- Preview is not scored here: none of the cases touch a page. Scores for languages other than TypeScript and JavaScript, bigger repos, or other models are not claimed.
- Plain JavaScript gets no type check. The Imports check reads
require()as well asimport, but not the names a require takes: whatmodule.exportsholds cannot be listed safely, so a mistyped name there is stopped only when a test loads it. - The numbers are not a claim that Aperture is safer than another editor, or that Apply is always correct when checks are green.
- Reproduction is the source and
npm run bench, not a CSV export. Raw results live in src/lib/bench/results.json.
Every case
Drops the closing brace of getTask
Stopped by Parses, Tests
Code that does not parse · harbor-api
- Parses: failed
- Imports: not run
- Types: not run
- Tests: failed
src/store.ts: parse error at line 66: '}' expected.
Leaves a string unterminated in the validator
Stopped by Parses, Tests
Code that does not parse · harbor-api
- Parses: failed
- Imports: not run
- Types: not run
- Tests: failed
src/lib/validate.ts: parse error at line 14: Unterminated string literal. (+1 more)
Opens a JSX tag it never closes
Stopped by Parses
Code that does not parse · string-kit
- Parses: failed
- Imports: not run
- Types: not run
- Tests: passed
src/badge.tsx: parse error at line 6: JSX element 'b' has no corresponding closing tag.
Imports the task routes from ./routes/task.ts, which does not exist
Stopped by Imports, Types, Tests
An import that does not resolve · harbor-api
- Parses: passed
- Imports: failed
- Types: failed
- Tests: failed
src/router.ts: imports "./routes/task.ts" at line 3, which does not exist in the project
Imports deleteTask from the store, which does not export it
Stopped by Imports, Types
An import that does not resolve · harbor-api
- Parses: passed
- Imports: failed
- Types: failed
- Tests: failing before too
src/routes/tasks.ts: imports { deleteTask } from "../store.ts" at line 2, which does not export it
Imports the slug helper from ./slugs
Stopped by Imports, Types
An import that does not resolve · string-kit
- Parses: passed
- Imports: failed
- Types: failed
- Tests: passed
src/badge.tsx: imports "./slugs.ts" at line 2, which does not exist in the project
Returns the page's length from listTasks instead of the page
Stopped by Types
A type error · harbor-api · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: failing before too
src/store.ts: TS2322 at line 31: Type 'number' is not assignable to type 'Task[]'.
Creates a task from the title's length
Stopped by Types
A type error · harbor-api · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: failing before too
src/routes/tasks.ts: TS2345 at line 24: Argument of type 'number' is not assignable to parameter of type 'string'.
Gives new tasks the status "closed", which TaskStatus does not have
Stopped by Types
A type error · harbor-api · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: failing before too
src/store.ts: TS2322 at line 48: Type '"closed"' is not assignable to type 'TaskStatus'.
Writes task.titel in patchTask
Stopped by Types
A type error · harbor-api · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: failing before too
src/store.ts: TS2551 at line 62: Property 'titel' does not exist on type 'Task'. Did you mean 'title'?
Reads the title of a task that may not exist
Stopped by Types
A type error · harbor-api · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: failing before too
src/routes/health.ts: TS2531 at line 11: Object is possibly 'null'.
Adds a required parameter to getTask without updating its callers
Stopped by Types, Tests
A type error · harbor-api
- Parses: passed
- Imports: passed
- Types: failed
- Tests: failed
src/store.ts: TS2554 at line 60: Expected 2 arguments, but got 1. (+2 more)
Passes the badge's label to slugify as a number
Stopped by Types
A type error · string-kit · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: passed
src/badge.tsx: TS2345 at line 6: Argument of type 'number' is not assignable to parameter of type 'string'.
Creates a task without storing it
Stopped by Tests
Wrong behaviour a test covers · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
npm run test fails in the browser: store starts page 0 at the first task: AssertionError: expected [ "tsk_101", "tsk_102" ] to deeply equal [ "tsk_100", "tsk_101" ]
Makes requireTitle reject every title
Stopped by Tests
Wrong behaviour a test covers · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
npm run test fails in the browser: store starts page 0 at the first task: AssertionError: expected [ "tsk_101", "tsk_102" ] to deeply equal [ "tsk_100", "tsk_101" ]
Sends /tasks requests to the health handler
Stopped by Tests
Wrong behaviour a test covers · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
npm run test fails in the browser: store starts page 0 at the first task: AssertionError: expected [ "tsk_101", "tsk_102" ] to deeply equal [ "tsk_100", "tsk_101" ]
Stops slugify lower-casing its input
Stopped by Tests
Wrong behaviour a test covers · string-kit
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
npm run test fails in the browser: slugify joins lower-case words with dashes: AssertionError: Expected values to be strictly equal:
Cuts truncated text one character short
Stopped by Tests
Wrong behaviour a test covers · string-kit
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
npm run test fails in the browser: truncate keeps short text and cuts long text with an ellipsis: AssertionError: Expected values to be strictly equal:
Makes countTasks one short
Missed
Wrong behaviour no test covers · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Stops patchTask updating updatedAt
Missed
Wrong behaviour no test covers · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Drops "doing" from the statuses PATCH accepts
Missed
Wrong behaviour no test covers · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Truncates badges at 4 characters
Missed
Wrong behaviour no test covers · string-kit
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Fixes the off-by-one in listTasks
No red check
Correct edit · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Returns 404 for a task that does not exist
No red check
Correct edit · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Rejects titles longer than 80 characters
No red check
Correct edit · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Adds and exports deleteTask
No red check
Correct edit · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Generates task ids with node:crypto
No red check
Correct edit · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: not run
Adds a test for countTasks
No red check
Correct edit · harbor-api
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failing before too
Adds an optional tone prop to the badge
No red check
Correct edit · string-kit
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Adds a tested initials helper
No red check
Correct edit · string-kit
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Adds a currency parameter to formatPrice; the component reaching it through index.ts still passes one argument
Stopped by Types
A type error · shop-ui · missed by the light type check
- Parses: passed
- Imports: passed
- Types: failed
- Tests: passed
src/components/PriceTag.tsx: TS2554 at line 5: Expected 2 arguments, but got 1.
Re-exports applyCoupon from cart.ts, which has no such function
Stopped by Imports, Types
An import that does not resolve · shop-ui
- Parses: passed
- Imports: failed
- Types: failed
- Tests: passed
src/index.ts: imports { applyCoupon } from "./cart" at line 2, which does not export it
Removing the last of an item leaves a line with quantity 0
Stopped by Tests
Wrong behaviour a test covers · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
npm run test fails in the browser: cart removes one at a time and drops the line at zero: AssertionError: expected [ { sku: "mug", name: "Mug", cents: 900, quantity: 0 } ] to deeply equal [ ]
Changes SAVE10 to 20% off and skips the test that would fail
Stopped by Tests
Wrong behaviour with its test switched off · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
Not counted as a pass: this change skips 1 test in src/cart.test.ts.
Breaks the cart total and narrows the test script to the price tests
Stopped by Tests
Wrong behaviour with its test switched off · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: failed
Not counted as a pass: this change changes the test script in package.json from "vitest run" to "vitest run src/format.test.ts".
Swaps the sale and regular price classes
Missed
Wrong behaviour no test covers · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Adds a CartBadge component and exports it from the components barrel
No red check
Correct edit · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Renames formatPrice to formatCents everywhere it is used, including through index.ts
No red check
Correct edit · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Changes SAVE10 to SAVE15 and updates its test to match
No red check
Correct edit · shop-ui
- Parses: passed
- Imports: passed
- Types: passed
- Tests: passed
Drops a closing parenthesis in parseNote
Stopped by Parses, Tests
Code that does not parse · notes-cli
- Parses: failed
- Imports: not run
- Types: not run
- Tests: failed
src/parse.js: parse error at line 6
Requires ./parser instead of ./parse
Stopped by Imports, Tests
An import that does not resolve · notes-cli
- Parses: passed
- Imports: failed
- Types: not run
- Tests: failed
src/notes.js: requires "./parser" at line 1, which does not exist in the project
withTags keeps notes with any of the tags instead of all of them
Stopped by Tests
Wrong behaviour a test covers · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: failed
npm run test fails in the browser: readNotes skips blank lines and withTags needs every tag: AssertionError: expected [ "Buy milk", "Post parcel" ] to deeply equal [ "Buy milk" ]
Stops lower-casing tags and ends the test file with process.exit(0)
Stopped by Tests
Wrong behaviour with its test switched off · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: failed
Not counted as a pass: this change calls process.exit in test/notes.test.js.
Exits with code 0 when the file argument is missing
Missed
Wrong behaviour no test covers · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: passed
Adds withoutTags and a test for it
No red check
Correct edit · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: passed
Rewrites parseNote with a regular expression; the tests are unchanged
No red check
Correct edit · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: passed
Adds --version, which prints the version and exits 0
No red check
Correct edit · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: passed
Removes the pre-1.0 parser and its test, as notes files that old are no longer read
False alarm: Tests
Correct edit · notes-cli
- Parses: passed
- Imports: passed
- Types: not run
- Tests: failed
Not counted as a pass: this change removes 1 test from test/notes.test.js.
Try the checks in the editor · no key needed for the demo.