Benchmark

How often the checks stop a bad edit

Aperture checks every change an agent stages before you can apply it. This is what those checks do with 30 edits an agent might stage: 22 with a mistake in them and 8 correct ones. Every case is listed below, misses included.

By kind of mistake

With the light type check the editor used before (one file at a time, not the compiler), 12 of 22 bad edits were stopped. With the TypeScript compiler, 18. The difference is type errors the light check cannot see: a wrong argument to a function from another file, a property that does not exist, a value that may be null, a status the type does not allow, a function returning the wrong type.

How it is measured

• Each case is a staged edit to a small real project: the editor's demo API (TypeScript, Vitest, three tests already failing) or a small TSX library (node:test). A case counts as stopped when any check turns red, because a red check is what holds Apply back. An amber check (failing the same way before the edit) does not count.
• The checks are the editor's own code: the same parse and import checks, the TypeScript compiler, and the same test runner the browser uses, run in Node. The preview check renders pages in a browser, and no case touches a page, so it is not part of this score.
• We wrote the cases, so they are not a random sample of agent mistakes. They are listed in src/lib/bench/cases.ts. Run npm run bench to reproduce this page. A test in CI fails when these numbers stop matching what the checks do.

What this shows

• On these 18 mistakes a check can see (parse, imports, types, or behaviour a test covers), the editor's checks stopped all of them before Apply: 18 of 18.
• On these 8 correct edits, no check turned red by mistake: 0 of 8 false alarms.
• Switching from a one-file light type check to the TypeScript compiler is what lifts catchable stops from 12 of 22 to 18 of 22 on this set.
• The misses are listed on purpose: 4 of 22 bad edits change behaviour no test covers, so no check here can stop them. A green suite is not a proof the edit is right.

What this does not show

• These cases are hand-written, not a random sample of agent mistakes. They are chosen to cover each check and to include the failure mode we know we miss.
• Preview is not scored here: none of the cases touch a page. Scores for other languages, bigger repos, or other models are not claimed.
• The numbers are not a claim that Aperture is safer than another editor, or that Apply is always correct when checks are green.
• Reproduction is the source and npm run bench, not a CSV export. Raw results live in src/lib/bench/results.json.

Every case

• Drops the closing brace of getTask

  Stopped by Parses, Tests

  Code that does not parse · harbor-api

  • Parses: failed
  • Imports: not run
  • Types: not run
  • Tests: failed

  src/store.ts: parse error at line 66

• Leaves a string unterminated in the validator

  Stopped by Parses, Tests

  Code that does not parse · harbor-api

  • Parses: failed
  • Imports: not run
  • Types: not run
  • Tests: failed

  src/lib/validate.ts: parse error at line 15

• Opens a JSX tag it never closes

  Stopped by Parses

  Code that does not parse · string-kit

  • Parses: failed
  • Imports: not run
  • Types: not run
  • Tests: passed

  src/badge.tsx: parse error at line 8

• Imports the task routes from ./routes/task.ts, which does not exist

  Stopped by Imports, Types, Tests

  An import that does not resolve · harbor-api

  • Parses: passed
  • Imports: failed
  • Types: failed
  • Tests: failed

  src/router.ts: imports "./routes/task.ts" at line 3, which does not exist in the project

• Imports deleteTask from the store, which does not export it

  Stopped by Imports, Types

  An import that does not resolve · harbor-api

  • Parses: passed
  • Imports: failed
  • Types: failed
  • Tests: failing before too

  src/routes/tasks.ts: imports { deleteTask } from "../store.ts" at line 2, which does not export it

• Imports the slug helper from ./slugs

  Stopped by Imports, Types

  An import that does not resolve · string-kit

  • Parses: passed
  • Imports: failed
  • Types: failed
  • Tests: passed

  src/badge.tsx: imports "./slugs.ts" at line 2, which does not exist in the project

• Returns the page's length from listTasks instead of the page

  Stopped by Types

  A type error · harbor-api · missed by the light type check

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: failing before too

  src/store.ts: TS2322 at line 31: Type 'number' is not assignable to type 'Task[]'.

• Creates a task from the title's length

  Stopped by Types

  A type error · harbor-api · missed by the light type check

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: failing before too

  src/routes/tasks.ts: TS2345 at line 24: Argument of type 'number' is not assignable to parameter of type 'string'.

• Gives new tasks the status "closed", which TaskStatus does not have

  Stopped by Types

  A type error · harbor-api · missed by the light type check

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: failing before too

  src/store.ts: TS2322 at line 48: Type '"closed"' is not assignable to type 'TaskStatus'.

• Writes task.titel in patchTask

  Stopped by Types

  A type error · harbor-api · missed by the light type check

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: failing before too

  src/store.ts: TS2551 at line 62: Property 'titel' does not exist on type 'Task'. Did you mean 'title'?

• Reads the title of a task that may not exist

  Stopped by Types

  A type error · harbor-api · missed by the light type check

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: failing before too

  src/routes/health.ts: TS2531 at line 11: Object is possibly 'null'.

• Adds a required parameter to getTask without updating its callers

  Stopped by Types, Tests

  A type error · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: failed

  src/store.ts: TS2554 at line 60: Expected 2 arguments, but got 1. (+2 more)

• Passes the badge's label to slugify as a number

  Stopped by Types

  A type error · string-kit · missed by the light type check

  • Parses: passed
  • Imports: passed
  • Types: failed
  • Tests: passed

  src/badge.tsx: TS2345 at line 6: Argument of type 'number' is not assignable to parameter of type 'string'.

• Creates a task without storing it

  Stopped by Tests

  Wrong behaviour a test covers · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failed

  npm run test fails in the browser: store starts page 0 at the first task: AssertionError: expected [ "tsk_101", "tsk_102" ] to deeply equal [ "tsk_100", "tsk_101" ]

• Makes requireTitle reject every title

  Stopped by Tests

  Wrong behaviour a test covers · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failed

  npm run test fails in the browser: store starts page 0 at the first task: AssertionError: expected [ "tsk_101", "tsk_102" ] to deeply equal [ "tsk_100", "tsk_101" ]

• Sends /tasks requests to the health handler

  Stopped by Tests

  Wrong behaviour a test covers · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failed

  npm run test fails in the browser: store starts page 0 at the first task: AssertionError: expected [ "tsk_101", "tsk_102" ] to deeply equal [ "tsk_100", "tsk_101" ]

• Stops slugify lower-casing its input

  Stopped by Tests

  Wrong behaviour a test covers · string-kit

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failed

  npm run test fails in the browser: slugify joins lower-case words with dashes: AssertionError: Expected values to be strictly equal:

• Cuts truncated text one character short

  Stopped by Tests

  Wrong behaviour a test covers · string-kit

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failed

  npm run test fails in the browser: truncate keeps short text and cuts long text with an ellipsis: AssertionError: Expected values to be strictly equal:

• Makes countTasks one short

  Missed

  Wrong behaviour no test covers · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Stops patchTask updating updatedAt

  Missed

  Wrong behaviour no test covers · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Drops "doing" from the statuses PATCH accepts

  Missed

  Wrong behaviour no test covers · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Truncates badges at 4 characters

  Missed

  Wrong behaviour no test covers · string-kit

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: passed

• Fixes the off-by-one in listTasks

  No red check

  Correct edit · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Returns 404 for a task that does not exist

  No red check

  Correct edit · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Rejects titles longer than 80 characters

  No red check

  Correct edit · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Adds and exports deleteTask

  No red check

  Correct edit · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Generates task ids with node:crypto

  No red check

  Correct edit · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: not run

• Adds a test for countTasks

  No red check

  Correct edit · harbor-api

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: failing before too

• Adds an optional tone prop to the badge

  No red check

  Correct edit · string-kit

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: passed

• Adds a tested initials helper

  No red check

  Correct edit · string-kit

  • Parses: passed
  • Imports: passed
  • Types: passed
  • Tests: passed

Try the checks in the editor · no key needed for the demo.