There are two ways to use AI for end-to-end testing, and they have very different economics.
In the first, AI runs the test. Every time you want to check that checkout works, a model looks at the screen, decides where to click, clicks, looks again and decides whether things look right. It looks great in a demo. As a regression suite, it has three problems.
- Cost grows with every run. Running nightly, before every release and across a dozen flows adds up to thousands of AI-driven sessions a month, and the bill grows with how often you test. That’s the wrong incentive: you want teams testing more.
- It’s slow. Every step waits for the model to reason about the screen, so a flow a script clicks through in seconds takes minutes.
- It isn’t deterministic. Ask the same question twice and you may get a slightly different answer. A regression test that sometimes passes and sometimes fails for no reason isn’t a regression test.
In the second, AI writes the test and a machine runs it. That’s the one we built.
Authoring: where AI earns its keep
Writing a good end-to-end test is where the intelligence is needed. You have to understand what the feature is supposed to do, work out a sensible path through it, pick selectors that will survive a redesign, and write assertions that check the thing that matters rather than something incidental.
When you point CodeReviewer at a module of your app, it does that work:
- It reads the code to learn what the feature should do: routes, forms, validation rules and permission checks.
- It proposes a test plan of positive and negative cases, and you review it. Keep what matters and cut what doesn’t.
- It signs in and drives your running app the way a user would, and records each case as a real, replayable test.
- It validates each test before promoting it. A test that can’t pass reliably against the current app doesn’t join the suite.
This costs AI time, and it’s worth it, because the output is an asset you keep.
Replay: where AI stays out of the way
Once the suite exists, replaying it is plain automation. No model sits in the loop, no reasoning happens per click, and nothing is metered. So the rule on every CodeReviewer plan is simple:
Authoring uses your plan’s AI runs. Replaying never does.
Run the suite nightly, before each release or on demand, as often as you like, for free. The same goes for the performance and security runs that ride on your suites.
That changes behavior. Teams stop rationing test runs. They schedule a nightly replay and run the suite before every release, because it costs nothing more. Coverage stops being a budget line.
“But the app changes”: healing
The classic objection to recorded tests is brittleness. Rename a button and half the suite goes red. Traditional script suites die this way: repairs cost more than the suite is worth, so people stop maintaining it.
This is the second place AI earns its keep. When a replay fails, CodeReviewer works out whether your app broke or the test went stale. If the button was only renamed, it repairs the test and re-verifies it against the current app before it counts again. If the app really broke, the failure stays red, with screenshots, video and a step-by-step trace attached so a developer can reproduce it.
Healing uses AI too, but only when something has changed. That’s a small fraction of runs, not every one.
Flaky tests get benched, not your suite
A single flaky test can wreck trust in a suite. After the third false alarm, people stop reading the failures. CodeReviewer quarantines a test that flips between pass and fail without a code change. The test is benched and retried on its own until it proves stable, and the rest of the suite keeps reporting clean signal.
The shape of the costs
Put together, the cost curve looks like this:
- Up front: AI time to author the suite, roughly in proportion to how much of your app you cover.
- Ongoing: close to zero AI time per replay, however often you run.
- Occasionally: a little AI time to heal tests when your app changes.
Compare that with AI-per-run testing, where cost rises in step with how often you test. Past a small number of runs, authoring once is far cheaper, and it’s faster and deterministic as well.
What it means in practice
Your QA team stops spending its week maintaining scripts and re-clicking the same flows. It spends that week curating test plans, investigating real failures and doing the exploratory testing no machine does well. The suite becomes something the team trusts, runs constantly and builds on.
That was the idea from day one. Use AI where it adds judgment, and plain automation everywhere else.