The problem with “we’ll add tests later”
End-to-end tests are the closest thing a team has to a user checking the product every day: sign in, click through a flow, confirm the right thing happened. Almost everyone agrees they’re valuable. Yet a lot of suites get abandoned within a year. They weren’t hard to write. They just got too expensive to keep.
Every interface change breaks tests that were correct the day before. Someone has to notice, work out whether it’s a real regression or a stale test, and fix it. That work competes with the roadmap and usually loses. The suite goes red, stays red and stops being trusted. At that point it costs money to run and hides real regressions in noise.
Where the hours actually go
The paper splits a test’s lifetime cost into four parts: authoring, running, maintenance and triage. Only authoring is visible when a team decides to “add tests”. Maintenance and triage arrive later, spread across every sprint, and they grow with how often the product changes. Fast-moving teams pay the most, and they’re exactly the teams that most need the safety net.
Separating authoring from running
AI changes the authoring step. An assistant that can read the code behind a feature and click through the running app can learn what the feature should do and record a real test. But if every run also needed AI, cost would grow with every replay, and nightly or per-change runs would become a budget argument.
The better design treats AI as the author and the recording as the artefact. The test is written once, reviewed, and saved as concrete steps and checks. From then on it replays deterministically without AI, as often as you like. AI comes back only when the app changes and a test genuinely breaks.
An illustrative model
The paper includes a twelve-month model you can fill in with your own numbers:
Annual human effort = authoring + 12 × (tests × monthly breakage rate × hours per repair) + 52 × weekly flaky-test triage
With the paper’s deliberately conservative example inputs, maintenance and triage outweigh authoring in a hand-scripted suite: it costs more to keep than to build. And because replays cost nothing in the author-once model, moving from nightly runs to every-change runs adds no human hours at all. It only shortens the time between a regression landing and someone knowing about it.
Also inside
- How flaky-test quarantine and automatic repair keep a suite trusted
- Why every replay should keep screenshots, video and a step-by-step trace
- A seven-step checklist for starting with one high-value flow
- How CodeReviewer applies the author-once model to web apps and to real Android phones
Download the full paper for the complete model, the assumptions behind it and the checklist. It’s free, 10 pages, and written for engineering and QA leads deciding how to invest in test automation.