Why web applications break after launch
An application that runs in a demo and an application that holds up under real users are two different things. This is the gap between them: the families of failure that live in it, and which layer of testing catches which.
The demo is the easy part
An application in a demo has one user. The data is clean because somebody typed it thirty seconds ago. The network is a metre of office wi-fi. The sequence of clicks is the one the builder had in mind, because the builder is the one clicking.
Production has none of those properties. Two people edit the same order inside the same second. A value arrives pasted from a spreadsheet with a trailing space and a non-breaking hyphen in it. A phone drops from 4G to nothing halfway through an upload. Someone presses the browser's back button after submitting, then submits again. A session expires while a long form sits open. A customer in another time zone books a slot that, where your server lives, was yesterday.
None of that is exotic. All of it is a normal Tuesday. And nothing in "it runs on my machine" says anything at all about how the software behaves in any of it.
The question was never whether the software runs. It is which of the thousand things that happen next it has been shown to survive.
Why this became more common
Writing code is faster than it has ever been. Writing the happy path is faster still, because the happy path is precisely what a description produces. Ask for a booking form and a booking form arrives: fields, a submit button, a success message. It will demo perfectly.
What a description rarely contains is the other half of a specification — the half that says what must *not* happen. What happens when the payment provider takes eleven seconds to answer. What happens when the same booking is submitted twice because the first response was slow. What happens when an administrator deletes the record between the page loading and the user pressing save.
That half is not technically difficult. It is simply invisible in a demo, so it does not get asked for, so it never gets written.
This is not new and it is not unique to generated code — the gap between a prototype and a product has always existed. What changed is the ratio. Producing the visible layer is now close to free, so far more software reaches real users having never had its invisible half written at all. It looks finished. Looking finished and being finished have never been further apart.
What actually breaks
The failures that surface weeks after launch fall into a small number of families. Recognising them is most of the work.
Lost updates. Two requests read the same stock level, both compute qty - 1 in application code, and both write the result. One decrement vanishes. Nothing errors and no log line appears; the number is just wrong from then on, and it stays wrong.
Worth being precise about why, because the fix depends on it. A single statement — UPDATE stock SET qty = qty - 1 WHERE id = $1 — is safe even at PostgreSQL's default READ COMMITTED, because the second transaction blocks on the row lock and then re-reads the updated value. It is the read-modify-write *in your code* that loses the write: the SELECT took a snapshot, and by the time the UPDATE lands that snapshot is stale. Four ways out, in rough order of how often they are the right one: do the arithmetic in SQL; take the row with SELECT ... FOR UPDATE first; carry a version column and write WHERE version = $seen, treating zero rows updated as a conflict; or run the transaction at SERIALIZABLE and retry on SQLSTATE 40001. All four are a few lines. None of them happens by accident.
Errors that get swallowed. A catch that catches everything and logs nothing. The feature stops working, the interface shows its empty state, and it reads as "no results" rather than "this is broken". These survive for months.
N+1 queries. A list that fires one extra query per row — one for the list, N for the rows — is imperceptible at fifty rows and fatal at fifty thousand. This is the usual cause of "it got slower and nobody knows why", and the giveaway is that the code did not change. The data did.
Two tools find it in an afternoon. pg_stat_statements ranks queries by total time across the whole database, which surfaces the cheap query being run ten thousand times rather than the expensive one being run once. EXPLAIN (ANALYZE, BUFFERS) on the offender tells you whether the planner is looping. The fix is a join, or one batched round trip with WHERE id = ANY($1).
Lists with no upper bound. A page that loads every record ever created felt instant in week one. It is the page that hangs in month nine.
Missing or unusable indexes. A query the database could have answered from an index is doing a sequential scan instead. Often the index exists and simply cannot be used: a composite index on (created_at, tenant_id) does not serve a query filtering on tenant_id alone, because a B-tree can only be entered from its leading column. Wrapping a column in a function — WHERE lower(email) = $1 — disqualifies a plain index too, unless there is an expression index to match it. Same symptom as the previous one, different cause, and invisible until somebody reads a plan.
Double submission. A slow response, an impatient user, two identical orders. Whether that costs money depends entirely on whether anyone wrote an idempotency key.
Time, dates and calendars. A slot stored without a time zone. A Persian date rendered through a Gregorian calendar. A "today" computed on a server in one country and shown to a user in another.
Permission checks that live in the screen. The button is hidden for the wrong role, so the feature looks secure. The endpoint behind the button checks nothing and will answer anybody who calls it directly.
Third parties with no timeout. An external service degrades instead of failing. Every call to it now takes thirty seconds. Your application does not fail either — it stops responding, because every worker is sitting in a queue waiting.
Resources that are never returned. A connection or a listener opened per request and never closed. Memory climbs across days. The application gets restarted every Sunday and nobody ever calls it a bug.
Every item on that list is catchable. Each one is caught by a different kind of test, which is the reason there is more than one kind.
The layers, and what each catches
Testing is not one activity. It is several, with different costs, different speeds and different blind spots. Treating them as interchangeable is what produces a suite that runs for an hour and still misses the obvious.
Unit tests — the logic One function. No database, no network. Arrange, act, assert. They run in single-digit milliseconds, so there can be thousands and they can run on every keystroke.
*Catches:* the VAT calculation that rounds half-down where the tax authority rounds half-up, the discount that can go negative on a returned line, the date arithmetic that breaks on 31 March.
*Misses:* everything about how the pieces are wired together. Every unit can be correct while the application is broken.
The trap here is substitution. Where a unit needs a collaborator, the useful replacement is a fake — a working in-memory implementation of the same interface — not a mock that asserts on the calls made to it. A mock that says "the repository's save was called once with this object" passes forever, including after save starts throwing on a constraint you added last week. It is testing that your code still calls a function, which nobody doubted.
Integration tests — the seams A real database, spun up per run — Testcontainers, a disposable schema, whatever your stack makes cheap — with the real migrations applied and the real constraints in force. This is the layer where most genuine defects actually live and the layer most often skipped, because it is the one that needs infrastructure.
*Catches:* the migration that never ran, the CHECK that lets half an order save, the transaction that turned out not to be a transaction because an async call escaped it, and the query that returns another tenant's rows.
*Misses:* how any of it looks or behaves in a browser.
If you use row-level security, this is the only layer that can prove it. The test has to connect as the restricted role and assert the empty result, because a service-role connection bypasses every policy and will hand you a green suite over a database that leaks. Write the negative case first: sign in as tenant A, request tenant B's row, require nothing back.
Contract tests — the agreement between parts The API says it returns a field; the client believes it. A contract test fails the moment one side changes without the other.
*Catches:* the renamed field that broke the mobile app three days after the web release shipped fine.
*Misses:* whether either side is correct. It only proves they still agree.
Component and UI tests — the interface in isolation A single component rendered with real props. Does the disabled state actually disable. Does the error message appear. Does an empty list render an empty state rather than nothing at all.
*Catches:* the button that stays enabled during submission — which is how the double order above happens — the form that loses its value on re-render, the spinner that never resolves.
*Misses:* whether that screen is wired to the right endpoint at all.
End-to-end tests — the journeys that earn money A real browser driving the real application: sign in, search, add to basket, pay, see the confirmation. Slow and expensive, so there should be few — and they should cover the paths that cost you money when they break.
*Catches:* the thing every other layer missed, because it is the only test that walks the whole path a customer walks.
*Misses:* nothing in particular, and that is the trap. Two hundred end-to-end tests take an hour and will be ignored inside a month.
They are also where flakiness is born. Almost all of it comes from waiting on a clock instead of on a condition: a fixed sleep passes on a fast machine and fails on a loaded CI runner. Wait for the element, the network response, the state — never for a duration. And when one does go intermittently red, quarantine it and fix the race it found. Adding an automatic retry converts a real concurrency bug into a green tick.
Smoke tests — is it alive A handful of checks that run after each deployment, against the real environment. The home page answers. Sign-in works. One record can be read and written.
*Catches:* the environment variable that was never set in production, the migration that ran everywhere except production, the build deployed to the wrong project. An entire class of failure that passes every test and still takes the site down.
*Misses:* anything subtle. That is the point — it is a pulse, not a diagnosis.
Regression tests — the bug that came back Every bug worth fixing is worth a test, written before the fix so that it fails first. A test that has never failed has not been shown to test anything.
*Catches:* the same defect returning in six months, after the person who fixed it has moved on.
This is also the only honest way to judge a suite. Coverage counts lines that were executed, which is not the same as lines that were *checked* — delete every assertion in your test suite and coverage does not move. If you want to know whether the tests would notice a bug, mutation testing is the measure: it flips a > to a >=, removes a line, inverts a condition, and reports how many of those deliberate breakages the suite caught. A codebase at ninety per cent line coverage and forty per cent mutation score is a codebase whose tests mostly watch.
Load and performance tests A hundred concurrent users against a realistic dataset rather than an empty one.
*Catches:* the query that grows with the data, the missing index, the connection pool that runs dry at thirty users — the whole slow-decline family.
*Misses:* correctness. Fast and wrong is still wrong.
Accessibility tests Automated checks for contrast, labels, focus order and keyboard reachability.
*Catches:* the unlabelled input, the dialog that traps focus, the button that is really a div and cannot be reached by keyboard at all.
*Misses:* whether the flow makes sense to somebody using a screen reader. Automation finds roughly half of it; a person is required for the rest.
Security tests Authorisation asserted rather than assumed: request another tenant's record, require a refusal.
*Catches:* the permission that was only ever enforced in the interface. Also the classics worth one test each: an object id swapped in a URL, a role field accepted from the request body during sign-up, an expired token still accepted because only the signature was verified.
*Misses:* design flaws. A test proves the rule you wrote is working, never that it was the right rule.
The shape of a suite that survives
Cheap and fast at the bottom, expensive and few at the top:
- Many unit tests — milliseconds, running constantly while the code is being written.
- Fewer integration tests — seconds, running on every change.
- A small number of end-to-end tests — minutes, covering only the journeys that matter.
- A few smoke checks — running after every deployment, against the real thing.
Inverting that shape is the common failure. A suite that is mostly end-to-end is slow; slow suites become flaky; flaky suites get ignored, and "just run it again" becomes the team habit. At that point the suite is worse than nothing, because it manufactures confidence without producing evidence.
A test that goes red for no reason teaches everybody to ignore red.
If you have no tests today
Do not start with a coverage number. Start with money and reputation, in this order:
- One smoke check on the critical path, running after every deploy. An afternoon of work, and it catches the most embarrassing class of failure there is.
- End-to-end tests for the two or three journeys that earn revenue. Sign-up, checkout, booking — whatever the equivalent is in your product.
- Integration tests around the rules that protect data: permissions, uniqueness, and anything touching money or stock.
- Unit tests for the logic that has already produced a bug once. History is the best available guide to where the next one is.
- A regression test with every bug fix, from now on, permanently.
That list is achievable in weeks on a codebase that already exists, and it covers most of what actually takes applications down. Coverage percentage is not the goal: it is entirely possible to hold ninety per cent coverage and have no test for the checkout.
A test nobody runs is not a test
A suite on one developer's laptop is not a safety net. It is a personal habit, and it leaves when they do. Tests only earn their cost once they run automatically, on every change, and are allowed to stop a release:
- Every push runs the full suite, without anyone deciding to.
- A failing suite blocks the merge. Not a warning — a block.
- Every change gets a preview environment, so review happens against something real.
- Deployment is automatic once the checks pass, and reversible in one step when the smoke check says otherwise.
That is continuous integration and delivery, and it is the machinery that turns a test suite from a good intention into a property of the system.
What testing does not do
Being honest about the limits is what makes the rest of it credible.
Tests prove the cases somebody thought of. They are not proof that bugs are absent, and no suite ever will be. They do not replace types, which catch a different class of mistake earlier and more cheaply. They do not replace monitoring, because the interesting production failures are the ones nobody predicted. And they will not rescue a data model that was wrong from the beginning — they will only pin the wrongness firmly in place.
They also cost something real: time to write, time to run, time to maintain as the code changes. A test that breaks every time anyone touches anything is testing the implementation rather than the behaviour, and deleting it is the correct response.
The goal is not to test everything. It is to know, deliberately, which failures you are protected from — and which ones you have quietly decided to hear about from a customer.