Your API docs are a second implementation

Someone read my API documentation properly for the first time recently. Not a customer. A reviewer I had cold-emailed the day before, who had no reason to be generous and was anyway. He sent back three corrections. All three were right. Applying them took one evening. The last thing they led me to did not turn up until the following morning.

The short version: reference documentation is a second implementation of your API. You write it once, by hand, against your understanding of the system on that day. Then the system moves. In most projects nothing ever compares the two again.

The symptom you have probably already seen

Open your Swagger UI and look at a successful response. If every field renders as string, your API is not returning strings. Your 200 is declaring application/json with an empty schema, and the renderer has nothing to work with, so it shows the only thing it can.

That was six of my seven public endpoints. Six operations telling every prospective integrator that a confidence score is a string, that a count is a string, that a nested object is a string. I had read that page dozens of times and registered it as "docs look a bit thin". It was not thin. It was wrong, in the specific way that makes a reader assume the API behind it is unfinished.

Absence would have been easier to spot

I also believed my operations were missing their summaries and that I would get to it eventually. They were not missing. FastAPI derives a summary from the function name, so a route called report becomes "Report" and a route called weekly_report becomes "Weekly Report". Every operation had a heading. Nothing was blank. The page looked finished.

This is worse than a gap. A gap is visible, and visible things get fixed. Auto-generated filler reads as an intentional decision, including to the person who made it, which in this case was nobody.

The document had drifted, and nothing was watching

My published parameters section told customers one set of per-plan limits. The code enforced another. The free tier matched. Nothing else did, and one entire paid tier was missing from the table.

Nobody lied. The document was written once, accurately, and then the numbers moved. I can find at least three separate changes to per-plan limits in my own changelog: a floor raised on the free tier, a ceiling lowered on the top tier, and a retention policy replaced outright. Every one of them was recorded at the time — in the changelog, and in the internal document that exists specifically to explain why those numbers are what they are. None of it reached the page a customer reads.

That is the part worth sitting with. The discipline was not missing. It was pointed inward. Documentation drift is not a writing problem or a diligence problem. It is a missing test.

Your internal notes are published

The operation descriptions came from the route docstrings. Docstrings in my codebase carry maintenance notes: shorthand references to past reviews, reminders that mean something to me and nothing to anyone else. FastAPI does not distinguish between a note to your future self and a sentence for someone deciding whether to integrate. It publishes both.

A docstring serves the maintainer. A description serves the buyer. One string cannot do both jobs, and when you make it try, the buyer gets the maintainer's version.

Why response_model was the wrong fix

The obvious repair for an empty response schema is to bind a Pydantic model and let the framework generate the shape. I planned to do exactly that. It would have been the most expensive mistake in the whole exercise, and it is worth walking through slowly, because the trap is not obvious until you write the two lists side by side.

I had a model. It had existed for months, it was accurate when written, and it declared five fields:

contract
metadata
sections
client_info
analytics

My published response documentation lists eleven top-level keys:

schema_version
org_id
plan
generated_at
contract
metadata
score
sections
analytics
trend_analysis
client_info

The model was not wrong so much as old. Six keys had been added to the payload over time by the layers that assemble it, and nobody had gone back to the schema, because nothing depended on the schema. It was a type hint that had quietly stopped describing anything.

Here is the part that matters. response_model does not describe a response. It filters one. Bind that model and FastAPI does not warn you that six keys are undeclared — it removes them. schema_version, org_id, plan, generated_at, score and trend_analysis would have disappeared from every reply, silently, in a commit whose message said the documentation had been improved.

That is a contract change shipped as a docs fix. It is the sort of thing you find out about from a customer, six weeks later, when their integration has been dropping a field it needed and neither of you can point at when it stopped arriving.

There was a second problem underneath the first. Some item-level fields are conditional. Evidence and confidence are only present when the calling plan includes them, and the published contract already advertises this: it carries explicit capability flags so a client can tell which fields to expect. Other fields appear only when the underlying data supports them at all. A single model represents that by declaring nearly everything optional — and a schema in which every field is optional has told the reader almost nothing. It says these keys may occur. That is strictly less information than one concrete example of a real response for a specific plan, which says here is what you get.

So the fix was not a model. It was a written summary and description for every operation, plus named examples per plan, each one checked in the test suite against what the producing code actually emits.

I would still bind a response_model on a new endpoint, where the model and the payload are born together and stay together. On an endpoint that has been shipping for a year, reach for it only after you have diffed its fields against a real response. If the two disagree, you have learned something important, and it is not that your documentation needs work.

If you take one thing from this: response_model is a filter, not a description. When your model and your payload disagree, binding it does not document your API. It changes your API.

The advice I ignored first and should have taken first

The reviewer's third point was that factoring documentation is usually a mistake, and that repetition in reference docs is often necessary because you never know where someone starts reading.

I had already done the opposite. I had pulled shared sentences into one place, because repeating them felt sloppy. It read well as a document and badly as a reference. Nobody arrives at your API docs on page one. They land on the endpoint they need, from a search result or a link in a support thread, and that page has to be complete on its own. Every sentence I had factored out was a sentence some reader would now have to go looking for, in a document they did not want to read.

DRY is a principle about code that has one reader at a time. Reference documentation has one reader per page.

Make the docs fail the build

Everything above is drift, and drift is only fixable once something is checking. Five things did the work:

Pick a definition and name it. I had three places describing my public surface: the allowlist in code, the docs page on the site, and the reference document in the repo. Three descriptions, no authority. The allowlist is now the definition, and the other two are checked against it.

Fail the suite when the other two disagree. Two tests compare the customer-facing document and the docs page against the allowlist. Adding an endpoint without documenting it now breaks the build. So does documenting one that does not exist.

Render facts from the code. Every per-plan number in the published description is now generated from the same constants the API enforces, at schema-build time. Not copied. Generated. The document cannot drift from the code because it no longer contains an independent copy of it.

Reject internal language automatically. A regex fails the suite when a published description contains the shorthand my docstrings use. It cannot judge whether prose is good, but it can guarantee that a note to myself never reaches a customer again.

Check every example against real output. Each published example is compared with what the producing code actually emits. An example is a claim about your product, and an unverified claim is a liability, especially the ones that look authoritative.

One habit made all of this trustworthy: I broke each test on purpose before believing it. Change a number, watch the suite fail, put it back. A documentation test that has never failed is decoration.

The parameter was lying too

The day after I shipped all of that, I found the last one. My API takes a query parameter that selects the shape of the response contract. It is named like a version, which is how the confusion started.

A default request answered with one version at the top level of the payload and a different one inside the contract block. Same response. Two values. And one of the values the parameter accepts had never appeared in any response the API had ever returned.

Nobody had complained, because nobody had looked closely enough. That is not reassurance. That is the absence of readers.

While fixing it I also found three curl examples in my usage guide pointing at a hostname that does not resolve. Copy them as printed and all three fail. I do not know how long they had been wrong, and that is its own answer.

Documentation drift does not stop at prose. It reaches the parameters, the examples, and the hostname in the snippet someone pastes into a terminal in the first ninety seconds of evaluating you.

What to check this afternoon

None of this needs a project. Open your own published docs and go through it:

1. Does any successful response render every field as string? Your 200 has an empty schema.
2. Are your summaries real sentences, or your function names with spaces inserted?
3. Do your descriptions come from docstrings? Read one as a stranger.
4. Does any number in your docs also exist in your code? One of them is stale. You do not know which.
5. List your model's fields and your real response's keys in two columns. Do they match?
6. Do endpoints that return files declare a file content type, and their failure codes?
7. Copy one curl from your docs and run it, unchanged, in a clean shell.

The seventh takes ten seconds and is the one I would do first.

My thanks to James Higginbotham of API Developer Weekly, who read a stranger's API and told him the truth about it. The three corrections in this article are his. The work of finding out how far each one reached was mine.

The public API described here is documented at /api-docs, and the scoring behind it at /methodology.