← Back to Academy
Engineering

One 403 threw away fourteen good answers

· GatiFlow Academy

Here is one run of one stage of the GatiFlow report, at 05:50 UTC on 10 November, from its own timing line and the report it fed:

phase                contributors
anchors              15
contributor calls    15
  answered           14   (slowest 0.44 s)
  still running       1   (a 403, at 29.03 s)
lowest rate limit    4,922 remaining
rate-limit wait      none
stage timeout        30 s
talent signals       0

Fourteen calls had finished almost thirty seconds earlier. Their results were in memory. The report shipped without any of them, because one call had not finished, and the timeout wrapped around the batch did what timeouts do: it cancelled the batch. Fourteen good answers went out with the one bad one.

That bad one was not slow because GitHub was struggling. It was slow because my client kept asking a question GitHub had already answered.

Two cycles with nothing to read

The stage finds developers worth knowing about: it takes the repositories trending that cycle, the anchors, asks GitHub for each one's contributors, then reads the profiles of the most promising ones. On 9 November the 11:50 and 17:50 UTC cycles both lost it to the same 30-second timeout, and the only thing in the logs was the timeout. No phase, no call, no status. A timeout tells you that the time ran out. It does not tell you where it went.

So before changing anything I made every GitHub call record itself: path, status, attempts, seconds, the rate limit left, and one line per run with the phase it reached, written in a finally so that a run the timeout kills still writes it. The table at the top is the first run after that. It answered in one read what two silent failures had not.

The 403 came from the contributors endpoint of one public repository, lexifina/global-law, whose GitHub page showed 343,596 commits. The rate limit was nowhere near: 4,922 requests left. The run's line did not keep GitHub's message, so it took one more cycle to read it, from a client that by then logged it:

/repos/lexifina/global-law/contributors status=403 attempts=1 seconds=9.24
rate_limit_remaining=4923
error=403: The history or contributor list is too large to list
contributors for this repository via the API.

That is not a transient fault. It is a definitive statement about the repository: this list will not be served, today or on the next attempt. And GitHub takes 9.24 seconds to say it.

The client retried an answer

The request code had two branches for failure. A 403 with the rate limit at zero waited for the reset, which is right. Everything else went to raise_for_status(), into a generic except, and was tried again after 1 second, then after 2, up to three attempts. A 5xx, a dropped connection and a 403 that means this repository is too large all took the same path.

The run's line records the call still running at 29.03 seconds, not how long each attempt took. But take the 9.24 seconds GitHub needed for its single answer the next cycle as the cost of one attempt, and three attempts plus the two waits come to 30.72 seconds. The arithmetic is mine; what is on record is that the call was still running when the 30-second timeout fired. Either way, the retry did not make a failure less likely. It made a known answer three times as expensive, and the bill landed on fourteen calls that had nothing to do with it.

GitHub's own documentation makes the trap easy to fall into. It says a primary rate limit answers 403 Forbidden or 429 Too Many Requests with x-ratelimit-remaining at 0, and a secondary rate limit answers 403 Forbidden or 429 too. So a 403 from GitHub can be a rate limit, and code written by someone who has seen a 403 rate limit retries 403s. The same endpoint's reference lists 403 as a response and does not say when; the contributor-list message above is how you find out. One status code carries a temporary condition and a permanent one, and the only way to tell them apart is to read the headers and the body.

This is the part that generalizes. Every retry policy is a claim about which failures go away by themselves. A 4xx is the server telling you that the request is the problem, and the request does not change between attempts. Retrying it costs time, quota and, on some APIs, money, and on GitHub it is explicitly risky: the rate-limit page says that continuing to make requests while you are rate limited may result in the banning of your integration.

The timeout threw away the work that finished

The retry made the call slow. The shape of the code made one slow call fatal. The contributor calls ran side by side under asyncio.gather, and the whole stage ran under asyncio.wait_for with the 30-second limit. When wait_for fires it cancels what it wraps, and a cancelled gather cancels its children and raises. It does not return the fourteen results it was holding. There is no partial answer to collect: the results exist only as the return value of a call that never returns.

This pattern is everywhere, in every language: Promise.all under a timeout, a WaitGroup with a context deadline, a thread pool joined with a limit. All of them say the same thing, all or nothing, and they say it implicitly. Nobody writing a batch of fifteen independent lookups means that one of them should be able to veto the other fourteen. But that is what the code says, and it is invisible until the slowest call crosses the line.

What shipped

Two changes, one for each mistake:

  1. A 4xx is final, with two exceptions: 429, and a 403 with the rate limit at zero. Those are the cases GitHub documents as rate limits, and they still wait, with one limit of their own: when the reset is more than 5 minutes away, the client gives up on that call instead of waiting, because no stage with a 25-second budget can wait that long. Everything else in the 4xx range is tried once, and GitHub's message goes into the log line, because that message is the diagnosis.
  2. The stage keeps its own deadline, 25 seconds, inside the orchestrator's 30. Each phase runs its calls until that deadline, keeps every result that came back, cancels whatever is still running, and returns what it has. The timing line counts what was cut, and a run that cut anything logs as a warning.

The first matters less than it looks. Even tried once, that 403 costs 9.24 seconds, and the next slow call will have a different cause: a large profile, a slow region, a server having a bad minute. Classifying statuses removes one way of being slow. The deadline is what makes being slow survivable. If I could keep one of the two, I would keep the deadline.

The code, and the test that reproduces the case

This is the minimum version, without the logging. The classification is small enough to read in one breath, and it is the part to argue with if your upstream documents its errors differently:

def is_final(status, remaining):
    # a 4xx is the server's answer; only a rate limit is worth waiting for
    if status == 429:
        return False
    if status == 403 and remaining == 0:
        return False  # primary limit: wait until x-ratelimit-reset
    return 400 <= status < 500

And the batch, run against a deadline instead of under one. It returns one entry per call, in order: the result, the exception the call raised, or None for a call that was still running when time ran out. And it says how many were cut, which is the number you want on a dashboard:

import asyncio


async def until_deadline(coros, seconds):
    # run the calls side by side; keep what answered, cancel the rest
    tasks = [asyncio.ensure_future(c) for c in coros]
    if not tasks:
        return [], 0
    done, pending = await asyncio.wait(tasks, timeout=seconds)
    for task in pending:
        task.cancel()
    await asyncio.gather(*pending, return_exceptions=True)
    results = []
    for task in tasks:
        if task not in done:
            results.append(None)  # still running at the deadline
        elif task.exception() is not None:
            results.append(task.exception())
        else:
            results.append(task.result())
    return results, len(pending)

Three details carry the weight. asyncio.wait with a timeout returns at the deadline without cancelling anything, which is the opposite of wait_for, so the cancelling is explicit and in one place. The cancelled tasks are awaited before returning, so no request outlives the function and keeps spending quota in the background. And an exception is returned, not raised, so one call that fails fast cannot take the batch down either. The shipped version adds one more thing: if the caller itself is cancelled, every call is cancelled too, as gather would have done.

The test is the 10 November run, scaled down a hundred times so that it takes a third of a second: fourteen calls that answer in 4.4 milliseconds, and one 403 that the old client tries three times, each attempt taking what GitHub's single answer took. It first runs the batch the old way, to pin the failure, then the new way:

import pytest


def test_a_4xx_is_an_answer_except_a_rate_limit():
    assert is_final(403, remaining=4922)  # the 10 November call
    assert is_final(404, remaining=4922)
    assert not is_final(403, remaining=0)  # rate limit: wait for the reset
    assert not is_final(429, remaining=4922)
    assert not is_final(502, remaining=4922)


def test_one_slow_403_does_not_empty_the_batch():
    # 10 November 2026, scaled 1:100 (30 s timeout -> 0.30 s)
    async def fast(i):
        await asyncio.sleep(0.0044)  # the slowest of the 14 took 0.44 s
        return [f'login-{i}']

    async def retried_403():
        for attempt in range(3):  # the old client: three tries
            await asyncio.sleep(0.0924)  # GitHub's 403 took 9.24 s
            if attempt < 2:
                await asyncio.sleep(0.01 * 2**attempt)  # waits of 1 s, 2 s
        return None

    def batch():
        return [fast(i) for i in range(14)] + [retried_403()]

    async def all_or_nothing():
        return await asyncio.wait_for(asyncio.gather(*batch()), timeout=0.30)

    with pytest.raises(TimeoutError):
        asyncio.run(all_or_nothing())  # and the 14 lists go with it

    results, cut = asyncio.run(until_deadline(batch(), 0.25))
    assert cut == 1
    assert [r for r in results if r] == [[f'login-{i}'] for i in range(14)]

The pytest.raises matters as much as the last two lines. It pins the old behaviour next to the new one, so the test cannot drift into a case where all-or-nothing happened to be harmless, and a reader sees the size of what is being refused: not one missing list, all fifteen. Every Python block on this page runs in this site's own test suite, against this page's text, and both functions are checked against the ones in production.

The first cycle after

At 11:50 UTC the same day, with both changes deployed: the 403 was tried once and logged with GitHub's message, the stage reached the end and returned 38 talents in 25.01 seconds. Fourteen of fifteen contributor lists, 52 profile lookups that all answered, and 51 repository lookups, of which 47 answered and 4 were cut at the deadline.

Read that last number twice, because it is the honest part. The run used its whole budget. The 403 was gone and the stage still took 25 seconds, now in the profile phase, where 51 calls summed 51.27 seconds and the slowest took 4.21. The deadline did its job, and four candidates were judged on less information than the others. A deadline converts a failure into a degraded answer. It does not make the degradation free, and the cut count is in the log so that it is a number someone watches rather than a cost nobody sees.

What is still open

Two things, and I would rather list them than let the fix look complete. First, GitHub says a secondary rate limit can also answer 403, with requests still remaining, and that when a retry-after header is present you should wait that many seconds before retrying. My client does not read retry-after. Under the new rule a secondary-limit 403 is final, so the cost is one missing list for that cycle, not a stalled stage, and for a batch that runs every six hours I accept that. For a client where one call is the whole job, I would not.

Second, nothing remembers a final answer. If that repository trends again, the next cycle asks again, and GitHub's single answer took 9.24 seconds of a 25-second budget to say something the stage already knew. The right fix is to remember a final answer for a while instead of asking again each cycle. It is not shipped. The deadline makes it a cost rather than an outage, which is exactly why it has been easy to leave for later.

Six things to check in your own clients

  1. Find your retry policy and write down which statuses it retries. If the answer is everything that raised, a 400, a 403 and a 404 are being retried, and each one costs the full backoff for an answer that will not change.
  2. For every upstream, read how it documents its rate limits. Where a limit shares a status with a permanent error, as 403 does on GitHub, the retry decision has to read the headers, not just the code.
  3. Log the body of an error response, cut to a line. The status says that something failed; the message usually says what, and on 10 November it was the whole diagnosis.
  4. Look for every timeout wrapped around a batch: gather, Promise.all, a joined pool. Ask what happens to the calls that finished when it fires. If the answer is that they are lost, one slow call decides the result of all of them.
  5. Give the batch a deadline below the outer timeout, keep what answered by then, and log how many calls were cut. A cut count that is usually zero and sometimes four is a metric. A stage that is sometimes empty is an incident.
  6. The ten-second one: when a timeout fires, can you say from the logs which call was still running? If not, time every call before you change anything. Two cycles of mine failed without a trace; the third, with timing, told me the cause in one line.

The second article I wrote here was about a retry loop that had never run. This one is about a retry loop that ran exactly as written, on the one kind of failure it should never have touched. Between them is the whole job of a retry policy: it is a list of failures you believe will go away by themselves, and every entry on it is a claim about someone else's API. A 4xx is that API telling you the claim is wrong. Believe it the first time, and do not let one call that is still arguing throw away the fourteen that already agreed.

Free 5-day course

Trend Reading 101

Five lessons by email, one per day, on how technology signals are measured: what an indicator can and cannot represent, how a composite score is built, and what a report does not say. Each lesson states its objectives, works through a documented case, cites its sources, and ends with an exercise.

We respect your inbox. Lessons only — no spam, no third-party sharing. What we store, and for how long, is in the privacy policy.

Tell me I am wrong

Corrections, the version of this you have lived through, or what you would like covered next. It reaches me directly and is never published. It is kept for two years so it can be read and answered; the privacy policy has the details.

0/2000


The public API described here is documented at /api-docs, and the scoring behind it at /methodology.