Your retry loop has never run

A scheduled job in this repo failed with Error: Process completed with exit code 28. It had printed exactly one line before it died: Attempt 1/3. The loop it died inside was written to try three times, thirty seconds apart, so that one bad minute would not cost a collection.

The collection it was triggering had finished. The data for that window landed around 09:41 UTC, with 218 signals from 13 of 13 sources — about two minutes after the workflow had given up at 09:39:49Z. Nothing was lost that day except the ability to tell, from the run, that nothing had been lost.

There were two defects. One was visible in the log. The other had been in the file since the day it was written and had never once done the thing it was there for.

The ceiling was measuring a request

The endpoint the job calls runs the whole cycle inline — collectors, the intelligence pipeline, the narrative, the retention purge — and answers when it is done. So the workflow holds a single HTTP request open for the duration of a collection, and --max-time, which reads like a network setting, is really a budget for the work behind it. Mine was 120 seconds. The slowest cycle I have measured is around 210.

Exit 28 is curl's timeout. It was not a hanging API. It was the job refusing to wait for its own work, and then reporting that refusal as a failure of the work.

This part generalizes past my repo. A timeout is a timeout only when the thing behind it is a request. The moment an endpoint does the work inline and answers at the end — a report build, a cache warm, an import — every client-side ceiling in front of it is a duration you have quietly promised the work will fit inside. If you have never measured that duration, the number in your config is a guess wearing the costume of a limit.

The retry loop had never run

The second defect is four characters wide. Here is the shape of the step, with the parts that do not matter removed:

MAX_RETRIES=3
for ATTEMPT in $(seq 1 $MAX_RETRIES); do
  echo "Attempt $ATTEMPT/$MAX_RETRIES"
  RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$URL" --max-time 120)
  HTTP_CODE=$(echo "$RESPONSE" | tail -1)
  if [ "$HTTP_CODE" = "200" ]; then exit 0; fi
  sleep 30
done

GitHub Actions runs a run: step under bash -e. In RESPONSE=$(curl ...) the exit status of the assignment is the exit status of the command substitution, so the moment curl exits 28 the shell is finished — not that iteration, the entire step. Control never reaches the retry. The three attempts on the tin were only ever delivered in one case: curl succeeding and the API answering something other than 200, which is the case the loop was least needed for.

Which means the file had been printing Attempt 1/3 on every failed run for as long as it had existed, and the number after the slash was a claim nobody had ever collected on. I had read that line in red logs before. I read it as three attempts, because that is what it says.

The fix is one line, and it is one line because set -e steps aside for a command that is part of a list:

CURL_EXIT=0
RESPONSE=$(curl -s -w "\n%{http_code}" -X POST "$URL" --max-time 420) || CURL_EXIT=$?
if [ "$CURL_EXIT" -ne 0 ]; then
  echo "curl exited $CURL_EXIT (28 = timed out waiting for the cycle)"
  HTTP_CODE="000"
fi

Reading the file would not have found it

The YAML is correct. The loop is correct as a loop. What is wrong is an interaction between two facts that are written down in different places: GitHub runs the step under bash -e, and the shell hands an assignment the exit status of whatever produced its value. Neither document mentions the other.

Running it found it in about a minute. Cut the step's script out of the YAML, put a fake curl earlier on PATH, and run it the way Actions runs it:

mkdir -p /tmp/bin
printf '#!/bin/bash\nexit 28\n' > /tmp/bin/curl
chmod +x /tmp/bin/curl
PATH=/tmp/bin:$PATH bash -e step.sh

One line of output. Attempt 1/3, and a shell that is already gone.

The test that guards it now does exactly that, against the workflow file rather than a copy of it: it cuts the step out between two markers, substitutes the two expressions no local shell can expand, stubs curl to exit 28, and asserts that every attempt the file advertises actually prints. A test that parsed the YAML and checked MAX_RETRIES would have been green through the entire life of the bug, which is a good description of what a test can be worth.

A retry that works is a bill

Fixing it created a question the broken version never had to answer. A loop that actually retries spends time, and Actions minutes are metered. At the first numbers I reached for — 600 seconds, three attempts — a wedged API would cost 32 minutes per bad run, four times a day, for a job whose successful runs take under two.

What shipped: two attempts, 30 seconds apart, 420 seconds each. Worst case is 14 and a half minutes for a run where nothing ever answers. The ceiling is twice the slowest cycle measured, so a merely slow collection still wins; the loop is short because the second attempt is the one that has any chance of helping. A test recomputes that worst case from the file and fails above fifteen minutes, so raising one number without shrinking another stops being a thing I can do by accident.

The job timeout belongs above the loop, not below it

There is a job-level timeout-minutes over all of this, and the tempting thing is to set it tight — under the loop's worst case, as a hard stop. That would quietly delete my alerting.

A job killed by its own timeout is cancelled, not failed. if: failure() does not run on a cancelled job. The step that posts to Discord when a collection dies would go silent in exactly the scenario where it matters most: the one where nothing is answering at all, for long enough to hit the cap.

So the backstop sits above the budget on purpose, at 20 minutes against a 14 and a half minute worst case. The loop always ends on its own terms, the step exits 1, the notification fires. The cap exists for a curl that ignores its own ceiling, not for the loop. That relationship between two numbers in two different parts of the file is now an assertion, because nothing about a green run would ever tell me it had been inverted.

The other number in that file that is not true

While I was in there I checked a second claim the same file makes. It is scheduled four times a day, ten minutes before each six-hour boundary. The offset is deliberate: the top of the hour is the busiest minute on GitHub's scheduler, so a job asking for 00:00 joins the longest queue of the day.

The four runs around the failure started 2 hours 14 minutes, 1 hour 46, 3 hours 47 and 2 hours 34 after the times in the cron expression. None were skipped. My own scheduler notes said to expect 30 to 60 minutes of drift, which was a figure I had written down once and never held up against the runs sitting in the Actions tab.

That is not a defect in Actions. Scheduled workflows are best-effort under load and everybody who runs one learns it eventually. It is a defect in what I believed, and it had a visible consequence I had been misreading: the report carries a badge for the age of its data, green up to three hours and amber past that. On a six-hour cadence, amber should be unusual. With a delay that can reach three and three-quarter hours, it is ordinary — and until I lined those four start times up next to the cron, I had been reading an amber badge as something wrong with the collector rather than something ordinary about the queue.

Two claims in one small file, wrong in the same direction. Both were the number I would have quoted if you had asked me what the system does, rather than the number I would have got if I had gone and measured.

What is still open

One thing did not get fixed. The idempotency key the job sends is written when the request finishes, not when it starts, and the orchestrator behind it takes no lock. A retry that arrives while the first cycle is still running starts a second one. The generous ceiling makes that unlikely — the retry only fires after 420 seconds of silence — but unlikely is a schedule, not a guarantee.

Closing it means deciding what the endpoint owes a request that arrives mid-cycle: a 409, a 202, or the same answer as the run already in flight. That is a contract decision rather than a patch, so it is written down and waiting instead of guessed at.

Seven things to check in your own workflows

1. Grep your workflow files for VAR=$(...) where the command can fail — curl, gh, aws, psql. Under bash -e, that assignment is a step-ending statement, not a variable.
2. If one of those sits inside a retry loop, the loop is decoration until you guard it. VAR=$(cmd) || VAR_EXIT=$? keeps the failure local to the iteration and gives you the code to branch on.
3. Ask what your timeout is actually bounding. If the endpoint does its work inline and answers at the end, --max-time is a promise about the work, not about the network.
4. Get the real duration from the runs you already have. The Actions tab carries the length of every past execution; the slowest one is your floor, not your ceiling.
5. Multiply before you raise anything. Attempts times ceiling, plus delay times attempts minus one, is what one run where nothing answers will bill. Multiply again by runs per day.
6. Check that the job's timeout-minutes sits above that worst case. A cancelled job does not run if: failure(), so a tight cap buys a hard stop and pays for it with your failure alerts.
7. The ten-second one, and the one I would do first: open your last failed run and count the lines it printed. One line where you expected three is the entire symptom, and it has probably been there for months.

The first article I wrote here argued that reference documentation is a second implementation of your API — written once, by hand, against the system as it was that day, and then never compared to it again. A retry loop is the same shape. It is written on the day the failure is hypothetical, and it is never run under the conditions it exists for, because those conditions are the ones you were hoping to avoid. Mine said Attempt 1/3 in production for months. I read it every time as three attempts.

The public API described here is documented at /api-docs, and the scoring behind it at /methodology.