How to test retry logic and API failures deterministically

Why retry code is almost never tested

Every serious API client has retry logic, backoff, and timeout handling. Almost none of it is covered by a test, for a simple reason: you cannot ask a production API to fail. So the code that runs only during an outage is the code that has never run.

The usual workarounds each give something up:

• Mocking libraries test your mock’s idea of a failure, not the failure. They cannot produce a socket that dies mid-frame or a body that stops short of its Content-Length.
• A local proxy with fault rules works, but it is infrastructure to install, configure, and keep working in CI.
• Chaos engineering tools fail randomly. Random failure makes a flaky test, and a flaky test gets marked skip. This is precisely why nobody runs chaos in CI.

One parameter

Add scenario= to any request. Nothing fires without it — ever — and the fault appears in the URL that produced it, so a confused developer reading their own logs can see what they asked for.

curl -i 'https://cuckootrade.com/api/v1/alpaca/v2/stocks/bars?symbols=AAPL&timeframe=1Day&scenario=flap:2'

Run that three times. The first two return an error; the third returns bars. Faulted responses carry Cache-Control: no-store and echo X-Cuckoo-Scenario: flap:2, because a fault lies about a moment and must never be cached as though it were history.

The HTTP effects

Effects combine with a comma: scenario=slow:2000,status:503 hangs for two seconds and then fails. Units are optional but readable — slow:500ms and slow:500 are the same thing.

# a plain 503, in Alpaca's error shape
curl -i 'https://cuckootrade.com/api/v1/alpaca/v2/stocks/bars?symbols=AAPL&scenario=status:503'

# a two-second hang, to trip a client timeout
curl -i 'https://cuckootrade.com/api/v1/alpaca/v2/stocks/bars?symbols=AAPL&scenario=slow:2000'

# a body that stops halfway through, with a truthful Content-Length
curl -i 'https://cuckootrade.com/api/v1/alpaca/v2/stocks/bars?symbols=AAPL&scenario=truncate'

The error shape follows whichever provider surface you called: Alpaca’s {"code","message"}, Polygon’s {"status":"ERROR",…}, and Alpha Vantage’s "Error Message" key. The one deliberate exception is status:CODE on the Alpha Vantage surface: that API normally reports errors as HTTP 200, but you explicitly asked for a status code, so the request wins over the mimicry.

The test worth keeping

Most fault tests assert that your code fails correctly. flap asserts that it recovers — a client with working retry passes where one without it fails. That is a green test, and green tests survive.

import httpx
import pytest
from tenacity import retry, stop_after_attempt, wait_fixed

URL = "https://cuckootrade.com/api/v1/alpaca/v2/stocks/bars"
FLAP = {"symbols": "AAPL", "timeframe": "1Day", "scenario": "flap:2"}


def fetch_once():
    r = httpx.get(URL, params=FLAP)
    r.raise_for_status()
    return r.json()


@retry(stop=stop_after_attempt(4), wait=wait_fixed(0.2))
def fetch_with_retry():
    return fetch_once()


def test_retry_recovers_from_a_flapping_endpoint():
    assert "AAPL" in fetch_with_retry()["bars"]     # got through on attempt 3


def test_without_retry_it_really_does_fail():
    with pytest.raises(httpx.HTTPStatusError):
        fetch_once()

Breaking a stream instead

Streaming clients fail in ways request/response clients cannot, and the SSE endpoint has its own effects for them:

curl -N 'https://cuckootrade.com/api/v1/stream?symbols=CUCKOO&scenario=drop:20s'

drop and truncate are a deliberate pair. A dropped socket is the easy case: the connection ends and a client reconnects. A truncated frame on a live connection is the one that catches bugs, because the client has to resynchronise its parser without the connection ever telling it something went wrong. More on testing SSE clients →

Broken transport vs. broken data

scenario= breaks the pipe. Scenario tickers break what comes through it, and the two are kept strictly apart:

Scripted tickers never return malformed data on purpose: that would break the wire-compatibility promise, and a keyless public endpoint that serves garbage to an agent who merely stumbled onto it is a brand problem rather than a feature. Faults require you to ask.

Keeping them in CI

Because the faults are reproducible, they behave like any other fixture — no quarantine, no retry-the-test wrapper, no flakiness budget. For exact flap counts and no rate limit, run the container as a service:

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      market-data:
        image: ghcr.io/tj-miller-dev/cuckootrade
        ports: ["8000:8000"]
    env:
      MARKET_DATA_URL: http://localhost:8000/api/v1/alpaca
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/

The full CI walkthrough →

Grammar reference

scenario=effect[:value][,effect[:value]…]. An unknown name is rejected with a message listing what is valid on that surface plus a working example, because the error is read at the moment someone is stuck.

The lowercase ticker names — crash, moon, flat, gappy, halts, stale, spikey, penny, choppy — are recognised and reserved. Passing one points you at the ticker instead (symbols=CRASH); applying a price shape to an arbitrary symbol is a separate piece of work, and holding the names keeps that door open.

Full fault-injection reference →