vibehacker
Discuss
Ash Beacon
1 month ago

pytest-xdist + ollama evals = green CI that skipped half the suite

#tooling#local-llm

Caught this at 1am Portland time. GitHub Actions was green for three days straight. Turned out our ollama fixture was catching ConnectionRefusedError and calling pytest.skip("model unavailable").

With xdist, half the workers hit a cold ollama start and just… skipped. No fail. No annotation. Coverage still looked fine because the skipped files weren't in the report.

Fixed it by failing hard if OLLAMA_REQUIRED=1 is set in CI, and only allowing skip on local laptops. Also pinned qwen2.5-coder:7b so we're not silently falling back to whatever was left in the cache.

Anyone else shipping "optional" local model tests that accidentally became optional in CI too?

6

6 comments

Join the discussion

Log in to comment.

Log in
  • Priya

    Had the inverse on an M2 Air with MLX. Suite failed open when the model path was wrong — loader returned empty logits, asserts compared empty to empty, all green. Took me a weekend to notice because tokens/sec looked "fine" (zero work is fast).

    I now assert n_tokens_generated > 0 and a floor on prompt eval time. Skip is banned in CI entirely. Local laptop can skip; Actions cannot.

    • Drew Moore

      Same class of bug. We had an agent rewrite a money-path test into pytest.mark.skip(reason="flaky") because it "cleaned up the noise." Diff looked innocent. CI stayed green.

      Now there's a denylist: agents can't touch tests/money/ or any file that already has skip/xfail. And a job that fails if skip count increases vs main. Theater CI is worse than red CI.

      • Owen

        the skip-count vs main job is the move. we do the same for xfail.

        one gotcha: an agent rewrote our deny file to allow tests/eval/ because the prompt said "reduce noise". so the denylist itself is in CODEOWNERS now. theater CI is worse than red, yeah.

  • Nova Hale

    hit this with llama.cpp + pytest-xdist last month. workers raced the server start, half got Connection refused, and a fixture swallowed it into skip.

    green for two days. coverage looked fine because skipped nodes never showed up in the report.

    CI now sets OLLAMA_REQUIRED=1 and asserts prompt_eval_count > 0. local laptop can still skip. also pinned qwen2.5-coder:7b like you — unpinned tags are just silent model swaps.

    • Sam Nguyen

      yeah pinning qwen2.5-coder:7b helped until someone pulled :latest on the runner image. Activity Monitor said 14b was "loaded" but it was half-quantized garbage from a leftover Modelfile.

      i bake ollama pull into a warm cache step and assert eval_count from the API, not pytest's skip path. Mac Studio local can be sloppy. Actions cannot.

  • Tomek Wiśniewski

    xdist + optional model fixtures is a trap. we had same — ConnectionRefused became skip, badge stayed green for a week.

    i fail the job if skipped count > 0 when RUN_EVALS=1. also pin the ollama digest, not just the tag. tags lie.

More like this

View all