Test Suite Integrity Audit
Find out why CI is green but tests aren't running: catches silently vanished tests, unexplained skips and suspiciously fast runs in test output or CI logs. Audits pasted test runner output, a CI log, a test file listing or config for three ways a suite quietly stops proving anything. First, planned versus executed versus passed counts, including a drop against a prior baseline. Second, newly skipped or disabled tests (.skip, xit, @pytest.mark.skip, #[ignore], commented-out blocks) that have no comment or tracked issue. Third, suspiciously fast or uniform run times. The report lists the three checks and ends with a verdict of CONFIRMED SILENT DROP, CLEAN or INCONCLUSIVE. It does not invent a baseline: without prior numbers it says which checks it could not run and what extra input is needed. It does not run the tests itself.
Example input
Use the Test Suite Integrity Audit on this (a sample I wrote). CI is green but a bug shipped.
CI log excerpt (fictional project "shopkit"):
> jest --ci
PASS src/cart/cart.test.js (2 tests)
PASS src/pricing/pricing.test.js (3 tests)
FAIL to-be-ignored: none
Test Suites: 2 passed, 2 total (4 planned)
Tests: 5 passed, 5 total
Time: 0.4 s
Done. exit code 0
Last week's run (from my notes): "Tests: 31 passed, 31 total, Time: 12.8 s".
Test files in repo: cart.test.js, pricing.test.js, checkout.test.js, coupons.test.js.
Snippets from checkout.test.js:
// it.skip("applies free shipping over $50", ...)
xit("rejects expired card", () => {...})
it.skip("handles tax for region EU", () => {...}) // TODO see issue #212Example output
## Test Suite Integrity Audit
### 1. Count mismatch
Planned: 4 suites | Executed: 2 suites | Passed: 5 tests (2 + 3)
FINDING — The log says 2 of 4 suites ran. The two missing suites match the repo listing: `checkout.test.js` and `coupons.test.js` do not appear in the output at all, not even as failed. Against the baseline from your notes, last week's run had 31 tests and this one has 5, so 26 tests are no longer executing and nothing failed. The visible evidence does not show why (the log has no collection or import error, and the odd line "FAIL to-be-ignored: none" is not a real runner message). Likely causes to check: a config `testPathIgnorePatterns` or `testMatch` change, an import failure in the two suites that the runner swallowed under `--ci`, or a bundler/config change that excludes them. Which tests were in those two files is not visible; the baseline of 31 is only your note, not a verified log.
### 2. Unexplained skips/disables
Only `checkout.test.js` snippets were provided, so `coupons.test.js`, `cart.test.js` and `pricing.test.js` were not checked for skips.
- `checkout.test.js` — `// it.skip("applies free shipping over $50", ...)`: commented-out test block, no comment or issue reference found (unexplained).
- `checkout.test.js` — `xit("rejects expired card", ...)`: disabled, no comment or issue reference found (unexplained).
- `checkout.test.js` — `it.skip("handles tax for region EU", ...)`: skipped, but has `// TODO see issue #212` (accounted-for; confirm the issue is still open).
The unexplained `xit("rejects expired card")` is relevant to a shipped bug if the bug involved expired cards, though this cannot be concluded from the material given.
### 3. Timing anomalies
FINDING — The suite finished in 0.4 s versus 12.8 s last week (about 30x faster), which matches the missing suites rather than a faster machine. Individual test durations were not given, so uniform 0 ms / 1 ms timing cannot be assessed.
## Verdict
**CONFIRMED SILENT DROP** — 26 of 31 baseline tests (suites `checkout` and `coupons`) stopped executing while CI stayed green; find out why those two files are no longer collected (check the jest config and run `npx jest --listTests` locally) before trusting this suite, then triage the 2 unexplained skips in `checkout.test.js`. To finish the audit, I'd need the jest config, a full local run with `--verbose`, and the contents of `coupons.test.js`.Real output from running the skill on a sample problem, shortened. The skill file itself is not published here.