Flaky Tests Detection

Tell real failures from noise — and keep flaky tests from blocking CI.

Try Flaky Tests Detection Free Request a Demo

Tests Dashboard listing the flakiest tests ranked by flaky rate, with durations and a trend line.

Feature overview

Flaky Tests Detection

Flaky Tests Detection identifies non-deterministic tests — those that pass and fail under the same code and inputs — and keeps their noise out of the CI signal developers rely on. Detection runs at two layers: within-build, where a test that fails then passes on retry inside one task execution is classified FLAKY; and cross-build, where Develocity compares outcomes across separate builds that share the same input fingerprint, catching tests that flake too rarely to ever fail twice in one run. On top of detection sits outcome control — whether a flaky result passes or fails the build — and the Tests API teams use to quarantine known-flaky tests, isolating them from the gate without deleting them. And cross-build detection adds nothing to configure — it runs server-side wherever Build Scans already publish.

Works great with

Classify flakiness with high confidence

Know a test is genuinely flaky, not just failing — a fail-then-pass on retry inside one task execution is recorded FLAKY, classified by same-environment retry rather than VCS heuristics.
Capture the signal from any retry source — the Develocity Test Retry plugin, Maven Surefire/Failsafe, Bazel, and sbt all feed the same FLAKY classification, however the retry was produced.
Retries don't fire when the IDE runs tests with its own runner — so debugging locally surfaces the real failure rather than a retry-masked pass.
Stop retry from hiding flakiness by failing the build on any flaky outcome, the discipline Gradle and Spring Boot adopted internally to force a fix.

Within-build retry classifying a fail-then-pass sequence as FLAKY inside one task execution.

Catch the tests that never flake twice in the same build

Find rare flakes retry never reveals — Develocity compares outcomes across builds with the same input fingerprint, flagging a test that passes in one and fails in another.
No client change and no retries required — cross-build detection is server-side and zero-configuration for any team already publishing Build Scans.
The input fingerprint hashes declared task inputs — source, classpath, configuration, environment — so "same conditions, different outcome" is a defensible signal, not a guess.

Cross-build detection comparing two builds with the same input fingerprint and different test outcomes.

Isolate known-flaky tests from the gate without deleting coverage

Move flaky tests to a non-blocking suite instead of deleting them — they keep running, so a fix is verified the moment a test goes consistently green again.
Automate it from the Tests API — scripts read each test's outcome distribution and quarantine above a threshold.
Fix the few tests that cause most of the pain — in Develocity's test suite, 10 tests produced 52% of flaky failures, so a ranked list beats suite-wide remediation.
Verify a fix holds before promoting — retry-until-failure stress-tests a now-passing test in the same JVM, confirming stability before it rejoins the blocking gate.

Quarantine workflow — threshold-based move to a non-blocking suite and promotion back once stable.

Observability

Flakiness context inline in every Build Scan, at the moment you are debugging

See a test's cross-build flakiness right beside its outcome in the Build Scan — an additive indicator that never overwrites what actually happened in this build.
Tell environment-driven flakiness from real non-determinism using the Build Scan's console output, stack traces, and resource timeline captured for every retry.
Develocity Analytics aggregates flaky signal across every build, team, and project — the per-build record becomes an organization-wide view.

Build Scan test view showing a test's outcome with the cross-build flaky indicator beside it.

Analytics

Rank your flakiest tests and watch stability trend over time

Surface your worst offenders — and what they cost in duration — by sorting the Tests Dashboard by flaky rate; the prioritization input every Fix-it Day starts from.
Trend flakiness over any window — outcomes are stored as structured records, not rolling counters, so history never decays and a past spike stays queryable.
Catch a real product bug, not just a flaky test — an unexpected spike can expose non-determinism in the code, like a race condition, confirmed fixed when it drops.
Trend these patterns in Develocity Analytics, or query them in natural language through Develocity's Model Context Protocol (MCP) servers.