Flaky Playwright AI Tests: When Self-Healing Lies

Self-healing is the most seductive feature in AI-assisted UI testing. A selector breaks, the tool quietly finds a "similar" element, the test goes green, and nobody gets paged. It feels like magic. It is often a lie.

The appeal — and the trap

The pitch is reasonable: UIs change, brittle selectors break, and re-authoring specs is tedious. So when #submit-btn disappears, an AI layer locates a button that looks like the old one and retargets the assertion. The run stays green.

The trap is that a green test is supposed to be a claim about behavior, and self-healing turns it into a claim about resemblance. Those are not the same thing. The moment your gate passes by finding *a* button instead of *the* button, it has stopped protecting you.

Three ways self-healing produces a false green

• It heals over a real regression. The "Save" button was removed in a refactor. Self-healing latches onto the "Cancel" button because it is visually adjacent and labeled similarly. The test passes. The save flow is broken in production.
• It masks a flaky test instead of fixing it. A race condition makes an element appear 1 in 5 runs. Self-healing retries against a fallback selector and reports success, so the underlying timing bug is never surfaced or fixed.
• It launders non-determinism. Each "heal" is a silent decision the agent made on your behalf. Over a few weeks, the spec no longer asserts what you think it asserts, and no diff ever recorded the drift.

Flaky is not the same as broken — and self-healing conflates them

A flaky Playwright test fails intermittently for reasons unrelated to a real defect: an unawaited animation, a network stub that resolves out of order, a hydration boundary. The correct response is to *diagnose and stabilize* — add the right wait, fix the race, pin the clock.

Self-healing does the opposite. It treats every failure, flaky or genuine, as a selector problem to route around. That is how a flaky test and a broken feature end up indistinguishable in your CI history: both show green.

A test that cannot fail for a real defect is not a test. It is a decoration.

Keep the gate honest: observe, do not paper over

The fix is not to ban automation — it is to make the runner's evidence visible and to let a defect actually fail.

1. Ground specs in recorded reality. Generate the spec from an actual session (the events that really happened) rather than from a guess about the DOM. The assertion targets what the user did, not what an element resembles.
2. Surface every repair as a reviewable change. If a selector must change, it should show up as a diff a human (or the coding agent) approves — never as a silent runtime substitution.
3. Let red mean red. When the behavior genuinely changed, the test should fail loudly, carrying the failing assertion and the console output back into the loop so the agent can decide whether it is a regression or an intended change.

This is the line CUIT draws. @cuit/spec-gen builds the spec from the recorded session, CUIT runs it for real, and a failure is reported as a failure — with the evidence attached — not quietly healed into a green you cannot trust.

The bottom line

If your flaky Playwright AI tests are passing because something healed them, you have traded a loud, fixable problem for a silent, expensive one. The whole value of a UI gate is that it tells the truth when behavior breaks. Self-healing that hides the break is worse than no test at all, because it costs you the one thing a test exists to provide: trust.