Testing And Verification

A workflow can advance through every stage and still produce a broken feature if its checks miss the defect. Verification connects the agent's claim of completion to evidence you can inspect. Tests provide some of that evidence; code review and trying the feature through its real interface answer other questions.

The example bug in Workflows / Pipelines changed the order of saved steps. A useful check must detect that change. A build that succeeds or a test that merely confirms the save function returned a value cannot establish that the steps reopen correctly.

Write Down The Expected Behavior First

Derive test cases from the requirements before using the implementation as a guide. This gives you a basis for noticing when the code and its tests share the same mistaken assumption. Given, when, and then are a simple way to describe a test case in ordinary language.

These describe observable results. Check the details against the planning decisions on duplicate names and updating by identifier. If a requirement is still ambiguous, resolve it before encoding an arbitrary choice into a test.

Exercise The Path The User Takes

A unit test can check a small function like validating an empty pipeline. An integration test can check that saving and loading preserve the same data. An end-to-end test can open the editor and save through the interface, then reload and reopen the pipeline.

These checks cover different failure points. A storage function may work while the Save button never calls it. A component test may show ordered steps while the loading code sorts them by name. Include a test through the actual connection where the reported failure occurs.

Be explicit about test substitutes. A mock is a stand-in for part of the system. It is useful when you need a controlled response such as a storage failure. If a persistence test replaces both saving and loading with mocks that return the expected data, it has bypassed the behavior it should check.

Run the regression test against the buggy version and confirm it fails for the expected reason. Then run it against the fix. A test that passes on both versions has not demonstrated that it detects this defect.

Check Whether The Tests Can Catch A Mistake

Agents can write tests that match their implementation while missing the intended behavior. Inspect the assertions, especially when the same agent wrote both. Check whether they compare the result with an independently stated expectation or simply repeat a calculation from the code.

Mutation testing probes this weakness by deliberately changing the source code and running the tests again. Temporarily reverse the loaded steps in an isolated copy for the pipeline example. The order-preservation test should fail; if it still passes, investigate whether the test reaches that code and checks the sequence.

Use mutations selectively around behavior you need confidence in. Some changes preserve observable behavior. When a mutation survives, inspect what changed and whether your assertions should have detected it. Remove the mutation and rerun the relevant checks on the code you intend to keep.

Inspect test and configuration changes as part of the patch. Deleting a failing assertion or skipping a test can make a run green without fixing the problem. So can disabling a check. Changes to the acceptance criteria deserve an explicit decision.

Review Without Supplying The Answer

Give the reviewer the requirements and current changes, plus the test evidence. Ask for concrete discrepancies and the code or reproduction that supports each finding. Allow “no issues found” so the reviewer need not invent a problem.

A fresh review session avoids carrying over the entire implementation discussion, but it can still repeat the same assumptions or miss a defect. Check its findings. Treat the review as another source of evidence with its own limitations.

Run quick checks while editing. Then run the broader checks relevant to the completed change. Record the commands and results, plus the code revision. If the patch changes after review or you combine it with other work, rerun the affected checks because the earlier result describes different code.

Keep the completion report precise. It should distinguish what passed from what failed, and what could not run. If the storage service was unavailable, preserve that as an unresolved verification step; a successful build does not cover it.

Exercise 2: Catch A Defect On Purpose

Have an agent add a small feature to a working project and write its tests in the same session. Before reading those tests, write the expected behavior in given, when, then form. Then copy the project to a separate directory and break the feature there by reversing a list or flipping a comparison.

Run the agent's tests against the broken copy and count how many defects get through. For each assertion, check whether it compares the result against the written expectation or repeats the calculation the code already does. Finish by adding a test that fails on the broken copy and passes once the original code is restored.

September, 2026