The PostgreSQL integration tests in my SQL question-answering project passed. But the test setup had added isolation rules that were missing from the product’s migrations.

The requirement was straightforward: company A must not see company B’s data.

The tests created their own database setup, including the rules they needed. Company A couldn’t see B’s data there. The test had supplied the very protection it was supposed to check.

We changed the tests to use the product’s migrations. Then we compared the old and new rules in a test database, keeping the read permissions fixed. With the old rules, the query returned another company’s data. With the new rules, it was blocked.

That reproduced the problem in a test environment. It did not establish that production data had leaked or that the fix had been deployed. Those are separate things to verify.

I used to ask AI to run the tests before telling me a task was done. This investigation made me look more closely at what those tests were proving.

Show me the failure

When AI fixes a bug, I want to know whether the test catches it while the bug is still present.

Take a simplified order query. Companies A and B each have an order. A request made as company A should return A’s order and exclude B’s. Checking only that B is absent is too weak: a broken query that returns nothing could pass.

In an isolated test environment, we could deliberately remove the protection. If B’s order appears, the test should fail. Restore the protection, and it should pass. A database connection error would not count as detecting this bug.

I also look at how a failing test became green. Did AI fix the implementation, remove an assertion, or change the expected result to match its code? If the requirement hasn’t changed, changing the answer doesn’t resolve the bug.

For this example, I’d put the acceptance criteria in the task before asking AI to implement it:

  • Build the test database using the product’s migrations.
  • Query as company A and verify that A’s order is returned and B’s is excluded.
  • With the isolation protection deliberately removed in an isolated test, verify that the test fails because B’s order appears.

Those expectations come from the access rule. They should hold regardless of how AI writes the query. Otherwise, it can write an implementation and a matching test that are both wrong.

Did the new tests actually run?

Another project had four standalone scripts for testing LLM behavior. CI only collected files that matched a particular path and naming pattern. Those four didn’t match, so the rest of the suite could pass without running them.

We added the missing checks to the automated suite. Now I ask AI to open the results and show me that the tests it just added actually ran.

Run the connected path

A database test still leaves the path from login to query unchecked. With browser control available, I can ask an agent to log in to a test environment and open the order page. Company A should see its own order, and the API response must not include B’s data. Opening a page without an error doesn’t prove that.

I don’t require TDD for every change or aim for 100% coverage. For changes to critical flows or behavior across services and devices, I want the affected path exercised in a safe environment, including the relevant failures.

If the path includes Kafka or Redis, the expected behavior needs to be agreed on before implementation too: what happens on duplicate delivery or interrupted processing, and how stale may a cached value be? Then the connected test needs to check those outcomes. If a service or configuration hasn’t been tested, I want that recorded.

I investigated and fixed these issues with AI; that doesn’t mean AI wrote the original defects. I’ve added these checks to my Codex and Claude rules, though I still need to verify that each project follows them.

When I review the next AI-assisted fix, I’ll ask for three things: the new test in the execution results, the product setup it exercised, and evidence that it catches the original defect. If one is missing, I’ll record what still needs checking before calling the fix verified.