Testing Kafka with AI: What Did the Passing Tests Actually Check?

When I ask AI to add tests, I also ask it to introduce the mistake those tests are supposed to catch. I want to see whether the existing tests fail. Otherwise, a green test run still leaves me wondering whether they'd notice if the code went wrong.

Trying this with Kafka gave us two different results. Swallowing an error made the rejection test fail. Changing the acknowledgement setting didn't: the rejection test still passed, and a separate unit test caught the change.

In an AI-assisted change to a kafka-go producer, we set RequireAll to wait for acknowledgements from the in-sync replicas. We also added a test to check that, when Kafka rejected a message, the calling code received the error.

The first idea was to use a topic with one replica and set the minimum in-sync replicas to two. With only one replica available, requiring two seemed like it should prevent the write.

But the local broker was Kafka 3.8.0. It uses the smaller of the configured minimum and the replication factor. With one replica, setting the minimum to two still means one; the source shows that calculation. That setup wouldn't make the broker reject the message.

We switched to a message-size limit. On a local test topic, we set max.message.bytes to one byte and sent an oversized message through the application. The test had to receive MessageSizeTooLarge to pass. A connection failure also produces an error, so checking for just any error wouldn't tell us what we'd tested. On a healthy topic, we sent a message and read it back to check that the same code could publish successfully.

While preparing this article, I asked AI to run mutation testing in an isolated copy of the code, using a single local Kafka 3.8.0 broker. We first checked that the tests passed, then changed one piece of production code at a time. The tests stayed unchanged, and we restored the code after each run.

First, we made the ordinary publish path return nil after receiving a rejection error. The rejection test failed: it expected an error and got none. Restoring error propagation made it pass again.

Next, we changed only RequireAll to acks=1, which waits for the leader. The rejection and healthy readback tests all passed. The unit test that checked RequiredAcks failed: it expected -1 (all) and got 1. After restoring the setting, both sets of tests passed again.

An oversized message can produce a rejection error with acks=1 too, so that test couldn't distinguish whether the producer waited for the in-sync replicas. It caught swallowed errors; a different test caught the changed setting. We didn't test whether data survived a replica going offline or a leader failure.

In another service, the test may need to observe a different part of the code. We use more than one Kafka package, so I ask AI to read each project's go.mod, then find where the client is created, where messages are sent, and where results are handled. With asynchronous publishing, the error may arrive later in a callback or delivery report. This table compares the documentation and source for the versions listed; I haven't run failure tests against all four clients.