I recently used AI to review several services that use Kafka. It found problems before notifications were sent, after database updates, and inside code that reported success.
A notification could be blocked by deduplication before it was ever sent. Kafka consumer progress could advance while background work was still running. A device state update could erase the transition needed to produce an alert on retry.
These findings had identifiable code paths and trigger conditions. They do not establish how many notifications, if any, were lost in production. AI helped investigate the code; that does not mean AI wrote the original implementations. This article describes the design problems and decisions from that review, not a claim that every fix has been deployed.
With AI doing more of the work, how do I avoid letting a bad decision turn into code? Ask it to add retries, deduplication, or parallel processing, and it can propose an implementation. Those instructions still leave a question unanswered: which outcomes are we willing to accept?
1. When can we call it successful?
In one admin backend, the code called the Kafka producer without checking the asynchronous result for that specific message. Even if the broker rejected it, the caller could continue down the success path.
In a Go backend serving Windows devices, the problem was more direct: a Kafka callback logged an error, failed to return, and then logged success. One send attempt produced contradictory records.
Those fixes can be judged against existing behavior and delivery results. They do not need a human decision for every line. But they expose a question worth making explicit: success at which stage?
Service accepts the request
↓
Kafka accepts the message
↓
Consumer finishes processing
↓
External push provider accepts the notification
↓
Device receives the notification
Any one of those steps can succeed while the workflow has not reached the point the user expects. If the interface says “Submitted,” a confirmed Kafka acknowledgment might satisfy that promise. If it says “Received by device,” we need evidence from the device too.
I have to decide what the product promises. AI has to establish what the code currently proves. That affects API responses, interface wording, wait times, and the state we need to retain. I do not want the agent to pick the easiest completion point and quietly call that success.
2. After a failure, does the system remember what it still owes?
One management API path saved a device change to the database and then published a Kafka message. If the process stopped after the database commit, the change remained, but the message needed by another service might never be sent. Without a saved record of the pending publication, a restarted process had no reliable way to recover that work.
Moving publication after the commit addressed the risk of another service reading uncommitted data. The failure window after the commit still existed.
A state-and-alert service had a different version of the problem. It examined changes in device state to decide whether to produce an alert. Its sequence looked like this:
First attempt
State A → update to B → alert processing fails
Replay the same event
Read state B → update leaves B → no change, no alert
The message was processed again, but the original alert was not recovered. The database retained the latest state, B. It did not durably retain the original A-to-B transition.
“Retry on failure” is not enough here. We need to recover the unfinished work, and rerunning the code may no longer calculate the same work.
I need to decide which work cannot be lost, how long recovery must remain possible, and what happens after that window expires. AI can then work out whether to persist a pending event, the before-and-after state, or the alert work already derived from the transition. It also has to ensure that saving this information does not introduce another crash window.
An outbox, which stores a data change and its pending publication in the same database transaction, is one technical option. I do not need to dictate every column. But I do need to answer whether this data change is allowed to happen without its downstream notification.
3. Can the next record proceed when the previous one fails?
The state-and-alert service handed records from the same Kafka partition to multiple workers. Each successful record independently marked consumer progress.
Record 10: fails, still unfinished
Record 11: succeeds, advances progress
Kafka stores a position within the partition. Committing a later position does not give the application a separate list saying “record 10 is still unfinished.” After a restart, that record can be skipped.
An obvious proposed fix is to process records sequentially. But if record 10 fails, the code continues to record 11, and then commits past it, the same gap remains. The design also needs to say when processing stops, how it recovers, and when it may continue.
That raises decisions for the person responsible for the workflow:
- Must events for the same device produce their effects in order?
- How long may later work wait for a failed record?
- If waiting cannot continue indefinitely, where is the unfinished work handed off, and who recovers it?
Throughput, ordering, and waiting time can require tradeoffs. AI can explain the options and their effects. If an established contract already settles the question, it should follow that contract. Otherwise, it should surface the conflict instead of deciding that ordering is unimportant because parallel processing is faster.
Ordering within a Kafka partition also does not automatically preserve execution order at an external API or device. The design has to cover the full boundary where ordering is required.
4. When the same work arrives again, should we run it or skip it?
The push service's deduplication bug was the easiest example to explain. After receiving a message, it first wrote a Redis deduplication record with a 24-hour lifetime. It then looked up the push token and sent the notification.
First attempt
Receive → write deduplication record → token lookup fails
Notification not sent
Replay the same message
Receive → find deduplication record → skip
Notification still not sent
This failure path had not yet entered the existing retry queue for failed sends. Saying “the service already has retries” did not cover it. A mechanism intended to stop work from happening twice was blocking work that had never happened once.
The design needs to distinguish work in progress from completed work. If two workers must not process the same event concurrently, the record granting one worker permission to try cannot also serve as proof of completion.
Moving deduplication to the completion point still leaves another question: how do we know whether the first attempt completed?
An external service may accept a command and then lose the response on its way back. A timeout tells us that we did not receive a response. It does not prove that the service did nothing.
The Google management adapter path we reviewed discarded the API's returned Operation result and lacked complete command-result reporting. That made later reconciliation harder.
A duplicate notification and a repeated device-wipe command have different consequences. Passing through Kafka does not make the same retry policy appropriate for both. For uncertain irreversible commands, my decision was:
Verify the outcome first. If it cannot be determined, persist it for manual handling and do not automatically resend it.
That decision accepts that some work may wait for a person. It also rules out automatic resubmission as a universal timeout fix. I need to choose the consequences I can accept before AI designs the recovery behavior.
Before implementation, have AI grill the human
By “grill,” I mean asking the unresolved questions that would change the design. A bare “Should I retry?” does not give me enough to decide. The agent should inspect the existing code and bring a concrete scenario:
The external service may already have accepted this command, but its response was lost. Resending may repeat the effect; stopping may leave work unfinished. Can we query the outcome? If we cannot determine it, who takes over?
Now I can see the decision I am making. Settled questions do not need to be asked again, and behavior already defined by an existing contract should not all be pushed back to me.
I would put those expectations in the Markdown instructions for the task:
# Design agreement before changing a Kafka workflow
## Inspect before asking
Trace the actual call path and record:
- Entry point, data changes, Kafka publication and consumption,
and the final side effect.
- The success condition at each stage.
- When consumer progress is committed.
- What deduplication records mean and how long they remain valid.
- Where unfinished work can be recovered after an interruption.
Reference the current revision and existing contracts.
Inspect relevant callers, guards, and recovery mechanisms.
Do not diagnose a defect from an isolated code fragment.
## Grill only unresolved decisions that affect the design
- Which work must not be lost? How much delay is acceptable?
- Which effects may be repeated, and which may not?
- How far through the workflow must ordering be preserved?
- How do we verify an unknown external outcome?
- Who takes over if reconciliation or automatic recovery fails?
- How long must recovery and deduplication information survive?
For each question, provide a concrete failure scenario,
the available options, and their consequences.
Reuse settled decisions instead of repeatedly asking.
Make routine implementation choices within established contracts.
## Turn human decisions into checkable constraints
- Success must correspond to an explicit completion condition.
- Committed progress must not skip unfinished work unless that work
has been reliably handed off under the agreed recovery contract.
- Required unfinished work must have a recoverable record
that survives the relevant interruptions.
- A failed attempt known to have produced no side effect must not
be blocked by a completed-work deduplication record.
- Handle completed duplicates within the agreed scope and window.
- Verify unknown irreversible-command outcomes first;
unresolved outcomes go to manual handling.
If new evidence conflicts with an existing decision,
explain the conflict and its impact before proceeding.
Do not silently change the product's guarantees.
Markdown preserves decisions; it does not automatically make the implementation conform to them. Changes still need evidence. I covered the limits of “the tests passed” in my previous Kafka article.
AI can help me find problems, trace workflows, propose options, and ask about failure cases I missed. My job is to understand the consequences of those options, decide which are acceptable, and turn those decisions into constraints the implementation has to satisfy.
The next time I ask AI to “add a retry,” I want it to establish one thing first: do we know the first attempt did nothing, or do we simply not know what it did?