What stayed with me after this MQTT incident was how a slowdown in one place could spread through the rest of the system.

Devices were not receiving remote control commands. Looking through EMQX and DIS logs, I found that authentication requests were arriving, but many answers came back after EMQX's ten-second timeout.

DIS had two tasks with 0.25 vCPU each. I estimated their combined capacity at roughly six logins per second, while devices were sending around fifty. Each login needed a bcrypt check. The CPU saturated quickly. DIS was still working, but the broker had stopped waiting.

How a CPU problem reached the database

The registration flow computed the bcrypt hash inside a database transaction. It held a connection throughout that computation. As the CPU became busier, hashing took longer, and database connections stayed occupied longer too.

DIS did not have an explicit upper bound on open database connections. It kept opening them until PgBouncer began rejecting new clients:

no more connections allowed (max_client_conn)

There are two kinds of connections here. DIS's pool manages connections from the application to PgBouncer. PgBouncer lets those clients share connections to PostgreSQL. Forty DIS connections do not necessarily mean forty PostgreSQL connections; that depends on the pooling mode and configuration.

When authentication failed or timed out, devices tried again. Edge was retrying every twelve seconds without backoff. New requests arrived before the previous batch had cleared.

Insufficient CPU
  → slower authentication
  → registration transactions hold connections longer
  → PgBouncer rejects new connections
  → device login fails
  → devices retry
  → more CPU pressure

Increasing the pool alone would not resolve that cycle. The CPU could still do only so much work. More connections might simply let more requests wait together.

Reducing the work entering the service

I first changed EMQX authentication pipelining from 100 to 1 to reduce outstanding requests on each HTTP connection. This did not restrict the whole broker to one device at a time: other HTTP connections could still operate.

The change did not add CPU capacity, but during the incident it allowed more authentication answers to arrive before the timeout.

I then bounded bcrypt concurrency and its waiting queue, limited the database pool, moved hashing outside the transaction, and added a four-second request deadline. A bcrypt operation that has already started cannot be interrupted just by cancelling its context, so I also adjusted the HTTP write timeout to default to eight seconds, leaving room to send the response.

DIS returns 503 for overload or temporary database unavailability, with separate log reasons for overload, deadline expiry, and backend unavailability. Genuine application defects still need attention, and an unknown device must not automatically be classified as overload. A 503 alone does not guarantee a device retry; EMQX's authentication chain and client behavior also matter.

A bounded pool still had a surprise

Initially, the open-connection limit was forty, but the idle limit was ten. Open connections include both busy and idle connections. An idle connection is already established and ready for another request.

At peak traffic the pool grew to forty, then closed surplus connections as requests finished. The next burst reopened them. PgBouncer takes time to clean up closed clients, so that churn could still briefly exceed its client limit.

I changed the idle default to follow the open limit. Established connections could remain available for reuse, with idle connections reclaimed after five quiet minutes. This does not open forty connections at startup; it allows connections created on demand to stay ready.

I also fixed an Android registration path that already held a transaction connection but requested a second connection from the pool. Once the pool is bounded, concurrent requests can each hold one connection while waiting for another. Using the existing transaction for that read avoids this deadlock pattern.

Separating the test result from production recovery

The recorded DEV test with 200 virtual users showed HTTP 500 counts falling from 5,481 to 29, and timeouts from 33 to zero. Those are development test counts, not a production error rate.

Production recovery also included DevOps increasing DIS CPU capacity and PgBouncer pools. Device backoff, reconnection, and credential recovery required separate coordination and verification.

This incident changed how I look at request paths. EMQX, DIS, and PgBouncer each have their own capacity and timeouts, but they share the consequences of waiting. My changes separated those waits: CPU work no longer held a database transaction, excess work had a limit, and requests that could not finish had a way to leave.