What stayed with me from this device-log incident is that two decisions, each reasonable on its own, only caused trouble after sitting together for a few months. Both were mine.
It started with the console's device log pages returning 500. Sentry showed Elasticsearch timeouts: single digits a day in early October, then 61 on 10/07 and 111 on 10/08. Traffic hadn't changed, and neither had ingest, still about 2.5 million documents a day.
I replayed the same query against the production cluster. It took 20 seconds. UIS waits 15 seconds for ES, so users saw a timeout. The ES response also carried a shard failure:
CircuitBreakingException: [fielddata] Data too large,
data for [_id] would be [1.6gb], which is larger than the limit of [1.5gb]
Why sorting on _id eats memory
Ordinary fields get doc values at write time, so a sort reads them straight from disk. _id has no doc values. To sort on it, ES loads the _id of every document in the shard into heap as fielddata, not just the matching ones, and keeps it there until someone clears it or the index is deleted.
ES guards that memory with a circuit breaker: 40% of heap per node, 1.5 GB here. Past that, queries are rejected.
The index being written at the time was at the end of its 14-day period: 39.5 million documents, 7.4 GB. The _id fielddata for its single shard came to almost 1 GB, and the cold load was those 20 seconds. The previous period's index still had 590 MB sitting on the same node. Together they crossed the line.
Both decisions were mine, in June and July
In June, ES heap sat at 70% while CPU was idle. The cause was device clocks. A device whose clock had drifted to 2036 uploaded logs, and Logstash created a 2036 index for them. Hundreds of nearly empty indices like that, each with 5 primary and 5 replica shards, added up to thousands of shards, each holding a slice of heap.
I changed the index template to 1 primary shard and let retention delete the old ones. No existing data was touched, and the heap target was met.
In July, deep pages in the log views were hitting the 10,000-result limit, so I moved pagination to search_after. That needs a field to break ties between documents with the same timestamp, and I used _id. I checked whether pages would skip or repeat entries. I didn't think about memory. ES 7.x emits a deprecation warning for sorting on _id; I never looked.
The two tickets were opened on the same day. I didn't read them together.
Later I worked out what 5 shards would have meant. 5 primary plus 5 replica is 10 copies across 3 nodes, about 3.3 per node at 200 MB each, plus about 390 MB spread over from the previous period: roughly 1 GB per node, a third below the 1.5 GB limit, loaded in parallel across 5 shards. With 1 shard, two nodes each carry the full 1 GB, and one node reads it end to end.
The total is the same. The difference is where it lands. The _id sort times data growth is the cost; 1 shard decides which period and which two nodes cross the line. This was computable in July. Nobody computed it.
In September I saw it, and measured the wrong thing
In mid-September I inventoried the production indices. Config says there should be 3 device-log indices at any time. There were 69, but only 73 shards. One index had 5 shards; the other 68 had 1 each, all created after June. Retention had in fact removed the hundreds of old ones.
Of the 68, 61 came from devices uploading backlogged logs: Logstash wrote them into old period indices by their old timestamps, recreating indices retention had just deleted, and retention deleted them again the next day. I only found that cause in October.
What retention genuinely missed was the 5-shard one: dls-device-logs-2026-p%{period}. Logstash couldn't compute the period, the placeholder was never substituted, and the literal string became the index name. The cleanup job derives age from the period number in the name; no match, skip. Its timestamp field was mapped as text, so any sort by time failed on it. UIS queries logs through a pattern that matches every index, so every query hit it: 13,620 shard failures in ten days.
I measured the heaviest device then, 2.4 seconds cold, and wrote "this is housekeeping, not latency." I was measuring a single query under normal conditions. Once _id took queries from 1 second to 20, each one also scanned a few dozen extra shards and failed once more on the bad index, and the noise became an amplifier. I don't have a number for how much; the search slow log was off.
The fix, and one ordering
The same-day mitigation was DevOps clearing the previous period's fielddata; queries went from 20 seconds back to 0.8. The current period still grows about 60 MB a day, so it needs clearing again in a week.
The sort is now _index plus _seq_no. Both have doc values, so nothing loads into heap. The assumption is that logs are append-only; _seq_no is stable only then. Other queries with the same sort changed with it.
Queries now hit only the indices the time range can fall in, not all of them. When Logstash can't compute a period it uses server time, so no more bad names. The bad index gets closed, then deleted.
DevOps proposed 3 shards per index, somewhere between thousands of tiny shards and one 40-million-document shard. But _seq_no is a per-shard counter; with multiple shards it collides. A globally unique field has to be written at ingest first, then shards added. The other way round, pagination starts dropping entries the moment shards split. The cross-check I didn't do in June, I did this time.
What I take from this: when changing shard count, go through the existing queries for operations whose cost scales with shard size. Fielddata is one; aggregations are another. When saying "retention will handle it," write down which names it can see. When measuring "not slow," write down the conditions. The fix is merged into QA and not yet in production; production is running on cleared memory and needs clearing again around 10/15. A merged PR is not the end of this.