Elasticsearch in Production Is Not a YouTube Tutorial

Every YouTube tutorial makes Elasticsearch look easy. Pull the image, point Kibana at it, watch the dashboards fill in, done in fifteen minutes. I believed that. Then you run it for real, on a shared box under a live ingest feed, and it teaches you everything those videos leave out. The tutorial ends exactly where production begins, and this is a tour of what lives on the other side of that line.

Here is the setup. A single-node Elasticsearch 8.15.3, fed continuously from Kafka through a Kafka Connect sink (the logs-camera_events-* dataset, ~2M+ documents a day), sharing one 4 vCPU / 7.8 GiB / 58 GB Docker host with Kibana, Kafka, and Filebeat. Same staging stack behind the CCTV ingestion pipeline. Then the query service started hanging and the node stopped responding.

The obvious reads were "the node is undersized" and "we're out of memory." Both were wrong. Elasticsearch itself was healthy. Everything that failed lived in the setup around it, and a single symptom, "Elasticsearch is slow," was hiding it.

The reframe: the node was healthy, the setup wasn't

One number reorganized the whole investigation. Once the node was breathing again, cluster health came back green: 93 of 93 shards active, no read_only_allow_delete block, and an Elasticsearch data footprint of 4.2 GB. On a 58 GB disk that had just hit 93%. Whatever filled that disk, it was not Elasticsearch.

That pointed the investigation away from Elasticsearch entirely. The node was not undersized and it was not out of data. What had failed was everything around it: how the data was modelled, how the host's storage was managed, how the service was deployed. Stand a stack up like a tutorial and those gaps stay invisible, right up until they collide.

How the diagnosis actually went

This resolved fast for one reason: the diagnosis was layered. Each command tested a hypothesis and threw it out before the next one, instead of committing to the first story that fit.

Step 1: df -h before anything clever

First command was disk. Not heap, not thread pools:

$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        58G   54G  4.4G  93% /

93%, 4.4 GB free, over Elasticsearch's 90% high watermark, which alone is enough to make a node thrash. Reclaiming ~17 GB of junk dropped / to 63%; the node fell back under the watermark and self-healed to green, no manual intervention. So the disk was a real trigger. But clearing it left the pegged CPU exactly where it was.

Step 2: cluster health killed the "data problem" theory

Green, all shards active, no read-only block, 4.2 GB of data. Disk used was 54 GB; Elasticsearch owned 4.2 of it. Roughly 50 GB belonged to something else. That pointed the flashlight away from Elasticsearch entirely.

Step 3: hot_threads disproved "memory pressure"

The tempting explanation for a pegged node is GC pressure. _nodes/hot_threads said otherwise. search_worker threads were burning CPU inside Painless, re-parsing _source.raw.EventID per document to feed a scripted_metric aggregation, next to a Lucene merge thread from the live ingest feed. Heap sat at 39%, GC negligible. Not memory-bound, CPU-bound. The node was not starved for RAM; it was doing expensive work on every document.

Three commands, three hypotheses tested. Disk confirmed as the trigger, "out of data space" rejected, "memory pressure" rejected. What remained was a clean split, and neither half was Elasticsearch's fault: a storage gap that filled the disk, and a schema gap that pinned the CPU.

Gap 1: the schema was never shaped for the queries

Start with the data model, because that is where this begins. The whole Dahua camera payload was stored in one un-indexed raw object (enabled: false). That was a reasonable call on its own, the heterogeneous payload would otherwise cause mapping explosion and type conflicts. The mistake was never doing the other half: promoting the fields the dashboards actually query to real indexed fields.

Because the hot fields lived only inside raw, every dashboard card had to lift them out of _source at query time with a Painless runtime field. Not just the per-event id, the whole drill set: vehicle_category, snap_category, plate_color, vehicle_color, object_type, and sex. Here is what that looked like for event_id:

// runtime field, evaluated per matching document
def r = params._source.raw;
if (r != null && r.EventID != null) {
    emit(r.EventID.toString());
}

For every matching document, Elasticsearch decompressed the stored _source, JSON-parsed the whole heterogeneous raw object, pulled one field, and fed it to a scripted_metric aggregation. Across millions of documents per query, that is brutally CPU-intensive, and it throws away the entire point of columnar doc-values. The query code even labelled it correct but slow, tens of seconds over a large match. It is the kind of pattern that looks fine on a tutorial's sample data and quietly saturates a node once real data lands.

The fix (commit ed5ea00) did the half that was missing from the start. It promoted the hot fields to real indexed keywords at ingest time, and rewrote the aggregations to read doc-values instead of scanning _source:

// after ed5ea00: event_id promoted to an indexed keyword
// at ingest time, aggregations read columnar doc-values
doc['event_id']

And here is where a schema gap turned into a deploy gap. That fix was already merged to main, and had been for a while. But staging was still running an older image built before it, so the slow query pattern kept hitting the cluster as if the fix did not exist. We only confirmed the drift by shelling into the container and finding the old Painless string still sitting in the deployed code.

The container ran the mutable :main tag with no record of the commit inside it. Ask "what version is running right now?" and there was no honest answer short of reading the code in the box. Redeploying from current main (revision 659c87a, which includes ed5ea00) took Elasticsearch CPU from ~158% to ~0.7%. The code was correct and merged. It just was not running.

Gap 2: the storage was never bounded

Docker's default json-file driver keeps unlimited log history unless you set max-size and max-file. Only Filebeat had an explicit logging config. Every other container inherited the uncapped default and wrote stdout forever into /var/lib/docker/containers/*/*-json.log. Those logs, not any data, walked the 58 GB root filesystem up to 93%.

Elasticsearch then did exactly what it is designed to do. Its disk-based shard allocator climbs a three-rung ladder, and the node had climbed most of it:

At 93% the node sat between the high watermark and flood stage, about two points from flipping every index read-only. It never got there, so no data was lost and nobody had to do manual watermark surgery. The tell is worth stating plainly: stop at "Elasticsearch is out of space" and you miss that the space went to logs from unrelated containers. The failing component's own storage was nearly empty.

Gap 3: tuned for the wrong workload

Two more defaults rounded out the tutorial setup, neither fatal on its own. First, refresh_interval sat at 1 second under a continuous Kafka-sink feed. That cuts a new Lucene segment every second and drives constant background merging, which fights search for the node's limited CPU. An append-heavy log workload does not need one-second search visibility. Setting index.refresh_interval: "30s" on the live camera_events indices, adding it to the index template so rollover indices inherit it, and committing it to the repo lifted a steady background tax. _simulate_index confirmed the next rollover picks it up.

Second, nothing watched the disk. It climbed to 93% in silence, and the first signal anyone got was the outage itself. A production setup pages at 80%, long before a watermark is ever in play.

What I actually took away

The failure is usually in the setup, not the datastore

Elasticsearch held 4.2 GB at 39% heap while other containers' logs filled its disk and a query pattern its own schema forced burned its CPU. It had not failed. It was the surface every setup gap showed up on. Follow the evidence to where the work is actually happening (df, hot_threads), not to the loudest process.

One symptom can hide more than one gap

"Elasticsearch is slow" was a storage gap and a schema gap at once, colliding by coincidence. Fix one and you would leave the node still degraded and the incident half-solved. That single 4.2 GB number forced the split; without it, clearing the disk would have looked like a full fix while the CPU stayed pegged.

Index what you query, before you query it

If a field is going to be aggregated or filtered on, it has to be a real indexed field with doc-values, not something lifted out of _source at query time. A runtime _source scan is a fine escape hatch for one-off exploration on data you already have, but it must not become the steady-state path behind a dashboard card. Storing a messy, heterogeneous payload blob un-indexed is fine; querying through it on every request is not. Promote every hot field at ingest, from day one, before you build the dashboard that leans on it.

Bound what a tutorial leaves unbounded

Any container on the default json-file driver with no rotation is a slow-motion disk fill waiting to cross a watermark. Logs, disk, retention, a tutorial leaves all of them unbounded because on sample data they never fill. Cap Docker logs globally (max-size, max-file), alert on disk before the watermark, and let ILM actually delete aged indices.

A merged fix that isn't deployed doesn't exist

A performance fix can be reviewed, merged, and celebrated, and still never reach the environment that needs it. A mutable image tag with no version visibility lets a system run old code indefinitely, and nothing surfaces the drift until someone inspects the running container by hand. Track the deployed image by commit SHA, and make redeploy-on-merge explicit.

The takeaway

The headline read like a sizing failure: a datastore pegged and unresponsive, queries timing out, ingestion stalled. The reality was a healthy node standing on a setup that was never finished. The schema was never shaped for its queries, the host's storage was never bounded, and the deploy pipeline could not say what it was running. None of that is exotic. It is exactly the work a fifteen-minute tutorial ends right before.

A green cluster is the start of the job, not the end of it. Index the fields you query before you build on them, bound every resource a default leaves open, tune for the workload you actually have, and always know what version is running. Do that and the next coincidence stays an inconvenience instead of an outage.