One Million CCTV Events a Day in a 512 MB Container

We run the event-ingestion backbone for a CCTV network strung along the main roads of Java, feeding a national traffic-monitoring platform. The cameras never stop talking. On a normal day they push about a million events into our pipeline: traffic-jam signals, over-speed flags, illegal-parking detections, stream connect/disconnect notices, every one of them landing in Kafka for downstream analytics. (A million a day isn't an estimate; it's the Kafka message count.)

The ingestion service that does this is boxed into 1 vCPU and 512 MB of RAM. Not a dedicated machine, a single container, capped that small, sharing one modest VM with the entire rest of the platform: Kafka, Elasticsearch, Kibana, Kafka Connect, two APIs, two frontends, plus host-level Postgres, MinIO, and Nginx. The whole stack lives on one 8 GB / 4 vCPU box, and the piece taking a firehose of camera events gets the smallest slice of it.

This post is about why that's not crazy, what the real hard problem was (it wasn't the one people assume), and why Go and the actor model turned out to fit the shape of this problem.

What the system actually is

Before the "why," the "what," because the setup is the part people picture wrong. When you hear "a thousand-camera network," you imagine the ingestion service opening a thousand connections. It does not. The roster is effectively one NVR, a single network video recorder that aggregates ~200 physical cameras behind it. The service holds one persistent HTTP stream to that NVR, and every one of the ~200 cameras' events comes down that single pipe, each one tagged with the camera that produced it.

From there the events flow through the ingestion container into Kafka, and on to Elasticsearch and Kibana for the analytics dashboards. Here's the whole pipeline, and where the ingestion service sits in it, the smallest tenant on a shared box:

First, kill the scary number

"One million events a day" sounds like a throughput problem. It isn't. Divide it out:

1,000,000 events / 86,400 seconds ≈ 11.6 events/second

Twelve events a second is nothing. A Raspberry Pi could serialize twelve JSON blobs a second while it sleeps. If steady-state throughput were the whole story, this would be a boring service and there'd be nothing to write about.

The load test we run against the pipeline pushes it to 1,022 events/second sustained, roughly 88 times the production average, using 100 emulated cameras, and the ingestion process stays under 16% of its one capped core and under 80 MiB of its 512 MB ceiling while doing it, with zero dropped events. Raw headroom is not the problem.

So where's the difficulty? It's in the shape of the workload, not its volume, and the shape is more interesting than the headline.

The shape that actually matters

You now have the whole picture: one stream, ~200 cameras behind it, twelve events a second with room to spare. So where's the difficulty? It's entirely in that word one. Holding a single stream that carries 200 cameras inverts the problem in a way that's worth sitting with:

One connection is a single point of blindness

If that stream dies and doesn't come back, you don't lose one camera, you lose all ~200 at once. There's no partial degradation, no "most cameras still reporting." Liveness of that one connection is the whole game. This is why the heartbeat watchdog and the reconnect logic (below) are load-bearing today, not as future insurance.

A dead stream doesn't announce itself

An NVR that has silently wedged doesn't send a TCP FIN; the socket sits there looking perfectly healthy while zero events flow. "Is the connection up?" and "are events actually arriving?" are different questions, and only the second one matters.

Bursts, not averages

Twelve events a second is the mean across the day. When traffic backs up at a junction, a cluster of cameras behind the NVR can fire a rapid series of detections down that one stream while the rest stay quiet. The pipeline has to absorb spikes without blocking the reader or falling over.

A tiny resource envelope, shared

512 MB total, one core, and it's contending for a box that's also running a JVM-heavy Kafka broker and an Elasticsearch node. "Uses little memory" isn't a nice-to-have here; it's a condition of coexistence.

And one more thing, which is where Go and the actor model come in: the service is designed to scale past the aggregator model entirely. The architecture targets 1,000 cameras per node, one actor per camera, a future where we subscribe to individual cameras directly instead of through an NVR. So the code has to be correct and cheap at one connection today and at a thousand tomorrow, with no rearchitecting in between. That dual requirement, bulletproof at N=1 and cheap at N=1000, is what drove the technology choices.

Why Go

The unit of work is a persistent stream: a long-lived socket, mostly idle, blocking on a read waiting for the next multipart chunk. Whether there's one of them or a thousand, that's the textbook case for Go's concurrency model, for three concrete reasons.

Goroutines make "one blocking loop per stream" affordable at any N

The natural way to write a stream reader is a straight-line blocking loop: read a part, parse it, hand it off, repeat. Today that's one goroutine on one NVR. At the 1,000-camera design target it's a thousand of them. In a thread-per-connection model, a thousand OS threads at megabytes of stack each would blow the memory budget before the first event arrived. In an async/callback model, you'd invert the same logic into a state machine and lose the readability. Goroutines let us keep the simple blocking loop and pay only a few KB of stack each, so the exact same code that costs nothing at one connection costs tens of MB, not hundreds, at a thousand. Scaling down to 1 is free; scaling up to 1000 doesn't change the code.

One static binary, no runtime to babysit

The service compiles to a single binary run with GOMAXPROCS=1 and a GOMEMLIMIT soft cap. No JVM to tune, no interpreter, no cgo; the Kafka client (franz-go) and the camera-auth library are both pure Go. On a box already hosting two JVMs (Kafka, Elasticsearch), adding one more heavyweight runtime for the ingestion tier would have been a poor neighbour. A lean static binary is a good one.

GOMEMLIMIT turns the memory cap into a first-class control

We set a soft limit of 450 MiB under the 512 MB container ceiling. The garbage collector treats that as a wall to work against rather than a cliff to fall off, so under memory pressure the runtime collects harder instead of letting the OOM killer make the decision for us. On a shared box where the OOM killer might otherwise reap the wrong container, that difference matters.

The profiler ships in the box

net/http/pprof is mounted on the observability port, so a live goroutine, heap, or CPU profile is one curl away: no rebuild, no special build, no attaching a debugger to production. For a long-lived streaming service, being able to ask "what is every goroutine doing right now?" without a redeploy is worth a lot.

Go didn't win here because it's fast. It won because the cheap-concurrency model makes the same blocking-loop code correct and affordable from one stream to a thousand, and the deployment model, one small static binary, makes it a good citizen on a crowded box.

Why the actor model

Go gives you goroutines and channels. That's raw material, not architecture. The moment you have more than one stream writing into a shared Kafka producer, or even one stream that has to be supervised, reconnected, and observed independently of the process lifecycle, you want a discipline on top of the primitives. That discipline is the actor model, via the Ergo framework, an OTP/Erlang-style supervision runtime for Go.

The core idea maps cleanly onto the domain:

One event source is one actor.

Each camera in the roster gets exactly one CameraActor, running in its own goroutine, owning exactly one thing: that source's connection and its lifecycle. In today's deployment that's a single actor holding the NVR stream. At the 1,000-camera target it's a thousand actors, each owning one camera. Either way, the actor owns its connection, parses its events, hands them to the sink, and shares no mutable state with any sibling.

Even at one actor, the model earns its keep, because its benefits aren't only about concurrency:

The supervisor encodes a restart policy, not just a restart

The actor sits under a CameraSupervisor with a bounded restart intensity: at most 5 restarts in 10 seconds before it gives up and escalates. That's a deliberate circuit breaker. A source that's crash-looping because of a real bug should get loud and stop, not silently thrash forever burning the one CPU we have.

Fault isolation is per-source, by construction

Which is what makes scaling to 1000 safe. The one-for-one supervision strategy means one actor's panic restarts only that actor. Today, with a single NVR, that's a modest benefit. But it's the property that lets us grow to a thousand direct-camera actors without a single misbehaving camera taking down the other 999. The isolation is banked now and cashed at scale.

A single shared sink, fed by mailboxes, gives clean backpressure boundaries

Actors don't each open their own Kafka producer; they hand events through a bounded channel (256 deep) into one shared sink. One producer is where batching and per-partition ordering work best, and the message-passing boundary between "reading" and "publishing" is exactly what makes the backpressure story below tractable, at one stream or a thousand.

A note on terminology, because precision matters here: these are classic, statically-provisioned actors, materialized at startup from the roster, not "virtual actors" in the Orleans/grains sense, where actors are activated on demand and garbage-collected when idle. Our source set is known and long-lived; there's no reason to activate lazily. A fixed fleet of always-on sources maps to a fixed fleet of always-on actors under a supervisor. Same lineage (Erlang/OTP), different activation model, worth being exact about, because the on-demand machinery of true virtual actors would be pure complexity for no benefit here.

The stream: one long-lived connection, many cameras

The service subscribes to the NVR over a single long-lived HTTP connection that never closes. The NVR streams events down it continuously, and each event is tagged with the index of the physical camera that produced it. That's how ~200 cameras multiplex down one pipe: one connection in, per-camera events out, each resolved to a stable camera identity downstream. Three properties of that stream drove real design decisions:

Authentication is handled once, correctly

The NVR requires an authentication scheme that Go's standard library doesn't cover out of the box, so we lean on a dedicated, well-tested library rather than hand-rolling the challenge/response. Auth is exactly the kind of code you don't want to "mostly" get right.

Liveness comes from heartbeats, not from TCP, and here it's everything

Because a wedged NVR leaves a healthy-looking socket, the stream interleaves periodic heartbeat markers. We set a read deadline of 2.5× the heartbeat interval (12.5s at the default 5s heartbeat). Miss that window and we declare the connection dead and reconnect. On a deployment where one stream carries 200 cameras, this watchdog is the single most important piece of liveness logic in the system: it's the difference between "we reconnected in 12 seconds" and "we went blind on a province's main roads and nobody noticed."

A malformed event must never kill the stream

Across ~200 cameras of varied firmware, you will receive a garbled body or an unexpected binary frame. The parser is layered (frame boundary, then filter, then decode), and a single bad frame is counted (parse_errors) and skipped, never fatal; the reader re-syncs at the next boundary. Frames are size-capped so a confused camera can't make us allocate unbounded memory. One bad frame is isolated to that one frame, critical when dropping the whole stream would blind all 200.

The decision that defines the system: reconnect is not a crash

This is the single most important design choice, and on a one-connection deployment it's the one that keeps the lights on.

Routine disconnects are handled inside the actor, not by the supervisor.

When the NVR reboots or the link flaps, the CameraActor does not panic and does not get restarted by the supervisor. It catches the disconnect in its own stream-supervision loop and reconnects with jittered exponential backoff (1s base, 60s cap, AWS-style full jitter). The actor stays alive; only its connection cycles.

The supervisor is reserved for the abnormal path: an actual panic, a real bug, something that shouldn't happen. That's the rare road.

Why draw the line there? Because if every ordinary disconnect were a supervisor restart, the restart-intensity limit (5-in-10s) would trip under normal network churn, and the supervisor would start giving up on a perfectly healthy NVR. The jitter matters even more at the 1,000-camera target: a network hiccup that drops hundreds of camera actors at once must not become a thundering herd of synchronized reconnects hammering the devices and our own CPU. The principle generalizes: the supervisor handles bugs; the actor handles the world being unreliable. Conflating the two is how supervised systems get a reputation for being fragile.

The sink: a shock absorber, and an honest durability tradeoff

Events flow out through a single shared Kafka producer into the camera-events topic (partition key = camera ID, so each physical camera's events stay ordered even though they share one inbound stream). Between the actor and that producer sits a ring buffer with a capacity of 100,000 events.

The ring buffer is the shock absorber. The actor drops events into it and moves on; a drain goroutine pulls from it and produces to Kafka. When a burst arrives, or when the co-located broker is briefly slow, the backlog piles up in the ring instead of blocking the stream reader, which, on one connection, would mean stalling all 200 cameras. The backpressure chain is explicit end to end:

Now the honest part. The ring's drop policy is oldest-first: if it fills completely (a long broker outage, say) the oldest buffered events are discarded to make room for the newest, and every drop is counted in a dropped_events metric.

That is a deliberate, documented decision, and it defines what kind of system this is:

This is analytics-grade ingestion, not evidence-grade.

The reasoning:

• The NVR cannot be backpressured. There is no "please slow down" in the stream protocol and no replay. If we block, we don't slow the cameras, we just fall behind and eventually lose our place in the stream anyway. Given that a total outage costs data no matter what, the only question is which data.
• For traffic analytics, fresh beats complete. If Kafka is unreachable for ten minutes, the freshest picture of what's happening on the road right now is worth more than a ten-minute-old snapshot. Drop-oldest preserves the most recent state.
• The loss is bounded, visible, and alertable. It can only happen when the ring is full, it's always counted, and it fires an alert. A bounded, observable gap is a categorically different thing from silent corruption.

If a customer needed evidence-grade capture, every event, guaranteed, for forensic use, this would be the wrong architecture, and the fix (a disk spool, acks-all, replayable offsets) is a different, heavier system. We're explicit that v1 is not that. Kafka is configured to match: leader-ack only (acks=1), idempotency off, LZ4 compression, tuned for throughput and freshness, not maximal durability. Naming the tradeoff out loud is what keeps it from becoming an unpleasant surprise later.

The numbers

The load test runs the ingestion process inside a container capped at exactly the production envelope, 1.0 CPU core, 512 MiB memory, and pushes it far past real load using 100 emulated cameras (a synthetic stand-in for the future direct-camera topology, not today's single NVR):

At 1,022 events/second sustained, ~88× the real daily average, the process holds under 16% of one core and under 80 MiB, dropping nothing. On the actual production workload (~12 events/second through one NVR), the container is effectively asleep, which is exactly what you want from the smallest tenant on a shared box. The headroom isn't luck; it's the payoff of cheap goroutines and a bounded, backpressure-aware pipeline.

Other decisions worth calling out

No images, no video, JSON events only

v1 forwards event metadata and deliberately skips binary parts (like image/jpeg snapshots). Moving image bytes through the same pipeline would blow both the memory budget and the "analytics-grade" scope. The parser tolerates binary parts, it just drops them, so an NVR configured to attach snapshots doesn't corrupt anything.

One topic, event type as a field

Everything lands in a single camera-events topic, with the event type in a code field rather than a topic-per-type. Partitioning by camera ID preserves per-camera ordering while letting consumers filter downstream. Fewer topics, less operational surface, and less to run on a shared box.

Scale-out is static sharding, not a cluster

If one node ever isn't enough, you run N instances, each owning a hash slice of the roster (fnv32(camera_id) % SHARD_COUNT). No gossip, no service discovery, no Kubernetes. Going from 1 to N is a config change and a restart. For a source set that changes slowly and is known in advance, coordination infrastructure would be pure overhead.

Observability treated as a feature, not an afterthought

The metrics are chosen to catch the silent failure modes that a one-connection deployment is most exposed to: last_event_age_seconds catches a stream that's technically connected but has gone quiet (the wedged-NVR case); camera_terminations flags actors the supervisor gave up on; dropped_events makes the durability tradeoff visible and alertable. Plus /healthz, /readyz, live pprof, and Ergo's own web UI showing the process tree and mailbox depths.

The takeaway

The impressive-sounding headline, a million events a day in a 512 MB container, rests on an unglamorous truth: a million events a day is only twelve a second, and the real work is keeping one fragile, all-important connection alive without letting the ordinary chaos of the physical world turn into data loss or a blind spot over 200 cameras.

Go made the streaming loop cheap and correct whether there's one connection or a thousand, which is what lets the same code serve today's single NVR and tomorrow's thousand-camera fleet. The actor model turned that stream into something supervised, isolated, and observable, with a single shared sink absorbing the backpressure. And the sharpest decision in the whole system was a boring one, a reconnect is not a crash, which keeps the constant low-grade failure of roadside infrastructure from ever reaching the machinery built for real bugs.

Match the tool to the shape of the problem, name your tradeoffs out loud, and you can put a province's worth of roadside cameras onto the smallest container on the box.