Highway to Scale Part 4 left BeeHero’s raw path on message queues and small JSON files in S3. This chapter is the next mile on the highway: HTTP from the field lands on API Gateway, durable Kinesis streams hold the traffic, and several systems — including the data lake — can read the same uploads without fighting over a single queue message.
In Highway to Scale Part 4 we told the story of getting raw IoT uploads off the monolith’s critical path. Gateways used to wait on a live HTTP call while the API checked JSON, wrote Postgres, and kicked off the next stage. Under load that meant timeouts, lost payloads that never made it to durable storage, and a was_read column that locked the very tables we needed to keep moving.
Queues were the right next step. Put the payload on a queue, tell the gateway “got it,” park broken messages that keep failing on a dead-letter queue, and let raw-processor and samples-processor scale on their own. We even stopped keeping raw samples in Postgres and parked them as JSON on S3. For a single pipeline with one main worker, a work queue is almost unfairly good.
It is also, by design, a one-reader world. Once a worker deletes a message, that upload event is gone. If the data lake, a reprocessor, and an enrichment job all want the same gateway traffic, you either invent extra queues and copies, or you accept that only one path “owns” the message. Replay means rebuilding from S3 files — and in our queue era those files were often tiny, one sample per object, which made querying possible but storage tiers and write costs painful.
Here is the shape we left behind after Part 4: field gateways hit an upload service, messages move through queues, raw files land as JSON in S3, and sample processing still ends in Postgres — with each queue message serving one consumer at a time.
Incoming traffic was still HTTP from gateways. We needed an edge that could accept those posts at any scale, turn them into durable records immediately, and let several downstream systems read the same history without stepping on each other.
That is the streams bargain: keep data for a configurable window (ours is measured in days; AWS allows much longer), split load across partitions for throughput, and let independent readers advance at their own pace. One stream can feed batch jobs, a data-lake writer, and richer analytics consumers later — without turning ingest into a chain of HTTP calls.
We also wanted less glue code at the edge. Instead of the upload service catching every POST and pushing to a queue, API Gateway could map HTTP routes straight into Kinesis, keep useful headers, and only forward leftovers back to the legacy upload service. The rollout itself was meant to be gradual — replace pieces of the flow from raw ingest toward insights, rather than flipping every gateway overnight.
Gateways on newer firmware hit devices.beehero.io. Amazon API Gateway sits there as a single entry point for sensor, gateway, discovery, weather-pack, interrupt, and related payloads.
For the routes that belong on streams — status uploads, discovery tests, weather-pack posts, interrupts, and the rest of the raw-data catalog — the integration is not a Lambda. It is a direct AWS service call that writes each request into Kinesis: which stream, how to spread the load, and a payload that carries a label, selected headers (remote MAC, firmware version, gateway-to-gateway flags, user agent), and the raw body.
Everything else still has a home: unmatched paths forward to the sensors-upload origin so configuration and legacy endpoints keep working while ingest moves stream-first. Binary media types are declared up front for the day audio and similar payloads ride the same entry point.
We did not pour every sample type into one shared pool. The infrastructure creates dedicated streams — among them raw_data_sensor, raw_data_gateway, raw_data_discovery, raw_data_wwp, plus audio, interrupt, and mesh status — with a two-week retention window.
Sensor traffic in production gets fixed capacity because that path is the busiest; other streams scale on demand so quieter data types do not pay for unused capacity. Each stream’s identity is exported so other stacks — notably the data lake — can subscribe without owning the ingest definition.
That split matters day to day. A discovery spike does not steal capacity from sensor ingest. Teams can turn on or tune readers per data type. And when we redesign one processor, we are not redeploying the entire IoT entry point.
The stream era looks different: API Gateway writes into named Kinesis streams, and independent readers share the same history — data-lake writers land partitioned Parquet, while the operational path continues toward sample processing and Postgres.
Early Kinesis work paired the stream with Amazon Kinesis Data Firehose into S3 and a Lambda that forwarded batches toward the samples-processor queue. That taught us the shape of the pipeline — and also where Firehose’s built-in transform fan-out would fight us. Internal demos sketched a phased path: first land records on a raw stream, then add Firehose, then grow toward pre-processing and a proper lake.
The redesign pushed responsibility to explicit consumers. Data-lake Lambdas, one per data type, read their stream, write partitioned Parquet into the raw zone, and publish onward so the lake and the operational path stay decoupled. The general BeeHero infrastructure owns the gateway and streams; the data-lake stack imports those stream identities and attaches triggers. Other Lambdas can attach with their own batch sizes, timing, retries, and partial-failure handling — without asking the gateway to know who is listening.
Amazon Kinesis Data Firehose was not the forever answer for our raw path; Parquet writers and stream-triggered Lambdas scaled more predictably for how we actually read and reprocess.
Implementation detail is where IoT gets opinionated. Firmware versions disagree about field names. Discovery payloads nest “number of devices” awkwardly. Interrupts arrive as arrays that need normalizing into gateway MAC, event id, and timestamp before anyone downstream cares.
The mapping templates absorb that ugliness at the edge: escape headers safely, attach a stable label (sensor, gateway, discovery, …), and hand consumers a consistent envelope. Spreading load by request id favors throughput over strict per-device ordering — ordered-per-device work stays in later stages if a consumer needs it.
Audio routes are scaffolded in infrastructure but still called out as transitioning through sensors-upload until the Postgres copy path is fully settled. Streams give us a place to land the design without forcing a big-bang cutover on every payload type.
The API era optimized for “handle it now in the web tier” and paid with timeouts and fragile persistence. The queue era optimized for “acknowledge fast and process later” for a primary worker, and paid with single-consumer ownership and awkward sharing. The stream era optimizes for “write once, read many, keep it for a while.”
Gateways get a quick success when the record is on the stream. The data lake can land untouched raw history as Parquet without blocking sample creation. Processors can batch, retry, and recover from partial failures. We can rewind within retention when a consumer bug ships — without asking gateways to re-upload the orchard.
Infrastructure as code keeps that contract reviewable: one nested stack wires gateway, write permissions, routes, and stream exports; other repos plug in as subscribers. That is the implementation behind the architecture diagrams — not a black box service, but an entry point and a set of named streams the rest of BeeHero can build on.
So what did this stretch of the highway actually deliver? We moved field HTTP onto API Gateway, wrote durable records into named Kinesis streams instead of one-reader queues, and let the data lake and operational processors share the same ingest without copying messages around. We tried Amazon Kinesis Data Firehose as a phase, then leaned on explicit stream consumers and Parquet writers when that fit how we read and reprocess better. Open questions from the first stream notes — custom readers versus managed tools, checkpoints, cost knobs, error playbooks — did not vanish; they moved downstream, where they belong. Ingest is no longer the bottleneck story.
What we still had to achieve next was a shift in how we think about storage itself: not “plain” S3 folders of tiny JSON files, and not Parquet alone as a file format checkbox, but the data lake as the place raw history lives — organized storage with zones, partitions, and readers that expect lake semantics rather than ad-hoc buckets.
