Highway to Scale – Part 9

For years, almost everything BeeHero knew lived in one hardworking database. This post is how we grew into a dual home for data — a familiar application database for the product, and a data lake for scale — without flipping a single big switch.

From One Database to a Data Lake architecture

‍

If you have been following Highway to Scale, you already know the plot: our sensors got chatty, our early web servers gasped, and we learned to put traffic on better roads — queues, streams, and smarter APIs. By Part 8, hive and gateway data could ride a dedicated stream so many systems could listen without fighting over the same line.

That still left a quieter problem. The product, the history, the science work, and the day-to-day dashboards were still leaning hard on one relational database. Volume, history, and analysis had outgrown a single home. This part is the story of how we added a second one — the data lake — and orchestrated our data across them.

‍

When one database did it all

‍

In the relational-only era, our production database was the town square. Customer and user records lived there. Processed hive samples lived there. Apps and APIs read and wrote there. Large tables were sliced by date so the place could keep breathing, and a read replica helped internal work stay out of the way of growers and beekeepers.

For deeper research and models, we had an early escape hatch: a continuous data sync into a separate analytics warehouse (AWS Redshift). It helped. It also meant analytics and the live product still orbited the same gravitational center. When that center got busy, everyone felt it — apps, science, and ops alike.

‍

Why a lake — not just a bigger database

‍

A bigger database can buy time. It does not change the job description. Product screens need fast, reliable answers about what is true right now. Science and heavy history need room to explore without elbowing the product. Trying to make one system perfect at both is how you end up perfect at neither.

A data lake is a different kind of home: a well-organized place for raw and cleaned history, built so you can store first and shape later. We planned it in layers — a raw zone for data as it arrives, a curated zone for data cleaned up enough to use, and space for experiments that should never rattle production — so scale could have its own address.

‍

Two homes, clear jobs

‍

We did not replace the database. We specialized.

The application database remains the home for application data — the living records apps need, and the low-latency reads that power grower and beekeeper experiences. The lake became the home for scale: as-received IoT payloads, long history, enriched samples at volume, and the pipelines that turn streams into something analysts and models can trust.

By 2024, the guideline was clear: keep the latest slice of data close to the product where needed, let the lake carry the weight of history, and aim analytics queries at the lake so the old warehouse path could eventually retire. Same company. Two homes. Clear jobs.

‍

What goes where

‍

Here is the goalpost:

  • Live entities — gateways, sensors, hives, orchards, users — stay anchored in the application database. The lake gets copies and change streams so analysis can join the story without poking the live app.
  • Raw IoT payloads land in the lake as they arrive. Enriched samples grow up in the curated lake, as well as extrapolated hive and orchard updates, and the relevant data is copied into the application database — so apps still read what they need from that source. Heavy history and research prefer the lake. Product apps stay on the database.

If a piece of data has to be correct for a customer click right now, it earns a seat in the application database. If it has to be huge, historical, or endlessly re-shaped, it earns a seat in the lake.

‍

How the two homes talk

‍

Field devices still send through our upload path. From there, early lake work put samples on a stream so the lake could keep a durable copy while processing continued — the same spirit as Part 8, aimed at storage and scale.

The critical rule is simple: data always flows one way. A given data type either originates in the lake and flows to the application database, or it originates in the application database and flows to the lake — never both ways for the same type. Orchards are a good example: orchard definitions live in application data, and read-only copies serve lake processes; orchard updates are computed in the lake, and read-only copies migrate to the application database to serve the apps.

We will not turn this post into a runbook. You can read more about change-data capture in All Data Flows to the Lake, and more about analytics modeling with dbt in Building BeeHero's Data Hive. The point here is the shape: parallel paths from the field, careful one-way bridges between homes, and lots of verification while both are live.

‍

‍

Data lake data ingestion
Data Lake - Ingestion and Storage

‍

The AWS shape of the lake

‍

On AWS, the lake is a set of building blocks. Amazon S3 holds the history in separate buckets for different stages, with partitioning, versioning, and lifecycle rules. The lake prefers to subscribe to streams and topics such as Amazon Kinesis and SNS — rather than asking every upstream service to write files directly — with Lambda functions often landing that traffic. AWS Glue catalogs tables and runs heavier transforms; Athena answers questions without a warehouse for every query; CloudWatch watches the machinery; and AWS CDK keeps the footprint repeatable as code.

‍

‍

Data lake data processing
Data Lake Vision - Data Procesing

‍

Two guidelines matter as much as the service names: the lake should subscribe and consume, and lake deployment tends to come late so it can import streams and resources that already exist elsewhere.

Inside that footprint we keep three zones. The raw zone is the landing pad for data as it arrives. The curated zone is cleaned and structured enough for analytics and for selected copies that return to the application database. The exploratory zone is a separate room for notebooks and experiments that should never rattle production. Same lake — different rules for each room.

‍

A gradual move, not a big bang

‍

First the lake showed up as a side-car — a place to keep raw history and start analysis without betting the product on day one. Sample flows came early: save samples through a stream into the lake, enrich and fan them out, while apps still leaned on the application database. Those sample flows were — and still are — enhanced using data accessed from the application database. That remaining dependency is a gap we are honest about.

Application-database-to-lake copies came later. They were built to support other lake analysis and compute — dbt layers, data analytics models, and later orchard updates and marker updates — not as the first step of the migration story.

The early bet was “Scale Samples Flow.” The work landed in pieces — upload, monitoring, pipelines, cost watch, then more systematic copies and lake-native compute — not as one dramatic cutover. Dual-run periods on purpose, table by table, until each workload has a clear home and a clear one-way path.

‍

The takeaway

‍

Scale is not “move everything to the lake.” Scale is knowing which home fits which data — and building trustworthy one-way bridges while you migrate.

One database got BeeHero running, but two homes, communicating carefully with each other, are how we keep growing without asking a single system to be everything at once. That is the data lake transformation: less drama, more design, and a highway wide enough for the years of hive data still ahead.