How data lake architecture works
One place to land any data now and decide what it means later.
A data lake ingests raw data from many sources — in batches and streams — and lands it untouched in cheap object storage. Layered pipelines then clean and curate it (bronze → silver → gold) so BI dashboards and ML models can read it. The defining trick is schema-on-read: you store first and impose structure only when you query.
Sources emit data
Operational databases, application events, clickstreams, IoT sensors and log files all produce data continuously and in wildly different shapes. A data lake's premise is that you shouldn't have to model or reshape any of this before capturing it. The goal is simply to get everything into one durable place cheaply.
Ingest batch and stream
An ingestion layer pulls data two ways: scheduled batch jobs that bulk-copy tables or files, and streaming pipelines (Kafka, Kinesis, CDC) that carry events in near real time. Both funnel into the same landing zone. Keeping ingestion dumb and lossless — copy first, question later — is what makes a lake flexible.
Land raw in bronze
Incoming data is written untouched to the bronze (raw) layer on object storage, preserving the exact source bytes so nothing is lost and everything is replayable. Storage is cheap and effectively unlimited, so you keep history in full. If a downstream transform has a bug, you can always reprocess from bronze.
Transform with schema-on-read
Processing engines like Spark read the raw files and apply structure at read time — parsing JSON, inferring types, validating rows — rather than requiring a fixed schema up front. This schema-on-read model is the core difference from a warehouse. It lets you store messy data immediately and decide how to interpret it later, per query.
Curate silver and gold
Cleaned and de-duplicated data is written to a silver layer, then aggregated and joined into business-ready gold tables (revenue by day, user features, KPIs). Each hop raises quality and trust while shrinking volume. Analysts and models consume gold; engineers debug in bronze and silver.
Serve BI and ML
Curated tables are exposed to SQL query engines and dashboards for analytics, and to feature stores and training jobs for machine learning. Because everything sits in open formats on shared storage, many tools can read the same data without copying it. One source of truth feeds both the BI team and the ML team.
Govern and catalog
A metadata catalog tracks what tables exist, their schemas, lineage and owners, while access controls and quality checks enforce policy across the lake. Without this, a lake silently degrades into a 'data swamp' nobody can find anything in. Governance is what keeps a lake usable as it grows to petabytes.
Why data lakes exist
For decades the default home for analytical data was the data warehouse — a structured, relational store you loaded through a rigid ETL pipeline. Warehouses are excellent for clean, well-understood business data, but they impose a heavy tax: you must define a schema and transform every record into it *before* you are allowed to store anything. That is called schema-on-write, and it becomes a bottleneck the moment your data is messy, high-volume, or of a type you haven't figured out how to use yet — semi-structured JSON, clickstreams, images, logs, sensor readings.
The data lake was the answer to a simple frustration: storage is now absurdly cheap, so why throw data away or delay capturing it just because you don't yet know its structure? A lake lets you land *any* data, in *any* format, immediately and cheaply, and defer the modelling until you actually have a question. The bet is that the raw material has value you can't foresee at ingest time, and the cost of keeping it is trivial compared to the cost of not having it when a new analysis or ML model needs it.
How data flows through the lake
A lake is a pipeline, not a single bucket. Data moves left to right through progressively cleaner stages, an arrangement usually called the medallion architecture:
- Ingestion captures data two ways. *Batch* jobs bulk-copy tables and files on a schedule; *streaming* pipelines (Kafka, Kinesis, change-data-capture) carry events in near real time. The ingestion layer is deliberately dumb — it copies faithfully and reshapes nothing.
- Bronze (raw) is where that data lands untouched, on cheap object storage. It is the durable, replayable source of truth.
- Silver (cleaned) holds data that has been parsed, de-duplicated, type-checked and conformed — trustworthy but still granular.
- Gold (curated) holds business-ready aggregates: KPIs, feature tables, star schemas the analytics team queries directly.
Each hop raises quality and trust while shrinking volume. The critical property is that every stage is derived from the one before it, so if a transform is buggy you fix the code and reprocess from bronze — you never lose the original.
Schema-on-read and open formats
The defining idea of a lake is schema-on-read: you store the bytes now and impose structure only when something reads them. A query engine like Spark, Trino or Athena parses the raw files at query time, inferring or applying a schema on the fly. This is the exact inversion of the warehouse's schema-on-write, and it's what lets a lake accept data whose shape you haven't modelled yet.
That flexibility only works because of open file formats. Storing raw CSV or JSON is convenient but slow to scan, so the workhorse of the lake is `Parquet` — a columnar format that stores data by column rather than row. Columnar layout means an analytical query that touches three columns out of fifty reads only those three, and it compresses far better because each column holds one data type. On top of the files sit open table formats like Apache Iceberg, Delta Lake and Hudi, which add a transaction log so a pile of Parquet files behaves like a real table — with ACID writes, schema evolution and time travel. These table formats are what make a lake queryable and reliable rather than just a folder of files.
Lake vs warehouse vs lakehouse
It helps to hold the three side by side:
- A warehouse stores clean, structured, business data with schema-on-write. It's fast and governed but expensive and inflexible, and it struggles with unstructured data.
- A lake stores anything, cheaply, with schema-on-read. It's flexible and scalable but historically weaker on performance, transactions and governance — and easy to turn into a swamp.
- A lakehouse is the modern convergence: a lake's cheap open storage plus a warehouse's transactions, governance and SQL performance, delivered through table formats like Delta and Iceberg. Platforms such as Databricks and Snowflake now let BI and ML run over the same governed tables on object storage, so you no longer copy data into a separate warehouse just to query it well.
The lakehouse exists because the two older models each solved half the problem. You wanted the warehouse's reliability *and* the lake's openness and cost, and the table-format layer finally made having both practical.
Governance, failure modes, and practical guidance
The greatest risk to a data lake is not technical, it's entropy. Because a lake accepts anything with almost no friction, it drifts toward becoming a data swamp: petabytes of undocumented, duplicated, unowned files that nobody trusts or can find. The antidote is governance you treat as a first-class part of the architecture, not an afterthought:
- A data catalog (Glue, Unity Catalog, a Hive metastore) records what tables exist, their schemas, owners and lineage, so people can discover and trust data.
- Access control and encryption enforce who can read what, which matters enormously once regulated or personal data lands in the lake.
- Data quality checks validate rows on the way from bronze to silver so bad data is caught, not propagated.
In practice, the guidance is to keep ingestion lossless and schema-light, push all cleaning into versioned transform code, standardise on an open table format early, and invest in the catalog before the lake gets big — retrofitting governance onto a swamp is far harder than building it in. Serve analysts and dashboards from gold, let data scientists explore silver and bronze, and remember the lake's whole reason for being: capture cheaply now so you can answer questions you haven't thought of yet.
Ingestion accepts both batch loads and real-time streams from databases, apps, logs and events.
Raw data lands untouched in the bronze layer on cheap object storage (S3, GCS, ADLS).
Transform pipelines refine bronze → silver (cleaned) → gold (curated, business-ready) tables.
Schema-on-read means structure is applied at query time, not at write time like a warehouse.
Open columnar formats like Parquet and table formats like Iceberg/Delta make the lake queryable.
A catalog plus governance stops the lake from rotting into an ungoverned 'data swamp'.
Quick quiz
What does 'schema-on-read' mean?
Why keep an untouched raw (bronze) layer at all?
Batch vs stream processing
Process the data after it piles up, or the instant it arrives.
How Amazon S3 works
An effectively infinite hard drive on the internet, with eleven nines of durability.
Database types you should know
A field guide to the eight database families and when each one wins.
How Apache Kafka works
A distributed commit log that decouples who produces data from who consumes it.