What is data lake architecture?
Data lake architecture is the structure that connects data sources, ingestion pipelines, data storage, processing services, metadata, security controls and consumption tools.
A data lake can store structured, semi-structured and unstructured data in its original format. This gives teams more flexibility during ingestion because data does not always need to fit a fixed schema before it reaches the storage layer.
The architecture usually separates storage from processing. Data can be stored in an object-based repository, then processed for different purposes such as reporting, business intelligence, machine learning or exploratory analysis.
A data lake architecture needs more than a storage location. It needs a set of connected layers that explain:
Where data enters the platform
How data is stored
How data is transformed
How users discover it
How quality is checked
How access is controlled
How data reaches its consumers
Core data lake components and layers
A common data lake architecture includes 6 logical layers. AWS describes a similar structure with ingestion, storage, processing, cataloging, consumption, security and governance layers.
Layer | Main responsibility |
|---|---|
Data sources | Generate operational, business and external data |
Ingestion | Move batch or streaming data into the lake |
Storage | Retain data in different formats and states |
Processing | Clean, transform and enrich datasets |
Catalog and metadata | Describe, classify and make data discoverable |
Consumption | Serve data to analytics, reporting and machine learning |
Security and governance | Control access, quality, lineage, retention and auditing |
Some organizations combine or rename these layers. The important point is that each function has a clear owner and a defined relationship with the others.
Data sources and ingestion
The ingestion layer brings data into the lake from different sources. These may include:
Relational databases
Business applications
APIs
Files
IoT devices
Application logs
Streaming platforms
External data providers
Batch ingestion moves data at scheduled intervals. Streaming ingestion handles events as they are generated. The right approach depends on how quickly the business needs the data and how much operational complexity the team can manage.
The ingestion design should record:
Source system
Data owner
Data format
Frequency
Expected volume
Security classification
Failure handling
Validation rules
A landing zone can preserve incoming data before transformation. This gives the team a reference copy and makes it easier to investigate errors or rerun a processing step.
Data lake storage layers
The storage layer usually keeps data in different states of preparation. A common pattern uses 3 zones:
Raw or bronze layer
The raw layer stores data close to its original format. It preserves the source record and provides a reference for later processing.
This layer may contain inconsistent values, duplicate records or incomplete metadata. Access should therefore be controlled, especially when the data includes personal or sensitive information.
Cleansed or silver layer
The silver layer contains validated and transformed data. Teams may standardize formats, remove duplicates, apply business rules and resolve basic quality issues.
The data is easier to use at this stage, but it still may require additional modelling for a specific business purpose.
Curated or gold layer
The gold layer contains business-ready datasets for reporting, dashboards, data products or machine learning workloads.
Microsoft describes this bronze, silver and gold pattern as a way to improve data quality, governance and performance as data moves through the platform.
These layers are useful design tools, but they should reflect the organization’s real workflows. Adding zones without clear ownership or quality rules simply creates more places for data to wait.
Data processing and quality
The processing layer turns stored data into datasets that people and applications can use.
Processing may include:
Schema validation
Data type conversion
Deduplication
Standardization
Enrichment
Aggregation
Data masking
Join operations
Business rule checks
Partitioning
Data quality checks should be placed at suitable points in the flow. A basic validation can happen during ingestion. More detailed checks can run after transformation, when the team has enough context to assess completeness, accuracy, freshness and consistency.
Each important dataset should have defined quality expectations. For example:
A financial dataset may require accurate totals and complete records.
A customer dataset may require consistent identifiers and current contact information.
A streaming dataset may require freshness and delivery monitoring.
Data quality results should be recorded with the dataset metadata. This lets users see whether the data is ready for their intended use.
Data catalog and metadata
A data lake can contain thousands of datasets, files and partitions. A catalog helps users find the right data and understand its meaning.
Useful metadata includes:
Dataset name
Business description
Owner
Source system
Update frequency
Data format
Sensitivity classification
Quality status
Lineage
Retention period
Approved use cases
AWS describes the catalog as a central place for technical and business metadata, including schemas, data locations, owners, definitions and sensitivity information.
Metadata also supports data lineage. Users should be able to trace a dataset back to its source and understand which transformations were applied before it reached the consumption layer.
Data lake governance and security
Data lake governance defines how data is owned, accessed, protected and reviewed.
The governance layer should cover:
Data ownership
Access roles
Encryption
Network protection
Retention
Deletion
Data classification
Quality rules
Lineage
Usage monitoring
Audit records
Access should reflect the sensitivity and purpose of the data. Some users may need access to raw records. Others may only need curated datasets with sensitive fields removed.
Governance also needs a clear operating model. A central team may define common policies and manage the platform. Business domains may own the datasets they create and remain responsible for their quality. The right distribution depends on the organization’s data architecture and maturity.
Organizations defining these responsibilities can connect their data lake plans with Data Strategy & Operating models.
How data lake architecture changes by operating model
The same technology can support different operating models.
Centralized data lake
A central data team manages the platform, ingestion standards, access rules and most quality controls. This model can suit organizations that need common definitions and central oversight.
Its main risk is that the central team becomes responsible for every dataset and every request.
Federated data lake
Business domains own their data while a central team manages shared standards, catalog rules, security requirements and platform services.
This model gives domains more responsibility for quality and documentation. It also requires clear decision rights and escalation paths.
Data mesh and data products
In a data mesh, domains publish data products with defined owners, documentation, quality expectations and access conditions.
Google Cloud describes a data mesh as a structure where domains expose data products through well-defined interfaces, supported by central catalog and governance functions.
A data lake can support this model when storage, metadata, access and processing services allow domains to publish and manage their own products.
How to design a data lake architecture
A practical design process can follow 6 steps:
Define business use cases. Start with the decisions, reports, applications or models the data must support.
Map sources and ownership. Identify where data comes from, who owns it and which systems depend on it.
Choose storage and processing patterns. Decide which data needs batch ingestion, streaming, raw storage, transformation or curated delivery.
Define the layer structure. Establish the rules for raw, cleansed and curated data, including retention and access.
Set metadata and quality requirements. Document business definitions, lineage, ownership, validation rules and quality expectations.
Test consumption and governance. Confirm that users can find, access and use the data under the right conditions.
The design should leave room for change. New sources, domains and use cases will appear over time, so the architecture needs clear rules for adding them without creating disconnected data stores.
Conclusion
A data lake architecture connects data ingestion, storage, processing, cataloging, consumption, security and governance.
Its success depends on the operating model behind it. Centralized, federated and data mesh approaches distribute ownership and decision-making differently, so the data lake should reflect those choices.
Organizations planning a data lake, reviewing data ownership or defining governance requirements can explore data strategy consulting.
