Enterprise Data Platforms, Explained — Part 1

Imagine a bank.

The core banking system manages accounts and ledger entries. Card systems handle authorizations. Other services process transfers, customer onboarding and identity checks. Customer conversations live in a CRM or support platform, while mobile and online banking produce their own events and logs.

Now the bank wants a daily payments dashboard. Later it wants a fraud model and an AI assistant that helps employees answer customer questions.

All three need data from several systems. Connecting each new consumer directly to every source quickly becomes a mess. Each team copies data differently, uses different field names and applies different business rules. Two reports can show different revenue for the same day.

A data platform solves this by creating a common path:

source systems → data movement → storage → preparation → consumption

The platform also controls who may use the data, checks whether it is complete and records where it came from.

This does not require one enormous product. A data platform is a set of responsibilities. A small company can implement them with a warehouse and several scheduled jobs. A larger organization may use object storage, streaming infrastructure and several processing systems.

This article explains the complete path before the rest of the series examines each part in detail.

Open the enterprise data platform diagram at full size.

Start with the systems that create the data

Data begins in systems that perform real business work:

  • a core banking system records accounts and ledger entries;
  • card and payment systems handle authorizations, transfers and settlements;
  • a CRM stores customer activity;
  • KYC systems keep onboarding and verification information;
  • mobile banking, ATMs and external providers produce events, files and API responses.

We call these source systems.

A source system is usually built to run a business process, not to answer every analytical question. Core banking and payment systems must process financial operations consistently. Allowing dozens of analysts, dashboards and AI applications to query them directly could affect critical services or expose information they should not see.

The data platform therefore takes controlled copies of the information that other teams need.

The source still owns the original business fact. Copying a posted transaction into an analytical platform does not make the platform part of the ledger or responsible for moving money.

Data has to enter the platform

The process of bringing data into the platform is called ingestion. There are three common ways to do it.

Batch: copy a group of records at once

A batch job might copy the previous day’s settled payments or download a reference file from a partner every night.

This is often the simplest option. If a report is updated once a day, it probably does not need a complicated real-time pipeline.

A good batch job remembers which period it copied, checks that the expected input arrived and can safely run again if something fails.

CDC: copy database changes

Change Data Capture, usually shortened to CDC, follows changes made to a database. When an account or transaction record is inserted or updated, the platform receives that change.

CDC can deliver updates faster than repeatedly copying the whole table. It also creates new questions: what happens after a long outage, how are deleted rows represented, and can the same change arrive twice?

Streaming: receive events as they happen

A banking service can publish an event such as CardAuthorized or TransferInitiated. Other systems subscribe to those events and react quickly. The event describes what happened at that stage; it does not automatically mean that settlement or final ledger posting has completed.

Streaming is useful when seconds matter. It is also more complex to operate. Events may arrive late, out of order or more than once. The platform has to be designed for that behaviour.

Most real platforms use a combination of batch, CDC and streaming. “Real time” is not automatically better. The right method depends on how quickly the data is needed and how much complexity the organization is prepared to operate.

The platform keeps a durable copy

After data enters the platform, it needs a safe place to live.

This may be:

  • object storage containing files;
  • a data warehouse optimized for analytical queries;
  • a database built for a particular application;
  • several systems used for different purposes.

Keeping an original or lightly changed copy is useful because mistakes happen. A transformation may contain a bug. A new report may need a field that was ignored before. A business rule may change.

If the original data is still available, the platform can process it again. If only the final report was stored, rebuilding the result may be difficult or impossible.

This does not mean keeping everything forever. Personal and sensitive data still needs clear retention and deletion rules.

Raw data is not ready for everyone

Source data often contains technical names, duplicate records, missing values and different formats.

For example:

  • one system stores a currency as USD, another as the numerical code 840;
  • a card authorization event can arrive before the related ledger entry;
  • the same event can be delivered twice;
  • a reversal can change the meaning of an earlier transaction;
  • different teams may define an “active customer” or “suspicious transaction” differently.

The platform uses processing jobs to clean, combine and check this data. The result is a dataset with a clear meaning.

A useful dataset should answer several simple questions:

  • What does one row represent?
  • Who owns the dataset?
  • Where did the values come from?
  • How fresh should the data be?
  • Who is allowed to read it?
  • What happens when its structure changes?

Teams sometimes call processing stages bronze, silver and gold, or raw, cleaned and curated. The names are less important than the rules. Creating three folders does not automatically make the data correct.

Processing should be repeatable

Data can be processed with SQL, distributed computing systems or stream processors. These tools perform operations such as:

  • removing duplicates;
  • matching authorizations with ledger postings and settlements;
  • converting currencies;
  • calculating totals;
  • hiding sensitive fields;
  • checking required values.

The important part is not the name of the tool. The important part is being able to understand and repeat the result.

If a revenue number is wrong, the team should be able to identify the source data, transformation code and execution that produced it. After fixing the problem, the affected period should be safe to process again.

An orchestrator starts jobs in the correct order, retries failed steps and runs historical periods again when needed. The transformation logic should remain visible and testable instead of being hidden inside a large collection of unrelated tasks.

Different consumers need different forms of data

Prepared data is useful only when consumers can access it in a suitable form.

Dashboards and analysts

They usually need tables with stable definitions and predictable query performance. An analyst should not have to reconstruct payment volume, balances or settlement state from raw events every time.

Applications

An application may need an API, an export or a stream of selected updates. It usually should not query every internal table in the platform.

Machine learning and AI

ML systems need reproducible training data. A fraud model may need transaction history and device signals. An AI assistant may need account information, product rules, statements or KYC documents—within the permissions of the employee and customer involved.

A vector database or search index is normally a prepared copy used for retrieval. It is not the original source of truth. The platform should still know which source record created an indexed item and update or remove it when the source changes.

This is how the data platform will later connect to the enterprise AI platform: through known datasets and controlled interfaces, not by giving a model unrestricted access to every database.

The platform also needs rules and operational controls

Moving and storing data is only half of the job.

Catalog

A catalog records which datasets exist, what they mean and who owns them. It helps people find the right data instead of creating another copy.

Quality

Quality checks detect missing, late, duplicated or invalid data. A pipeline can finish successfully while producing an incomplete dataset, so infrastructure monitoring alone is not enough.

Security

Not every user should see every field. The platform applies access rules and can mask sensitive values. Copies and derived datasets must follow the same privacy boundary as their sources.

Lineage

Lineage shows how a result was created: which source it came from and which transformations changed it. This helps when a number is wrong or a source schema changes.

Operations

The platform team watches failures, freshness, capacity and cost. It also needs a recovery plan for broken pipelines and unavailable systems.

Follow one card payment through the platform

Consider a bank that needs a daily payments dashboard and a fast fraud signal.

  1. A customer pays by card.
  2. The card processing system authorizes or declines the operation and publishes an event.
  3. The relevant hold or transaction is later recorded in the bank’s operational records and ledger according to the payment flow.
  4. CDC copies the committed database change into the data platform.
  5. The platform keeps the original authorization event and ledger record.
  6. A processing job matches the records and removes duplicates.
  7. Quality checks look for unmatched authorizations, invalid amounts and inconsistent currencies.
  8. A prepared payments dataset feeds the daily operations and reconciliation dashboard.
  9. A lower-latency stream sends selected signals to the fraud system.

The dashboard and fraud system use related business facts, but they do not need the same speed or storage technology. The platform gives them a shared foundation without pretending that an authorization event, a settlement and a ledger posting are the same thing.

What a data platform is not

It is not one database. Different workloads may need different stores.

It is not a folder full of files. Data without ownership and meaning is difficult to trust.

It is not real time by default. A daily batch is often the better engineering choice.

It is not a replacement for source systems. The platform prepares their data for additional uses.

It is not useful without consumers. The platform should grow from real reporting, application, ML and AI needs.

A sensible way to start

  1. Choose one useful report, application or model.
  2. Identify the source data it needs.
  3. Decide how fresh the data must be.
  4. Use the simplest ingestion method that meets that requirement.
  5. Keep a recoverable copy.
  6. Build clear and repeatable transformations.
  7. Add quality checks, ownership and access rules.
  8. Publish the result in a form the consumer can use.

This is usually safer than building a large central platform before anyone has defined what it should deliver.

The whole system in one sentence

Source systems create data, ingestion brings it in, storage keeps it, processing prepares it, serving delivers it, and governance keeps the full path understandable and controlled.

In Part 2, we will look more closely at batch, CDC and streaming and explain when each method is useful.

Enterprise Data Platforms, Explained

  1. How Enterprise Data Platforms Work
  2. How Data Enters the Platform: Batch, CDC and Streaming
  3. Data Lake, Warehouse and Lakehouse: What Is the Difference?
  4. How Data Is Stored: Object Storage, Parquet and Table Formats
  5. How Data Is Processed: ETL, ELT and Orchestration
  6. How Data Quality, Catalogs, Lineage and Contracts Work
  7. How Security and Authorization Work in Data Platforms
  8. How the Whole Enterprise Data Platform Fits Together

Sources and further reading