AI Data Infrastructure: Building a Governed Foundation for Enterprise AI

Reliable AI infrastructure begins with a well-governed data layer. Whether the AI layer consists of a single model or several working together, its answers depend on the data it can access, making storage, sensitivity separation, pipelines, indexing, and retrieval fundamental to system trustworthiness long before model selection begins.  

At ASSIST Software, this data-first view shapes how we architect AI-augmented systems for enterprise clients. 

Most conversations about "adding AI" to a system start and end at the AI layer: which model, which API, which prompts. That framing skips the harder question underneath it: what data is that AI layer allowed to see, and how did it get there?

A model with an API key is not an AI system. An AI system is a full data path: raw sources, ingestion pipelines, a governed store, an index, and an orchestration layer that determines what the AI layer can read before it answers. Get any one of those wrong (unclassified data reaching a public chatbot, a stale index, an ungoverned pipeline) and the AI layer's own quality stops mattering.

The rest of this article works through that data path layer by layer, using a reference architecture and a concrete cloud implementation as evidence. 

Access tiers: public, internal, confidential

Data must be classified into access tiers, typically public, internal, and confidential, before it ever reaches an ingestion pipeline, not filtered afterward. Each tier then has its own pipeline, storage, and index, so a permission error in one tier cannot propagate to another.

This requires physical separation instead of an access-control list bolted onto a shared pipeline. Practically, that means:

• Public data flows through an elastic, cost-optimized pipeline meant for broad access.
• Internal data gets moderate, balanced compute and storage, sized for internal analytics and operations.
• Confidential data runs through encrypted storage and restricted compute concurrency, sized to meet strict security service-level agreements.

Classification at ingestion, not after

Automated tagging and metadata enrichment classify data as it lands, which is also what makes compliance frameworks like GDPR and HIPAA fall outside the architecture itself, rather than a separate review step. An AI layer or agent querying the public tier cannot, by design, reach the confidential tier because there is no shared pipeline or index to cross. 

A data lakehouse (DLH) is a governed storage foundation that holds all three sensitivity tiers, keeps storage, compute, and pipelines decoupled, and provides every downstream AI or BI tool with a single place to query rather than many disconnected systems. Storage stays cheap and durable; pipelines run ELT transformations on their own schedule, and compute never has to wait on ingestion jobs it does not depend on.

ETL versus ELT, and why the difference matters

ETL and ELT are not interchangeable, even though many teams use "ETL" as a catch-all word for any data pipeline. ETL (extract, transform, load) transforms data before it lands anywhere permanently. ELT (extract, load, transform) loads raw data first and transforms it afterward, using the destination's own compute.

Calling every pipeline "ETL" hides a real architectural decision: whether an untouched copy of the source data ever gets stored at all. A raw, unmodified landing zone or bronze layer only exists if the pipeline is ELT. An ETL pipeline that transforms before loading has nothing to fall back on if a transformation proves incorrect.

ELT is what makes the loading-and-staging and medallion patterns described below possible in the first place. Cheap object storage makes storing an untouched raw copy affordable, which is also why lakehouses default to ELT, while older, storage-constrained data warehouses have leaned on ETL instead.

ELT's emphasis on loading data first, then transforming it on a schedule, implies batch processing rather than real-time data. That assumption is correct: an ELT pipeline is not built for the millisecond-level or per-event freshness a live system might need. That gap is exactly what query federation, covered later in this article, is for: reaching a CRM's own API for live customer data, an incident management system for open incidents, or an eShop's shopping cart backend for a cart's current contents, all without waiting for a batch ELT job to land that data in the lakehouse first.

Two common storage-layer patterns: loading/staging and medallion

In that lakehouse, teams do not usually dump everything into a single undifferentiated container. Two patterns account for most real designs.

The simpler pattern uses two containers: a loading area (also called a landing zone) that holds an exact, unmodified copy of source data, and a staging area that holds the same data after cleaning, validation, and deduplication. Nothing is queried directly out of the loading area. It exists purely so that a bad transformation can be rerun from an untouched copy of the source.

The more layered pattern is the medallion architecture: a bronze layer for raw data (the same role as the loading area above), a silver layer for cleaned and conformed data, and a gold layer for aggregated, business-ready data. Some teams add a platinum layer on top of gold for highly curated, externally shared, or ML-feature-ready data. Medallion is really the loading/staging pattern extended with one more explicit stage of refinement between "raw" and "ready to use."

What the curated data layer is, in data engineering terms

"Curated layer" is not its own separate technology. It is the name for whichever layer in the pattern above is safe to query directly: the staging area in the two-container pattern, or the gold (and platinum, if present) layer in medallion. Curated means validated, deduplicated, documented in the metadata catalog, and considered safe for broad consumption, including by the AI layer.

That curated layer is also where the split covered later in this article occurs: a curated layer is not a single store; it is a structured store (query-ready tables) and a semantic store (a vector index) sitting side by side, both fed by the same upstream bronze or raw data.

Storage, compute, and pipeline decoupling

That separation between raw and curated data is also what keeps an AI system's performance predictable. If a batch of ELT jobs and a live chat query shared the same compute, a heavy nightly transform could slow down every user-facing answer that hour. Decoupling storage, compute, and pipelines means the two never contend for the same resources.

The diagram below shows this end-to-end: three sensitivity tiers, each with its own pipeline, store, and index, converging on a shared AI and data retrieval (RAG) orchestration layer. 

What lives inside the AI layer

The AI layer is not one model. Depending on the system, it typically houses several specialized models working together, for example:

• A model that picks the most relevant retrieved data and builds an answer from it, given the user's input. This is the model shown, interpreting the question and drafting the answer in the diagram above.
• A model that creates the embeddings stored in the semantic (vector) layer, so content becomes searchable by similarity in the first place.
• A model that pre-processes user input, working alongside traditional, non-AI pre-processing techniques such as normalization or intent routing rules.
• A model that assists in processing unstructured raw data on the way into the lakehouse, for example, during chunking, language detection, or entity recognition.

Calling this single box "the AI model" understates how much is actually happening inside it. Calling it the AI layer keeps the architecture honest about where multiple models plug in independently, each one upgradable, swappable, or in some designs replaceable with a rules-based step, without touching the rest of the system. 

What RAG retrieves

Retrieval-Augmented Generation (RAG) lets the AI layer answer questions using indexed content it was never trained on by retrieving the most relevant chunks of the question and passing them in as context. The AI layer never gets direct, unfiltered access to the data lake itself. It only sees what the index returns for that specific query.

That distinction is what makes the layered architecture in the previous section enforceable. Because each sensitivity tier has its own index, a chat interface scoped to public data can only retrieve from the public index, no matter how the question is phrased. The access boundary lives in the index, not in a prompt instruction that a clever question could talk its way around.

Chunking and embedding strategy

Indexing itself is not one step. A typical pipeline chunks content into retrievable pieces, generates embeddings for semantic search, and often enriches the content with language detection or entity recognition before anything becomes searchable. Each of those choices, chunk size especially, directly affects whether the AI layer retrieves the right context or a plausible-sounding wrong one. 

Catalog-grounded query generation

The AI layer queries structured data by having a model first read the metadata catalog's schema, then generate SQL against that schema, rather than guessing table and column names from the question alone. This is a second, parallel grounding mechanism to RAG. RAG grounds a generated answer in a vector search result. Catalog-grounded querying grounds a generated query in the catalog's actual schema.

This distinction matters because a data lakehouse's curated layer is not one thing. It splits into two:

• A structured layer, held in an open table format such as Apache Iceberg or Delta Lake, queried with generated SQL. "What was Q3 revenue by region" belongs here.
• A semantic layer, held in a vector store such as Milvus, Azure AI Search, or an S3 vector bucket, queried by similarity. "Summarize what customers said about the Q3 outage" belongs here. 

The metadata catalog

A metadata catalog (Unity Catalog, AWS Glue Data Catalog, and the Iceberg REST Catalog are three implementations of the same idea) enables a structured path. It tracks table and column names, types, relationships, and access permissions, without holding the data itself, and every compute engine that touches the structured layer reads through it first.

Federated query and the data mesh

The same catalog is also how a federated query engine, the backbone of a data mesh, reaches data that never moved into the lakehouse at all. Instead of copying an external source into the lake first, a federated query joins the lakehouse's structured layer with that external source at query time, as if both were colocated.

This is the real-time complement to ELT mentioned earlier. A federated query does not wait for a batch job. It can reach a CRM, an incident management system, or an eShop's cart backend in real time at query time, and combine that result with whatever the lakehouse's own ELT pipelines have already loaded.

KAG as an alternative to RAG

One further alternative worth naming: Knowledge-Graph Augmented Generation (KAG) grounds an answer in a knowledge graph's explicit entity relationships rather than a vector index, which can follow a connection that a similarity search would miss entirely. That knowledge graph is not a file or a document store. It usually lives in a dedicated graph database, such as Neo4j, Amazon Neptune, using the Gremlin query language, or Azure Cosmos DB's Gremlin API, chosen specifically because traversing relationships is what those engines are built to do quickly.

KAG is a real alternative to RAG, not a replacement for it, and the right choice depends on whether the underlying knowledge is more naturally represented as a set of documents (RAG, backed by a vector store) or as a set of explicit relationships (KAG, backed by a graph database).

KAG and RAG can also live together in the same system, rather than being an either-or choice. The graph database stores entities and their relationships, and each node can carry a reference to its corresponding document in the vector store. A document's metadata can even be represented as an entire subgraph, with references pointing back into the vector store. Either way, the graph database stays free of raw unstructured content: it holds structure and references, while the vector store holds the text and embeddings that those references point to.

The AI layer never queries either store directly. It always goes through the same data retrieval orchestrator introduced earlier, the "AI and data retrieval (RAG) orchestration" box from the layer-separation diagram, which decides whether a given question calls for graph traversal, similarity search, or both, before handing anything back to the AI layer. 

Azure implementation

On Azure, this pattern maps directly onto Azure Data Lake Storage Gen2 for storage, Azure Data Factory or Databricks for pipelines, Databricks for compute, Unity Catalog for governance, Azure AI Search for indexing, and Microsoft Foundry for AI hosting and guardrails. ASSIST Software's data engineering practice uses this exact mapping as a reference architecture when standing up AI-augmented systems for clients on Azure. 

Unity Catalog is a useful concrete data point for  this governance layer is not optional forever. Databricks has announced that the older standard tier (and its legacy Hive metastore) is being deprecated in favor of the premium tier, in which Unity Catalog is the default. Architecture decisions made today about governance are not a "nice to have for later"; they are the default, and a workload will be running on within a couple of years.

AWS implementation

The same pattern holds for other clouds, just with different-named services doing the same job. This is not a second worked example; it is the same table with one AWS alternative per row. 

The point is not which cloud wins. It is that every row in that table answers the same architectural question, regardless of which vendor implements it.

An AI system's reliability is decided in its data layer long before its AI layer ever answers a question. Separating data by sensitivity, decoupling storage from compute and pipelines, and grounding retrieval through indexes and metadata catalogs are what make an AI-augmented system auditable, compliant, and safe to scale. Whether that architecture runs on Azure, AWS, or any other cloud, the same underlying questions apply, which is why ASSIST Software treats this data-first view as the starting point for every AI-augmented system it architects, not an afterthought once the model is chosen. 

Lucian Cucoș is Head of AI at ASSIST Software, where he leads the company’s AI and data engineering practice. His background spans software engineering, data engineering, Microsoft technologies, and enterprise application architecture. During his career at ASSIST Software, he has held several technical and leadership roles, including Deputy Head of AI and Senior .NET Software Development Engineer.

With more than a decade of experience in software development, Lucian has worked extensively with .NET, ASP.NET Web API, Microsoft CRM, and enterprise systems. He currently oversees the development of enterprise AI solutions, governed data architectures, and scalable software that connects AI capabilities with operational processes. His work supports ASSIST Software’s broader focus on Deep Tech and Physical AI, including secure and auditable AI systems designed for real-world applications.

1. Do I need a data lakehouse before I can add AI to my product?

   You need governed, sensitivity-classified storage before you add AI, whether you call it a data lakehouse by name. The lakehouse pattern is simply the most common way teams achieve that governance today, since it keeps storage, compute, and pipelines decoupled and auditable from day one.

    

2. What is the difference between RAG and giving the AI layer direct database access?

   RAG retrieves only the chunks relevant to a specific question and passes them as context, while direct database access gives the AI layer the ability to query anything it is credentialed for. RAG is what keeps a public-facing chat interface from ever seeing confidential rows it was never meant to reach.

    

3. When should I use RAG versus letting the AI layer generate SQL against structured tables?

   Use RAG when the answer lives in unstructured or semi-structured content, such as documents, tickets, or transcripts, and use catalog-grounded SQL generation when the answer is a specific, structured fact, such as a revenue total or a row count. Most production systems need both, depending on the kind of question that came in.

    

4. Is KAG a replacement for RAG?

   No. KAG is an alternative retrieval mechanism for cases where the underlying knowledge is naturally a set of explicit relationships (a knowledge graph, typically stored in a graph database such as Neo4j, Amazon Neptune, or Azure Cosmos DB's Gremlin API) rather than a set of documents (a vector index). Most systems will keep using RAG as the default and reach for KAG specifically where relationship-following matters more than semantic similarity. The two can also run side by side, with the graph database's nodes referencing the matching documents or embeddings in the vector store, so the graph stays limited to structure and references rather than holding unstructured content itself.

    

5. Does this architecture work the same way on AWS as it does on Azure?

   Yes, the layers are the same (storage, pipelines, compute, governance, indexing, AI hosting), only the named services change. Azure Data Lake Storage Gen2 becomes Amazon S3, Unity Catalog becomes the AWS Glue Data Catalog with AWS Lake Formation, and Microsoft Foundry becomes Amazon Bedrock, but the underlying architectural decisions are identical.

    

6. Why call it an "AI layer" instead of "the AI model"?

   A: Because in most real systems, it is not one model. A single orchestration point typically calls on several specialized models: one to interpret input and draft answers, one to generate embeddings, one to assist with preprocessing, and one to help parse unstructured raw data, so the term "AI layer" describes what is there.