Blueprint to help non-tech executives assess AI vendors

CASE STUDY 12 min read CEO, CFO, Board Members Thought Leadership Cross-industry

Why enterprise AI is fundamentally different

According to an MIT 2025 report, 95% of AI initiatives fail to deliver value.

The reason is simple: building AI agentic solutions for real enterprise use cases requires a blend of business acumen, technical expertise, and software craftsmanship that is just uncommon.

This is because Enterprises face constraints that don’t apply equally to consumer, LLM-products: reliability and consistency, auditability, replayability, data confidentiality, and data quality grading.

Off-the-shelf GPT-wrappers fail on all these dimensions.

Non-tech executives need AI mental models

Companies who partner with vendors with real-world AI expertise are twice as successful as those who DIY or hire hand-waving consultants (MIT, 2025). But how do you assess such vendors when you’re a non-tech Exec?

Since there is no ‘CFA of AI building’, you cannot use certificates as a proxy. What you must do instead is asking the right questions, about recent solutions they’ve deployed and probe how they think about your specific problem.

To do that well, you need blueprints: mental models that help you to probe the right way.

Ask the right questions. Choose the right vendor.

There are multiple blueprints non-tech executives should know: data ingestion, AI orchestration, data confidentiality, deployment strategy, etc.

This breakdown focuses on the most important blueprint of all: the data ingestion blueprint. It sits upstream of any AI systems and will make or break your workflows.

We will give you the vocabulary and specific craftsmanship questions to separate experienced AI vendors from those selling GPT-wrappers dressed up as enterprise solutions.

By the end of this article, you will understand what your Enterprise data ingestion pipeline should look like, and know exactly what good answers from experienced vendors look like.

Introduction

When using consumer LLMs like GPT, Gemini, etc, the workflow is pretty straightforward: users pass a prompt + docs (sometimes) → LLM does its thinking → LLM responds.

Real-world workflows are radically different. Unfortunately, lots of non-tech executives and board members are sold on the idea that implementing AI in their organizations is as simple as GPT. This leads to disappointment and failed projects. According to the MIT 2025 report, 95% of AI initiatives fail to deliver value.

Why?

Building AI agentic solutions for real use cases is extremely hard. It requires a blend of business acumen, technical expertise and software craftsmanship. Perhaps most importantly, it requires understanding the constraints that Enterprise/Mid Market companies are facing that do not apply equally to consumer products. Here are just 5 (there are more):

Reliability and consistency: broadly similar input/prompts should have similar outputs across multiple runs
Auditability and traceability: one needs to explain how the system thought about a particular problem and ended up with a particular output, step by step, almost in a “debug” style
Replayability: one must be able to reconstruct past decisions using the same workflow, and arrive at the same result (which is different than merely explaining or re-tracing the steps)
Data confidentiality: PII (Personally Identifiable Info), MNPI (Material Non Public Info) should not leave the org boundary (eg don’t pass it to frontier LLM for reasoning)
Data filtering and quality grading: one must be able to find the signal in a large amount of noise, and flag the quality level of certain data instead of assuming that “it comes from the Client so it must be good quality”

Notice that when using LLM as consumers, we typically don’t care much about any of this (we don’t need re-runs, we don’t audit the LLM reasoning, we routinely pass private data to the LLM, we let the LLM sort out the signal from the noise, etc.) - which is one of the reasons why GPT-wrappers always fail in Enterprise environments.

The solution: find the right AI partner / vendor

Without surprise, quoting MIT, companies who partner with vendors who have real-world expertise in building AI solutions are twice as successful as those who try to DIY (or worse, hire hand-waving consultants who build GPT-wrappers).

But unlike many other industries, you can’t tell how good a vendor is based on certificates (eg, there is no “CFA of AI building”). All you can do is ask about recent AI solutions they’ve deployed, and probe to see how they think about your problem, based on a blueprint (eg mental model) that you understand.

There are multiple layers of blueprint that non-tech executives should have. Here’s a non exhaustive list:

The data ingestion blueprint
The AI orchestration blueprint
The ontology blueprint
The data confidentiality and solution deployment blueprint
The train vs fine tune vs orchestrate blueprint

In this breakdown, we want to help the non-tech executives understand the most important blueprint of all: the data ingestion blueprint. The goal is to understand its first principles and be able to ask the right questions to any vendors, so that you choose the right one and increase your chances of success with AI.

The data ingestion blueprint

Note: when we say Enterprise, we include large Enterprise (think MNCs) but also mid-market companies (say, a company with more than 10 $M in AR or an investment fund managing upwards of 100 $M in AUM).

Every AI workflow problem starts with a data problem. It is useful to think of this as 3 big sub-layers:

The ingestion layer
The data storage layer
The data preparation layer

Some people will tell you that they follow the ELT framework (Extract Load Transform, as opposed to the old school ETL Extract Transform Load). We think that just confuses executives, so we just prefer to talk in raw language: ingest, store, prepare.

That’s really all there is to it. Of course, any vendor can tell you that, so the secret is to ask them the real AI craftsmanship questions - which we are appending at the end of this page.

Here is a visual blueprint of what any data ingestion pipeline should - at the minimum - look like for an AI solution in the real-world. A description of each step is provided below the diagram.

    
C4Container
    title Investment Intelligence Ingestion Architecture

    System_Ext(extSources, "External Sources", "APIs, filings, news, market data, expert sources")
    System_Ext(intSources, "Internal Sources", "CRM, IC memos, portfolio updates, internal documents")

    Container_Boundary(ingestion, "1. Controlled Ingestion") {
        Container(scheduler, "Ingestion Scheduler", "Orchestration Layer", "Schedules ingestion jobs, controls frequency, prioritizes sources")
        Container(intake, "Source Intake", "Connector Layer", "Access auth, format checks, malware screening, retries")
        Container(policy, "Policy & Confidentiality Gate", "Control Layer", "PII/MNPI detection, sensitivity classification, access tiering")
    }

    Container_Boundary(storage, "2. Data Storage Layer") {
        ContainerDb(raw, "Canonical Raw Zone", "Immutable Restricted Storage", "Original source records preserved for auditability, replayability, and traceability")
        ContainerDb(redacted, "Redacted Working Store", "Sanitized Operational Storage", "Policy-cleared records for downstream processing and controlled operational use")
    }

    Container_Boundary(preparation, "3. Data Preparation Layer") {
        Container(gov, "Data Reliability Controls", "Governance Layer", "Deduplication, schema validation, lineage, audit trail, quality scoring")
        Container(struct, "Analytical Preparation", "Processing Layer", "Entity extraction, metadata tagging, structuring, signal extraction")
        Container(materiality, "Materiality Engine", "Decision Layer", "Change detection, threshold rules, priority scoring, routing")
    }

    Container_Boundary(consumption, "4. Consumption") {
        Container(orchestration, "AI Orchestration", "Agent Layer", "Orchestrates AI agent tasks")
    }

    %% Ingestion flows
    Rel(extSources, scheduler, "Available data sources")
    Rel(intSources, scheduler, "Available data sources")

    Rel(scheduler, intake, "Triggers ingestion jobs")
    Rel(intake, policy, "Validates and admits")

    %% Storage outputs from policy gate
    Rel(policy, raw, "Preserves original records")
    Rel(policy, redacted, "Writes policy-cleared records")

    %% Preparation flow uses redacted working data
    Rel(redacted, gov, "Provides working records")
    Rel(gov, struct, "Passes trusted records")
    Rel(struct, materiality, "Produces signals")

    %% Consumption
    Rel(materiality, orchestration, "Triggers workflows")

Step-by-step explanation of the data ingestion pipeline

Ingestion Scheduler

This layer determines when / how often to ingest data.

Example: daily news ingestion, weekly internal doc ingestion, continuous monitoring for new regulatory filings.

In real enterprise environments, not everything should run in real-time. Timing creates predictability in workflows (teams know when updates happen), avoids unnecessary noise or system instability. It also allows us prioritize sources in terms of quality.

Source Intake

This is triggered by the scheduler, controls how data is actually pulled in (APIs, scraping, file drops, internal systems, etc.) and what is authorized by the Client for ingestion.

Sanitization is applied to prevent malicious content (e.g. used for prompt injection risks during AI orchestration), and to standardize formats so downstream systems can process it reliably.

In practice, ingestion is rarely plug-and-play. Many sources are non-standard (PDFs, emails, spreadsheets, etc.), and building robust ingestion pipelines is often one of the hardest engineering parts.

Policy & Confidentiality Gate

This layer determines what is allowed to enter the system and how it can be used.

It detects and redacts sensitive content (PII, MNPI), assigns sensitivity levels (so not all data is used equally by agents later but only the type of data you allow), and access levels (you don’t want an LLM to disclose a particular information to a certain category of employee while they use the system for example).

In real investment environments, this is critical. Without it, sensitive information can leak into search, AI workflows, or broader team access. This layer allows the system to scale without creating compliance or reputational risk.

Canonical Raw Zone

This is where the original data is stored as-is, with timestamps, source tracking, and versioning.

Example: the exact filing or memo that was ingested, preserved unchanged.

This store is restricted and not used in day-to-day workflows. Its purpose is to preserve a verifiable source of truth.

This enables:

going back to what was known at a given time
replaying past analyses with the same inputs
supporting audits and dispute resolution

In practice, this layer acts as a static repo of the known knowledge at any point of time.

Redacted Working Store

This is the policy-cleared version of the data, produced by the Policy & Confidentiality Gate.

Sensitive elements (PII, MNPI, restricted content) are removed, masked, or access-controlled before data is written here.

This is the version of the data that:

downstream systems operate on
AI workflows consume
broader teams can access under controlled permissions

This separation allows the system to remain operationally usable without exposing sensitive raw data.

Data Reliability Controls

This layer operates on the redacted working data and ensures it is consistent, traceable, and usable:

removes duplicates across sources
re-validates formats and structure
tracks provenance (where each piece of data came from)
assigns quality / confidence indicators

In practice, multiple sources often report the same information differently, or with inconsistencies. This layer ensures the system does not propagate conflicting or low-quality signals.

It also enables:

traceability → understanding exactly where a signal came from
consistency → avoiding multiple versions of the same fact
explainability → showing how outputs were derived

Without this layer, downstream systems may still function, but output quality and explainability will suffer.

Analytical Preparation

This layer converts raw data into usable signals.

Example:

extracting company names, metrics, events
tagging documents for later retrieval
structuring unstructured content

Without this step, data remains fragmented and hard to use. This step enables AI Agents to use the relevant info given a specific task, instead of looking for a needle in a haystack.

Again, a high degree of craftsmanship and business acumen is required here, since we start entering the domain of bespoke work for different clients. For example, in works we have done for an investment fund, this is where we start shaping their incoming data in a way that is as useful as possible for investment due diligence by downstream AI agents.

Materiality Engine

This layer determines what actually should trigger an action. Not every data change requires re-running an entire agentic pipeline, otherwise clients end up with unnecessary information overload.

Example:

Re-run an analysis but only if a revenue number drop beyond a threshold
A new regulatory guideline is published and must trigger an update of risks grading

While not all materiality criteria are explicitly known by Clients, we found that they actually know most of them. A tight feedback loop with the Client post-deployment allows us to adjust this materiality plane and amend it as we go.

AI orchestration

This is where the data ingestion layer ends and the actual AI Agentic work begins, turning static data into actionable insights. All the orchestration logic sits here, and consumes the data from the redacted store.

We will cover the entire domain of AI orchestration in a separate article.

Final thoughts

We hope this simplified data ingestion blueprint helps non tech-executives understand what differentiate what they truly need in their organization from what most hand-waving, GPT-wrapping consultants sell.

Bear in mind that this is only the first step in a long list of blueprints that non-tech executives need to understand in order to make an informed decision regarding AI implementation.

In later articles, we’ll break down the other important blueprints/mental models, namely:

The AI orchestration blueprint - where the actual business workflows happen
The ontology blueprint - or how to scale and institutionalize an AI solution across a portfolio
The data confidentiality and solution deployment blueprint - answering questions about data residency, governance and model confidentiality
The train vs fine tune vs orchestrate blueprint - answering the question of what type of AI implementation strategy one should pursue

Below is a cheat sheet with questions you can use to probe any AI vendor, and assess their degree of expertise based on their answers.

And if you have any doubt what next step to take to start your AI transformation, you can always reach out to us.

Questions to detect hand-waving vendors from real experienced AI vendors

Question to ask	Hand-waving answers	Answer you want to hear
We have A LOT of different data sources. Should we give you everything?	Yes give us access to everything, we will then tell you later what data we don’t need. You never know!	Of course not, we’ll start with a diagnosis of your problem first, and figure out the minimum data access requirements. If we need more later, we’ll tell you why
How do you know which data source actually matters for a specific task?	The LLM/autonomous AI agent will know based on our expert prompt what to use, so you don’t need to worry about it	We’ll define selection rules with your SMEs, encode them as deterministic filters and scoring logic, and validate them through test cases and ongoing evaluation metrics. We advise to keep a human-in-the-loop
How can you replay a certain scenario if my auditor needs to?	We store the prompts and LLM responses in clear text, so your auditor can always see what happened (vendor is confusing traceability with replayability, and is actually using a naive-version of traceability which can leak sensitive data in logs)	In an audit scenario, we can re-run a specific workflow with the provided input, and your auditor can compare the live replay output with the historical version of what was produced. Note that it may not be 100% the same (eg not word by word) because LLM are non deterministic machines. But we will provide the evaluation rubric (eg used for answer scoring and grade output quality) for your auditor to verify and understand the output variance
How do you handle PII and MNPI from our internal docs?	The LLM is smart enough to know what constitutes a name, bank account number, amount of money, and all other PII / MNPI. It will redact such data automatically	We never pass internal docs directly to LLM to redact, since that would essentially expose your PII / MNPI in clear text to them, creating a risk exposure We do a 3-pass process: standard regular expression redaction (email, phones, etc), custom redaction (that we must define with your team) and offline LLM redaction (this depends per project)
How do you make sure the data you store from our docs is not accidentally leaked or used by an AI Agent that goes rogue?	The AI agents don’t have access to your docs directly, but to the redacted data we store, so you don’t need to worry (vendor is not answering the question but trying to confuse the non-tech executive. It is technically correct but still the wrong answer) And we will make sure AI agents never go rogue anyway (clear hand waving)	The data layer has an authorization layer baked in. Each agent has its own permissions and cannot self escalate to grant itself more permissions. In the unlikely event of an agent “going rogue”, your data would have been redacted in our 3-pass redaction process already, so only non sensitive information would be exposed
Since you store the data, can your dev read it?	Of course not, no one can read your data. It is safely encrypted (technically true but clear hand-waving, since LLM will need the decrypted version to do their work, which can be made available via logs in clear text)	Once in prod, we can’t read any of your data. Access controls and encryption will prevent us from doing that, unless you give us a special privilege. While building the solution for you, we’ll need to spend time understanding your data, to make sure the system works as intended. So we’ll need limited and confidential access to it in the beginning, otherwise we can’t build the thing.
If no dev can read our data, how will you debug when things go wrong?	We will rely on your expertise to tell us when the output is not appropriate to fix it (operationally, this doesn’t mean anything if they don’t know what went wrong and where)	During development, we’ll see the execution trace and the redacted input / outputs, which is enough for us to fix things. In prod, drift will be monitored against the scoring framework we would have put in place with your team in the beginning for each workflow. Above a certain threshold of variance, we’ll know we need to fix things, and will be able to identify which workflow to investigate. But for prod data access and reproducing the issue, we will work with your team for a forensic analysis (eg since we don’t have access to your prod data, we will need to check with them what we can access for forensic purposes and if prior redaction on their end is required)
I heard about prompt injection. How do you guard against that?	This is not a data ingestion problem but a prompt engineering problem. We will make sure no external prompt can be provided to our AI agents	During data ingestion, we’ll sanitize data for malicious/suspicious phrases. We’ll inform you on flagged risks just so you can check internally. Often times, employees don’t mean harm, but it is possible that some motivated and skilled employees want to extract some of your data by incorporating prompt injection attacks directly into some data sources that will be ingested We also have prompts sanitization at the orchestration layer combined with restricted tool access by agents, to ensure they can only take authorized actions

Blueprint to Help Non-Tech Executives Assess AI Vendors