For the past few years, the AI conversation has been dominated by increasingly powerful large language models (LLMs). Every new model release seems to bring bigger context windows, better reasoning capabilities, and new benchmarks showing impressive gains over the previous generation. As a result, it's easy to walk away with the impression that once the models become smart enough, the rest of the puzzle will solve itself.
Unfortunately, that's not how most enterprise environments work.
Ask an AI model to explain the causes of customer churn, identify supply chain risks, recommend inventory adjustments, or predict maintenance issues, and you'll quickly discover that the challenge isn't necessarily the model itself. More often than not, the real problem is context. The model may be able to read your data, but it doesn't understand what the data means. It doesn't know your business processes, your organizational structure, your industry terminology, or the countless relationships that exist between your customers, products, suppliers, assets, and employees.
This stark reality is forcing a shift in how software vendors think about AI. The conversation is gradually moving away from simply making models smarter and toward making them better informed. Across the industry, companies like Microsoft, SAP, Salesforce, ServiceNow, Databricks, and others are investing heavily in new intelligence layers designed to give AI systems a deeper understanding of how businesses actually operate. Whether they're called knowledge graphs, semantic layers, ontologies, business data fabrics, or IQ systems, they're all attempting to solve the same fundamental problem: helping AI move beyond information and toward understanding.
In this article, we'll explore why large language models often struggle to make sense of enterprise data, why business context has emerged as the missing ingredient for AI success, and how modern data platforms are evolving to bridge these gaps.
The LLM Illusion
One of the most persistent myths surrounding AI is the idea that if you simply connect an LLM to enough enterprise data, meaningful business insights will magically emerge.
Figure 1: Misconceptions About the Abilities of LLMs
To be fair, this is not an unreasonable assumption. After all, modern LLMs can summarize documents, answer questions, write software, analyze spreadsheets, and carry on surprisingly sophisticated conversations. If they can do all of that, surely they can look at a company's ERP, CRM, and operational data and tell executives exactly what's happening in the business.
Unfortunately, the reality is a bit more complicated.
Most enterprise data wasn't created to be understood by humans, much less AI models. It's fragmented across dozens of systems, buried inside thousands of tables, and represented using terminology that only makes sense to the people who work with it every day. An ERP system might contain fields named "VBELN," "AUART," or "MATNR." A CRM system might use abbreviations, custom fields, and internal codes that have accumulated over years or even decades. To the model, these are simply tokens with no inherent meaning.
Figure 2: Decoding the Language of ERP Systems
Even when the data is accessible, another problem emerges. Data by itself rarely explains how a business actually operates. A customer record doesn't tell the model which customers are strategic accounts. A work order doesn't explain how it relates to a maintenance program. A sales order doesn't reveal the business rules, exceptions, approvals, and organizational relationships that surround it. Those details exist outside of the system somewhere in the collective knowledge of the organization.
This is where many AI initiatives encounter their first reality check. Organizations discover that access to data is not the same thing as understanding data. The model may be able to retrieve records, summarize reports, and identify patterns, but without additional context it often struggles to determine which relationships matter, which metrics are important, and which conclusions can be trusted.
In other words, AI doesn't become a business expert simply because it can see your data.
The challenge isn't that the models aren't intelligent. The challenge is that intelligence without context has limits. And as organizations move beyond experimentation and begin deploying AI to support real business decisions, that distinction becomes increasingly important.
LLMs Aren't Business Experts
To understand why AI struggles with enterprise data, it helps to understand what LLMs are actually designed to do.
At their core, LLMs are prediction engines. They're trained on vast amounts of publicly available text and learn to recognize patterns in language. This gives them an impressive ability to summarize information, answer questions, generate content, reason through problems, and even write software. In many ways, they can feel remarkably intelligent. Kind of like the world's smartest tourist.
What they are not, however, is experts in your business.
An LLM doesn't arrive with an understanding of your organizational structure, your business processes, your customer relationships, or the unique terminology that has evolved within your company over time. It doesn't know that one customer segment is strategically more important than another. It doesn't understand why a particular KPI matters to operations but not finance. It has no inherent knowledge of how work flows through your organization or how decisions are made.
Figure 3: How LLMs "Think" - Ref: Towards Data Science
Consider a simple question: "Which customers are most at risk?"
For a business leader, answering that question may involve a combination of factors such as declining revenue, increasing support incidents, contract renewal dates, payment history, asset performance, customer sentiment, and dozens of other signals. The answer often requires understanding relationships that span multiple systems and departments.
To an LLM, however, these relationships are not automatically obvious. Unless that context has been explicitly provided, the model has no way of knowing which signals matter, how they relate to one another, or how your organization defines customer risk in the first place.
This distinction is important because many organizations assume that more data automatically leads to better answers. In reality, the opposite can occur. As data volumes grow, the challenge of understanding the meaning behind that data becomes even more difficult. The model may have access to millions of records, but it still lacks the business context needed to interpret them correctly.
A useful way to think about it is this: an LLM is often like a highly intelligent new employee on their first day of work. They can read every document you give them. They can ask thoughtful questions. They can identify patterns that others may miss. But they still don't know how your business works. That knowledge comes from experience, relationships, definitions, policies, processes, and institutional memory.
And that is precisely the challenge the industry is now trying to solve.
The Missing Layer: Business Context
Enterprise software products mostly focus on capturing transactions: orders, invoices, financial postings, and so forth. What's often missing is a structured way to describe what that data means, how it relates to other data, and why it matters to the business. That gap is becoming a much bigger problem as organizations look to push the innovation envelope with agentic AI.
When a human analyst looks at a report, they bring years of experience and institutional knowledge to the table. They understand which metrics matter, which relationships are important, and which business rules should influence decision-making. AI systems lack this understanding.
As organizations move from experimentation to production AI deployments, a new priority is emerging: building explicit business context that AI systems can use to reason more effectively.
This is where technologies such as semantic models, knowledge graphs, and ontologies enter the picture.
Semantic Models: Defining What Matters
One of the simplest ways to think about a semantic model is as a business-friendly layer that sits on top of raw data.
Consider something as seemingly straightforward as revenue. Different departments may calculate it differently. One report might include returns while another excludes them. One team may focus on booked revenue while another looks at recognized revenue.
Humans often navigate these inconsistencies through experience and conversation. AI systems cannot.
Semantic models help solve this problem by establishing common (and human-readable) definitions for business concepts, metrics, and calculations. Instead of forcing users and AI systems to interpret hundreds of underlying tables and fields, semantic models provide a shared language for understanding the business.
Figure 4: Adding Context to Raw Data with Semantic Models
This is one reason semantic modeling has become such a critical capability in modern analytics platforms. Before AI can reason about a business, it first needs to understand what the business means by terms such as customer, revenue, margin, inventory, utilization, or churn.
Knowledge Graphs: Understanding Relationships
While semantic models help define business concepts, knowledge graphs help explain how those concepts relate to one another.
A graph represents information as a network of interconnected entities and relationships. Customers are connected to orders. Orders are connected to products. Products are broken down by categories. Figure 5 visualizes what this graph looks like when everything comes together.
Figure 5: Building a Knowledge Graph in Microsoft Fabric
This may sound simple, but it represents a meaningful shift in how information is organized. Traditional databases excel at storing records. Graphs excel at representing relationships.
For AI systems, those relationships provide critical context. Instead of viewing information as isolated records, AI can begin to understand how people, products, assets, processes, and events interact across the enterprise.
In many ways, knowledge graphs serve as a map of the business.
Ontologies: Creating a Shared Understanding
If semantic models define business concepts and knowledge graphs define relationships, ontologies define the rules and vocabulary that govern them.
An ontology establishes what entities exist within a business domain and how they should be interpreted. It answers questions such as:
What is a customer?
What is an asset?
What is a supplier?
How do these entities relate to one another?
Which attributes define them?
While the concept may sound academic, the goal is practical. Ontologies help ensure that people, systems, and AI models are all speaking the same language. As organizations increasingly rely on AI to reason across multiple systems and data sources, that shared understanding becomes essential.
Figure 6: Defining Ontologies in Microsoft Fabric
From Systems of Record to Systems of Understanding
Viewed individually, semantic models, knowledge graphs, and ontologies may appear to be separate technologies. In reality, they're all attempting to solve the same problem. They provide structure, meaning, and context that does not naturally exist within transactional systems.
This is why we're seeing so much investment in these areas across the software industry. Vendors are increasingly recognizing that the next wave of AI innovation won't come solely from larger models or bigger context windows. Instead, it will come from helping AI understand how businesses actually work.
The Intelligence Layer Revolution
These ideas aren't just being bandied about in a lab somewhere. Across the software industry, companies that have spent decades building systems of record are now making serious investments in something different: systems of understanding. The terminology varies by platform, but the underlying objective is remarkably consistent: create a layer of intelligence that helps AI understand how a business actually operates.
Microsoft: Building an IQ Layer for the Enterprise
Microsoft's recent introduction of concepts such as Work IQ, Fabric IQ, and Foundry IQ offers one of the clearest examples of this shift. Rather than treating AI as a standalone capability, Microsoft's vision revolves around connecting AI systems to multiple layers of organizational knowledge.
Figure 7: Microsoft's IQ Concept
Work IQ focuses on the information created through daily work, including emails, meetings, chats, documents, and business activities. Fabric IQ extends that understanding into the data estate by leveraging semantic models, business metrics, analytics, and organizational data. Foundry IQ aims to provide a common context layer for AI applications, agents, and enterprise workflows.
Viewed collectively, these initiatives represent an attempt to build a comprehensive understanding of both how a business operates and how work gets done within it.
Figure 8: Introducing an Intelligence Layer for Agents with Fabric IQ
SAP: Teaching AI to Understand the Shape of Enterprise Data
SAP is approaching the challenge from a different perspective: the structure of business data itself.
The centerpiece of this strategy is SAP's investment in Tabular Foundation Models (TFMs). TFMs are AI models trained not on text, but on the kind of structured, relational data that lives inside ERP systems. In other words, purchase orders, material master records, vendor relationships, cost centers, plant assignments. This is the data that actually runs the business, and it looks nothing like the documents and web pages that power most foundation models today.
The premise is significant. Relational data in an ERP system carries meaning not just in its values, but also in its relationships. A sales order connects to a customer, a delivery, a billing document, a cost object, and a material — each tied together through a web of foreign keys and business rules that encode decades of process logic. A model that can natively understand that structure doesn't just retrieve records; it can also reason about business state.
Figure 8: Tabular Foundational Model Concepts - Ref: Towards Data Science
SAP's work in this area aims to give AI systems a working knowledge of how enterprise data is organized, what the relationships between entities mean in a business context, and how to navigate that structure to answer real operational questions.
This isn't a new direction for SAP. The company has been investing in the SAP Knowledge Graph for several years, a structured representation of the semantic relationships across its entire application portfolio. The Knowledge Graph maps how objects like customers, materials, purchase orders, and cost centers relate to one another across modules. It gives AI systems a shared ontology to reason against, rather than treating each data element in isolation.
That foundational work is now paying dividends. As SAP layers Joule and its broader AI capabilities on top of the new SAP Business AI Platform, the Knowledge Graph provides the semantic backbone. It encodes the meaning behind the data, not just field names and foreign keys.
Salesforce, ServiceNow, and the Data Platform Vendors
Similar patterns can be found elsewhere across the software industry.
Salesforce is positioning Data 360 as a unified customer intelligence layer that connects information across customer touchpoints and business systems. Meanwhile, ServiceNow continues to expand its enterprise knowledge graph capabilities to help AI reason across workflows, services, assets, and operational processes.
Meanwhile, data platform vendors such as Databricks and Snowflake are investing heavily in metadata management, governance, semantic understanding, and cataloging technologies. While these capabilities are often discussed through the lens of data management, they ultimately serve the same purpose: creating a richer understanding of enterprise information.
In many respects, the lines between data platforms, knowledge platforms, and AI platforms are beginning to blur.
The Emergence of Enterprise Memory
If we take take a step back, an even broader pattern comes into focus.
The gap AI systems have historically struggled to close isn't capability, it's context. Most enterprise AI deployments sit on top of systems that capture what happened but not why decisions were made, how work actually flows, or what institutional knowledge exists inside the people doing the work.
Enterprise memory is the effort to close that gap.
When AI systems have access to structured context — defined relationships, documented processes, organizational terminology, historical decisions — they can do more than just generate responses. They can support decisions with relevant precedent, route work based on actual business rules, and surface knowledge that would otherwise live only in someone's inbox or institutional memory.
The practical value is straightforward. Organizations spend considerable time and money rebuilding context that already exists somewhere: in past projects, in tribal knowledge, in undocumented workflows. Systems that capture and retain that context reduce onboarding friction, accelerate decision-making, and make automation more reliable because the underlying logic is grounded in how the business actually operates.
This is what distinguishes the next generation of AI deployments from earlier waves. The question is no longer whether a model can generate a reasonable answer. The question is whether the system surrounding that model knows enough about your business to make the answer useful.
Closing Thoughts
One of the clearest lessons from the past few years of AI adoption is that intelligence alone isn't enough.
Most organizations have spent decades building systems that capture transactions and generate data. What they're discovering now is that data rarely provides the context AI needs to reason effectively. Before a model can recommend an action or surface a risk, it first needs to understand what the business actually does. In other words, how processes work, how data relates, what terms mean in practice.
This is why so many vendors are investing in semantic layers, knowledge graphs, and ontologies. The names may differ, but the underlying goal is the same: turning enterprise data into enterprise understanding.
For business leaders, the implication is straightforward. The value you get from AI will depend less on which model you choose and more on how well you have organized your business knowledge. Trusted definitions, connected data, and clear relationships are not just IT concerns. They're the foundation that determines whether AI can do anything useful with what you have built.
Bottom line: if the AI doesn't understand your business, it cannot meaningfully improve it.
