Retrieval-Augmented Generation, or RAG, is now one of the most practical ways for organizations to connect AI with their own business information.

Instead of relying only on what a large language model learned during its original training, a RAG system searches approved company sources for information relevant to a user’s question. It then adds that information to the prompt used to generate an answer.

For Dynamics 365 customers, this can help AI work with customer records, policies, product information, service documents and other business knowledge. A user might ask, “What discount agreements do we have with ACME Corporation?” and receive an answer grounded in current company information, without retraining the underlying model.

When it is designed and governed well, RAG can improve relevance, reduce unsupported answers and give organizations more control over the information available to AI. But there is an important limitation: RAG retrieves information; it does not repair it.

---

Why RAG is so powerful

RAG addresses a basic challenge with general-purpose AI models: they do not automatically know an organization’s private information or the latest changes to its policies, customers and operations.

By retrieving information at the time of a request, RAG can provide:

• Current, company-specific knowledge
• More relevant and supportable answers
• Lower risk of unsupported or fabricated responses
• Greater control over approved information sources
• Improved traceability and governance

These benefits, however, depend on retrieval finding the right information. A sophisticated AI model cannot resolve every ambiguity created by a poorly managed knowledge base.

What happens when the data is trusted

Imagine an organization with a well-managed knowledge base. Its policies are current, duplicate documents have been removed, metadata is consistent and every important document has a clear owner. Superseded material is archived or excluded from retrieval.

A service manager asks: “What is our customer escalation process?”

The RAG system identifies three highly relevant documents and supplies approximately 12,000 tokens of supporting content to the model. Because the retrieved material is focused and consistent, the system can generate a fast, accurate response at a relatively low processing cost.

The user may still need to apply professional judgment, but the answer has a strong and clearly governed foundation.

What changes when the data is poor

Now imagine that the same knowledge base contains five versions of the escalation policy, duplicated procedure documents, outdated instructions, inconsistent metadata and no reliable indication of which document is authoritative.

The same question might retrieve 15 documents and more than 50,000 tokens of content. Some documents may agree, while others conflict on response times, responsibilities or approval requirements.

The consequences extend beyond a larger prompt:

• More content must be searched, ranked and processed
• Responses may take longer to generate
• The model has more opportunities to combine conflicting instructions
• Users may need to locate and verify the original documents
• Confidence and adoption decline when answers cannot be trusted

The biggest problem is token waste

Organizations shouldn’t aim to minimize AI interactions across the board. The goal is to eliminate consumption that yields no business value.

Common sources of waste include:

• Repeated retrieval of duplicate customer records
• Old or irrelevant activities included in prompts
• Multiple versions of the same document
• Poor metadata that causes broad, unfocused searches
• Agents repeating unsuccessful steps
• Overly large prompts and unnecessary conversation history
• Users verifying answers because the underlying data is unreliable

The organization pays more to obtain a worse result and employees still perform the manual verification that AI was intended to reduce.

How Dynamics 365 customers can respond

Effective AI cost governance begins with measurement. Organizations should monitor consumption by environment, agent, process and business outcome, not merely as one monthly total.

They should also identify which Dynamics 365 records and fields are repeatedly supplied to AI. Duplicate accounts and contacts, incomplete records, stale activities and inconsistent classifications can increase processing while weakening results.

Before scaling Copilot or autonomous agents, establish baseline metrics for consumption, response quality, and human-in-the-loop intervention. From there, prioritize cleaning the records and tightening the retrieval logic that drive high-frequency tasks. Spending caps and threshold alerts protect budgets from runaway surprises, but clean data and precise retrieval eliminate the root cause of compute waste.

The hidden cost of RAG

Discussions about AI costs often focus on model inference: the tokens processed when a user submits a request and receives an answer. In a RAG environment, costs can arise across the entire information lifecycle.

1. Data preparation

Documents and records must be collected, cleansed, parsed and assigned useful metadata. Poor source data increases the work required before information can be made searchable.

2. Chunking and embedding

Content is divided into smaller sections, or chunks, and converted into numerical representations called embeddings. Duplicate and obsolete content creates unnecessary chunks and embeddings that must be generated, managed and potentially regenerated whenever the source changes.

3. Storage and indexing

Embeddings, document content and search indexes require storage. If the source contains repeated or low-value information, the organization pays to store and index material that does not improve the answer.

4. Retrieval and reranking

Each query searches for potentially relevant content. Many systems then re-rank the results to decide which chunks should be included. Weak metadata, duplicates and contradictory versions increase the volume of candidates and make it harder to identify the authoritative source.

5. Prompt expansion

Retrieved chunks are added to the model’s prompt. Irrelevant or repetitive content consumes additional tokens and can push useful information deeper into an increasingly crowded context.

6. Human review

This is often the largest hidden cost. When users do not trust an AI answer, they check source documents, compare versions, correct the response or repeat the search. The direct technology cost may be visible; lost employee time and delayed decisions often are not visible.

An illustrative cost comparison

Consider a company knowledge base containing one million documents.

In a clean environment, a typical request might retrieve five focused chunks and create a prompt containing approximately 15,000 tokens. This becomes the organization’s baseline for response time and processing cost.

Now add 20% duplicate documents, 15% obsolete content and inconsistent metadata. A similar request might retrieve 18 chunks and produce a prompt containing approximately 55,000 tokens.

The exact financial impact will depend on the models, architecture, retrieval settings, query volume and commercial pricing involved. However, the direction is clear: more unnecessary content must be prepared, embedded, stored, searched, reranked and processed. Monthly AI costs can become several times higher even though the quality of the answer declines.

For Dynamics 365 customers, the same problem can extend beyond documents. Duplicate accounts, outdated contacts, incomplete activities and inconsistent customer classifications can all weaken retrieval and distort the context provided to Copilot or an AI agent.

Build data readiness into the RAG strategy

RAG should not begin with the question, “Which model should we use?” It should begin by asking whether the information being connected to the model is accurate, current, relevant and governed.

Organizations should identify authoritative sources, remove unnecessary duplication, improve metadata, assign document ownership and prevent obsolete content from being retrieved. Within Dynamics 365, this also means addressing duplicate records, validating important contact information, completing priority fields and monitoring data quality over time.

The objective is not to clean every piece of information indiscriminately. It is to improve the data most likely to influence important AI answers and actions.

---

Final Takeaway

RAG can make AI more useful by grounding it in company knowledge. It cannot make unreliable information trustworthy simply by retrieving it.

Poor-quality data creates costs at every stage, from preparation and embedding to retrieval, prompt processing and human verification. It also increases the risk that users receive conflicting, outdated or misleading answers.

Before scaling RAG across Dynamics 365 and other business systems, organizations should assess the quality of the information on which it will depend. An AI Readiness and Data Quality Health Check can identify high-risk data, duplicate content and governance gaps, then provide a prioritized improvement plan.

The reality is simple: bad input doesn’t just produce bad results. It creates more expensive bad results.

This article is the second in our AI Economics and Readiness Series, read further articles on this topic here:

The AI Cost Paradox.