Your AI Problem Might Be an Architecture Problem

When an AI application gives inconsistent results, prompt tuning is usually the first place people go. Teams rewrite the instructions, add examples, make the rules more explicit, tell the model to double-check itself, and add another paragraph explaining exactly what it got wrong.

Sometimes that works. Other times, it keeps everyone busy while the underlying design problem goes untouched.

A recent production case study from DiDi and AWS offers a useful example. DiDi rebuilt part of its international contact-center quality-assurance system using Amazon Bedrock. According to the companies' published results, intent-verification accuracy increased from 38 percent to 86 percent, compliance-scoring accuracy exceeded 90 percent, and Voice of Customer analysis that previously took hours could be completed in minutes. AWS details the architecture and results in its September 2026 case study.

DiDi had already tried multiple rounds of prompt tuning. The major improvement came when the team narrowed each model call and controlled the information available at each stage.

The design flaw

Part of DiDi's system verifies the contact reason assigned to a customer-service interaction. Its taxonomy branches into increasingly specific categories, so the application has to decide whether the representative selected a reasonable label and, when necessary, recommend a better one.

The first implementation gave the language model the conversation and the complete classification tree, then asked it to evaluate the representative's choice. It performed poorly.

According to the DiDi and AWS case study, the model tended to compare the selected category with every available alternative. If it found something slightly more precise, it could reject the original classification even when that choice was perfectly reasonable. Multiple rounds of prompt tuning did not correct the behavior.

Nothing was technically broken. The model was doing the broader job it had been given: consider all available options and decide which one fits best. The application only needed to know whether the selected label was reasonable.

DiDi split verification and classification into separate stages. During verification, the model sees the conversation and the category that was selected. It judges that choice without the full taxonomy competing for attention. Only when verification fails does the system provide the complete classification tree and ask for an alternative. Tickets labeled "Other" also follow a separate path because identifying a gap in the taxonomy is a different job from validating an existing category.

After the redesign, DiDi reported that intent-verification accuracy increased from 38 percent to 86 percent. The model did better because it received a smaller, more precise job.

Context overload

AI teams often assume that more information will produce a better answer. Large context windows make it possible to send enormous amounts of documentation, history, rules, metadata, and examples in a single request, but capacity is not the same as relevance.

Every piece of context affects the model's decision space. Information that matters somewhere in the workflow can still interfere with the decision being made right now. DiDi's classification tree was necessary when the system had to recommend a new label. It was noise when the system only had to validate the current one.

The same problem shows up in other applications. A support assistant may need access to an entire knowledge base, but a billing question probably does not require every troubleshooting document. An agent may have twenty available tools even though a particular step should use only two. A document-review system may rely on several policies across the full workflow while one compliance decision depends on a single policy section.

Context is part of the architecture. The better design question is: What information does the model need to make this specific decision correctly?

Deterministic logic

Another part of DiDi's architecture applies to almost any AI system. The system evaluates customer interactions against compliance criteria, but it does not ask the model to determine every fact used in the evaluation.

For computable values such as agent response-wait times, DiDi calculates the result in code and gives that value to the model. For rule-based criteria, a programmatic validation layer can check the model's judgment against the original conversation. AWS describes both patterns in the implementation.

If the application can calculate something reliably, calculate it. Code can count occurrences, compare timestamps, check whether a database value exceeds a known threshold, and apply fixed business rules. Spending tokens to make a probabilistic system recreate deterministic logic adds cost and another place for errors to creep in.

Models are useful when interpretation is required. Was the representative dismissive? Did the conversation address the customer's underlying problem? Which contact reason best describes an unusual request? Are differently worded complaints describing the same issue?

Those are judgment problems. Whether the customer waited 47 seconds for a reply is arithmetic. The distinction sounds obvious on paper, yet production systems blur it all the time.

Model safeguards

DiDi also built controls around the model instead of assuming stronger instructions would make every output trustworthy. Personally identifiable information is masked before it reaches the model, grounding checks flag unsupported responses, and some judgments are validated programmatically afterward.

The spelling-error check is a useful example. The model identifies possible errors, but the application verifies them against the representative's messages and applies the pass/fail threshold using the confirmed count. The case study describes these validation and privacy controls as part of the production architecture.

That division of responsibility is practical. A model can identify candidates. Code can verify objective facts and enforce known thresholds. Humans can review decisions when ambiguity or impact warrants it.

These controls matter more as organizations move beyond chat interfaces into systems that classify records, score interactions, update databases, call tools, or influence business decisions. A slightly wrong chatbot answer is one kind of risk. Recording that answer as a business decision creates a larger one. The architecture determines how far the error can travel.

Smaller workflows

DiDi's QA system uses separate pipelines for intent verification, compliance evaluation, and Voice of Customer analysis. Some of those pipelines divide the work further. The compliance system assembles the criteria relevant to a ticket's language and business line, while the Voice of Customer workflow separates structured issue extraction, semantic clustering, statistical ranking, and report generation. AWS's architecture diagrams show how those responsibilities are divided.

This looks a lot like ordinary software engineering. Different problems get different components. Deterministic operations stay deterministic. Configuration lives outside the prompt when appropriate. Structured outputs create predictable interfaces between steps. The model handles the work that benefits from its capabilities instead of becoming responsible for the entire application.

The boundaries also make failures easier to trace. When one giant model call produces a bad answer, the cause could be the instructions, retrieved information, classification logic, business rules, model behavior, output formatting, or an interaction among several of them. A staged system lets the team measure where the failure occurred. That is usually more valuable than squeezing another clever sentence into the system prompt.

Before prompt tuning

Prompt changes are cheap, so they deserve a place in the debugging process. They should not be the automatic answer to every quality problem. When an AI workflow underperforms, check the application around the prompt:

1. Is the model doing several distinct jobs at once? Verification, classification, extraction, calculation, summarization, and decision-making may belong in different steps. Splitting them makes each decision easier to evaluate and control.

2. Is the model seeing information it does not need yet? Relevant information can still create noise. Give each step the smallest useful context, not the largest available context.

3. Is the model inferring something the software can determine? Calculate deterministic facts in code, query authoritative systems directly, and apply fixed rules programmatically. Use the model for work that requires interpretation.

4. Can important outputs be checked after generation? Validate IDs, calculations, thresholds, schemas, permissions, policy conditions, and other objective claims outside the model whenever possible.

5. Can you tell which part of the workflow failed? If individual steps cannot be isolated and measured, improvement becomes guesswork. Instrument the workflow so a bad final result can be traced to the component that produced it.

Once those responsibilities are clear, prompt tuning becomes more useful because the prompt sits inside a system with sensible boundaries.

About the results

The 38-to-86-percent improvement comes from DiDi's production validation as reported in a joint DiDi and AWS case study. It is not an independent benchmark, and the published information does not support assuming that another organization would see the same gain from a similar redesign.

The value of the example is the sequence of events: repeated prompt tuning failed to correct a specific production behavior, while restructuring the workflow produced a substantial improvement. The team described context management and data definitions as important capabilities developed through the project. Its implementation shows what that means in practice: isolate tasks, control the information available at each stage, keep deterministic processing in code, validate what can be checked, and make model decisions traceable. The full implementation is documented in the AWS case study.

Those principles hold whether the system uses Amazon Bedrock, OpenAI, Anthropic, Google, or a provider that has not launched yet.

Architecture first

Prompt engineering matters, but it sits inside a larger system. When a workflow keeps producing unreliable results, rewriting the prompt for the fifteenth time can delay the harder question: Did we design the workflow correctly?

DiDi's system shows how much accuracy depends on decisions made before and after the model call. The application determines which problem the model receives, which information it sees, which calculations happen elsewhere, what gets validated, and what happens to the answer afterward. Weak decisions in those areas leave even an excellent prompt with too much cleanup to do.

Before rewriting the prompt again, check the architecture.

From demo to production

ShruggieTech helps organizations design practical AI systems around real business processes, including context management, RAG, automation, integrations, evaluation, and the controls that belong around the model.

If an AI workflow keeps producing inconsistent results, we can help determine whether the prompt needs work or the surrounding system needs a better design.