DIY-ing Your Own AI Agent? Consider the Maintenance Burden

In our last article, we explored  what it takes to scale an AI agent from a simple demo to a production-grade system. Spoiler alert: vibes won’t get you there. Shipping a robust AI agent is a formidable systems engineering challenge, demanding expertise far beyond linking an LLM to a vector database. But for those who succeed, launching an agent isn’t the end of the journey. In fact, it’s the beginning of an even more complex one.

If building your agent was “Day 1,” welcome to  “Day 2”: keeping it alive, relevant, and performant. Many assume that once deployed, an AI agent is a static asset. The truth is that it’s a dynamic, rapidly degrading system battling against the constant flux of the digital world. The engineering discipline required to maintain these systems is a new frontier, one that makes the initial deployment look like the easy part.

From API drift to gaps in data governance, there’s a lot that can (and probably will) go wrong throughout the agent lifecycle, especially if you’re DIY’ing rather than taking a platform approach. Let’s explore some common Day 2 challenges.

1. The mechanics of model and embedding obsolescence

Migrating an agent to a different LLM is not a configuration change — it’s a micro-migration project fraught with technical peril. (Those of us who have done it for experiments, hackathons, and tinkering builds know this!) Every layer, from hardware to prompts, presents a source of drift.

• Tokenizer and context misalignment: Swapping a model from one family to another (e.g., OpenAI’s GPT series to Meta’s Llama series) introduces a tokenizer mismatch. The same text string can tokenize into a different number of tokens with different boundaries, potentially causing context window overflows or subtle shifts in model attention. A prompt that was 3500 tokens under cl100k_base might be 4100 tokens under Llama’s sentencepiece tokenizer, breaking your 4k context window.
• Structured output instability: The reliability of forcing structured output (e.g., JSON) varies wildly between models. A production system cannot simply trust the LLM to return valid JSON. Robust solutions require implementing a validation and repair loop, often using a library like instructor to bind the LLM’s output to a Pydantic schema, or even employing a second, smaller model tasked specifically with correcting the primary model’s malformed output.
• Quantization and inference engine drift: An LLM is not just its weights but also the runtime that executes it. Moving an agent from a fp16 precision model served on vLLM to a 4-bit AWQ quantized version on a TensorRT-LLM backend to reduce costs can cause significant shifts in output logits. This seemingly minor change can alter the probability distribution of tokens enough to break deterministic sampling (temperature=0) and subtly change the agent’s behavior in unpredictable ways. This is a hardware- and software-stack-dependent form of drift that requires re-evaluation on every infrastructure change.
• The fine-tuning vs. meta-prompting dilemma: When a new base model is released, maintenance teams face a recurring, complex technical decision. Option A: Spend hundreds of engineering hours meticulously re-crafting complex, few-shot, chain-of-thought meta-prompts. Option B: Spend weeks and significant budget fine-tuning the new base model on your corpus of old prompts and ideal completions to teach it your required formats. Choosing the wrong path results in massive wasted engineering cycles.
• Zero-downtime re-indexing and multi-modal complexity: When upgrading an embedding model, the “great re-indexing” is a major SRE challenge. The standard production pattern is to implement a shadow index: your application writes to both old and new indices while a backfill process populates the new index. A routing layer then directs traffic to the new index, and only after validation is the final cutover made. This multi-week project explodes in complexity as RAG evolves towards multi-modal embeddings. Upgrading now requires not only re-indexing all text but also implementing a new image processing pipeline, potentially increasing storage costs and indexing time by an order of magnitude.

2. The granular failure modes of data and tooling contracts

An agent’s tools are its lifelines, but these connections are subject to constant, low-level failures that simple retries cannot solve. If you’re building a DIY agent, that means having to worry about maintaining a steady API state, ensuring your tooling can manage errors and streamline latency, and constantly updating RAG pipelines as new business data is integrated.

• Semantic API drift: Syntactic drift (a schema change) is the easy problem. The far more insidious issue is semantic drift. A financial API might change its definition of a “risk score” from a 0.0-1.0 float to a categorical “LOW” | “MEDIUM” | “HIGH” string. The API contract is still valid and won’t throw a 400 Bad Request, but the agent’s internal logic, which expected a float for comparison, is now broken. This necessitates semantic monitoring and versioned tool definitions.
• Stateful fault tolerance and latency-aware planning: Simple, stateless retries are insufficient for multi-step agentic chains or directed acyclic graphs (DAGs). For context, DAGs are a model for representing dependencies between tasks in a workflow. If a tool in step 3 of a 5-step plan fails, the orchestration engine must be stateful enough to not only retry but also potentially re-plan the entire downstream path. A truly sophisticated planner must also be a cost-aware optimizer. To choose between a fast, cached tool (p99: 50ms) and a slow, comprehensive one (p99: 2500ms), the planner needs access to near-real-time observability data about its own tools, requiring a feedback loop from your monitoring stack into the agent’s decision-making context.
• Recursive RAG and vector DB maintenance: Advanced agents perform recursive retrieval — retrieving a document, finding a reference, and then retrieving that entity. This risks runaway execution from circular references. Production-grade recursive RAG requires explicit depth counters, visited-node tracking, and token budget controls. This is compounded by vector database maintenance. When vectors are deleted (e.g., for GDPR), they often leave behind broken edges in the HNSW (Hierarchical Navigable Small World) search graph. This degrades recall and latency over time, necessitating periodic, resource-intensive VACUUM or OPTIMIZE commands to re-prune the graph, a critical but often overlooked operational task.

3. The nitty-gritty of continuous evaluation (CI/CE)

A CI/CD pipeline ensures your code runs,while a CI/CE pipeline ensures your agent thinks correctly. In the context of a traditional software DevOps lifecycle, this means consistently building and maintaining evaluation sets (synthetic data) to continuously assess model performance. This process involves extensive manual and automated testing, including using an LLM as a judge to determine the optimal model. 

• The “golden set” treadmill and synthetic data generation: Your evaluation dataset (the “golden set”) requires constant, manual curation to add new failure modes as they’re discovered. To overcome the inherent lack of real-world edge cases, the state-of-the-art solution is to build an adversarial loop using another LLM to generate challenging synthetic data (e.g., “Create 100 user queries that are intentionally ambiguous”). This synthetic dataset is then fed into your candidate agent, allowing you to automatically discover and patch weaknesses.
• Implementing LLM-as-an-evaluator and mitigating bias: For abstract metrics like “faithfulness” or “relevance,” using a powerful LLM as a judge is the standard approach (frameworks like Ragas provide a starting point). However, these judge models often exhibit positional bias — a tendency to prefer the response listed first. To achieve a statistically sound result, every A/B evaluation must be run twice, swapping the order of the responses ([A, B] then [B, A]), and only trusting a consistent preference. This doubles evaluation cost but is essential for trustworthy metrics.
• Component-level metrics and precise cost attribution: You must monitor metrics at multiple levels. A drop in your retriever’s precision from 0.85 to 0.75 is a critical leading indicator of system degradation, even if end-to-end user satisfaction hasn’t moved yet. This requires precise, per-step cost and performance attribution. In a complex agentic chain, attributing cost is a distributed tracing nightmare. Accurately summing the prompt_tokens and completion_tokens for each distinct LLM call and associating that total cost back to the initial user query requires a meticulous, context-propagated tracing system.

The Agentforce approach: A deeply integrated, multi-layered solution for the agent lifecycle

The sheer technical depth of these “Day 2” maintenance problems makes it clear that building a solution from scratch is not just an engineering project— it’s a commitment to building and maintaining a complex internal platform for years to come. The alternative is to leverage a pre-built, integrated stack where each component is specifically designed to solve a distinct part of this lifecycle puzzle.

1. The data foundation: Salesforce Data Cloud

Data Cloud’s role is to solve the formidable data and RAG maintenance challenges. It is the managed, enterprise-grade grounding layer for your agents.

Data Cloud ingests, cleans, and harmonizes data from across the enterprise into a unified data model. This means your agent’s RAG system queries a single, governed source of truth. It handles the entire low-level retrieval pipeline as a service:

• Unified and contextual real-time data integration: Data Cloud unifies structured and unstructured data across enterprise systems, lakes, warehouses, and Customer 360 in real time using over 270 connectors and zero-copy architecture. This enables Agentforce to access a single source of truth enriched with a rich metadata layer for deep contextual understanding, crucial for accurate, informed decision-making and personalized autonomous actions.
• Industry-leading RAG and hybrid search capabilities: Salesforce Data Cloud incorporates advanced retrieval-augmented generation techniques coupled with hybrid search (combining semantic vector search with exact keyword matching). This allows Agentforceto retrieve, augment, and summarize relevant data efficiently from both structured and unstructured sources (emails, tickets, images, voicemails), achieving superior accuracy and context-aware responses beyond traditional RAG systems.
• Scalable, governed, and extensible platform for autonomous actions: As a hyperscale data engine, Data Cloud supports real-time indexing, search, analytics, and immediate calls to action within agentic workflows. Built-in data governance ensures secure, compliant operation with access control and regulatory adherence. Its open ecosystem and integration with Salesforce’s Zero Copy Partner Network allow extensible, scalable deployment and integration into diverse enterprise architectures, enabling complex autonomous workflows and hyper-personalized customer experiences.

2. Enterprise connectivity layer: Magic of MuleSoft

MuleSoft’s role is to solve the tool and integration brittleness problem. It acts as the secure, stable, and managed “connective tissue” between your agent and the chaotic world of backend systems and third-party APIs.

• API abstraction and insulation: MuleSoft provides a crucial abstraction facade. Instead of an agent making brittle, direct calls to a dozen different APIs, it makes calls to a single, stable set of MuleSoft APIs. When a backend system’s API undergoes a breaking change (e.g., schema drift), the transformation logic is updated within the MuleSoft integration layer. The API contract presented to the agent remains unchanged, effectively insulating the agent’s tool from downstream churn. And more recently, with MuleSoft MCP support developers can transform any API to be exposed as a structured, agent-ready asset. This enables AI agents to not only gather context from your systems but also perform tasks across them — securely, reliably, and at scale.
• Centralized security and governance: MuleSoft centralizes all API security. The agent authenticates once to the MuleSoft layer, which then securely manages credentials, authentication flows (e.g., OAuth 2.0), and authorization for all backend systems. This is where policies for rate limiting, threat protection, and request validation are enforced, providing a unified security posture for all of the agent’s tools.
• Discoverable tool marketplace via Anypoint Exchange: MuleSoft’s Anypoint Exchange functions as a private marketplace for your company’s APIs. Agent developers don’t have to build tool connectors from scratch. Instead, they can browse a catalog of pre-built, documented, and governed APIs, find the capability they need (e.g., lookup_inventory), and immediately integrate it into their agent.

3. The intelligence and lifecycle hub: Agentforce

With data and connectivity managed by Data Cloud and MuleSoft, Agentforce serves as the “cockpit” for designing, orchestrating, and — most importantly — maintaining the agent itself.

It solves the model and lifecycle challenges.

• Stateful orchestration engine: Agentforce provides the framework for designing the agent’s reasoning process (its DAG). This is where you chain together calls to LLMs, invoke tools via the MuleSoft layer, and query for knowledge from Data Cloud. The engine is inherently stateful, providing built-in primitives for complex fault tolerance, such as re-planning execution paths based on real-time tool latency data provided by the observability feedback loop.
• Model abstraction and adaptation: The Agentforce platform features a “model adapter” layer that makes migrating between LLMs a managed process. When you select a new model, this adapter automatically recompiles the agent’s abstract prompt definitions into the specific, optimized format required by the target model — handling everything from tokenizer-aware prompt construction to applying quantization-aware inference parameters.
• Integrated continuous evaluation (CI/CE) suite: Agentforce directly tackles the core maintenance challenge with a built-in evaluation suite. It provides an agent testing center, which allows you to run scale tests of how your agents will perform qualitatively even before you deploy them. Built-in version control helps guide continous upgrades and capability changes to your agent.

By clearly delineating these responsibilities, the Salesforce stack transforms agent maintenance from a chaotic, reactive fire drill into a structured, managed, and sustainable engineering discipline. It allows organizations to bypass the immense cost of building this foundational platform themselves and focus on what matters: creating intelligent, reliable, and secure AI experiences.

Can your agent serve TEA?

As you build and maintain your AI agents, you must ask yourself: Is this approach steeped in TEA? What’s that, you may ask? It stands for the three pillars of trusted AI: transparency, explainability, and auditability.

Transparency into the cost and performance of every component. Explainability into why the agent made a specific decision or chose a particular tool. And auditability to provide an immutable, step-by-step record for compliance, security, and debugging. Without these three pillars, an AI agent remains a clever but dangerous prototype. With them, it can become a trusted, enterprise-grade asset.