Evals Are the New Uptime: Salesforce’s Agent Fabric Just Made Agent Quality an Operations Metric

<p></p> <p><strong>Key findings</strong></p> <ul> <li>Salesforce says its Agent Fabric control plane handles <strong>more than 1.2 billion requests daily</strong>. On October 27, 2026, it announced production evaluations and cost controls designed to detect agent failures and intervene before they affect customers.</li> <li><strong>Fewer than half of organizations with agents evaluate them on live traffic</strong>, according to Salesforce’s announcement. The gap is operational: teams may observe activity without continuously measuring whether agents complete work correctly.</li> <li><strong>96% of enterprises run agents on more than one platform</strong>, according to research sponsored by Salesforce and published by <em>The Atlantic</em>. Agent reliability therefore needs controls that can work across platforms, not just inside an individual model or agent builder.</li> <li>The executive response is an <strong>agent SLO</strong> for each production use case: a defined quality threshold, hard spending limit, intervention path, and re-evaluation gate before the agent returns to service.</li> </ul> <h2>What You’ll Learn</h2> <p>How to translate familiar uptime and error-budget practices into agent operations, what Salesforce announced and when its capabilities are expected to be available, and how to account for evaluation work when assessing the economics of a production agent.</p> <h2>Who Should Read This</h2> <p>CIOs, CTOs, COOs, chief data and AI officers, customer-service executives, and platform leaders accountable for customer-facing or business-critical AI agents.</p> <h2>The operational problem is silent failure</h2> <p>An agent can remain technically online while doing the wrong job. It may give an inaccurate answer, mishandle a tool call, or continue consuming tokens after a model or API change. Unlike a conventional service outage, these failures can be quiet: requests still complete, but quality slips and costs accumulate before anyone notices.</p> <p>That is why uptime alone is insufficient as a measure of agent health. Reliability engineering already tracks service-level indicators such as availability and latency against service-level objectives. Production agents need comparable operational measures for the outcomes they produce: <strong>quality, cost, and safe intervention</strong>.</p> <p>Research reinforces the gap. In its <em>State of Agent Engineering</em> report, published June 12, 2026, LangChain reported that 89% of respondents had implemented some form of agent observability, while 37.3% reported online evaluations. The survey collected 1,340 responses in late 2025. Visibility into agent activity is becoming common; continuous assessment of live performance remains less established.</p> <h2>Salesforce’s announcement brings evaluation and action closer together</h2> <p>Salesforce’s October 27 announcement expands Agent Fabric, its control plane for managing and governing agents across platforms. The new <strong>Agent Evals</strong> capability is designed to generate tests before deployment, then evaluate agents, large language models, and MCP servers in production against defined quality thresholds. When results fall short, it can trigger predefined interventions, including a kill switch, with human approval where policy requires it.</p> <p><strong>API Evals</strong> target a different failure source: changes in the APIs agents depend on. The feature is designed to generate functional and performance tests from API specifications to catch breaking changes before they reach production.</p> <p>The availability dates matter. Salesforce said Agent Evals and API Evals are planned for beta in <strong>November 2026</strong> and general availability in <strong>January 2027</strong>. The Agent Kill Switch is generally available as of the announcement; the triggered-agent kill-switch capability is planned for January 2027. These are not all the same release status, and executives should distinguish generally available controls from upcoming evaluation features.</p> <p><img src="https://cdn.marblism.com/VzGhDqnKr-4.webp" alt="Minimal illustration of an agent control loop with quality thresholds, spend caps, intervention and re-evaluation" style="max-width: 100%; height: auto;"></p> <p>The announcement also includes cost controls generally available on October 27: a FinOps Dashboard, Model Wallets, and Cost Optimization Policies. Salesforce says hard spending limits can stop requests before costs exceed budget. Its policies use semantic caching, payload compression, and response trimming to reduce token usage by <strong>up to 75% in test scenarios</strong>. A test result, not a production-savings guarantee.</p> <p>Together, these changes reflect a shift in how agent operations can work: measure behavior, define an acceptable boundary, then connect a breach to a response. The tools do not define the right quality threshold for a particular business. The organization must.</p> <h2>Define an agent SLO for every production use case</h2> <p>An agent SLO is an operational contract between the business owner and the team responsible for the system. It specifies what acceptable service means for one use case, and what should happen when performance falls outside the agreed range.</p> <p>A practical agent SLO has four parts:</p> <ol> <li><strong>Quality threshold on live traffic.</strong> Define measurable criteria for task completion, factual accuracy, policy compliance, tool selection, and escalation. Set thresholds according to the consequences of failure; a scheduling assistant and a regulated advice workflow do not have the same risk profile.</li> <li><strong>Hard spend cap.</strong> Set a budget per agent, application, or business unit. Specify what happens at the limit, such as blocking further requests, routing to a fallback, or requiring an authorized override.</li> <li><strong>Intervention path.</strong> Identify the owner and response for a breach: alert, restrict, quarantine, or shut down. Keep human approval in the loop when risk or policy requires it.</li> <li><strong>Re-evaluation gate.</strong> Require the agent to pass the relevant quality tests after a prompt, model, tool, or API change before it returns to production.</li> </ol> <p><img src="https://cdn.marblism.com/KVOotRNV6sQ.webp" alt="Illustration of an agent SLO linking live quality thresholds, spend caps, intervention, and re-evaluation" style="max-width: 100%; height: auto;"></p> <p>These controls should be defined at the use-case level, not as a single enterprise-wide accuracy number. A customer-service agent answering order-status questions should be measured on whether it retrieves the correct order, gives a supported answer, and escalates exceptions appropriately. An internal research agent may need a different quality rubric and a different cost ceiling.</p> <h2>Multi-platform operations require a layer above the agents</h2> <p>With 96% of enterprises reportedly running agents across more than one platform, reliability policies tied to a single vendor leave gaps. A business may have distinct agents, models, APIs, and MCP servers. Each with its own release cycle and observability tools. The operational objective is consistent measurement and escalation across that estate.</p> <p>This is where a control plane can help, but technology does not substitute for clear ownership. Each agent needs a named business owner, an accountable technical operator, a defined data source for evaluation, and a documented escalation route. API testing must also be included: an agent’s behavior can change when a dependency changes even if its prompt and model remain untouched.</p> <p>For the executive team, the key question is not simply, “Can we see our agents?” It is: <strong>Can we detect a material quality or cost breach, identify who owns it, and contain it within an agreed time?</strong></p> <h2>Measure the economics of verified work</h2> <p>A lower-cost model is not necessarily a lower-cost operation if it produces more wrong answers, retries, human reviews, or customer contacts. Salesforce’s retailer example illustrates the trade-off: after switching an order-support agent to a cheaper model, Agent Evals detects inaccurate answers about delayed orders. The agent is quarantined with human approval, its instructions are refined, and it is evaluated again before returning to production.</p> <p>Finance and operations teams should therefore include evaluation runs, monitoring, remediation, and human review in their assessment of production economics. The useful comparison is not model price alone, but the full operating cost of work that meets the business’s acceptance criteria. If the evaluation process is omitted, a team can report lower inference spending while overlooking the cost of failed tasks.</p> <p>RAPID’s operational and monitoring stage is where a system proves itself against real traffic and business outcomes. That means confirming the quality signal is meaningful, cost limits behave as intended, and intervention works under realistic conditions, not merely validating that an agent can complete a demo.</p> <h2>The cost of waiting is an unmeasured production risk</h2> <p>The gap between broad observability and limited live evaluation creates a predictable exposure: an organization can know an agent is running without knowing whether it remains effective. The longer a silent failure goes undetected, the more customer interactions, manual corrections, and model spend may accumulate.</p> <p>A disciplined first step is to inventory production agents and assign an owner to each. Then define a quality threshold, spending ceiling, and response path for the highest-impact use cases. Expand live evaluation in stages, using production-appropriate privacy and approval controls. Track whether quality breaches are detected and contained, not just how many dashboards are deployed.</p> <p>Salesforce’s announcement is part of a broader shift. On October 27, Oracle also announced AI Data Platform capabilities involving AI-assisted business ontologies, a Semantic Reasoning Engine, Deep Insight Agents, and an AI Registry for agents, tools, and MCP servers. <strong>Oracle said these new capabilities are planned within the next 12 months</strong>; they are not generally available today. The direction is consistent: enterprise agent operations will need both business context and ongoing control.</p> <h2>Make agent reliability an operating decision</h2> <p>The tools are beginning to support continuous evaluation, budget enforcement, and intervention. The discipline must come from the business: set thresholds before deployment, monitor real outcomes, and make re-entry contingent on passing evaluation.</p> <p>Mahlum Innovations helps organizations map AI to business goals and implement systems from strategy through production. Our <a href="https://mahluminnovations.com/ai-strategy-consulting/">AI strategy consulting</a> connects use cases to measurable objectives; our company-stated average of <strong>3.5x ROI on client projects is not a guarantee</strong>. Teams ready to operationalize a use case can explore <a href="https://mahluminnovations.com/ai-implementation-services/">AI implementation services</a>, <a href="https://mahluminnovations.com/ai-automation-consulting/">AI automation consulting</a>, and our <a href="https://mahluminnovations.com/services">services overview</a>.</p> <p>The executive decision is straightforward: if an agent is important enough to stay in production, it is important enough to have an SLO. Define its quality bar, cap its spend, specify how it is stopped, and require proof before it returns. That is how agent quality becomes an operations metric, not a post-incident discovery.</p>