Agentic Systems: Learning Loop

An agentic system creates value by completing business work reliably at an acceptable cost. Once the system runs in production, the domain application sees evidence that a model provider cannot: whether the work succeeded, what people corrected, and what the complete attempt cost. That evidence can drive a learning loop that improves quality and economics as models change.

In Ownership Boundaries, we described how the domain application, process orchestration, agent harness, and model inference connect agents to a real business. That gave us our third principle:

Principle #3: Business-owned orchestration

Let the model handle variation. Keep business invariants in deterministic code.

We call this feedback cycle the learning loop. Vertical SaaS provides a useful recurring example because each business process exposes both quality and inference cost.

This gives us our fourth principle:

Principle #4: Own the learning loop

Treat the model as replaceable. Own the evals, traces, and feedback mechanisms that improve the system for your domain.

Cost per successful outcome

Traditional SaaS often has low marginal delivery costs because the same application code serves many customers, even though infrastructure and support costs still grow with usage. Agentic SaaS adds a potentially material variable cost: inference. Every model call consumes paid compute. A vertical SaaS company can deliver valuable automation while depending on expensive foundation models for every interaction, so more usage produces more revenue and more inference cost at the same time.

Cost per token measures model efficiency. Cost per successful outcome measures the economics of the complete business process. A smaller model can be inexpensive per request and still produce poor economics if it causes more retries, failures, or human review. A larger model may cost more per call while completing difficult work correctly on the first attempt.

Measure cost per successful business outcome.

The numerator includes the variable cost of every attempt, including inference, tool execution, retries, and human review. The denominator counts outcomes that meet the domain’s success criteria within the expected time. Failed attempts remain in the cost total.

The goal is to find the least expensive path that reliably reaches the required business outcome.

Connecting traces to business outcomes

The domain application sees information a generic model provider does not have. It sees which tasks completed successfully, which recommendations a human approved or rejected, which fields an operator corrected, which policies caused an escalation, which tool calls failed, how long the process took, how much it cost, and whether the final business outcome was correct.

Production traces become valuable once the application connects them to these outcomes. A collection of prompts and responses is hard to learn from. A trace labeled claim_approved_correctly, human_revision_required, or policy_violation carries the signal. This instrumentation belongs in the domain application, because that is where success has business meaning, and it should be part of normal operation rather than a separate analytics project.

These traces and corrections supply candidate cases for a private eval: a domain-specific scorecard for the product’s own work. Each case needs a validated expected outcome and scoring criteria before it can measure a system change.

Concrete example: Dental intake

A new patient submits an intake request. The agent classifies it, proposes an appointment, and drafts a confirmation message. A staff member corrects the classification, changes the appointment, and edits the message.

The application records the original output, the corrections, the final approved result, and the cost. After the team validates why the corrections were necessary, the reviewed case can enter an eval set with clear success criteria: correct classification, appropriate scheduling, required information, and acceptable cost.

Some cases help the team improve the policy context, prompt, workflow, routing, or model. A separate held-out set tests whether the change also improves cases that the team did not tune against. As representative cases accumulate, eval runs show whether the system is becoming more accurate and less expensive.

Private evals are the product’s scorecard. A team can change a prompt, a workflow, a retrieval strategy, a tool, or a model, then measure whether the whole process improved against the current baseline. Evals need to include both quality and economics. Optimizing only for accuracy can produce a system that is too expensive to operate, while optimizing only for token cost can push more work onto human reviewers.

Improve the system before training a model

Fine-tuning is one way to improve the system. It is rarely the first available improvement.

A practical sequence:

1. Remove unnecessary model calls.
2. Improve the workflow and tool boundaries.
3. Improve prompts, context, and retrieval.
4. Cache repeated context or results where appropriate.
5. Route routine and difficult cases to different models.
6. Fine-tune or distill a smaller model for stable, repeated tasks.

Development evals show where the system fails and guide each change. Held-out evals test whether the candidate improves without overfitting to the cases used during development.

Fine-tuning becomes useful once a task repeats often, the desired behavior is stable, and the company has enough high-quality examples. A smaller domain-specific model can then handle routine cases at lower cost, while a frontier model handles the ambiguous ones. The resulting system is a model portfolio rather than a single-model strategy: a small specialized model for common classifications and drafts, a frontier model for high-value or uncertain cases, a human reviewer for risky decisions, and an eval-driven router that picks the path.

The learning environment becomes the moat

At Microsoft Build 2026, Mustafa Suleyman argued that companies should retain the benefits of their workflows, know-how, institutional data, and resulting models. In Microsoft’s frontier-tuning approach, the reinforcement-learning environments and models “become your moat” (keynote transcript, p. 37).

Satya Nadella later connected private evals, outcomes, reinforcement-learning environments, traces, and enterprise knowledge to differentiated intellectual property that the company owns and controls (keynote transcript, pp. 40-41).

The product framing is specific to Microsoft’s frontier tuning. The broader strategic point is that a business should retain the improvements produced by its workflows, evals, traces, and data. These assets preserve the business’s evidence and domain knowledge when the underlying model changes, even though prompts, adapters, routing, or tuned weights may require rework.

The model is replaceable. The learning environment compounds.

Domain learning and tenant learning

A vertical SaaS company can learn at two levels.

Domain learning captures patterns shared across the vertical: terminology, document structures, common workflows, and standard decision criteria.

Tenant learning captures one customer’s policies, tone, exceptions, and historical decisions.

Tenant-specific retrieval and configuration are often simpler than maintaining a separate model for every customer. Tenant-specific fine-tuning becomes more attractive when a customer has enough examples and usage to justify the additional training, deployment, evaluation, and governance work.

Data rights are part of the architecture. A SaaS provider needs clear permission to use customer interactions for training or cross-tenant improvement. Private data from one tenant should not silently become training data for another.

The compounding loop

The complete loop looks like this:

Run domain processes → capture outcomes and corrections → build private evals → identify weak or expensive steps → improve workflows, routing, or models → measure again

This loop creates two forms of advantage.

Product advantage

The system learns the domain’s terminology, policies, exceptions, and desired outcomes. It requires less correction and becomes more useful than a generic agent connected to the same foundation model.

Economic advantage

Routine work moves to cheaper paths. Unnecessary calls disappear. Context becomes smaller. Human review focuses on cases where it adds value. The cost per successful outcome declines.

Fine-tuned model weights can lose relative value as better base models become available. The durable assets are the private evals, outcome-labeled traces, workflow knowledge, and feedback mechanisms used to improve any model.

Conclusion

The four posts describe one progression: choose the minimum autonomy required, choose the minimum abstraction that solves the engineering problem, keep domain policy and business state above the agent runtime, and treat the resulting system as a learning loop rather than a finished product.

The same principle applies to any business adopting AI: own the evidence and feedback mechanisms that improve domain decisions. For a vertical agentic SaaS company, the model is a rented capability. The moat is the private evals, the outcome-labeled traces, and the workflow knowledge built by running real business processes and correcting what the model gets wrong. Own that loop, and the underlying model can change without the business losing what it has learned.

A practical starting point is one bounded process. Define success and business invariants, record outcomes, corrections, and cost, curate development and held-out eval cases, then compare one system change against the current baseline.

This is the last of four posts on building agentic systems. Read the others: Autonomy Spectrum, Abstraction Layers, and Ownership Boundaries.