Agentic Systems: Abstraction Layers

You do not need a complex gateway or runtime to start building. This guide breaks down the five layers of LLM infrastructure so you can choose the simplest set for your application. Start with a direct model call, and add abstractions only to solve recurring problems.

The previous post, Autonomy Spectrum, asked how much authority a model needs and established our first principle:

Principle #1: Minimum sufficient autonomy

Give the model only the authority it needs to achieve the desired outcome.

Autonomy and abstraction answer different questions. Autonomy determines what the model can decide and do. Abstraction determines how much of the surrounding software your application controls.

Once you choose the model’s authority, decide which engineering responsibilities the application should keep and which an abstraction should handle. That decision gives us the second principle:

Principle #2: Minimum sufficient abstractions

Add a layer only when it solves a recurring engineering problem.

How the abstraction layers work

Start with a direct model call

Imagine a command-line tool that sends an incident report to a model and prints a summary. The application formats an HTTP request, sends it to one model provider, parses the response, and handles request errors.

CLI → Model provider

A model provider hosts or loads a model, runs inference , and exposes an interface that the application can call. OpenAI, Anthropic, Google, hosts of open models, and local inference runtimes all fill this role.

The provider determines which capabilities the application can rely on. The incident application may need structured output for predictable fields or an image modality to read screenshots. To query an incident system, the application can give the model a controlled capability called a tool . The provider must support tool calling so the model can request that capability. Long reports need a suitable context size . The user experience depends on latency , while sensitive data may require a specific deployment boundary .

The application controls the prompt, request code, state, error handling, and evaluation. If this direct call meets the product’s needs, stop here. Each layer that follows should remove a new source of recurring complexity.

Give the model tools when it needs to act

A model generates responses from the information in its context. The model needs tools to retrieve current information or act on another system. A tool exposes one application-defined capability, such as getIncident, searchLogs, or postStatusUpdate.

The application sends the model a definition for each available tool: its name, description, and the structure of its arguments. The model sees this interface; the underlying function code stays in the application. When the model selects a tool, it returns a structured request instead of a final response:

{
  "name": "getIncident",
  "arguments": {
    "incidentId": "INC-2048"
  }
}

The application matches the name to an allowed function, validates the arguments and permissions, runs the function, and sends the result back to the model. The model then uses the result to answer or request another tool.

Application sends prompt and tool definitions
                 ↓
Model requests a tool with structured arguments
                 ↓
Application validates and executes the tool
                 ↓
Application returns the result to the model
                 ↓
Model answers or requests another tool

Application developers implement the function and describe its interface. Production tools also need input validation, permission checks, timeouts, and error handling. This boundary keeps authority in application code even when the model chooses which available tool to request.

Add a model SDK when calls become repetitive

The incident tool now streams partial responses, returns structured fields, and asks the application to run tools. Raw provider requests force the application to parse stream events, validate tool-call arguments, and translate provider-specific errors.

A model SDK packages that protocol work inside the application. The SDK standardizes how application code sends messages, streams responses , produces structured output, and handles tool calls.

Application [Model SDK] → Model provider

The SDK handles request and response conventions. The application retains prompts, model selection, context strategy, state, the agent loop, and product behavior.

SDK consistency ends at the code boundary. Models interpret prompts differently, select different tools, and produce different results. A model change requires evaluation.

Vercel AI SDK and Pi AI (@earendil-works/pi-ai) are examples of Model SDK and they provide two different API designs.

Vercel AI SDK accepts prompts or messages and returns generation results. Its result-oriented design fits applications that generate, stream, and render responses. Pi AI passes a serializable context through a streamed model interaction. Its context-oriented design fits work that spans several calls or processes.

Add an AI gateway when policy spans applications

If we now extend the command-line tool to have a web interface and scheduled batch jobs, each of these applications carries provider credentials, budgets, logs, and fallback rules. A key rotation or routing change now requires several deployments.

An AI gateway moves those shared policies onto the network request path. The gateway receives model requests, applies routing and budget rules, and forwards each request to a provider.

Applications [Model SDK] → AI gateway → Model provider

An SDK simplifies model calls inside one application. A gateway governs model access across applications. The two layers work together: the SDK sends a request through the gateway, and the gateway selects a provider.

The gateway centralizes credentials, usage logs, budgets, rate limits, retries, and fallback rules. The application sets data policy, routing intent, and acceptable fallback behavior.

Fallback routing preserves availability. A different model may select different tools, accept less context, expose different reasoning controls, or apply different safety behavior. Routing policy therefore affects product correctness as well as operations. Switching hosting providers for the same model (e.g., from Anthropic’s API to AWS Bedrock for Claude) carries less behavioral risk than switching model families (e.g., from Claude to GPT).

OpenRouter orders and filters hosting providers, then falls back when an endpoint is unavailable. Vercel AI Gateway provides a unified endpoint with usage monitoring, budgets, routing, and model fallbacks. Cloudflare AI Gateway adds controls such as caching, rate limits, retries, and dynamic routing.

Add an agent runtime when work outlives a request

To continue with the analogy, let’s extend the incident tool to do long-running investigation and pauses for approval. A deployment in the middle of such an investigation ends the server process and erases the task’s in-memory progress.

Two main control-flow patterns frame this choice, and many systems combine them. The core question is who chooses the next step.

When the sequence is known in advance, use a workflow . The application defines the overall path and stopping point. Individual steps may call a model and produce variable output, but application code decides which step runs next. Workflows fit processes you can map in advance, such as ingesting a document, analyzing it, requesting approval, and publishing the result.

When the path depends on what the system discovers, use agentic mode . The application gives the model a goal, a set of tools, and operating boundaries. On each turn, the model examines the current context, chooses a tool or response, receives the result, and decides what to do next. The path emerges as the work progresses, so two runs can take different routes to the same goal.

Agentic mode gives the model control over the sequence, while the application controls the available tools, permissions, budgets, approval points, and stopping conditions. Developers often describe workflow orchestration as deterministic and agentic orchestration as non-deterministic. These labels describe who chooses the sequence; model calls inside a workflow can still produce variable results.

An agent runtime runs the model-and-tool loop and manages its lifecycle. A loop library may coordinate only the current model and tool turns. A durable runtime also stores persistent state , supports recovery after interruptions, pauses for approvals, and records observability data.

Agent runtime [Model SDK] → AI gateway → Model provider

A durable runtime records enough progress to resume or safely retry work. The application retains the agent’s instructions, authority, approval rules, tool design, evaluation, and any capabilities the runtime leaves open.

Full agent frameworks often bundle tools, sandboxes , persistence, channels , credentials, and deployment around a runtime. Framework defaults save engineering work and make architectural decisions. Evaluate whether those defaults fit the product and whether the framework provides a supported way to replace a component or reach the underlying service when the abstraction falls short. This direct route is an escape hatch .

Pi Agent Core (@earendil-works/pi-agent-core) adds a stateful agent loop, tool execution, context transformation, and lifecycle events above Pi AI.

Vercel Eve combines Vercel services for model access, durable workflows, sandboxes, authentication, channels, schedules, and subagents. Flue provides durable agents and finite workflows across Node.js, Cloudflare Workers, CI systems, and other environments.

Eve makes more infrastructure choices through an integrated Vercel stack. Flue leaves more choices about persistence, sandboxes, credentials, and deployment with the team.

Add a channel abstraction when interactions multiply

Let’s extend the incident management tool to support Slack, Teams, GitHub, and web as interfaces through which users can interact with the tool. Each platform supplies different event payloads, signatures, identities, thread identifiers, streaming behavior, and interactive components.

A channel abstraction translates those platform-specific interactions into common events, threads, messages, and replies. Platform adapters then translate application responses back into each platform’s API.

Slack / Teams / GitHub / Web
              ↓
Channel → Agent runtime [Model SDK] → AI gateway → Model provider

Channel adapters also determine which external events can start work. A mention, issue, webhook , reaction, or schedule can trigger an agent loop. These triggers fall into three groups:

• User-triggered: A user explicitly starts the interaction.
• Event-triggered: A change in the environment starts the interaction.
• Policy-driven: The runtime decides whether and how the system responds.

Every channel creates a trust boundary. Incoming messages, attachments, linked documents, webhook payloads, and retrieved history supply untrusted context. The application owns identity checks, permission rules, input limits, conversation design, and the authority granted to each trigger.

Platform differences limit portability. Slack, Teams, Discord, and WhatsApp expose different capabilities. Some stream natively, while others edit an existing message. Some support threads, modals, or rich cards that others lack.

A shared interface usually covers the common subset. For a platform-specific feature, such as a Slack modal, a strong channel abstraction also exposes the underlying platform API. This escape hatch lets the application use the native feature without abandoning the shared interface.

Vercel Chat SDK provides common interfaces for channels, threads, messages, mentions, reactions, direct messages, subscriptions, cards, and modals. Platform adapters handle webhook verification, event parsing, formatting, and API calls. State adapters persist subscriptions and coordinate distributed processing.

Chat SDK also converts channel history into AI SDK messages and exposes channel actions as tools.

How the complete system fits together

The article introduced the layers in a likely adoption order: provider, SDK, gateway, runtime, then channel. A request through the complete system travels in the opposite direction, from the interaction surface toward model execution:

Responses travel back through the same path. Every system needs a way to execute a model. The other layers remain optional, and a framework may bundle several of them into one product.

Choose the smallest useful set

Choose each layer from the problem it solves, rather than from the complete architecture diagram.

Configure control flow inside the runtime

After choosing a runtime, choose how the application will control work:

Three rules that travel across layers

Abstractions transfer responsibility

An abstraction takes over part of the system and applies defaults on the application’s behalf. The application writes less code and inherits new limits and failure modes. Check whether the abstraction’s decisions fit the product.

Durability requires stored progress

An in-memory loop works while a task begins and ends within one request. Approvals, deployments, later events, and process failures force the system to store progress. Stored progress often earns a durable runtime its place.

Every boundary needs visibility and an escape hatch

Each SDK, gateway, runtime, and channel hides part of the system. A clear boundary lets the team inspect what crosses it, trace failures through it, and reach the underlying provider or platform when the abstraction falls short.

Conclusion

Start with the product outcome. Give the model enough authority to achieve it, then add abstractions for recurring engineering problems. Keep the responsibilities where your judgment shapes the product. Reuse infrastructure where a standard implementation fits.

This is the second of four posts on building agentic systems. Read the others: Autonomy Spectrum, Ownership Boundaries, and Learning Loop.