Kategorie: Cloud Native

Veröffentlicht am

Pods Are Workers, Not Agents: Designing the Runtime Boundary for Enterprise Agent Platforms

Kubernetes Pods are excellent execution units. They provide scheduling, resource controls, networking, workload identity integration, and a natural boundary for security and observability.

That does not automatically make a Pod the right representation of an AI agent.

Enterprise agent platforms need to distinguish two concepts that are easy to collapse during early implementations: the logical agent and the runtime worker executing its current task. Treating them as the same object can work for prototypes and continuously running agents. At scale, it creates idle infrastructure, slow burst handling, fragmented identity, and weak lifecycle semantics.

The durable pattern is to let Kubernetes manage execution workers while an agent control plane manages agent identity, state, policy, placement, and lifecycle. Pods remain essential. They become workers rather than the agent itself.

Why one Pod per agent is an attractive first design

The one-agent-per-Pod model solves several real problems quickly.

  • A Pod provides a process and container isolation boundary.
  • A ServiceAccount gives the workload a Kubernetes identity.
  • NetworkPolicy and admission policy can constrain its environment.
  • CPU and memory requests make resource consumption schedulable.
  • Logs, metrics, and traces can be attributed to a workload instance.
  • Existing GitOps, deployment, and incident-response practices remain usable.

For a small number of high-value agents, those benefits may outweigh the overhead. The model is understandable and conservative. It uses boundaries that platform and security teams already know how to operate.

The problem appears when the organization assumes that the execution container is also the durable identity and lifecycle of the agent.

Agents do not behave like ordinary services

A typical service is expected to remain available and handle a continuing stream of requests. An agent may wake up for a task, run for seconds or minutes, wait for a human decision, delegate work to subagents, and then remain idle for hours.

These characteristics create a different workload shape:

  • Bursty demand: a single business event can fan out into many parallel agent tasks.
  • Long idle periods: logical agents may exist without needing compute.
  • External waiting: execution may pause for approval, data, or another system.
  • Variable duration: tasks range from short tool calls to extended research or coding sessions.
  • Delegated authority: an agent often acts on behalf of a user or workflow rather than only as itself.
  • Stateful continuation: a later execution may need to resume the same logical conversation or plan on a different worker.

Keeping one Pod alive for every logical agent reserves capacity for identities that are not doing work. Creating a fresh Pod for every short task can introduce startup latency and control-plane churn. Encoding state inside the Pod makes rescheduling and recovery harder.

The architectural question is therefore not whether Kubernetes should run agents. It is which responsibilities belong to Kubernetes and which belong to an agent-specific control plane.

The runtime boundary: agents, actors, and workers

A recent CNCF article describing kagent’s agent-substrate architecture illustrates this separation. Kubernetes continues to manage Pods, networking, storage, and compute. A higher-level control plane manages logical actors and places them onto a pool of execution workers.

In that model:

  • The logical agent has durable identity, ownership, policy, configuration, and state.
  • An agent task or actor instance represents a unit of active execution.
  • A worker is a sandboxed runtime capable of executing one or more assigned actors.
  • A worker pool defines capacity, runtime profile, isolation class, and placement characteristics.

Agent-substrate is one implementation, not a universal enterprise standard. Its value for platform design is the principle it demonstrates: logical lifecycle can be decoupled from Pod lifecycle without removing Kubernetes from the architecture.

Six contracts the control plane must preserve

Decoupling an agent from a Pod improves efficiency only if the platform preserves the controls that dedicated Pods made easy.

1. Durable agent identity

An agent needs an identity that survives worker replacement. That identity should identify the agent definition, tenant, owner, environment, risk tier, and approved capabilities.

The worker also needs its own workload identity. The two must not be confused. A worker identity proves which runtime is communicating with the platform. The agent identity determines which business permissions and policies apply to the assigned execution.

When an agent acts for a person, the authorization decision should include delegated user context with explicit scope and expiry. Copying a user’s full credentials into a worker is not delegation.

2. Execution leases

Placement should create a time-bound execution lease binding an agent task to a specific worker. The lease should include the agent identity, policy revision, tool permissions, state reference, deadline, and expected resource profile.

Leases make reassignment and failure handling explicit. If a worker disappears, the control plane can determine whether the task is safe to retry, must resume from a checkpoint, or requires human review.

3. Isolation classes

Sharing workers does not mean sharing trust. The platform needs multiple runtime profiles based on risk.

  • Low-risk, read-only tasks may use a warm multi-tenant worker pool.
  • Tasks handling confidential data may require stronger sandboxing and tenant-dedicated workers.
  • Agents with write access to production systems may require a dedicated Pod or ephemeral sandbox per execution.
  • Untrusted code execution may require gVisor, microVMs, or another hardened isolation boundary.

The scheduling decision should derive from policy. Developers should request a workload class rather than select a weaker runtime to reduce latency.

4. Policy attribution

Kubernetes policy usually sees the Pod, namespace, and ServiceAccount. A shared worker introduces another logical principal inside that boundary. The platform must propagate agent, tenant, task, and delegated-user context to every policy enforcement point.

Tool gateways, model gateways, data APIs, and egress proxies should authorize the logical execution, not merely trust the worker’s network location. Audit events should record both worker identity and agent identity so investigators can reconstruct who did what and where it ran.

5. Externalized state and checkpoints

Agent state should not depend on the continued existence of a worker Pod. Conversation state, plans, artifacts, approval state, and checkpoints need durable storage with tenant-aware encryption and retention controls.

Externalizing state allows the platform to release compute while an agent is idle and rehydrate it when work resumes. It also creates a controlled recovery point instead of treating the worker filesystem as an accidental system of record.

6. End-to-end observability

Pod-level telemetry remains necessary but is no longer sufficient. Operators need to follow a logical agent across workers and over time.

Every execution should carry stable correlation fields such as:

  • agent, tenant, task, session, and parent-task identifiers;
  • worker and worker-pool identity;
  • policy, prompt, model, and tool versions;
  • delegated user and approval references where permitted;
  • token, latency, tool-call, cost, and outcome signals;
  • checkpoint, retry, reassignment, and termination reasons.

This creates observability for the business execution rather than only for the container currently hosting it.

A reference enterprise architecture

A practical runtime separates responsibilities across four layers.

Agent control plane

The control plane stores agent definitions, ownership, policy, lifecycle, state references, and desired runtime class. It accepts tasks, decides placement, issues leases, tracks execution, and coordinates retries or resumptions.

Worker pools

Kubernetes Deployments or other controllers maintain warm capacity for defined execution profiles. Pools may differ by tenant, geography, accelerator, sandbox technology, network access, or data classification.

Shared platform gateways

Model, tool, MCP, data, and egress gateways enforce logical identity and policy. They keep privileged credentials out of agent code and provide consistent rate limits, approval checks, observability, and revocation.

Durable state and evidence

State services store checkpoints and artifacts. An evidence plane records immutable links between the agent definition, execution lease, policy decision, worker, model interaction, tool call, and outcome.

Kubernetes remains the infrastructure substrate. The agent control plane provides semantics Kubernetes was not designed to infer.

Multi-tenancy must shape worker placement

Worker utilization can improve dramatically when idle logical agents do not retain Pods. That benefit should not override tenant boundaries.

Platform teams should define placement rules covering:

  • whether tenants may share a worker process, Pod, node, or cluster;
  • which data classifications require dedicated runtime capacity;
  • how memory, filesystems, caches, and credentials are cleared between assignments;
  • whether agent-generated code can execute and under which sandbox;
  • which tools and destinations each pool can reach;
  • how noisy-neighbor behavior is detected and constrained;
  • where state and inference traffic may be processed geographically.

There is no single correct sharing boundary. The platform should offer a small set of reviewed isolation classes and make the selected class visible in cost, latency, and risk reporting.

When one Pod per agent is still the right answer

Decoupling should not become an objective by itself. A dedicated Pod remains a strong choice when:

  • the agent is continuously active or exposes a stable service endpoint;
  • startup latency is acceptable and the fleet is small;
  • the workload needs strong tenant or process isolation;
  • it runs untrusted code or privileged tools;
  • its memory and resource profile do not fit a shared pool;
  • existing Kubernetes controls provide sufficient lifecycle semantics;
  • the added agent scheduler would cost more to operate than it saves.

The mature platform supports more than one runtime pattern. It chooses the boundary based on workload behavior and risk rather than forcing every agent into the same optimization.

Measure the runtime as a platform product

Worker density is useful, but cost efficiency alone is an incomplete success measure. Track flow, reliability, isolation, and control together.

  • Task queue time and time to first execution
  • Warm-start and cold-start latency
  • Active versus idle worker utilization
  • Logical agents per worker and per isolation class
  • Checkpoint, resume, retry, and reassignment success rates
  • Policy denials and unauthorized cross-tenant attempts
  • State cleanup and credential revocation failures
  • Cost per successful agent task
  • Trace and audit coverage from task request to external side effect

A cheaper runtime that cannot explain an agent’s actions is not an enterprise improvement.

A staged adoption path

1. Separate identifiers before changing runtime

Introduce stable agent, task, tenant, and worker identifiers in the current platform. Propagate them through logs, traces, policy decisions, and tool calls. This exposes hidden coupling before a scheduler is introduced.

2. Externalize state

Move durable state and artifacts out of the Pod. Define checkpoint, retry, expiry, encryption, and deletion semantics. Test recovery from worker termination.

3. Add one low-risk worker pool

Select bursty, read-only tasks with clear resource limits. Compare queue time, utilization, cost, and operational effort with the dedicated-Pod baseline.

4. Add policy-aware placement

Introduce reviewed isolation classes and execution leases. Integrate logical identity with tool, model, data, and egress gateways. Exercise tenant separation and credential revocation.

5. Expand only with evidence

Move higher-risk agents after proving state hygiene, observability, rollback, and incident response. Keep dedicated Pods as an explicit option rather than treating them as a failed legacy design.

Pods should host work, not define the agent

The Pod remains one of the strongest execution boundaries available to cloud-native platforms. The mistake is asking it to carry semantics it does not own: durable agent identity, delegated authority, conversation lifecycle, human approval, and cross-execution state.

Enterprise agent platforms should model those concerns explicitly. Kubernetes can then do what it does best — schedule and isolate execution — while the agent control plane decides which logical work runs where, under whose authority, with which policy, and with what evidence.

That separation improves utilization, but its greater value is governance. It allows the platform to scale agents without losing the identity and accountability that production systems require.

Sources

Veröffentlicht am

The AI-Native Platform Contract: Expanding Golden Paths Beyond Application Delivery

Platform engineering earned its place by turning application delivery into a repeatable product. Golden paths combined infrastructure, security, deployment, and operational standards into a paved route that developers could use without learning every platform detail.

AI-native workloads do not invalidate that model. They expose where it stops too early.

A conventional golden path typically starts with source code and ends with a running service. An AI-native product depends on a wider chain: governed data, accelerator capacity, models and prompts, evaluation evidence, inference controls, agent identities, external tools, and continuous cost and risk feedback. If each of those capabilities arrives through a separate specialist portal, the organization has not created an AI platform. It has created another integration problem.

The next platform contract should therefore extend the golden path rather than build a parallel AI silo. The goal is not to hide every AI decision behind automation. It is to make safe defaults easy, exceptions explicit, and every promoted artifact traceable.

The application delivery contract is no longer enough

Platform Engineering 1.0 concentrated on a familiar delivery unit: an application packaged as a container, deployed through a pipeline, and operated with standard observability and security controls. That remains valuable, but AI changes both the workload and its consumers.

ML engineers need experiment tracking, model registries, feature and data access, and specialized compute. Application teams need stable inference endpoints and predictable latency. Security teams need controls for model provenance, prompt injection, data leakage, and non-human identities. FinOps teams need to attribute expensive training and inference usage. AI agents themselves become platform consumers that request tools, credentials, and runtime actions.

The CNCF discussion of evolving platform engineering for AI-native workloads captures this expansion through capabilities such as GPU and TPU allocation, model serving, MCP gateways, agentic guardrails, embedded FinOps, and policy-driven governance. The important organizational point is that these should not become an isolated platform owned by a small AI team. They should become extensions of the same product model, interfaces, and control philosophy used by the enterprise platform.

Define a platform contract, not a catalog of tools

A platform contract describes what a product team can request, what evidence it must provide, what the platform guarantees, and which controls are automatically applied. It is stronger than a service catalog entry and more flexible than a single mandatory implementation.

For an AI-native workload, that contract should cover at least six dimensions.

1. Governed data access

The path should make data classification, residency, retention, and permitted use visible before a workload reaches production. A request for a dataset should resolve to an approved identity, purpose, environment, and audit trail. The platform can automate access, but the product team remains accountable for whether the data is appropriate for the use case.

2. Compute and accelerator intent

Teams should request capabilities rather than hard-code a particular GPU model into every manifest. The contract can express workload class, memory, performance objective, duration, geographic constraints, and cost ceiling. Kubernetes mechanisms such as Dynamic Resource Allocation can support more structured resource claims, but the platform still needs policy for quotas, scarcity, preemption, and approved hardware profiles.

3. Model, prompt, and artifact provenance

Container images are not the only production artifacts. The platform must track model version, source, license, evaluation result, prompt bundle, retrieval configuration, tool definitions, and deployment policy. Promotion should be based on an immutable set of linked artifacts, not a model name copied into an environment variable.

4. Evaluation as a release gate

AI quality is probabilistic and context-dependent. A successful build does not prove production fitness. Golden paths should provide standard evaluation suites for task quality, safety, latency, robustness, and cost. Teams can add domain-specific tests, while the platform supplies the execution environment, evidence format, thresholds, and promotion workflow.

5. Runtime identity and guardrails

An inference service or autonomous agent needs a workload identity, scoped data access, approved tools, network boundaries, and observable policy decisions. The contract should distinguish a human user’s authority from an agent’s delegated authority. It should also define what happens when a model, tool, or policy is unavailable rather than allowing silent fallback to an uncontrolled path.

6. Cost and operational accountability

AI infrastructure introduces different cost behavior from ordinary stateless services. Training jobs can consume scarce capacity in bursts. Inference cost depends on model choice, token volume, batching, cache efficiency, and service-level objectives. Cost attribution and budgets should therefore be part of provisioning and release decisions, not a dashboard reviewed after the invoice arrives.

What an AI-native golden path looks like

A useful golden path follows the product lifecycle rather than exposing a collection of disconnected infrastructure forms.

  1. Declare the workload. The team selects an archetype such as batch training, online inference, retrieval-augmented generation, or tool-using agent. It declares data class, expected scale, latency objective, risk tier, and ownership.
  2. Provision an isolated workspace. The platform creates namespaces, identities, network boundaries, secrets references, storage, accelerator claims, quotas, and standard telemetry.
  3. Develop with approved building blocks. Teams consume versioned model endpoints, registries, feature services, MCP or tool gateways, and evaluation templates through stable APIs.
  4. Produce evidence. CI records model and data lineage, software dependencies, evaluation results, policy decisions, security findings, and predicted operating cost.
  5. Promote as a release set. GitOps promotes the linked application, model, prompt, policy, and tool configuration together. A rollback restores the complete known-good set.
  6. Operate with continuous feedback. Runtime telemetry covers service health, model quality indicators, policy denials, data drift, tool calls, accelerator utilization, and unit economics.

This lifecycle gives specialists room to innovate without forcing every product team to assemble the control plane themselves.

Avoid the separate AI platform trap

A dedicated AI enablement team may be necessary, but a separate delivery system should not be the default. Parallel identity models, pipelines, policy engines, and observability stacks increase cost and weaken governance. They also create a handoff between application engineers and AI specialists exactly where the product needs shared accountability.

A better operating model separates platform ownership by capability while preserving one product contract:

  • The core platform team owns common interfaces, workload identity, delivery workflows, policy integration, and the developer experience.
  • The AI platform capability team owns model-serving patterns, evaluation services, accelerator profiles, registries, and AI-specific runtime controls.
  • Data teams own governed data products and access semantics.
  • Security and risk teams define control objectives and approval boundaries as policy and evidence requirements.
  • Product teams own business fitness, domain evaluations, production outcomes, and accepted residual risk.

The teams collaborate through APIs, schemas, policy bundles, and service-level objectives rather than tickets and undocumented exceptions.

Measure whether the contract creates value

An AI-native platform should not be measured by the number of services in its catalog. Measure whether teams can deliver trustworthy outcomes faster.

  • Time from approved use case to first governed experiment
  • Time from candidate model to production release
  • Percentage of releases with complete model, data, prompt, and policy provenance
  • Evaluation failure escape rate
  • Percentage of agent tool calls using approved identities and gateways
  • Accelerator utilization and queue time by workload class
  • Inference cost per business transaction
  • Rollback time for a complete AI release set
  • Adoption and exception rates for each golden path

These metrics reveal whether the platform improves flow and control together. High adoption with slow delivery signals an overloaded path. Fast delivery with weak evidence signals unmanaged risk.

A practical 90-day starting point

Do not begin by designing a universal AI platform. Choose one real workload and use it to define the minimum viable contract.

Days 1–30: map the lifecycle

  • Select one representative AI product with a committed owner.
  • Map every artifact, identity, environment, approval, and operational dependency.
  • Classify which existing platform capabilities can be reused and where AI-specific gaps exist.
  • Define the workload’s risk tier, evaluation evidence, and cost objectives.

Days 31–60: build one vertical path

  • Create one workload template and governed workspace.
  • Connect model and prompt provenance to the existing GitOps release flow.
  • Add standard telemetry, policy checks, evaluation execution, and cost labels.
  • Document escape hatches with owners, expiry dates, and review requirements.

Days 61–90: prove and productize

  • Run a production-like release and rollback.
  • Measure lead time, evidence completeness, operational quality, and unit cost.
  • Interview the platform consumers and remove unnecessary steps.
  • Publish the contract as versioned schemas, APIs, examples, and service-level expectations.

The platform becomes the organizational control surface

AI-native platform engineering is not a race to add GPUs and model registries to an internal portal. It is the work of extending a proven product contract across a more complex value stream.

The strongest platforms will preserve what already works: product thinking, self-service, golden paths, policy automation, and composable cloud-native interfaces. They will add the missing contracts for data, models, evaluations, agents, specialized compute, and cost. That approach avoids a new silo while giving teams a credible path from experimentation to governed production.

Sources

Veröffentlicht am

OpenTelemetry Fleet Management: Why OpAMP Belongs in the Enterprise Observability Control Plane

OpenTelemetry can standardize how an enterprise collects and exports telemetry, but standardization alone does not make the collection layer operable.

At small scale, teams can manage Collector configuration through deployment manifests, virtual machine tooling, or a handful of automation scripts. At enterprise scale, the fleet becomes heterogeneous: Kubernetes DaemonSets, centralized gateways, virtual machines, laptops, point-of-sale devices, edge systems, and embedded environments. Different teams deploy the agents, while a central observability group remains accountable for data quality and service reliability.

That creates a control-plane problem. The organization needs to know which agents exist, what they are running, whether their configuration is current, whether a rollout succeeded, and how to recover without losing telemetry. The Open Agent Management Protocol, or OpAMP, provides a vendor-neutral protocol for that management relationship.

The strategic point is bigger than remote configuration. OpAMP belongs in the enterprise observability control plane because telemetry collection is production infrastructure. It needs identity, desired state, health feedback, controlled rollout, auditability, and rollback just like any other critical fleet.

Telemetry standardization exposes the management gap

OpenTelemetry adoption often begins with a sensible objective: remove proprietary instrumentation and normalize traces, metrics, and logs around open standards. The Collector becomes a flexible processing and export layer between workloads and one or more observability backends.

Success creates a new operating challenge. Collector configurations diverge by environment and team. Components run different versions. Credentials rotate at different times. Pipelines fail silently or begin dropping data. A change that looks safe in a development cluster can overload a regional gateway or remove a critical security log source.

GitOps helps with Kubernetes-managed Collectors, but it does not automatically cover agents on virtual machines, workstations, edge locations, or devices. It also tells the platform what was declared, not necessarily what every agent loaded or whether the resulting pipeline is healthy.

An enterprise control plane must connect declared intent with runtime evidence across the entire fleet.

What OpAMP actually provides

The OpenTelemetry specification describes OpAMP as a network protocol for remotely managing large fleets of data collection agents. It is vendor-agnostic and supports communication between an OpAMP server and clients associated with managed agents.

Core capabilities include:

  • Reporting agent identity, description, version, capabilities, and health
  • Receiving and acknowledging remote configuration
  • Reporting effective configuration and configuration status
  • Reporting package or component inventory
  • Receiving package update offers where the implementation supports them
  • Establishing bidirectional management communication over WebSocket or HTTP

OpAMP is a protocol, not a complete fleet-management product. It does not decide who may approve a production configuration, how rollout rings are selected, what policy is acceptable, or how a failed change should be escalated. Those are control-plane responsibilities that an enterprise platform must implement around the protocol.

The specification is currently marked beta. Newer Collector management work, including an alpha OpAMP Gateway Extension discussed by the CNCF, is promising but should be treated according to its maturity. Protocol adoption and production rollout should be deliberately separated from assumptions about experimental components.

The observability control plane needs a clear contract

A useful control plane maintains two views of every managed agent.

Desired state describes what the organization intends: approved Collector version, component set, configuration bundle, certificates, export destinations, and rollout assignment.

Observed state describes what the agent reports: identity, capabilities, effective configuration, health, errors, version, and last successful communication.

The difference between these views is configuration drift. Drift is not automatically a failure. An agent may be offline, a rollout may be paused, or a local emergency override may be permitted. The control plane should classify the difference, assign an owner, and decide whether to reconcile, roll back, or escalate.

This is why OpAMP complements rather than replaces GitOps. Git remains the reviewable source of approved configuration. OpAMP provides a standardized delivery and feedback channel for agents that cannot all be managed through the same deployment mechanism.

A reference enterprise architecture

A practical architecture separates policy, rollout orchestration, and protocol transport.

  1. Configuration repository. Versioned Collector templates, component allow lists, routing policy, environment overlays, and rollout metadata are reviewed through pull requests.
  2. Build and validation service. Every bundle is parsed, semantically validated, policy-checked, and tested against representative telemetry before promotion.
  3. Fleet inventory. The platform records agent identity, owner, environment, workload class, capabilities, current version, desired version, and health.
  4. Rollout controller. A change is assigned to cohorts, advanced through rings, paused on thresholds, and linked to an immutable configuration revision.
  5. OpAMP server. The server communicates desired state to clients and receives acknowledgements and status. It should not become the only system of record for policy decisions.
  6. Managed agents. Collectors or supervisors authenticate to the control plane, apply supported changes, and report effective state and health.
  7. Control-plane observability. The management system emits its own metrics, logs, and traces to an independent path so a fleet failure remains visible.

This architecture keeps configuration governance in familiar enterprise workflows while using OpAMP for standardized fleet interaction.

Identity is the first security boundary

A remote management channel can change what telemetry is collected, where it is sent, and which components execute. It is therefore a high-value security boundary.

Each client needs a stable identity tied to an owner and expected environment. Transport encryption is necessary but not sufficient. The server must authorize what that identity may receive, which cohort it belongs to, and whether it can accept sensitive configuration.

Recommended controls include:

  • Mutual authentication with short-lived, automatically rotated credentials
  • Per-agent or narrowly scoped workload identities rather than shared fleet secrets
  • Authorization by tenant, environment, geography, and workload class
  • Signed or integrity-protected configuration artifacts
  • Strict separation between configuration authors, approvers, and rollout operators
  • Audit records linking every server instruction to a reviewed revision and actor
  • Egress restrictions so agents communicate only with approved management and telemetry endpoints
  • Safe local behavior when the management server is unavailable

The server also needs protection from compromised agents. Rate limits, message-size limits, tenant isolation, replay resistance, input validation, and anomaly detection should be part of the threat model.

Configuration rollout should look like progressive delivery

Collector configuration can affect the visibility of an entire production estate. Treating a fleet-wide change as a simple push is an operational risk.

A safer workflow uses progressive rollout rings:

  1. Validation. Parse the configuration, resolve components, verify endpoints, run policy checks, and exercise representative telemetry.
  2. Development cohort. Apply the revision to disposable or low-risk agents and verify configuration acknowledgement.
  3. Canary cohort. Select a small production group that represents important environments and traffic patterns.
  4. Regional or workload rings. Expand only while health, drop rate, queue pressure, and backend load remain within thresholds.
  5. Fleet completion. Record coverage and identify offline or incompatible agents as explicit exceptions.
  6. Rollback. Restore the last known-good revision automatically when defined safety conditions fail.

A rollout is not successful because the server sent a configuration. It is successful when the intended agents report the expected effective state and the telemetry pipeline remains healthy.

Observe the observability fleet

The Collector layer is part of the monitoring system, so its management telemetry must not disappear into the same failure domain.

Track at least:

  • Active, offline, unknown, and quarantined agents
  • Desired-versus-effective configuration drift
  • Configuration acknowledgement and failure rates
  • Rollout duration and rollback frequency
  • Agent version and component-version distribution
  • Telemetry receive, drop, retry, queue, and export-failure rates
  • Credential age and failed authentication attempts
  • Management-channel latency and reconnect rate
  • Coverage by business service, environment, and data type

Business-level objectives matter as well. How quickly can the organization deploy a new security log source? How long does it take to revoke a compromised exporter credential? What percentage of critical services has a healthy, policy-compliant collection path?

An operating model for shared ownership

Fleet management spans organizational boundaries. Clear ownership prevents the control plane from becoming either an unresponsive central bottleneck or an uncontrolled self-service system.

  • The observability platform team owns the protocol service, supported agent profiles, configuration schemas, rollout automation, and service-level objectives.
  • Security owns control objectives, management-plane threat modeling, credential requirements, and sensitive destination policy.
  • Service and infrastructure teams own agent coverage, local dependencies, and declared business criticality.
  • Backend owners publish capacity constraints and compatibility requirements.
  • A change advisory model is encoded through risk tiers, automated evidence, and approval rules rather than a universal manual meeting.

Teams should be able to request supported pipelines and processors through a controlled interface. They should not need permission for every low-risk change, but they also should not be able to redirect enterprise telemetry to an arbitrary endpoint.

A staged adoption plan

Phase 1: establish inventory and evidence

  • Enumerate Collector deployments and other managed agents.
  • Assign ownership, environment, and criticality.
  • Define a small set of supported configurations and component versions.
  • Measure current drift, rollout time, and blind spots before adding remote control.

Phase 2: introduce OpAMP in read-oriented mode

  • Connect a non-critical cohort.
  • Collect agent descriptions, versions, effective configuration, and health.
  • Validate identity and tenant boundaries.
  • Compare observed state with the Git-approved desired state.

Phase 3: controlled configuration delivery

  • Enable remote configuration for one standardized agent profile.
  • Use signed revisions, canary rings, automated thresholds, and rollback.
  • Exercise server outage, invalid configuration, expired credentials, and incompatible-agent scenarios.

Phase 4: expand deliberately

  • Add heterogeneous environments and additional agent capabilities.
  • Integrate package updates only after configuration delivery is reliable.
  • Publish service objectives and an exception process.
  • Keep experimental extensions behind explicit maturity and risk gates.

Standard telemetry needs standard operations

OpenTelemetry solves an important portability problem, but enterprises also need a portable way to operate the collection fleet. OpAMP creates the protocol foundation for that control plane.

The durable design is not a central server that can push arbitrary files. It is a governed system that connects reviewed intent to agent identity, progressive rollout, effective state, health evidence, and safe recovery. Organizations that build those capabilities can scale OpenTelemetry without replacing proprietary telemetry agents with a new collection layer that is open but operationally opaque.

Sources

Veröffentlicht am

Dapr Agents v1.0: Resilient Multi-Agent Orchestration on Kubernetes

The Distributed Systems Foundation for AI Agents

When LangGraph introduced stateful agents and CrewAI popularized role-based collaboration, they solved the what of multi-agent AI systems. But as organizations move from demos to production, a critical question emerges: how do you run these systems reliably at scale?

Enter Dapr Agents, which reached v1.0 GA in March 2026. Built on the battle-tested Dapr runtime—a CNCF graduated project—this Python framework takes a fundamentally different approach: instead of bolting reliability onto AI frameworks, it brings AI agents to proven distributed systems primitives.

The result? AI agents that inherit decades of distributed systems wisdom: durable execution, exactly-once semantics, automatic retries, and the ability to survive node failures without losing state.

Why Traditional Agent Frameworks Struggle in Production

Most AI agent frameworks were designed for prototyping. They work brilliantly in Jupyter notebooks but encounter friction when deployed to Kubernetes:

  • State Loss on Restart: LangGraph checkpoints require manual persistence configuration. A pod restart can lose agent memory mid-conversation.
  • No Native Retry Semantics: When an LLM API returns a 429, most frameworks fail or require custom retry logic.
  • Coordination Complexity: Multi-agent communication typically requires custom message queues or REST endpoints.
  • Observability Gaps: Tracing an agent’s reasoning across multiple tool calls often means stitching together fragmented logs.

Dapr Agents addresses each of these by standing on the shoulders of infrastructure patterns that have been production-hardened since the early days of microservices.

Architecture: Agents as Distributed Actors

At its core, Dapr Agents builds on three Dapr building blocks:

1. Workflows for Durable Execution

Every agent interaction—LLM calls, tool invocations, state updates—is persisted as a workflow step. If the agent crashes mid-reasoning, it resumes exactly where it left off:

from dapr_agents import DurableAgent, tool

class ResearchAgent(DurableAgent):
    @tool
    def search_arxiv(self, query: str) -> list:
        return arxiv_client.search(query)
    
    async def research(self, topic: str):
        papers = await self.search_arxiv(topic)
        summary = await self.llm.summarize(papers)
        return summary

Under the hood, Dapr Workflows use the Virtual Actor model—the same pattern that powers Orleans and Akka. Each agent is a stateful actor that can be deactivated when idle and reactivated on demand, enabling thousands of agents to run on a single node.

2. Pub/Sub for Event-Driven Coordination

Multi-agent systems need reliable communication. Dapr’s Pub/Sub abstraction lets agents publish events and subscribe to topics without knowing about the underlying message broker:

from dapr_agents import AgentRunner

await agent_a.publish("research-complete", {
    "topic": "quantum computing",
    "findings": summary
})

@runner.subscribe("research-complete")
async def handle_research(event):
    await writer_agent.draft_article(event["findings"])

Swap Redis for Kafka or RabbitMQ without changing agent code.

3. State Management for Agent Memory

Conversation history, tool results, reasoning traces—all flow through Dapr’s State API with pluggable backends:

from dapr_agents import memory

agent = ResearchAgent(memory=memory.InMemory())

agent = ResearchAgent(
    memory=memory.PostgreSQL(
        connection_string=os.environ["PG_CONN"],
        enable_vector_search=True
    )
)

Agentic Patterns Out of the Box

Dapr Agents ships with implementations of common multi-agent patterns:

Pattern Description Use Case
Prompt Chaining Sequential LLM calls where each output feeds the next Document processing
Evaluator-Optimizer One LLM generates, another critiques in a loop Code review
Parallelization Fan-out work to multiple agents, aggregate results Research synthesis
Routing Classify input and delegate to specialist agents Customer support
Orchestrator-Workers Central coordinator delegates subtasks dynamically Complex workflows

MCP and Cross-Framework Interoperability

A standout feature is native support for the Model Context Protocol (MCP):

from dapr_agents import MCPToolProvider

tools = MCPToolProvider("http://mcp-server:8080")
agent = DurableAgent(tools=[tools])

Dapr Agents can also invoke agents from other frameworks as tools:

from dapr_agents.interop import CrewAITool

research_crew = CrewAITool(crew=research_crew, name="research_team")
coordinator = DurableAgent(tools=[research_crew])

Kubernetes-Native Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: research-agent
  annotations:
    dapr.io/enabled: "true"
    dapr.io/app-id: "research-agent"
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: agent
        image: myregistry/research-agent:v1

Comparison: Dapr Agents vs. LangGraph vs. CrewAI

Capability Dapr Agents LangGraph CrewAI
Durable Execution Built-in Requires config Limited
Auto Retry Built-in Manual Manual
State Persistence 50+ backends SQLite, PG In-memory
Kubernetes Native Sidecar Manual Manual
Observability OpenTelemetry LangSmith Limited

When to Choose Dapr Agents

Dapr Agents makes sense when:

  • You’re already running Dapr for microservices
  • Your agents must survive node failures without state loss
  • You need to scale to thousands of concurrent agents
  • Enterprise observability requirements demand OpenTelemetry

Getting Started

pip install dapr-agents
dapr init

from dapr_agents import DurableAgent, AgentRunner

class GreeterAgent(DurableAgent):
    system_prompt = "You are a helpful assistant."

runner = AgentRunner(agent=GreeterAgent())
runner.start()

The Bigger Picture

Dapr Agents represents a broader trend: AI frameworks are maturing from „make it work“ to „make it work reliably.“ The CNCF ecosystem is converging on this need—KubeCon 2026 showcased kagent, AgentGateway, and the AI Gateway Working Group.

For platform teams, Dapr Agents offers a familiar operational model: sidecars, state stores, message brokers, and observability pipelines. The agents are new; the infrastructure patterns are proven.

Dapr Agents v1.0 is available now at github.com/dapr/dapr-agents.