Beiträge

Veröffentlicht am

AI Code Review Needs a Platform Contract: Governing Instructions, Network Access, and Runners

AI-assisted code review is moving beyond a simple product toggle. Once a reviewer can read repository instructions, execute setup steps, access networks, and run on different compute environments, it becomes part of the software delivery platform. That changes the governance question from “Should we enable AI review?” to “Under which versioned and auditable contract may it operate?”

GitHub’s latest Copilot code review improvements make that shift visible. Copilot code review can now read instructions from the pull request’s head branch, consume additional instruction formats such as REVIEW.md, GEMINI.md, and CLAUDE.md, execute custom setup steps, run behind a configurable firewall, and use runner settings that are independent from the Copilot cloud agent.

These capabilities can improve relevance and reduce false positives. They also expand the execution surface around every review. Enterprises should therefore treat the configuration as platform policy rather than scattered repository preferences.

The reviewer now has an execution environment

Traditional static analysis usually operates under a relatively fixed contract: a known rule set evaluates a defined artifact and emits deterministic results. An AI reviewer behaves differently. Its output depends on context, instructions, available tools, network access, and the environment in which it runs.

AI review can work in an enterprise, but model selection is only one part of the operating model.

Four control planes now shape the result:

  • Instruction policy: which repository files influence the review and who may change them.
  • Setup policy: which commands and dependencies prepare the review environment.
  • Network policy: which external destinations the reviewer may reach.
  • Runner policy: where the review executes and which identities, secrets, and internal resources are available there.

If these controls evolve independently, two repositories using the same AI reviewer can have very different security and quality properties. A platform contract makes those differences explicit.

Instructions are executable review policy

Reading custom instructions from the head branch is useful because teams can test review guidance before merging it. It also creates a trust-boundary question: a pull request can propose code changes and changes to the instructions used to review that code.

This is not inherently unsafe. Pull requests already modify workflows, build scripts, and policy files. The important point is that instruction files deserve comparable governance.

A practical enterprise baseline should:

  • define which instruction formats are supported and where they may live;
  • assign CODEOWNERS to organization-critical instruction and agent-skill paths;
  • require human review when a pull request changes both application code and reviewer policy;
  • separate mandatory organization guidance from repository-specific context;
  • test instruction changes against representative pull requests before broad rollout;
  • retain the exact instruction revision with review evidence where audit requirements justify it.

Product teams still need room for domain-specific guidance. The platform team should own the contract, validation, and protected baseline, while repository owners maintain local context within those boundaries.

Setup steps belong to the software supply chain

Custom setup steps can make reviews substantially better. A reviewer may need generated code, dependency metadata, project-specific linters, or a compiled schema before it can understand a change.

However, setup steps are code execution. They can download dependencies, run scripts from the branch, and interact with credentials exposed to the runner. Treating them as harmless customization would recreate risks that enterprises already learned to manage in CI/CD.

The platform contract should require:

  • minimal and explicit setup steps;
  • pinned versions or immutable references for downloaded tools and reusable actions;
  • dependency caching that does not weaken provenance checks;
  • timeouts and resource limits;
  • no production credentials in the review environment;
  • clear separation between untrusted pull-request code and trusted setup logic;
  • logs that show what ran without exposing secrets.

Where setup behavior is common across many repositories, publish a maintained golden path instead of copying shell fragments. Teams can then inherit a reviewed baseline and add only the repository-specific preparation they need.

Network access needs an exception model

GitHub states that Copilot code review now runs behind a firewall by default and that its network rules can be configured independently from the Copilot cloud agent. This is an important default because review tasks often need less network access than general-purpose coding agents.

The right enterprise pattern is deny by default with narrow, observable exceptions. Typical allowed destinations may include an internal package repository, a schema registry, or an approved documentation endpoint. General internet access should require a documented reason.

Every exception should answer four questions:

  1. Which destination is required?
  2. Which review capability depends on it?
  3. What data may leave the runner?
  4. Who owns and periodically revalidates the exception?

The current GitHub notice also matters operationally: self-hosted runners do not support the code-review firewall. Organizations using self-hosted runners therefore need compensating controls at the network, workload, or runner-pool level. “Self-hosted” should not be interpreted as “automatically safer.”

Runner selection is a data-boundary decision

Separating runner configuration for Copilot code review from the Copilot cloud agent allows enterprises to match infrastructure to the task. It also prevents an overly broad agent configuration from becoming the default for review workloads.

Runner policy should be based on data classification and required connectivity:

  • use isolated hosted runners when repositories need no private network access;
  • use hardened self-hosted pools only when internal dependencies make them necessary;
  • keep runner images minimal and reproducible;
  • use short-lived credentials with repository-scoped permissions;
  • prevent persistence between review jobs;
  • separate runner pools for different trust zones.

Runner selection affects far more than performance. It determines what source code, internal services, credentials, and telemetry the review process can reach.

A platform contract for AI code review

A useful contract should be small enough for teams to understand and strict enough for governance teams to trust. It can be implemented as policy-as-code, reusable repository configuration, protected settings, and automated conformance checks.

The contract should define at least:

  1. Purpose: AI review supplements human review and deterministic controls; it does not become an unmeasured release gate.
  2. Instruction ownership: approved formats, protected paths, and required reviewers.
  3. Execution policy: allowed setup operations, dependency provenance, resource limits, and secret handling.
  4. Egress policy: default network posture, approved destinations, and exception ownership.
  5. Runner policy: permitted runner classes by repository classification and connectivity need.
  6. Evidence: the configuration version, runner class, policy result, and operational outcome retained for each review where needed.
  7. Escalation: how developers report incorrect findings, missing context, or policy constraints that reduce review quality.

This contract should be versioned. A change to review instructions, setup behavior, firewall rules, or runner selection can alter both quality and risk. It deserves the same change-management discipline as a CI template or deployment policy.

Measure the operating model, not just adoption

Counting how many repositories enabled AI review says little about whether it works safely. Platform and engineering leaders need evidence across quality, reliability, and governance.

Useful measures include:

  • acceptance and dismissal rates for review findings;
  • time from finding to resolution;
  • setup-step failure and timeout rates;
  • repositories compliant with the approved runner and firewall baseline;
  • number and age of network exceptions;
  • frequency of instruction changes and their effect on finding quality;
  • cases where deterministic checks or human reviewers contradicted the AI reviewer.

These measures create a feedback loop. Teams can improve instructions and setup context while the platform team identifies systemic policy or reliability problems.

A pragmatic rollout sequence

Enterprises do not need to design the final governance model before gaining value. A controlled rollout can proceed in four steps:

  1. Inventory: identify existing instruction files, setup workflows, runner types, and network dependencies.
  2. Baseline: define one default contract with protected instructions, minimal setup, restricted egress, and an approved runner class.
  3. Pilot: onboard a representative set of repositories and measure finding quality, setup reliability, and exceptions.
  4. Scale: publish golden-path configuration, automate conformance checks, and review exceptions periodically.

This approach lets the organization learn without turning every repository into a separate governance experiment.

From feature configuration to delivery policy

GitHub’s new controls are useful because they expose the real shape of AI-assisted review: it is a configurable workload operating inside the delivery system. Instructions influence judgment. Setup steps execute code. Network rules define egress. Runners define the data boundary.

Enterprises now have to make this customization reviewable, reusable, and measurable.

Platform teams should provide the paved road: a versioned contract, secure defaults, approved extension points, and evidence that the reviewer operated within policy. Product teams should own the domain context that makes reviews useful. Security and engineering leadership should define the exceptions and success measures.

That division of responsibility allows AI code review to improve delivery quality without creating an invisible parallel control plane.

GitHub’s implementation details are described in the Copilot code review customization announcement, the code review environment documentation, the repository custom instructions guidance, and the firewall configuration guidance.

Veröffentlicht am

AI Security Findings in Pull Requests: Treat the Model as a Reviewer, Not a Release Gate

GitHub now surfaces AI-powered security detections directly in pull requests. The feature extends coverage to languages and frameworks that CodeQL does not currently support, including examples such as PHP, Shell, Terraform configuration, Dockerfiles, JSP, and Blazor.

That is useful coverage. It is not the same thing as a release control.

GitHub explicitly describes these findings as advisory. They are labeled as AI-generated, may contain false positives, appear only on pull requests, and cannot currently be used in rulesets to enforce merge requirements. The feature is also in public preview and its supported languages and detection categories may change.

Enterprise engineering organizations should preserve that distinction. Use the model as an additional security reviewer that broadens visibility. Keep release gates grounded in deterministic controls, validated policies, and accountable human decisions.

The value is coverage, not certainty

CodeQL provides high-precision static analysis for supported languages and queries. AI-powered security detections address a different problem: codebases contain languages, frameworks, infrastructure definitions, and integration patterns that deterministic analyzers may not cover.

GitHub’s AI engine analyzes changes when a pull request is opened or updated. It can use code search to gather repository context and reports findings as they become available. The results appear alongside CodeQL alerts but carry an AI label so reviewers can distinguish the evidence source.

This creates practical value in three areas:

  • Coverage expansion: teams receive security signals in previously unscanned parts of the repository.
  • Workflow placement: findings appear where developers already discuss and approve changes.
  • Contextual explanation: a finding includes a risk explanation and often a suggested remediation.

None of those benefits requires treating the model’s conclusion as an objective fact. The useful product is a prioritized question for the reviewer: Is this change unsafe, and what evidence confirms or rejects that assessment?

Why an AI finding should not become an automatic release gate

A release gate is an enforcement mechanism. When it fails, delivery stops. That makes consistency, explainability, availability, and predictable remediation essential operational properties.

AI-generated findings have different characteristics.

Model output is probabilistic

The same flexibility that lets a model reason across unfamiliar frameworks also introduces uncertainty. GitHub’s documentation acknowledges that findings may include false positives. A noisy blocking control creates alert fatigue, encourages bypasses, and can reduce trust in the entire security program.

The detection surface can evolve

The feature is in public preview. Detection categories and language coverage may change as the product evolves. A release policy tied directly to an evolving model can change effective enforcement without a corresponding policy review inside the enterprise.

Results are asynchronous

AI and CodeQL analysis run independently, and findings are posted as each engine returns them. A fast-moving pull request may therefore see one source before the other. A release process must define whether it waits, who evaluates late findings, and what happens when a result arrives after approval.

The product itself treats them as advisory

GitHub states that AI-powered findings do not block merges and cannot currently be used in rulesets for merge enforcement. Recreating an improvised hard gate around an advisory preview feature transfers the operational risk to the platform team without improving the underlying evidence quality.

The right conclusion is not to ignore AI findings. It is to design a decision process appropriate to their evidence class.

Build an explicit security evidence hierarchy

A mature pull-request policy should distinguish how a finding was produced and how much confidence the organization has earned in it.

Class 1: deterministic blocking controls

These controls have clear pass or fail semantics and an agreed relationship to release risk. Examples include required tests, policy-as-code checks, secret push protection, approved dependency rules, and configured code-scanning merge protection for validated analyzers and severities.

Failures block the merge because the organization has intentionally accepted the trade-off between delivery speed and risk reduction.

Class 2: deterministic advisory findings

Some scanner results are reliable but not severe enough to stop every change. They remain visible, receive an owner and service-level expectation, and may be promoted to blocking after the policy is validated.

Class 3: AI advisory findings

These are hypotheses that require triage. They should be labeled, routed, measured, and resolved with a documented outcome. They extend the reviewer’s attention but do not replace the reviewer’s judgment.

Class 4: confirmed risk decisions

Once a qualified reviewer confirms a material vulnerability, the decision is no longer merely a model output. The team can require remediation, accept the risk through an accountable exception, or stop the release under the existing security policy.

This hierarchy prevents a common category error: confusing the mechanism that discovered a concern with the governance decision that determines whether software may ship.

A practical pull-request operating model

The operating model should connect detection, triage, decision, and learning without creating a parallel workflow outside the pull request.

1. Preserve the source label

Do not normalize every result into a generic „security failed“ status. Keep the AI indicator and record the detection source, category, repository, language, commit, and time. Reviewers need to know whether they are evaluating a deterministic query, a model-generated hypothesis, or a human-confirmed issue.

2. Route by risk context

Not every repository needs the same handling. Use repository criticality, data classification, deployment target, and change ownership to determine the triage path.

  • A low-risk internal tool may let the author resolve the finding with peer review.
  • A customer-facing service may require a security champion for high-impact categories.
  • An identity, payment, or production-control component may require application-security review before approval.

The model can suggest severity, but enterprise routing should also use deterministic context the organization owns.

3. Require a recorded disposition

Each material finding should end with one of a small number of outcomes:

  • confirmed and fixed;
  • confirmed and accepted through the risk process;
  • false positive with a short technical rationale;
  • duplicate of an existing issue;
  • deferred to a tracked remediation item.

A thumbs-up or thumbs-down signal can help improve detection quality, but the enterprise also needs its own auditable disposition when the finding influenced a release decision.

4. Keep enforcement in the existing control plane

Use branch rulesets, required status checks, approval requirements, and policy-as-code for actual merge protection. If a confirmed AI finding should stop a release, translate that human decision into an existing accountable control rather than letting an unreviewed model response become the gate.

5. Handle late findings deliberately

Because results can arrive asynchronously, define a policy for high-risk repositories. Options include a minimum review stage, a named security owner, or post-merge follow-up when an advisory result arrives after approval. The policy should reflect system criticality instead of imposing the same delay on every repository.

Measure detection quality before changing policy

AI security coverage should be managed as an evidence-producing service. Usage counts alone do not show whether it reduces risk.

Track at least:

  • findings by repository, language, framework, and category;
  • confirmed findings and false-positive dispositions;
  • time from finding to first triage and final resolution;
  • findings fixed before merge versus deferred or accepted;
  • late findings that arrived after approval or merge;
  • repeat findings for the same weakness or component;
  • review effort per confirmed vulnerability;
  • AI credit consumption per run and per confirmed finding;
  • developer feedback and bypass behavior.

Precision is especially important for workflow trust: of the findings reviewed, how many were confirmed? Recall is harder because the organization does not automatically know what the model missed. Periodic expert review, penetration tests, incident data, and comparison with other scanners can provide partial evidence.

Do not create an arbitrary global threshold and call the model „validated.“ Quality can differ by language, repository pattern, and vulnerability category. Evaluate the segments that matter to your environment.

Promote patterns, not raw model confidence

If repeated AI findings reveal a reliable class of defects, the long-term goal should be to convert that learning into a deterministic control where possible.

For example:

  • Add a CodeQL query or another static-analysis rule for a recurring unsafe pattern.
  • Create a policy-as-code rule for an insecure infrastructure configuration.
  • Improve a secure library or platform template so teams avoid the defect by default.
  • Add a focused test to the affected component.
  • Update a golden path, coding standard, or reviewer checklist.

This is how AI improves the release system without becoming its single point of judgment. The model discovers weak signals; the platform team turns proven patterns into repeatable controls.

Govern the feature as an enterprise capability

GitHub requires enterprise policy permission, organization-level opt-in, CodeQL default setup, GitHub Advanced Security, and during public preview a GitHub Copilot license. Runs also consume AI credits.

That makes enablement a portfolio decision, not merely a repository toggle.

Enterprise owners should define:

  • which organizations and repository classes may use the feature;
  • who owns configuration, cost, triage policy, and support;
  • which repositories provide the initial evaluation cohort;
  • how findings and dispositions are retained for audit purposes;
  • how credit consumption is budgeted and attributed;
  • how product-preview changes are reviewed before broader rollout;
  • how teams report harmful noise, gaps, or inconsistent behavior.

Platform engineering, application security, developer experience, and service owners all have a role. Security defines risk policy. Platform engineering integrates the workflow and evidence. Developer experience monitors friction. Service owners remain accountable for the code they merge.

A staged rollout plan

Phase 1: baseline

Document existing CodeQL coverage, merge protection, security-review responsibilities, and unsupported languages. Keep current release gates unchanged.

Phase 2: advisory pilot

Enable AI detections for a representative set of repositories with meaningful coverage gaps. Train reviewers to distinguish AI findings from CodeQL alerts and require a simple disposition for reviewed findings.

Phase 3: measure and tune

Review confirmed findings, false positives, triage time, late results, developer effort, and AI credit consumption. Segment the results by language and finding category.

Phase 4: institutionalize learning

Turn recurring confirmed patterns into deterministic queries, tests, policies, secure defaults, or platform templates. Define escalation paths for categories that consistently indicate material risk.

Phase 5: expand with evidence

Extend the feature to additional repository classes only when the organization can support the triage load and demonstrate useful detection quality. Reassess the operating model as the public-preview capability changes.

The model reviews; the organization decides

AI-powered security detections can close meaningful coverage gaps and bring more security context into the pull request. Their value is strongest when they broaden human attention and feed continuous improvement.

A release gate carries a different responsibility. It must represent an explicit, accountable policy backed by evidence the organization understands and can operate reliably.

Treat the model as a security reviewer. Measure its findings. Confirm the risk. Convert repeatable lessons into deterministic controls. Then let people and policy decide whether the software is ready to ship.

Sources

Veröffentlicht am

Pods Are Workers, Not Agents: Designing the Runtime Boundary for Enterprise Agent Platforms

Kubernetes Pods are excellent execution units. They provide scheduling, resource controls, networking, workload identity integration, and a natural boundary for security and observability.

That does not automatically make a Pod the right representation of an AI agent.

Enterprise agent platforms need to distinguish two concepts that are easy to collapse during early implementations: the logical agent and the runtime worker executing its current task. Treating them as the same object can work for prototypes and continuously running agents. At scale, it creates idle infrastructure, slow burst handling, fragmented identity, and weak lifecycle semantics.

The durable pattern is to let Kubernetes manage execution workers while an agent control plane manages agent identity, state, policy, placement, and lifecycle. Pods remain essential. They become workers rather than the agent itself.

Why one Pod per agent is an attractive first design

The one-agent-per-Pod model solves several real problems quickly.

  • A Pod provides a process and container isolation boundary.
  • A ServiceAccount gives the workload a Kubernetes identity.
  • NetworkPolicy and admission policy can constrain its environment.
  • CPU and memory requests make resource consumption schedulable.
  • Logs, metrics, and traces can be attributed to a workload instance.
  • Existing GitOps, deployment, and incident-response practices remain usable.

For a small number of high-value agents, those benefits may outweigh the overhead. The model is understandable and conservative. It uses boundaries that platform and security teams already know how to operate.

The problem appears when the organization assumes that the execution container is also the durable identity and lifecycle of the agent.

Agents do not behave like ordinary services

A typical service is expected to remain available and handle a continuing stream of requests. An agent may wake up for a task, run for seconds or minutes, wait for a human decision, delegate work to subagents, and then remain idle for hours.

These characteristics create a different workload shape:

  • Bursty demand: a single business event can fan out into many parallel agent tasks.
  • Long idle periods: logical agents may exist without needing compute.
  • External waiting: execution may pause for approval, data, or another system.
  • Variable duration: tasks range from short tool calls to extended research or coding sessions.
  • Delegated authority: an agent often acts on behalf of a user or workflow rather than only as itself.
  • Stateful continuation: a later execution may need to resume the same logical conversation or plan on a different worker.

Keeping one Pod alive for every logical agent reserves capacity for identities that are not doing work. Creating a fresh Pod for every short task can introduce startup latency and control-plane churn. Encoding state inside the Pod makes rescheduling and recovery harder.

The architectural question is therefore not whether Kubernetes should run agents. It is which responsibilities belong to Kubernetes and which belong to an agent-specific control plane.

The runtime boundary: agents, actors, and workers

A recent CNCF article describing kagent’s agent-substrate architecture illustrates this separation. Kubernetes continues to manage Pods, networking, storage, and compute. A higher-level control plane manages logical actors and places them onto a pool of execution workers.

In that model:

  • The logical agent has durable identity, ownership, policy, configuration, and state.
  • An agent task or actor instance represents a unit of active execution.
  • A worker is a sandboxed runtime capable of executing one or more assigned actors.
  • A worker pool defines capacity, runtime profile, isolation class, and placement characteristics.

Agent-substrate is one implementation, not a universal enterprise standard. Its value for platform design is the principle it demonstrates: logical lifecycle can be decoupled from Pod lifecycle without removing Kubernetes from the architecture.

Six contracts the control plane must preserve

Decoupling an agent from a Pod improves efficiency only if the platform preserves the controls that dedicated Pods made easy.

1. Durable agent identity

An agent needs an identity that survives worker replacement. That identity should identify the agent definition, tenant, owner, environment, risk tier, and approved capabilities.

The worker also needs its own workload identity. The two must not be confused. A worker identity proves which runtime is communicating with the platform. The agent identity determines which business permissions and policies apply to the assigned execution.

When an agent acts for a person, the authorization decision should include delegated user context with explicit scope and expiry. Copying a user’s full credentials into a worker is not delegation.

2. Execution leases

Placement should create a time-bound execution lease binding an agent task to a specific worker. The lease should include the agent identity, policy revision, tool permissions, state reference, deadline, and expected resource profile.

Leases make reassignment and failure handling explicit. If a worker disappears, the control plane can determine whether the task is safe to retry, must resume from a checkpoint, or requires human review.

3. Isolation classes

Sharing workers does not mean sharing trust. The platform needs multiple runtime profiles based on risk.

  • Low-risk, read-only tasks may use a warm multi-tenant worker pool.
  • Tasks handling confidential data may require stronger sandboxing and tenant-dedicated workers.
  • Agents with write access to production systems may require a dedicated Pod or ephemeral sandbox per execution.
  • Untrusted code execution may require gVisor, microVMs, or another hardened isolation boundary.

The scheduling decision should derive from policy. Developers should request a workload class rather than select a weaker runtime to reduce latency.

4. Policy attribution

Kubernetes policy usually sees the Pod, namespace, and ServiceAccount. A shared worker introduces another logical principal inside that boundary. The platform must propagate agent, tenant, task, and delegated-user context to every policy enforcement point.

Tool gateways, model gateways, data APIs, and egress proxies should authorize the logical execution, not merely trust the worker’s network location. Audit events should record both worker identity and agent identity so investigators can reconstruct who did what and where it ran.

5. Externalized state and checkpoints

Agent state should not depend on the continued existence of a worker Pod. Conversation state, plans, artifacts, approval state, and checkpoints need durable storage with tenant-aware encryption and retention controls.

Externalizing state allows the platform to release compute while an agent is idle and rehydrate it when work resumes. It also creates a controlled recovery point instead of treating the worker filesystem as an accidental system of record.

6. End-to-end observability

Pod-level telemetry remains necessary but is no longer sufficient. Operators need to follow a logical agent across workers and over time.

Every execution should carry stable correlation fields such as:

  • agent, tenant, task, session, and parent-task identifiers;
  • worker and worker-pool identity;
  • policy, prompt, model, and tool versions;
  • delegated user and approval references where permitted;
  • token, latency, tool-call, cost, and outcome signals;
  • checkpoint, retry, reassignment, and termination reasons.

This creates observability for the business execution rather than only for the container currently hosting it.

A reference enterprise architecture

A practical runtime separates responsibilities across four layers.

Agent control plane

The control plane stores agent definitions, ownership, policy, lifecycle, state references, and desired runtime class. It accepts tasks, decides placement, issues leases, tracks execution, and coordinates retries or resumptions.

Worker pools

Kubernetes Deployments or other controllers maintain warm capacity for defined execution profiles. Pools may differ by tenant, geography, accelerator, sandbox technology, network access, or data classification.

Shared platform gateways

Model, tool, MCP, data, and egress gateways enforce logical identity and policy. They keep privileged credentials out of agent code and provide consistent rate limits, approval checks, observability, and revocation.

Durable state and evidence

State services store checkpoints and artifacts. An evidence plane records immutable links between the agent definition, execution lease, policy decision, worker, model interaction, tool call, and outcome.

Kubernetes remains the infrastructure substrate. The agent control plane provides semantics Kubernetes was not designed to infer.

Multi-tenancy must shape worker placement

Worker utilization can improve dramatically when idle logical agents do not retain Pods. That benefit should not override tenant boundaries.

Platform teams should define placement rules covering:

  • whether tenants may share a worker process, Pod, node, or cluster;
  • which data classifications require dedicated runtime capacity;
  • how memory, filesystems, caches, and credentials are cleared between assignments;
  • whether agent-generated code can execute and under which sandbox;
  • which tools and destinations each pool can reach;
  • how noisy-neighbor behavior is detected and constrained;
  • where state and inference traffic may be processed geographically.

There is no single correct sharing boundary. The platform should offer a small set of reviewed isolation classes and make the selected class visible in cost, latency, and risk reporting.

When one Pod per agent is still the right answer

Decoupling should not become an objective by itself. A dedicated Pod remains a strong choice when:

  • the agent is continuously active or exposes a stable service endpoint;
  • startup latency is acceptable and the fleet is small;
  • the workload needs strong tenant or process isolation;
  • it runs untrusted code or privileged tools;
  • its memory and resource profile do not fit a shared pool;
  • existing Kubernetes controls provide sufficient lifecycle semantics;
  • the added agent scheduler would cost more to operate than it saves.

The mature platform supports more than one runtime pattern. It chooses the boundary based on workload behavior and risk rather than forcing every agent into the same optimization.

Measure the runtime as a platform product

Worker density is useful, but cost efficiency alone is an incomplete success measure. Track flow, reliability, isolation, and control together.

  • Task queue time and time to first execution
  • Warm-start and cold-start latency
  • Active versus idle worker utilization
  • Logical agents per worker and per isolation class
  • Checkpoint, resume, retry, and reassignment success rates
  • Policy denials and unauthorized cross-tenant attempts
  • State cleanup and credential revocation failures
  • Cost per successful agent task
  • Trace and audit coverage from task request to external side effect

A cheaper runtime that cannot explain an agent’s actions is not an enterprise improvement.

A staged adoption path

1. Separate identifiers before changing runtime

Introduce stable agent, task, tenant, and worker identifiers in the current platform. Propagate them through logs, traces, policy decisions, and tool calls. This exposes hidden coupling before a scheduler is introduced.

2. Externalize state

Move durable state and artifacts out of the Pod. Define checkpoint, retry, expiry, encryption, and deletion semantics. Test recovery from worker termination.

3. Add one low-risk worker pool

Select bursty, read-only tasks with clear resource limits. Compare queue time, utilization, cost, and operational effort with the dedicated-Pod baseline.

4. Add policy-aware placement

Introduce reviewed isolation classes and execution leases. Integrate logical identity with tool, model, data, and egress gateways. Exercise tenant separation and credential revocation.

5. Expand only with evidence

Move higher-risk agents after proving state hygiene, observability, rollback, and incident response. Keep dedicated Pods as an explicit option rather than treating them as a failed legacy design.

Pods should host work, not define the agent

The Pod remains one of the strongest execution boundaries available to cloud-native platforms. The mistake is asking it to carry semantics it does not own: durable agent identity, delegated authority, conversation lifecycle, human approval, and cross-execution state.

Enterprise agent platforms should model those concerns explicitly. Kubernetes can then do what it does best — schedule and isolate execution — while the agent control plane decides which logical work runs where, under whose authority, with which policy, and with what evidence.

That separation improves utilization, but its greater value is governance. It allows the platform to scale agents without losing the identity and accountability that production systems require.

Sources

Veröffentlicht am

The Agent Egress Boundary: Making Every AI Tool Call Enforceable and Observable

AI agents do not create risk only when they generate the wrong answer. They create operational risk when they turn that answer into an outbound action: calling an API, querying a search service, downloading content, opening a ticket, sending a message, or changing a production system.

Most enterprise controls still focus on the agent’s intent. Prompts, guardrails, and model policies describe what the agent should do. They do not guarantee which destinations the workload can reach, which request was sent, or whether an unapproved path was used.

That gap calls for an agent egress boundary: a platform-enforced control through which every external tool call must pass, combined with traceable evidence that links the call to the originating agent interaction.

Guardrails are necessary, but they are not enforcement

Prompt-level guardrails are useful for shaping behavior. They can tell an agent not to disclose sensitive information, not to call unknown services, or to request human approval before a consequential action. But those controls operate inside the reasoning path they are intended to constrain.

Production systems need an independent layer. If an agent is compromised through prompt injection, a poisoned tool response, a vulnerable dependency, or a simple implementation mistake, the network should still prevent access to destinations outside the approved contract.

The distinction is familiar from other areas of security:

  • application authorization expresses intended access;
  • network enforcement limits reachable destinations;
  • observability records what actually happened;
  • human approval controls high-impact exceptions.

No single layer is sufficient. Together, they create defense in depth.

The platform contract

An agent egress boundary should answer four questions for every outbound request:

  1. Who initiated it? Identify the workload, agent, tenant, and user or workflow context.
  2. Where is it going? Resolve the approved destination, protocol, port, and application-level route.
  3. Was it allowed? Evaluate the call against a versioned policy rather than an application convention.
  4. What evidence remains? Record a traceable decision without leaking secrets or sensitive payloads.

This turns outbound connectivity into a platform contract. An agent receives only the network access required by its tools, while the platform provides a consistent control and evidence plane.

A practical cloud-native pattern

A recent CNCF implementation demonstrates the core idea using NGINX, Kubernetes, and OpenTelemetry. NGINX acts as both the inbound reverse proxy and the outbound forward proxy for an agent workload. Network rules drop direct egress so the proxy becomes the only approved path. The NGINX OpenTelemetry module emits a span for each request, and an OpenTelemetry Collector forwards the evidence to observability or security systems.

The important principle is architectural: the boundary is not a library the agent may choose to call. It is the only network path available.

A production-oriented request flow can look like this:

  1. A user or system invokes the agent through an authenticated gateway.
  2. The gateway propagates a trace context and workload identity.
  3. The agent selects a tool and issues an outbound request.
  4. Kubernetes egress controls permit traffic only to the designated proxy.
  5. The proxy evaluates destination, protocol, identity, and policy.
  6. Allowed traffic is forwarded; denied traffic returns a controlled error.
  7. OpenTelemetry records the decision and correlates it with the originating interaction.

The result is a chain of evidence from user request to external side effect.

Why Kubernetes NetworkPolicy alone is not enough

Kubernetes NetworkPolicy is a strong foundation. It can isolate workloads and restrict egress by IP block, port, and selected peers, provided the cluster’s network plugin enforces the policy. A default-deny egress policy should be the starting point for sensitive agent workloads.

However, many agent tools call dynamic external services over HTTPS. IP addresses change, destinations share infrastructure, and business rules are usually expressed in terms of domains, API routes, methods, or tool identities rather than static addresses.

That is why a layered design is useful:

  • NetworkPolicy or equivalent CNI controls ensure the workload can only reach the approved proxy and essential platform services.
  • The egress proxy enforces destination and application-aware rules.
  • Workload identity distinguishes agents and tenants without relying only on source IP.
  • OpenTelemetry provides correlated evidence for operations, security, and audit.

The network layer prevents bypass. The proxy layer understands enough context to make a useful decision.

Policy should follow the tool contract

Allowing an agent to reach an entire domain is often broader than the tool definition requires. A better policy starts with the declared tool contract.

For example, an incident-analysis agent may need to:

  • read selected observability APIs;
  • create, but not delete, incident tickets;
  • query a controlled knowledge source;
  • send notifications only to an approved channel;
  • never call arbitrary internet destinations.

The platform can translate that contract into an egress policy covering destination, method, route, identity, rate, and approval requirements. High-risk actions can be routed through a separate approval service rather than granted as normal network access.

This also creates a cleaner ownership model. Domain teams define which tools are necessary. Security teams define control requirements. Platform teams provide the reusable enforcement mechanism.

Observability must produce evidence, not surveillance

OpenTelemetry is well suited to correlating inbound interactions with outbound HTTP client activity. Standard HTTP span conventions provide consistent attributes for requests and responses, while trace context links multiple services into one transaction.

But recording everything is not automatically safe. Agent traffic can include credentials, personal data, customer information, prompts, and tool payloads. The audit plane therefore needs its own policy.

Useful evidence

  • trace and request identifiers;
  • agent, workload, tenant, and tool identity;
  • policy version and allow or deny decision;
  • destination service and approved route classification;
  • HTTP method and status class;
  • latency, retries, and byte counts;
  • model or agent configuration version;
  • human approval reference where required.

Data to avoid by default

  • authorization headers and API keys;
  • full request or response bodies;
  • raw prompts containing confidential data;
  • URL query parameters unless explicitly sanitized;
  • unbounded high-cardinality attributes.

The purpose is to prove and investigate behavior, not to create a second uncontrolled copy of sensitive data.

Controls that make the boundary credible

A proxy is only a boundary when bypass is demonstrably difficult. Platform teams should validate at least the following controls:

  • Default-deny egress: direct external connectivity fails.
  • DNS control: workloads cannot switch to an unmonitored resolver or exploit unexpected resolution paths.
  • IPv4 and IPv6 parity: policy applies consistently to both address families.
  • Protocol coverage: non-HTTP tools, WebSockets, streaming APIs, and message protocols have explicit handling.
  • TLS design: the organization decides where TLS terminates and what metadata can be inspected without undermining privacy.
  • Identity: decisions rely on authenticated workload identity, not only mutable labels or network location.
  • Fail-closed behavior: proxy, collector, or policy failures do not silently open direct access.
  • High availability: the control plane does not become an avoidable single point of failure.

These details determine whether the pattern is an architectural control or merely a useful demonstration.

Operational signals for platform teams

Once all tool traffic crosses the boundary, the same telemetry can improve reliability and cost control.

Useful service-level indicators include:

  • allowed and denied tool calls by agent and policy version;
  • unexpected destinations or repeated policy violations;
  • external dependency latency and error rates;
  • retry storms and rate-limit responses;
  • egress volume and estimated third-party API cost;
  • calls that required human approval;
  • trace gaps where an outbound action lacks an originating interaction.

This gives security and operations teams a shared view. The same denied request may indicate an attack, an outdated policy, or a legitimate new tool requirement.

A phased adoption plan

  1. Inventory agent egress. Identify destinations, protocols, credentials, and business owners for each production tool.
  2. Introduce observation first. Capture sanitized outbound traces to understand real behavior before enforcing a narrow policy.
  3. Define tool-level contracts. Document approved destinations and actions rather than granting general internet access.
  4. Apply default deny. Force a low-risk agent through the proxy and prove that direct egress fails.
  5. Add policy-as-code. Version destination rules, ownership, exceptions, and approval conditions in Git.
  6. Connect the audit plane. Send sanitized OpenTelemetry data to the organization’s observability and SIEM platforms.
  7. Test failure modes. Validate DNS bypass, IPv6, proxy outage, collector outage, policy rollback, and certificate rotation.
  8. Scale by platform product. Offer the boundary as a reusable golden-path capability rather than a custom design for every agent.

Conclusion

Enterprises should not have to trust that an AI agent will respect its network boundaries. Those boundaries should be enforced by the platform and evidenced through telemetry.

NGINX, Kubernetes, and OpenTelemetry show that the core pattern can be built from mature cloud-native components: default-deny connectivity, an application-aware egress proxy, and correlated traces. The exact implementation will vary, but the platform contract should remain consistent.

Every agent tool call should be attributable, policy-checked, observable, and reversible where the downstream system allows it. That is the difference between experimenting with autonomous software and operating it responsibly.

Sources and further reading

Veröffentlicht am

The AI-Native Platform Contract: Expanding Golden Paths Beyond Application Delivery

Platform engineering earned its place by turning application delivery into a repeatable product. Golden paths combined infrastructure, security, deployment, and operational standards into a paved route that developers could use without learning every platform detail.

AI-native workloads do not invalidate that model. They expose where it stops too early.

A conventional golden path typically starts with source code and ends with a running service. An AI-native product depends on a wider chain: governed data, accelerator capacity, models and prompts, evaluation evidence, inference controls, agent identities, external tools, and continuous cost and risk feedback. If each of those capabilities arrives through a separate specialist portal, the organization has not created an AI platform. It has created another integration problem.

The next platform contract should therefore extend the golden path rather than build a parallel AI silo. The goal is not to hide every AI decision behind automation. It is to make safe defaults easy, exceptions explicit, and every promoted artifact traceable.

The application delivery contract is no longer enough

Platform Engineering 1.0 concentrated on a familiar delivery unit: an application packaged as a container, deployed through a pipeline, and operated with standard observability and security controls. That remains valuable, but AI changes both the workload and its consumers.

ML engineers need experiment tracking, model registries, feature and data access, and specialized compute. Application teams need stable inference endpoints and predictable latency. Security teams need controls for model provenance, prompt injection, data leakage, and non-human identities. FinOps teams need to attribute expensive training and inference usage. AI agents themselves become platform consumers that request tools, credentials, and runtime actions.

The CNCF discussion of evolving platform engineering for AI-native workloads captures this expansion through capabilities such as GPU and TPU allocation, model serving, MCP gateways, agentic guardrails, embedded FinOps, and policy-driven governance. The important organizational point is that these should not become an isolated platform owned by a small AI team. They should become extensions of the same product model, interfaces, and control philosophy used by the enterprise platform.

Define a platform contract, not a catalog of tools

A platform contract describes what a product team can request, what evidence it must provide, what the platform guarantees, and which controls are automatically applied. It is stronger than a service catalog entry and more flexible than a single mandatory implementation.

For an AI-native workload, that contract should cover at least six dimensions.

1. Governed data access

The path should make data classification, residency, retention, and permitted use visible before a workload reaches production. A request for a dataset should resolve to an approved identity, purpose, environment, and audit trail. The platform can automate access, but the product team remains accountable for whether the data is appropriate for the use case.

2. Compute and accelerator intent

Teams should request capabilities rather than hard-code a particular GPU model into every manifest. The contract can express workload class, memory, performance objective, duration, geographic constraints, and cost ceiling. Kubernetes mechanisms such as Dynamic Resource Allocation can support more structured resource claims, but the platform still needs policy for quotas, scarcity, preemption, and approved hardware profiles.

3. Model, prompt, and artifact provenance

Container images are not the only production artifacts. The platform must track model version, source, license, evaluation result, prompt bundle, retrieval configuration, tool definitions, and deployment policy. Promotion should be based on an immutable set of linked artifacts, not a model name copied into an environment variable.

4. Evaluation as a release gate

AI quality is probabilistic and context-dependent. A successful build does not prove production fitness. Golden paths should provide standard evaluation suites for task quality, safety, latency, robustness, and cost. Teams can add domain-specific tests, while the platform supplies the execution environment, evidence format, thresholds, and promotion workflow.

5. Runtime identity and guardrails

An inference service or autonomous agent needs a workload identity, scoped data access, approved tools, network boundaries, and observable policy decisions. The contract should distinguish a human user’s authority from an agent’s delegated authority. It should also define what happens when a model, tool, or policy is unavailable rather than allowing silent fallback to an uncontrolled path.

6. Cost and operational accountability

AI infrastructure introduces different cost behavior from ordinary stateless services. Training jobs can consume scarce capacity in bursts. Inference cost depends on model choice, token volume, batching, cache efficiency, and service-level objectives. Cost attribution and budgets should therefore be part of provisioning and release decisions, not a dashboard reviewed after the invoice arrives.

What an AI-native golden path looks like

A useful golden path follows the product lifecycle rather than exposing a collection of disconnected infrastructure forms.

  1. Declare the workload. The team selects an archetype such as batch training, online inference, retrieval-augmented generation, or tool-using agent. It declares data class, expected scale, latency objective, risk tier, and ownership.
  2. Provision an isolated workspace. The platform creates namespaces, identities, network boundaries, secrets references, storage, accelerator claims, quotas, and standard telemetry.
  3. Develop with approved building blocks. Teams consume versioned model endpoints, registries, feature services, MCP or tool gateways, and evaluation templates through stable APIs.
  4. Produce evidence. CI records model and data lineage, software dependencies, evaluation results, policy decisions, security findings, and predicted operating cost.
  5. Promote as a release set. GitOps promotes the linked application, model, prompt, policy, and tool configuration together. A rollback restores the complete known-good set.
  6. Operate with continuous feedback. Runtime telemetry covers service health, model quality indicators, policy denials, data drift, tool calls, accelerator utilization, and unit economics.

This lifecycle gives specialists room to innovate without forcing every product team to assemble the control plane themselves.

Avoid the separate AI platform trap

A dedicated AI enablement team may be necessary, but a separate delivery system should not be the default. Parallel identity models, pipelines, policy engines, and observability stacks increase cost and weaken governance. They also create a handoff between application engineers and AI specialists exactly where the product needs shared accountability.

A better operating model separates platform ownership by capability while preserving one product contract:

  • The core platform team owns common interfaces, workload identity, delivery workflows, policy integration, and the developer experience.
  • The AI platform capability team owns model-serving patterns, evaluation services, accelerator profiles, registries, and AI-specific runtime controls.
  • Data teams own governed data products and access semantics.
  • Security and risk teams define control objectives and approval boundaries as policy and evidence requirements.
  • Product teams own business fitness, domain evaluations, production outcomes, and accepted residual risk.

The teams collaborate through APIs, schemas, policy bundles, and service-level objectives rather than tickets and undocumented exceptions.

Measure whether the contract creates value

An AI-native platform should not be measured by the number of services in its catalog. Measure whether teams can deliver trustworthy outcomes faster.

  • Time from approved use case to first governed experiment
  • Time from candidate model to production release
  • Percentage of releases with complete model, data, prompt, and policy provenance
  • Evaluation failure escape rate
  • Percentage of agent tool calls using approved identities and gateways
  • Accelerator utilization and queue time by workload class
  • Inference cost per business transaction
  • Rollback time for a complete AI release set
  • Adoption and exception rates for each golden path

These metrics reveal whether the platform improves flow and control together. High adoption with slow delivery signals an overloaded path. Fast delivery with weak evidence signals unmanaged risk.

A practical 90-day starting point

Do not begin by designing a universal AI platform. Choose one real workload and use it to define the minimum viable contract.

Days 1–30: map the lifecycle

  • Select one representative AI product with a committed owner.
  • Map every artifact, identity, environment, approval, and operational dependency.
  • Classify which existing platform capabilities can be reused and where AI-specific gaps exist.
  • Define the workload’s risk tier, evaluation evidence, and cost objectives.

Days 31–60: build one vertical path

  • Create one workload template and governed workspace.
  • Connect model and prompt provenance to the existing GitOps release flow.
  • Add standard telemetry, policy checks, evaluation execution, and cost labels.
  • Document escape hatches with owners, expiry dates, and review requirements.

Days 61–90: prove and productize

  • Run a production-like release and rollback.
  • Measure lead time, evidence completeness, operational quality, and unit cost.
  • Interview the platform consumers and remove unnecessary steps.
  • Publish the contract as versioned schemas, APIs, examples, and service-level expectations.

The platform becomes the organizational control surface

AI-native platform engineering is not a race to add GPUs and model registries to an internal portal. It is the work of extending a proven product contract across a more complex value stream.

The strongest platforms will preserve what already works: product thinking, self-service, golden paths, policy automation, and composable cloud-native interfaces. They will add the missing contracts for data, models, evaluations, agents, specialized compute, and cost. That approach avoids a new silo while giving teams a credible path from experimentation to governed production.

Sources

Veröffentlicht am

OpenTelemetry Fleet Management: Why OpAMP Belongs in the Enterprise Observability Control Plane

OpenTelemetry can standardize how an enterprise collects and exports telemetry, but standardization alone does not make the collection layer operable.

At small scale, teams can manage Collector configuration through deployment manifests, virtual machine tooling, or a handful of automation scripts. At enterprise scale, the fleet becomes heterogeneous: Kubernetes DaemonSets, centralized gateways, virtual machines, laptops, point-of-sale devices, edge systems, and embedded environments. Different teams deploy the agents, while a central observability group remains accountable for data quality and service reliability.

That creates a control-plane problem. The organization needs to know which agents exist, what they are running, whether their configuration is current, whether a rollout succeeded, and how to recover without losing telemetry. The Open Agent Management Protocol, or OpAMP, provides a vendor-neutral protocol for that management relationship.

The strategic point is bigger than remote configuration. OpAMP belongs in the enterprise observability control plane because telemetry collection is production infrastructure. It needs identity, desired state, health feedback, controlled rollout, auditability, and rollback just like any other critical fleet.

Telemetry standardization exposes the management gap

OpenTelemetry adoption often begins with a sensible objective: remove proprietary instrumentation and normalize traces, metrics, and logs around open standards. The Collector becomes a flexible processing and export layer between workloads and one or more observability backends.

Success creates a new operating challenge. Collector configurations diverge by environment and team. Components run different versions. Credentials rotate at different times. Pipelines fail silently or begin dropping data. A change that looks safe in a development cluster can overload a regional gateway or remove a critical security log source.

GitOps helps with Kubernetes-managed Collectors, but it does not automatically cover agents on virtual machines, workstations, edge locations, or devices. It also tells the platform what was declared, not necessarily what every agent loaded or whether the resulting pipeline is healthy.

An enterprise control plane must connect declared intent with runtime evidence across the entire fleet.

What OpAMP actually provides

The OpenTelemetry specification describes OpAMP as a network protocol for remotely managing large fleets of data collection agents. It is vendor-agnostic and supports communication between an OpAMP server and clients associated with managed agents.

Core capabilities include:

  • Reporting agent identity, description, version, capabilities, and health
  • Receiving and acknowledging remote configuration
  • Reporting effective configuration and configuration status
  • Reporting package or component inventory
  • Receiving package update offers where the implementation supports them
  • Establishing bidirectional management communication over WebSocket or HTTP

OpAMP is a protocol, not a complete fleet-management product. It does not decide who may approve a production configuration, how rollout rings are selected, what policy is acceptable, or how a failed change should be escalated. Those are control-plane responsibilities that an enterprise platform must implement around the protocol.

The specification is currently marked beta. Newer Collector management work, including an alpha OpAMP Gateway Extension discussed by the CNCF, is promising but should be treated according to its maturity. Protocol adoption and production rollout should be deliberately separated from assumptions about experimental components.

The observability control plane needs a clear contract

A useful control plane maintains two views of every managed agent.

Desired state describes what the organization intends: approved Collector version, component set, configuration bundle, certificates, export destinations, and rollout assignment.

Observed state describes what the agent reports: identity, capabilities, effective configuration, health, errors, version, and last successful communication.

The difference between these views is configuration drift. Drift is not automatically a failure. An agent may be offline, a rollout may be paused, or a local emergency override may be permitted. The control plane should classify the difference, assign an owner, and decide whether to reconcile, roll back, or escalate.

This is why OpAMP complements rather than replaces GitOps. Git remains the reviewable source of approved configuration. OpAMP provides a standardized delivery and feedback channel for agents that cannot all be managed through the same deployment mechanism.

A reference enterprise architecture

A practical architecture separates policy, rollout orchestration, and protocol transport.

  1. Configuration repository. Versioned Collector templates, component allow lists, routing policy, environment overlays, and rollout metadata are reviewed through pull requests.
  2. Build and validation service. Every bundle is parsed, semantically validated, policy-checked, and tested against representative telemetry before promotion.
  3. Fleet inventory. The platform records agent identity, owner, environment, workload class, capabilities, current version, desired version, and health.
  4. Rollout controller. A change is assigned to cohorts, advanced through rings, paused on thresholds, and linked to an immutable configuration revision.
  5. OpAMP server. The server communicates desired state to clients and receives acknowledgements and status. It should not become the only system of record for policy decisions.
  6. Managed agents. Collectors or supervisors authenticate to the control plane, apply supported changes, and report effective state and health.
  7. Control-plane observability. The management system emits its own metrics, logs, and traces to an independent path so a fleet failure remains visible.

This architecture keeps configuration governance in familiar enterprise workflows while using OpAMP for standardized fleet interaction.

Identity is the first security boundary

A remote management channel can change what telemetry is collected, where it is sent, and which components execute. It is therefore a high-value security boundary.

Each client needs a stable identity tied to an owner and expected environment. Transport encryption is necessary but not sufficient. The server must authorize what that identity may receive, which cohort it belongs to, and whether it can accept sensitive configuration.

Recommended controls include:

  • Mutual authentication with short-lived, automatically rotated credentials
  • Per-agent or narrowly scoped workload identities rather than shared fleet secrets
  • Authorization by tenant, environment, geography, and workload class
  • Signed or integrity-protected configuration artifacts
  • Strict separation between configuration authors, approvers, and rollout operators
  • Audit records linking every server instruction to a reviewed revision and actor
  • Egress restrictions so agents communicate only with approved management and telemetry endpoints
  • Safe local behavior when the management server is unavailable

The server also needs protection from compromised agents. Rate limits, message-size limits, tenant isolation, replay resistance, input validation, and anomaly detection should be part of the threat model.

Configuration rollout should look like progressive delivery

Collector configuration can affect the visibility of an entire production estate. Treating a fleet-wide change as a simple push is an operational risk.

A safer workflow uses progressive rollout rings:

  1. Validation. Parse the configuration, resolve components, verify endpoints, run policy checks, and exercise representative telemetry.
  2. Development cohort. Apply the revision to disposable or low-risk agents and verify configuration acknowledgement.
  3. Canary cohort. Select a small production group that represents important environments and traffic patterns.
  4. Regional or workload rings. Expand only while health, drop rate, queue pressure, and backend load remain within thresholds.
  5. Fleet completion. Record coverage and identify offline or incompatible agents as explicit exceptions.
  6. Rollback. Restore the last known-good revision automatically when defined safety conditions fail.

A rollout is not successful because the server sent a configuration. It is successful when the intended agents report the expected effective state and the telemetry pipeline remains healthy.

Observe the observability fleet

The Collector layer is part of the monitoring system, so its management telemetry must not disappear into the same failure domain.

Track at least:

  • Active, offline, unknown, and quarantined agents
  • Desired-versus-effective configuration drift
  • Configuration acknowledgement and failure rates
  • Rollout duration and rollback frequency
  • Agent version and component-version distribution
  • Telemetry receive, drop, retry, queue, and export-failure rates
  • Credential age and failed authentication attempts
  • Management-channel latency and reconnect rate
  • Coverage by business service, environment, and data type

Business-level objectives matter as well. How quickly can the organization deploy a new security log source? How long does it take to revoke a compromised exporter credential? What percentage of critical services has a healthy, policy-compliant collection path?

An operating model for shared ownership

Fleet management spans organizational boundaries. Clear ownership prevents the control plane from becoming either an unresponsive central bottleneck or an uncontrolled self-service system.

  • The observability platform team owns the protocol service, supported agent profiles, configuration schemas, rollout automation, and service-level objectives.
  • Security owns control objectives, management-plane threat modeling, credential requirements, and sensitive destination policy.
  • Service and infrastructure teams own agent coverage, local dependencies, and declared business criticality.
  • Backend owners publish capacity constraints and compatibility requirements.
  • A change advisory model is encoded through risk tiers, automated evidence, and approval rules rather than a universal manual meeting.

Teams should be able to request supported pipelines and processors through a controlled interface. They should not need permission for every low-risk change, but they also should not be able to redirect enterprise telemetry to an arbitrary endpoint.

A staged adoption plan

Phase 1: establish inventory and evidence

  • Enumerate Collector deployments and other managed agents.
  • Assign ownership, environment, and criticality.
  • Define a small set of supported configurations and component versions.
  • Measure current drift, rollout time, and blind spots before adding remote control.

Phase 2: introduce OpAMP in read-oriented mode

  • Connect a non-critical cohort.
  • Collect agent descriptions, versions, effective configuration, and health.
  • Validate identity and tenant boundaries.
  • Compare observed state with the Git-approved desired state.

Phase 3: controlled configuration delivery

  • Enable remote configuration for one standardized agent profile.
  • Use signed revisions, canary rings, automated thresholds, and rollback.
  • Exercise server outage, invalid configuration, expired credentials, and incompatible-agent scenarios.

Phase 4: expand deliberately

  • Add heterogeneous environments and additional agent capabilities.
  • Integrate package updates only after configuration delivery is reliable.
  • Publish service objectives and an exception process.
  • Keep experimental extensions behind explicit maturity and risk gates.

Standard telemetry needs standard operations

OpenTelemetry solves an important portability problem, but enterprises also need a portable way to operate the collection fleet. OpAMP creates the protocol foundation for that control plane.

The durable design is not a central server that can push arbitrary files. It is a governed system that connects reviewed intent to agent identity, progressive rollout, effective state, health evidence, and safe recovery. Organizations that build those capabilities can scale OpenTelemetry without replacing proprietary telemetry agents with a new collection layer that is open but operationally opaque.

Sources

Veröffentlicht am

GitOps for AI Agents: Why Prompts, Tools, and Policies Belong in Your Platform Repository

AI agents are increasingly moving from experiments into production workflows. They can inspect systems, call tools, change infrastructure, open pull requests, and trigger operational actions. Yet many teams still manage the most important parts of an agent—its system prompt, tool permissions, output contract, and safety rules—as scattered text in notebooks, environment variables, or application code.

That is not just inconvenient. It is a governance problem.

If agent configuration influences production behavior, it should be managed like any other form of production configuration: declarative, versioned, reviewed, testable, and reversible. This is where GitOps becomes relevant—not as another fashionable label, but as a practical operating model for agentic systems.

Agent configuration is production behavior

For a conventional service, teams already treat deployment manifests, network policies, resource limits, and feature flags as controlled artifacts. An AI agent adds another behavioral layer:

  • the system prompt defines role, boundaries, and decision priorities;
  • the tool list determines which actions the agent can perform;
  • the output schema defines what downstream systems may trust;
  • policy bundles decide which actions are allowed, denied, or escalated;
  • model and routing settings affect cost, latency, and risk;
  • confidence and blast-radius thresholds determine when a human must intervene.

A change to any of these elements can alter production outcomes without changing a single line of traditional application code. Treating them as informal configuration creates an audit gap: teams may know which container image ran, but not which instructions or tool permissions shaped the agent’s decision.

What GitOps adds

The OpenGitOps principles describe desired state as declarative, versioned and immutable, automatically pulled, and continuously reconciled. Applied to agents, these principles create a clear chain from intent to runtime behavior.

A practical model looks like this:

  1. Agent configuration is stored in Git as structured data.
  2. A pull request shows the exact behavioral change.
  3. Automated checks validate schemas, policies, permissions, and evaluation results.
  4. Reviewers approve the change based on ownership and risk.
  5. A GitOps controller reconciles the approved state into the runtime platform.
  6. Telemetry confirms which version is active and how it behaves.
  7. A rollback restores the last known-good configuration when required.

This is already being applied in real cloud-native agent platforms. In a CNCF case study from Orange Innovation, each agent’s system prompt, tool list, and output schema is represented as a Kubernetes Custom Resource and reconciled from Git through Argo CD. Safety policies live in the same repository, making promotion code-reviewed, auditable, and reversible.

What should live in Git?

The goal is not to put every piece of runtime context into a repository. Git should contain the stable desired state that governs the agent.

Good candidates

  • system prompts and instruction templates;
  • allowed and denied tool definitions;
  • input and output schemas;
  • policy-as-code bundles;
  • model selection and fallback rules;
  • human-approval thresholds;
  • resource limits and deployment settings;
  • evaluation datasets and acceptance thresholds;
  • ownership metadata and escalation routes.

What should not live in Git?

  • API keys, tokens, and credentials;
  • personal or customer-sensitive conversation data;
  • short-lived runtime context;
  • unfiltered model traces containing confidential data;
  • mutable operational state that belongs in a database or event stream.

Secrets should be referenced through a secret-management system. Dynamic context should be retrieved through controlled tools with explicit identity, authorization, and audit trails.

An illustrative Kubernetes resource

Kubernetes Custom Resources provide one possible way to model agent desired state. The following example is illustrative rather than a proposed standard:

apiVersion: agents.platform.it-stud.io/v1alpha1
kind: AgentConfiguration
metadata:
  name: incident-reviewer
spec:
  promptRef: prompts/incident-reviewer-v12
  modelPolicy:
    primary: approved-enterprise-model
    fallback: approved-low-latency-model
  tools:
    allow:
      - read-observability-data
      - create-incident-ticket
    deny:
      - execute-production-change
  outputSchemaRef: schemas/incident-review-v3.json
  policyBundleRef: policies/soc-reviewer-v8
  humanApproval:
    requiredFor:
      - customer-facing-assets
      - identity-systems
      - actions-above-blast-radius-threshold

The value is not the YAML itself. The value is that the desired behavior becomes visible, reviewable, and reconcilable. A platform controller can translate this resource into runtime configuration while policy engines validate what teams are allowed to change.

The pull request becomes a governance control

A prompt review should not be treated like a copy-editing exercise. It is closer to reviewing infrastructure or authorization policy.

Different changes need different reviewers:

  • domain owners review whether instructions reflect the intended business process;
  • platform teams review runtime, deployment, and operational impact;
  • security teams review tool permissions, policy rules, identity, and blast radius;
  • AI engineers review model behavior, schemas, and evaluation results.

Branch protection and CODEOWNERS can turn this responsibility model into an enforceable workflow. A tool-permission change may require security approval, while a wording clarification within an existing boundary may only require the domain owner.

CI must test behavior, not just syntax

Schema validation is necessary but insufficient. An agent configuration can be valid YAML and still create unsafe or ineffective behavior.

A useful CI pipeline should combine:

  • schema and policy validation;
  • checks for forbidden tools or excessive permissions;
  • prompt-injection and adversarial test cases;
  • regression evaluations against representative scenarios;
  • cost and latency budgets;
  • output-schema conformance;
  • evidence that required human escalation still occurs.

The result should be an evaluation report attached to the pull request. Reviewers then see not only what changed, but how the agent’s measured behavior changed.

Deployment needs progressive delivery

GitOps makes rollback possible, but production agent changes should still be introduced gradually. A prompt or policy update can pass offline evaluations and fail under real operational conditions.

Platform teams can apply familiar delivery patterns:

  • shadow mode, where the new version makes decisions without executing them;
  • canary rollout to a limited workload or user group;
  • automatic rollback on quality, safety, latency, or cost regression;
  • version labels in traces so behavior can be tied to the exact Git revision;
  • human approval for changes that expand tool access or blast radius.

This is where agent operations begin to look less like prompt experimentation and more like mature platform engineering.

A practical operating model

Teams do not need a new organizational silo for every agent. They need clear contracts between existing responsibilities.

  • Domain teams own desired outcomes and business constraints.
  • AI engineering owns agent contracts, evaluations, and model behavior.
  • Platform engineering owns the runtime, GitOps reconciliation, observability, and deployment controls.
  • Security and risk own policy requirements, privileged actions, and evidence.

Machine-readable contracts—schemas, policies, Custom Resources, and evaluation thresholds—reduce coordination overhead. Teams can evolve their area without relying on undocumented meetings or hidden configuration.

A 30-day starting plan

  1. Inventory: identify production agents and locate their prompts, tools, policies, and schemas.
  2. Structure: move stable behavioral configuration into a versioned repository without migrating secrets or sensitive runtime data.
  3. Protect: add CODEOWNERS, branch protection, and approval requirements for high-risk fields.
  4. Validate: introduce schema checks, policy tests, and a small regression evaluation suite.
  5. Reconcile: automate deployment through an existing GitOps controller or equivalent reconciliation process.
  6. Observe: attach configuration version, model version, tool calls, cost, latency, and escalation outcomes to telemetry.
  7. Roll back: test restoration of the last known-good configuration before the first production incident.

Conclusion

AI agents should not be governed through scattered prompts and tribal knowledge. The configuration that shapes their behavior belongs in the same disciplined operating model used for other production systems.

GitOps provides a practical foundation: declared intent, version history, peer review, automated validation, continuous reconciliation, and fast rollback. Combined with policy-as-code, behavioral evaluations, progressive delivery, and human approval boundaries, it gives platform teams a credible way to scale agentic systems without losing control.

The core principle is simple: if a configuration change can alter what an agent is allowed to decide or do, it deserves the same engineering rigor as a production code change.

Sources and further reading

Veröffentlicht am

llm-d: Kubernetes-Native Distributed LLM Inference and CNCF’s Answer to the AI Inference Gap

For the last few years, Kubernetes has been the default control plane for almost everything except one increasingly important workload: large language model inference. Teams that adopted Kubernetes for microservices, batch jobs, and stateful systems often ran their GenAI inference on bespoke setups, vendor platforms, or hand-tuned vLLM deployments glued together with custom scripts. That gap is now closing.

At KubeCon EU 2026, the CNCF accepted llm-d into its Sandbox. Backed by Red Hat, Google, IBM, CoreWeave, and NVIDIA, llm-d is a purpose-built, Kubernetes-native framework for distributed LLM inference. It is not another model server competing with vLLM. It is the orchestration and serving layer that turns a fleet of accelerators into a coherent, scalable inference platform on Kubernetes.

This matters because the economics and operational reality of LLM inference are fundamentally different from traditional web services. A single large model can exceed the memory of one GPU. Request patterns are bursty. The compute profile of generating the first token is completely different from generating the rest. Treating inference like a stateless HTTP service leaves enormous performance and cost on the table. llm-d is the CNCF community’s answer to that problem.

The Inference Gap: Why Kubernetes Needed a Native Answer

According to the CNCF’s 2026 survey, around 66% of organizations now run GenAI inference on Kubernetes. That adoption happened faster than the tooling matured. Most teams stitched together their own stack: a model server such as vLLM, a custom Deployment or StatefulSet, an Ingress or Gateway in front, a homegrown autoscaler keyed off queue depth, and a lot of YAML.

This works until it doesn’t. The problems show up at scale:

  • Model size vs. accelerator memory: Large models must be sharded across multiple GPUs or nodes, which standard Deployments do not coordinate well.
  • Prefill vs. decode imbalance: The compute-heavy prefill phase and the memory-bandwidth-bound decode phase compete for the same resources when colocated.
  • Scheduling fragility: Multi-GPU inference pods need all their resources at once. Partial scheduling wastes expensive accelerators.
  • Routing blindness: Standard HTTP load balancing does not understand KV-cache locality, model affinity, or queue depth.
  • Cost opacity: Without per-accelerator metrics, FinOps for inference is guesswork.

llm-d addresses these as first-class concerns rather than afterthoughts. It aligns with the emerging Kubernetes AI Requirements (KARs) and positions itself as the reference inference stack for the cloud-native ecosystem.

Prefill/Decode Disaggregation: The Core Idea

The single most important architectural concept in llm-d is the disaggregation of the prefill and decode phases of inference.

When an LLM processes a request, it first reads the entire prompt and builds a key-value (KV) cache. This is the prefill phase, and it is compute-bound: it benefits from raw GPU throughput. Then the model generates tokens one at a time, each step depending on the KV cache. This is the decode phase, and it is largely memory-bandwidth-bound and latency-sensitive.

When both phases run on the same GPU, they interfere. A long prompt being prefilled can stall token generation for other requests, hurting time-to-first-token and inter-token latency simultaneously. Disaggregation separates these phases onto different pools of accelerators that can be scaled and tuned independently.

Phase Characteristic Bottleneck Scaling Strategy
Prefill Processes the full prompt, builds KV cache Compute (FLOPs) Scale for throughput on high-FLOP accelerators
Decode Generates tokens iteratively Memory bandwidth, latency Scale for concurrency and low latency

By splitting these across nodes, llm-d lets platform teams right-size each pool. You can throw high-throughput accelerators at prefill and optimize a separate pool for low-latency decode, transferring the KV cache between them. The result is better GPU utilization and more predictable latency under load.

How llm-d Fits the Kubernetes Stack

llm-d is not a monolith. It is designed to compose with the newest Kubernetes primitives for AI workloads. This is what makes it cloud-native rather than just another inference server wrapped in a container.

Dynamic Resource Allocation (DRA)

Kubernetes v1.36 matured Dynamic Resource Allocation, which replaces the aging device plugin framework for GPUs and accelerators. NVIDIA and Google have contributed CNCF-donated DRA drivers that act as the accelerator abstraction layer. llm-d uses DRA to request GPUs declaratively, with structured parameters the scheduler and autoscaler can actually understand. This means topology-aware allocation, partitionable devices, and cleaner multi-accelerator scheduling.

Gang Scheduling

Distributed inference needs all its pods at once. A model sharded across four GPUs is useless with three. llm-d relies on gang scheduling, available in the Kubernetes v1.36 workload-aware scheduling features, to ensure that a distributed inference deployment either gets all its resources or waits, rather than partially allocating and stranding expensive accelerators.

Kueue for Job Queuing

For multi-tenant inference pools and batch-style inference, llm-d integrates with Kueue. This brings quota management, fair sharing, and queuing across teams. Platform teams can define ClusterQueues that cap GPU budgets per team while keeping the pool efficiently shared.

Gateway API Inference Extension

Above llm-d sits the Gateway API Inference Extension (GIE), which provides intelligent routing for LLM traffic. Unlike standard HTTP routing, GIE understands inference-specific signals: model affinity, KV-cache locality, queue depth, and load. It routes requests to the right pool and the right replica, which is essential once prefill and decode are disaggregated.

llm-d vs. vLLM, KServe, and Ray Serve

A common point of confusion is where llm-d sits relative to existing tools. It does not replace all of them; it orchestrates and complements them.

Tool Primary Role When to Use
vLLM High-performance model server / inference engine As the underlying engine, often used by llm-d itself
KServe General model serving framework on Kubernetes Mixed model types, classic ML plus LLMs, standardized CRDs
Ray Serve Python-native distributed serving Teams already invested in the Ray ecosystem
llm-d Kubernetes-native distributed LLM inference orchestration Large-scale LLM inference with prefill/decode disaggregation and multi-node scaling

The practical mental model: vLLM is the engine, llm-d is the distributed serving and orchestration layer that runs engines like vLLM across a fleet, and the Gateway API Inference Extension is the smart front door. KServe and Ray Serve remain valid choices, especially for mixed workloads, but llm-d is purpose-built for the specific challenges of large-scale LLM inference on Kubernetes.

Observability and Cost: Making Inference Accountable

One of the most underrated aspects of running inference at scale is knowing what it costs and how it performs per accelerator. llm-d treats observability as a requirement, aligning with AI Conformance expectations.

Key signals platform teams should capture:

  • Time-to-first-token (TTFT): Dominated by prefill and queueing; the primary latency SLO for interactive use.
  • Inter-token latency: Reflects decode performance and concurrency pressure.
  • Per-accelerator utilization: GPU compute and memory usage, exported via Prometheus.
  • Queue depth and batch size: Drive autoscaling decisions far better than CPU metrics.
  • Throughput (tokens/sec): The real unit of inference work.

With OpenTelemetry and Prometheus wired in, these metrics feed both autoscaling and FinOps. The cost question llm-d helps answer is concrete: is it cheaper to run distributed inference across several smaller accelerators with disaggregation, or to consolidate on fewer large GPU nodes? The answer depends on model size, request mix, and latency targets, but for the first time the data to decide is available natively.

Security and Multi-Tenancy

Inference pools are expensive shared resources, which makes isolation important. llm-d supports multi-tenant inference pools with workload isolation and RBAC. Combined with Kueue quotas and namespace-scoped policies, platform teams can let multiple AI teams share a GPU fleet without one team starving another or accessing another team’s models. This is the same multi-tenancy discipline platform teams already apply to compute, extended to accelerators.

A Production Readiness Checklist

llm-d is in the CNCF Sandbox, which means it is early. Sandbox status signals direction and community backing, not production maturity. Platform teams evaluating it should be deliberate.

  1. Validate your Kubernetes version. You need v1.36-era features: mature DRA, gang scheduling, and workload-aware scheduling. Confirm your managed provider (EKS/GKE/AKS) exposes the required feature gates and DRA drivers.
  2. Confirm accelerator drivers. Ensure the NVIDIA or Google DRA drivers are installed and supported on your node pools.
  3. Start with a single model and pool. Prove the basic serving path before introducing prefill/decode disaggregation across nodes.
  4. Wire observability first. Export TTFT, inter-token latency, throughput, and per-GPU metrics to Prometheus before scaling up.
  5. Introduce the Gateway API Inference Extension early. Smart routing is what makes disaggregation pay off.
  6. Layer in Kueue for multi-tenancy. Define GPU quotas per team before opening the pool to multiple consumers.
  7. Run a cost comparison. Benchmark distributed inference against your current setup with real traffic, not synthetic load.
  8. Plan for sandbox churn. APIs may change. Pin versions, track releases, and avoid hard-coupling your platform to unstable interfaces.

Why This Is a Platform Engineering Story

It is tempting to file llm-d under „AI infrastructure“ and leave it to ML teams. That would be a mistake. The whole point of llm-d is to make distributed LLM inference a self-service IDP primitive. Instead of every AI team building bespoke serving stacks, the platform team offers inference as a paved road: declare a model, a pool, and an SLO, and the platform handles GPU allocation, scheduling, routing, scaling, and observability.

This is the same shift platform engineering brought to application deployment, now applied to inference. The golden path for an AI team becomes a governed, observable, cost-aware inference service rather than a pile of custom YAML and tribal knowledge.

The Bottom Line

llm-d represents the cloud-native ecosystem catching up to where AI workloads actually are. By disaggregating prefill and decode, integrating with DRA, gang scheduling, Kueue, and the Gateway API Inference Extension, and treating observability and multi-tenancy as requirements, it offers a coherent answer to the inference gap on Kubernetes.

It is early, and Sandbox status means platform teams should pilot rather than bet the farm. But the backing from Red Hat, Google, IBM, CoreWeave, and NVIDIA, plus alignment with the Kubernetes AI Requirements, makes llm-d the most credible candidate to become the standard distributed inference layer for Kubernetes. For platform teams whose organizations are already running GenAI on Kubernetes, now is the time to start evaluating it, building the observability foundation, and planning the golden path for inference-as-a-service.

Veröffentlicht am

Internal Developer Portals 2.0: How AI Copilots Inside Backstage and Port Are Transforming Developer Self-Service

Internal Developer Portals have spent the past three years earning their place in the platform engineering stack. Backstage — now a CNCF Graduated project — established the blueprint: a service catalog, software templates, TechDocs, and a plugin ecosystem exceeding 900 integrations. For many organizations, that was enough. But static catalogs have hit a ceiling. Developers still context-switch between Backstage, Slack, their IDE, and a dozen dashboards to scaffold a service, request infrastructure, or troubleshoot an incident. The portal that was supposed to unify developer experience became just another tab.

2025 and 2026 have introduced a different paradigm: AI copilots embedded directly inside IDPs. Not chatbots bolted onto the side, but intelligent agents that understand your service catalog, your golden paths, and your organizational policies — and let developers interact with infrastructure through natural language instead of form-driven UIs. This is Internal Developer Portals 2.0, and it changes the economics of platform engineering.

From Catalog-Centric to Action-Centric Portals

The first generation of IDPs was catalog-centric. You browsed a list of services, looked up ownership, maybe triggered a pre-built template. The developer experience was better than nothing, but it still required knowing where to click and which template to use. For a senior engineer who helped build the portal, that was fine. For a new hire on day three, it was another maze.

Action-centric IDPs flip the model. Instead of navigating a catalog hierarchy, a developer types:

"Deploy my payment-service to staging with the new database migration"

The AI copilot inside the portal understands the intent, resolves the service from the catalog, identifies the correct deployment pipeline, checks RBAC policies, and either executes or presents a confirmation step. The catalog is still there — it’s the knowledge backbone — but the interaction layer has fundamentally changed.

This isn’t speculative. Port has shipped an AI assistant that queries its internal software catalog and executes self-service actions through natural language. Cortex integrates LLM-driven recommendations directly into its scorecards. Humanitec has taken an API-first approach that makes AI orchestration a first-class integration pattern. Even Backstage itself is seeing community plugins that expose catalog data to AI agents via standardized protocols.

The Knowledge Graph Advantage

What makes IDP-embedded AI fundamentally different from a generic ChatGPT wrapper is context. An Internal Developer Portal already holds a rich knowledge graph:

  • Service dependencies: which services call which, what databases they use, what message queues connect them
  • Team ownership: who owns what, who’s on-call, escalation paths
  • Runbooks and documentation: operational playbooks indexed per service
  • Deployment history: what was deployed when, by whom, with what configuration
  • Scorecards: production readiness, security posture, cost allocation

When an AI copilot has access to this graph, its responses move from generic to surgical. Ask it "Why is checkout-service latency spiking?" and it can correlate recent deployments, check the dependency graph for upstream changes, pull relevant runbooks, and suggest specific remediation steps — all without the developer leaving the portal.

Compare this to ChatOps bots in Slack that operate with minimal context, or IDE-integrated copilots that understand your code but not your infrastructure. The IDP sits at the intersection of code, infrastructure, and organizational knowledge. That’s where AI adds the most leverage.

MCP Servers: The Bridge Between AI Agents and IDP APIs

The technical glue making this possible is increasingly the Model Context Protocol (MCP). Originally open-sourced by Anthropic in 2024 and now seeing broad adoption, MCP provides a standardized interface for AI agents to discover and invoke tools — including IDP APIs.

An MCP server wrapping your Backstage or Port API exposes capabilities like:

  • Querying the service catalog (list services, get owners, check dependencies)
  • Triggering software templates (scaffold a new microservice, provision a database)
  • Reading and updating scorecards
  • Executing self-service actions (deploy, rollback, scale)
  • Fetching TechDocs and runbooks for context

This decouples the AI layer from the IDP implementation. Your platform team maintains the MCP server as a thin adapter. The AI agent — whether it’s embedded in the portal UI, accessible via Slack, or running inside an IDE — connects through the same protocol. You get a single source of truth for what actions are available and what permissions govern them.

For teams already running Backstage, this is particularly powerful. The existing plugin ecosystem handles data aggregation; an MCP server adds an AI-native interaction layer on top without replacing the portal itself.

Scorecards Meet AI Analysis

Scorecards have been one of the quiet successes of the IDP movement. Tools like Backstage (via the Scorecards plugin), Port, and Cortex let platform teams define maturity criteria — production readiness, security compliance, documentation coverage, cost efficiency — and track every service against them.

AI transforms scorecards from passive dashboards into active recommendation engines:

  • Service maturity gaps: „Your order-service scores 62% on production readiness. Adding health check endpoints and configuring pod disruption budgets would bring it to 85%.“
  • Security posture: „Three services in the payments domain are running container images older than 90 days. Here are the specific CVEs affecting them.“
  • Cost optimization: „Based on CPU utilization patterns over the last 30 days, analytics-worker is over-provisioned by 3x. Recommended resource requests: 200m CPU, 256Mi memory.“

The shift is from „here’s your score“ to „here’s what to do about it.“ When combined with self-service actions, the AI can even generate the pull request to implement the recommendation — turning insight into action in a single interaction.

Dynamic Golden Paths: AI-Generated Templates

Golden paths — the blessed, paved roads for common developer tasks — have traditionally been static. Your platform team creates a service template, a database provisioning workflow, a CI/CD pipeline configuration. Developers pick from the menu.

AI-powered IDPs make golden paths dynamic. Instead of maintaining 15 slightly different service templates for different tech stacks and deployment targets, you maintain a smaller set of composable building blocks. The AI assembles them based on the developer’s intent:

"I need a new Go microservice with a PostgreSQL database, deployed to our EU region, with PII data handling compliance"

The copilot generates a tailored template that includes the correct Helm values for the EU cluster, enables encryption-at-rest annotations for PII compliance, configures the appropriate network policies, and sets up the CI/CD pipeline with the required security scanning stages. The golden path isn’t a fixed road anymore — it’s a GPS that calculates the route based on where you’re going.

This has real implications for template maintenance. Platform teams spend significant effort keeping templates current across Kubernetes versions, policy changes, and infrastructure updates. AI-generated templates that compose from maintained primitives reduce that burden substantially.

Incident Response: The Killer Use Case

If there’s a single scenario where IDP-embedded AI proves its ROI overnight, it’s incident response. Consider the typical flow today:

  1. Alert fires in PagerDuty or Opsgenie
  2. On-call engineer opens the monitoring dashboard
  3. Checks recent deployments in the CI/CD tool
  4. Looks up service ownership in the IDP
  5. Searches for relevant runbooks
  6. Correlates with dependency graph to identify blast radius
  7. Begins remediation

Steps 2 through 6 are pure context gathering — and they happen under pressure at 3 AM. An AI agent inside the IDP can perform all of them in seconds:

  • Correlate the alert with the service catalog entry
  • Identify recent changes (deployments, config updates, dependency upgrades)
  • Pull the relevant runbook and highlight the most likely remediation steps
  • Map the blast radius through the dependency graph
  • Suggest or auto-execute a rollback if the confidence is high enough

The on-call engineer still makes the decision, but the mean time to context drops from 15 minutes to 15 seconds. For organizations running hundreds of microservices, that’s not a nice-to-have — it’s a competitive advantage.

Developer Experience Metrics: Measuring What Matters

AI-powered IDPs also change how we measure developer experience. The DORA metrics (deployment frequency, lead time for changes, change failure rate, mean time to recovery) and the SPACE framework (satisfaction, performance, activity, communication, efficiency) are becoming first-class citizens in IDP dashboards.

The AI layer adds predictive and diagnostic capabilities:

  • Trend analysis: „Deployment frequency for the checkout team has dropped 30% over the past sprint. The primary bottleneck appears to be flaky integration tests in the payment-gateway pipeline.“
  • Correlation: „Teams using the v3 service template have 40% lower change failure rates than those on v2. Consider migrating remaining v2 services.“
  • Forecasting: „Based on current velocity, the platform migration will complete in Q3 — two weeks later than planned. The blocker is database schema migrations for three legacy services.“

This is where platform engineering ROI becomes measurable. When you can demonstrate that AI-assisted self-service reduces time-to-production for new services from five days to four hours, the investment case writes itself.

Backstage’s Plugin Ecosystem vs. AI-Native Platforms

The market is splitting into two camps, and platform teams need to understand the tradeoffs:

Dimension Backstage + AI Plugins AI-Native Platforms (Port, Cortex)
Flexibility 900+ plugins, infinite customization Fewer but deeper integrations
AI integration Community-driven, via MCP/plugins Built-in, first-class AI features
Maintenance burden High (self-hosted, plugin compatibility) Lower (SaaS, managed updates)
Data ownership Full control (self-hosted) Vendor-dependent
Time to value Weeks to months Days to weeks
Vendor lock-in Low (CNCF, open source) Moderate to high
Knowledge graph depth As deep as you build it Pre-built entity models

Neither approach is universally better. If your organization has strong platform engineering capacity and wants full control, Backstage with AI plugins and MCP servers gives you maximum flexibility. If you want faster time-to-value and your team is lean, an AI-native platform like Port gets you to production-grade IDP faster — at the cost of some flexibility and data sovereignty.

ChatOps vs. IDP-Embedded AI vs. IDE Copilots

It’s worth clarifying where IDP-embedded AI fits relative to other AI integration points:

  • ChatOps (Slack/Teams bots): Good for notifications and simple commands. Limited context about your infrastructure. Works well for quick queries but struggles with complex multi-step workflows.
  • IDE-integrated copilots (GitHub Copilot, Cursor): Excellent for code generation. No awareness of your deployment topology, service catalog, or organizational policies. Wrong tool for infrastructure tasks.
  • IDP-embedded AI: Sits at the intersection of organizational knowledge, infrastructure state, and developer workflows. Best for self-service actions, incident response, and cross-cutting concerns that span multiple services.

The ideal setup uses all three — but the IDP is the orchestration layer. Your Slack bot calls the IDP’s AI capabilities through MCP. Your IDE copilot references the service catalog for context. The IDP is the brain; everything else is an interface.

Multi-Tenancy, RBAC, and Governance

Here’s where many early AI-in-IDP implementations fall short: governance. When an AI agent can trigger deployments, modify infrastructure, or scaffold services, you need the same (or stricter) access controls as your existing self-service workflows.

Critical requirements:

  • RBAC for AI actions: The AI copilot should inherit the requesting user’s permissions, not operate with elevated privileges
  • Audit trails: Every AI-initiated action must be logged with the full context — who asked, what was requested, what was executed, what was the outcome
  • Approval gates: Destructive or high-risk actions (production deployments, database migrations, security policy changes) should require human approval, even when AI-initiated
  • Multi-tenancy: In organizations with multiple teams sharing an IDP, AI actions must respect tenant boundaries. Team A’s copilot cannot access Team B’s secrets or deploy to Team B’s namespaces
  • Rate limiting: Prevent AI agents from executing runaway loops of infrastructure changes

Without these controls, you’re trading developer friction for security risk — not a trade worth making.

The Risks: What Can Go Wrong

Let’s be direct about the failure modes:

  • Hallucinated infrastructure configurations: An AI that generates a Kubernetes manifest with incorrect resource limits, missing security contexts, or wrong network policies can cause outages. Every AI-generated configuration must pass through the same validation pipelines (OPA/Kyverno, CI checks) as human-authored configs.
  • Insufficient audit trails: If an AI agent makes a change and the audit log only shows „AI modified resource X,“ you’ve lost forensic capability. Log the full chain: user prompt → AI interpretation → action taken → result.
  • Shadow IT acceleration: If self-service becomes too easy, developers spin up resources without proper tagging, cost allocation, or lifecycle management. AI-powered IDPs need to enforce organizational policies at the point of creation, not after the fact.
  • Over-reliance on AI recommendations: Scorecard suggestions and incident response playbooks should augment human judgment, not replace it. Build a culture where AI recommendations are validated, not blindly accepted.

Getting Started: A Practical Roadmap

If you’re running Backstage today and want to add AI capabilities, here’s a pragmatic path:

  1. Start with read-only: Build an MCP server that exposes your service catalog, scorecards, and documentation to an AI agent. Let developers query the catalog through natural language. Zero risk, immediate value.
  2. Add scorecard analysis: Connect the AI to your scorecard data and let it generate improvement recommendations. Still read-only, but now actively useful.
  3. Enable template generation: Allow the AI to compose software templates based on developer intent. Route the output through your existing PR review process.
  4. Introduce action execution: Wire up deployment, scaling, and provisioning actions with approval gates. Start with non-production environments.
  5. Extend to incident response: Connect alerting systems and let the AI perform context gathering and remediation suggestions during incidents.

Each step builds on the previous one, and each can be rolled back independently. The key is maintaining human oversight throughout — AI copilots augment your platform team, they don’t replace it.

The Bottom Line

Internal Developer Portals 2.0 aren’t about replacing Backstage or rebuilding your IDP from scratch. They’re about adding an intelligence layer that transforms the portal from a passive catalog into an active assistant. The service catalog becomes a knowledge graph. Templates become dynamic. Scorecards become recommendation engines. Incident response becomes proactive.

The organizations that get this right will see measurable improvements in developer productivity, onboarding speed, and operational resilience. The ones that don’t will keep maintaining static portals that developers tolerate rather than love.

The technology is ready. The protocols (MCP) are standardizing. The question isn’t whether AI belongs in your IDP — it’s how quickly you can integrate it without compromising the governance that makes your platform trustworthy.

Veröffentlicht am

The .de DNSSEC Meltdown: What Platform Teams Can Learn from Germany’s TLD Outage

TL;DR — On May 5 2026, DENIC pushed broken DNSSEC signatures into the .de zone. Because DNSSEC validation is a strict chain-of-trust model, every validating resolver on the planet began returning SERVFAIL for all .de domains. Millions of websites, APIs, and mail servers went dark. Resolvers that had deployed Serve-Stale (RFC 8767) and Negative Trust Anchors (RFC 7646) recovered within minutes; everyone else waited hours. This article breaks down the incident, the mitigation patterns, and the concrete steps platform teams should take so a single TLD mistake doesn’t take down their stack.

What Happened on May 5, 2026

At approximately 10:42 UTC on Monday, May 5, monitoring dashboards across Europe lit up. DNS resolution for .de domains — one of the world’s largest country-code TLDs, consistently ranking in the Top 5 at Cloudflare Radar — started failing en masse. The root cause: DENIC, the registry operator for .de, had published DNSSEC signatures that did not match the zone’s active Zone Signing Key (ZSK).

The timing was no coincidence. The faulty signatures surfaced during a scheduled ZSK rotation — one of the most operationally sensitive windows in DNSSEC key management. A misconfiguration in the signing pipeline meant that the new signatures were generated with a key that validating resolvers could not verify against the published DS records in the root zone. The result was catastrophic: the entire .de chain of trust was broken.

Within minutes, every DNSSEC-validating resolver worldwide — including Cloudflare’s 1.1.1.1, Google’s 8.8.8.8, and Quad9’s 9.9.9.9 — began returning SERVFAIL for queries to .de domains. Non-validating resolvers continued to work, which created a confusing split-brain situation where some users could reach German websites and others couldn’t, depending on their configured resolver.

The DNSSEC Chain of Trust: One Link Breaks, Everything Falls

To understand why a single registry mistake can have such a massive blast radius, you need to understand how DNSSEC validation works.

DNSSEC adds cryptographic signatures to DNS records. Resolvers verify these signatures by walking a chain of trust from the root zone (.) down through the TLD (.de) to the individual domain (example.de). Each level delegates trust to the next via DS (Delegation Signer) records. If any link in this chain produces an invalid signature, a validating resolver must return SERVFAIL. That’s not a bug — it’s the design. DNSSEC was built to prevent cache poisoning, and treating unverifiable answers as failures is the entire point.

The double-edged nature of this design becomes painfully clear during operator errors at the TLD level. When DENIC’s signatures broke, it wasn’t just one domain that failed — it was every single .de domain, regardless of whether the individual domain owner had done everything right. The TLD is a single point of cryptographic failure for all domains beneath it.

ZSK/KSK Rotation: The Critical Window

DNSSEC uses two types of keys: the Key Signing Key (KSK), which signs the DNSKEY RRset, and the Zone Signing Key (ZSK), which signs the actual zone data. ZSK rotations happen more frequently and involve a carefully choreographed dance: pre-publish the new key, wait for caches to expire, sign with the new key, remove the old one. Get any step wrong — wrong timing, wrong key reference, stale DS record — and you shatter the chain of trust. This is exactly what happened with .de.

How Major Resolvers Responded

The incident provided a real-world stress test for two mitigation techniques that the DNS community has been advocating for years: Serve-Stale and Negative Trust Anchors.

Serve-Stale (RFC 8767)

Serve-Stale allows a resolver to return expired (stale) cached records instead of failing with SERVFAIL when it cannot fetch a fresh, valid answer from upstream. Cloudflare’s 1.1.1.1 had Serve-Stale enabled, and their detailed incident report showed that users hitting warm caches continued to get working answers for .de domains — stale data, but functional. For most use cases (websites, APIs, mail routing), a stale A or AAAA record from five minutes ago is infinitely better than SERVFAIL.

The limitation: Serve-Stale only works if the record was previously cached. Cold caches — new queries for domains the resolver hadn’t seen recently — still failed. And once stale TTLs expired (typically capped at 1–3 days depending on implementation), even warm caches would stop serving.

Negative Trust Anchors (RFC 7646)

Negative Trust Anchors (NTAs) are the emergency brake for DNSSEC. An NTA tells a resolver: „Stop validating DNSSEC for this specific domain or zone.“ When applied to .de, it effectively disables signature verification for the entire TLD, allowing queries to resolve normally — at the cost of losing DNSSEC protection.

Cloudflare, Google, and Quad9 all deployed NTAs for .de within the first hour of the incident. This was the fastest path to restoring service for end users. The NTAs were removed once DENIC republished correct signatures later that day.

The Third Option: Disabling DNSSEC Validation Entirely

Some smaller operators chose the nuclear option: disabling DNSSEC validation on their resolvers entirely. This restored service for all domains immediately but removed cryptographic protection for every zone, not just the broken one. This is the equivalent of disabling your firewall because one rule is misconfigured — it works, but the security implications are severe. NTAs are strictly preferable because they scope the trust bypass to the affected zone.

The Amplification Problem

DNS outages create a vicious feedback loop. When resolvers return SERVFAIL, clients retry — aggressively. Applications retry. Browsers retry. Stub resolvers retry. Monitoring systems fire off their own queries. Cloudflare reported a 10x spike in query volume for .de during the incident, as retry storms amplified the load on authoritative servers and resolvers alike.

This client-retry amplification is a well-known pattern in distributed systems, but it’s especially brutal in DNS because retries happen at multiple layers simultaneously. It delays recovery because even after the root cause is fixed, the query flood continues until retry backoffs settle.

Parallels to Prior TLD Outages

The .de incident wasn’t the first time a TLD’s DNSSEC misconfiguration caused widespread outages. In 2024, New Zealand’s .nz experienced a similar DNSSEC signing failure that took down domains across the country. Sweden’s .se has had its own DNSSEC-related incidents. Each time, the pattern is the same: a key management error at the TLD level cascades into a nationwide or zone-wide outage, and the community rediscovers that DNSSEC’s strict validation model trades availability for integrity.

The lesson keeps repeating because the operational complexity of DNSSEC key management is genuinely hard, and the failure mode is binary: it either validates or it doesn’t. There’s no graceful degradation built into the protocol itself.

Platform Engineering Lessons

If you’re running a platform team — especially one operating in the EU — the .de incident should be a wake-up call. DNS is deeply embedded in every layer of a modern cloud-native stack: ExternalDNS syncs records, cert-manager validates domain ownership via DNS-01 challenges, Ingress controllers rely on DNS routing, service meshes resolve endpoints. A DNS outage isn’t just „websites are down“ — it can break certificate issuance, deployment pipelines, service discovery, and monitoring.

1. Monitor DNSSEC Validation, Not Just Resolution

Most teams monitor whether DNS resolution works. Few monitor whether DNSSEC validation is healthy. Set up checks that specifically test DNSSEC signature validity for your critical domains and their parent zones. Tools like DNSViz, Zonemaster, and RIPE Atlas probes can automate this. Alert on validation failures before your users notice.

2. Implement a Multi-Resolver Strategy

Don’t depend on a single upstream resolver. Configure failover across multiple providers: Cloudflare (1.1.1.1), Google (8.8.8.8), Quad9 (9.9.9.9). Each operator has different NTA deployment speeds and Serve-Stale configurations. During the .de incident, the window between „Cloudflare deployed NTA“ and „smaller ISP resolvers deployed NTA“ was measured in hours. A multi-resolver setup lets you ride the fastest responder.

3. Deploy Serve-Stale in Your Own Resolvers

If you run local resolvers (CoreDNS, Unbound, BIND), enable Serve-Stale. In CoreDNS, this means configuring the cache plugin with serve_stale. In Unbound, set serve-expired: yes with appropriate serve-expired-ttl and serve-expired-client-timeout values. This single configuration change is your best passive defense against upstream DNSSEC failures.

# Unbound example
server:
    serve-expired: yes
    serve-expired-ttl: 86400
    serve-expired-client-timeout: 1800
# CoreDNS example
.:53 {
    forward . 1.1.1.1 8.8.8.8 9.9.9.9
    cache 3600 {
        serve_stale 86400
    }
}

4. Treat DNS as a Critical Dependency in Your Architecture

Map out every component in your stack that depends on DNS resolution. ExternalDNS, cert-manager (DNS-01 challenges), Ingress controllers, external API calls, webhook endpoints, OAuth/OIDC provider discovery — all of these break when DNS breaks. Document these dependencies and include DNS failure scenarios in your chaos engineering practice.

5. Build a DNS Incident Response Playbook

Your runbook should5 include:

  • Detection: Automated alerts for DNSSEC validation failures and elevated SERVFAIL rates
  • Triage: Is the issue local, resolver-level, or TLD-level? Use dig +dnssec and delv to isolate
  • Mitigation: Pre-approved steps to deploy NTAs on local resolvers, switch upstream resolvers, or enable Serve-Stale
  • Communication: Templates for status page updates that explain DNS issues to non-technical stakeholders
  • Recovery: Validation that DNSSEC signatures are correct before removing NTAs

6. NIS2 and DORA: DNS Resilience Is Now a Compliance Issue

For organizations operating in the EU, the NIS2 Directive and the Digital Operational Resilience Act (DORA) explicitly require resilience measures for critical infrastructure, including ICT supply chain risks. DNS is a foundational ICT service. A TLD-level outage that takes down your platform because you had no failover, no Serve-Stale, and no incident playbook is now a compliance gap, not just an operational one. Document your DNS resilience measures as part of your NIS2/DORA risk assessments.

The Bigger Picture

The .de DNSSEC meltdown highlights a fundamental tension in internet infrastructure: the systems designed to protect us (DNSSEC, certificate validation, strict security policies) can also become single points of failure when they break. The answer isn’t to disable security — it’s to build resilience layers that absorb the impact of failures without sacrificing protection during normal operations.

Serve-Stale and Negative Trust Anchors are exactly this kind of resilience layer. They don’t weaken DNSSEC; they give operators a controlled way to maintain availability while the underlying issue is fixed. Every platform team should have both in their toolkit.

Conclusion: Your DNS Is Only as Strong as Your Weakest Trust Anchor

The .de outage wasn’t caused by a sophisticated attack. It was a configuration error during routine key rotation — the kind of mistake that can happen to any registry, any operator, at any time. What separated the teams that weathered it from those that scrambled was preparation: multi-resolver setups, Serve-Stale configurations, DNSSEC monitoring, and tested incident playbooks.

Your action items for this week:

  1. Check if your resolvers have Serve-Stale enabled. If not, enable it today.
  2. Set up DNSSEC validation monitoring for your critical domains and their parent TLDs.
  3. Document your DNS dependencies and add DNS failure to your incident response playbook.
  4. Test a multi-resolver failover — don’t wait for the next TLD outage to find out if it works.

The next DNSSEC meltdown isn’t a matter of if — it’s a matter of which TLD and when. Be ready.