Kategorie: Platform Engineering

Veröffentlicht am

The .de DNSSEC Meltdown: What Platform Teams Can Learn from Germany’s TLD Outage

TL;DR — On May 5 2026, DENIC pushed broken DNSSEC signatures into the .de zone. Because DNSSEC validation is a strict chain-of-trust model, every validating resolver on the planet began returning SERVFAIL for all .de domains. Millions of websites, APIs, and mail servers went dark. Resolvers that had deployed Serve-Stale (RFC 8767) and Negative Trust Anchors (RFC 7646) recovered within minutes; everyone else waited hours. This article breaks down the incident, the mitigation patterns, and the concrete steps platform teams should take so a single TLD mistake doesn’t take down their stack.

What Happened on May 5, 2026

At approximately 10:42 UTC on Monday, May 5, monitoring dashboards across Europe lit up. DNS resolution for .de domains — one of the world’s largest country-code TLDs, consistently ranking in the Top 5 at Cloudflare Radar — started failing en masse. The root cause: DENIC, the registry operator for .de, had published DNSSEC signatures that did not match the zone’s active Zone Signing Key (ZSK).

The timing was no coincidence. The faulty signatures surfaced during a scheduled ZSK rotation — one of the most operationally sensitive windows in DNSSEC key management. A misconfiguration in the signing pipeline meant that the new signatures were generated with a key that validating resolvers could not verify against the published DS records in the root zone. The result was catastrophic: the entire .de chain of trust was broken.

Within minutes, every DNSSEC-validating resolver worldwide — including Cloudflare’s 1.1.1.1, Google’s 8.8.8.8, and Quad9’s 9.9.9.9 — began returning SERVFAIL for queries to .de domains. Non-validating resolvers continued to work, which created a confusing split-brain situation where some users could reach German websites and others couldn’t, depending on their configured resolver.

The DNSSEC Chain of Trust: One Link Breaks, Everything Falls

To understand why a single registry mistake can have such a massive blast radius, you need to understand how DNSSEC validation works.

DNSSEC adds cryptographic signatures to DNS records. Resolvers verify these signatures by walking a chain of trust from the root zone (.) down through the TLD (.de) to the individual domain (example.de). Each level delegates trust to the next via DS (Delegation Signer) records. If any link in this chain produces an invalid signature, a validating resolver must return SERVFAIL. That’s not a bug — it’s the design. DNSSEC was built to prevent cache poisoning, and treating unverifiable answers as failures is the entire point.

The double-edged nature of this design becomes painfully clear during operator errors at the TLD level. When DENIC’s signatures broke, it wasn’t just one domain that failed — it was every single .de domain, regardless of whether the individual domain owner had done everything right. The TLD is a single point of cryptographic failure for all domains beneath it.

ZSK/KSK Rotation: The Critical Window

DNSSEC uses two types of keys: the Key Signing Key (KSK), which signs the DNSKEY RRset, and the Zone Signing Key (ZSK), which signs the actual zone data. ZSK rotations happen more frequently and involve a carefully choreographed dance: pre-publish the new key, wait for caches to expire, sign with the new key, remove the old one. Get any step wrong — wrong timing, wrong key reference, stale DS record — and you shatter the chain of trust. This is exactly what happened with .de.

How Major Resolvers Responded

The incident provided a real-world stress test for two mitigation techniques that the DNS community has been advocating for years: Serve-Stale and Negative Trust Anchors.

Serve-Stale (RFC 8767)

Serve-Stale allows a resolver to return expired (stale) cached records instead of failing with SERVFAIL when it cannot fetch a fresh, valid answer from upstream. Cloudflare’s 1.1.1.1 had Serve-Stale enabled, and their detailed incident report showed that users hitting warm caches continued to get working answers for .de domains — stale data, but functional. For most use cases (websites, APIs, mail routing), a stale A or AAAA record from five minutes ago is infinitely better than SERVFAIL.

The limitation: Serve-Stale only works if the record was previously cached. Cold caches — new queries for domains the resolver hadn’t seen recently — still failed. And once stale TTLs expired (typically capped at 1–3 days depending on implementation), even warm caches would stop serving.

Negative Trust Anchors (RFC 7646)

Negative Trust Anchors (NTAs) are the emergency brake for DNSSEC. An NTA tells a resolver: „Stop validating DNSSEC for this specific domain or zone.“ When applied to .de, it effectively disables signature verification for the entire TLD, allowing queries to resolve normally — at the cost of losing DNSSEC protection.

Cloudflare, Google, and Quad9 all deployed NTAs for .de within the first hour of the incident. This was the fastest path to restoring service for end users. The NTAs were removed once DENIC republished correct signatures later that day.

The Third Option: Disabling DNSSEC Validation Entirely

Some smaller operators chose the nuclear option: disabling DNSSEC validation on their resolvers entirely. This restored service for all domains immediately but removed cryptographic protection for every zone, not just the broken one. This is the equivalent of disabling your firewall because one rule is misconfigured — it works, but the security implications are severe. NTAs are strictly preferable because they scope the trust bypass to the affected zone.

The Amplification Problem

DNS outages create a vicious feedback loop. When resolvers return SERVFAIL, clients retry — aggressively. Applications retry. Browsers retry. Stub resolvers retry. Monitoring systems fire off their own queries. Cloudflare reported a 10x spike in query volume for .de during the incident, as retry storms amplified the load on authoritative servers and resolvers alike.

This client-retry amplification is a well-known pattern in distributed systems, but it’s especially brutal in DNS because retries happen at multiple layers simultaneously. It delays recovery because even after the root cause is fixed, the query flood continues until retry backoffs settle.

Parallels to Prior TLD Outages

The .de incident wasn’t the first time a TLD’s DNSSEC misconfiguration caused widespread outages. In 2024, New Zealand’s .nz experienced a similar DNSSEC signing failure that took down domains across the country. Sweden’s .se has had its own DNSSEC-related incidents. Each time, the pattern is the same: a key management error at the TLD level cascades into a nationwide or zone-wide outage, and the community rediscovers that DNSSEC’s strict validation model trades availability for integrity.

The lesson keeps repeating because the operational complexity of DNSSEC key management is genuinely hard, and the failure mode is binary: it either validates or it doesn’t. There’s no graceful degradation built into the protocol itself.

Platform Engineering Lessons

If you’re running a platform team — especially one operating in the EU — the .de incident should be a wake-up call. DNS is deeply embedded in every layer of a modern cloud-native stack: ExternalDNS syncs records, cert-manager validates domain ownership via DNS-01 challenges, Ingress controllers rely on DNS routing, service meshes resolve endpoints. A DNS outage isn’t just „websites are down“ — it can break certificate issuance, deployment pipelines, service discovery, and monitoring.

1. Monitor DNSSEC Validation, Not Just Resolution

Most teams monitor whether DNS resolution works. Few monitor whether DNSSEC validation is healthy. Set up checks that specifically test DNSSEC signature validity for your critical domains and their parent zones. Tools like DNSViz, Zonemaster, and RIPE Atlas probes can automate this. Alert on validation failures before your users notice.

2. Implement a Multi-Resolver Strategy

Don’t depend on a single upstream resolver. Configure failover across multiple providers: Cloudflare (1.1.1.1), Google (8.8.8.8), Quad9 (9.9.9.9). Each operator has different NTA deployment speeds and Serve-Stale configurations. During the .de incident, the window between „Cloudflare deployed NTA“ and „smaller ISP resolvers deployed NTA“ was measured in hours. A multi-resolver setup lets you ride the fastest responder.

3. Deploy Serve-Stale in Your Own Resolvers

If you run local resolvers (CoreDNS, Unbound, BIND), enable Serve-Stale. In CoreDNS, this means configuring the cache plugin with serve_stale. In Unbound, set serve-expired: yes with appropriate serve-expired-ttl and serve-expired-client-timeout values. This single configuration change is your best passive defense against upstream DNSSEC failures.

# Unbound example
server:
    serve-expired: yes
    serve-expired-ttl: 86400
    serve-expired-client-timeout: 1800
# CoreDNS example
.:53 {
    forward . 1.1.1.1 8.8.8.8 9.9.9.9
    cache 3600 {
        serve_stale 86400
    }
}

4. Treat DNS as a Critical Dependency in Your Architecture

Map out every component in your stack that depends on DNS resolution. ExternalDNS, cert-manager (DNS-01 challenges), Ingress controllers, external API calls, webhook endpoints, OAuth/OIDC provider discovery — all of these break when DNS breaks. Document these dependencies and include DNS failure scenarios in your chaos engineering practice.

5. Build a DNS Incident Response Playbook

Your runbook should5 include:

  • Detection: Automated alerts for DNSSEC validation failures and elevated SERVFAIL rates
  • Triage: Is the issue local, resolver-level, or TLD-level? Use dig +dnssec and delv to isolate
  • Mitigation: Pre-approved steps to deploy NTAs on local resolvers, switch upstream resolvers, or enable Serve-Stale
  • Communication: Templates for status page updates that explain DNS issues to non-technical stakeholders
  • Recovery: Validation that DNSSEC signatures are correct before removing NTAs

6. NIS2 and DORA: DNS Resilience Is Now a Compliance Issue

For organizations operating in the EU, the NIS2 Directive and the Digital Operational Resilience Act (DORA) explicitly require resilience measures for critical infrastructure, including ICT supply chain risks. DNS is a foundational ICT service. A TLD-level outage that takes down your platform because you had no failover, no Serve-Stale, and no incident playbook is now a compliance gap, not just an operational one. Document your DNS resilience measures as part of your NIS2/DORA risk assessments.

The Bigger Picture

The .de DNSSEC meltdown highlights a fundamental tension in internet infrastructure: the systems designed to protect us (DNSSEC, certificate validation, strict security policies) can also become single points of failure when they break. The answer isn’t to disable security — it’s to build resilience layers that absorb the impact of failures without sacrificing protection during normal operations.

Serve-Stale and Negative Trust Anchors are exactly this kind of resilience layer. They don’t weaken DNSSEC; they give operators a controlled way to maintain availability while the underlying issue is fixed. Every platform team should have both in their toolkit.

Conclusion: Your DNS Is Only as Strong as Your Weakest Trust Anchor

The .de outage wasn’t caused by a sophisticated attack. It was a configuration error during routine key rotation — the kind of mistake that can happen to any registry, any operator, at any time. What separated the teams that weathered it from those that scrambled was preparation: multi-resolver setups, Serve-Stale configurations, DNSSEC monitoring, and tested incident playbooks.

Your action items for this week:

  1. Check if your resolvers have Serve-Stale enabled. If not, enable it today.
  2. Set up DNSSEC validation monitoring for your critical domains and their parent TLDs.
  3. Document your DNS dependencies and add DNS failure to your incident response playbook.
  4. Test a multi-resolver failover — don’t wait for the next TLD outage to find out if it works.

The next DNSSEC meltdown isn’t a matter of if — it’s a matter of which TLD and when. Be ready.

Veröffentlicht am

Small Language Models for Platform Engineering: Why 8B Parameters Beat API Dependencies

The economics of AI in platform engineering are shifting — fast. For the past two years, the default answer to „how do we add AI to our internal platform?“ has been „call an API.“ But with inference costs rising, data governance getting stricter, and a new generation of compact models matching much larger counterparts on critical benchmarks, that default is worth questioning. Small Language Models (SLMs) — particularly in the 7B–9B parameter range — have reached a threshold where they can handle the majority of platform engineering workloads without ever leaving your network.

The Benchmark Reality Check: 8B Is Not a Compromise

IBM’s Granite 4.1 8B, released in April 2026 under Apache 2.0, is a useful anchor for this conversation. On enterprise coding benchmarks, the 8B model matches IBM’s own 32B Mixture-of-Experts (MoE) variant. On HumanEval pass@1, the 8B scores 87.2% compared to 89.6% for the 30B model — a gap of less than 3 percentage points that is largely irrelevant for the deterministic, constrained tasks that platform teams actually run.

This pattern holds across the SLM landscape:

  • Phi-4 (14B) — Microsoft’s model excels at reasoning-heavy tasks, punching well above its weight on MATH and GPQA
  • Qwen-3 (8B) — Strong multilingual coding support, excellent for polyglot infrastructure codebases
  • Llama-3.3 (8B) — Meta’s workhorse, widely supported across inference frameworks
  • Mistral-Small (22B) — A good middle ground when you need more capacity without the frontier price tag

The takeaway: if you are still reaching for GPT-4 or Claude Sonnet to answer „why is this Helm chart failing?“ you are likely overspending.

Dense Non-Thinking Architecture: Why It Matters for Operations

Granite 4.1 uses what IBM calls a Dense Non-Thinking Architecture. In practice, this means the model does not execute an internal chain-of-thought (CoT) reasoning step before responding. For frontier models solving novel math problems, CoT is valuable. For a platform engineer asking „summarize this PagerDuty alert and suggest the top three actions,“ CoT overhead is pure latency and token cost with zero benefit.

Platform tasks are largely pattern-matching with context, not novel reasoning. Alert triage, PR description generation, runbook execution, code review comments — these are well-defined, repetitive, structured tasks where a fast, confident response beats a slow, deeply deliberative one. Dense models optimized for inference speed are a natural fit.

The FinOps Case: What Self-Hosting an 8B Model Actually Costs

Let’s put numbers on this. A mid-tier platform team might generate 50,000 LLM calls per month for internal tooling: PR review summaries, alert enrichment, documentation queries, CI/CD pipeline diagnostics.

At $0.002 per 1K tokens (input + output average), 50,000 calls at ~500 tokens each = $50/month in API costs. Manageable — until agents arrive.

Agentic workflows are not single API calls. A single „investigate this alert“ agent might issue 15–25 tool calls, each with full context. That same 50,000-event scenario becomes 750,000–1,250,000 LLM calls. At $0.002/1K tokens, that is now $1,500–$2,500/month — and growing linearly with adoption.

Self-hosting an 8B model on a single RTX 4090 (~$1,800 hardware) or a Mac Studio M4 Max (~$2,000) delivers:

  • ~30–50 tokens/second throughput (sufficient for internal tooling)
  • Zero marginal cost per call after hardware amortization
  • Full data residency — no tokens leave your network
  • Instant availability without rate limits or provider outages

At an agentic scale, the hardware pays for itself within 1–2 months. Beyond that, it is pure savings.

Platform Engineering Use Cases Where SLMs Shine

1. Alert Triage and Runbook Execution

The HolmesGPT pattern (CNCF Sandbox) demonstrates the right approach: give an SLM access to kubectl, PromQL, and Loki, and a structured Markdown runbook. With a well-crafted runbook, tool calls per investigation drop from 16+ to 2–4. An 8B model running locally handles this at millisecond latency with no data leaving the cluster.

2. CI/CD Pipeline Assistance

PR description generation, test coverage summaries, changelog drafting — these are low-complexity, high-volume tasks. An SLM integrated directly into your CI/CD pipeline (via Ollama’s REST API or a vLLM endpoint) can run as a pipeline step without any external dependency. No API key rotation. No rate limiting during a big release crunch.

3. Code Review Comments

Automated first-pass code review — style enforcement, security pattern flagging, documentation gaps — is exactly the kind of task where an 8B model is sufficient. The model does not need to understand your entire business domain; it needs to apply consistent rules to code diffs. Fine-tuning on your internal codebase further improves relevance.

4. Documentation and Runbook Generation

Keeping runbooks current is a perennial platform team pain point. An SLM that can read infrastructure-as-code, observe recent incident patterns, and generate or update Markdown documentation solves a real operational problem — without requiring a cloud API call for every update.

Enterprise Trust: Granite’s Compliance Credentials

IBM Granite 4.1 ships with two features that matter disproportionately in regulated industries: Guardian Models and cryptographic signing.

Guardian Models are companion classifiers that can check model inputs and outputs for compliance — harmful content, PII exposure, prompt injection attempts. This is built into the model ecosystem, not bolted on afterward. For financial services or healthcare platform teams, this is a significant differentiator versus a generic open-source model.

The cryptographic signing (with ISO certification) means you can verify model provenance. In an era where supply chain security is central to platform governance (see SLSA, Sigstore, in-toto), being able to verify that the model running in your cluster is exactly the model IBM published is not a minor detail.

The Multi-Model Strategy: SLM + Cloud for 80/20 Coverage

The most practical approach is not „replace all cloud APIs with SLMs“ — it is to route intelligently:

  • ~80% of tasks → Local SLM: Alert triage, CI/CD assistance, doc generation, code review, runbook execution, structured queries against internal data
  • ~20% of tasks → Cloud frontier model: Novel architecture decisions, complex multi-step reasoning, tasks requiring broad world knowledge not captured in your fine-tuned model

This mirrors how mature platform teams already think about compute: use the right tool at the right cost tier. An internal platform that routes requests based on complexity signals (task type, token budget, confidence threshold) gives you both cost efficiency and capability headroom.

Getting Started: Self-Hosting in the Platform Engineering Stack

The barrier to running an 8B model is lower than most teams expect:

  • Ollama — Single-command model serving, REST API, model library with one-line pulls (ollama pull granite3.3:8b)
  • LM Studio — Desktop GUI for evaluation, good for initial benchmarking before committing to infrastructure
  • vLLM — Production-grade serving with OpenAI-compatible API, batching, and quantization support; the right choice for Kubernetes-native deployments

For Kubernetes, vLLM running as a Deployment with a GPU node selector and an HPA on request queue depth is a reasonable production starting point. Pair it with an OpenAI-compatible API shim and your existing LLM-integrated tooling requires zero code changes to switch endpoints.

The Connection to Agentic Infrastructure

The Agentic Compute Cliff is real: GitHub Copilot paused new signups in April 2026 due to capacity constraints, and multiple cloud providers are experiencing GPU shortages. As agentic workloads scale — where a single developer workflow might trigger hundreds of LLM calls per hour — dependency on cloud inference is a reliability and cost risk.

SLMs running on internal infrastructure are not just a cost play. They are a resilience play. Your internal platform keeps working when the cloud provider has an outage. Your agents are not rate-limited during a major incident response. Your data never transits a network boundary you do not control.

When 8B Is Not Enough

Intellectual honesty matters here. SLMs are not the answer for everything:

  • Novel architecture decisions requiring broad reasoning across domains
  • Complex multi-step debugging across large, unfamiliar codebases
  • Tasks requiring deep world knowledge beyond your training/fine-tuning window
  • High-stakes customer-facing generation where quality variance is unacceptable

The skill is in classification — building a platform that knows when to route locally and when to escalate to a frontier model. That routing logic, often just a simple task classifier, is itself a good candidate to run on a local SLM.

Conclusion: Make the Economics Argument

The conversation about SLMs in platform engineering is no longer theoretical. The benchmarks have arrived. The tooling (Ollama, vLLM, LM Studio) is mature. The hardware cost is justified within months at agentic scale. And the privacy and compliance benefits — data residency, Guardian Models, cryptographic provenance — increasingly matter as organizations bring AI deeper into their software delivery lifecycle.

The 8B parameter class is not a compromise. It is a deliberate choice that aligns cost, performance, privacy, and operational simplicity for the tasks that platform teams actually run. Start with one use case — alert triage is a natural first target — measure the results, and expand from there. The API dependency you are paying for today may be entirely optional.

Veröffentlicht am

The Vercel Breach Playbook: What Platform Teams Must Do When Their PaaS Provider Gets Compromised

Today — April 19, 2026 — Vercel disclosed a security incident involving unauthorized access to its internal systems. The breach has been linked to the ShinyHunters group, a threat actor known for targeting SaaS platforms via social engineering and vulnerability exploitation. Vercel says a „limited subset of customers“ was impacted and recommends reviewing environment variables — particularly urging use of their Sensitive Environment Variable feature.

If you’re a platform engineer running production workloads on Vercel, this is your signal to act. Not tomorrow. Now.

But this post isn’t just about Vercel. It’s about what every platform team should do when the infrastructure they trust gets compromised — because this has happened before, and it will happen again.

We’ve Been Here Before

The Vercel breach follows a pattern that platform teams should recognize by now:

  • CircleCI (January 2023) — An engineer’s laptop was compromised, giving attackers access to customer environment variables, tokens, and keys. CircleCI’s guidance was unambiguous: rotate every secret, immediately. Teams that delayed paid the price.
  • Codecov (April 2021) — Attackers modified Codecov’s Bash Uploader script, exfiltrating environment variables from CI pipelines for two months before detection. Thousands of repositories had their credentials silently harvested.
  • Travis CI (September 2021) — A vulnerability exposed secrets from public repositories, including signing keys and access tokens. The scope was enormous because the trust boundary had been quietly violated for years.

The common thread: environment variables are the crown jewels, and PaaS providers are the vault. When the vault gets cracked, every secret inside is potentially compromised.

The Shared Responsibility Blind Spot

Most teams understand the shared responsibility model for IaaS — you secure your workloads, AWS secures the hypervisor. But with PaaS providers like Vercel, Netlify, or Railway, the trust boundary is far murkier.

Consider what Vercel has access to in a typical deployment:

  • Your source code (pulled from Git during builds)
  • Every environment variable you’ve configured — database URLs, API keys, signing secrets
  • Build-time and runtime secrets
  • Deployment metadata and audit logs
  • DNS configuration and SSL certificates

When Vercel’s internal systems are breached, all of these become part of the blast radius. You didn’t misconfigure anything. You didn’t leak a credential. Your provider’s security posture became your security posture.

This is the platform trust boundary problem: the more convenience your PaaS offers, the more implicit trust you’ve delegated.

Immediate Response: The First 24 Hours

If you’re running on Vercel right now, here’s the checklist. Don’t wait for their investigation to conclude — assume the worst and work backward.

1. Audit Your Environment Variables

Vercel’s own advisory specifically calls out environment variables. Start here:

# List all Vercel projects and their env vars
vercel env ls --environment production
vercel env ls --environment preview
vercel env ls --environment development

Or use the consolidated environment variables page Vercel provides. Document every secret. You need to know what’s potentially exposed before you can rotate.

2. Rotate Every Secret — No Exceptions

This is the lesson from CircleCI: partial rotation is no rotation. If a secret was accessible to your PaaS provider, treat it as compromised.

  • Database credentials (connection strings, passwords)
  • API keys (Stripe, Twilio, SendGrid, any third-party service)
  • OAuth client secrets
  • JWT signing keys
  • Webhook secrets
  • Encryption keys

Prioritize by blast radius: payment processing keys and database credentials first, monitoring API keys last.

3. Review Deployment History

Check for unauthorized deployments or unexpected build activity:

# Review recent deployments via Vercel CLI
vercel ls --limit 50

# Check for deployments from unexpected branches or commits
vercel inspect <deployment-url>

Look for deployments that don’t correlate with your Git history. An attacker with access to Vercel’s internals could potentially trigger builds with modified environment variables or injected build steps.

4. Revoke and Regenerate Tokens

Beyond environment variables, rotate all integration tokens:

  • Vercel API tokens (personal and team)
  • Git integration tokens (GitHub/GitLab app installations)
  • Any webhook endpoints that use shared secrets for verification
  • CI/CD integration tokens that connect to Vercel

5. Check Downstream Systems

If your database credentials were in Vercel env vars, check your database audit logs for unusual access patterns. If your AWS keys were stored there, review CloudTrail. Every secret that was in Vercel is a thread to pull.

Stop Storing Secrets in Environment Variables

The deeper lesson here is architectural. Environment variables are the de facto standard for passing configuration to applications — but they were never designed as a secrets management system. They’re plaintext, they get logged, they get copied into build caches, and they’re only as secure as the system storing them.

External Secrets Operator

If you’re running Kubernetes workloads (even alongside a PaaS), the External Secrets Operator lets you reference secrets from external stores without ever putting them in your deployment platform:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: vault-backend
    kind: ClusterSecretStore
  target:
    name: db-creds
  data:
    - secretKey: password
      remoteRef:
        key: secret/data/production/database
        property: password

The secret lives in Vault or AWS Secrets Manager. Your PaaS never sees it. If the PaaS is breached, the secret isn’t in the blast radius.

HashiCorp Vault with Dynamic Secrets

Even better: don’t store long-lived credentials at all. Vault’s dynamic secrets generate short-lived database credentials on demand:

# Application requests temporary database credentials at startup
vault read database/creds/my-role
# Returns credentials valid for 1 hour
# Automatically revoked after TTL expires

When your PaaS is breached, there’s nothing useful to steal — the credentials expired hours ago.

CI/CD Credential Hygiene: Kill the Static Tokens

Static API keys and long-lived tokens are the gift that keeps giving — to attackers. Every major PaaS breach has involved harvesting static credentials. The fix is structural.

OIDC Federation: Identity Without Secrets

Instead of storing cloud provider credentials in your CI/CD platform, use OIDC federation. Your pipeline proves its identity to the cloud provider directly, receiving short-lived tokens that can’t be stolen from the PaaS:

# GitHub Actions example — no AWS keys stored anywhere
- uses: aws-actions/configure-aws-credentials@v4
  with:
    role-to-assume: arn:aws:iam::123456789:role/deploy-role
    aws-region: eu-central-1
    # No access-key-id or secret-access-key needed
    # GitHub's OIDC token proves the workflow's identity

All major cloud providers support OIDC federation from GitHub Actions, GitLab CI, and most CI/CD platforms. There is no good reason to store static cloud credentials in your PaaS in 2026.

Workload Identity and SPIFFE7.

For more complex deployments, SPIFFE (Secure Production Identity Framework for Everyone) and its reference implementation SPIRE provide cryptographic identity attestation for workloads. Every workload gets a verifiable identity (SVID) without static credentials, and identity is attested based on the workload’s environment — not a secret that can be exfiltrated.

This is zero-trust for deployment pipelines: trust is established through verifiable identity, not shared secrets.

SBOM and Provenance: Know What You Shipped

When your build platform is compromised, one critical question emerges: can you prove that what’s running in production is what you intended to ship?

Build provenance — cryptographic attestations that link a deployed artifact to its source code, build parameters, and builder identity — becomes essential during incident response:

# Verify build provenance with cosign
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity builder@your-org.iam.gserviceaccount.com \
  --certificate-oidc-issuer https://accounts.google.com \
  ghcr.io/your-org/your-app:latest

If you maintain SBOMs (Software Bills of Materials) and SLSA provenance attestations, you can forensically verify whether a compromised build platform injected anything into your artifacts. Without them, you’re flying blind.

Long-Term: Multi-Provider Resilience

The uncomfortable truth is that every PaaS provider will eventually have a security incident. The question isn’t if — it’s whether your architecture limits the blast radius when it happens.

Reduce Single Points of Trust

  • Secrets in an external vault, not in the PaaS — Vault, AWS Secrets Manager, Azure Key Vault
  • Build artifacts signed independently — don’t rely on the build platform’s integrity alone
  • DNS and TLS managed separately — if your PaaS controls your DNS, a breach can redirect traffic
  • Audit logs forwarded in real-time — ship PaaS audit logs to your own SIEM before the provider can tamper with them

Portable Deployments

If your deployment is tightly coupled to a single PaaS, you can’t move quickly during an incident. Containerized workloads with Infrastructure-as-Code configuration give you the option to shift to another platform within hours, not weeks. You don’t need to be multi-cloud on day one — but you need the capability to move when the trust relationship breaks.

The Incident Response Checklist

Pin this somewhere visible. When your next PaaS breach notification lands in your inbox:

Timeframe Action
0-1 hours Inventory all secrets stored in the provider. Begin rotating critical credentials (database, payment, auth).
1-4 hours Revoke all API tokens and integration credentials. Review deployment history for anomalies.
4-12 hours Complete rotation of all remaining secrets. Check downstream system audit logs. Verify build artifact integrity.
12-24 hours Confirm no unauthorized deployments occurred. Brief stakeholders. Document timeline.
1-7 days Conduct full post-incident review. Implement architectural improvements (external secrets, OIDC federation). Update runbooks.

Trust, but Architect for Betrayal

The Vercel breach is a reminder that platform trust is borrowed, not owned. Every convenience a PaaS provides — environment variable storage, built-in secrets, managed DNS — is a trust delegation that becomes a liability during a breach.

The platforms you depend on will get compromised. The question is whether you’ve architected your systems so that a provider breach is a inconvenience you handle in hours — or a catastrophe that takes weeks to untangle.

Start rotating your secrets now. Then start building the architecture that means you won’t have to do it so urgently next time.

Veröffentlicht am

Dapr Agents v1.0: Resilient Multi-Agent Orchestration on Kubernetes

The Distributed Systems Foundation for AI Agents

When LangGraph introduced stateful agents and CrewAI popularized role-based collaboration, they solved the what of multi-agent AI systems. But as organizations move from demos to production, a critical question emerges: how do you run these systems reliably at scale?

Enter Dapr Agents, which reached v1.0 GA in March 2026. Built on the battle-tested Dapr runtime—a CNCF graduated project—this Python framework takes a fundamentally different approach: instead of bolting reliability onto AI frameworks, it brings AI agents to proven distributed systems primitives.

The result? AI agents that inherit decades of distributed systems wisdom: durable execution, exactly-once semantics, automatic retries, and the ability to survive node failures without losing state.

Why Traditional Agent Frameworks Struggle in Production

Most AI agent frameworks were designed for prototyping. They work brilliantly in Jupyter notebooks but encounter friction when deployed to Kubernetes:

  • State Loss on Restart: LangGraph checkpoints require manual persistence configuration. A pod restart can lose agent memory mid-conversation.
  • No Native Retry Semantics: When an LLM API returns a 429, most frameworks fail or require custom retry logic.
  • Coordination Complexity: Multi-agent communication typically requires custom message queues or REST endpoints.
  • Observability Gaps: Tracing an agent’s reasoning across multiple tool calls often means stitching together fragmented logs.

Dapr Agents addresses each of these by standing on the shoulders of infrastructure patterns that have been production-hardened since the early days of microservices.

Architecture: Agents as Distributed Actors

At its core, Dapr Agents builds on three Dapr building blocks:

1. Workflows for Durable Execution

Every agent interaction—LLM calls, tool invocations, state updates—is persisted as a workflow step. If the agent crashes mid-reasoning, it resumes exactly where it left off:

from dapr_agents import DurableAgent, tool

class ResearchAgent(DurableAgent):
    @tool
    def search_arxiv(self, query: str) -> list:
        return arxiv_client.search(query)
    
    async def research(self, topic: str):
        papers = await self.search_arxiv(topic)
        summary = await self.llm.summarize(papers)
        return summary

Under the hood, Dapr Workflows use the Virtual Actor model—the same pattern that powers Orleans and Akka. Each agent is a stateful actor that can be deactivated when idle and reactivated on demand, enabling thousands of agents to run on a single node.

2. Pub/Sub for Event-Driven Coordination

Multi-agent systems need reliable communication. Dapr’s Pub/Sub abstraction lets agents publish events and subscribe to topics without knowing about the underlying message broker:

from dapr_agents import AgentRunner

await agent_a.publish("research-complete", {
    "topic": "quantum computing",
    "findings": summary
})

@runner.subscribe("research-complete")
async def handle_research(event):
    await writer_agent.draft_article(event["findings"])

Swap Redis for Kafka or RabbitMQ without changing agent code.

3. State Management for Agent Memory

Conversation history, tool results, reasoning traces—all flow through Dapr’s State API with pluggable backends:

from dapr_agents import memory

agent = ResearchAgent(memory=memory.InMemory())

agent = ResearchAgent(
    memory=memory.PostgreSQL(
        connection_string=os.environ["PG_CONN"],
        enable_vector_search=True
    )
)

Agentic Patterns Out of the Box

Dapr Agents ships with implementations of common multi-agent patterns:

Pattern Description Use Case
Prompt Chaining Sequential LLM calls where each output feeds the next Document processing
Evaluator-Optimizer One LLM generates, another critiques in a loop Code review
Parallelization Fan-out work to multiple agents, aggregate results Research synthesis
Routing Classify input and delegate to specialist agents Customer support
Orchestrator-Workers Central coordinator delegates subtasks dynamically Complex workflows

MCP and Cross-Framework Interoperability

A standout feature is native support for the Model Context Protocol (MCP):

from dapr_agents import MCPToolProvider

tools = MCPToolProvider("http://mcp-server:8080")
agent = DurableAgent(tools=[tools])

Dapr Agents can also invoke agents from other frameworks as tools:

from dapr_agents.interop import CrewAITool

research_crew = CrewAITool(crew=research_crew, name="research_team")
coordinator = DurableAgent(tools=[research_crew])

Kubernetes-Native Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: research-agent
  annotations:
    dapr.io/enabled: "true"
    dapr.io/app-id: "research-agent"
spec:
  replicas: 3
  template:
    spec:
      containers:
      - name: agent
        image: myregistry/research-agent:v1

Comparison: Dapr Agents vs. LangGraph vs. CrewAI

Capability Dapr Agents LangGraph CrewAI
Durable Execution Built-in Requires config Limited
Auto Retry Built-in Manual Manual
State Persistence 50+ backends SQLite, PG In-memory
Kubernetes Native Sidecar Manual Manual
Observability OpenTelemetry LangSmith Limited

When to Choose Dapr Agents

Dapr Agents makes sense when:

  • You’re already running Dapr for microservices
  • Your agents must survive node failures without state loss
  • You need to scale to thousands of concurrent agents
  • Enterprise observability requirements demand OpenTelemetry

Getting Started

pip install dapr-agents
dapr init

from dapr_agents import DurableAgent, AgentRunner

class GreeterAgent(DurableAgent):
    system_prompt = "You are a helpful assistant."

runner = AgentRunner(agent=GreeterAgent())
runner.start()

The Bigger Picture

Dapr Agents represents a broader trend: AI frameworks are maturing from „make it work“ to „make it work reliably.“ The CNCF ecosystem is converging on this need—KubeCon 2026 showcased kagent, AgentGateway, and the AI Gateway Working Group.

For platform teams, Dapr Agents offers a familiar operational model: sidecars, state stores, message brokers, and observability pipelines. The agents are new; the infrastructure patterns are proven.

Dapr Agents v1.0 is available now at github.com/dapr/dapr-agents.

Veröffentlicht am

MCP Security: Securing the Model Context Protocol for Enterprise AI Agents

The Model Context Protocol (MCP) has rapidly become the de facto standard for connecting AI agents to enterprise systems. Originally developed by Anthropic and released in November 2024, MCP provides a standardized interface for AI models to interact with databases, APIs, file systems, and external services. It’s the protocol that powers Claude’s ability to read your files, query your databases, and execute tools on your behalf.

But with adoption accelerating—Gartner predicts 40% of enterprise applications will integrate MCP servers by end of 2026—security researchers are discovering critical vulnerabilities that could turn your helpful AI assistant into a gateway for attackers.

The Protocol That Connects Everything

MCP works by establishing a client-server architecture where AI models (the clients) connect to MCP servers that expose „tools“ and „resources.“ When you ask Claude to read a file or query a database, it’s making MCP calls to servers that have been granted access to those systems.

The protocol is elegant in its simplicity: JSON-RPC messages over standard transports (stdio, HTTP, WebSocket). But this simplicity also means that a single compromised MCP server can potentially access everything it’s been granted permission to touch.

Consider a typical enterprise setup: an MCP server connected to your GitHub repositories, another to your production database, a third to your internal documentation. Each server aggregates credentials and access tokens. An attacker who compromises one server doesn’t just get access to that service—they get access to the aggregated credentials that service holds.

Recent CVEs: A Wake-Up Call

The first quarter of 2026 has already seen two critical CVEs in official MCP SDK implementations:

CVE-2026-34742 (CVSS 8.1) affects the official Go SDK. A DNS rebinding vulnerability allows attackers to bypass localhost restrictions by resolving to 127.0.0.1 after initial CORS checks pass. This means a malicious website could potentially interact with MCP servers running on a developer’s machine, even when those servers are configured to only accept local connections.

CVE-2026-34237 (CVSS 7.5) in the Java SDK involves improper CORS wildcard handling. The SDK accepted overly permissive origin configurations that could be exploited to bypass same-origin protections, potentially allowing cross-site request forgery against MCP endpoints.

These aren’t theoretical vulnerabilities—they’re implementation bugs in the official SDKs that thousands of developers use to build MCP integrations. The patches are available, but how many custom MCP servers in production environments are still running vulnerable versions?

Attack Vectors Unique to MCP

Beyond SDK vulnerabilities, MCP introduces new attack surfaces that security teams need to understand:

Tool Poisoning and Rug Pulls

MCP’s tool discovery mechanism allows servers to dynamically advertise available tools. A compromised server can change its tool definitions at runtime—a „rug pull“ attack. Your AI agent thinks it’s calling read_file, but the server has silently replaced it with a tool that exfiltrates data before returning results.

More subtle: tool descriptions influence how AI models use them. A malicious server could manipulate descriptions to guide the AI toward dangerous actions. „Use this tool for all sensitive operations“ could be embedded in a description, influencing the model’s behavior without changing the tool’s apparent functionality.

The Confused Deputy Problem

AI agents operate with the combined permissions of their MCP connections. When an agent uses multiple tools in sequence, it can inadvertently transfer data between contexts in ways that violate security boundaries.

Example: A user asks an AI to „summarize the Q1 financials and post a summary to Slack.“ The agent reads confidential data from a financial database (MCP server A) and posts it to a public channel (MCP server B). Neither MCP server violated its permissions—but the agent performed an unauthorized data transfer.

Shadow AI via Uncontrolled MCP Servers

Developers love convenience. When official MCP integrations are locked down by IT, they’ll spin up their own servers on localhost. These shadow MCP servers often have overly permissive configurations, skip authentication entirely, and connect to production systems using personal credentials.

The result: an invisible attack surface that security teams can’t monitor because they don’t know it exists.

Defense in Depth: Securing MCP Deployments

Authentication: OAuth 2.1 with PKCE

MCP’s transport layer supports OAuth 2.1, but many deployments still rely on API keys or skip authentication for „internal“ servers. This is insufficient.

Implement OAuth 2.1 with PKCE (Proof Key for Code Exchange) for all MCP connections, even internal ones. PKCE prevents authorization code interception attacks that could allow attackers to hijack MCP sessions.

# Example MCP server configuration
auth:
  type: oauth2
  issuer: https://auth.company.com
  client_id: mcp-database-server
  pkce: required
  scopes:
    - mcp:tools:read
    - mcp:tools:execute

Every MCP server should validate tokens on every request—don’t cache authentication decisions.

Centralized MCP Gateways

Rather than allowing AI agents to connect directly to MCP servers, route all traffic through a centralized gateway. This provides several security benefits:

Traffic visibility: Log every tool call, including parameters and results. This audit trail is essential for detecting anomalies and investigating incidents.

Policy enforcement: Implement fine-grained access controls that go beyond what individual MCP servers support. Block specific tool calls based on user identity, time of day, or risk scoring.

Rate limiting: Prevent credential stuffing and abuse by throttling requests at the gateway level.

This pattern mirrors what we discussed in our AI Gateways post—the same architectural principles apply. Products like Aurascape, TrueFoundry, and Bifrost are beginning to offer MCP-specific gateway capabilities.

Behavioral Analysis for Anomaly Detection

MCP call patterns are highly predictable for legitimate use cases. A developer’s AI assistant will typically make similar calls day after day: reading code files, querying documentation, creating pull requests.

Sudden changes in behavior—a new tool being called for the first time, unusual data volumes, calls at unexpected hours—should trigger alerts. This is where AI can help secure AI: use machine learning models to baseline normal MCP activity and flag deviations.

Key signals to monitor:

  • First-time tool usage by an established user
  • Data volume anomalies (reading entire databases vs. specific records)
  • Tool call sequences that don’t match known workflows
  • Geographic or temporal anomalies in API calls

Supply Chain Validation

Many organizations install MCP servers from package managers (npm, pip) without verifying integrity. The LiteLLM supply chain attack in March 2026 demonstrated how a compromised package could inject malicious code into AI infrastructure.

For MCP servers:

  1. Pin specific versions in your dependency files
  2. Verify package signatures where available
  3. Scan MCP server code for malicious patterns before deployment
  4. Maintain an inventory of all MCP servers and their versions
  5. Subscribe to security advisories for SDKs you use

Principle of Least Privilege

Each MCP server should have the minimum permissions necessary for its function. This seems obvious, but the convenience of MCP makes it tempting to create „god servers“ that can access everything.

Instead:

  • Create separate MCP servers for different data classifications
  • Use short-lived credentials that are rotated frequently
  • Implement time-based access windows where possible
  • Regularly audit and revoke unused permissions

The Path Forward

MCP is too useful to avoid. The productivity gains from giving AI agents structured access to enterprise systems are substantial. But we’re in the early days of understanding MCP’s security implications.

The organizations that will thrive are those that treat MCP security as a first-class concern from day one. Don’t wait for a breach to implement proper authentication, monitoring, and access controls.

Start here:

  1. Inventory: Know every MCP server in your environment, official and shadow
  2. Authenticate: Deploy OAuth 2.1 with PKCE for all MCP connections
  3. Monitor: Route MCP traffic through a centralized gateway with logging
  4. Validate: Implement supply chain security for MCP server dependencies
  5. Limit: Apply least-privilege principles to every MCP server’s permissions

The Model Context Protocol represents a fundamental shift in how AI agents interact with enterprise infrastructure. Getting security right now—while the ecosystem is still maturing—is far easier than retrofitting it later.

This post builds on our earlier exploration of AI Gateways. For more on protecting AI infrastructure, see our series on Guardrails for Agentic Systems and Non-Human Identity.