Schlagwort: Incident Response

What Happened on May 5, 2026

At approximately 10:42 UTC on Monday, May 5, monitoring dashboards across Europe lit up. DNS resolution for .de domains — one of the world’s largest country-code TLDs, consistently ranking in the Top 5 at Cloudflare Radar — started failing en masse. The root cause: DENIC, the registry operator for .de, had published DNSSEC signatures that did not match the zone’s active Zone Signing Key (ZSK).

The timing was no coincidence. The faulty signatures surfaced during a scheduled ZSK rotation — one of the most operationally sensitive windows in DNSSEC key management. A misconfiguration in the signing pipeline meant that the new signatures were generated with a key that validating resolvers could not verify against the published DS records in the root zone. The result was catastrophic: the entire .de chain of trust was broken.

Within minutes, every DNSSEC-validating resolver worldwide — including Cloudflare’s 1.1.1.1, Google’s 8.8.8.8, and Quad9’s 9.9.9.9 — began returning SERVFAIL for queries to .de domains. Non-validating resolvers continued to work, which created a confusing split-brain situation where some users could reach German websites and others couldn’t, depending on their configured resolver.

The DNSSEC Chain of Trust: One Link Breaks, Everything Falls

To understand why a single registry mistake can have such a massive blast radius, you need to understand how DNSSEC validation works.

DNSSEC adds cryptographic signatures to DNS records. Resolvers verify these signatures by walking a chain of trust from the root zone (.) down through the TLD (.de) to the individual domain (example.de). Each level delegates trust to the next via DS (Delegation Signer) records. If any link in this chain produces an invalid signature, a validating resolver must return SERVFAIL. That’s not a bug — it’s the design. DNSSEC was built to prevent cache poisoning, and treating unverifiable answers as failures is the entire point.

The double-edged nature of this design becomes painfully clear during operator errors at the TLD level. When DENIC’s signatures broke, it wasn’t just one domain that failed — it was every single .de domain, regardless of whether the individual domain owner had done everything right. The TLD is a single point of cryptographic failure for all domains beneath it.

ZSK/KSK Rotation: The Critical Window

DNSSEC uses two types of keys: the Key Signing Key (KSK), which signs the DNSKEY RRset, and the Zone Signing Key (ZSK), which signs the actual zone data. ZSK rotations happen more frequently and involve a carefully choreographed dance: pre-publish the new key, wait for caches to expire, sign with the new key, remove the old one. Get any step wrong — wrong timing, wrong key reference, stale DS record — and you shatter the chain of trust. This is exactly what happened with .de.

How Major Resolvers Responded

The incident provided a real-world stress test for two mitigation techniques that the DNS community has been advocating for years: Serve-Stale and Negative Trust Anchors.

Serve-Stale (RFC 8767)

Serve-Stale allows a resolver to return expired (stale) cached records instead of failing with SERVFAIL when it cannot fetch a fresh, valid answer from upstream. Cloudflare’s 1.1.1.1 had Serve-Stale enabled, and their detailed incident report showed that users hitting warm caches continued to get working answers for .de domains — stale data, but functional. For most use cases (websites, APIs, mail routing), a stale A or AAAA record from five minutes ago is infinitely better than SERVFAIL.

The limitation: Serve-Stale only works if the record was previously cached. Cold caches — new queries for domains the resolver hadn’t seen recently — still failed. And once stale TTLs expired (typically capped at 1–3 days depending on implementation), even warm caches would stop serving.

Negative Trust Anchors (RFC 7646)

Negative Trust Anchors (NTAs) are the emergency brake for DNSSEC. An NTA tells a resolver: „Stop validating DNSSEC for this specific domain or zone.“ When applied to .de, it effectively disables signature verification for the entire TLD, allowing queries to resolve normally — at the cost of losing DNSSEC protection.

Cloudflare, Google, and Quad9 all deployed NTAs for .de within the first hour of the incident. This was the fastest path to restoring service for end users. The NTAs were removed once DENIC republished correct signatures later that day.

The Third Option: Disabling DNSSEC Validation Entirely

Some smaller operators chose the nuclear option: disabling DNSSEC validation on their resolvers entirely. This restored service for all domains immediately but removed cryptographic protection for every zone, not just the broken one. This is the equivalent of disabling your firewall because one rule is misconfigured — it works, but the security implications are severe. NTAs are strictly preferable because they scope the trust bypass to the affected zone.

The Amplification Problem

DNS outages create a vicious feedback loop. When resolvers return SERVFAIL, clients retry — aggressively. Applications retry. Browsers retry. Stub resolvers retry. Monitoring systems fire off their own queries. Cloudflare reported a 10x spike in query volume for .de during the incident, as retry storms amplified the load on authoritative servers and resolvers alike.

This client-retry amplification is a well-known pattern in distributed systems, but it’s especially brutal in DNS because retries happen at multiple layers simultaneously. It delays recovery because even after the root cause is fixed, the query flood continues until retry backoffs settle.

Parallels to Prior TLD Outages

The .de incident wasn’t the first time a TLD’s DNSSEC misconfiguration caused widespread outages. In 2024, New Zealand’s .nz experienced a similar DNSSEC signing failure that took down domains across the country. Sweden’s .se has had its own DNSSEC-related incidents. Each time, the pattern is the same: a key management error at the TLD level cascades into a nationwide or zone-wide outage, and the community rediscovers that DNSSEC’s strict validation model trades availability for integrity.

The lesson keeps repeating because the operational complexity of DNSSEC key management is genuinely hard, and the failure mode is binary: it either validates or it doesn’t. There’s no graceful degradation built into the protocol itself.

Platform Engineering Lessons

If you’re running a platform team — especially one operating in the EU — the .de incident should be a wake-up call. DNS is deeply embedded in every layer of a modern cloud-native stack: ExternalDNS syncs records, cert-manager validates domain ownership via DNS-01 challenges, Ingress controllers rely on DNS routing, service meshes resolve endpoints. A DNS outage isn’t just „websites are down“ — it can break certificate issuance, deployment pipelines, service discovery, and monitoring.

1. Monitor DNSSEC Validation, Not Just Resolution

Most teams monitor whether DNS resolution works. Few monitor whether DNSSEC validation is healthy. Set up checks that specifically test DNSSEC signature validity for your critical domains and their parent zones. Tools like DNSViz, Zonemaster, and RIPE Atlas probes can automate this. Alert on validation failures before your users notice.

2. Implement a Multi-Resolver Strategy

Don’t depend on a single upstream resolver. Configure failover across multiple providers: Cloudflare (1.1.1.1), Google (8.8.8.8), Quad9 (9.9.9.9). Each operator has different NTA deployment speeds and Serve-Stale configurations. During the .de incident, the window between „Cloudflare deployed NTA“ and „smaller ISP resolvers deployed NTA“ was measured in hours. A multi-resolver setup lets you ride the fastest responder.

3. Deploy Serve-Stale in Your Own Resolvers

If you run local resolvers (CoreDNS, Unbound, BIND), enable Serve-Stale. In CoreDNS, this means configuring the cache plugin with serve_stale. In Unbound, set serve-expired: yes with appropriate serve-expired-ttl and serve-expired-client-timeout values. This single configuration change is your best passive defense against upstream DNSSEC failures.

# Unbound example
server:
    serve-expired: yes
    serve-expired-ttl: 86400
    serve-expired-client-timeout: 1800

# CoreDNS example
.:53 {
    forward . 1.1.1.1 8.8.8.8 9.9.9.9
    cache 3600 {
        serve_stale 86400
    }
}

4. Treat DNS as a Critical Dependency in Your Architecture

Map out every component in your stack that depends on DNS resolution. ExternalDNS, cert-manager (DNS-01 challenges), Ingress controllers, external API calls, webhook endpoints, OAuth/OIDC provider discovery — all of these break when DNS breaks. Document these dependencies and include DNS failure scenarios in your chaos engineering practice.

5. Build a DNS Incident Response Playbook

Your runbook should5 include:

• Detection: Automated alerts for DNSSEC validation failures and elevated SERVFAIL rates
• Triage: Is the issue local, resolver-level, or TLD-level? Use dig +dnssec and delv to isolate
• Mitigation: Pre-approved steps to deploy NTAs on local resolvers, switch upstream resolvers, or enable Serve-Stale
• Communication: Templates for status page updates that explain DNS issues to non-technical stakeholders
• Recovery: Validation that DNSSEC signatures are correct before removing NTAs

6. NIS2 and DORA: DNS Resilience Is Now a Compliance Issue

For organizations operating in the EU, the NIS2 Directive and the Digital Operational Resilience Act (DORA) explicitly require resilience measures for critical infrastructure, including ICT supply chain risks. DNS is a foundational ICT service. A TLD-level outage that takes down your platform because you had no failover, no Serve-Stale, and no incident playbook is now a compliance gap, not just an operational one. Document your DNS resilience measures as part of your NIS2/DORA risk assessments.

The Bigger Picture

The .de DNSSEC meltdown highlights a fundamental tension in internet infrastructure: the systems designed to protect us (DNSSEC, certificate validation, strict security policies) can also become single points of failure when they break. The answer isn’t to disable security — it’s to build resilience layers that absorb the impact of failures without sacrificing protection during normal operations.

Serve-Stale and Negative Trust Anchors are exactly this kind of resilience layer. They don’t weaken DNSSEC; they give operators a controlled way to maintain availability while the underlying issue is fixed. Every platform team should have both in their toolkit.

Conclusion: Your DNS Is Only as Strong as Your Weakest Trust Anchor

The .de outage wasn’t caused by a sophisticated attack. It was a configuration error during routine key rotation — the kind of mistake that can happen to any registry, any operator, at any time. What separated the teams that weathered it from those that scrambled was preparation: multi-resolver setups, Serve-Stale configurations, DNSSEC monitoring, and tested incident playbooks.

Your action items for this week:

1. Check if your resolvers have Serve-Stale enabled. If not, enable it today.
2. Set up DNSSEC validation monitoring for your critical domains and their parent TLDs.
3. Document your DNS dependencies and add DNS failure to your incident response playbook.
4. Test a multi-resolver failover — don’t wait for the next TLD outage to find out if it works.

The next DNSSEC meltdown isn’t a matter of if — it’s a matter of which TLD and when. Be ready.