Schlagwort: FinOps

Veröffentlicht am

The AI-Native Platform Contract: Expanding Golden Paths Beyond Application Delivery

Platform engineering earned its place by turning application delivery into a repeatable product. Golden paths combined infrastructure, security, deployment, and operational standards into a paved route that developers could use without learning every platform detail.

AI-native workloads do not invalidate that model. They expose where it stops too early.

A conventional golden path typically starts with source code and ends with a running service. An AI-native product depends on a wider chain: governed data, accelerator capacity, models and prompts, evaluation evidence, inference controls, agent identities, external tools, and continuous cost and risk feedback. If each of those capabilities arrives through a separate specialist portal, the organization has not created an AI platform. It has created another integration problem.

The next platform contract should therefore extend the golden path rather than build a parallel AI silo. The goal is not to hide every AI decision behind automation. It is to make safe defaults easy, exceptions explicit, and every promoted artifact traceable.

The application delivery contract is no longer enough

Platform Engineering 1.0 concentrated on a familiar delivery unit: an application packaged as a container, deployed through a pipeline, and operated with standard observability and security controls. That remains valuable, but AI changes both the workload and its consumers.

ML engineers need experiment tracking, model registries, feature and data access, and specialized compute. Application teams need stable inference endpoints and predictable latency. Security teams need controls for model provenance, prompt injection, data leakage, and non-human identities. FinOps teams need to attribute expensive training and inference usage. AI agents themselves become platform consumers that request tools, credentials, and runtime actions.

The CNCF discussion of evolving platform engineering for AI-native workloads captures this expansion through capabilities such as GPU and TPU allocation, model serving, MCP gateways, agentic guardrails, embedded FinOps, and policy-driven governance. The important organizational point is that these should not become an isolated platform owned by a small AI team. They should become extensions of the same product model, interfaces, and control philosophy used by the enterprise platform.

Define a platform contract, not a catalog of tools

A platform contract describes what a product team can request, what evidence it must provide, what the platform guarantees, and which controls are automatically applied. It is stronger than a service catalog entry and more flexible than a single mandatory implementation.

For an AI-native workload, that contract should cover at least six dimensions.

1. Governed data access

The path should make data classification, residency, retention, and permitted use visible before a workload reaches production. A request for a dataset should resolve to an approved identity, purpose, environment, and audit trail. The platform can automate access, but the product team remains accountable for whether the data is appropriate for the use case.

2. Compute and accelerator intent

Teams should request capabilities rather than hard-code a particular GPU model into every manifest. The contract can express workload class, memory, performance objective, duration, geographic constraints, and cost ceiling. Kubernetes mechanisms such as Dynamic Resource Allocation can support more structured resource claims, but the platform still needs policy for quotas, scarcity, preemption, and approved hardware profiles.

3. Model, prompt, and artifact provenance

Container images are not the only production artifacts. The platform must track model version, source, license, evaluation result, prompt bundle, retrieval configuration, tool definitions, and deployment policy. Promotion should be based on an immutable set of linked artifacts, not a model name copied into an environment variable.

4. Evaluation as a release gate

AI quality is probabilistic and context-dependent. A successful build does not prove production fitness. Golden paths should provide standard evaluation suites for task quality, safety, latency, robustness, and cost. Teams can add domain-specific tests, while the platform supplies the execution environment, evidence format, thresholds, and promotion workflow.

5. Runtime identity and guardrails

An inference service or autonomous agent needs a workload identity, scoped data access, approved tools, network boundaries, and observable policy decisions. The contract should distinguish a human user’s authority from an agent’s delegated authority. It should also define what happens when a model, tool, or policy is unavailable rather than allowing silent fallback to an uncontrolled path.

6. Cost and operational accountability

AI infrastructure introduces different cost behavior from ordinary stateless services. Training jobs can consume scarce capacity in bursts. Inference cost depends on model choice, token volume, batching, cache efficiency, and service-level objectives. Cost attribution and budgets should therefore be part of provisioning and release decisions, not a dashboard reviewed after the invoice arrives.

What an AI-native golden path looks like

A useful golden path follows the product lifecycle rather than exposing a collection of disconnected infrastructure forms.

  1. Declare the workload. The team selects an archetype such as batch training, online inference, retrieval-augmented generation, or tool-using agent. It declares data class, expected scale, latency objective, risk tier, and ownership.
  2. Provision an isolated workspace. The platform creates namespaces, identities, network boundaries, secrets references, storage, accelerator claims, quotas, and standard telemetry.
  3. Develop with approved building blocks. Teams consume versioned model endpoints, registries, feature services, MCP or tool gateways, and evaluation templates through stable APIs.
  4. Produce evidence. CI records model and data lineage, software dependencies, evaluation results, policy decisions, security findings, and predicted operating cost.
  5. Promote as a release set. GitOps promotes the linked application, model, prompt, policy, and tool configuration together. A rollback restores the complete known-good set.
  6. Operate with continuous feedback. Runtime telemetry covers service health, model quality indicators, policy denials, data drift, tool calls, accelerator utilization, and unit economics.

This lifecycle gives specialists room to innovate without forcing every product team to assemble the control plane themselves.

Avoid the separate AI platform trap

A dedicated AI enablement team may be necessary, but a separate delivery system should not be the default. Parallel identity models, pipelines, policy engines, and observability stacks increase cost and weaken governance. They also create a handoff between application engineers and AI specialists exactly where the product needs shared accountability.

A better operating model separates platform ownership by capability while preserving one product contract:

  • The core platform team owns common interfaces, workload identity, delivery workflows, policy integration, and the developer experience.
  • The AI platform capability team owns model-serving patterns, evaluation services, accelerator profiles, registries, and AI-specific runtime controls.
  • Data teams own governed data products and access semantics.
  • Security and risk teams define control objectives and approval boundaries as policy and evidence requirements.
  • Product teams own business fitness, domain evaluations, production outcomes, and accepted residual risk.

The teams collaborate through APIs, schemas, policy bundles, and service-level objectives rather than tickets and undocumented exceptions.

Measure whether the contract creates value

An AI-native platform should not be measured by the number of services in its catalog. Measure whether teams can deliver trustworthy outcomes faster.

  • Time from approved use case to first governed experiment
  • Time from candidate model to production release
  • Percentage of releases with complete model, data, prompt, and policy provenance
  • Evaluation failure escape rate
  • Percentage of agent tool calls using approved identities and gateways
  • Accelerator utilization and queue time by workload class
  • Inference cost per business transaction
  • Rollback time for a complete AI release set
  • Adoption and exception rates for each golden path

These metrics reveal whether the platform improves flow and control together. High adoption with slow delivery signals an overloaded path. Fast delivery with weak evidence signals unmanaged risk.

A practical 90-day starting point

Do not begin by designing a universal AI platform. Choose one real workload and use it to define the minimum viable contract.

Days 1–30: map the lifecycle

  • Select one representative AI product with a committed owner.
  • Map every artifact, identity, environment, approval, and operational dependency.
  • Classify which existing platform capabilities can be reused and where AI-specific gaps exist.
  • Define the workload’s risk tier, evaluation evidence, and cost objectives.

Days 31–60: build one vertical path

  • Create one workload template and governed workspace.
  • Connect model and prompt provenance to the existing GitOps release flow.
  • Add standard telemetry, policy checks, evaluation execution, and cost labels.
  • Document escape hatches with owners, expiry dates, and review requirements.

Days 61–90: prove and productize

  • Run a production-like release and rollback.
  • Measure lead time, evidence completeness, operational quality, and unit cost.
  • Interview the platform consumers and remove unnecessary steps.
  • Publish the contract as versioned schemas, APIs, examples, and service-level expectations.

The platform becomes the organizational control surface

AI-native platform engineering is not a race to add GPUs and model registries to an internal portal. It is the work of extending a proven product contract across a more complex value stream.

The strongest platforms will preserve what already works: product thinking, self-service, golden paths, policy automation, and composable cloud-native interfaces. They will add the missing contracts for data, models, evaluations, agents, specialized compute, and cost. That approach avoids a new silo while giving teams a credible path from experimentation to governed production.

Sources

Veröffentlicht am

Small Language Models for Platform Engineering: Why 8B Parameters Beat API Dependencies

The economics of AI in platform engineering are shifting — fast. For the past two years, the default answer to „how do we add AI to our internal platform?“ has been „call an API.“ But with inference costs rising, data governance getting stricter, and a new generation of compact models matching much larger counterparts on critical benchmarks, that default is worth questioning. Small Language Models (SLMs) — particularly in the 7B–9B parameter range — have reached a threshold where they can handle the majority of platform engineering workloads without ever leaving your network.

The Benchmark Reality Check: 8B Is Not a Compromise

IBM’s Granite 4.1 8B, released in April 2026 under Apache 2.0, is a useful anchor for this conversation. On enterprise coding benchmarks, the 8B model matches IBM’s own 32B Mixture-of-Experts (MoE) variant. On HumanEval pass@1, the 8B scores 87.2% compared to 89.6% for the 30B model — a gap of less than 3 percentage points that is largely irrelevant for the deterministic, constrained tasks that platform teams actually run.

This pattern holds across the SLM landscape:

  • Phi-4 (14B) — Microsoft’s model excels at reasoning-heavy tasks, punching well above its weight on MATH and GPQA
  • Qwen-3 (8B) — Strong multilingual coding support, excellent for polyglot infrastructure codebases
  • Llama-3.3 (8B) — Meta’s workhorse, widely supported across inference frameworks
  • Mistral-Small (22B) — A good middle ground when you need more capacity without the frontier price tag

The takeaway: if you are still reaching for GPT-4 or Claude Sonnet to answer „why is this Helm chart failing?“ you are likely overspending.

Dense Non-Thinking Architecture: Why It Matters for Operations

Granite 4.1 uses what IBM calls a Dense Non-Thinking Architecture. In practice, this means the model does not execute an internal chain-of-thought (CoT) reasoning step before responding. For frontier models solving novel math problems, CoT is valuable. For a platform engineer asking „summarize this PagerDuty alert and suggest the top three actions,“ CoT overhead is pure latency and token cost with zero benefit.

Platform tasks are largely pattern-matching with context, not novel reasoning. Alert triage, PR description generation, runbook execution, code review comments — these are well-defined, repetitive, structured tasks where a fast, confident response beats a slow, deeply deliberative one. Dense models optimized for inference speed are a natural fit.

The FinOps Case: What Self-Hosting an 8B Model Actually Costs

Let’s put numbers on this. A mid-tier platform team might generate 50,000 LLM calls per month for internal tooling: PR review summaries, alert enrichment, documentation queries, CI/CD pipeline diagnostics.

At $0.002 per 1K tokens (input + output average), 50,000 calls at ~500 tokens each = $50/month in API costs. Manageable — until agents arrive.

Agentic workflows are not single API calls. A single „investigate this alert“ agent might issue 15–25 tool calls, each with full context. That same 50,000-event scenario becomes 750,000–1,250,000 LLM calls. At $0.002/1K tokens, that is now $1,500–$2,500/month — and growing linearly with adoption.

Self-hosting an 8B model on a single RTX 4090 (~$1,800 hardware) or a Mac Studio M4 Max (~$2,000) delivers:

  • ~30–50 tokens/second throughput (sufficient for internal tooling)
  • Zero marginal cost per call after hardware amortization
  • Full data residency — no tokens leave your network
  • Instant availability without rate limits or provider outages

At an agentic scale, the hardware pays for itself within 1–2 months. Beyond that, it is pure savings.

Platform Engineering Use Cases Where SLMs Shine

1. Alert Triage and Runbook Execution

The HolmesGPT pattern (CNCF Sandbox) demonstrates the right approach: give an SLM access to kubectl, PromQL, and Loki, and a structured Markdown runbook. With a well-crafted runbook, tool calls per investigation drop from 16+ to 2–4. An 8B model running locally handles this at millisecond latency with no data leaving the cluster.

2. CI/CD Pipeline Assistance

PR description generation, test coverage summaries, changelog drafting — these are low-complexity, high-volume tasks. An SLM integrated directly into your CI/CD pipeline (via Ollama’s REST API or a vLLM endpoint) can run as a pipeline step without any external dependency. No API key rotation. No rate limiting during a big release crunch.

3. Code Review Comments

Automated first-pass code review — style enforcement, security pattern flagging, documentation gaps — is exactly the kind of task where an 8B model is sufficient. The model does not need to understand your entire business domain; it needs to apply consistent rules to code diffs. Fine-tuning on your internal codebase further improves relevance.

4. Documentation and Runbook Generation

Keeping runbooks current is a perennial platform team pain point. An SLM that can read infrastructure-as-code, observe recent incident patterns, and generate or update Markdown documentation solves a real operational problem — without requiring a cloud API call for every update.

Enterprise Trust: Granite’s Compliance Credentials

IBM Granite 4.1 ships with two features that matter disproportionately in regulated industries: Guardian Models and cryptographic signing.

Guardian Models are companion classifiers that can check model inputs and outputs for compliance — harmful content, PII exposure, prompt injection attempts. This is built into the model ecosystem, not bolted on afterward. For financial services or healthcare platform teams, this is a significant differentiator versus a generic open-source model.

The cryptographic signing (with ISO certification) means you can verify model provenance. In an era where supply chain security is central to platform governance (see SLSA, Sigstore, in-toto), being able to verify that the model running in your cluster is exactly the model IBM published is not a minor detail.

The Multi-Model Strategy: SLM + Cloud for 80/20 Coverage

The most practical approach is not „replace all cloud APIs with SLMs“ — it is to route intelligently:

  • ~80% of tasks → Local SLM: Alert triage, CI/CD assistance, doc generation, code review, runbook execution, structured queries against internal data
  • ~20% of tasks → Cloud frontier model: Novel architecture decisions, complex multi-step reasoning, tasks requiring broad world knowledge not captured in your fine-tuned model

This mirrors how mature platform teams already think about compute: use the right tool at the right cost tier. An internal platform that routes requests based on complexity signals (task type, token budget, confidence threshold) gives you both cost efficiency and capability headroom.

Getting Started: Self-Hosting in the Platform Engineering Stack

The barrier to running an 8B model is lower than most teams expect:

  • Ollama — Single-command model serving, REST API, model library with one-line pulls (ollama pull granite3.3:8b)
  • LM Studio — Desktop GUI for evaluation, good for initial benchmarking before committing to infrastructure
  • vLLM — Production-grade serving with OpenAI-compatible API, batching, and quantization support; the right choice for Kubernetes-native deployments

For Kubernetes, vLLM running as a Deployment with a GPU node selector and an HPA on request queue depth is a reasonable production starting point. Pair it with an OpenAI-compatible API shim and your existing LLM-integrated tooling requires zero code changes to switch endpoints.

The Connection to Agentic Infrastructure

The Agentic Compute Cliff is real: GitHub Copilot paused new signups in April 2026 due to capacity constraints, and multiple cloud providers are experiencing GPU shortages. As agentic workloads scale — where a single developer workflow might trigger hundreds of LLM calls per hour — dependency on cloud inference is a reliability and cost risk.

SLMs running on internal infrastructure are not just a cost play. They are a resilience play. Your internal platform keeps working when the cloud provider has an outage. Your agents are not rate-limited during a major incident response. Your data never transits a network boundary you do not control.

When 8B Is Not Enough

Intellectual honesty matters here. SLMs are not the answer for everything:

  • Novel architecture decisions requiring broad reasoning across domains
  • Complex multi-step debugging across large, unfamiliar codebases
  • Tasks requiring deep world knowledge beyond your training/fine-tuning window
  • High-stakes customer-facing generation where quality variance is unacceptable

The skill is in classification — building a platform that knows when to route locally and when to escalate to a frontier model. That routing logic, often just a simple task classifier, is itself a good candidate to run on a local SLM.

Conclusion: Make the Economics Argument

The conversation about SLMs in platform engineering is no longer theoretical. The benchmarks have arrived. The tooling (Ollama, vLLM, LM Studio) is mature. The hardware cost is justified within months at agentic scale. And the privacy and compliance benefits — data residency, Guardian Models, cryptographic provenance — increasingly matter as organizations bring AI deeper into their software delivery lifecycle.

The 8B parameter class is not a compromise. It is a deliberate choice that aligns cost, performance, privacy, and operational simplicity for the tasks that platform teams actually run. Start with one use case — alert triage is a natural first target — measure the results, and expand from there. The API dependency you are paying for today may be entirely optional.