Category Archives: ai

Testing Kubernetes sandboxing technologies

Posted on by Carlos Sanchez
Reply

There are several flavours of “give a workload its own isolated runtime on Kubernetes” floating around right now. We wanted a head-to-head comparison — not a slide-deck comparison, but one with real numbers against a non-trivial workload that exercises identity routing, port forwarding, WebSocket upgrades, and a heavy long-lived process. Playwright driving Chromium is exactly that workload. This post walks through how we tested four sandboxing technologies — agent-sandbox, OpenShell, substrate, and KarsSandbox — against the same Playwright harness, what we found, and what each option is actually good at.

The harness and bench live at github.com/carlossg/playwright-k8s-sandbox. The deep-dive architecture doc with full sequence diagrams is at docs/ARCHITECTURE.md.

Why Playwright as the test workload

Most sandboxing demos use traefik/whoami or nginx. Both are useful for a smoke test and useless for telling sandboxing options apart, because they don’t stress anything. Real agent workloads do. Playwright gives us, in one process tree:

  • A long-lived stateful process (Chromium with ~6 child processes, hundreds of MB of memory).
  • A native WebSocket protocol (chromium.connect(wsEndpoint)), which requires HTTP upgrade handling end-to-end through the data plane.
  • A non-trivial cold-start cost that’s measurable and stable (Node + Chromium boot is ~3 seconds).
  • A real possibility of checkpoint/restore value — Chromium with a warm page tree is genuinely expensive to recreate, so the snapshot story is interesting to test rather than theoretical.
  • A clear correctness oracle — page.goto(url) either fetches the page or it doesn’t.

So each “test” in our harness is: instantiate a sandbox per tenant, get a Playwright client to connect over WebSocket, open a page, fetch a URL, measure each phase.

The four sandboxing technologies under test

Sandbox unitIsolationPersistence model
agent-sandboxPod from a SandboxWarmPool, bound by a SandboxClaim CRDPluggable per RuntimeClass — runc by default, gVisor or Kata if you point the SandboxTemplate at the corresponding RuntimeClassStateless. Claim is the pod’s lifecycle.
OpenShellSame machinery as agent-sandbox; with added process level isolationStateless.
substrategVisor sandbox on a worker pod, managed as an “Actor”gVisor (runsc, systrap platform); built in, not pluggableDesigned for full sandbox checkpoint/restore to S3.
KarsSandboxNamespaced pod per KarsSandbox CR (KARS controller)Namespace-level isolation + optional Azure runtime sandboxingStateless. CR deletion destroys both namespace and pod.

The first two are mechanically identical — same CRDs, same controller — and both can run with runc, gVisor (runsc), or Kata Containers by pointing the SandboxTemplate at the appropriate RuntimeClass. We ran them with the cluster default (runc) for the bench.

KarsSandbox takes a different approach: each sandbox gets its own dedicated namespace (not just a pod), providing stronger isolation boundaries and compatibility with Azure-specific runtime features like InferencePolicy for AI/GPU workloads. Unlike agent-sandbox’s warmpool model, KARS provisions sandboxes on-demand.

The interesting comparison isn’t really “container vs gVisor” — multiple models can do gVisor — it’s warmpool of pre-bound pods vs on-demand namespace provisioning vs substrate’s actor lifecycle with snapshot/restore.

The test harness

To make the comparison a bit similar we built a small proxy that abstracts the four backends behind one interface. Each backend implements Ensure(id) → Endpoint + Delete(id); the proxy handles caller identification, session caching, idle reaping, and WebSocket upgrade forwarding identically across all four. That way, when we compare bench numbers, we’re comparing the sandboxing technology, not four different ad-hoc client implementations.

┌─ test client pod ┐                ┌─ proxy ─────────┐                ┌─ backend ─────────┐
│  labels:         │                │ identify        │                │  one of:          │
│   playwright-id  ├──HTTP / WS────▶│ session.Manager ├──Ensure(id)───▶│  - SandboxClaim   │
│   = bench-X      │                │  (singleflight) │                │  - SandboxClaim   │
└──────────────────┘                │ reverse proxy   │                │  - Actor (gRPC)   │
                                    │ idle reaper     │                │  - KarsSandbox CR │
                                    └────────┬────────┘                └─────────┬─────────┘
                                             │                                   │
                                             │                         ┌─────────▼─────────┐
                                             └──HTTP / WS upgrade─────▶│ Chromium sandbox  │
                                                                       └───────────────────┘

The proxy identifies callers by pod label: the test client sets metadata.labels.playwright-id on its Deployment, the proxy looks up the caller’s pod IP via a client-go informer and resolves it to that id. No agent-side SDK, no token plumbing — just one label. Each unique id gets its own sandbox.

Three scenarios per backend:

ScenarioSetupMeasures
coldDelete any prior sandbox, then connect for the first time.Full provisioning cost: CreateClaim/CreateActor + Resume + WS upgrade + handshake.
warmConnect again with the same id, sandbox still alive.Steady-state cost: proxy hop + WS upgrade only.
restoreOut-of-band suspend (substrate) or wipe (sandboxclaim), then a fresh request.The persistence story: does the sandbox come back faster than a cold start?

What the agent-sandbox / OpenShell flow looks like

Both back ends share the same CRD lifecycle: the proxy creates a SandboxClaim, the agent-sandbox controller picks a warm pod from the pool, binds it to the claim, and the proxy gets back an endpoint.

https://viewscreen.githubusercontent.com/markdown/mermaid?docs_host=https%3A%2F%2Fdocs.github.com&color_mode=light#b580a2bf-2583-4a74-a63f-7f41726379e1

There is no checkpoint/restore in this model. A claim’s life is the sandbox’s life; deleting the claim destroys the pod, and the next call for the same id gets a fresh warm pod from the pool. The “restore” scenario therefore re-creates the claim and behaves identically to cold. The interesting question this design answers well is: how cheap can a cold-start be when you have warm capacity pre-allocated? Answer below.

OpenShell’s flow is the same shape; the only difference is the added process isolation and OpenShell features.

What substrate’s flow looks like

Substrate is a different beast. Each tenant gets an “Actor” living inside a gVisor sandbox on a worker pod. The data plane is atenet-router (Envoy with an ext_proc filter) which dispatches to the right worker pod by Host: <actor-id>.actors.resources.substrate.ate.dev. Actor lifecycle (Create, Resume, Suspend, Delete) is a gRPC API on ate-api-server.

https://viewscreen.githubusercontent.com/markdown/mermaid?docs_host=https%3A%2F%2Fdocs.github.com&color_mode=light#adea08fd-13b7-4d8c-828f-6540909bfc0a

In principle, substrate gives you persistent sandboxes — suspend an actor mid-session, restore it later, and Chromium picks up where it left off with all its in-memory state intact. That’s the headline feature you don’t get from container-with-warmpool or namespace-scoped sandboxes. Whether it actually works is the interesting test result.

What KarsSandbox’s flow looks like

KarsSandbox uses Azure’s KARS (Kubernetes Azure Runtime Sandboxes) controller to provision a dedicated namespace per tenant. Each KarsSandbox CR (kars.azure.com/v1alpha1, runtime: BYO) triggers the controller to create both a namespace and the sandbox pod within it. The proxy polls status.phase=Running then locates the pod IP via the CoreV1 API.

https://viewscreen.githubusercontent.com/markdown/mermaid?docs_host=https%3A%2F%2Fdocs.github.com&color_mode=light#9d2da80b-ec46-4bc1-a2bc-e4b2bdc22d1c

Unlike agent-sandbox’s warmpool or substrate’s actor pool, KARS provisions resources on-demand. The tradeoff is no pre-warmed capacity, but you get namespace-level isolation that plays well with Azure-specific features like InferencePolicy for GPU scheduling.

Configuration:

BACKEND=karssandbox
KARS_SANDBOX_IMAGE=<your-playwright-image>     # Required: sandbox container image
KARS_INFERENCE_REF=<inference-policy-name>     # Optional: for AI/GPU workloads

State across runs: None, like agent-sandbox. A KarsSandbox CR creates a dedicated namespace and pod; when the CR is deleted (idle reap or explicit Delete), both the namespace and pod are destroyed by the KARS controller. The next caller for the same id gets a brand-new isolated sandbox. Reuse only happens while the sandbox is alive.

RBAC requirements: The proxy needs additional permissions beyond the base ClusterRole:

- apiGroups: ["kars.azure.com"]
  resources: ["karssandboxes"]
  verbs: ["get", "list", "watch", "create", "delete"]
- apiGroups: ["kars.azure.com"]
  resources: ["karssandboxes/status"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list"]  # To locate pod IP after KarsSandbox is Running

See deploy/examples/kars/ for complete deployment manifests including proxy configuration, RBAC patches, and InferencePolicy examples.

The numbers

Run on Colima 16 GiB / 6 CPU, kind 1.33, arm64. All scenarios pass (KARS results pending).

| backend       | scenario | result | connect_ms | newPage_ms | goto_ms | total_ms |
|---------------|----------|--------|-----------:|-----------:|--------:|---------:|
| agent-sandbox | cold     | PASS   |        579 |        192 |      34 |      805 |
| agent-sandbox | warm     | PASS   |         23 |         23 |      13 |       59 |
| agent-sandbox | restore  | PASS   |        544 |         37 |      14 |      595 |
| openshell     | cold     | PASS   |        556 |         42 |      19 |      617 |
| openshell     | warm     | PASS   |         23 |         32 |      13 |       68 |
| openshell     | restore  | PASS   |        549 |         47 |      16 |      612 |
| substrate     | cold     | PASS   |       3610 |         72 |      33 |     3715 |
| substrate     | warm     | PASS   |         29 |         48 |      27 |      104 |
| substrate     | restore  | PASS   |        133 |         50 |      15 |      198 |

What this comparison says:

  • Container-with-warmpool wins on cold-start by ~6×. agent-sandbox and OpenShell come in ~580–600ms cold; substrate is 3.6s. The warmpool model amortizes container start-up at provisioning time; substrate has pre-warmed worker pods, but the user workload (Node + Chromium) still boots cold inside the gVisor sandbox on every cold-start.
  • All three are essentially free at warm. 60–100ms total at warm — the proxy hop and WS handshake dominate. The choice of backend doesn’t matter once the sandbox is up.
  • Substrate’s “restore” is fast (198ms) for the wrong reason. The bench’s suspend leaves the worker pod hot — the OCI bundle is already extracted on disk — so the boot-from-spec restore reuses cached state. With working snapshot restore, this would be the most interesting cell in the table; today, it doesn’t prove much.
  • The 3.6s substrate cold-start isn’t just gVisor. A meaningful chunk of it is Node + Chromium boot itself, plus the actor lifecycle workflow (CreateActor → AssignWorker → AteletRestore → URPC into the sentry). Running agent-sandbox under gVisor via RuntimeClass would add some runsc-specific overhead to its ~580ms cold, but not the full 3s gap — the rest is substrate’s per-tenant actor setup vs agent-sandbox’s “the warm pod already exists, just bind it” model.

When to use which

Based on what testing actually surfaced:

agent-sandbox is the safe default for browser-style workloads. Sub-second cold-start, trivial to operate (one CRD, one controller, one warmpool per template), and the model is easy to reason about — claim’s life is the pod’s life. If you need gVisor or Kata isolation, swap the RuntimeClass on the SandboxTemplate; you keep the same controller and the same warmpool semantics. The OpenShell flavor demonstrates how easy it is to fork the image story without touching the controller.

OpenShell adds little value with agent-sandbox gVisor isolation. Adds process isolation when using default RuntimeClass.

substrate is the right answer when you need per-tenant snapshot/ restore — suspend an actor mid-session, ship the checkpoint elsewhere, restore later with browser state intact. That’s the capability nothing else in this comparison offers. gVisor isolation alone is not the differentiator (agent-sandbox can do that too via RuntimeClass); the actor lifecycle + S3-backed snapshots is. Today the snapshot path needs work in our environment, so we’re paying substrate’s per-tenant boot cost without yet getting the persistence benefit; once snapshot restore is reliable end-to-end, the substrate story becomes very compelling.

KarsSandbox is the choice for Azure/AKS environments where you need stronger isolation boundaries than pod-level (each tenant gets its own namespace) or integration with Azure-specific features like InferencePolicy for AI/GPU workloads. The on-demand provisioning model means no warmpool capacity planning, but cold-starts will be slower than agent-sandbox since KARS must create both namespace and pod from scratch. Best fit for multi-tenant scenarios on AKS where namespace-level RBAC and resource quotas matter, or when targeting Azure’s runtime sandbox extensions.

Try it yourself

git clone https://github.com/carlossg/playwright-k8s-sandbox
cd playwright-k8s-sandbox
./test/harness.sh up         # spin up the agent-sandbox kind cluster
./test/harness.sh up-kars    # spin up the KARS kind cluster
./test/bench.sh all          # run cold/warm/restore against all backends
./test/bench.sh kars         # run KARS-specific benchmarks

For substrate you’ll also need its own kind cluster and the ate.dev control plane installed (hack/install-ate-kind.sh in the substrate repo). The full architecture deep-dive — including the sequence diagrams, identity model, idle-reap policy, and the bench methodology — is at docs/ARCHITECTURE.md.

KARS test harness commands:

./test/harness.sh up-kars      # Create KARS cluster with controller
./test/harness.sh test kars    # Run integration tests
./test/bench.sh kars           # Run cold/warm/restore benchmarks
./test/harness.sh down-kars    # Cleanup

If you’re picking a Kubernetes sandboxing technology for a new workload, the meta-takeaway is: build a small harness around your actual workload (whatever it is), put it through cold/warm/restore on the candidates, and let the numbers + the debugging stories decide. The harness in this repo is built around Playwright; the same shape works for anything with a WebSocket or HTTP frontend.

Building a TripIt Visualizer in a few hours with Google Antigravity

Posted on by Carlos Sanchez
Reply

We travel to see the world, but we track our travels to remember it. For years, I relied on TripIt to keep my travel history organized. But the API is no longer available and in their UI is hard to find older trips and get cool stats and visualizations.

Instead of losing that history, I spent a recent holiday afternoon building a custom solution: a TripIt Data Visualization site.

The initial motivation was simple: data ownership. I knew I could still get a JSON export of my data (thanks to GDPR), but a JSON file isn’t exactly “visual.”

🌍 See it in Action (Try the Sample Data)

You don’t need your own data export to see how it works. I’ve included a “Sample Data” mode so anyone can explore the dashboard immediately.

👉 Check out the demo here: tripit.csanchez.org

A web dashboard titled "Viewing Sample Data" showing four key statistics cards: 23 total trips, 34 flights (123,420 miles), 23 unique countries visited, and 206 days traveling. Below the stats is a dark world map with visited countries like the USA, Brazil, Australia, and parts of Europe highlighted in bright blue.
A dark-themed interactive world map displaying "Flight Paths." Numerous purple arced lines connect cities across the globe, showing a dense network of flight routes primarily originating from Europe and connecting to North America, South America, Africa, and Asia.
A "Yearly Statistics" bar chart tracking travel data from 2007 to 2026. The chart uses multi-colored bars to compare Days, Flights, Miles, and Trips per year, showing a significant peak in travel around 2019 and a visible dip during 2020-2021.
A clean user interface showing a list of "Past Trips" with cards for Auckland Nature 2025, Sydney NYE 2025, Oslo Summer 2024, and Summer in Europe 2024. Each card displays the destination, date range, duration in days, and number of flights.

The Goal: Beyond the Itinerary

I wanted more than just a list of past trips. I wanted a comprehensive dashboard that felt like a mix of the best travel apps out there:

  • Countries Visited: A high-level view of global coverage.
  • Deep Statistics: Yearly and monthly breakdowns of flights, miles, and airline preferences.
  • Interactive Flight Paths: View every take-off and landing as a beautiful arc on a global map.
  • Country Tracking: Automatically highlights every country you’ve visited based on your trip history.
  • Deep-Dive Statistics: Automated breakdowns of your travels by year, month, and even specific airlines.
  • Privacy-First Architecture: Your data never leaves your browser. There is no backend server storing your history; it’s all processed locally using your tripit export.
  • Zero Friction: Don’t have your data yet? You can explore every feature using the Gemini generated Sample Data .

The Secret Sauce: Building with Google Antigravity & Gemini

The most remarkable part of this project wasn’t the code itself, but how fast it came together. The entire project took just a few hours, and complex visualizations were trivial to add. Which also hooks you into adding more and more features as it is too easy!

I used the Antigravity browser, which allowed for a feedback loop with Gemini. Instead of manually debugging CSS or layout issues, I could:

  1. Ask Gemini for a design change or a new feature.
  2. The AI would “see” the current state of the app via screenshots.
  3. It would provide the fix or the code block instantly based on the visual context.

This “visual-first” development meant I could spend more time on the logic of the data and less time wrestling with the UI.

Privacy First: Your Data Stays Yours

I wanted to ensure this tool was 100% private.

  • Since there is no API, you need to request a JSON export from TripIt (thank you, GDPR!).
  • Your data never leaves your browser. It’s processed locally, visualized, and saved for the next time you open the page.

📁 GitHub: carlossg/tripit-view

Managing the Machine: A Practical Look at Google Antigravity

Posted on by Carlos Sanchez
Reply

By now, most of us have experimented with AI coding. But while tools like Cursor have mastered the “co-pilot” experience—keeping you in the flow of writing code—Google Antigravity (released late 2025) is trying something different. It is an agentic IDE where your primary job is reviewing “Artifacts” rather than syntax.

One of the killer features is the Browser Integration.

The Browser as a “Feedback Loop”

In Antigravity, the built-in browser isn’t just for documentation; it’s a sandbox for the AI agents. When an agent implements a UI change or a new route, it doesn’t just hope the code works. It can:

  • Self-Verify: Open a headless window to check if a component actually renders.
  • Visual Debugging: Use DOM inspection to fix styling bugs that text-based models usually struggle with.
  • Artifact Generation: It produces browser recordings and screenshots so you can verify the “Mission” was successful without having to manually run the build yourself.

It’s less about “the AI browsing the web” and more about the AI having a way to prove its work to you before you merge.

The Reality of the Free Tier: Knowing Your Quotas

The biggest change in the 2026 version of Antigravity is the move to Weekly Quotas. Instead of a daily reset that might cut you off mid-task, you now have a larger “bucket” of usage that refreshes weekly.

Here is how those limits are structured for Free Tier users:

CategoryQuota TypeKey Detail
Gemini ModelsDedicated per ModelGemini 3 Pro and Gemini 3 Flash each have their own separate weekly allowance. If you run out of Pro, you can still use Flash for lighter tasks.
Claude ModelsShared PoolAll non-Google models (Claude 3.5/4.5 Sonnet and Opus) share a single combined bucket. High-intensity work with Opus will drain the same quota you use for Sonnet.
CompletionsUnlimitedStandard inline tab-completions do not count toward your weekly agent/chat limits.

The Strategy: Staggering Your Usage

Because of this “separate vs. shared” structure, the smartest way to use Antigravity is to stagger your models:

  1. Use Gemini 3 Pro for your initial architectural planning and browser-heavy UI tasks.
  2. Switch to the Claude pool only when you need a “second opinion” on complex logic or specific refactoring patterns where Anthropic’s reasoning shines.
  3. Save Gemini 3 Flash for documentation questions and basic unit test generation to preserve your “Heavy Reasoning” buckets.

Final Verdict

Antigravity isn’t a “Cursor killer”—it’s a different workflow. While Cursor is for when you want to be the pilot, Antigravity is for when you want to be the air traffic controller. Just keep a close eye on your Claude shared pool; if you’re not careful, a single complex feature build on Monday can leave you without Claude access for the rest of the week.

Self-Healing Rollouts: Automating Production Fixes with Agentic AI and Argo Rollouts

Posted on by Carlos Sanchez
Reply

Rolling out changes to all users at once in production is risky—we’ve all learned this lesson at some point. But what if we could combine progressive delivery techniques with AI agents to automatically detect, analyze, and fix deployment issues? In this article, I’ll show you how to implement self-healing rollouts using Argo Rollouts and agentic AI to create a fully automated feedback loop that can fix production issues while you grab a coffee.

The Case for Progressive Delivery

Progressive Delivery is a term that encompasses deployment strategies designed to avoid the pitfalls of all-or-nothing deployments. The concept gained significant attention after the CrowdStrike incident, where a faulty update took down a substantial portion of the internet. Their post-mortem revealed a crucial lesson: they should have deployed to progressive “rings” or “waves” of customers, with time between deployments to gather metrics and telemetry.

The key principles of progressive delivery are:

  • Avoiding downtime: Deploy changes gradually with quick rollback capabilities
  • Limiting the blast radius: Only a small percentage of users are affected if something goes wrong
  • Shorter time to production: Safety nets enable faster, more confident deployments

As I like to say: “If you haven’t automatically destroyed something by mistake, you’re not automating enough.”

Progressive Delivery Techniques

Rolling Updates

Kubernetes provides rolling updates by default. As new pods come up, old pods are gradually deleted, automatically shifting traffic to the new version. If issues arise, you can roll back quickly, affecting only the percentage of traffic that hit the new pods during the update window.

Blue-Green Deployment

This technique involves deploying a complete copy of your application (the “blue” version) alongside the existing production version (the “green” version). After testing, you switch all traffic to the new version. While this provides quick rollbacks, it requires twice the resources and switches all traffic at once, potentially affecting all users before you can react.

Canary Deployment

Canary deployments offer more granular control. You deploy a new version alongside the stable version and gradually increase the percentage of traffic going to the new version—perhaps starting with 5%, then 10%, and so on. You can route traffic based on various parameters: internal employees, IP ranges, or random percentages. This approach allows you to detect issues early while minimizing user impact.

Feature Flags

Feature flags provide even more granular control at the application level. You can deploy code with new features disabled by default, then enable them selectively for specific user groups. This decouples deployment from feature activation, allowing you to:

  • Ship faster without immediate risk
  • Enable features for specific customers or user segments
  • Quickly disable problematic features without redeployment

You can implement feature flags using dedicated services like OpenFeature or simpler approaches like environment variables.

Progressive Delivery in Kubernetes

Kubernetes provides two main architectures for traffic routing:

Service Architecture

The traditional approach uses load balancers directing traffic to services, which then route to pods based on labels. This works well for basic scenarios but lacks flexibility for advanced routing.

Ingress Architecture

The Ingress layer provides more sophisticated traffic management. You can route traffic based on domains, paths, headers, and other criteria, enabling fine-grained control essential for canary deployments. Popular ingress controllers include:

Enter Argo Rollouts

Argo Rollouts is a Kubernetes controller that provides advanced deployment capabilities including blue-green deployments, canary releases, analysis, and experimentation. It’s a powerful tool for implementing progressive delivery in Kubernetes environments.

How Argo Rollouts Works

The architecture includes:

  1. Rollout Controller: Manages the deployment process
  2. Rollout Object: Defines the deployment strategy and analysis configuration
  3. Analysis Templates: Specify metrics and success criteria
  4. Replica Sets: Manages stable and canary versions with automatic traffic shifting

When you update a Rollout, it creates separate replica sets for stable and canary versions, gradually increasing canary pods while decreasing stable pods based on your defined rules. If you’re using a service mesh or advanced ingress, you can implement fine-grained routing—sending specific headers, paths, or user segments to the canary version.

Analysis Options

Argo Rollouts supports various analysis methods:

  • Prometheus: Query metrics to determine rollout health
  • Datadog: Integration with Datadog monitoring
  • Kubernetes Jobs: Run custom analysis logic—check databases, call APIs, or perform any custom validation

The experimentation feature is particularly interesting. We considered using it to test Java upgrades: deploy a new Java version, run it for a few hours gathering metrics on response times and latency, then decide whether to proceed with the full rollout—all before affecting real users.

Adding AI to the Mix

Now, here’s where it gets interesting: what if we use AI to analyze logs and automatically make rollout decisions?

The AI-Powered Analysis Plugin

I developed a plugin for Argo Rollouts that uses Large Language Models (specifically Google’s Gemini) to analyze deployment logs and make intelligent decisions about whether to promote or rollback a deployment. The workflow is:

  1. Log Collection: Gather logs from stable and canary versions
  2. AI Analysis: Send logs to an LLM with a structured prompt
  3. Decision Making: The AI responds with a promote/rollback recommendation and confidence level
  4. Automated Action: Argo Rollouts automatically promotes or rolls back based on the AI’s decision

The prompt asks the LLM to:

  • Analyze canary behavior compared to the stable version
  • Respond in JSON format with a boolean promotion decision
  • Provide a confidence level (0-100%)

For example, if the confidence threshold is set to 50%, any recommendation with confidence above 50% is executed automatically.

The Complete Self-Healing Loop

But we can go further. When a rollout fails and rolls back, the plugin automatically:

  1. Creates a GitHub Issue: The LLM generates an appropriate title and detailed description of the problem, including log analysis and recommended fixes
  2. Assigns a Coding Agent: Labels the issue to trigger agents like Jules, GitHub Copilot, or similar tools
  3. Automatic Fix: The coding agent analyzes the issue, creates a fix, and submits a pull request
  4. Continuous Loop: Once merged, the new version goes through the same rollout process

Live Demo Results

In my live demonstration, I showed this complete workflow in action:

Successful Deployment: When deploying a working version (changing from “blue” to “green”), the rollout progressed smoothly through the defined steps (20%, 40%, 60%, 80%, 100%) at 10-second intervals. The AI analyzed the logs and determined: “The stable version consistently returns 100 blue, the canary version returns 100 green, both versions return 200 status codes. Based on the logs, the canary version seems stable.”

Failed Deployment: When deploying a broken version that returned random colors and threw panic errors, the system:

  • Detected the issue during the canary phase
  • Automatically rolled back to the stable version
  • The AI analysis identified: “The canary version returns a mix of colors (purple, blue, green, orange, yellow) along with several panic errors due to runtime error index out of range with length zero”
  • Provided a confidence level of 95% that the deployment should not be promoted
  • Automatically created a GitHub issue with detailed analysis
  • Assigned the issue to Jules (coding agent)
  • Within 3-5 minutes, received a pull request with a fix

The coding agents (I demonstrated both Jules and GitHub Copilot) analyzed the code, identified the problem in the getColor() function, fixed the bug, added tests, and created well-documented pull requests with proper commit messages.

Technical Implementation

The Rollout Configuration

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: canary-demo
spec:
  strategy:
    canary:
      analysis:
        templates:
          - templateName: canary-analysis-ai

The Analysis Template

The template configures the AI plugin to check every 10 seconds and require a confidence level above 50% for promotion:

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: canary-analysis-ai
spec:
  metrics:
    - name: success-rate
      interval: 10s
      successCondition: result > 0.50
      provider:
        plugin:
          argoproj-labs/metric-ai:
            model: gemini-2.0-flash
            githubUrl: https://github.com/carlossg/rollouts-demo
            extraPrompt: |
              Ignore color changes.

Agent-to-Agent Communication

The plugin supports two modes:

  1. Inline Mode: The plugin directly calls the LLM, makes decisions, and creates GitHub issues
  2. Agent Mode: Uses agent-to-agent (A2A) communication to call specialized agents with domain-specific knowledge and tools

The native mode is particularly powerful because you can build agents that understand your specific problem space, with access to internal databases, monitoring tools, or other specialized resources.

The Future of Self-Healing Systems

This approach demonstrates the practical application of AI agents in production environments. The key insight is creating a continuous feedback loop:

  1. Deploy changes progressively
  2. Automatically detect issues
  3. Roll back when necessary
  4. Generate detailed issue reports
  5. Let AI agents propose fixes
  6. Review and merge fixes
  7. Repeat

The beauty of this system is that it works continuously. You can have multiple issues being addressed simultaneously by different agents, working 24/7 to keep your systems healthy. As humans, we just need to review and ensure the proposed fixes align with our intentions.

Practical Considerations

While this technology is impressive, it’s important to note:

  • AI isn’t perfect: The agents don’t always get it right on the first try (as demonstrated when the AI ignored my instruction about color variations)
  • Human oversight is still crucial: Review pull requests before merging
  • Start simple: Begin with basic metrics before adding AI analysis
  • Tune your confidence thresholds: Adjust based on your risk tolerance
  • Monitor the monitors: Ensure your analysis systems are reliable

Getting Started

If you want to implement similar systems:

  1. Start with Argo Rollouts: Learn basic canary deployments without AI
  2. Implement analysis: Use Prometheus or custom jobs for analysis
  3. Add AI gradually: Experiment with AI analysis for non-critical deployments
  4. Build the feedback loop: Integrate issue creation and coding agents
  5. Iterate and improve: Refine your prompts and confidence thresholds

Conclusion

Progressive delivery isn’t new, but combining it with agentic AI creates powerful new possibilities for self-healing systems. While we’re not at full autonomous production management yet, we’re getting closer. The technology exists today to automatically detect, analyze, and fix many production issues without human intervention.

As I showed in the demo, you can literally watch the system detect a problem, roll back automatically, create an issue, and have a fix ready for review—all while you’re having coffee. That’s the future I want to work toward: systems that heal themselves and learn from their mistakes.

Resources