Guardrailed Agentic Kubernetes Operations Use Read-Only Tools and Approval for exec

Why Kubernetes Exec Is Secret-Adjacent in Agentic Operations

When teams connect AI agents to Kubernetes, they often start the security review with one question: Can this agent read Kubernetes Secrets? That is a good question, but it is not sufficient. An agent may be blocked from the Kubernetes Secrets API and still be able to expose sensitive data indirectly through exec. That makes Kubernetes exec a secret-adjacent capability. The misleading comfort of blocked Secret API access Imagine a Kubernetes tool service account with this posture: get/list/watch secrets: no create pods/exec: yes At the API layer, this looks reassuring. The agent cannot run: kubectl get secrets kubectl describe secret But if the same agent can exec into a running pod, it may be able to inspect what the container can see at runtime. That can include: mounted Secret volumes service account tokens environment variables application config files credentials cached on disk cloud identity tokens files written by init containers or sidecars So the real posture is: Kubernetes Secret API access: blocked Runtime secret exposure through pod exec: possible That distinction matters a lot when the user interface is natural language. Why exec is powerful Exec is not one permission. It is a door into a process environment. Even harmless-looking prompts can become risky: Can you check the environment of this pod? Can you cat the config directory? Can you print the token file? The Kubernetes API may not expose Secrets to the agent, but the application container might. This is why pods/exec should not be treated like a normal read-only permission. It is interactive runtime access. Practical risk classification For agentic Kubernetes workflows, I like to classify exec requests separately from ordinary reads. Green: Pure Kubernetes API reads Yellow: Exec with safe diagnostic intent Red: Exec that may expose secrets, credentials, tokens, or broad filesystem state Examples: Request Suggested Risk Reason List pods Green Kubernetes API read Describe deployment Green Kubernetes API read Get pod logs Green/Yellow Logs may contain sensitive data Exec uname -a Yellow Runtime access, but narrow output Exec ps aux Yellow Process details may be sensitive Exec printenv Red Environment variables often contain secrets Exec cat /var/run/secrets/... Red Explicit token/secret access Exec find / -type f Red Broad filesystem inspection This model is intentionally simple. It is designed to make risk visible before execution. Use approval gates, not just polite prompts An LLM asking "Do you approve?" is not an enforcement mechanism. For sensitive tools, the platform should pause the actual tool call and require a real approval decision before execution. A better flow: User: Run uname in this pod. Agent: Risk: Yellow Reason: pod exec enters a live container, but the requested command is narrow. Approval gate: Approve / Reject Tool: Runs only after approval. For Red requests, the platform can either block them outright or require a stronger approval path. Add command-level guardrails Start with obvious patterns: secret token password credential kubeconfig printenv /var/run/secrets This will not catch everything, but it catches the requests that should never slip through casually. Then add context: Is the namespace sensitive? Is the pod part of the platform control plane? Is the user asking for broad filesystem access? Is the command trying to dump environment variables? Is the command wrapped in a shell? An exec request is not just a command. It is a command plus a target plus a user plus an operational context. RBAC still matters Guardrails do not replace Kubernetes RBAC. RBAC should still be scoped tightly: avoid broad cluster-wide exec permissions prefer namespace scoping where possible separate read-only tools from write/exec tools use different service accounts for different risk tiers keep destructive tools out of default agents The agent should not have more power than the platform team is willing to expose through a normal operational workflow. Final thought Agentic Kubernetes platforms should treat exec with respect. It is not the same as get pods. It is closer to temporary access into the runtime boundary of an application. Sometimes that access is exactly what an incident needs, but it should be visible, approved, logged, and risk-classified. The safest rule is simple: If an agent can exec into a pod, assume it may be able to see what that pod can see.

2 hours ago

Guardrails Before Write Access: Building Agentic Kubernetes Operations with Human Approval

Agentic operations for Kubernetes sound exciting until the agent asks to run a production command. That is the moment the architecture stops being a demo and starts becoming a platform problem. It is not enough to connect an LLM to Kubernetes tools. A real platform needs boundaries: What can the agent read? What can it change? Who approves risky actions? Can the agent retrieve secrets? How do we stop a chat interface from becoming a cluster-admin shortcut? How do we prove the request actually went through the intended path? This post walks through a practical pattern for building a guarded Kubernetes agent workflow using chat-based invocation, MCP tools, risk classification, RBAC, and human approval. The examples are intentionally anonymized and use generic names, but the lessons come from real implementation work. The platform shape The architecture had three layers: Chat surface | | user asks for inspection/remediation v Agent runtime | | calls approved tools through MCP v Kubernetes tool server | | uses a scoped Kubernetes service account v Application cluster The chat surface can be Slack, Microsoft Teams, or another collaboration tool. The important part is that the agent is invoked where incident response already happens. The agent runtime is responsible for interpreting the request, selecting tools, summarizing results, and asking for approval when needed. The Kubernetes tool server is the dangerous part. It is where natural language becomes Kubernetes API access, so it must be designed as a platform boundary rather than a convenience script. Start with read-only operations The safest starting point is read-only inspection: list pods describe deployments read events fetch logs inspect services, ingresses, gateways, and routes compare resource state across namespaces or clusters This already supports many incident workflows. During an incident, a useful agent does not need to delete a pod immediately. It first needs to gather context quickly: What changed? Which pods are unhealthy? Are there recent restarts? Are events showing image pull errors, scheduling failures, or failed probes? Is the service routing to ready endpoints? Is this isolated to one namespace, one cluster, or many clusters? Read-only tools let the team gain speed without handing the agent mutation rights on day one. Add write tools behind approval Eventually, teams will want remediation actions: restart a deployment scale a workload patch a resource execute a command in a container apply a manifest delete a stuck pod These tools should not be exposed as ordinary chat commands. They should be protected tools with explicit approval. A good flow looks like this: User: Run a diagnostic command in this pod. Agent: Risk: Yellow Operation: pod exec Target: namespace/app-pod Reason: exec is read-only in intent, but the target namespace is sensitive. Approval gate: Approve / Reject Agent: Runs the tool only after approval and returns the result. The agent should explain the risk before the tool runs, but the actual enforcement should live in the platform approval mechanism. That distinction matters. A model politely asking "Do you approve?" is not the same as an approval gate that pauses the tool call. Use simple risk classes You do not need a perfect policy engine on the first day. A simple risk model is already useful: Green Low-risk read-only inspection Yellow Sensitive read-only action or limited operational change Red Destructive, secret-adjacent, broad, or hard-to-reverse action Example classification: Request Risk Why List pods in a namespace Green Read-only inspection Get logs from an application pod Green/Yellow Depends on log sensitivity Exec uname -a in a pod Yellow Exec is powerful even if command is harmless Scale a deployment Yellow Mutates live workload capacity Delete a pod Red or Yellow Depends on namespace, controller, and blast radius Print environment variables Red Secret-adjacent Read mounted service account token Red Secret exposure The goal is not to make the agent "smart" in a vague way. The goal is to make risk visible before execution. Treat exec as secret-adjacent One of the most important lessons: blocking Kubernetes Secret API access is not enough. A service account may have: get/list/watch secrets: no create pods/exec: yes At first glance, that sounds safe. The agent cannot call kubectl get secret. But exec can still enter a running container. From inside that container, a user might read: mounted secret files service account tokens environment variables app config files credentials written to disk So the real security posture is: Kubernetes Secret API access: blocked Secret exposure through pod runtime/exec: still possible That does not mean exec must be forbidden everywhere. It means exec deserves special handling: require approval classify suspicious commands as Red block obvious secret-reading patterns scope RBAC carefully prefer diagnostic commands with predictable output avoid exec into sensitive namespaces unless there is a clear reason Good guardrails understand that Kubernetes permissions and runtime access are related but not identical. RBAC is part of the product experience When an agent fails to execute a command, the model may guess why. For example, it might say: The operation may have been blocked by policy or GitOps drift prevention. But the actual reason could simply be Kubernetes RBAC: cannot patch resource "deployments/scale" That distinction matters. If the agent guesses, users lose trust. The platform should capture and surface the real tool error: RBAC denied resource not found container binary not found namespace blocked by tool guardrail admission denied network timeout invalid command arguments The agent can summarize, but it should not invent the root cause when the tool already returned one. Distroless containers change your demo plan Another practical lesson: many containers do not have a shell. Commands like these may fail: sh -c "..." bash ps curl That does not mean exec is broken. It may just mean the target image is distroless or minimal. For a reliable demo or diagnostic workflow: pick commands that exist in the target image do not assume a shell is available test against known-good pods treat "executable file not found" as a container/runtime fact, not an agent failure This is mundane, but it saves a lot of confusion. Chat permissions need their own boundary If an agent is available from a collaboration tool, the chat surface also needs access control. There are two separate layers: Where the app can be installed or invoked. Which teams, channels, or conversations the bot will actually serve. The first layer is app configuration. The second layer is runtime enforcement. For sensitive operational agents, runtime allowlists are useful: ALLOWED_TEAM_IDS ALLOWED_CHANNEL_IDS ALLOWED_CONVERSATION_IDS The agent should reject messages and approval actions from unapproved locations. This matters because an approval button is also an operational action. It should not be usable from a random personal chat or copied into an uncontrolled place. Thread context is subtle Chat threads look obvious to humans, but bot frameworks may not automatically provide the full thread history to the bot. If the agent is mentioned halfway through a long incident thread, it may only receive the message where it was tagged, not every previous reply. That means the platform has to choose: require users to include relevant context in the prompt fetch thread history through the chat provider API summarize thread history before passing it to the agent map each chat thread to a separate agent context For incident response, the safest model is: one chat thread = one agent context That prevents unrelated incidents from accidentally sharing memory. What I would build first For an early production version, I would keep the first iteration intentionally conservative: read-only Kubernetes tools enabled by default write tools disabled until RBAC and approvals are ready exec treated as approval-required secret-looking commands classified as Red chat channel allowlists enabled one chat thread mapped to one agent context real Kubernetes errors surfaced clearly all tool calls logged with actor, target, risk, and decision Then I would add automation gradually: Read-only triage. Approved exec for diagnostic commands. Approved rollout restart or scale operations. Policy-backed remediation. Pull-request based fixes for configuration changes. Final thought Agentic Kubernetes operations should not start with "let the AI fix production." They should start with: Can we help humans gather context faster, classify risk clearly, and execute approved actions through a controlled platform path? That framing is less flashy, but it is much closer to how platform teams can actually adopt agentic systems. The interesting work is not just connecting an LLM to Kubernetes. The interesting work is building the boundaries around it.

2 hours ago