Working with agents
Reliability work includes availability, performance and recovery. When agents interact with a system, also check what they can access, which actions they take and whether their results are useful.
A service can stay reachable while an agent repeatedly calls a failing tool or produces incorrect results. Choose monitoring that can reveal those problems and agree when a person should intervene.
Distinguish an agent helping investigate an incident from an agent allowed to change production. Each needs clear access limits, review and recovery arrangements.
Experience to discuss
Discuss monitoring, incident response and recovery using examples from relevant systems. Ask how the person would notice a failure, communicate its impact and check that the system has recovered. Add questions about agent behavior and cost when the role covers them.
Responsibilities and skills
Purpose
Own observability, cost measurement, failure recovery, and guardrail enforcement across both traditional infrastructure and agent operations.
Responsibilities to discuss
- Define and implement the Guardrail Stack (all five layers) in collaboration with the AI Architect
- Monitor cost per agent execution and investigate unexpected spending
- Build observability for agent operations: execution traces, token usage, failure rates, and latency per agent workflow
- Detect and recover from incorrect outputs, actions outside scope, repeated retries and cost overruns
- Run incident response for agent-related failures, including postmortems that improve guardrails, not only runbooks
- Implement and enforce policy guardrails: secret scanning, PII filtering, safety classification, and dependency policies
- Agree service-level objectives for agent tasks, including quality, response time and cost
- Enforce governance policies at runtime (Layer 5): monitor agent compliance with registry rules, access boundaries, and cost budgets; escalate violations
Skills for this work
These role-specific skills complement the five shared competencies. Choose examples relevant to the work and support people as they practice.
- Agent failure mode expertise — Recognize incorrect outputs, repeated actions and scope errors that ordinary health checks may miss.
- Observability design — Monitor the quality and behavior of agent workflows as well as service availability.
- Cost engineering — Attribute costs to workflows, set useful budget alerts and compare savings with any effect on quality.
- Guardrail implementation — Translating policy into automated enforcement, from secret scanning and PII detection to safety classification and dependency rules.
- Incident response for AI systems — Adapting detection, communication, and postmortem practice when the trigger is an agent workflow rather than a failed deploy.
- Governance enforcement — Check access, ownership records and budgets during operation. Coordinate with the people who maintain the shared infrastructure.
Compare the skills in this guide and choose an area to discuss or practice.
Signals that need more context
Use work examples alongside these signals; none is a complete measure of someone’s ability.
- Purely infrastructure-focused experience with no application-layer or data-flow awareness
- Expertise limited to container orchestration and CI/CD pipelines without ownership of behavioral or economic SLOs
- Incident response habits that assume deterministic failure modes and static blast-radius models
- Cost management confined to compute, storage, and network with no fluency in token economics and agent run patterns
- Monitoring strategies that stop at binary up/down checks and miss drift, abuse, and quality erosion
Questions to discuss
Adapt these example questions to the role and the person’s opportunities to do the work.
- Incident scenario — "An agent generated and merged a pull request overnight that passes all tests but introduced a subtle security vulnerability. Walk through your detection and response process."
- Observability design — "Design the monitoring dashboard for a team running five different agent workflows. What metrics do you track? What alerts do you set?"
- Cost analysis — "Agent costs increased three hundred percent this month. Walk through your investigation and mitigation approach."
- Guardrail design — "Define the guardrail stack for an agent with access to a production database. Which layers do you implement, and in what order?"
- Failure mode analysis — "List five ways an autonomous coding agent can fail that a traditional CI/CD pipeline would not catch."
An example day
Illustrative scenario. Use this example to discuss how the responsibilities fit together.
A workflow passes its health checks, but its total cost has risen. You trace repeated calls to a tool that times out. You limit retries, confirm the workflow still completes useful tasks and add an alert for the total spending pattern.
Later, you review an incident caused by a migration that failed on production data. You work with engineering on the fix, recovery steps and a test for the missing case, then update the runbook.
Related HELM guidance
The AI Reliability Engineer responsibilities connect to the Guardrail Stack. Use the Decision Rights Matrix to agree who can approve access, accept risk and decide on recovery.
Silent Quality Drift and Governance Gap describe problems to investigate. Principle 4 asks teams to put appropriate limits and checks in place before expanding agent use.