AI Agents Development — Production Skill Hub

Modern Best Practices (March 2026): deterministic control flow, bounded tools, auditable state, MCP-based tool integration, handoff-first orchestration, multi-layer guardrails, OpenTelemetry tracing, and human-in-the-loop controls (OWASP LLM Top 10: https://owasp.org/www-project-top-10-for-large-language-model-applications/).

This skill provides production-ready operational patterns for designing, building, evaluating, and deploying AI agents. It centralizes procedures, checklists, decision rules, and templates used across RAG agents, tool-using agents, OS agents, and multi-agent systems.

No theory. No narrative. Only operational steps and templates.

When to Use This Skill

Codex should activate this skill whenever the user asks for:

• Designing an agent (LLM-based, tool-based, OS-based, or multi-agent).

• Scoping capability maturity and rollout risk for new agent behaviors.

• Creating action loops, plans, workflows, or delegation logic.

• Writing tool definitions, MCP tools, schemas, or validation logic.

• Generating RAG pipelines, retrieval modules, or context injection.

• Building memory systems (session, long-term, episodic, task).

• Creating evaluation harnesses, observability plans, or safety gates.

• Preparing CI/CD, rollout, deployment, or production operational specs.

• Producing any template in /references/ or /assets/ .

• Implementing MCP servers or integrating Model Context Protocol.

• Setting up agent handoffs and orchestration patterns.

• Configuring multi-layer guardrails and safety controls.

• Evaluating whether to build an agent (build vs not decision).

• Calculating agent ROI, token costs, or cost/benefit analysis.

• Assessing hallucination risk and mitigation strategies.

• Deciding when to kill an agent project (kill triggers).

• For prompt scaffolds, retrieval tuning, or security depth, see Scope Boundaries below.

Scope Boundaries (Use These Skills for Depth)

• Prompt scaffolds & structured outputs → ai-prompt-engineering

• RAG retrieval & chunking → ai-rag

• Search tuning (BM25/HNSW/hybrid) → ai-rag

• Security/guardrails → ai-mlops

• Inference optimization → ai-llm-inference

Default Workflow (Production)

• Pick an architecture with the Decision Tree (below); default to workflow/FSM/DAG for production.

• Draft an agent spec with assets/core/agent-template-standard.md (or assets/core/agent-template-quick.md ).

• Specify tools and handoffs with JSON Schema using assets/tools/tool-definition.md and references/api-contracts-for-agents.md .

• Add retrieval only when needed; start with assets/rag/rag-basic.md and scale via assets/rag/rag-advanced.md

• references/rag-patterns.md .

• Add eval + telemetry early via references/evaluation-and-observability.md .

• Run the go/no-go gate with assets/checklists/agent-safety-checklist.md .

• Plan deploy/rollback and safety controls via references/deployment-ci-cd-and-safety.md .

Quick Reference

Agent Type Core Control Flow Interfaces MCP/A2A When to Use

Workflow Agent (FSM/DAG) Explicit state transitions State store, tool allowlist MCP Deterministic, auditable flows

Tool-Using Agent Route → call tool → observe Tool schemas, retries/timeouts MCP External actions (APIs, DB, files)

RAG Agent Retrieve → answer → cite Retriever, citations, ACLs MCP Knowledge-grounded responses

Planner/Executor Plan → execute steps with caps Planner prompts, step budget MCP (+A2A) Multi-step problems with bounded autonomy

Multi-Agent (Orchestrated) Delegate → merge → validate Handoff contracts, eval gates A2A Specialization with explicit handoffs

OS Agent Observe UI → act → verify Sandbox, UI grounding MCP Desktop/browser control under strict guardrails

Code/SWE Agent Branch → edit → test → PR Repo access, CI gates MCP Coding tasks with review/merge controls

Framework Selection (March 2026)

Tier 1 — Production-Grade

Framework Architecture Best For Languages Ease

LangGraph Graph-based, stateful Enterprise, compliance, auditability Python, JS Medium

Claude Agent SDK Event-driven, tool-centric Anthropic ecosystem, Computer Use, MCP-native Python, TS Easy

OpenAI Agents SDK Tool-centric, lightweight Fast prototyping, OpenAI ecosystem Python Easy

Google ADK Code-first, multi-language Gemini/Vertex AI, polyglot teams Python, TS, Go, Java Medium

Pydantic AI Type-safe, graph FSM Production Python, type safety, MCP+A2A native Python Medium

MS Agent Framework Kernel + multi-agent Enterprise Azure, .NET/Java teams Python, .NET, Java Medium

Tier 2 — Specialized

Framework Architecture Best For Languages Ease

LlamaIndex Event-driven workflows RAG-native agents, retrieval-heavy Python, TS Medium

CrewAI Role-based crews Team workflows, content generation Python Easiest

Mastra Vercel AI SDK-based TypeScript/Next.js teams TypeScript Easy

SmolAgents Code-first, minimalist Lightweight, fewer LLM calls Python Easy

Agno FastAPI-native runtime Production Python, 100+ integrations Python Easy

AWS Bedrock Agents Managed infrastructure Enterprise AWS, knowledge bases Python Easy

Tier 3 — Niche

Framework Niche

Haystack Enterprise RAG+agents pipeline (Airbus, NVIDIA)

DSPy Declarative optimization — compiles programs into prompts/weights

See references/modern-best-practices.md for detailed comparison and selection guide.

Framework Deep Dives

• Claude Agent SDK - references/claude-agent-sdk-patterns.md

Agent definition, built-in tools (Bash, TextEditor, Computer), MCP servers, guardrails, multi-agent, streaming events

• Pydantic AI - references/pydantic-ai-patterns.md

Type-safe agents, MCP toolsets, native A2A, pydantic-graph FSM, durable execution, HITL, TestModel testing

Decision Tree: Choosing Agent Architecture

What does the agent need to do? ├─ Answer questions from knowledge base? │ ├─ Simple lookup? → RAG Agent (LangChain/LlamaIndex + vector DB) │ └─ Complex multi-step? → Agentic RAG (iterative retrieval + reasoning) │ ├─ Perform external actions (APIs, tools, functions)? │ ├─ 1-3 tools, linear flow? → Tool-Using Agent (LangGraph + MCP) │ └─ Complex workflows, branching? → Planning Agent (ReAct/Plan-Execute) │ ├─ Write/modify code autonomously? │ ├─ Single file edits? → Tool-Using Agent with code tools │ └─ Multi-file, issue resolution? → Code/SWE Agent (HyperAgent pattern) │ ├─ Delegate tasks to specialists? │ ├─ Fixed workflow? → Multi-Agent Sequential (A → B → C) │ ├─ Manager-Worker? → Multi-Agent Hierarchical (Manager + Workers) │ └─ Dynamic routing? → Multi-Agent Group Chat (collaborative) │ ├─ Control desktop/browser? │ └─ OS Agent (Anthropic Computer Use + MCP for system access) │ └─ Hybrid (combination of above)? └─ Planning Agent that coordinates: - Tool-using for actions (MCP) - RAG for knowledge (MCP) - Multi-agent for delegation (A2A) - Code agents for implementation

Protocol Selection:

• Use MCP for: Tool access, data retrieval, single-agent integration

• Use A2A for: Agent-to-agent handoffs, multi-agent coordination, task delegation

Framework Selection (after choosing architecture):

Which framework? ├─ MVP/Prototyping? │ ├─ Python → OpenAI Agents SDK or CrewAI │ └─ TypeScript → Mastra or Claude Agent SDK │ ├─ Production → │ ├─ Auditability/compliance? → LangGraph │ ├─ Type safety + MCP/A2A native? → Pydantic AI │ ├─ Anthropic models + Computer Use? → Claude Agent SDK │ ├─ Google Cloud / Gemini? → Google ADK │ ├─ Azure / .NET / Java? → MS Agent Framework │ ├─ AWS managed? → Bedrock Agents │ └─ RAG-heavy? → LlamaIndex Workflows │ ├─ Minimalist / Research → │ ├─ Fewest LLM calls? → SmolAgents │ └─ Optimize prompts automatically? → DSPy │ └─ Enterprise pipeline → Haystack

Core Concepts (Vendor-Agnostic)

Control Flow Options

• Reactive: direct tool routing per user request (fast, brittle if unbounded).

• Workflow (FSM/DAG): explicit states and transitions (default for deterministic production).

• Planner/Executor: plan with strict budgets, then execute step-by-step (use when branching is unavoidable).

• Orchestrated multi-agent: separate roles with validated handoffs (use when specialization is required).

Memory Types (Tradeoffs)

• Short-term (session): cheap, ephemeral; best for conversational continuity.

• Episodic (task): scoped to a case/ticket; supports audit and replay.

• Long-term (profile/knowledge): high risk; requires consent, retention limits, and provenance.

Failure Handling (Production Defaults)

• Classify errors: retriable vs fatal vs needs-human.

• Bound retries: max attempts, backoff, jitter; avoid retry storms.

• Fallbacks: degraded mode, smaller model, cached answers, or safe refusal.

Do / Avoid

Do

• Do keep state explicit and serializable (replayable runs).

• Do enforce tool allowlists, scopes, and idempotency for side effects.

• Do log traces/metrics for model calls and tool calls (OpenTelemetry GenAI semantic conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/).

Avoid

• Avoid runaway autonomy (unbounded loops or step counts).

• Avoid hidden state (implicit memory that cannot be audited).

• Avoid untrusted tool outputs without validation/sanitization.

Navigation: Economics & Decision Framework

Should You Build an Agent?

• Build vs Not Decision Framework - references/build-vs-not-decision.md

• 10-second test (volume, cost, error tolerance)

• Red flags and immediate disqualifiers

• Alternatives to agents (usually better)

• Full decision tree with stage gates

• Kill triggers during development and post-launch

• Pre-build validation checklist

Agent ROI & Token Economics

• Agent Economics - references/agent-economics.md

• Token pricing by model (January 2026)

• Cost per task by agent type

• ROI calculation formula and tiers

• Hallucination cost framework and mitigation ROI

• Investment decision matrix

• Monthly tracking dashboard

Navigation: AI Engine Layers

Five-layer architecture for production agent systems. Start with the overview, then drill into layer-specific patterns.

AI Engine Architecture — references/ai-engine-layers.md

5-layer composition model, layer interaction matrix, implementation phases

Context Graph Patterns — references/context-graph-patterns.md

Node/edge schema, traversal patterns, graph-RAG, memory tiers, conflict detection

Inbox Engine Patterns — references/inbox-engine-patterns.md

Event-driven intake, signal classification, deduplication, priority routing, dead letter

Knowledge Base Architecture — assets/knowledge-base/kb-architecture.md

Unified KB schema (vector + graph + doc index), provenance, freshness, multi-tenant

Action Graph → covered by references/operational-patterns.md

• references/agent-operations-best-practices.md

Data Agent → covered by ../ai-rag/SKILL.md

• references/rag-patterns.md

Navigation: Core Concepts & Patterns

Governance & Maturity

• Agent Maturity & Governance - references/agent-maturity-governance.md

• Capability maturity levels (L0-L4)

• Identity & policy enforcement

• Fleet control and registry management

• Deprecation rules and kill switches

Modern Best Practices

• Modern Best Practices - references/modern-best-practices.md

• Model Context Protocol (MCP)

• Agent-to-Agent Protocol (A2A)

• Agentic RAG (Dynamic Retrieval)

• Multi-layer guardrails

• LangGraph over LangChain

• OpenTelemetry for agents

Context Management

• Context Engineering - references/context-engineering.md

• Progressive disclosure

• Session management

• Memory provenance

• Retrieval timing

• Multimodal context

Core Operational Patterns

• Operational Patterns - references/operational-patterns.md

• Agent loop pattern (PLAN → ACT → OBSERVE → UPDATE)

• OS agent action loop

• RAG pipeline pattern

• Tool specification

• Memory system pattern

• Multi-agent workflow

• Safety & guardrails

• Observability

• Evaluation patterns

• Deployment & CI/CD

Navigation: Protocol Implementation

MCP Practical Guide - references/mcp-practical-guide.md

Building MCP servers, tool integration, and standardized data access

MCP Server Builder - references/mcp-server-builder.md

End-to-end checklist for workflow-focused MCP servers (design → build → test)

A2A Handoff Patterns - references/a2a-handoff-patterns.md

Agent-to-agent communication, task delegation, and coordination protocols

Protocol Decision Tree - references/protocol-decision-tree.md

When to use MCP vs A2A, decision framework, and selection criteria

Navigation: Agent Capabilities

Agent Operations - references/agent-operations-best-practices.md

Action loops, planning, observation, and execution patterns

RAG Patterns - references/rag-patterns.md

Contextual retrieval, agentic RAG, and hybrid search strategies

Memory Systems - references/memory-systems.md

Session, long-term, episodic, and task memory architectures

Tool Design & Validation - references/tool-design-specs.md

Tool schemas, validation, error handling, and MCP integration

Skill Packaging & Sharing

Skill Lifecycle - references/skill-lifecycle.md

Scaffold, validate, package, and share skills with teams (Slack-ready)

API Contracts for Agents - references/api-contracts-for-agents.md

Request/response envelopes, safety gates, streaming/async patterns, error taxonomy

Multi-Agent Patterns - references/multi-agent-patterns.md

Manager-worker, sequential, handoff, and group chat orchestration

OS Agent Capabilities - references/os-agent-capabilities.md

Desktop automation, UI grounding, and computer use patterns

Code/SWE Agents - references/code-swe-agents.md

SE 3.0 paradigm, autonomous coding patterns, SWE-Bench, HyperAgent architecture

Framework-Specific Patterns

• Pydantic AI Patterns - references/pydantic-ai-patterns.md

Type-safe agents, MCP toolsets (Stdio/SSE/StreamableHTTP), A2A via to_a2a() , pydantic-graph FSM, durable execution, TestModel testing

Navigation: Production Operations

Evaluation & Observability - references/evaluation-and-observability.md

OpenTelemetry GenAI, metrics, LLM-as-judge, and monitoring

Deployment, CI/CD & Safety - references/deployment-ci-cd-and-safety.md

Multi-layer guardrails, HITL controls, NIST AI RMF, production checklists

Agent Debugging Patterns - references/agent-debugging-patterns.md

Systematic debugging for agentic systems: trace analysis, tool call failures, loop detection, state corruption

Voice & Multimodal Agents - references/voice-multimodal-agents.md

Voice-first and multimodal agent patterns: speech pipelines, vision grounding, cross-modal orchestration

Guardrails Implementation - references/guardrails-implementation.md

Multi-layer guardrail patterns: input/output validation, content filtering, PII detection, cost caps

Navigation: Templates (Copy-Paste Ready)

Checklists

• Agent Design & Safety Checklist - assets/checklists/agent-safety-checklist.md

Go/No-Go safety gate: permissions, HITL triggers, eval gates, observability, rollback

Core Agent Templates

Standard Agent Template - assets/core/agent-template-standard.md

Full production spec: memory, tools, RAG, evaluation, observability, safety

Specialized Agent Template - assets/core/agent-template-specialized.md

Domain-specific agents with custom capabilities and constraints

Quick Agent Template - assets/core/agent-template-quick.md

Minimal viable agent for rapid prototyping

RAG Templates

Basic RAG - assets/rag/rag-basic.md

Simple retrieval-augmented generation pipeline

Advanced RAG - assets/rag/rag-advanced.md

Contextual retrieval, reranking, and agentic RAG patterns

Hybrid Retrieval - assets/rag/hybrid-retrieval.md

Semantic + keyword search with BM25 fusion

Tool Templates

Tool Definition - assets/tools/tool-definition.md

MCP-compatible tool schemas with validation and error handling

Tool Validation Checklist - assets/tools/tool-validation-checklist.md

Testing, security, and production readiness checks

Multi-Agent Templates

Manager-Worker Template - assets/multi-agent/manager-worker-template.md

Orchestration pattern with task delegation and result aggregation

Evaluator-Router Template - assets/multi-agent/evaluator-router-template.md

Dynamic routing with quality assessment and domain classification

Service Layer Templates

• FastAPI Agent Service - ../dev-api-design/assets/fastapi/fastapi-complete-api.md

Auth, pagination, validation, error handling; extend with model lifespan loads, SSE, background tasks

External Sources Metadata

• Curated References - data/sources.json

Authoritative sources spanning standards, protocols, and production agent frameworks

Shared Utilities (Centralized patterns — extract, don't duplicate)

• ../software-clean-code-standard/utilities/llm-utilities.md — Token counting, streaming, cost estimation

• ../software-clean-code-standard/utilities/error-handling.md — Effect Result types, correlation IDs

• ../software-clean-code-standard/utilities/resilience-utilities.md — p-retry v6, circuit breaker for API calls

• ../software-clean-code-standard/utilities/logging-utilities.md — pino v9 + OpenTelemetry integration

• ../software-clean-code-standard/utilities/observability-utilities.md — OpenTelemetry SDK, tracing, metrics

• ../software-clean-code-standard/utilities/testing-utilities.md — Test factories, fixtures, mocks

• ../software-clean-code-standard/references/clean-code-standard.md — Canonical clean code rules (CC-* ) for citation

Trend Awareness Protocol

IMPORTANT: When users ask framework recommendations or "what's best for X" questions, use WebSearch to verify current landscape before answering. If unavailable, use data/sources.json and state what was verified vs assumed.

Trigger: framework comparisons, "best for [use case]", "is X still relevant?", "latest in AI agents", MCP server availability.

Report: current landscape, emerging trends, deprecated patterns, recommendation with rationale.

Related Skills

This skill integrates with complementary skills:

Core Dependencies

• ../ai-llm/

• LLM patterns, prompt engineering, and model selection for agents

• ../ai-rag/

• Deep RAG implementation: chunking, embedding, reranking

• ../ai-prompt-engineering/

• System prompt design, few-shot patterns, reasoning strategies

Production & Operations

• ../qa-observability/

• OpenTelemetry, metrics, distributed tracing

• ../software-security-appsec/

• OWASP Top 10, input validation, secure tool design

• ../ops-devops-platform/

• CI/CD pipelines, deployment strategies, infrastructure

Supporting Patterns

• ../dev-api-design/

• REST/GraphQL design for agent APIs and tool interfaces

• ../ai-mlops/

• Model deployment, monitoring, drift detection

• ../qa-debugging/

• Agent debugging, error analysis, root cause investigation

• ../dev-ai-coding-metrics/

• Team-level AI coding metrics: adoption, DORA/SPACE, ROI, DX surveys (this skill covers per-task agent economics)

Usage pattern: Start here for agent architecture, then reference specialized skills for deep implementation details.

Usage Notes

• Modern Standards: Default to MCP for tools, agentic RAG for retrieval, handoff-first for multi-agent

• Lightweight SKILL.md: Use this file for quick reference and navigation

• Drill-down resources: Reference detailed resources for implementation guidance

• Copy-paste templates: Use templates when the user asks for structured artifacts

• External sources: Reference data/sources.json for authoritative documentation links

• No theory: Never include theoretical explanations; only operational steps

AI-Native SDLC Template

• Use assets/agent-template-ainative-sdlc.md for the Delegate → Review → Own runbook (guardrails + outputs checklist).

Fact-Checking

• Use web search/web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.

• Prefer primary sources; report source links and dates for volatile information.

• If web access is unavailable, state the limitation and mark guidance as unverified.