AI agent architecture: the building blocks of an LLM agent, and how they fit together
An AI agent is a language model running in a loop: it reads the task, calls tools, reads the results and decides the next step until the job is done. Around that loop sit six building blocks: tools (often over MCP), context and memory, orchestration across steps or agents, guardrails in code, a human in the loop, and evaluation. Good agent architecture starts with the simplest design that works.
What an AI agent is
Anthropic separates two kinds of LLM systems. Workflows follow "predefined code paths". Agents are systems where "LLMs dynamically direct their own processes and tool usage". The architecture of an agent is everything that makes that self-direction useful and safe.
The core block is what Anthropic calls the augmented LLM: a model "enhanced with augmentations such as retrieval, tools, and memory". The rest of the architecture decides how that block is driven, constrained and checked.
1. The loop
The agent runs in a loop: send the conversation and tool definitions, read the response, run any tools it asks for, append the results, repeat until it answers without asking for a tool. With Claude, stop_reason drives the loop (tool_use to continue, end_turn to stop), and turn or budget caps act only as safety guards. Details in the agentic loop explained.
2. Tools and MCP
Tools are how the agent acts: look up a record, run a query, send a message. The model knows a tool only by its name, description and input schema, so tool design is architecture, not plumbing. Anthropic calls this the agent-computer interface and puts it among its three core principles. The Model Context Protocol packages tools and data once for every compatible app; see LLM tool calling for the mechanics.
3. Context and memory
Everything the agent sees lives in its context window: instructions, history, tool results. More is not better: accuracy drops as context grows, so the architecture must curate it. Common pieces: a short system prompt and project memory file, trimmed tool outputs, a block of exact case facts, compaction for long sessions, and files on disk for state that must survive across sessions.
4. Orchestration: one agent or several
| Design | Use when |
|---|---|
| A single prompt or prompt chain | The steps are known in advance |
| Routing or parallel calls | Inputs fall into categories, or parts are independent |
| One agent with a loop | The path must be discovered step by step |
| A coordinator with subagents | The work splits into independent parts too large for one context |
A multi-agent system can do far more work but costs far more tokens; Anthropic measured about 15 times the tokens of a chat. Start with the simplest design that meets the goal.
5. Guardrails in code
Prompts shape behavior; they don't guarantee it. Rules that must always hold (a refund limit, a forbidden directory, a required verification step) belong in code: hooks that intercept tool calls, permission rules, and gates between steps. Tools should also fail honestly: an error the agent can see beats an empty result it will believe.
6. A human in the loop
People approve risky actions, answer clarifying questions, take over cases the agent should not decide, and review a sample of outputs; see human in the loop AI.
7. Evaluation
An agent changes behavior whenever its prompt, tools or model change. A fixed test set of real cases, run before every change, is how you know whether a change helped. Measure by segment, not only overall averages.
Building the architecture with Claude
| Layer | Claude option |
|---|---|
| You run the loop | Claude API with tool use |
| The loop runs for you, in your app | Claude Agent SDK |
| A finished agent for code | Claude Code |
| Tools and data for any of them | MCP servers |
Which to pick is the subject of Claude API vs Agent SDK vs Claude Code vs MCP.
Agent architecture on the Claude Certified Architect exam
Domain 1, Agentic Architecture & Orchestration, is the largest part of the CCAR-F exam at 27%, and the other four domains each cover one of the building blocks above: tools and MCP, Claude Code, prompts and structured output, and context and reliability.
More in all articles.
Sources: Building effective agents; How we built our multi-agent research system; How the agent loop works; Context windows; Claude Certified Architect Foundations exam guide v1.0.
FAQ
What is the architecture of an AI agent?
A language model augmented with tools, retrieval and memory, running in a loop that decides each next step from the results so far, plus the pieces around it: orchestration, guardrails enforced in code, human checkpoints and evaluation.
What is the difference between an AI agent and a workflow?
In a workflow, your code fixes the sequence of LLM calls and tools. In an agent, the model directs its own process and tool use, which suits open-ended tasks whose steps can't be known in advance.
Do I need a multi-agent architecture?
Only when the work splits into independent parts too big for one context window, such as broad research. Multi-agent systems use many more tokens and add coordination cost, so start with one agent.
How do you keep an AI agent safe?
Enforce hard rules in code with hooks and permission rules rather than prompts, put a human checkpoint on risky actions, return tool failures honestly, and test changes against an eval set.