An AI agent is not a chatbot with function calling. It’s a system that perceives, reasons, acts, and learns —autonomously, across multiple steps, with state that persists between actions. Anthropic’s guide to building effective agents is the best starting point for understanding agent architecture patterns. A chatbot answers your question. An agent books your flight, reschedules your meetings, and updates your team’s Slack while you’re in the air.
Everyone is building agents in 2026. Many break in production —looping forever, calling the wrong tools, forgetting what they were doing three steps ago. This guide covers the architecture that prevents those failures. From the tool-calling loop to memory systems to multi-agent orchestration, every section has working code. By the end, you’ll have a working research assistant agent you can fork and extend.
What Makes an AI Agent —and What Doesn’t
Definition. An AI agent has three layers: a reasoning core (the LLM), a tool layer (APIs, databases, code execution), and a memory layer (short-term conversation, long-term knowledge, working state). The key difference from a simple LLM call: the agent makes autonomous decisions in a loop. It doesn’t just respond. It plans, acts, observes the result, and decides what to do next.
The three layers:
User Query → Reasoning Core (LLM) → Decision → Tool Execution → Observation → Memory Update → Next Decision → ... → Final ResponseWhen you need an agent vs. a simple LLM call. Agent: multi-step tasks where the model needs to gather information, execute actions, and adapt based on results. “Research this topic and write a report” —agent. “Summarize this article” —simple call. “Debug this error, check the logs, and open a PR with the fix” —agent. “Explain this error message” —simple call.
If the task can be completed in one API call with no external tool use, you don’t need an agent. If the task requires gathering information from multiple sources, executing actions, and making decisions based on intermediate results, you need an agent. The decision tree is: one step? —simple call. Multiple steps with tools? —agent.
The Tool-Calling Loop: Your Agent’s Hands
The core agent loop. Every agent framework —LangChain, CrewAI, AutoGen, raw code —implements some version of this.
import json
from openai import OpenAI
client = OpenAI(
base_url="https://api.tokspan.com/v1",
api_key="ts-your-key-here"
)
class Agent:
def __init__(self, model: str, tools: list, max_iterations: int = 10):
self.model = model
self.tools = {t["function"]["name"]: t for t in tools}
self.max_iterations = max_iterations
self.memory = [] # Working memory —the agent's scratchpad
def run(self, user_query: str) -> str:
messages = [
{"role": "system", "content": "You are a research assistant. Use tools to gather information, then synthesize a report."},
{"role": "user", "content": user_query}
]
for iteration in range(self.max_iterations):
response = client.chat.completions.create(
model=self.model,
messages=messages,
tools=list(self.tools.values()),
tool_choice="auto"
)
msg = response.choices[0].message
# Agent decided to respond with text —done
if msg.content and not msg.tool_calls:
return msg.content
# Agent decided to call tools —execute and continue
if msg.tool_calls:
messages.append(msg)
for tool_call in msg.tool_calls:
tool_name = tool_call.function.name
tool_args = json.loads(tool_call.function.arguments)
result = self._execute_tool(tool_name, tool_args)
messages.append({
"role": "tool",
"tool_call_id": tool_call.id,
"content": str(result)
})
return "Agent reached maximum iterations without completing the task."
def _execute_tool(self, name: str, args: dict):
# In production: dispatch to actual functions
print(f"Calling tool: {name}({args})")
return f"Result from {name}"Tool definition best practices. A tool’s description is a prompt —write it clearly. Include examples of when to use each tool. Constrain parameters tightly —enums instead of free-text strings. Make tools idempotent —calling the same tool twice with the same parameters should produce the same result. When a tool execution fails, send the error message back to the model —it can often recover by trying different parameters.
For the complete cross-provider function calling comparison —including OpenAI vs. Anthropic vs. Google vs. DeepSeek implementations —see our cross-provider function calling guide.
Error handling in the loop. Tool execution fails —send error back to model —model decides: retry with different parameters, try a different tool, or inform the user. The model is surprisingly good at recovering from tool errors —but only if you send the error back. Silently swallowing tool failures produces agents that fail mysteriously.
Memory Systems: Teaching Your Agent to Remember
An agent with no memory is a goldfish —it forgets everything between steps. Three memory types, each with a specific purpose.
Short-term memory —conversation history. The messages array. What the user said, what the agent did, what tools returned. Managed as a sliding window —when the history approaches the model’s context limit, trim the oldest messages or summarize them. Summarization trigger: when total tokens exceed 80% of the context window, summarize the oldest 50% of the conversation into a single system message.
Long-term memory —vector store. Past interactions, user preferences, learned facts —stored as embeddings in a vector database, retrieved by similarity to the current query. Implementation: embed each significant interaction —store in ChromaDB or Pinecone —on each new query, retrieve the top 3–5 most similar past interactions —include in the system prompt as context. This is the difference between “the agent knows what we talked about last week” and “the agent starts fresh every conversation.”
Working memory —the scratchpad. The agent’s current plan, intermediate results, and hypotheses —stored as a JSON object updated at each loop iteration. What is the agent trying to accomplish right now? What has it tried? What did it learn? The scratchpad is the agent’s “train of thought” —externalized so it survives across tool calls and can be inspected for debugging.
Memory architecture. All three memories feed into the agent at each loop iteration: conversation history (what happened) + retrieved long-term memories (what’s relevant from the past) + working memory (what we’re doing right now). The LLM synthesizes these into the next decision.
Multi-Agent Systems: When One Agent Isn’t Enough
A single agent with 15 tools makes poor decisions —too many options, too much context, degraded selection accuracy. The solution: specialized agents, each with a focused tool set and clear responsibility.
Orchestration patterns:
- Supervisor/Worker. One orchestrator agent assigns tasks to specialized worker agents. The supervisor doesn’t do the work —it coordinates. “Research this topic” —supervisor dispatches to a researcher agent, an analyst agent, and a writer agent —supervisor compiles results.
- Peer-to-peer debate. Two agents argue opposite sides of a decision, then converge. “Should we approve this loan?” —agent A argues yes, agent B argues no —both review each other’s arguments —produce a joint recommendation.
- Sequential pipeline. Agent A’s output is Agent B’s input. “Analyze this codebase” —Code Analyzer outputs a report —Bug Finder uses the report to identify issues —Fix Generator proposes solutions.
Multi-agent communication. Use a shared message bus —each agent publishes its output as a structured message with {from: "researcher", to: "analyst", content: "...", type: "report"}. This makes the agent interaction graph observable and debuggable. When something goes wrong, you can trace exactly which agent produced which output and why.
Cost of multi-agent. Each agent is making its own LLM calls. A three-agent system makes 3x the API calls of a single-agent system. Mitigation: use cheap models for worker agents (DeepSeek V4 Flash, $0.14/$0.28) and reserve frontier models (Claude Opus, GPT-5.5) for the orchestrator. The orchestrator makes the high-stakes decisions. The workers execute.
Here’s a concrete 3-agent research pipeline you can deploy today. The Researcher agent uses Gemini 3.1 Pro ($2.00/M input tokens) with exactly two tools —web search and document retrieval —so it never gets lost in tool-selection paralysis. Its output is a structured JSON brief: {sources: [...], key_facts: [...], gaps: [...]}.
The Writer agent runs GPT-5.5 ($5.00/M input) to transform that brief into a draft, using only a formatting tool. The Reviewer agent uses Claude Opus 4.8 ($5.00/M input) to fact-check every claim against the original sources, flag hallucinations, and return a scored review: {score: 1-10, issues: [...], corrected_draft: "..."}. Per-task cost runs $0.12–0.35 —the Researcher consumes ~40% of tokens, the Writer ~35%, the Reviewer ~25%.
The message bus is a simple Python dict passed between agents —no framework required. Log {timestamp, from_agent, to_agent, payload_type, token_count} at each transition and you can trace every handoff when something breaks.
Debugging Agent Failures
Agents fail in predictable ways. The three most common failure modes: infinite loops (the agent calls tools but never converges), wrong tool selection (the model picks an irrelevant tool with incorrect parameters), and context overflow (conversation history exceeds the model’s context window, silently dropping earlier messages). Each has a diagnostic pattern.
For infinite loops, log the agent’s actions for repetition —if the same tool is called with the same parameters three times in a row, the agent is stuck. Intervention: inject a system message saying “You have called {tool} with {args} multiple times. The result has not changed. Try a different approach or report what you have so far.”
For wrong tool selection, log the tool name, parameters, and result at each iteration —you will spot patterns. An agent that calls search_web when it should call query_database reveals a tool description that needs rewriting, not a model problem.
For context overflow, track total_tokens at each iteration using the API’s usage field. When tokens exceed 80% of the context limit —1M for GPT-5.5, 200K for Claude Opus —summarize the oldest 50% of messages before the next iteration. The most common developer mistake: never checking response.usage.total_tokens until the agent starts producing garbled output, unaware the context was silently truncated five iterations ago.
Structured logging is the single most effective debugging tool you have. At minimum, log per iteration: {iteration, model, tool_calls, tokens_used, latency_ms, error}. After 20 agent runs you will have enough data to identify which failure mode bites you most often.
You will also notice performance regressions immediately —a tool call that normally takes 200ms suddenly taking 2 seconds is a signal before it becomes an incident.
Which Model for Which Agent Role?
| Agent Role | Best Model | Why |
|---|---|---|
| Orchestrator | GPT-5.5 | Most reliable tool use, best parallel tool calling |
| Code Agent | Claude Opus 4.8 | Highest SWE-bench, best architectural reasoning |
| Research Agent | Gemini 3.1 Pro | 2M context for document analysis, multimodal |
| Cost-Efficient Worker | DeepSeek V4 Pro | 92% HumanEval at $0.44/M output |
| Writer Agent | GPT-5.5 | Best prose quality and stylistic range |
The aggregation platform advantage: access all five models through one API key. Route each agent role to its optimal model. Change models without changing agent code. The orchestrator, code agent, research agent, and workers all use the same OpenAI SDK —just different model parameters.
For a deeper architecture discussion on routing different tasks to different models —a pattern every production agent system uses —see our guide to using multiple AI models in one app.
FAQ
Do I need a framework like LangChain to build agents?
No. The core agent loop is ~50 lines of Python —the code in this article is a complete working agent. Frameworks add conveniences (pre-built tools, tracing, memory backends) and complexity (abstraction layers, dependency trees, breaking changes between versions). Start raw. Add a framework only when you have a specific problem it solves. Developers who jump straight to LangChain often regret it —they spend more time debugging the framework than building the agent.
Which model is best for agents?
GPT-5.5 for tool-use reliability —it’s the most consistent at calling the right tool with the right parameters. Claude Opus for complex multi-step reasoning where depth matters more than reliability. Gemini for long-context tasks. DeepSeek V4 Pro for cost-efficient agents. Most production agent systems use 2–3 models: a reliable orchestrator (GPT-5.5), a deep-reasoning specialist (Claude Opus) for complex steps, and a cost-efficient worker (DeepSeek) for high-volume simple tasks.
How do I prevent my agent from looping forever?
Three safeguards. Set max_iterations (10–20 is reasonable for most tasks). Track task completion —if the agent’s last 3 actions produced no new information, it’s stuck; terminate and return a partial result. Budget cap per agent session —a $0.50 spending limit catches infinite loops before they become $50 problems (see our key management and budget control guide for setting up spending caps and rate limits as guardrails). Always have a timeout + graceful fallback response.
How much do AI agents cost to run?
Simple agent (3–5 tool calls): $0.05–0.20 per task using DeepSeek V4 Pro. Complex multi-agent (10–20 tool calls): $0.50–2.00 per task with mixed models. Use cost-based routing: simple tasks —cheap models, complex tasks —frontier models. The cost per task should be measured and optimized like any other infrastructure cost.
How do I test agent reliability before deploying to production?
Build an eval harness with 20–50 hand-labeled test cases covering your agent’s expected task range. Run each case 5 times —agent behavior is non-deterministic, so a single pass proves nothing. Measure two metrics: task completion rate (did the agent produce a valid output?) and tool-selection accuracy (did it call the right tools in the right order?).
A completion rate below 85% means your prompts or tool descriptions need work. For multi-agent systems, add a third metric: handoff correctness —did each agent receive the expected input format from the upstream agent? One bad handoff cascades into failures downstream.
When should I use a single agent vs. a multi-agent system?
Start with a single agent. Add a second agent only when you hit one of three thresholds: the tool list exceeds 8–20 functions (tool-selection accuracy degrades above this count according to OpenAI’s function-calling benchmarks), the task has clearly separable sub-tasks with different expertise requirements (research vs. writing vs. review), or you need independent safety verification (a reviewer agent that checks the primary agent’s output).
Premature multi-agent architecture is the most common over-engineering mistake in agent development —it adds latency, cost, and debugging complexity without proportional benefit for simple workflows.
Step one: copy the 50-line agent loop from this article, swap in your own tool definitions, set max_iterations to 10, and deploy it against a non-critical internal task —something low-stakes where a failure is a learning opportunity, not an incident. Watch the logs. See where it gets stuck. Add memory when it forgets context. Add multi-agent when the tool list gets unwieldy. The only way to learn what breaks is to ship something and observe.
The 50-line agent loop above is a working starting point. When you are ready to assign each agent role to its optimal model —the mapping table earlier in this article is a reference —an aggregation endpoint lets you change the model per agent by editing a string parameter. No per-provider SDKs, no separate billing relationships. Deploy the loop against a low-stakes internal task first, watch the logs, and iterate from there.