Single-agent architectures hit a ceiling. Your coding agent writes —but cannot review its own output. Your billing agent resolves —but cannot talk to shipping. Same perspective, same blind spots.
You pile on more tools. Latency balloons. Context windows overflow. The ceiling is structural, not prompt-deep.
The fix: specialized agents —narrow expertise, parallel execution, defined protocols.
You’ll leave with three orchestration patterns (Supervisor-Worker, Peer-to-Peer, Hierarchical) with Python code, role-to-model mapping that cuts costs 50-70%, and trace topology that turns a flat span list into a system map.
When Does Single Agent Break?
The Complexity Ceiling
Single agents fail predictably in four scenarios:
Multi-domain expertise. One agent asked to simultaneously be a coding expert, a legal reviewer, and a finance analyst. Prompt bloat —the system prompt grows to accommodate all three domains. Role confusion —the model blends tones and priorities. A legal review written in casual engineering slang. A financial analysis that skips compliance checks because the “coding expert” persona dominates.
Parallel execution. Three independent subtasks that could run concurrently —research competitor pricing, analyze user reviews, summarize product specs —processed serially by a single agent. Total latency is the sum of all three. With three specialized agents running in parallel, latency is the max of the three.
Self-critique. A single agent generates output and evaluates its own output. The same cognitive process that produced the answer is asked to find its flaws. It can’t. Peer review —a separate agent with a different prompt, different model, different perspective —catches errors the generating agent is blind to.
Tool scope explosion. Fifteen tools is fine. Twenty-five tools and the model’s tool selection accuracy drops measurably —it picks the wrong tool more often, misses the right tool, or calls tools in the wrong order. Specialized agents with 5-8 tools each outperform one generalist with 25.
Multi-Agent Isn’t Always the Answer
Multi-agent systems add real overhead: inter-agent communication latency, token costs for messages between agents, consistency risk when two agents produce conflicting answers, and debugging complexity when you can’t tell which agent in a 5-agent chain produced the wrong output. If your task can be handled by one well-prompted agent with 5-10 well-scoped tools, don’t introduce multi-agent complexity. Start single. Go multi only when single demonstrably fails —tool confusion, role inconsistency, or serial latency that parallel execution would eliminate.
Our single agent architecture guide covers the foundations. This article assumes you’ve pushed single-agent design to its limit and are now hitting the ceiling.
Three Orchestration Patterns
Pattern 1: Supervisor-Worker
One supervisor agent orchestrates. It decomposes the task, assigns subtasks to workers, aggregates results, and runs a quality gate before returning output. Workers are specialized —each has its own system prompt, tool set, and model. Workers don’t talk to each other. They only communicate with the supervisor.
class SupervisorAgent:
def __init__(self, model: str, workers: list[WorkerAgent]):
self.model = model
self.workers = {w.name: w for w in workers}
async def execute(self, task: str) -> dict:
plan = await self._decompose(task)
results = {}
for subtask in plan["subtasks"]:
worker = self.workers[subtask["assigned_to"]]
results[subtask["id"]] = await worker.execute(subtask)
synthesis = await self._synthesize(task, results)
quality = await self._quality_gate(synthesis)
if quality["passed"]:
return synthesis
else:
return await self._retry_or_escalate(task, quality["issues"])Best for: tasks that decompose cleanly into independent subtasks, need central quality control, and have a relatively fixed team structure. Trade-off: supervisor becomes a bottleneck and a single point of failure. Not suitable when workers need to negotiate dynamically with each other.
Pattern 2: Peer-to-Peer
No hierarchy. All agents are peers. Any agent can initiate communication with any other agent through a shared message bus. Agents discover each other’s capabilities and negotiate task handoffs dynamically.
class PeerAgent:
def __init__(self, name: str, role: str, model: str, message_bus: MessageBus):
self.name = name
self.role = role
self.model = model
self.bus = message_bus
self.bus.subscribe(self.name, self._handle_message)
async def _handle_message(self, msg: Message):
if msg.type == "request":
result = await self._execute_task(msg.content)
await self.bus.send(Message(
to=msg.from_agent,
type="response",
content=result,
correlation_id=msg.correlation_id
))Best for: dynamic negotiation —a code review agent discovers a bug and directly messages the coding agent with the fix, no supervisor in the loop. No bottleneck. Flexible workflows. Trade-off: debugging is harder —the conversation graph can become complex, and infinite message loops or conflicting outputs between peers require explicit loop detection and conflict resolution mechanisms.
Pattern 3: Hierarchical
Multi-tier escalation. Front-line agents handle routine cases. Complex or unusual cases escalate to senior agents. The hardest cases reach a principal agent or a human. Model quality and cost scale with tier —Tier 1 uses cheap models, Tier 2 uses mid-tier, Tier 3 uses frontier.
class TieredAgentSystem:
def __init__(self, tiers: list[AgentTier]):
self.tiers = sorted(tiers, key=lambda t: t.level)
async def execute(self, task: str) -> dict:
for tier in self.tiers:
result = await tier.agent.execute(task)
if result["confidence"] >= tier.confidence_threshold:
return result
return await self._escalate_to_human(task)Best for: support workflows with natural complexity tiers, content moderation with auto-filter —senior review —human escalation, and any domain where most requests are simple and a minority require deep expertise. Trade-off: escalation logic requires tuning from production data —you need a feedback loop to calibrate confidence thresholds per tier.
Agent Communication Protocols
Function Calling as Inter-Agent Protocol
Simplest approach: Agent A calls a send_message_to_agent_b tool. The tool is defined in Agent A’s function calling schema. Compatible with existing infrastructure. Zero new protocols to learn. Limitation: synchronous blocking. Each message is a full tool-call round-trip. For multi-turn agent conversations, latency accumulates linearly. When the tool definitions themselves need to span multiple providers with inconsistent schemas, our function calling and tool use guide covers the normalization layer that makes inter-agent tool calls portable across models.
MCP for Agent Communication
Each agent exposes its capabilities as an MCP server. Other agents, acting as MCP clients, discover and invoke those capabilities. Our MCP guide covers the protocol fundamentals. In multi-agent systems, MCP provides standardized capability discovery —an agent doesn’t need to know in advance what other agents can do. It queries the MCP ecosystem and discovers capabilities dynamically.
Google A2A
Purpose-built for agent-to-agent communication (A2A specification). Agent Cards for capability discovery. Structured Task lifecycle: submitted —working —completed or failed. Streaming updates during task execution. Multi-modal content exchange. Cross-framework —agents built with different frameworks can interoperate if they speak A2A. Best for heterogeneous multi-agent systems where different teams use different stacks.
Custom Message Bus
Redis Pub/Sub or NATS with a structured message schema. Maximum control over routing, persistence, retry, and observability. Maximum implementation effort. The schema: {agent_id, correlation_id, message_type, payload, timestamp}. Choose this when your inter-agent communication patterns are unique enough that no standard protocol fits —or when you need guarantees (exactly-once delivery, ordered delivery) that higher-level protocols don’t provide.
Role-to-Model Mapping
The biggest cost trap in multi-agent systems: every agent gets a frontier model. Your classifier agent that routes “billing” vs. “technical support” does not need Claude Opus at $15/M input tokens. It needs GPT-4o Mini at $0.15 —and the classification accuracy is identical.
| Agent Role | Recommended Model | Cost/1M Input | Rationale |
|---|---|---|---|
| Classifier/Router | DeepSeek V3.2 / GPT-4o Mini | $0.14-0.15 | Simple classification. Cheap models perform identically to frontier. |
| Basic Q&A / FAQ | GPT-4o Mini / Claude Haiku | $0.15-0.25 | Retrieval-augmented factual responses. Mid-tier capability, floor pricing. |
| Content Generation | GPT-4o / Claude Sonnet 4 | $2.50-3.00 | Tone, structure, creativity matter. Worth the mid-tier premium. |
| Code Generation | Claude Sonnet 4 / DeepSeek V4 Pro | $0.42-3.00 | SWE-bench data drives this choice. DeepSeek’s coding performance per dollar is exceptional. |
| Code Review / Quality Gate | Claude Opus 4 / GPT-5.5 | $10-15.00 | Finding bugs requires frontier reasoning. One missed bug costs more than the model premium. |
| Final Synthesis | Claude Opus 4 | $15.00 | The output the user sees. Worth the premium for tone, accuracy, and instruction following. |
Per-agent cost tracking —gen_ai.cost.total on every LLM span with agent_id as a span attribute —gives you a per-agent cost report. You’ll quickly spot the agent consuming 40% of your multi-agent budget. Downgrade it from Opus to Sonnet and check if output quality actually changes. Often, it doesn’t. Beyond per-agent model selection, our 12-way LLM API cost optimization guide covers prompt caching, batch processing, and semantic caching patterns that compound across your agent fleet.
Debugging and Monitoring Multi-Agent Systems
The Trace Topology Problem
One user request can trigger: Agent A (classify) —Agent B (research) + Agent C (code) in parallel —Agent A (synthesize) —six LLM calls, ten tool calls, three inter-agent messages. A flat span list of 19 entries is undebuggable. You can’t see the execution graph.
The solution: OpenInference AGENT span kind with hierarchical modeling. Root span: user session. Level 1: orchestrator decision. Level 2: per-agent execution. Level 3: per-agent tool calls. Your trace viewer renders the execution graph, not a flat list. When Agent C’s tool call fails, you see it in context —which agent, which step, what happened before and after.
See our observability guide for the full OpenTelemetry setup.
Common Multi-Agent Failure Modes
Infinite message loop. Agent A —Agent B —Agent A —Agent B. Prevention: conversation depth limit and loop detection at the message bus level. At the API layer, per-agent rate limits add a budget ceiling —when an agent exceeds its request quota, the loop stops regardless of what the message bus logic permits.
Conflicting outputs. Agent A says refund. Agent B says no refund. No resolution mechanism. Prevention: supervisor pattern with explicit conflict resolution, or voting mechanism across agents.
Tool permission leak. The coding agent accidentally gets access to the billing agent’s refund tool through a misconfigured tool registry. Prevention: per-agent tool allowlists. Agent identity in every tool audit trail.
Silent agent failure. One agent throws an unhandled error. The orchestrator doesn’t notice. The user gets a partial response missing critical information. Prevention: structured error reporting in the inter-agent protocol. Orchestrator-level health checks before synthesis.
FAQ
When should I NOT use multi-agent architecture?
If a single well-prompted agent with 10 or fewer well-defined tools handles your task, don’t introduce multi-agent complexity. The communication overhead, debugging difficulty, and cost of inter-agent messages outweigh the benefits. Single-agent first. Multi-agent only when single-agent demonstrably fails —tool confusion, role inconsistency, or serial latency that parallel execution would fix.
How many agents should my system start with?
Two to three, covering clearly distinct roles. Don’t start with six. Coordination complexity scales non-linearly —a 6-agent system is not 3×harder than a 2-agent system. It’s closer to 9× Add agents only when an existing agent shows consistent failure on a specific subtask that requires distinct expertise.
Which orchestration pattern should I start with?
Supervisor-Worker. Centralized control, clear state management, simplest debugging. Upgrade to Peer-to-Peer only when workers genuinely need dynamic negotiation with each other. Upgrade to Hierarchical only when you have clear complexity tiers with measurable confidence thresholds. Start simple. Add complexity only when the data proves you need it.
How do I handle an agent that consistently makes mistakes?
Don’t start by tweaking its prompt. First: trace analysis. Find the failure pattern —which inputs trigger the error? Which step in its execution goes wrong? Second: narrow the agent’s scope. It may be handling too many task types. Third: add a quality gate agent —a reviewer that checks outputs before they reach the user. Fourth: if the agent is missing domain knowledge, add RAG —not fine-tuning, not more prompting.
What’s the actual operational cost of a 5-agent system running across 3 model tiers?
The operational cost isn’t the tokens —it’s the fragmentation. Each agent tier potentially hits a different provider. That means separate API keys, separate rate-limit tracking, separate cost dashboards. When your classifier agent (DeepSeek Flash at $0.14/M) and your code reviewer (Claude Opus at $15/M) run through different providers, you spend more time managing credentials than optimizing agent behavior. The fix is architectural: one endpoint for all agents regardless of model. Per-agent cost attribution becomes a span attribute, not a cross-provider spreadsheet exercise. For the observability and latency tuning that makes this single-endpoint approach production-ready, TokSpan’s production optimization guide covers connection pooling, request queuing, and per-model timeout configuration across your agent fleet.
Multi-agent architecture isn’t more sophisticated —it’s a response to specific failure modes. When your single agent confuses tools across domains, when serial execution makes latency unacceptable, when self-review can’t catch its own errors —those are the signals. Not before.
Start with Supervisor-Worker. Map roles to models ruthlessly —your classifier doesn’t need Opus. Instrument every agent’s spans with agent_id. Watch the per-agent cost report for a week. You’ll find at least one agent that’s over-modeled and can drop a tier with zero quality impact.
The orchestration pattern matters less than the discipline of knowing when not to add another agent.
Your agent fleet’s cost report shouldn’t require stitching together CSV exports from four different provider dashboards. Deploy your agents on TokSpan —per-agent cost attribution, centralized rate limits, and every model your agents need behind one endpoint.