Agents
How to choose a model for an AI agent
Choose an agent’s model by testing a few candidates on your own tasks, and weigh four things: how reliably it calls tools, how well it follows instructions, what it costs and how fast it is, and where your data may go. Start with a strong model, measure the results, then try cheaper ones where they hold up.
Updated · 5 min read
What makes a model good for an AI agent?
An agent asks more of a model than a chat does. It must choose tools, fill in their arguments correctly, read the results and keep going for many steps without losing the thread.
- Tool calling: the model must support tool calls (also called function calling). A model without it cannot run an agent at all.
- Instruction following: it should keep to your rules on step twenty as well as on step one.
- Reasoning over several steps: planning, noticing a wrong result and recovering from it.
- Context length: enough room for the instructions, the conversation and the tool results.
- Speed: people waiting in a chat notice every second.
- Cost: price per token, multiplied by the number of steps a typical run takes.
- Data terms: where the provider runs the model, whether it keeps your data, and whether your contracts allow it.
How do you compare models for an agent?
Benchmarks give a first shortlist, but your own tasks decide. A small, honest test beats a long list of scores:
- Write down 20 to 50 real requests the agent will get, each with what a good outcome looks like.
- Run every request with each candidate model, with the same instructions and tools.
- Check the tool calls, not only the final answers: the right tool, the right arguments, no needless steps.
- Count the steps, the cost and the failures for each model.
- Run the same test again whenever you change the instructions, the tools or the model.
Should you use a hosted, cloud or local model?
| Option | Examples | Good for | Watch for |
|---|---|---|---|
| Model makers’ own APIs | Anthropic, OpenAI, Google, Mistral, xAI, DeepSeek, Cohere | The strongest models, first | Data goes to the provider under its terms |
| Your cloud provider | Amazon Bedrock, Google Vertex AI, Microsoft Foundry, Azure OpenAI | Your existing cloud contracts and agreements | Which models and regions your account can use |
| Hosts for open models | Groq, Together AI, Fireworks AI, OpenRouter | Speed and lower prices | Tool-calling quality varies by model |
| Your own server | Ollama, or any OpenAI-compatible server | Full control of where the model runs | You run it, scale it and keep it up |
Model makers’ own APIs
- Examples
- Anthropic, OpenAI, Google, Mistral, xAI, DeepSeek, Cohere
- Good for
- The strongest models, first
- Watch for
- Data goes to the provider under its terms
Your cloud provider
- Examples
- Amazon Bedrock, Google Vertex AI, Microsoft Foundry, Azure OpenAI
- Good for
- Your existing cloud contracts and agreements
- Watch for
- Which models and regions your account can use
Hosts for open models
- Examples
- Groq, Together AI, Fireworks AI, OpenRouter
- Good for
- Speed and lower prices
- Watch for
- Tool-calling quality varies by model
Your own server
- Examples
- Ollama, or any OpenAI-compatible server
- Good for
- Full control of where the model runs
- Watch for
- You run it, scale it and keep it up
Should you use one model or several?
Usually several. A strong model for the agent that talks to people and makes decisions. A smaller, cheaper model for simple steps such as sorting messages into kinds. And a fallback model from another provider, so an outage at one provider does not stop your agents.
Which models can MyDeepAgents agents use?
Agents can use models from 13 named providers: Anthropic, OpenAI, Google, Mistral AI, Groq, xAI, DeepSeek, Cohere, Amazon Bedrock, OpenRouter, Together AI, Fireworks AI and Ollama, plus any server that speaks the OpenAI-compatible API, added by its address.
- Several routes to the same model: Claude through Anthropic’s API, Amazon Bedrock, Google Vertex AI or Microsoft Foundry; OpenAI models through OpenAI or Azure OpenAI; Gemini through Google’s Gemini API or Vertex AI.
- Amazon Bedrock connects with a Bedrock API key, an assumed role with an external ID, or access keys, and reaches other Bedrock models such as Llama, Amazon Nova and Mistral by their model ID.
- A default: the console suggests Claude Opus 5 for new agents, unless your organization has chosen its own default model.
- Fallbacks: each agent can list fallback models. The platform tries each model’s routes in order, with retries, and moves on to the next fallback when a model cannot answer.
- Your own keys: each organization adds its own key for each provider, and the platform uses it before any key a parent organization shares.
- Allow lists: an organization can limit the providers and models its agents may use, and organizations below it can only narrow that list.
- Your own server: a custom endpoint or an Ollama server must be reachable over HTTPS at a public address; private addresses are refused. Its model must support tool calling to run an agent.
Workflows choose their own model for each Ask AI step, so a cheap model can sort messages while the agent keeps a stronger one.
How do compliance rules affect the choice?
Rules about data can narrow the list before quality does. In MyDeepAgents, an agent under the HIPAA profile may only use routes covered by a business associate agreement, which today are Claude models on Amazon Bedrock. An organization’s allow list can narrow the choice further for every agent below it.
How do you keep model costs under control?
- Measure: MyDeepAgents meters every model call with its provider, model, input, output and cached tokens, and cost.
- Cap each run: every agent run has a cost cap, $0.50 by default, counted at the model’s list price. Models the platform has no price for, such as one on your own server, are not counted toward it, so keep a step limit on those.
- Cap the steps: 40 model calls per run by default; a run that reaches its limit stops.
- Budgets: monthly and daily budgets per organization and workspace, soft or hard, for calls on keys the platform provides.
- Use the right size: route simple steps to smaller models, and keep instructions short enough that every step does not resend pages of text.
Key points
- The model must support tool calling; after that, test candidates on your own tasks.
- Judge tool calls and steps, not only final answers.
- Use a strong model for decisions, a cheap one for simple steps, and a fallback from another provider.
- MyDeepAgents works with 13 named providers and any OpenAI-compatible server, with per-agent fallbacks and your own keys.
- Compliance profiles, allow lists, run caps and budgets keep the choice safe and affordable.
Questions people also ask
Can I use a local or open model, such as Llama, for an agent?
Yes, through Ollama or any OpenAI-compatible server, or Llama on Amazon Bedrock. In MyDeepAgents your own server must be reachable over HTTPS at a public address, and the model must support tool calling to run an agent.
What happens if a model provider is down?
If the agent lists fallback models, the platform retries, then moves on to the next fallback. Choose a fallback from a different provider, so one outage cannot stop the agent.
Can different agents use different models?
Yes. Each agent has its own model and fallbacks, and each Ask AI step in a workflow chooses its own model too.
Is the most expensive model always the best choice?
No. It is a good place to start measuring. Many steps, such as sorting messages or pulling fields out of text, work as well on smaller models at a fraction of the cost.