How to Set Up an AI Agent with the Agno Framework: A Complete Guide

You can set up an AI agent with the Agno framework by installing the dependencies, instantiating the Agent class with a model like OpenAIChat and optional tools, then calling the run() method to process user queries.

The Agno framework (formerly Phidata) provides a lightweight, plug-and-play architecture for building AI agents that interact with LLMs and external tools. In the Shubhamsaboo/awesome-llm-apps repository, you'll find production-ready examples demonstrating how to set up an AI agent with the Agno framework for data analysis, media processing, and autonomous task execution.

Core Concepts and Architecture

The Agent Class

The Agent class serves as the central orchestrator in the Agno framework. Located throughout the repository (notably in starter_ai_agents/ai_data_analysis_agent/ai_data_analyst.py), this class accepts a model, tools list, system message, and provides run() and stream() methods for executing queries.

Models

Agno supports multiple providers through unified wrappers. You can use OpenAIChat for OpenAI models, Gemini for Google models, or DeepSeek for reasoning tasks. Each wrapper accepts an API key and model identifier.

Tools

Tools extend agent capabilities beyond text generation. The repository demonstrates DuckDbTools for SQL queries, PandasTools for dataframe operations, and custom browser automation tools. These are passed as a list to the Agent constructor.

Media Handling

For multimodal inputs, Agno provides AgnoImage and AgnoAudio classes in agno/media/. These wrap file paths to ensure proper encoding for vision-capable models.

Step-by-Step Setup Guide

1. Install Dependencies

Clone the repository and install the required packages:

git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git
cd awesome-llm-apps/starter_ai_agents/ai_data_analysis_agent
pip install -r requirements.txt

The requirements.txt files pin Agno to version ≥2.0.4.

2. Configure API Keys

Obtain API keys from your chosen provider:

• OpenAI: platform.openai.com/account/api-keys
• Gemini: makersuite.google.com/app/apikey
• DeepSeek: deepseek.com/api-keys

Store these securely in environment variables or Streamlit session state. Never hard-code secrets in production code.

3. Initialize the Agent

Create your agent by combining a model, tools, and system instructions:

from agno.agent import Agent
from agno.models.openai import OpenAIChat
from agno.tools.duckdb import DuckDbTools
from agno.tools.pandas import PandasTools

data_analyst_agent = Agent(
    model=OpenAIChat(id="gpt-4o", api_key=st.session_state.openai_key),
    tools=[DuckDbTools(), PandasTools()],
    system_message=(
        "You are an expert data analyst. Use the 'uploaded_data' table to answer user queries. "
        "Generate SQL queries using DuckDB tools to solve the user's query."
    ),
    markdown=True,
)

The system_message parameter defines the agent's persona and constraints.

4. Handle Media Inputs (Optional)

For image analysis, wrap files using AgnoImage:

from agno.media import Image as AgnoImage
from pathlib import Path

image = AgnoImage(filepath=Path("screenshot.png"))
response = agent.run("Describe this image", images=[image])

This pattern appears in starter_ai_agents/ai_breakup_recovery_agent/ai_breakup_recovery_agent.py for processing user-uploaded images.

5. Execute and Display Results

Trigger the agent from your UI:

if st.button("Analyze"):
    with st.spinner("Processing..."):
        result = data_analyst_agent.run(user_query)
    st.markdown(result.content)

The run() method returns a response object with a .content attribute containing the formatted output.

Production Examples from the Repository

AI Data Analyst

Located in starter_ai_agents/ai_data_analysis_agent/ai_data_analyst.py, this agent combines DuckDbTools and PandasTools to let users query CSV/Excel files using natural language. The agent automatically generates and executes SQL queries based on user questions.

Breakup Recovery Squad

Found in starter_ai_agents/ai_breakup_recovery_agent/ai_breakup_recovery_agent.py, this demonstrates multi-agent orchestration and image handling via AgnoImage. It showcases how to build empathetic AI interactions with multimodal inputs.

3D PyGame Code Generator

In advanced_ai_agents/autonomous_game_playing_agent_apps/ai_3dpygame_r1/ai_3dpygame_r1.py, this example shows mixed-model orchestration where DeepSeek handles reasoning tasks while OpenAI manages code extraction. It also implements async browser automation tools.

Common Pitfalls and Solutions

Issue	Cause	Solution
Agent ignores tools	Missing tools= parameter or uninstantiated tool classes	Pass instantiated tool objects like [DuckDbTools(), PandasTools()] to the Agent constructor
Images not processed	Using raw file paths instead of AgnoImage wrapper	Import AgnoImage from agno.media and wrap file paths before passing to run()
API rate limits	Concurrent calls exceeding quota	Enable debug_mode=True to inspect raw calls, implement retry logic, or upgrade API tier
UI freezing	Synchronous blocking without feedback	Wrap agent.run() in st.spinner() or execute in background threads

Summary

• Install Agno framework via pip (version ≥2.0.4) from the repository requirements.
• Configure API keys for OpenAI, Gemini, or DeepSeek without hard-coding secrets.
• Instantiate the Agent class with a model wrapper, tool list, and system message.
• Execute queries using run() for synchronous responses or stream() for real-time output.
• Extend capabilities by adding tools like DuckDbTools or media handlers like AgnoImage.

Frequently Asked Questions

What is the Agno framework and how does it differ from other agent frameworks?

Agno (formerly Phidata) is a lightweight, plug-and-play Python framework designed specifically for building LLM agents with minimal boilerplate. Unlike heavier orchestration frameworks, Agno focuses on unified model wrappers and first-class tool integration, allowing you to set up an AI agent with just a few lines of code in starter_ai_agents/ai_data_analysis_agent/ai_data_analyst.py.

Can I use Agno with models other than OpenAI?

Yes. Agno supports multiple providers including Gemini, DeepSeek, and others through provider-specific wrappers. The repository demonstrates mixed-model orchestration in advanced_ai_agents/autonomous_game_playing_agent_apps/ai_3dpygame_r1/ai_3dpygame_r1.py, where DeepSeek handles reasoning tasks while OpenAI manages code extraction.

How do I add custom tools to my Agno agent?

You instantiate tool classes like DuckDbTools() or PandasTools() and pass them as a list to the tools= parameter when creating your Agent. For custom functionality, you can write Python functions and register them as tools using Agno's decorator pattern, though the repository primarily uses built-in tool classes for database and media operations.

What is the best way to handle images in Agno agents?

Always wrap image file paths using the AgnoImage class from agno.media before passing them to the run() method. This ensures proper encoding and metadata formatting for multimodal models. The breakup-recovery agent in starter_ai_agents/ai_breakup_recovery_agent/ai_breakup_recovery_agent.py demonstrates this pattern for processing user-uploaded images.

{
  "mcpServers": {
    "instagit": {
      "command": "npx",
      "args": ["-y", "instagit@latest"]
    }
  }
}