Multimodal Agents

Guide for creating and using multimodal AI agents in PraisonAI for processing images, videos, and other media types

Multimodal agents process text, images, and video in a single workflow — attach media to tasks or pass ephemeral attachments at chat time.

from praisonaiagents import Agent

agent = Agent(
    name="VisionAnalyst",
    instructions="Analyse images and video and describe what you see.",
    llm="gpt-4o-mini",
)
agent.start("Describe the landmark in this image.")

The user attaches media; the agent returns a single analysis covering text and visuals.

How It Works

Quick Start

Code

1. Install Package

   First, install the PraisonAI Agents package:

   pip install praisonaiagents opencv-python moviepy
   
2. Set API Key

   Set your OpenAI API key as an environment variable in your terminal:

   export OPENAI_API_KEY="${OPENAI_API_KEY:?Set OPENAI_API_KEY in your shell}"
   
3. Create a file

   Create a new file app.py with the basic setup:

   from praisonaiagents import Agent, Task, AgentTeam
   
   # Create Vision Analysis Agent
   vision_agent = Agent(
       name="VisionAnalyst",
       role="Computer Vision Specialist",
       goal="Analyze images and videos to extract meaningful information",
       backstory="""You are an expert in computer vision and image analysis.
       You excel at describing images, detecting objects, and understanding visual content.""",
       llm="gpt-4o-mini",
       reflection=False
   )
   
   # Create tasks with different media types
   task = Task(
       name="analyze_landmark",
       description="Describe this famous landmark and its architectural features.",
       expected_output="Detailed description of the landmark's architecture and significance",
       agent=vision_agent,
       images=["https://upload.wikimedia.org/wikipedia/commons/b/bf/Krakow_-_Kosciol_Mariacki.jpg"]
   )
   
   # Run the agents
   agents = AgentTeam(
       agents=[vision_agent],
       tasks=[task],
       process="sequential",
       
   )
   
   agents.start()
   
4. Start Agents

   Type this in your terminal to run your agents:

   python app.py
   

No Code

1. Install Package

   Install the PraisonAI package:

   pip install praisonai opencv-python moviepy
   
2. Set API Key

   Set your OpenAI API key as an environment variable in your terminal:

   export OPENAI_API_KEY="${OPENAI_API_KEY:?Set OPENAI_API_KEY in your shell}"
   
3. Create a file

   Create a new file agents.yaml with the basic setup:

   framework: praisonai
   process: sequential
   topic: analyze landmark image
   agents:  # Canonical: use 'agents' instead of 'roles'
     vision_analyst:
       name: VisionAnalyst
       role: Computer Vision Specialist
       goal: Analyze images and videos to extract meaningful information
       instructions:  # Canonical: use 'instructions' instead of 'backstory' |
         You are an expert in computer vision and image analysis.
         You excel at describing images, detecting objects, and understanding visual content.
       llm: gpt-4o-mini
       self_reflect: false
       tasks:
         analyze_landmark:
           description: Describe this famous landmark and its architectural features.
           expected_output: Detailed description of the landmark's architecture and significance
           images:
             - https://upload.wikimedia.org/wikipedia/commons/b/bf/Krakow_-_Kosciol_Mariacki.jpg
   
4. Start Agents

   Type this in your terminal to run your agents:

   praisonai agents.yaml
   

Requirements

• Python 3.10 or higher
• OpenAI API key with vision model access
• Basic understanding of Python and media handling

Understanding Multimodal Agents

What are Multimodal Agents?

Multimodal agents are designed to:

• Process multiple types of data (text, images, videos)
• Understand context across different modalities
• Generate insights from diverse media sources
• Handle complex multimedia tasks

Features

Vision Processing

Analyze images, detect objects, and understand visual content.

Video Analysis

Process video content for events and actions.

Text Extraction

Extract and analyze text from images and documents.

Cross-Modal Understanding

Integrate insights across different media types.

Need the other direction — tools that hand images back to the agent? See Multimodal Tool Output.

Multi-Agent Media Processing

Code

from praisonaiagents import Agent, Task, AgentTeam

# Create first agent for image analysis
vision_agent = Agent(
    role="Image Analyst",
    goal="Analyze visual content and extract key information",
    backstory="Expert in visual analysis and image understanding",
    llm="gpt-4o-mini",
    reflection=False
)

# Create second agent for content writing
writer_agent = Agent(
    role="Content Writer",
    goal="Create engaging content based on image analysis",
    backstory="Expert in creating compelling content from visual insights",
    llm="gpt-4o-mini"
)

# Create tasks for different media types
document_task = Task(
    description="Extract and summarize text from this document image",
    expected_output="Structured text content with key information highlighted",
    agent=vision_agent,
    images=["document.jpg"]
)

writing_task = Task(
    description="Create engaging content based on image analysis",
    expected_output="Compelling article incorporating visual insights",
    agent=writer_agent
)

# Create and start the agents
agents = AgentTeam(
    agents=[vision_agent, writer_agent],
    tasks=[document_task, writing_task],
    process="sequential"
)

result = agents.start()

No Code

framework: praisonai
process: sequential
topic: document analysis and content creation
agents:  # Canonical
  vision_analyst:
    role: Image Analyst
    goal: Analyze visual content and extract key information
    instructions:  # Canonical: use 'instructions' instead of 'backstory' Expert in visual analysis and image understanding
    llm: gpt-4o-mini
    self_reflect: false
    tasks:
      document_task:
        description: Extract and summarize text from this document image
        expected_output: Structured text content with key information highlighted
        images:
          - document.jpg

  content_writer:
    role: Content Writer
    goal: Create engaging content based on image analysis
    instructions:  # Canonical: use 'instructions' instead of 'backstory' Expert in creating compelling content from visual insights
    llm: gpt-4o-mini
    tasks:
      writing_task:
        description: Create engaging content based on image analysis
        expected_output: Compelling article incorporating visual insights

Configuration Options

# Create an agent with multimodal configuration
agent = Agent(
    role="Media Analyst",
    goal="Process multiple types of media",
    backstory="Expert in multimedia analysis",
    llm="gpt-4o-mini",  # Must support vision capabilities
      # Enable detailed logging
    reflection=False  # Optional: disable self-reflection
)

# Task with media requirements
task = Task(
    description="Analyze media content",
    expected_output="Comprehensive analysis",
    agent=agent,
    images=[  # Support multiple media sources
        "https://example.com/image1.jpg",
        "path/to/local/image.jpg",
        "path/to/video.mp4"
    ]
)

Best Practices

Use supported media formats

Prefer JPEG and PNG for images; keep file sizes reasonable and validate paths before processing.

Choose vision-capable models

Set llm to a model with vision support (e.g. gpt-4o-mini) when attaching images or video.

Write clear task descriptions

Break complex analyses into steps and specify expected output format in each task.

Prefer ephemeral attachments for large media

Use attachments= or ephemeral() so image bytes do not fill chat history.

Ephemeral Attachments

Send images to the agent for analysis without storing them in chat history. Essential for preventing context window overflow when processing multiple images.

Attachments Parameter

from praisonaiagents import Agent

agent = Agent(
    instructions="You analyze images and remember context",
    memory=True
)

# Image is analyzed but NOT stored in history
response = agent.chat(
    prompt="What's in this image?",     # ← Stored in history
    attachments=["photo.jpg"],           # ← NOT stored (ephemeral)
)

# Agent remembers the question, not the image data
response = agent.chat("What did I ask about earlier?")
# Agent: "You asked 'What's in this image?' and I told you..."

Ephemeral Context Manager

from praisonaiagents import Agent

agent = Agent(instructions="Analyze images", memory=True)

# Pre-image conversation
agent.chat("Hello, I have some photos to show you")

# Ephemeral block - nothing stored permanently
with agent.ephemeral():
    response = agent.chat(
        "Analyze this",
        attachments=["image1.jpg"]
    )
    followup = agent.chat("What about the colors?")

# After block, history is restored - images NOT persisted
agent.chat("What have we discussed?")  # Only remembers pre-image chat

History Management Methods

Clean up chat history after image analysis sessions:

from praisonaiagents import Agent

agent = Agent(instructions="Image analyst", memory=True)

# After image analysis, clean up history
agent.prune_history(keep_last=5)           # Keep only last 5 messages
agent.delete_history(-1)                    # Delete last message
agent.delete_history_matching("[IMAGE]")   # Delete all image-related messages

# Check history size
print(f"History size: {agent.get_history_size()}")

Method	Description
prune_history(keep_last=N)	Keep only last N messages
delete_history(index)	Delete message by index
delete_history_matching(pattern)	Delete messages containing pattern
get_history_size()	Get current history length
ephemeral()	Context manager for temporary conversations

Use attachments= for one-time image analysis, or ephemeral() for multi-turn image conversations that shouldn’t persist.

Example Use Cases

Document Analysis

Extract and analyze text from document images.

Security Monitoring

Monitor security feeds for suspicious activity.

Medical Imaging

Analyze medical scans for abnormalities.

Architectural Analysis

Study architectural features and designs.

Related

Multimodal Tool Output

Return images and media from tools back to the agent.

Media Pipelines

Chain AudioAgent, ImageAgent, and VideoAgent in YAML workflows.

For optimal results, ensure your media files are in supported formats and sizes for processing.