AgentOps logo AgentOps provides session replays, metrics, and monitoring for AI agents. At a high level, AgentOps gives you the ability to monitor LLM calls, costs, latency, agent failures, multi-agent interactions, tool usage, session-wide statistics, and more. For more info, check out the AgentOps Repo.
📊 Replay Analytics and DebuggingStep-by-step agent execution graphs
💸 LLM Cost ManagementTrack spend with LLM foundation model providers
🧪 Agent BenchmarkingTest your agents against 1,000+ evals
🔐 Compliance and SecurityDetect common prompt injection and data exfiltration exploits
🤝 Framework IntegrationsNative Integrations with CrewAI, AG2, & LangChain

Agent Dashboard

Agent Dashboard
Session Analytics
Session Replays

​
Installation

AgentOps works seamlessly with applications built using AG2.
  1. Install AgentOps
pip install agentops
  1. Create an API Key: Create a user API key here: Create API Key
  2. Configure Your Environment: Add your API key to your environment variables 
AGENTOPS_API_KEY=<YOUR_AGENTOPS_API_KEY>
  1. Initialize AgentOps
To start tracking all available data on AG2 runs, simply add two lines of code before using AG2. 
import agentops
agentops.init() # Or: agentops.init(api_key="your-api-key-here")
After initializing AgentOps, AG2 will now start automatically tracking your agent runs.

​
Features

  • LLM Costs: Track spend with foundation model providers
  • Replay Analytics: Watch step-by-step agent execution graphs
  • Recursive Thought Detection: Identify when agents fall into infinite loops
  • Custom Reporting: Create custom analytics on agent performance
  • Analytics Dashboard: Monitor high level statistics about agents in development and production
  • Public Model Testing: Test your agents against benchmarks and leaderboards
  • Custom Tests: Run your agents against domain specific tests
  • Time Travel Debugging: Save snapshots of session states to rewind and replay agent runs from chosen checkpoints.
  • Compliance and Security: Create audit logs and detect potential threats such as profanity and PII leaks
  • Prompt Injection Detection: Identify potential code injection and secret leaks

​
AG2 + AgentOps examples

Edit this page