Roo-Code/packages/evals/ARCHITECTURE.md

Evals System Architecture

Overview

The evals system is a distributed evaluation platform that runs AI coding tasks in isolated VS Code environments. It solves two critical problems in AI evaluation:

1. Dependency Management: Eliminates the complexity of setting up multiple programming language environments by packaging everything into pre-configured containers
2. Resource Isolation: Prevents memory exhaustion and state contamination by running each task in a fresh, isolated container instead of sequentially in a single VS Code instance

The architecture consists of three main components: a Next.js web application for management, a controller container that orchestrates evaluation runs, and multiple runner containers that execute individual tasks.

Problems Solved

Simplified Setup and Deployment

Traditional AI evaluation setups require complex dependency management across multiple programming languages, development tools, and VS Code extensions. The evals system eliminates this friction by:

• One-Command Deployment: Single docker compose up command starts the entire evaluation infrastructure
• Pre-configured Environments: Runner containers include all necessary language runtimes, tools, and VS Code extensions
• Dependency Isolation: No host system contamination or version conflicts between different language requirements
• Reproducible Environments: Identical evaluation conditions across different machines and deployments

Resource Management and Isolation

Running multiple AI evaluation tasks sequentially in a single VS Code instance creates several problems:

• Memory Accumulation: VS Code instances gradually consume more memory with each task, eventually leading to crashes
• State Contamination: Previous tasks can leave behind files, settings, or processes that affect subsequent evaluations
• Resource Contention: Multiple tasks competing for the same VS Code instance create bottlenecks and inconsistent performance
• Failure Propagation: A single problematic task can crash the entire evaluation session

The containerized approach solves these issues by:

• Fresh Environments: Each task starts with a clean VS Code instance and workspace
• Memory Reset: Container termination automatically reclaims all memory and resources
• Parallel Execution: Multiple tasks can run simultaneously without interference
• Fault Isolation: Individual task failures don't affect other running evaluations

Architecture Components

graph TB
    Web[Admin Web App] <--> Redis[(Redis<br/>PubSub & Registration)]
    Web <--> DB[(PostgreSQL<br/>Runs & Tasks)]
    Web --> Controller[Run Controller / PQueue]

    Controller <--> DB
    Controller --> Runner1[Task Runner 1]
    Controller --> Runner2[...]
    Controller --> RunnerN[Task Runner N]

    Runner1 <--> Redis
    Runner2 <--> Redis
    RunnerN <--> Redis

    Redis <--> Web

Core Components

Next.js Web Application

The web application serves as the primary interface for creating and monitoring evaluation runs. It provides:

• Run Management: Create evaluation runs with configurable parameters (model, concurrency, exercise selection)
• Real-time Monitoring: Live progress tracking via Server-Sent Events
• Results Dashboard: View task completion status, metrics, and outcomes
• Container Orchestration: Spawns controller containers for new runs

Controller Container

A specialized instance of the evals-runner container that acts as the run orchestrator. The controller:

• In-Memory Task Queue: Uses the p-queue npm package to manage task distribution with configurable concurrency limits
• Git Workspace Setup: Prepares exercise repositories and manages version control
• Runner Coordination: Spawns and monitors individual task runner containers
• Heartbeat Monitoring: Maintains Redis heartbeat to track controller health
• Result Aggregation: Collects task results and finalizes run metrics

Runner Containers

Individual containers that execute single evaluation tasks. Each runner:

• Isolated Environment: Fresh VS Code instance with pre-installed language tools and extensions
• Task Execution: Runs AI agent with evaluation prompt in VS Code environment
• IPC Communication: Connects to VS Code via Unix socket for real-time interaction
• Unit Testing: Validates task completion using language-specific test suites
• Metrics Collection: Tracks token usage, costs, tool usage, and execution time

Supporting Infrastructure

• Redis: Provides pub/sub messaging for real-time events and runner registration tracking (not used for task queuing)
• PostgreSQL: Stores run configurations, task definitions, execution metrics, and results
• Docker: Container orchestration for isolation and scalability

Execution Flow

1. Run Initialization

The web application creates an evaluation run with specified parameters:

• Suite Type: Full evaluation (all exercises) or partial (selected exercises)
• Model Configuration: AI model selection and settings
• Concurrency: Number of parallel task executions (1-25)
• Exercise Selection: Programming language and specific coding challenges

2. Controller Deployment

The web application spawns a controller container that:

• Loads Run Configuration: Retrieves run parameters and associated tasks from database
• Prepares Workspace: Sets up git repository with exercise code and test suites
• Establishes Monitoring: Starts Redis heartbeat and event publishing
• Creates Task Queue: Initializes concurrent task processing with specified limits

3. Task Distribution

The controller distributes tasks across runner containers using an in-memory queue:

• p-queue Management: Uses the p-queue npm package to manage task concurrency in memory
• Container Spawning: Creates isolated runner containers for each task
• Resource Management: Enforces concurrency limits to prevent resource exhaustion
• Task Assignment: Each runner receives a single task with full context
• Progress Tracking: Monitors runner registration and task status via Redis pub/sub

4. Task Execution

Individual runners execute evaluation tasks:

• Environment Setup: Launches VS Code with Roo extension in isolated container
• Prompt Delivery: Sends evaluation prompt to AI agent via IPC
• Code Generation: AI agent writes code using available tools and context
• Real-time Events: Publishes progress updates, token usage, and completion status
• Validation: Runs language-specific unit tests to verify task completion

5. Result Collection

The system aggregates and reports results:

• Event Streaming: Real-time progress updates flow from runners through Redis to web interface
• Metrics Aggregation: Controller collects execution metrics, costs, and success rates
• Run Completion: Final results stored in database with comprehensive analytics
• Cleanup: Containers terminated and resources released

Technical Implementation

CLI System

The evaluation system is driven by a command-line interface that can operate in two modes:

• Run Mode: Orchestrates complete evaluation runs with multiple tasks
• Task Mode: Executes individual tasks within runner containers

The CLI automatically detects its execution environment and adapts behavior accordingly, using containerized task execution when running within Docker.

Container Architecture

Both controller and runner containers use the same base image but serve different purposes:

Runner Container Features

• Multi-language Support: Pre-installed runtimes for Go, Java, JavaScript, Python, and Rust
• Development Tools: VS Code with language-specific extensions and Roo Code extension
• Containerization: Docker-in-Docker capability for nested container execution
• Exercise Repository: Git clone of evaluation exercises with test suites

Container Isolation

Each task executes in complete isolation with:

• Fresh VS Code Instance: Clean environment with no shared state
• Dedicated Workspace: Task-specific directory with relevant exercise files
• Resource Limits: Controlled CPU and memory allocation
• Network Isolation: Containers communicate only through Redis pub/sub

Communication Architecture

The system uses multiple communication channels:

IPC (Inter-Process Communication)

• Unix Sockets: Direct communication between CLI and VS Code extension
• Event Streaming: Real-time task progress and AI agent interactions
• Command Interface: Task lifecycle management (start, cancel, close)

Redis Pub/Sub

• Event Broadcasting: Task events published to run-specific channels
• Runner Registration: Active runner tracking per evaluation run
• Heartbeat Monitoring: Controller health and availability status
• Not Used for Queuing: Task queue management is handled in-memory by the controller using p-queue

HTTP/SSE

• Web Interface: REST API for run management and configuration
• Real-time Updates: Server-Sent Events for live progress monitoring
• Result Retrieval: Task metrics and completion status

Task Lifecycle Management

Each evaluation task follows a structured lifecycle:

1. Initialization: Container startup and VS Code launch
2. Connection: IPC socket establishment and extension activation
3. Prompt Delivery: Evaluation challenge sent to AI agent
4. Execution: AI agent writes code using available tools
5. Validation: Unit test execution to verify correctness
6. Cleanup: Container termination and resource cleanup

Error Handling and Timeouts

The system implements comprehensive error handling:

• Task Timeouts: 30-minute maximum execution time per task
• Process Cleanup: Automatic termination of hung processes
• Container Recovery: Failed containers are cleaned up and resources released
• Graceful Degradation: Individual task failures don't affect other tasks in the run

Metrics and Monitoring

Comprehensive tracking of evaluation performance:

• Token Usage: Input/output tokens and context size tracking
• Cost Analysis: API costs per task and aggregated run costs
• Tool Usage: Frequency and success rates of different AI tools
• Execution Time: Task duration and queue wait times
• Success Rates: Pass/fail statistics across languages and exercises

Configuration and Customization

Run Configuration

Evaluation runs support extensive customization:

• Model Selection: Choose from available AI models via OpenRouter integration
• Concurrency Control: 1-25 parallel task executions based on resource availability
• Exercise Selection: Full suite (all exercises) or partial (selected exercises)
• Custom Settings: Override default AI agent configuration and behavior
• System Prompts: Optional custom prompts for specialized evaluation scenarios

Exercise Management

The system uses a separate Git repository containing:

• Language-specific Exercises: Coding challenges organized by programming language
• Test Suites: Automated validation for each exercise
• Prompt Templates: Standardized evaluation instructions per language
• Workspace Configuration: Language-specific development environment setup

Scalability Considerations

The architecture supports horizontal scaling:

• Container Orchestration: Multiple controller instances can run simultaneously
• Resource Management: Configurable concurrency prevents resource exhaustion
• Database Optimization: Efficient task querying and result storage
• Redis Clustering: Pub/sub system can scale with message volume

Operational Characteristics

Performance

• Task Isolation: Complete environment isolation prevents interference between tasks
• Parallel Execution: Configurable concurrency maximizes resource utilization
• Efficient Communication: Unix sockets and Redis provide low-latency messaging
• Resource Cleanup: Automatic container termination prevents resource leaks

Reliability

• Fault Tolerance: Individual task failures don't impact other tasks
• Timeout Management: Prevents hung tasks from consuming resources indefinitely
• Health Monitoring: Controller heartbeat and runner registration tracking
• Graceful Shutdown: Proper cleanup of containers and database connections

Observability

• Real-time Monitoring: Live progress tracking through web interface
• Comprehensive Logging: Detailed execution logs for debugging and analysis
• Metrics Collection: Performance and cost analytics for optimization
• Event Auditing: Complete task lifecycle tracking for accountability

This architecture provides a robust, scalable platform for evaluating AI coding capabilities across multiple programming languages while maintaining strict isolation and comprehensive monitoring.