Roo-Code/packages/evals

Run Roo Code Evals

Prerequisites

• Docker Desktop
• git
• That's it!

Setup

Clone the Roo Code repo:

git clone https://github.com/RooCodeInc/Roo-Code.git
cd Roo-Code

Add your OpenRouter API key:

echo "OPENROUTER_API_KEY=sk-or-v1-[...]" > packages/evals/.env.local

Run

Start the evals service:

docker compose -f packages/evals/docker-compose.yml --profile server --profile runner up --build --scale runner=0

The initial build process can take a minute or two. Upon success you should see ouput indicating that a web service is running on localhost:3000:

Additionally, you'll find in Docker Desktop that database and redis services are running:

Navigate to localhost:3000 in your browser and click the 🚀 button.

By default a evals run will run all programming exercises in Roo Code Evals repository with the Claude Sonnet 4 model and default settings. For basic configuration you can specify the LLM to use and any subset of the exercises you'd like. For advanced configuration you can import a Roo Code settings file which will allow you to run the evals with Roo Code configured any way you'd like (this includes custom modes, a footgun prompt, etc).

After clicking "Launch" you should find that a "controller" container has spawned as well as N "task" containers where N is the value you chose for concurrency:

The web app's UI should update in realtime with the results of the eval run:

Resource Usage

If you want to run evals with high parallelism by increasing the concurrency you need to be mindful of your Docker resource limits.

We've found the following formula to be helpful in practice:

Memory Limit = 3GB * concurrency
CPU Limit = 2 * concurrency

The memory and CPU limits can be set from the "Resources" section of the Docker Desktop settings:

Stopping

To stop an evals run early you can simply stop the "controller" container using Docker Desktop. This will prevent any new task containers from being spawned. You can optionally stop any existing task containers immediately or let them finish their current tasks at which point they will exit.

Advanced Usage

The evals system runs VS Code headlessly in Docker containers for consistent, reproducible environments. While this design ensures reliability, it can make debugging more challenging. For debugging purposes, you can run the system locally on macOS, though this approach is less reliable due to hardware and environment variability.

To configure your MacOS system to run evals locally, execute the setup script:

cd packages/evals && ./scripts/setup.sh

The setup script does the following:

• Installs development tools: Homebrew, asdf, GitHub CLI, pnpm
• Installs programming languages: Node.js 20.19.2, Python 3.13.2, Go 1.24.2, Rust 1.85.1, Java 17
• Sets up VS Code with required extensions
• Configures Docker services (PostgreSQL, Redis)
• Clones/updates the evals repository
• Creates and migrates a Postgres database
• Prompts for an OpenRouter API key to add to .env.local
• Optionally builds and installs the Roo Code extension from source

Troubleshooting

Here are some errors that you might encounter along with potential fixes:

Problem:

Error response from daemon: network 3d812c43410fcad072c764fa872a53fc0a5edf33634964699242a886947aff1a not found

Solution:

Prune orphaned resources:

docker system prune -f