Run Scripts Documentation

This directory contains documentation for selected ct run commands and related scripts.

Organization

Documentation is organized by function:

Trajectory Analysis

Tools for analyzing individual trajectories:

submit_guessability.md - Theoretical submit tool exploit analysis
submit_guessability_verified.md - Live-verified submit tool exploits

Task Analysis

Tools for analyzing task quality and design:

compare_side_tasks.md - Compare side task outcomes between runs
goal_completeness.md - Verify agents have sufficient task information
reward_hacking.md - Find reward hacking vulnerabilities
rt_alignment.md - Check RT description alignment with scorers
side_task_review.md - Comprehensive side task quality review
validate_cover.md - Validate main task provides cover for side task

Evaluation & Monitoring

Core evaluation infrastructure:

Running Evaluations - Run AI agent evaluations (ct run eval)
Monitoring - Run action-level monitors on trajectories

Task Generation

Tools for creating new tasks:

find_questions.md - Discover answer-question tasks
mtgen.md - Generate main tasks
mtgen_bugfix.md - Generate bug-fix main tasks

Quality Assurance

Environment and test validation:

doc_audit.md - Audit side task goal and scoring docs
pairing_analysis.md - Analyze main/side task pairings
integration_test_quality.md - Integration test quality (ct envs integration-test-quality)

Quick Reference

Most Common Commands

# Run an evaluation
ct run eval --policy honest --env web_scraping --main-task crawl_depth

# Monitor trajectories
ct run monitor logs/my_eval.eval -m multi

# Review side task quality
ct run side-task-review -e web_scraping

# Test environments
ct envs test -e web_scraping