Tree of Agents (TOA)

This project is associated with the paper "Tree of Agents: Improving Long-Context Capabilities of Large Language Models through Multi-Perspective Reasoning," which has been accepted for publication as part of EMNLP 2025 Findings. 📄

This project provides a modular framework for building and evaluating multi-agent reasoning systems on long documents such as novels or stories. It is designed to work with large language models (LLMs) through APIs (OpenAI-compatible, local deployment, etc.), allowing agents to collaborate and debate over document comprehension tasks.

• Modular and extensible agent design (agent.py)
• Dataset loader with preprocessing utilities (data_utils.py)
• Multi-round collaborative reasoning pipeline (run.py)
• Unified and structured prompting logic (utils.py)
• Supports OpenAI API and local LLM deployments
• Automatic retry and logging for robustness

├── run.py                # Entry point for running experiments
├── agent.py              # Defines agent classes for different backends
├── data_utils.py         # Dataset loading and context preparation
├── utils.py              # Prompt templates, parsing, voting, etc.
├── results/              # Stores experiment outputs
├── logs/                 # Stores runtime logs
├── .env.example          # Sample environment variable configuration
└── requirements.txt      # Python dependencies

pip install -r requirements.txt

Create a .env file:

API_KEY=your-api-key
BASE_URL=http://127.0.0.1:8000/v1
TOKENIZER_PATH=/path/to/llama/tokenizer

You can load this via:

from dotenv import load_dotenv
load_dotenv()

Place datasets like DetectiveQA or NovelQA under the datasets/ directory. These should be in .pkl format with the expected structure.

You can find datasets in Phospheneser/DetectiveQA and NovelQA/NovelQA.

python run.py   --model llama   --dataset DetectiveQA   --sample_num 100   --agent_num 5   --repetition_num 1

• --model: model name (e.g., llama, deepseek)
• --dataset: dataset name (DetectiveQA, NovelQA)
• --sample_num: number of examples to process
• --agent_num: number of agents in the reasoning group
• --repetition_num: how many times to repeat the full run

• Logs are saved to logs/
• Result JSONs are saved to results/
• Each decision contains the final prediction and ground truth for evaluation

If you use this codebase in your research, please cite appropriately or link back to the repository.

This project is licensed under the MIT License. See the LICENSE file for details. 📝

Feel free to extend the agents, customize the prompts, or plug in your own LLMs for research or product development. 🚀