arXiv-2025 The AI Scientist-v2：Workshop-Level Automated Scientific Discovery via Agentic Tree Search

Paper: The AI Scientist-v2：Workshop-Level Automated Scientific Discovery via Agentic Tree Search

Code: https://github.com/SakanaAI/AI-Scientist-v2

AI Scientist-v2: Tree-Search Experiment Manager Agent + VLM Experiment Checking

Abstract

Artificial intelligence (AI) is playing an increasingly central role in transforming how scientific discovery is conducted. This work introduces THE AI SCIENTIST-V2, an end-to-end agentic system that produced the first fully AI-generated manuscript accepted through peer review at a workshop.

The system can:

• Iteratively formulate scientific hypotheses.
• Design and run experiments.
• Analyze and visualize data.
• Autonomously write scientific manuscripts.

Compared with the prior system (v1, Lu et al., 2024), THE AI SCIENTIST-V2 offers the following improvements:

• It removes reliance on human-written code templates.
• It generalizes effectively across diverse machine learning domains.
• It employs an innovative progressive agentic tree search approach managed by a dedicated experiment manager agent.
• It strengthens the AI review component by integrating a vision–language model (VLM) feedback loop, enabling iterative refinement of figure content and visual design.

The team evaluated THE AI SCIENTIST-V2 by submitting three fully autonomously generated papers to a peer-reviewed ICLR workshop. Notably, one paper scored above the average human acceptance threshold, marking the first time a fully AI-generated paper successfully passed peer review. This milestone highlights the growing capability of AI across the full scientific research pipeline.

The authors expect further advances in autonomous scientific discovery to profoundly affect how human knowledge is produced, enabling unprecedented scalability of research productivity and accelerating scientific breakthroughs for society. They have open-sourced the code to support future development. The paper also discusses the role of AI in science, including AI safety.

1 Introduction

1. Background and motivation

In recent years, automated scientific discovery driven by artificial intelligence (AI) has attracted broad attention. End-to-end frameworks that can autonomously formulate hypotheses, run experiments, analyze results, and write manuscripts promise to reshape the scientific process. A landmark step in this area is THE AI SCIENTIST-V1 (Lu et al., 2024), which demonstrated the feasibility of a fully automated scientific workflow and downstream paper output.

However, V1 has significant limitations that constrain broad applicability and autonomy:

• Template dependence: It relies heavily on human-written code templates; each new topic area requires manually creating a new template.
• Shallow exploration: Its linear, shallow experimental procedure limits deep exploration of scientific hypotheses.

2. Technical advances in THE AI SCIENTIST-V2

As the successor to V1, THE AI SCIENTIST-V2 makes substantial improvements that directly address these limitations. Its core contributions span three dimensions:

1. Removing template dependence: The system no longer depends on human-provided code templates, greatly increasing autonomy and enabling out-of-the-box use across multiple machine learning domains.
2. Experiment progress manager and tree search: It introduces an experiment manager agent and a novel Agentic Tree-Search Algorithm, enabling deeper, more systematic exploration of complex hypotheses.
3. Multimodal feedback: It improves the review and refinement stages by integrating vision–language model (VLM) feedback, raising the quality, clarity, and consistency of generated figures, captions, and textual interpretation.

For a concrete comparison of V1 and V2 on code drafting, execution planning, and parallel experiments, see Table 1.

3. Peer-review experiment and results

To rigorously evaluate fully autonomous paper generation, the team ran a controlled experiment: three complete manuscripts generated by THE AI SCIENTIST-V2 were submitted to a peer-reviewed ICLR workshop.

The results are striking:

• First breakthrough: One manuscript received an average review score of 6.33 (roughly top 45% of submissions).
• Acceptance threshold: That score would be sufficient for meta-review acceptance if the paper were human-written.
• Milestone: This marks the first fully AI-generated manuscript to pass a formal peer-review process.

4. Content and review analysis of the accepted paper

The accepted paper investigates whether adding an explicit compositional regularization term during neural network training improves compositional generalization. It hypothesizes that penalizing large deviations between consecutive timestep embeddings in sequence models can encourage compositional behavior.

Experiments and feedback:

• Experimental conclusion: Evaluation on a synthetic arithmetic expression dataset shows compositional regularization did not yield significant gains and occasionally hurt performance.
• Reviewer assessment: Reviewers appreciated clear identification of challenges with compositional regularization and reporting of negative results.
• Remaining gaps: Reviewers noted insufficient argumentation and intuition for why the chosen regularizer should enhance compositionality.
• Internal author assessment: Post-hoc analysis found room to improve method descriptions (e.g., which component is regularized), dataset overlap, and figure caption accuracy.
• Overall qualitative view: Reviewers viewed the work as an interesting, technically solid workshop contribution, but further development and broader experiments would be needed for main-conference rigor.

5. Introduction summary: four overarching contributions

The paper summarizes the main contributions of THE AI SCIENTIST-V2 as follows:

1. System innovation: An enhanced automated scientific discovery framework with agentic tree search, VLM feedback, and parallel experiments, improving autonomy and exploration depth.
2. Milestone: First demonstration that an AI-generated manuscript can pass peer review at a recognized machine learning workshop.
3. In-depth insight: Comprehensive internal evaluation of peer-review feedback and system outputs, clarifying the current state and limits of AI papers relative to conventional human papers.
4. Open-source release: Full codebase and ICLR 2025 workshop experiment data are open-sourced to foster community discussion on the evolving role of AI in science.

2 Background

The paper notes that THE AI SCIENTIST-V1 (Lu et al., 2024) was the first AI system to automate the full pipeline from scientific discovery to presentation of results.

• Core capability: Given a benchmark code template, V1 could autonomously write code, run experiments, visualize results, and produce a complete scientific paper.
• Main limitations:
  • Template dependence: V1 relied heavily on human-designed benchmark templates, limiting autonomy and preventing true out-of-the-box use because humans still had to pre-write outline code for experiments.
  • Linear workflow: Experiments followed a strict linear hypothesis-testing procedure, lacking depth and flexibility on complex research questions.

1. Language model agent scaffolding

To improve large language models (LLMs) on complex reasoning tasks, researchers have developed several agent architectures:

• Reflexion-style scaffolding: For example Reflexion (Shinn et al., 2024), which iteratively reflects on prior responses and self-evaluates to drive improvement. This improves robustness but adds compute cost and slower inference.
• Tree search strategies: Another promising direction combines tree search with LLMs (Jiang et al., 2025), enabling structured exploration of reasoning paths. This improves comprehensiveness at the cost of higher complexity and compute.

2. Combining LLMs with tree search

The team observed that automated research in V1 often exhibited “short-sighted” experimentation. Human science, by contrast, relies on open-ended hypothesis generation, finding stepping-stones, and iterative hypothesis refinement.

• AIDE: Inspired by recent work treating code generation as an action space, AIDE (Jiang et al., 2025) combines LLM-based code generation with tree search and leads on the MLEBench machine learning engineering benchmark.
• Mechanism: In AIDE, each tree node represents a candidate solution state with a scalar evaluation score (e.g., validation accuracy). The system iteratively selects nodes to debug or refine based on these scores.

THE AI SCIENTIST-V2 draws on this approach, integrating a similar tree-search exploration strategy into its automated scientific discovery framework, adapted to the multi-stage nature of scientific experiments.

3 The AI Scientist-v2

3.1 More General Idea Generation

Unlike the prior system, which focused mainly on incremental edits to an existing codebase, V2 begins research at a higher level of abstraction.

• Open-ended thinking: The system is prompted to think more openly about potential research directions, hypotheses, and experimental designs, akin to writing a research abstract or grant proposal before implementation.
• Beyond code constraints: This encourages more novel or foundational ideas rather than being limited by predefined code structure.
• Literature retrieval: This stage integrates tools such as Semantic Scholar, allowing the system to query the literature to assess novelty and identify prior work.

3.2 Removing Template Dependency

3.2.1 Experiment Progress Manager

To mirror structured real-world experimentation, the system introduces an experiment progress manager agent that coordinates four defined experimental stages:

• Stage 1: Preliminary Investigation: Build a minimal viable prototype from the generated research idea to establish initial feasibility and correctness.
• Stage 2: Hyperparameter Tuning: Refine the initial implementation by optimizing key hyperparameters (e.g., learning rate, epochs) to create a robust experimental baseline.
• Stage 3: Research Agenda Execution: Systematically implement the core research agenda on top of the tuned baseline.
• Stage 4: Ablation Studies: Systematically assess the importance of research components to support main findings rigorously.

Each stage has explicit stopping criteria. Stage 1 ends after the prototype runs successfully; stage 2 ends when training curves converge and runs succeed on at least two datasets; stages 3 and 4 end when the allocated compute budget is exhausted. At the end of each stage, the manager uses a dedicated LLM evaluator to select the best-performing node as the seed for the next stage and launches repeated runs to obtain statistics such as mean and standard deviation.

3.2.2 Parallelized Agentic Tree Search

THE AI SCIENTIST-V2 abandons V1’s strict linear pipeline in favor of a more flexible, exploratory tree search approach inspired by recent work. The method spans the four experimental stages above and supports more systematic exploration of scientific hypotheses.

Node execution cycle: Each experimental node in the tree undergoes a full cycle:

• Generation and execution: The LLM first produces a concrete experiment plan and corresponding Python code, then runs it immediately in an interpreter.
• Plotting and VLM review: If execution succeeds, the system generates visualizations from stored results and sends them to a vision–language model (VLM) for review.
• Status labeling: Nodes that fail at runtime or fail VLM review (e.g., missing labels, visually misleading plots) are marked Buggy; others are Non-buggy.

Search and expansion strategy: At each iteration the system selects multiple nodes in parallel for expansion. With a predefined probability it selects Buggy nodes for debugging, or uses an LLM-guided best-first strategy to select Non-buggy nodes for refinement.

Specialized node variants: To meet specific experimental needs, the system defines several node types:

• Hyperparameter nodes: Explore configurations in stage 2 while avoiding duplicate experiments.
• Ablation nodes: Assess component importance in stage 4.
• Replication nodes: Repeat parent experiments with different random seeds to strengthen robustness.
• Aggregation nodes: Perform no new experiment; only generate scripts to aggregate replication results and plot statistics with mean and standard deviation.

This structured staging and customized node types let the system steer the full research cycle flexibly while preserving coherence across iterative stages. See Figure 2 for stage-specific tree-search flow diagrams.

3.3 Dataset Loading via Hugging Face

Unified framework: Hugging Face Hub provides a convenient, unified way to access many common datasets, often with predefined train/validation/test splits.

Automated download: In THE AI SCIENTIST-V2, the system is prompted to use Hugging Face Hub whenever possible. It downloads required datasets automatically via the standard one-line datasets.load_dataset call.

Benefits and limitations:

• Simplified handling: This standardized approach greatly simplifies dataset handling.
• Compatibility issues: The authors acknowledge this is somewhat ad hoc because not all dataset repositories support this download path.

3.4 Vision-Language Model Reviewer

Unlike THE AI SCIENTIST-V1, which lacks vision, V2 introduces vision–language models (VLMs) at two points in the research workflow:

1. Two application stages

• Experiment stage: During tree-based experimentation, the VLM gives immediate feedback on generated plots, ensuring visualizations communicate results effectively and accurately.
• Paper writing stage: During manuscript reflection, the VLM evaluates figures and captions, improving visual clarity and coherence in the final paper.

2. Concrete workflow

During paper writing, the system:

• Extracts information: Captures screenshots of figures together with their captions.
• Links context: Identifies and extracts in-text references to each figure via the keyword “Figure X”.
• Holistic evaluation: Supplies both image content and textual references to the VLM for multiple quality checks.

3. VLM quality checks

The VLM reviewer performs rigorous checks including:

• Alignment verification: Whether figure content matches the caption.
• Visual clarity: Issues such as missing legends or ambiguous labels.
• Duplicate detection: Whether the same figure appears in both the main text and appendix.

4. Effect

Iterative VLM feedback substantially improves visual quality and clarity in manuscripts from THE AI SCIENTIST-V2. The placement of this pipeline in the overall workflow is shown in Figure 1.

4 Human Evaluation of Manuscripts Generated by The AI Scientist -v2

The team describes how they validated the system through real interaction with the human research community. In collaboration with organizers of the ICLR 2025 “I Can’t Believe It’s Not Better” (ICBINB) workshop, they ran a human evaluation study by submitting fully automatically generated papers to the official peer-review process.

4.1 Evaluation Methodology

The evaluation followed four controlled steps:

1. AI-generated submissions:
   • The team used THE AI SCIENTIST-V2 to generate three complete manuscripts.
   • Generation used only high-level topic prompts aligned with the workshop scope; hypothesis formulation, experiment design, programming, data analysis, visualization, and writing were fully autonomous, with no human editing or intervention.
2. Blind peer review:
   • The three AI papers were mixed among 43 total ICBINB submissions.
   • Reviewers were told in advance that some submissions might be AI-generated but not which ones came from THE AI SCIENTIST-V2. Reviewers could decline to review such papers.
3. Review outcomes and acceptance decisions:
   • Among the three submissions, one received scores high enough for acceptance, with an average of $6.33$ (out of 10, individual scores 6, 6, 7), above the workshop acceptance threshold.
   • The other two scored lower and were not accepted. The accepted paper is illustrated in Figure 3.
4. Post-review withdrawal:
   • The team agreed in advance with ICLR leadership that all accepted AI papers would be withdrawn after review and would not enter the formal scientific record.
   • This avoids setting precedent before the community reaches consensus. Reviewers were informed of the experiment only after reviews concluded.

Internal evaluation and observations

Beyond official reviews, the team conducted in-depth internal assessment:

• Quality judgment: All three papers did not yet meet main-track acceptance standards at top international conferences.
• Level alignment: Internal conclusions matched official reviews: one paper did reach the bar for a top machine learning workshop.
• Observed flaws:
  • The system occasionally introduced citation errors, similar to common LLM “hallucination”.
  • Despite running standard experimental pipelines, methodological rigor and depth of analysis remained limited.

Transparency and ethical considerations

• Ethics approval: The study received institutional review board (IRB) approval from the University of British Columbia (ID: H24-02652).
• Transparency requirements: The authors stress that all AI-generated papers should be clearly labeled, and THE AI SCIENTIST is designed to ensure this.

4.2 The first AI-generated peer-reviewed workshop paper

1. Paper generation process

Generating the accepted paper involved multiple automated stages under high-level human guidance:

• Initial idea generation: The system first produced about 20 initial ideas aligned with the workshop (ICBINB) theme.
• Prompt adjustment: Researchers modified system prompts to emphasize deep learning in real-world domains such as finance and psychology; the system then generated about 20 additional ideas.
• Idea selection: From this pool, human researchers chose three of the most promising starting ideas—analogous to a PI deciding which directions to fund, without editing the ideas themselves.
• Autonomous execution: For selected ideas, the system autonomously ran the full pipeline: code generation, execution, analysis, and writing.
• Multi-seed runs: The system ran multiple times with different random seeds; human researchers ultimately selected the most coherent complete manuscript for submission.

2. Core content of the accepted paper

The paper Enhancing Compositional Generalization in Neural Networks via Compositional Regularization studies the effect of explicit regularization on model capabilities:

• Core hypothesis: Adding a compositional regularization term to the training loss, by penalizing large changes between consecutive timestep embeddings in sequence models, encourages compositional representations.
• Experimental results: Tests on a synthetic arithmetic expression dataset yielded negative conclusions: compositional regularization did not materially improve generalization and worsened as expression complexity increased.
• Final conclusion: Enforcing compositional structure through regularization alone is insufficient; the paper notes potential conflict between the regularizer and the primary learning objective. The paper’s appearance is shown in Figure 3.

3. Authors’ internal assessment

The team performed strict self-review and identified the following strengths and weaknesses:

• Strengths: Appropriate experimental task and informative exploration of negative results.
• Technical issues:
  • Ambiguous description: Unclear whether regularization applies to “input embeddings” or “LSTM hidden states”.
  • Citation problems: Missing core references (e.g., Hochreiter & Schmidhuber, 1997) and over-reliance on generic textbook citations.
  • Data overlap: Code review found roughly 57% overlap between training and test sets, severely undermining credibility.
  • Misinterpretation: Figure 3’s caption misread validation loss; Figure 5’s conclusion contradicted what the plot showed.

4. Human reviewer comments

The paper received a strong score of 6.33 at ICBINB (reviewer scores 6, 7, 6), placing it in the top 45%:

• Reviewer #1 (7 / accept): Found the idea of compressing hidden representations to encourage compositionality very interesting; ablations were well done and explained why the method underperformed.
• Reviewer #2 (6 / accept): Viewed the paper as a perfect fit for the workshop focus on “negative or surprising results” and an excellent discussion case.
• Common suggestions: Reviewers generally recommended more intuitive explanation and validation on additional architectures such as Transformers.

Overall, although the paper did not yet meet main-conference rigor, reviewers agreed that analysis of the negative results had scientific value sufficient for workshop acceptance.

5 Limitations & Ethical Considerations

1. Interpreting the achievement soberly

Although THE AI SCIENTIST-V2 produced an accepted workshop paper, the authors emphasize keeping the milestone in perspective:

• Venue level: Acceptance was at workshop level, not the main conference track, and only one of three submissions was accepted.
• Acceptance-rate context: Workshop papers often report preliminary or exploratory work; acceptance rates (typically 60–80%) are much higher than top ML conferences (main-track rates at ICLR, ICML, and NeurIPS are often 20–30%).
• Consistency: The current system cannot reliably meet main-conference rigor and does not guarantee consistent quality even at workshop level.

2. Deeper challenges in scientific inquiry

Even with structured agentic tree search, core aspects of research remain very hard for fully automated systems:

• Formulating genuinely novel, high-impact hypotheses.
• Designing truly innovative experimental methodology.
• Applying deep domain expertise to justify design choices rigorously.
• Addressing these gaps is key to moving from preliminary or incremental results toward high-quality, conference-level contributions.

3. Governance, ethics, and community consensus

The team took a cautious approach and calls for community norms:

• Oversight and compliance: The study received University of British Columbia IRB approval (H24-02652) and was conducted in close coordination with ICLR leadership and workshop organizers.
• Withdrawal and transparency: To avoid premature precedent before community consensus, all accepted AI-generated papers were withdrawn from the public OpenReview forum per agreement and do not appear in the formal record.
• Industry standards: Science needs norms for AI-generated research, including disclosure requirements and timing of release. Transparency is encouraged, but whether anonymous review should precede disclosure to reduce bias remains open.

4. Guarding against research misconduct

The authors warn about potential misuse:

• Gaming peer review: The technology must not become a tool to game peer-review systems.
• CV padding: Unethical use to inflate citations or résumé credentials would undermine the meaning of peer review and scientific evaluation.