Zochi: The World's First Artificial Scientist

Project Lead: Andy Zhou

Core Contributors: Ron Arel, Soren Dunn, Nikhil Khandekar

1. Introduction

Updated May 27, 2025: The technical report and code cover an earlier version of Zochi. Zochi’s capabilities have greatly expanded, culminating in acceptance to ACL 2025!

Zochi is an artificial scientist system capable of end-to-end scientific discovery, from hypothesis generation through experimentation to peer-reviewed publication. Unlike previous systems that automate isolated aspects of scientific research, Zochi demonstrates comprehensive capabilities across the complete research lifecycle.

We present empirical validation through multiple peer-reviewed publications accepted at ICLR 2025 workshops and ACL 2025, each containing novel methodological contributions and state-of-the-art experimental results. These include Compositional Subspace Representation Fine-tuning (CS-ReFT), which achieved a 93.94% win rate on the AlpacaEval benchmark on Llama-2-7b while using only 0.0098% of model parameters, the Tempest (formerly Siege) framework, a state-of-the-art jailbreak which identified critical vulnerabilities in language model safety measures through multi-turn adversarial testing.

Figure 1: Comparative analysis of automated reviewer ratings across AI research systems. Zochi achieves an average score of 7.67 on the NeurIPS guidelines scale, significantly exceeding the human acceptance threshold of 6, while previous systems fall below this threshold.

2. Research Publications

Tempest: Autonomous Multi-Turn Jailbreaking of Large Language Models with Tree Search (ACL 2025)

Note: This is an updated version of Siege, which was accepted to ICLR 2025 BuildingTrust

Tempest represents a significant advancement in safety testing methodology by formalizing how minor policy breaches can accumulate over successive conversation turns and by employing beam search to explore multiple attack strategies in parallel. The framework treats each conversation state as a node in a search tree, with the central innovation being a sophisticated partial compliance tracking mechanism that identifies and exploits incremental policy leaks.

Results

CS-ReFT: Compositional Subspace Representation Fine-tuning for Adaptive Large Language Models (ICLR 2025 SCOPE)

CS-ReFT embodies a fundamentally different paradigm compared to existing approaches. While methods like LoRA implement orthogonality constraints at the weight level, CS-ReFT applies these constraints directly to hidden-state representations. This innovation allows each task to have its dedicated subspace transformation, which eliminates interference while still enabling composition through a lightweight router mechanism. Part of codebase adapted from ReFT.

Results

3. Automated Review Scores

Our evaluation framework is built on an automated reviewer system from the AI Scientist that processes research papers based on the NeurIPS conference review guidelines, assigning numerical scores for soundness, presentation, contribution, and overall quality. The scoring scale ranges from 1 to 10, with 6 representing the acceptance threshold at top machine learning conferences.

4. Reproducing Zochi's Results

Zochi generated the code and main results for both papers, starting from publicly available repositories that were retrieved from baseline methods. Some baseline results were retrieved from existing papers using the same experimental setting. The final codebase was cleaned up and modified to remove traces of Zochi's intermediate research process and allow for easier reproducibility.

CS-ReFT

# Clone the repository
git clone https://github.com/zochi-ai/zochi.git
cd zochi/csreft

# Install dependencies
pip install -r requirements.txt

# Train CS-ReFT on Llama-2-7B and get outputs for AlpacaEval
python csrf_train_instruct.py --output_dir <output_dir> --run_eval

Tempest

# Move back to the repository root
cd ..

# Install Tempest dependencies
pip install -r tempest/requirements.txt

# Enter the Tempest directory
cd tempest

# Run Tempest on a target model
python tempest_pipeline.py --target_model <target_model> --pipeline_model <pipeline_model> --results_json <results_json>

# Evaluate results
python get_metrics.py <results_json>

Using a Local Ollama Model

If you run a model locally with Ollama, prefix the model name with local/ and ensure the Ollama server is running. For example:

# Use a locally hosted DeepSeek model
python tempest_pipeline.py --target_model local/deepseek-llm-r1-8b --pipeline_model local/deepseek-llm-r1-8b --results_json results.json

Set the OLLAMA_BASE_URL environment variable if your server is not available at http://localhost:11434.

6. Citation

@article{zochi2025,
  title={Zochi Technical Report},
  author={Intology},
  journal={arXiv},
  year={2025}
}

7. License

This repository is released under the MIT License. See the LICENSE file for more details.