๐ Introduction
March 19, 2025 ยท View on GitHub
๐ Introduction
DeepLiterature is an open-source science deep research framework, jointly launched by the Institute of Automation, Chinese Academy of Sciences; the National Science Library, Chinese Academy of Sciences; the Computer Network Information Center, Chinese Academy of Sciences; Beijing Wenge Technology Co., Ltd.; and Zidong Taichu (Beijing) Technology Co., Ltd. It aims to provide various tool support for intelligent agent systems. Currently, DeepLiterature supports four types of tools: Search Engine, Code Runner, Link Parser, and Information expander.
๐ฅ News
- [2025/03] DeepLiterature officially released, supporting tools such as Search Engine, Code Runner, Link Parser, and Information expander.
๐ Table of Contents
- ๐ Introduction
- ๐ฅ News
- ๐ Table of Contents
- ๐ค Agent Framework
- ๐ Evaluation
- ๐ฌ Case Study
- โก Quickstart
- ๐ TODO
- ๐ Acknowledgement
- ๐ License
๐ค Agent Framework
The core of DeepLiterature lies in building a multi-agent interactive collaboration framework, which integrates various tools and achieves flexible expansion and efficient collaboration through modular design.
๐ Flowchart
Below is a schematic diagram of the main internal processes of the system, showing the information transfer and call flow between modules:
flowchart TD
A[Task๐] --> B[๐ค Planner]
B --> C[๐ง Optimizer]
C --> B
B --> D[๐ Searcher]
D --> G[๐ง Discriminator]
G --> H[URL Parser ๐]
H --> B
B --> E[๐ค Code Generator]
E --> F[๐ฅ๏ธ Code Runner]
F --> B
G --> B
F --> I[โ๏ธ Writer]
B --> I
D --> I
H --> I
I --> J[๐ฐ Publisher]
Module Description
- Planner: Responsible for receiving user input, parsing requests, and coordinating module calls.
- Optimizer: Uses external search interfaces to retrieve and return relevant information summaries.
- Code Generator: Generates code based on the code description distributed by the planner, supporting multi-language environments.
- Code Runner: Executes the code generated by the code generator in a secure sandbox, supporting multi-language environments.
- Searcher: Uses search engines to retrieve relevant web content.
- Discriminator: Analyzes search result summaries and determines whether to obtain the full text to improve answer accuracy.
- URL Parser: Parses link content and retrieves the full text of web pages.
- Writer: Writes output raw data based on user instructions and obtained materials.
- Publisher: Processes the raw data from the writer to form output that meets user instructions.
๐ ๏ธ Tool Library
This framework integrates the following tool libraries to achieve the above agents:
- ๐ Search Engine: Efficiently retrieves network information.
- ๐ฅ๏ธ Code Runner: Supports online code running and debugging.
- ๐ Link Parser: Parses, verifies, and extracts data from links.
- ๐ง Information Expansion: Based on search results, determines whether the summary information is relevant; when the summary information is insufficient to answer the question, automatically obtains the full text.
๐ Evaluation
We tested on the smolagents-benchmark, which aggregates subsets from multiple datasets: GAIA (32 items), MATH (50 items), SimpleQA (50 items). Below are the results of our DeepLiterature and smolagent(HF).
๐ฌ Case Study
The following examples demonstrate the application effect of DeepLiterature in actual conversations, providing both Chinese and English versions (the following GIF files are for illustration only, please replace them with corresponding examples in actual use).
Chinese Example
Query 1 (GPT-4o)๏ผๅๆ2024ๅนดไธญๅฝ็็ตๅๅ ฌๅธ๏ผๆช่ณๅฐQ3ๅญฃๅบฆ็็ๅฉๆ ๅต๏ผๅฏไปฅๆ็ งๅญฃๅบฆๅฏนๆฏ๏ผๅฏไปฅๆจชๅๅฏนๆฏไธๅ็ๅ ฌๅธใๅๆๅไธชๅ ฌๅธ็ๅธๅบไปฝ้ขๆ ๅตใ
Query 2 (GPT-4o)๏ผๅธฎๆๆพๅฐ2024ๅนดไธญๅฝ็ไบงๆปๅผtop10็ๅๅธ๏ผไปฅๆฑ็ถๅพๅ็ฐtop10็ๅๅธๅ็ไบงๆปๅผใๅฏนๆฏ่ฟไบๅๅธๅจ่ฟๅปไธๅนด็ไบบๅฃๅๅๆ ๅต๏ผไปฅๆ็บฟๅพ็ๆนๅผๅฑ็คบไบบๅฃๅๅใ
English Example
Query 1 (DeepSeek-R1)๏ผStatistics on the number of Nobel Prize winners by country in recent years, and analyze the age groups of Nobel Prize recipients.
Query 2 (DeepSeek-R1)๏ผIdentify the top 3 most profitable cloud providers globally. Use a bar chart to illustrate the profitability of these three companies. Locate the top 1 cloud provider and analyze its profit changes over the past three years.
โก Quickstart
The following content will help you quickly deploy and experience DeepLiterature.
๐๏ธ Code Structure
DeepLiterature adopts a modular design, with the main directory structure as follows:
./src/
โโโ agents # Agent modules, defining different role agents and corresponding prompt templates
โโโ config # Configuration files, configurable for search engine API, web parsing, and code execution modules
โโโ llms # LLM invocation modules
โโโ logo # Front-end display library
โโโ service # Platform and REST-ful API services
โโโ tools # Executor module, which implements code runner and search engine
โโโ utils # General utility library
โโโ workflow # Task workflow modules
โ๏ธ Configure API
- Configuration files are located in the
src/configs/directory, you can modify the default parameters as needed. - Each module provides a unified interface for configuration, see the documentation for each module.
๐ Select Prompt Template
-
Default prompt templates are provided in both Chinese and English.
-
Template files are stored in
src/agents/prompts/, users can customize them according to actual applications. The prompt file names correspond to the agent file names: for example,src/agents/orchestrator_agent.pycorresponds tosrc/agents/prompts/orchestrator_agent_prompts.py. -
Switch between Chinese and English prompt templates: through the
languageconfiguration inconfigs/config.yml.deepliterature: language: <LANG> # ["zh", "en"]
๐ Configure Search Engine
-
Modify
src/configs/config.ymlto configure the search API key, request parameters, etc. -
Supports parallel calls to multiple search engines to improve retrieval efficiency and accuracy, with SerpAPI used by default.
search-engine: search_engine: "serp" serp: api_url: "https://serpapi.com/search" api_key: <YOUR_API_KEY> gl: <GOOGLE_LOCATION> # such as "cn" hl: <GOOGLE_LANGUAGE> # such as "zh-cn" -
SerpAPI is an API service that provides search engine results, allowing developers to integrate search engine functionalities like Google and Bing into their applications. It supports various parameter settings and can perform customized searches based on region and language. With SerpAPI, users can easily obtain Google search results without directly interacting with Google Search or dealing with complex crawlers and CAPTCHA issues. Specifically:
- gl: This parameter defines the country for using Google Search. It is a two-letter country code (e.g., us for the United States, uk for the United Kingdom, fr for France). See Google country page for more details.
- hl: This parameter defines the language for using Google Search. It is a two-letter language code (e.g., en for English, es for Spanish, fr for French). See Google language page for more details.
๐ฅ๏ธ Configure URL Parser
- Modify
src/configs/config.ymlto configure the API key, request parameters, etc. Jina API is used by default.jina: api_url: "https://r.jina.ai/" api_key: <YOUR_API_KEY> - JinaAI is dedicated to building search systems for any type of data, including text, images, audio, and video. Through modular design and multiple layers of abstraction, users can effectively build systems by parts or link them into a stream for an end-to-end experience.
โ๏ธ Configure LLM
- Modify
src/configs/config.ymlto configure the remote LLM API key, request parameters, etc., and switch the LLM through thellm_modeloption.llm: max_context_length: <MAX_CONTEXT_LENGTH> llm_model: <LLM_MODEL_TYPE> # ["remote-llm", "remote-reasoning-llm"] remote-llm: api_url: <YOUR_API_URL> api_key: <YOUR_API_KEY> model_name: <YOUR_MODEL_NAME> tokenizer: tokenizer_class: <TOKENIZER_CLASS> tokenizer_name_or_path: <TOKENIZER_NAME_OR_PATH> remote-reasoning-llm: api_url: <YOUR_API_URL> api_key: <YOUR_API_KEY> model_name: <YOUR_MODEL_NAME> tokenizer: tokenizer_class: <TOKENIZER_CLASS> tokenizer_name_or_path: <TOKENIZER_NAME_OR_PATH>
๐ป Configure Code Runner
-
In
src/configs/config.yml, modifyapi_urlto your custom code runner API.code-runner: api_url: <YOUR_CODE_RUNNER_API>The code runner API is called via POST request and must conform to the following request and response formats.
- Request format: The request body is in JSON format and includes the following fields.
{ "id": <REQUEST_ID>, # Unique identifier for the request, type: string "code_text": <CODE>, # Code text, type: string "code_type": <PROGRAMMING_LANGUAGE>, # Programming language type, e.g., "python" } - Response format: The response body is in JSON format, with the code execution text output in the
textfield and image output in theimgfield. Example response:{ "id": <REQUEST_ID>, # Unique identifier for the request, type: string "code": 200, # Status code: 200 for success, 500 for failure, type: int "data": [ {"text": "Hello world!\n"}, {"img": "iVBORw0KGgoAAAANSUhEUgAAAjMAAAHFCAYAAAAHcXhbAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjYuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8o6BhiAAAACXBIWXMAAA9hAAAPYโฆโฆggg=="}, {"text": "Hello DeepLiterature"} ], "msg": "success", # Execution status message, type: string "time": 0.711 # Execution time in seconds, type: float }
- Request format: The request body is in JSON format and includes the following fields.
-
Ensure security: The code runner uses a sandbox mechanism to ensure user code runs in an isolated environment.
๐ Launch Platform
-
After cloning the repository, install dependencies:
git clone https://github.com/ScienceOne-AI/DeepLiterature.git cd DeepLiterature pip install -r requirements.txt -
Start the platform service:
streamlit run server.py --server.port=36667 -
Start the API service:
# Default port 36668 python api.py# Test sample curl --location 'http://127.0.0.1:36668/stream' \ --header 'Content-Type: application/json' \ --data '{ "query": "Which country ranked first in the medal tally of the 2025 Winter Asian Games, and display the number of each medal in a bar chart." }'# Output format: data: { "id": "978210f0-23fd-4f43-9d82-4837e4cb8e28", "created": 1741312834, "choices": [ { "index": 0, "delta": { "reasoning_content": "", "content": "" } } ], "stage": "" }
๐ฌ Start Chat
- Access the local service (such as http://localhost:36667), enter the chat page, and experience the intelligent Q&A service provided by DeepLiterature.
๐ TODO
- Launch the DeepLiterature experience platform.
- Provide a search engine focused on scientific literature.
- Provide a general code executor interface.
- Complete HLE benchmark testing and display detailed test results.
- Support more customization options for each module, allowing users to configure flexibly according to their needs.
๐ Acknowledgement
The front-end display part of this framework is implemented through the Streamlit lightweight application platform, thanks to their contribution.
๐ License
DeepLiterature is open-sourced under the Apache 2.0 License, welcome to use and contribute code under the license agreement.