Install

Parrot can be installed as a Python library from the github repository.

Environment Settings

• OS: Linux, Ubuntu 20.04
• GPU: cc >= 7.0 (Ours: NVIDIA A100, NVIDIA A6000)
• CUDA version: >= 12.1
• DL Framework: PyTorch >= 2.1.0 with CUDA 12.1.

pip install torch==2.1.0 --upgrade --index-url https://download.pytorch.org/whl/cu121

Clone the Project

git clone --recursive https://github.com/microsoft/ParrotServe.git

Configure the Environment

source .env

Install Dependencies

• Step 1: Install basic requirements.

pip install -r requirements.txt

• Step 2: Install necessary dependencies listed in 3rdparty folder.

cd 3rdparty/vllm
pip install -e .

• Step 3 (Optional): Install Optional dependencies.

(Optional) FastChat and Langchain are used only in our benchmark.

cd 3rdparty/FastChat
pip install -e ".[model_worker,webui]"

cd 3rdparty/langchain/libs/langchain
pip install -e .

! Important Notes:

Triton 2.0.0 has some bugs in Kernel memory issues. So we enforce the version to be 2.1.0 here. You will see some dependencies warnings, but it will not affect the common usages. (The similar error also happens in LightLLM kernels.)

pip install triton==2.1.0

Install Parrot

(In the root folder of Parrot)

pip install -e .

When installing is done, check your installation by

import parrot