README.md

A unified workflow orchestration and distributed KV cache sharing framework for multimodal inference.

About

As multimodal models rapidly evolve from text-only to image, audio, and video, inference systems face three core challenges: flexible workflow orchestration across heterogeneous components, low-latency data transfer between roles, and safe KV cache & weight sharing among multiple consumers. Omni-Flow addresses these through a three-layer abstraction:

• Control Flow — Exposes a Python DSL for composing heterogeneous computing units into a unified dataflow graph. It supports both static DAGs and dynamic conditional routing, and ships with built-in service discovery and a suite of load-balancing strategies, making it straightforward to wire up arbitrarily complex multimodal pipelines.
• Data Flow — Provides a distributed KV cache abstraction that transcends the conventional prefill/decode boundary. It unifies cache allocation across roles, delivers direct cross-role transmission over a three-tier paged storage hierarchy (GPU → CPU → SSD), and employs zero-copy channels to minimise end-to-end transfer latency.
• Compute Flow — Implements sophisticated multimodal prefix matching for KV reuse across multi-turn dialogues, and assumes full control of KV cache management and sampling logic through a unified SGLang interface. This lets diffusion models share the same LLM forward path and operate under identical parallel semantics without any bespoke adaptation.

📄 See the technical report: arxiv

Quick Start

1. Pull the Base Docker Image

Omni-Flow builds on top of the official SGLang image. Pull the recommended version before proceeding:

docker pull lmsysorg/sglang:v0.5.10.post1-cu130

Note: Use v0.5.10.post1 (without -cu130) if your NVIDIA driver is older than 570 (CUDA < 13.0).

Start a container (adjust volume mounts and GPU flags as needed):

docker run -it --gpus all \
  -v /path/to/omni-flow:/workspace/omni-flow \
  lmsysorg/sglang:v0.5.10.post1-cu130 bash

2. Install Dependencies

Inside the container (or directly on the host if not using Docker), run:

cd /workspace/omni-flow
git submodule update --init
bash install.sh

This will install the project in editable mode along with all required dependencies (including sglang-kernel).

3. Redis Setup (Optional)

Redis is used for distributed coordination. Choose one of the following options:

Option A — Local Redis (for development / single-node testing):

bash omni_flow/deploy/start_redis.sh

Option B — External Redis cluster (recommended for production):

If you already have a Redis cluster, set the connection info via environment variables before launching any service:

export REDIS_HOST=<your-redis-host>
export REDIS_PORT=<your-redis-port>
# export REDIS_PASSWORD=<your-redis-password>  # if auth is enabled

4. Launch a Service (DeepSeek-V2 example)

bash omni_flow/examples/deepseek_v2_llm_only/start.sh --foreground

Other launch options:

bash omni_flow/examples/deepseek_v2_llm_only/start.sh              # daemon mode
bash omni_flow/examples/deepseek_v2_llm_only/start.sh --foreground # foreground debug
bash omni_flow/examples/deepseek_v2_llm_only/start.sh --stop       # stop
bash omni_flow/examples/deepseek_v2_llm_only/start.sh --status     # check status

5. Send a Request

bash omni_flow/examples/deepseek_v2_llm_only/request.sh

6. (Optional) Try the WebUI

Omni-Flow ships a browser chat UI on top of the running backend:

bash omni_flow/deploy/webui/start.sh

Then open http://localhost:7860/, and in the chat page:

• Step 1: pick the target model in the top-right selector — wait for its status dot to turn from red to green (service is up and reachable) before sending anything; if the service is already up but the dot is still red, just refresh the page
• Step 2: choose a mode (Understanding / Generation / Automatic)
• Step 3: type your message and send

See deploy/webui/README.md for details (custom catalog root/port via OMNI_WEBUI_CATALOG_ROOTS / OMNI_WEBUI_PORT).

Supported Models

Model	Scenario	Example	Description
DeepSeek-V2	LLM-only streaming inference	deepseek_v2_llm_only/ · README	Single graph, single role, streaming token output
LongCat-Next	Full-modality conversation	longcat_next/ · README	LLM + Vision Encoder + Audio Encoder + Image Decoder + Audio Decoder (5 roles)
HunyuanImage-3	Image generation pipeline	hunyuan_image3_pipeline/e2e/ · README	AR + Diffusion, LLM Backbone DiT, t2i/ti2i + CFG

Deployment

Omni-Flow uses Ray for distributed deployment, with resource topology declared via resource.yaml:

• Elastic scaling: modify replicas to scale within a resource group
• Layer co-location: layer: 0 determines PG shape; layer ≥ 1 layers onto the same physical GPUs
• KV Barrier synchronization: ensures accurate HBM evaluation timing in co-located deployments to prevent OOM

See SCHEMA_V2.md and launch/README.md for detailed configuration.

Contact

For collaboration or hiring inquiries, please contact: xiaobin14@meituan.com.

Citation

@misc{omni-flow2026,
  title  = {Omni-Flow: A Unified Workflow Orchestration and Distributed KV Cache Sharing Framework for Multimodal Inference},
  author = {Bin Xiao and Jingfu Dong and Changran Wang and Yitian Chen and Xiaoyu Zhao and Yuqi Peng and Jianping Lin and Yuchen Xie},
  year   = {2026},
  eprint = {2606.31093},
  archivePrefix = {arXiv},
  primaryClass  = {cs.DC}
}