Awesome AI System

This repo is motivated by awesome tensor compilers.

Contents

• Paper-Code
  • Researcher
  • LLM Serving Framework
  • LLM Evaluation Platform
  • LLM Robustness and Debugging
  • LLM Inference System Side)
  • Compiler
  • Attention
  • RAG And ANNS
  • RLHF
  • Video
  • LLM Inference AI Side)
  • LLM MoE
  • LoRA
  • Framework
  • Parallellism Training
  • Training
  • Communication
  • Serving-Inference
  • MoE
  • GPU Cluster Management
  • Schedule and Resource Management
  • Optimization
  • GNN
  • Fine-Tune
  • Energy
  • Misc
• Contribute

Paper-Code

Researcher

Name	University	Homepage
Ion Stoica	UC Berkeley
Joseph E. Gonzalez	UC Berkeley
Matei Zaharia	UC Berkeley
Zhihao Jia	CMU
Tianqi Chen	CMU
Stephanie Wang	UW
Xingda Wei	SJTU
Zeyu Min	SJTU
Xin Jin	PKU
Harry Xu	UCLA
Anand Iyer	Georgia Tech
Ravi Netravali	Princeton
Christos Kozyrakis	Stanford
Christopher Ré	Stanford
Tri Dao	Princeton
Mosharaf Chowdhury	UMich
Shivaram Venkataraman	Wisc
Hao Zhang	UCSD
Yiying Zhang	UCSD
Ana Klimovic	ETH
Fan Lai	UIUC
Lianmin Zheng	UC Berkeley
Ying Sheng	Stanford
Zhuohan Li	UC Berkeley
Woosuk Kwon	UC Berkeley
Zihao Ye	University of Washington
Amey Agrawal	Georgia Tech

LLM Serving Framework

Title	Github
MLC LLM
TensorRT-LLM
xFasterTransformer
CTranslate2(low latency)
llama2.c

LLM Evaluation Platform

Title	Github	Website
FastChat

LLM Robustness and Debugging

Title	Paper	Github	Pub. & Date
WFGY 1.0: Self-healing LLM Systems Framework	PDF		Tech report, Oct 13 2025

LLM Inference (System Side)

Title	Paper	Github	WebSite	Pub. & Date
SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling			-	SIGMETRICS'26
{HydraServe}: Minimizing Cold Start Latency for Serverless {LLM} Serving in Public Clouds			-	NSDI'26
BulletServe:Boosting LLM Serving through Spatial-Temporal GPU Resource Sharing			-	ASPLOS'26
Aegaeon: Effective GPU Pooling for Concurrent LLM Serving on the Market			-	SOSP'25
DynaPipe: Dynamic Layer Redistribution for Efficient Serving of LLMs with Pipeline Parallelism			-	NeurIPS'25
DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compaction			-	SOSP'25
Pie: A Programmable Serving System for Emerging LLM Applications			-	SOSP'25
KTransformers: Unleashing the Full Potential of CPU/GPU Hybrid Inference for MoE Models			-	SOSP'25
XSched: Preemptive Scheduling for Diverse XPUs			-	OSDI 25
TokenWeave: Efficient Compute-Communication Overlap for Distributed LLM Inference			-	Arxiv 25
ServeGen: Workload Characterization and Generation of Large Language Model Serving in Production			-	Arxiv 25
Resource Multiplexing in Tuning and Serving Large Language Models			-	ATC'25
RetroInfer: A Vector-Storage Approach for Scalable Long-Context LLM Inference			-	Arxiv May 2025
SpecEE: Accelerating Large Language Model Inference with Speculative Early Exiting			-	ISCA'25
LIA: A Single-GPU LLM Inference Acceleration with Cooperative AMX-Enabled CPU-GPU Computation and CXL Offloading			-	ISCA'25
Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving			-	SIGMOD'25
Marconi: Prefix Caching for the Era of Hybrid LLMs			-	MLSys'25
SpInfer: Leveraging Low-Level Sparsity for Efficient Large Language Model Inference on GPUs			-	Eurosys'25 Best Paper
NeuStream: Bridging Deep Learning Serving and Stream Processing			-	Eurosys'25
Towards End-to-End Optimization of LLM-based Applications with Ayo			-	ASPLOS'25
NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference			-	MLSYS'25
CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion			-	Eurosys'25 Best Paper
Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flow			-	ASPLOS'25
GLINTHAWK: A Two-Tiered Architecture for High-Throughput LLM Inference			-	Arxiv'25,Jan
Queue Management for SLO-Oriented Large Language Model Serving			-	SOCC'24
NanoFlow: Towards Optimal Large Language Model Serving Throughput			-	OSDI'25
PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU			-	SOSP'24
LoongServe: Efficiently Serving Long-context Large Language Models with Elastic Sequence Parallelism			-	SOSP'24
Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference			-	MLSYS'24
PLLMCompass: Enabling Efficient Hardware Design for Large Language Model Inference			-	ISCA'24
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference			-	ISCA'24
Prompt Cache: Modular Attention Reuse for Low-Latency Inference			-	MLSYS'24
Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve			-	OSDI'24
DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving			-	OSDI'24
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving			-	July'24
Llumnix: Dynamic Scheduling for Large Language Model Serving			-	OSDI'24
Parrot: Efficient Serving of LLM-based Application with Semantic Variables			-	OSDI'24
CacheGen: Fast Context Loading for Language Model Applications via KV Cache Streaming			-	SIGCOMM'24
Efficiently Programming Large Language Models using SGLang			-	Jan, 2024
Efficient Memory Management for Large Language Model Serving with PagedAttention			-	SOSP'23
SpecInfer: Accelerating Generative Large Language Model Serving with Speculative Inference and Token Tree Verification			-	Dec,2023
Liger: Interleaving Intra- and Inter-Operator Parallelism for Distributed Large Model Inference	-		-	PPOPP'24
Efficiently Programming Large Language Models using SGLang			-	Nurips'24
Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsity			-	VLDB'24

Compiler

Title	Paper	Github	WebSite	Pub. & Date
Mercury: Unlocking Multi-GPU Operator Optimization for LLMs via Remote Memory Scheduling			-	SOSP'25
Mirage: A Multi-Level Superoptimizer for Tensor Programs			-	OSDI'25

Attention

Title	Paper	Github	WebSite	Pub. & Date
UltraAttn: Efficiently Parallelizing Attention through Hierarchical Context-Tiling			-	SC'25
TASP: Topology-aware Sequence Parallelism			-	Arxiv'25
Ring Attn			-

RAG And ANNS

Title	Paper	Github	WebSite	Pub. & Date
HedraRAG: Co-Optimizing Generation and Retrieval for Heterogeneous RAG Workflows			-	SOSP'25
LEANN: A Low-Storage Vector Index			-	Arxiv 25
OdinANN: Direct Insert for Consistently Stable Performance in Billion-Scale Graph-Based Vector Search			-	FAST'26
Achieving Low-Latency Graph-Based Vector Search via Aligning Best-First Search Algorithm with SSD			-	OSDI'25
Quake: Adaptive Indexing for Vector Search			-	OSDI'25
Hermes: Algorithm-System Co-design for Efficient Retrieval Augmented Generation At-Scale			-	ISCA'25
PathWeaver: A High-Throughput Multi-GPU System for Graph-Based Approximate Nearest Neighbor Search			-	ATC'25
In-Storage Acceleration of Retrieval Augmented Generation as a Service: Artifact Evaluation README			-	ISCA'25
RAGO: Systematic Performance Optimization for Retrieval-Augmented Generation Serving			-	ISCA'25

RLHF

Title	Paper	Github	WebSite	Pub. & Date
Optimizing RLHF Training for Large Language Models with Stage Fusion			-	NSDI'25
HybridFlow: A Flexible and Efficient RLHF Framework			-	Eurosys'25
ReaLHF: Optimized RLHF Training for Large Language Models through Parameter Reallocation			-	June. 2024
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework			-	May. 2024

Video

Title	Paper	Github	WebSite	Pub. & Date
Katz: Efficient Workflow Serving for Diffusion Models with Many Adapters			-	ATC'25
PPipe: Efficient Video Analytics Serving on Heterogeneous GPU Clusters via Pool-Based Pipeline Parallelism			-	Nov. 2024
xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism			-	Nov. 2024
FastVideo			-	Dec. 2024

LLM Inference(AI Side)

Title	Paper	Github	WebSite	Pub. & Date
InferCept: Efficient Intercept Support for Augmented Large Language Model Inference			-	ICML'24
Online Speculative Decoding			-	ICML'24
MuxServe: Flexible Spatial-Temporal Multiplexing for LLM Serving			-	ICML'24
BitDelta: Your Fine-Tune May Only Be Worth One Bit			-	Feb,2024
Medusa: Simple Framework for Accelerating LLM Generation with Multiple Decoding Heads			-	Jan,2024
LLMCompiler: An LLM Compiler for Parallel Function Calling			-	Dec,2023
Mamba: Linear-Time Sequence Modeling with Selective State Spaces			-	Dec,2023
Teaching LLMs memory management for unbounded context			-	Oct,2023
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding			-	Feb,2024
EAGLE: Lossless Acceleration of LLM Decoding by Feature Extrapolation			-	Jan,2024

LLM MoE

Title	Paper	Github	WebSite	Pub. & Date
Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference			-	ISCA'24
SIDA-MOE: SPARSITY-INSPIRED DATA-AWARE SERVING FOR EFFICIENT AND SCALABLE LARGE MIXTURE-OF-EXPERTS MODELS			-	MLSYS'24
ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling			-	Eurosys'24

LoRA

Title	Paper	Github	WebSite	Pub. & Date
oRAFusion: Efficient LoRA Fine-Tuning for LLMs			-	Eurosys'26
dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving			-	OSDI'24
S-LoRA: Serving Thousands of Concurrent LoRA Adapters			-	Nov,2023
Punica: Serving multiple LoRA finetuned LLM as one			-	Oct,2023

Framework

• code Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22

  paper Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22

• code Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22 OSDI'22

  paper Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22

• code Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM SC21

  paper Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM SC21

• code A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters OSDI'20

  paper A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters OSDI'20

• code Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training ICPP'23

  paper Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training ICPP'23

• code HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework VLDB'22

  paper HET: Scaling out Huge Embedding Model Training via Cache-enabled Distributed Framework VLDB'22

Parallellism Training

• code zero-bubble-pipeline-parallelism

  paper NEAR ZERO BUBBLE PIPELINE PARALLELISM ICLR'24

• code DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines Eurosys'24

  paper DynaPipe: Optimizing Multi-task Training through Dynamic Pipelines Eurosys'24

• Aceso: Efficient Parallel DNN Training through Iterative Bottleneck Alleviation Eurosys'24

• code HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis Eurosys'24

  paper HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis Eurosys'24

• code Calculon: A Methodology and Tool for High-Level Co-Design of Systems and Large Language Models SC'23

  paper Calculon: A Methodology and Tool for High-Level Co-Design of Systems and Large Language Models SC'23

• code PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices MLSYS'23

  paper PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices MLSYS'23

• code Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs NSDI'23

  paper Bamboo: Making Preemptible Instances Resilient for Affordable Training of Large DNNs NSDI'23

• code MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23

  paper MPress: Democratizing Billion-Scale Model Training on Multi-GPU Servers via Memory-Saving Inter-Operator Parallelism HPCA'23

• code Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression ASPLOS'23

  paper Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compression ASPLOS'23

• code Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22

  paper Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning OSDI'22

• code AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS '22

  paper AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness NeurIPS '22

• code Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22

  paper Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization OSDI'22

• code NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism ASPLOS'22

  paper NASPipe: High Performance and Reproducible Pipeline Parallel Supernet Training via Causal Synchronous Parallelism

• code Varuna: Scalable, Low-cost Training of Massive Deep Learning Models Eurosys'22

  paper Varuna: Scalable, Low-cost Training of Massive Deep Learning Models

• code Chimera: efficiently training large-scale neural networks with bidirectional pipelines SC'21

  paper Chimera: efficiently training large-scale neural networks with bidirectional pipelines SC'21

• code Piper: Multidimensional Planner for DNN Parallelization NeurIPS'21

  paper Piper: Multidimensional Planner for DNN Parallelization NeurIPS'21

• code PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models ICML'21

  paper PipeTransformer: Automated Elastic Pipelining for Distributed Training of Large-scale Models ICML'21

• code DAPPLE: An Efficient Pipelined Data Parallel Approach for Large Models Training PPOPP'21

  paper DAPPLE: An Efficient Pipelined Data Parallel Approach for Large Models Training PPOPP'21

• code TeraPipe:Large-Scale Language Modeling with Pipeline Parallelism ICML'21

  paper TeraPipe:Large-Scale Language Modeling with Pipeline Parallelism ICML'21

• code PipeDream: Pipeline Parallelism for DNN Training SOSP'19

  paper PipeDream: Pipeline Parallelism for DNN Training SOSP'19

• code SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

  paper SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient

• code Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

  paper Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models

• awesome distributed deep learning

• awsome parallelism

Training

• code ModelKeeper: Accelerating DNN Training via Automated Training Warmup NSDI'23

  paper ModelKeeper: Accelerating DNN Training via Automated Training Warmup NSDI'23

• code STRONGHOLD: Fast and Affordable Billion-scale Deep Learning Model Training SC'22

  paper STRONGHOLD: Fast and Affordable Billion-scale Deep Learning Model Training SC'22

• code Whale: Efficient Giant Model Training over Heterogeneous {GPUs}ATC'22

  paper Whale: Efficient Giant Model Training over Heterogeneous {GPUs}ATC'22

• code GeePS: Scalable Deep Learning on Distributed GPUs with a GPU-Specialized Parameter Server Eurosys'16

  paper GeePS: Scalable Deep Learning on Distributed GPUs with a GPU-Specialized Parameter Server Eurosys'16

Communication

• code ARK: GPU-driven Code Execution for Distributed Deep Learning NSDI'23

  paper ARK: GPU-driven Code Execution for Distributed Deep Learning NSDI'23

• code TopoOpt: Optimizing the Network Topology for Distributed DNN Training NSDI'23

  paper TopoOpt: Optimizing the Network Topology for Distributed DNN Training NSDI'23

• code Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads ASPLOS'22

  paper Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads ASPLOS'22

• code Efficient Sparse Collective Communication and its application to Accelerate Distributed Deep Learning SIGCOMM'21

  paper Efficient Sparse Collective Communication and its application to Accelerate Distributed Deep Learning SIGCOMM'21

Serving-Inference

• code Paella: Low-latency Model Serving with Virtualized GPU Scheduling SOSP'23

  paper Paella: Low-latency Model Serving with Virtualized GPU Scheduling SOSP'23

• code AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving OSDI'23

  paper AlpaServe: Statistical Multiplexing with Model Parallelism for Deep Learning Serving OSDI'23

• code Optimizing Dynamic Neural Networks with Brainstorm OSDI'23

  paper Optimizing Dynamic Neural Networks with Brainstorm OSDI'23

• code Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access Eurosys'23

  paper Fast and Efficient Model Serving Using Multi-GPUs with Direct-Host-Access Eurosys'23

• code Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs.ASPLOS'23

  paper Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs.ASPLOS'23

• code MPCFormer: fast, performant, and private transformer inference with MPC ICLR'23

  paper MPCFormer: fast, performant, and private transformer inference with MPC ICLR'23

• code High-throughput Generative Inference of Large Language Modelwith a Single GPU ICML'23

  paper High-throughput Generative Inference of Large Language Modelwith a Single GPU ICML'23

• code VELTAIR: Towards High-Performance Multi-Tenant Deep Learning Serving via Adaptive Compilation and Scheduling ASPLOS'22

  paper VELTAIR: Towards High-Performance Multi-Tenant Deep Learning Serving via Adaptive Compilation and Scheduling ASPLOS'22

• code DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs ATC'22

  paper DVABatch: Diversity-aware Multi-Entry Multi-Exit Batching for Efficient Processing of DNN Services on GPUs ATC'22

• code Cocktail: A Multidimensional Optimization for Model Serving in Cloud NSDI'22

  paper Cocktail: A Multidimensional Optimization for Model Serving in Cloud NSDI'22

• code Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing ATC'22

  paper Serving Heterogeneous Machine Learning Models on Multi-GPU Servers with Spatio-Temporal Sharing ATC'22

• code RIBBON: cost-effective and qos-aware deep learning model inference using a diverse pool of cloud computing instances SC'21

  paper RIBBON: cost-effective and qos-aware deep learning model inference using a diverse pool of cloud computing instances SC'21

• code INFaaS: Automated Model-less Inference Serving ATC'21

  paper INFaaS: Automated Model-less Inference Serving ATC'21

• code Enable Simultaneous DNN Services Based on Deterministic Operator Overlap and Precise Latency Prediction SC'21

  paper Enable Simultaneous DNN Services Based on Deterministic Operator Overlap and Precise Latency Prediction SC'21

• code Serving DNNs like Clockwork: Performance Predictability from the Bottom Up OSDI'20

  paper Serving DNNs like Clockwork: Performance Predictability from the Bottom Up OSDI'20

• code Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving ATC'19

  paper Exploiting Cloud Services for Cost-Effective, SLO-Aware Machine Learning Inference Serving ATC'19

• code Nexus: a GPU cluster engine for accelerating DNN-based video analysis SOSP'19

  paper Nexus: a GPU cluster engine for accelerating DNN-based video analysis SOSP'19

• code Clipper:A low-latency prediction-serving system NSDI'17

  paper Clipper:A low-latency prediction-serving system NSDI'17

MoE

• code SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Static and Dynamic Parallelization ATC'23

  paper SmartMoE: Efficiently Training Sparsely-Activated Models through Combining Static and Dynamic Parallelization ATC'23

• code MegaBlocks: Efficient Sparse Training with Mixture-of-Experts MLSYS'23

  paper MegaBlocks: Efficient Sparse Training with Mixture-of-Experts MLSYS'23

• code Tutel: Adaptive Mixture-of-Experts at Scale MLSYS'23

  paper Tutel: Adaptive Mixture-of-Experts at Scale MLSYS'23

• code FastMoE: A Fast Mixture-of-Expert Training System PPOPP'22

  paper FastMoE: A Fast Mixture-of-Expert Training System PPOPP'22

• code AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers ICLR'23

  paper AutoMoE: Neural Architecture Search for Efficient Sparsely Activated Transformers ICLR'23

• awesome MoE

• MoE Paper

GPU Cluster Management

• code Lucid: A Non-Intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs ASPLOS'23

  paper Lucid: A Non-Intrusive, Scalable and Interpretable Scheduler for Deep Learning Training Jobs ASPLOS'23

• code Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning NSDI'23

  paper Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learning NSDI'23

• code Synergy : Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters OSDI'22

  paper Synergy : Looking Beyond GPUs for DNN Scheduling on Multi-Tenant Clusters OSDI'22

• code Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning OSDI'21

  paper Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning OSDI'21

• code Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads OSDI'20

  paper Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloads OSDI'20

• code Tiresias -- A GPU Cluster Manager for Distributed Deep Learning Training without complete job information NSDI'19

  paper Tiresias -- A GPU Cluster Manager for Distributed Deep Learning Training without complete job information NSDI'19

• code Chronus: A Novel Deadline-aware Scheduler for Deep Learning Training Jobs SOCC'21

  paper Chronus: A Novel Deadline-aware Scheduler for Deep Learning Training Jobs SOCC'21

• awesome DL scheduler

Schedule and Resource Management

• code An interference-aware scheduler for fine-grained GPU sharing Resources Eurosys'24

  paper An interference-aware scheduler for fine-grained GPU sharing Resources Eurosys'24

• code ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning ASPLOS'23

  paper ElasticFlow: An Elastic Serverless Training Platform for Distributed Deep Learning ASPLOS'23

• code Multi-Resource Interleaving for Deep Learning Training SIGCOMM'22

  paper Multi-Resource Interleaving for Deep Learning Training SIGCOMM'22

• code Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training ASPLOS'24

  paper Slapo: A Schedule Language for Progressive Optimization of Large Deep Learning Model Training ASPLOS'24

• code Out-of-order backprop: an effective scheduling technique for deep learning Eurosys'22

  paper Out-of-order backprop: an effective scheduling technique for deep learning Eurosys'22

• code KungFu: Making Training in Distributed Machine Learning Adaptive OSDI'20

  paper KungFu: Making Training in Distributed Machine Learning Adaptive OSDI'20

• code PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications OSDI'20

  paper PipeSwitch: Fast Pipelined Context Switching for Deep Learning Applications OSDI'20

Optimization

• code GLake: optimizing GPU memory management and IO transmission ASPLOS'24

• code Spada: Accelerating Sparse Matrix Multiplication with Adaptive Dataflow ASPLOS'23

  paper Spada: Accelerating Sparse Matrix Multiplication with Adaptive Dataflow ASPLOS'23

• code MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters SOCC'22

  paper MISO: Exploiting Multi-Instance GPU Capability on Multi-Tenant GPU Clusters SOCC'22

• code Accpar: Tensor partitioning for heterogeneous deep learning accelerators HPCA'20

  paper Accpar: Tensor partitioning for heterogeneous deep learning accelerators HPCA'20

• code Hidet: Task Mapping Programming Paradigm for Deep Learning Tensor Programs ASPLOS'23

  paper Hidet: Task Mapping Programming Paradigm for Deep Learning Tensor Programs ASPLOS'23

• code iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud TPDS'22

  paper iGniter: Interference-Aware GPU Resource Provisioning for Predictable DNN Inference in the Cloud TPDS'22

• code CheckFreq: Frequent, Fine-Grained DNN Checkpointing FAST'22

  paper CheckFreq: Frequent, Fine-Grained DNN Checkpointing FAST'22

• code Efficient Quantized Sparse Matrix Operations on Tensor Cores SC'22

  paper Efficient Quantized Sparse Matrix Operations on Tensor Cores SC'22

• code Harmony: Overcoming the hurdles of GPU memory capacity to train massive DNN models on commodity servers VLDB'22

  paper Harmony: Overcoming the hurdles of GPU memory capacity to train massive DNN models on commodity servers VLDB'22

• code PetS: A Unified Framework for Parameter-Efficient Transformers Serving ATC'22

  paper PetS: A Unified Framework for Parameter-Efficient Transformers Serving ATC'22

• code PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections OSDI'21

  paper PET: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections OSDI'21

• code APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Core SC'21

  paper APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Core SC'21

• code iGUARD: In-GPU Advanced Race Detection SOSP'21

  paper iGUARD: In-GPU Advanced Race Detection SOSP'21

• code Fluid: Resource-Aware Hyperparameter Tuning Engine MLSYS'21

  paper Fluid: Resource-Aware Hyperparameter Tuning Engine MLSYS'21

• code Baechi: Fast Device Placement on Machine Learning Graphs SOCC'20

  paper Baechi: Fast Device Placement on Machine Learning Graphs SOCC'20

• code Dynamic Parameter Allocation in Parameter Servers VLDB'20

  paper Dynamic Parameter Allocation in Parameter Servers VLDB'20

• code Data Movement Is All You Need: A Case Study on Optimizing Transformers paper Data Movement Is All You Need: A Case Study on Optimizing Transformers

GNN

• code gSampler: Efficient GPU-Based Graph Sampling for Graph Learning SOSP'23

  paper gSampler: Efficient GPU-Based Graph Sampling for Graph Learning SOSP'23

• code Legion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale GNN Training ATC'23

  paper Legion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale GNN Training ATC'23

• code TC-GNN: Accelerating Sparse Graph Neural Network Computation Via Dense Tensor Core on GPUs ATC'23

  paper TC-GNN: Accelerating Sparse Graph Neural Network Computation Via Dense Tensor Core on GPUs ATC'23

• code Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms OSDI'23

  paper Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platforms OSDI'23

• code CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs SC'22

  paper CoGNN: Efficient Scheduling for Concurrent GNN Training on GPUs SC'22

• code GNNAdvisor: An Efficient Runtime System for GNN Acceleration on GPUs OSDI'21

  paper GNNAdvisor: An Efficient Runtime System for GNN Acceleration on GPUs OSDI'21

• code Marius: Learning Massive Graph Embeddings on a Single Machine OSDI'21

  paper Marius: Learning Massive Graph Embeddings on a Single Machine OSDI'21

• code Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads OSDI'21

  paper Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads OSDI'21

• code BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling MLSYS'22

  paper BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling MLSYS'22

• code Accelerating Large Scale Real-Time GNN Inference Using Channel Pruning VLDB'21

  paper Accelerating Large Scale Real-Time GNN Inference Using Channel Pruning VLDB'21

• code Reducing Communication in Graph Neural Network Training SC'20

  paper Reducing Communication in Graph Neural Network Training SC'20

• awesome GNN

Fine-Tune

• code Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism ATC'21

  paper Fine-tuning giant neural networks on commodity hardware with automatic pipeline model parallelism ATC'21

Energy

• code Zeus: Understanding and Optimizing {GPU} Energy Consumption of {DNN} Training NSDI'23

  paper Zeus: Understanding and Optimizing {GPU} Energy Consumption of {DNN} Training NSDI'23

• code EnvPipe: Performance-preserving DNN Training Framework for Saving Energy ATC'23

  paper EnvPipe: Performance-preserving DNN Training Framework for Saving Energy ATC'23

Misc

• code Characterizing Variability in Large-Scale, Accelerator-Rich Systems SC'22

  paper Characterizing Variability in Large-Scale, Accelerator-Rich Systems SC'22

• code Prediction of the Resource Consumption of Distributed Deep Learning Systems SIGMETRICS'22

  paper Prediction of the Resource Consumption of Distributed Deep Learning Systems SIGMETRICS'22

• code AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications HPCA'22

  paper AI-Enabling Workloads on Large-Scale GPU-Accelerated System: Characterization, Opportunities, and Implications HPCA'22

Contribute

We encourage all contributions to this repository. Open an issue or send a pull request.