LLM App Stack

February 15, 2024 · View on GitHub

aka Emerging Architectures for LLM Applications

2657 Emerging LLM App Stack R2 Clean

This is a list of available tools, projects, and vendors at each layer of the LLM app stack.

Our original article included only the most popular options, based on user interviews. This repo is meant to be more comprehensive, covering all available options in each category. We probably still missed some important projects, so please open a PR if you see anything missing.

We also included Perplexity and Cursor.sh prompts to make searching and markdown table formatting easier.

Data Pipelines
Embedding Models
Vector Databases
Playgrounds
Orchestrators
APIs / Plugins
LLM Caches
Logging / Monitoring / Eval
Validators
LLM APIs (proprietary)
LLM APIs (open source)
App Hosting Platforms
Cloud Providers
Opinionated Clouds

Project List

Data Pipelines

Name (site)	Description	Github	Pip Installs
Databricks	A unified data platform for building, deploying, and maintaining enterprise data solutions, including products (like MosaicML and MLflow) purpose-built for AI	<a href=https://github.com/apache/spark>	<a href=https://pypi.org/project/pyspark>
Airflow	A data pipeline framework to programmatically author, schedule, and monitor data pipelines and workflows, including for LLMs	<a href=https://github.com/apache/airflow>	<a href=https://pypi.org/project/apache-airflow>
Unstructured.io	Open-source components for pre-processing documents such as PDFs, HTML, and Word documents for usage with LLM apps	<a href=https://github.com/Unstructured-IO/unstructured>	<a href=https://pypi.org/project/unstructured>
Fivetran	A platform that extracts, loads, and transforms data from various sources for analytics, AI, and operations	N/A	<a href=https://pypi.org/project/fivetran>
Airbyte	An open-source data integration engine that helps consolidate data in data warehouses, lakes, and databases	<a href=https://github.com/airbytehq/airbyte>	<a href=https://pypi.org/project/airbyte-cdk>
Anyscale	An AI compute platform that allows developers to scale data ingest, preprocessing, embedding, and inference computations using Ray	<a href=https://github.com/ray-project/ray>	<a href=https://pypi.org/project/ray>
Alluxio	An open-source data platform at the intersection of compute and storage, bringing data closer to compute, to accelerate model training and serving, boost GPU utilization, and reduce costs for AI workloads	<a href=https://github.com/Alluxio/alluxio>	<a href=https://pypi.org/project/alluxio-python-library/>

Name (site)	Description	Github	Pip Installs
OpenAI Ada Embedding 2	OpenAI's most popular embedding model for capturing semantic relationships in text	n/a	<a href=https://pypi.org/project/openai>
Cohere AI	An independent commerical provider of LLMs, with particular focus on embeddings for semantic search, topic clustering, and vertical applications	<a href=https://github.com/cohere-ai/notebooks>	<a href=https://pypi.org/project/cohere>
Sentence Transformers	An open-source Python framework for sentence, text, and image embeddings	<a href=https://github.com/UKPLab/sentence-transformers>	<a href=https://pypi.org/project/sentence-transformers>

Name (site)	Description	Github	Pip Installs
Pinecone	A managed, cloud-native vector database with a simple API for high-performance AI applications	n/a	<a href=https://pypi.org/project/pinecone-client>
Weaviate	An open-source vector database that stores both objects and vectors	<a href=https://github.com/weaviate/weaviate>	<a href=https://pypi.org/project/weaviate-client>
ChromaDB	An AI-native, open-source embedding database platform for developers	<a href=https://github.com/chroma-core/chroma>	<a href=https://pypi.org/project/chromadb>
Pgvector	An open-source vector similarity search for Postgres, allowing for exact and approximate nearest-neighbor search	<a href=https://github.com/pgvector/pgvector>	<a href=https://pypi.org/project/pgvector>
Zilliz (Milvus)	An open-source vector database, built for developing and maintaining AI applications	<a href=https://github.com/milvus-io/milvus>	<a href=https://pypi.org/project/pymilvus>
Qdrant	A vector database and vector similarity search engine	<a href=https://github.com/qdrant/qdrant>	<a href=https://pypi.org/project/qdrant-client>
Metal io	A managed service for developers to build applications with ML embeddings	N/A	<a href=https://pypi.org/project/metal-python>
LanceDB	A serverless vector database for AI applications	<a href=https://github.com/lancedb/lancedb>	<a href=https://pypi.org/project/lancedb>

Name (site)	Description	Github	Pip Installs
OpenAI Playground	A web-based platform for experimenting with various machine-learning models developed by OpenAI	N/A	N/A
nat.dev	A platform that allows users to test prompts with multiple language models and compare their performance	<a href=https://github.com/nat/openplayground>	<a href=https://pypi.org/project/openplayground>
Humanloop	A platform that helps developers build applications on top of LLMs	<a href=https://github.com/humanloop/humanloop-tutorial-python>	<a href=https://pypi.org/project/humanloop>
Parea AI	Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground.	<a href=https://github.com/parea-ai>	<a href=https://pypi.org/project/parea-ai/>

Name (site)	Description	Github	Pip Installs
Langchain	An open-source library that gives developers the tools to build applications powered by LLMs	<a href=https://github.com/langchain-ai/langchain>	<a href=https://pypi.org/project/langchain>
LlamaIndex	A data framework for LLM applications to ingest, structure, and access private or domain-specific data	<a href=https://github.com/jerryjliu/llama_index>	<a href=https://pypi.org/project/llama-index>
Autogen	A framework for automating and streamlining LLM workflows using customizable, conversable agents for complex AI applications	<a href=https://github.com/microsoft/autogen>	<a href=https://pypi.org/project/pyautogen>
Microsoft Semantic Kernel	A lightweight open-source orchestration SDK	<a href=https://github.com/microsoft/semantic-kernel>	<a href=https://pypi.org/project/semantic-kernel>
Haystack	LLM orchestration framework to build customizable, production-ready LLM applications	<a href=https://github.com/deepset-ai/haystack>	<a href=https://pypi.org/project/farm-haystack/>
Vercel AI SDK	An open-source library for developers to build streaming UIs in JavaScript and TypeScript	<a href=https://github.com/vercel/ai>	<a href=https://pypi.org/project/vercel-ai-sdk>(node/npm)
Vectara AI	A search and discovery platform for AI conversations utilizing your own data	<a href=https://github.com/vectara/vectara-ingest>	N/A
ChatGPT	An AI chatbot that uses natural language processing to create humanlike conversational dialogue	N/A	N/A

Name (site)	Description	Github	Pip Installs
Serp API	A real-time API to access Google search results, as well as handling proxies, solving captchas, and parsing structured data	<a href=https://github.com/serpapi/google-search-results-python>	<a href=https://pypi.org/project/google-search-results>
Wolfram Alpha API	A web-based API providing computational and presentation capabilities for integration into various applications	N/A	<a href=https://pypi.org/project/wolframalpha>
Zapier API AI Plugin	A plugin that allows you to connect 5,000+ apps and interact with them directly inside ChatGPT	N/A	N/A

Name (site)	Description	Github	Pip Installs
Redis	An in-memory data structure store used as a database, cache, message broker, and streaming engine	<a href=https://github.com/redis/redis>	<a href=https://pypi.org/project/redis/>
SQLite	A self-contained, serverless, zero-configuration, transactional SQL database engine	<a href=https://github.com/sqlite/sqlite>	<a href=https://pypi.org/project/pysqlite3/>
GPTCache	An open-source tool for improving the efficiency and speed of GPT-based applications by implementing a cache to store the responses	<a href=https://github.com/zilliztech/GPTCache>	N/A

Name (site)	Description	Github	Pip Installs
Braintrust Data	An AI product stack featuring evaluations, prompt playgrounds, continuous integration, dataset management, and access to various AI models through a single API	<a href=https://github.com/braintrustdata/braintrust-proxy>	<a href=https://pypi.org/project/braintrust>
Arize AI	An observability platform for both LLMs and supervised ML	<a href=https://github.com/Arize-ai>	<a href=https://pypi.org/project/arize>
Weights & Biases	An MLOps platform for streamlining ML workflows	<a href=https://github.com/wandb/wandb>	<a href=https://pypi.org/project/wandb>
MLflow	A platform to streamline ML development	<a href=https://github.com/mlflow/mlflow>	<a href=https://pypi.org/project/mlflow>
PromptLayer	A platform for tracking, managing, and sharing LLM prompt engineering	<a href=https://github.com/MagnivOrg/prompt-layer-library>	<a href=https://pypi.org/project/promptlayer>
Helicone	An open-source observability platform for LLMs	<a href=https://github.com/Helicone/helicone>	<a href=https://pypi.org/project/helicone>
Quotient AI	Quotient AI is a platform for evaluating AI products on real-world use-cases, during research, development, and in production	N/A	N/A
Portkey AI	A platform to develop, launch, maintain, and iterate generative AI apps and features	N/A	N/A
Freeplay AI	A platform to prototype, test, and optimize LLM features for customers	N/A	N/A
Gentrace	An API and SDKs for evaluating and observing generative data, with features like AI, heuristic, and human grading evaluations, as well as production data observation	N/A	<a href=https://pypi.org/project/gentrace-py>
Patronus AI	An automated evaluation and benchmarking platform for LLMs, providing tools for testing, scoring, and evaluating LLMs in real-world scenarios	N/A	N/A
Autoblocks AI	A collaborative cloud-based workspace designed for rapid iteration on GenAI products, offering features like prompt management, observability, continuous evaluations, fine-tuning, prototyping, debugging, and scalable data ingestion & search, all in a provider-agnostic environment	N/A	<a href=https://pypi.org/project/autoblocksai>
Context AI	Tools for pre-launch LLM evaluations and post-launch analytics, with features such as testing, performance monitoring, user conversation analysis, and support for various models and libraries	N/A	<a href=https://pypi.org/project/context-python>
E2b dev	Services to deploy, test, and monitor AI agents, including a sandbox with a secure, long-running cloud environment for various LLMs with features like internet access	<a href=https://github.com/e2b-dev/e2b>	<a href=https://pypi.org/project/e2b>
Agentops	Toolkit for evaluating and developing AI agents, providing tools for agent development, monitoring capabilities, and replay analytics	<a href=https://github.com/AgentOps-AI/agentops>	<a href=https://pypi.org/project/agentops>
Zenoml	AI evaluation platform that enables data visualization, model performance analysis, and the creation of interactive reports for various data types	<a href=https://github.com/zeno-ml/zeno-build>	<a href=https://pypi.org/project/zeno-client>
Baserun	Tools for model configuration, prompt playground, monitoring, and prototype workflow, as well as features for full visibility into LLM workflows and end-to-end testing	<a href=https://github.com/baserun-ai/baserun-py>	<a href=https://pypi.org/project/baserun>
WhyLabs	AI Observability platform for ML and GenAI including LLM monitoring, guardrails and security	<a href=https://github.com/whylabs>	<a href=https://pypi.org/project/whylabs-client>
Log10	AI-powered LLMOps platform that automatically optimizes prompts and models with built-in logging, debugging, metrics, feedback, evaluations and fine-tuning	<a href=https://github.com/log10-io/log10>	<a href=https://pypi.org/project/log10-io> <a href=https://pypi.org/project/llmeval>
promptfoo	Open-source LLM eval framework with support for model/prompt/RAG eval, dataset generation, local models, and self-hosting.	<a href=https://github.com/promptfoo/promptfoo>	<a href=https://www.npmjs.com/package/promptfoo> (node/npm)
Parea AI	Platform and SDK for AI Engineers providing tools for LLM evaluation, observability, and a version-controlled enhanced prompt playground.	<a href=https://github.com/parea-ai>	<a href=https://pypi.org/project/parea-ai/>
Galileo	Galileo is a platform for evaluation, fine-tuning and real-time observability, powered by high-accuracy hallucination guardrails.	N/A	N/A

Name (site)	Description	Github	Pip Installs
Guardrails AI	An open-source Python package for specifying structure and type, validating, and correcting the outputs of LLMs	<a href=https://github.com/ShreyaR/guardrails>	<a href=https://pypi.org/project/guardrails-ai>
Rebuff	An open-source framework designed to detect and protect against prompt injection attacks in LLM apps	<a href=https://github.com/woop/rebuff>	<a href=https://pypi.org/project/rebuff>
Microsoft Guidance	A guidance language for controlling LLMs, providing a syntax for architecting LLM workflows	<a href=https://github.com/microsoft/guidance>	<a href=https://pypi.org/project/guidance>
LMQL	An open-source programming language and platform for language model interaction	<a href=https://github.com/eth-sri/lmql>	<a href=https://pypi.org/project/lmql>
Outlines	A tool for helping developers guide text generation to build robust interfaces with external systems and guarantee that outputs match a regex or JSON schema	<a href=https://github.com/outlines-dev/outlines>	<a href=https://pypi.org/project/outlines>
LLM Guard	An open-source, comprehensive tool designed to fortify the security of Large Language Models (LLMs).	<a href=https://github.com/laiyer-ai/llm-guard>	<a href=https://pypi.org/project/llm-guard>

Name (site)	Description	Github	Pip Installs
OpenAI	A company providing many leading LLMs, including the GPT-3.5 and GPT-4 families	N/A	<a href=https://pypi.org/project/openai>
Anthropic	The developer of Claude, an AI assistant based on Anthropic’s research	N/A	<a href=https://pypi.org/project/anthropic>
Cohere AI	An LLM vendor with particular focus on embeddings for semantic search, topic clustering, and vertical applications	<a href=https://github.com/cohere-ai/notebooks>	<a href=https://pypi.org/project/cohere>
LLM	A CLI utility and Python library for interacting with Large Language Models, both via remote APIs and models that can be installed and run on your own machine.	<a href=https://github.com/simonw/llm>	<a href=https://pypi.org/project/llm/>

Name (site)	Description	Github	Pip Installs
Hugging Face	A hub for open-source AI models and inference endpoints, including leading base LLMs and LoRAs/fine-tunes	<a href=https://github.com/huggingface/transformers>	<a href=https://pypi.org/project/transformers>
Replicate	An AI hosting platform and model inference hub that allows software developers to integrate AI models into their apps	<a href=https://github.com/replicate/cog>	<a href=https://pypi.org/project/replicate>
Anyscale	An AI API and compute platform that allows developers to scale inference, training, and embedding computations with any model using Ray	<a href=https://github.com/ray-project/ray>	<a href=https://pypi.org/project/ray>
Ollama	Get up and running with large language models locally	<a href=https://github.com/ollama/ollama>	<a href=https://pypi.org/project/ollama/>
GPT4ALL	An ecosystem of open-source on-edge large language models.	<a href=https://github.com/nomic-ai/gpt4all>	<a href=https://pypi.org/project/gpt4all/>

Name (site)	Description
Amazon Web Services	A cloud computing platform, offering services from data centers globally
Google Cloud Platform	A cloud computing platform, offering services from data centers globally
Microsoft Azure	A cloud computing platform, offering services from data centers globally
CoreWeave	A specialized cloud provider that delivers GPUs on top of flexible deployment infrastructure

Name (site)	Description
Vercel	A cloud platform designed for front-end engineers, built with first-class support for LLM apps
Netlify	An enterprise cloud computing company that offers a development platform for web applications and dynamic websites
Steamship	An SDK and hosting platform for AI agents and tools, both a package manager and package hosting service for AI
Streamlit	An open-source Python library designed for creating and sharing custom web apps for ML and data science
Modal	A platform that enables running distributed applications using the modal Python package

Name (site)	Description
Databricks (MosaicML)	Databricks acquired Mosaic ML in 2023, along with its tooling and platform for efficient pre-trainining, fine-tuning and inferencing LLMs
Anyscale	An AI compute platform that enables developers to scale inference, training, and embedding computations with any model using Ray
Modal	A platform that eables running distributed applications using the Modal Python package
Runpod	A cloud computing platform designed for AI and ML applications
OctoML	A compute service that allows users to run, tune, and scale generative models
Baseten	A inference service that allows users to deploy, serve, and scale custom and open-source models
E2B	Secure sandboxed cloud environments made for AI agents and AI apps