stack-tools.md

Frameworks & Tools

Modern Python for Gen-AI

• Modern Python for Gen-AI

Agent Frameworks

• OpenAI Agents
  • OpenAI Agents (Guides)
  • OpenAI Agents (Python SDK)
  • OpenAI Agents Cookbook (Agent Orchestration)
• LlamaIndex: Agentic RAG and workflow orchestration
• LangGraph: Agent and workflow management
• Crew AI (GitHub): Multi-agent orchestration framework
• AutoGen: Multi-agent patterns and chatbot development (Core & AgentChat)
• HuggingFace SmolAgents: Lightweight code generation agents

UI Development Tools

• Streamlit: Rapid prototyping for data apps
• Gradio: ML model interfaces
• Chainlit: Agent-specific UI development

Model Rankings & Evaluation

LLM Performance Rankings

• Hugging Face LLM Leaderboard
• OpenRouter Trending
• LMSYS Chatbot Arena: Crowdsourced head-to-head LLM comparisons

Benchmark Leaderboards

• Hugging Face Open LLM Leaderboard: Comprehensive open-source model benchmarks
• AlpacaEval Leaderboard: LLM alignment evaluation
• HELM by Stanford: Holistic model evaluation framework
• LLMPerf: Automated LLM benchmarking
• Hugging Face Arena-Hard Leaderboard: Advanced model robustness testing

Gen-AI Infrastructure

LLM APIs

• OpenAI API: GPT-4, tools, and assistants
• Anthropic Claude API: Extended context, safety-focused
• Google Vertex AI / Gemini API: Enterprise multimodal capabilities
• Mistral: Available via OpenRouter or Hugging Face
• Meta LLaMA: Available via Together.ai or Hugging Face
• Cohere API: RAG-optimized with strong embeddings
• OpenRouter API: Multi-model access layer
• LittleLLM: Lightweight agent framework

API Usage Considerations

• Monitor rate limits:
  • Requests per minute (RPM)
  • Tokens per minute (TPM)
  • Requests per day (RPD)

Embedding Solutions

• OpenAI Embeddings
• Cohere Embeddings
• HuggingFace Sentence Transformers
• LangChain Embeddings
• VoyageAI: Embedding + re-rankers as a service

Vector Databases

• FAISS: High-performance similarity search
• ChromaDB: Embedded vector store
• Pinecone: Managed vector service
• Weaviate: Vector search engine
• Haystack: End-to-end search pipeline

Local Deployment

• Ollama: Local model deployment and management

Orchestration

• Agent Orchestration Primitives:

  • LangGraph: uses graph nodes (LLMs/tools/memory) with edge-controlled execution
  • CrewAI: defines Agents, Tasks, Tools, and Crew structured execution like a task force
  • Autogen: supports Agent-to-Agent and Human-in-the-loop collaboration

• Workflow Orchestration:

  • Prefect: Workflow orchestration
  • Airflow: Workflow orchestration

• Infrastructure Orchestration:

  • Kubernetes: Container orchestration
  • Docker: Containerization
  • CloudInfra: Cloud infrastructure as code
  • AWS: Cloud infrastructure

• AI Orchestration Tools, Akka.io

Observability

• Opik: Monitoring and evaluation
• todo add more

Specialized Agent Tools

• mem0: Agent memory management
• Embedchain: LLM application framework
• Agno: Knowledge-centric agent framework
• Firecrawl: Web data extraction (FIRE-1 crawler)

Examples

Example Stacks

Philoagent Simulation Architecture

Layer	Tools / Frameworks
Memory	MongoDB
Inference	FastAPI, LangGraph
Vector Indexing	Custom Embedding + Vector DB
Orchestration	Agentic Layer, State Client
LLM API	Groq
Observability	Opik (Monitoring & Evaluation)
UI	Custom Game UI

graph TB
  subgraph L1[Infra Orchestration Layer]
    K8s[Kubernetes] --> Deploy
    Docker --> K8s
    Terraform --> CloudInfra
  end

  subgraph L2[Workflow Orchestration Layer]
    Prefect --> WorkerNode1
    Temporal --> AgentTask1
    Airflow --> DAG1
  end

  subgraph L3[Agent Reasoning Layer]
    LangGraph --> PlanTask
    CrewAI --> AssignAgents
    Autogen --> CodeAgent
  end

  UserQuery --> LangGraph
  LangGraph --> Prefect
  Prefect --> Docker
  Docker --> K8s

graph TD
  A[User Query] --> B[LangGraph: Plans Task Nodes]
  B --> C[Prefect: Orchestrates Steps]
  C --> D1[LangChain: Tool - Web Search]
  C --> D2[LangChain: Tool - PDF Extractor]
  C --> D3[LangChain: Tool - Blog Generator]
  D1 --> E[Redis/Pinecone: Stores Embeddings]
  D2 --> E
  D3 --> F[Draft Output]
  C --> G[Prometheus Logs + Prefect Retry Logic]
  F --> H[LangGraph Final Node: Output Response]

graph TD
  A[User Query] --> B[Agent Reasoning Layer<br/>Plans task, selects tools, sets goal]
  B --> C[Workflow Layer<br/>Coordinates multi-step execution like fetch, summarize, email]
  C --> D[Tool Execution Layer<br/>Calls APIs, plugins, or internal tools]
  D --> E[Data Movement Layer<br/>Moves inputs and outputs between components]
  E --> F[Infrastructure Layer<br/>Ensures services are running and scaled]
  F --> G[Monitoring & Feedback Layer<br/>Logs, evaluates, and informs agent refinement]