---

memU compiles conversations, documents, code, images, audio, video, URLs, and tool traces into human-readable Markdown files (INDEX.md, MEMORY.md, SKILL.md). Agents traverse the tree and load only what the moment needs — instead of rescanning everything or stuffing long histories into every prompt.

await service.memorize(resource_url="workspace/meeting-notes.md", modality="document")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What should I know about this user's launch preferences?"}}],
    where={"user_id": "123"},
)

That's it. Instead of one giant prompt about a person or their workspace, your agent gets three durable layers it can traverse:

workspace/
├── INDEX.md              ← Index: a map of everything — raw sources and summaries
├── MEMORY.md             ← Memory: an overview that links into memory/
├── SKILL.md              ← Skill: an overview that links into skill/
├── resource/             ← the raw source files, copied verbatim
├── memory/
│   └── <topic>.md        ← one memory file per topic: facts, preferences, goals, events
└── skill/
    └── <name>.md         ← one skill file per learned pattern, workflow, or mistake to avoid

• Index (INDEX.md) — a map of your memories: what exists, where it came from, and where to look first
• Memory (MEMORY.md) — personal facts, preferences, goals, events, and decisions extracted from source data
• Skill (SKILL.md) — auto-extracted from tool traces and refined on every memorize() so the agent improves at recurring tasks

Three things make it different from stuffing everything into the prompt:

• Fast retrieval — walk to the right folder and rank the right files instead of scanning everything every time.
• Higher accuracy — scope by user, task, or session, and trace every item back to the exact conversation, document, image, or log it came from.
• Lower cost — retrieve compact, scoped context instead of reinjecting long histories, documents, logs, and media-derived text into every prompt.
• Yours to inspect — a human-readable file tree you can audit, edit, scope, and route through your own storage (inmemory, sqlite, postgres) and LLM providers.

---

⭐️ Star the repository

If you find memU useful or interesting, a GitHub Star ⭐️ would be greatly appreciated.

---

✨ Core Features

Capability	Description
🗂️ Multimodal Ingestion	Write conversations, documents, images, video, audio, URLs, logs, and local files into memory
📁 Compiled Memory Workspace	Persist the Index, Skill, and Memory layers — folders (categories), files (items), source artifacts, links, summaries, and embeddings
🧠 Typed Memory Extraction	Extract profile, event, knowledge, behavior, skill, and tool memories from raw sources
🛠️ Self-Evolving Skills	Auto-extract reusable tool patterns and workflows from tool traces, then merge and refine them on every memorize() instead of relearning
🧭 Self-Organizing Folders	Auto-build categories, links, summaries, and embeddings without manual tagging
🤖 Agent-Ready Retrieval	Find the right files quickly — scoped, ranked context for any agent workflow
🧱 Pluggable Storage	Use in-memory, SQLite, or Postgres backends with the same repository contracts
🔀 Profile-Based LLM Routing	Route chat, embedding, vision, and transcription work through configurable LLM profiles

---

🎯 Use Cases

1. Personal Memory

Turn chat logs and workspace notes into user preferences, goals, events, decisions, and relationship context.

await service.memorize(
    resource_url="examples/resources/conversations/conv1.json",
    modality="conversation",
    user={"user_id": "123"},
)

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What should I remember about this user?"}}],
    where={"user_id": "123"},
)

2. Workspace Context for Coding Agents

Convert docs, PR notes, logs, and design decisions into reusable project memory.

await service.memorize(resource_url="docs/architecture.md", modality="document")
await service.memorize(resource_url="examples/resources/logs/log1.txt", modality="document")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "How should I structure this module?"}}],
)

3. Multimodal Knowledge Layer

Extract searchable facts from documents, screenshots, images, videos, and audio notes.

await service.memorize(resource_url="examples/resources/docs/doc1.txt", modality="document")
# Rich documents (PDF/Word/PowerPoint/Excel/HTML) are converted to Markdown via
# MarkItDown — install the extra with: pip install 'memu-py[document]'
await service.memorize(resource_url="reports/q3-summary.pdf", modality="document")
await service.memorize(resource_url="examples/resources/images/image1.png", modality="image")
# Audio is supported for your own .mp3/.wav/.m4a files.
await service.memorize(resource_url="meeting-audio.mp3", modality="audio")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What matters for the next research plan?"}}],
)

4. Tool and Agent Learning

Turn execution traces into tool memories that tell future agents when to use a tool and what mistakes to avoid.

await service.memorize(resource_url="examples/resources/logs/log1.txt", modality="document")

context = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "Which tools worked for config editing?"}}],
)

---

🗂️ Architecture

The compiled workspace is hierarchical enough for browsing and structured enough for direct retrieval:

Layer	Primary Role	Retrieval Role
Category (folder)	Maintain topic-level summaries	Assemble compact context for broad queries
Item (file)	Store typed atomic memories	Load precise facts, events, preferences, skills, and tool patterns
Resource (source)	Preserve source artifacts and captions	Recall original context when item/category summaries are not enough

See docs/architecture.md for the runtime view of MemoryService, workflow pipelines, storage backends, and LLM routing.

---

🚀 Quick Start

Option 1: Cloud Version

👉 memu.so — Hosted API for managed ingestion, structured memory, and retrieval

For enterprise deployment: info@nevamind.ai

Cloud API (v3)

Base URL	https://api.memu.so
Auth	Authorization: Bearer <token>

Method	Endpoint	Description
POST	/api/v3/memory/memorize	Ingest raw data and build structured memory
GET	/api/v3/memory/memorize/status/{task_id}	Check processing status
POST	/api/v3/memory/categories	List auto-generated categories
POST	/api/v3/memory/retrieve	Query memory for agent context

📚 Full API Documentation

---

Option 2: Self-Hosted

Installation

From a clone of this repository:

uv sync
# or, for the full development setup:
make install

To install the published package instead:

pip install memu-py

Requirements: Python 3.13+. The default examples use OpenAI, so set OPENAI_API_KEY or pass another provider through llm_profiles.

Run an in-memory smoke script:

export OPENAI_API_KEY=your_key
cd tests
uv run python test_inmemory.py

Run with PostgreSQL + pgvector:

uv sync --extra postgres
docker run -d --name memu-postgres \
  -e POSTGRES_USER=postgres \
  -e POSTGRES_PASSWORD=postgres \
  -e POSTGRES_DB=memu \
  -p 5432:5432 \
  pgvector/pgvector:pg16

export OPENAI_API_KEY=your_key
export POSTGRES_DSN=postgresql+psycopg://postgres:postgres@127.0.0.1:5432/memu
cd tests
uv run python test_postgres.py

---

Custom LLM and Embedding Providers

from memu import MemUService

service = MemUService(
    llm_profiles={
        "default": {
            "base_url": "https://dashscope.aliyuncs.com/compatible-mode/v1",
            "api_key": "your_key",
            "chat_model": "qwen3-max",
            "client_backend": "sdk"
        },
        "embedding": {
            "base_url": "https://api.voyageai.com/v1",
            "api_key": "your_key",
            "embed_model": "voyage-3.5-lite"
        }
    },
)

---

OpenRouter Integration

from memu import MemoryService

service = MemoryService(
    llm_profiles={
        "default": {
            "provider": "openrouter",
            "client_backend": "httpx",
            "base_url": "https://openrouter.ai",
            "api_key": "your_key",
            "chat_model": "anthropic/claude-3.5-sonnet",
            "embed_model": "openai/text-embedding-3-small",
        },
    },
    database_config={"metadata_store": {"provider": "inmemory"}},
)

---

📖 Core APIs

memorize() — Structure Raw Data

result = await service.memorize(
    resource_url="path/to/file.json",    # local file path or HTTP URL
    modality="conversation",            # conversation | document | image | video | audio
    user={"user_id": "123"},            # optional: scope to a user or agent
)
# Returns after processing completes:
# { "resource": {...}, "items": [...], "categories": [...], "relations": [...] }

• Converts raw input into typed memory items
• Categorizes and embeds items without manual tagging
• Preserves source resources and item-category relations

---

retrieve() — Load Agent Context

# The retrieval strategy is set once on the service via retrieve_config:
#   MemoryService(retrieve_config={"method": "rag"})   # vector-first recall
#   MemoryService(retrieve_config={"method": "llm"})   # LLM-ranked recall
result = await service.retrieve(
    queries=[{"role": "user", "content": {"text": "What are their preferences?"}}],
    where={"user_id": "123"},   # scope filter
)
# Returns:
# {
#   "needs_retrieval": true,
#   "original_query": "...",
#   "rewritten_query": "...",
#   "next_step_query": "...",
#   "categories": [...],
#   "items": [...],
#   "resources": [...]
# }

retrieve_config.method	Behavior	Cost	Best For
rag	Vector-first category/item/resource recall, with optional LLM routing and sufficiency checks enabled by default	Embeddings plus LLM calls unless route_intention and sufficiency_check are disabled	Fast scoped recall with controllable reasoning
llm	LLM-ranked category/item/resource recall	LLM ranking at each tier	Deeper semantic ranking

---

💡 Example Workflows

Always-Learning Assistant

export OPENAI_API_KEY=your_key
uv run python examples/example_1_conversation_memory.py

Automatically extracts preferences, builds relationship models, and surfaces relevant context in future conversations.

Self-Improving Agent

uv run python examples/example_2_skill_extraction.py

Monitors agent actions, identifies patterns in successes and failures, auto-generates skill guides from experience.

Multimodal Context Builder

uv run python examples/example_3_multimodal_memory.py

Cross-references text, images, and documents automatically into a unified memory layer.

---

📊 Performance

memU achieves 92.09% average accuracy on the Locomo benchmark across all reasoning tasks.

View detailed results: memU-experiment

---

🧩 Ecosystem

Repository	Description
memU	Personal memory as files — fast retrieval, higher accuracy, lower cost
memU-server	Backend with real-time sync and webhook triggers
memU-ui	Visual dashboard for browsing and monitoring memory

Quick Links:

• 🚀 Try MemU Cloud
• 📚 API Documentation
• 💬 Discord Community

---

🤝 Partners

---

🤝 Contributing

# Fork and clone
git clone https://github.com/YOUR_USERNAME/memU.git
cd memU

# Install dev dependencies
make install

# Run quality checks before submitting
make check

See CONTRIBUTING.md for full guidelines.

Prerequisites: Python 3.13+, uv, Git

---

📄 License

Apache License 2.0

---

🌍 Community

• GitHub Issues: Report bugs & request features
• Discord: Join the community
• X (Twitter): Follow @memU_ai
• Contact: info@nevamind.ai

---