Storage Layout

June 5, 2026 · View on GitHub

How everos lays out a memory-root on disk: directory tree, file naming, frontmatter chassis, and entry-id encoding.

The contents are the source of truth; SQLite and LanceDB are derived indexes that can be rebuilt from markdown alone.

1. Memory-root tree

A memory-root is a single directory holding all persisted memory. The default location is ~/.everos/; override via EVEROS_MEMORY__ROOT env var or [memory] root in the TOML config.

Memory is partitioned by <app_id>/<project_id> before the user-visible scope dirs, so different (app, project) spaces never share a directory. The reserved id "default" materialises as default_app / default_project on disk. The scope is encoded in the path, not in the frontmatter (see §3).

<memory-root>/                              default ~/.everos
│
├── <app_id>/                               user-visible; "default" → default_app
│   └── <project_id>/                       "default" → default_project
│       ├── users/
│       │   └── <user_id>/
│       │       ├── user.md                          single-file rewrite (profile)
│       │       ├── episodes/                         daily-log append
│       │       │   └── episode-<YYYY-MM-DD>.md
│       │       ├── .atomic_facts/                    daily-log append (hidden)
│       │       │   └── atomic_fact-<YYYY-MM-DD>.md
│       │       └── .foresights/                      daily-log append (hidden)
│       │           └── foresight-<YYYY-MM-DD>.md
│       ├── agents/
│       │   └── <agent_id>/
│       │       ├── .cases/                           daily-log append (hidden)
│       │       │   └── agent_case-<YYYY-MM-DD>.md
│       │       └── skills/                           skill-named dir
│       │           └── skill_<name>/
│       │               ├── SKILL.md
│       │               ├── references/               (optional)
│       │               └── scripts/                  (optional)
│       └── knowledge/                                user-visible (shared / global, reserved)
│
├── .index/                              system-managed, rebuildable (gitignore)
│   ├── sqlite/
│   │   ├── system.db                    state / cascade queue (md_change_state) / buffer / audit / LSN  (+ -wal / -shm)
│   │   ├── ome.db                        Offline Memory Engine state
│   │   ├── ome.aps.db                    APScheduler jobstore (split to avoid lock contention)
│   │   └── ome.db.lock                   OME single-engine guard (portalocker)
│   └── lancedb/
│       └── <kind>.lance/                one directory per LanceDB table
│
├── ome.toml                             user-editable OME strategy overrides (hot-reloaded)
└── .tmp/                                staging dir for batch / multi-step writes

Cascade queue state, the LSN watermark, and the change audit all live in SQLite (system.db, table md_change_state) — crash-recovery replays from that durable queue, not a log file. (MemoryRoot also exposes a .lock anchor for the memory_root_lock primitive; there is no .cascade.log / .manifest.json.)

The path manager is MemoryRoot, exposing every path as a property. MemoryRoot.ensure() creates the runtime-required dirs (.index/{sqlite,lancedb}/, .tmp/) and copies the OME template to ome.toml; the user-visible dirs are not pre-created — they appear on first write.

The single-file writer also supports agent.md / soul.md / tools.md / behaviors.md, but no shipped strategy produces those today — only user.md is written. memcell is a SQLite-only kind (the boundary ledger); it has no markdown file.

2. Three storage strategies

Each business memory kind picks one of three on-disk patterns:

StrategyFilenameMutationExamples
Daily-log append<FILE_PREFIX>-<YYYY-MM-DD>.md under <DIR_NAME>/append entriesepisode / atomic_fact / foresight / agent_case
Skill-named dirskills/skill_<name>/SKILL.md (+ references/ scripts/)overwrite the fileagent skills (procedural memory)
Single-file rewriteuser.md (writer also supports agent.md / soul.md / tools.md / behaviors.md, not yet produced)overwrite the fileuser profile

Markdown IO primitives live in core/persistence/markdown/; business-aware writers live in infra/persistence/markdown/writers/ and pick the right strategy via a base class.

To add a new memory kind, define its per-kind frontmatter schema under infra/persistence/markdown/mds/ and add a matching writer/reader pair under writers/ and readers/.

3. Frontmatter chassis (YAML)

Every markdown file carries a YAML frontmatter block at the top:

---
id: episode_log_alice_2026-06-01
type: episode_daily
file_type: episode_daily
schema_version: 1
user_id: alice
track: user
date: '2026-06-01'
entry_count: 11
last_appended_at: '2026-06-01T09:12:13+00:00'
---
<!-- entry:ep_20260601_00000001 -->
...content...
<!-- /entry:ep_20260601_00000001 -->

Scope (app_id / project_id) is not a frontmatter field — it is carried by the <app>/<project> path segments and recovered by the cascade path parser. The frontmatter only holds the file-level owner (user_id / agent_id) and track.

The chassis lives in core/persistence/markdown/frontmatter.py (Pydantic v2):

BaseFrontmatter            id / type / schema_version + SCOPE_DIR ClassVar
   ├─ UserScopedFrontmatter   + user_id / track="user" + SCOPE_DIR="users"
   └─ AgentScopedFrontmatter  + agent_id / track="agent" + SCOPE_DIR="agents"

Concrete business schemas subclass one of the scope mixins and add per-kind fields plus three more ClassVars that drive path resolution

  • entry-id assembly:
class EpisodeDailyFrontmatter(DailyLogPathMixin, UserScopedFrontmatter):
    ENTRY_ID_PREFIX: ClassVar[str] = "ep"
    DIR_NAME: ClassVar[str] = "episodes"
    FILE_PREFIX: ClassVar[str] = "episode"
    type: Literal["episode_daily"] = "episode_daily"
    date: dt.date
    entry_count: int = 0
    last_appended_at: dt.datetime | None = None

4. Entry-id encoding

Inside daily-log files each entry is bracketed by HTML-comment markers so the raw markdown stays clean for human readers:

<!-- entry:<entry_id> -->
...content...
<!-- /entry:<entry_id> -->

<entry_id> is <prefix>_<YYYYMMDD>_<NNNNNNNN> (8-digit sequence), e.g. ep_20260601_00000001:

SegmentSource
prefixFrontmatter.ENTRY_ID_PREFIX (declared by the schema subclass)
<YYYYMMDD>The daily-log file's date bucket
NNNNNNNNPer-file sequence, 8-digit zero-padded, restarts at 00000001 each day per scope

Implementation: core/persistence/markdown/entries.py (EntryId.parse / format / next_for).

File-level seq, not global: the same ep_20260601_00000001 may appear across two different user_ids (each user has its own daily file). Cross-table joins must therefore key on (scope_id, entry_id) rather than entry_id alone — see SQLite/LanceDB tables that follow.

5. SQLite + LanceDB derived indexes

.index/
├── sqlite/
│   └── system.db          state / audit log / task queue / LSN watermark
│                           + per-kind business state tables (composite key)
└── lancedb/
    └── <kind>.lance/      one Arrow-based table per kind
                            stores text / vector / tags / metadata

Both layers are fully derivable from markdown — wipe .index/ and the in-process cascade subsystem re-builds everything by scanning the user-visible tree (the durable md_change_state SQLite queue covers crash-recovery replay).

6. Atomic write semantics

MarkdownWriter uses a same-directory temp file (.<name>.tmp.<uuid>) + os.replace for atomicity. Keeping the temp file in the same directory guarantees os.replace is atomic on POSIX (the rename is only atomic within a single filesystem).

MarkdownWriter.append_entry reads → merges frontmatter → appends an entry block → atomic write back. The caller passes a full EntryId (built via EntryId.next_for(prefix, date, current_count)); this primitive is schema-agnostic — field-level semantics (entry_count / last_appended_at) are a business writer's job (see BaseDailyAppender._frontmatter_updates in infra/persistence/markdown/writers/base.py).

7. References