Architecture Notes: "Waterfall" Context Management For Personal LLM Use

18-12 (Dec)-2024

Notes

The purpose of this repository is to hold some notes and schematics for an architecture that I am working on for managing personal contextual data and feeding it into large language models and LLM agents.

The architecture envisioned here is not a sophisticated RAG workflow intended for large-scale enterprise use. Rather, it's intended as a more pragmatic way of managing and routing contextual data intended for interested end-users although it could perhaps be adapted for a smaller business use.

Since I began exploring the field of large language models and working with them intensively throughout this year, I've been intrigued by the question of how context can be carefully utilized to improve the quality, relevance and accuracy of inference.

My experience has been that contextual data can be extremely small but highly useful. As an example, a single markdown file that I use which denotes my hardware specification has proven incredibly useful in everything from immediately setting context with a memory free LLM . The process can be replicated ad infinitum across an essentially endless breadth of subjects reflecting both one's personal life and professional endeavors.

The issue with managing personal context in this manner is that it's not very user-friendly or scalable. Thus my feelings have been that creating workflows which mimic a little more closely the pipelines seen in RAG and vector storage makes more sense. This would actually leave users in a powerful position - they could assert ownership over a contextual data store, as I think is proper and necessary, and this contextual repository, as I call it, is not wedded to any one LLM platform or provider.

The process of ideating, generating, collecting and then vectorizing these context snippets as I call them (a “context snippet” representing one discreet set of notes on a topic, stored in markdown) is not at all difficult. For those interested in this topic, I've created some tools like Hugging Chat Assistants, which actually proactively probe the user by asking questions and generate structured context notes as outputs. But even taking a less active approach, it would be possible to develop a quite comprehensive bank or reservoir of contextual data about one's life and career.

The question becomes, however, what's the easiest way to front-end these data stores?

The architecture described in this note is a “first thought” at one way that this could be done. Platforms like OpenAI make it really easy to create abundant assistants and vector stores to create segregated resources with granular API key configurations. To my mind this is a very wise approach and the idea in this architecture is to work with it to feed specific parts of one's contextual imprint into specific vector stores.

Submodules For Context "Flow" From Specific To General

In simple terms, the architecture planned here uses GitHub sub-modules and workflows in order to provide the user with a system for making the various moving parts of the context picture make sense.

The architecture here moves from specific context repos to the master context repo. It does so to because of how GitHub submodules work. A repository can be set up to hold context for a specific project feeding into a specific vector store. And then sub-modules are used to gather all these smaller and subject specific pieces of context into the one master store.

Vector store pipelines can be set up laterally at all the levels of this architecture. It's assumed that they exist between the specific subject repositories. And once those are gathered into the main context store, a single pipeline can be used to copy all that data into one master context vector store. This might mean that the user has duplicate or replicate context data stored in different vector stores. But given the very affordable price at which storage in these technologies is currently priced and the very lightweight nature of textual file storage this doesn't seem like a duplication worth chasing down.

Notes and screenshots might be added as I work through implementing this approach in my own experiments with using context more systematically.

Author

Daniel Rosehill
(public at danielrosehill dot com)

Licensing

This repository is licensed under CC-BY-4.0 (Attribution 4.0 International) License

Summary of the License

The Creative Commons Attribution 4.0 International (CC BY 4.0) license allows others to:

• Share: Copy and redistribute the material in any medium or format.
• Adapt: Remix, transform, and build upon the material for any purpose, even commercially.

The licensor cannot revoke these freedoms as long as you follow the license terms.

License Terms

• Attribution: You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
• No additional restrictions: You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.

For the full legal code, please visit the Creative Commons website.