Local Llama

July 12, 2024 ยท View on GitHub

This project enables you to chat with your PDFs, TXT files, or Docx files entirely offline, free from OpenAI dependencies. It's an evolution of the gpt_chatwithPDF project, now leveraging local LLMs for enhanced privacy and offline functionality.

Features

  • Offline operation: Run in airplane mode
  • Local LLM integration: Uses Ollama for improved performance
  • Multiple file format support: PDF, TXT, DOCX, MD
  • Persistent vector database: Reusable indexed documents
  • Streamlit-based user interface

New Updates

  • Ollama integration for significant performance improvements
  • Uses nomic-embed-text and llama3:8b models (can be changed to your liking)
  • Upgraded to Haystack 2.0
  • Persistent Chroma vector database to enable re-use of previously updloaded docs

Installation

  1. Install Ollama from https://ollama.ai/download
  2. Clone this repository
  3. Install dependencies: 
    pip install -r requirements.txt
  4. Pull required Ollama models: 
    ollama pull nomic-embed-text
ollama pull llama3:8b

Usage

  1. Start the Ollama server: 
    ollama serve
  2. Run the Streamlit app: 
    python -m streamlit run local_llama_v3.py
  3. Upload your documents and start chatting!

How It Works

  1. Document Indexing: Uploaded files are processed, split, and embedded using Ollama.
  2. Vector Storage: Embeddings are stored in a local Chroma vector database.
  3. Query Processing: User queries are embedded and relevant document chunks are retrieved.
  4. Response Generation: Ollama generates responses based on the retrieved context and chat history.

License

This project is licensed under the Apache 2.0 License.

Acknowledgements

  • Ollama team for their excellent local LLM solution
  • Haystack for providing the RAG framework
  • The-Bloke for the GGUF models