Exploration Log

August 30, 2026 · View on GitHub

Chinese | English

Exploration Log

This document records the web searches, technical decisions, and validation conclusions made during the implementation of the DeepSeek-OCR1 memory plugin.

1. DeepSeek-OCR Paper

  • Paper: DeepSeek-OCR: Contexts Optical Compression
  • arXiv: https://arxiv.org/abs/2510.18234
  • Core concept: Use DeepEncoder to compress document images into a small number of visual tokens, then decode them with DeepSeek-3B-MoE-A570M.
  • Official resolution modes:
    • Tiny: 512×512 → 64 tokens
    • Small: 640×640 → 100 tokens
    • Base: 1024×1024 → 256 tokens
    • Large: 1280×1280 → 400 tokens
    • Gundam: dynamic tiling
  • OCR-Memory (ACL 2026): Explicitly uses DeepSeek-OCR 3B as the backbone, with SoM + Locate-and-Transcribe + age-aware multi-resolution + active recall.
  • AgentOCR (ACL 2026 Oral): Optical self-compression + segment optical caching, with open-source code, but its backbone is Qwen2.5-VL.
  • MemOCR (Meituan): A visual memory agent whose backbone is Qwen2.5-VL; it follows a similar approach rather than being a direct OCR1 implementation.
  • VTC-R1: Uses optical memory for long-context reasoning; its backbone is Glyph/Qwen3-VL, not DeepSeek-OCR.

3. Local Deployment of DeepSeek-OCR

  • vLLM officially supports DeepSeek-OCR, but this machine has an AMD RX 7800 XT and no NVIDIA CUDA, so the official vLLM route is not recommended.
  • Community GGUF sources:
    • Ollama: ollama run deepseek-ocr (requires v0.13.0+)
    • Hugging Face:
      • ggml-org/DeepSeek-OCR-GGUF (Q8_0, recommended by the official llama.cpp blog)
      • sabafallah/DeepSeek-OCR-GGUF (Q4_K_M / Q8_0 / BF16)
      • NexaAI/DeepSeek-OCR-GGUF (Q4_K / Q5_K / Q6_K / Q8_0, etc.)
  • Empirical results:
    • ollama pull deepseek-ocr stalled at 84%, with speed dropping to 100KB/s, so it was abandoned.
    • The download succeeded using aria2c with multiple threads from hf-mirror.com, reaching several MB/s.
    • The sabafallah Q4_K_M triggered a crash in the llama.cpp CLI and may have been converted from a PR branch; however, it runs stably under llama-server.
    • Currently, sabafallah's deepseek-ocr-Q4_K_M.gguf + mmproj-deepseek-ocr-q8_0.gguf run stably through llama-server.

4. llama.cpp / AMD Route

  • Downloads: llama-b10453-bin-win-vulkan-x64.zip and llama-b10453-bin-win-cpu-x64.zip
  • Paths: <models>\llama.cpp\, <models>\llama.cpp-cpu\
  • Runtime:
    • llama-server listens on 127.0.0.1:18080
    • Uses deepseek-ocr-Q4_K_M.gguf + mmproj-deepseek-ocr-q8_0.gguf (uses less VRAM)
  • Key parameters:
    • A \nFree OCR.-style prompt must be used
    • repeat_penalty and no_repeat_ngram_size must be added to prevent repetitive output
  • Notes:
    • llama-mtmd-cli crashes in both Vulkan and CPU modes (0xC0000409), but llama-server works normally.
    • This was observed empirically on this machine and may be related to Windows, the driver, or the build.

5. Local Environment

  • GPU: AMD Radeon RX 7800 XT (16GB) + Radeon 780M
  • No NVIDIA CUDA
  • Python 3.12 / Node 24 / bun 1.3
  • Pillow is installed; torch is not installed
  • Model files: <models>\deepseek-ocr-gguf\

6. llama-server Runtime Notes

  • -c 8192 is more suitable for a standalone OCR service handling 1280×1280 image tokens; when OCR and embeddings share an endpoint, the plugin defaults to ocrEmbeddingContextSize=2048 and ocrEmbeddingUbatchSize=2048.
  • The plugin includes built-in automatic startup: lib/ocr-server.js + the autoStartOcrServer configuration.
  • Automatic startup uses a detached direct spawn of llama-server.exe, not a PowerShell wrapper; concurrent starts for one endpoint are merged, and cancellation/failure/disposal cleans only plugin-owned processes.
  • Equivalent current combined CPU-service arguments: 
    llama-server.exe --host 127.0.0.1 --port 18080 \
  -m deepseek-ocr-Q4_K_M.gguf \
  --mmproj mmproj-deepseek-ocr-q8_0.gguf \
  --alias deepseek-ocr -c 2048 -np 1 -n 1024 --embeddings --pooling mean -b 2048 -ub 2048

7. llama.cpp Multimodal Embedding (Real Visual Embedding)

  • Conclusion: The llama.cpp llama-server /v1/embeddings endpoint supports image input, but the request format is not the OpenAI-standard input string. Instead, it is: 
    { "input": [ { "prompt_string": "<media_marker>", "multimodal_data": ["<raw base64>"] } ] }
  • <media_marker> must be obtained dynamically from the media_marker field returned by GET /props (randomized on every startup).
  • multimodal_data must be raw base64, not data:image/png;base64,... (the latter reports Failed to load image or audio file).
  • Some llama.cpp builds default -ub (physical batch size) to 512. A 1024×200 SoM memory image has approximately 784 visual tokens and reports input too large; the plugin's embedding/combined auto-start passes -ub 2048 by default.
  • Empirical results:
    • The current llama.cpp build supports both /v1/chat/completions and /v1/embeddings under --embeddings --pooling mean, so OCR and visual embedding can share the same service on 18080;
    • A marker-only request has prompt_tokens=785;
    • Empty text input:"" has prompt_tokens=1;
    • Direct visual tokens = 784, embedding dimension = 1280;
    • The server normalizes by Euclidean norm (--embd-normalize 2 by default).
  • Integrated into the plugin: measureImageEmbedding / createEmbeddingHttpClient, with embedding, embeddingDim, and visualTokensDirect persisted in visualMemory.

8. Key Conclusions

  • The plugin core can now genuinely invoke DeepSeek-OCR to read images.
  • Isolated temporary testing passes; the current npm test run is 88/88 with the live backend (eight live-backend cases skip when it is unavailable).
  • Added the ocr1_mem_metrics tool: estimates the text-token/visual-token compression ratio according to the official resolution modes and records usage.prompt_tokens from real OCR requests and the approximate number of visual tokens.
  • Added the ocr1_mem_update tool: supports explicit memory updates for conflict resolution/latest-value overwrite.
  • Added scripts/start-ocr-server.ps1, scripts/ensure-ocr-server.mjs, and lib/ocr-server.js.
  • Added the autoStartOcrServer configuration to the plugin: when enabled, the plugin automatically ensures that llama-server is online when the plugin loads; an explicit port in ocrBaseUrl controls health checks and launch, and an external already-running service is not stopped.
  • Comparative benchmark (BENCHMARK.md, scripts/compare-memory.mjs): R1–R6 were fully rerun with the corrected script; dsh-ocr1-memory passed all tests; dsh-memory also passed all tests this time, but R5 had previously been marked FAIL during manual validation (it remained readable after archiving), indicating unstable behavior; R4 now has an explicit update capability.
  • DSH-level R1–R6 completed a full comparison in an isolated headless environment (without killing processes, running in the background): dsh-ocr1-memory passed all tests.
  • Robustness enhancements have been implemented: a shared store for multiple Agents (sharedStore), automatic recovery from missing images/corrupted caches, boundary testing for overlong input, pure-query list, bounded tier refresh, and renderer cancellation propagation.
  • Maintenance is bounded, cancellable, and single-flight per namespace; maintenanceBatchSize defaults to 8, and plugin disposal cancels and drains automatic maintenance.
  • Real 1280-dimensional DeepSeek-OCR visual embeddings are now stored through marker-only requests to llama.cpp /v1/embeddings, and the number of direct visual tokens is measured (marker-only prompt_tokens − empty-text baseline); the current probe is 785−1=784.
  • Visual embedding similarity retrieval has been implemented: measureTextEmbedding embeds queries, and retrieveSegmentsWithEmbeddings incorporates cosine similarity into ranking. Because cross-modal discriminability has not been sufficiently validated, the plugin disables embeddingRetrieval by default and does not use it as an unconditional primary retrieval signal.
  • General agent memory testing specifications (MemoryAgentBench / LongMemEval / LoCoMo / AMB) were investigated and organized into TEST_SPEC.md; R1–R6 map one-to-one to these specifications.
  • An experiment attempted to have DeepSeek-OCR directly output SoM numbers without fine-tuning: with the current llama.cpp backend, output under a custom locate prompt is unreliable (it returns irrelevant text rather than numbers), so the paper's original Locate method (model-generated numbers) cannot proceed without LoRA. The plugin's opticalLocatorStrict path still requires a compatible trained locator and rejects malformed output instead of falling back to text.

9. LoRA Optical Locator: Progress from Local Execution (2026-08)

9.1 Environment and Toolchain (WSL2 / RX 7800 XT / ROCm)

  • Independent training venv /root/ocr1-train-env: torch 2.11.0+rocm7.2 + torchvision 0.26.0+rocm7.2 + transformers 4.57.2 + trl 0.24 + unsloth 2026.8.22 + bitsandbytes 0.50 + datasets 5.0.1.
  • Key pitfalls (see the memory SOP for details):
    • Under WSL, the torch/lib/libhsa-runtime64.so* bundled with the PyTorch wheel must be deleted for the GPU to become visible (otherwise get_device_capability is None);
    • transformers ≥5 is incompatible with DeepSeek-OCR's old remote code (strict dataclass kv_lora_rank=None, naming differences for DeepseekV2Moe/MoE), so 4.57.x + a MoE alias patch must be used;
    • Unsloth UnslothVisionDataCollator only supports models with an image_processor; DeepSeek-OCR does not have one → a custom collator is required.

9.2 Training Input (Official Format, from HF Remote Code and the vLLM Processor)

  • images=[(patches, global_view)], where normalized global_view=(1,3,1024,1024) and, when empty, patches is (0,3,640,640); paired with images_seq_mask (True at <image> id 128815) and images_spatial_crop=[[1,1]].
  • Tokenizer single-token supervision: 0→18, 1→19; only the target interval is supervised (prompt/image/space/EOS are all -100). Each HotpotQA sample has approximately 2 positive/8 negative labels. The code defaults to w+=4,w-=1 for frequency balancing (the paper only discloses w+>w- and does not give specific values).
  • Generation must follow the training grammar digit space digit ... digit EOS; if only 0/1 are allowed and spaces are prohibited, the autoregressive context immediately drifts.

9.3 Empirical Results (2026-08-28)

The training set of 300 entries (seed=7) and evaluation set of 30 entries (seed=42) have zero ID overlap. Evaluation strictly uses the same grammar constraints; the table below shows the first 12 independent evaluation samples:

StageConfigurationResult
base (without LoRA)Same binary grammar constraintsexact 0/12, mean F1 0.139
LoRA intermediate milestone300 entries×3 epochs, lr 1e-4, old w+=2exact 1/12, mean F1 0.375
Single-sample overfitting100 steps, lr 1e-3, w+=4exact 1/1, F1 1.000; loss≈0 after 15 steps

Conclusion: The complete training/vision/supervision/autoregressive decoding loop is functional; the mean F1 of the 300-entry LoRA is approximately 2.7× that of the base model. The current default code has been further corrected with target-alignment self-tests, no EOS supervision, space grammar constraints, w+=4 class balancing, and correct gradient accumulation. Reproduction of the paper's main table is not yet claimed because the paper uses a larger HotpotQA training scale and evaluations such as Mind2Web/AppWorld/RULER.

9.4 Separation of Runtime and Training

  • Training: WSL ROCm, allowed to occupy the discrete GPU (this validation took from several minutes to approximately 1 hour).
  • Runtime (retrieval): llama-server uses the CPU-only build (<models>\llama.cpp-cpu\) by default and does not occupy the discrete GPU; CPU OCR/embedding has been empirically verified end to end, and isolated DSH headless validation with the official V4 Flash passed.
  • The following are still required for a complete reproduction at the level of the paper's main table:
    • The number of pure visual tokens output layer by layer within DeepEncoder (the current llama.cpp runtime only provides interface-level statistics);
    • Run the full HotpotQA training scale used in the paper and reproduce the Mind2Web/AppWorld/RULER/Mind2Web retrieval subset metrics;
    • Merge/quantize the LoRA into a CPU-runnable GGUF and integrate it with opticalLocatorEnabled for end-to-end validation of the Locate tool in DSH.