[ECCV 2026] InternVL3.5-8B Combined Scanpath Model

July 5, 2026 · View on GitHub

📄 Paper: https://arxiv.org/abs/2607.02083

A rank-32 LoRA adapter on top of OpenGVLab/InternVL3_5-8B-HF, fine-tuned to predict human free-viewing scanpaths as coordinate sequences on a 100x100 grid. Given an image, the model emits a sequence of fixation coordinates that approximate where a human observer would look. This is the combined model: it was trained jointly on five eye-tracking datasets (MIT, CAT, COCO, Daemons, Figrim). The weights bundled here are the best checkpoint.

Models

Two LoRA adapters are bundled, both on top of OpenGVLab/InternVL3_5-8B-HF:

model/combined_adapter/ — free-viewing scanpath prediction (rank 32), trained jointly on MIT, CAT, COCO, Daemons and Figrim. Used by run_eval.sh.
model/visual_search_adapter/ — goal-directed visual search (rank 8), trained on COCO-Search18 (target-present and target-absent trials); given a search target it predicts the search scanpath.

Choose which one to load with --adapter-path. The bundled 5-image sample and run_eval.sh target the free-viewing model; the visual-search model expects COCO-Search18-style inputs (a target category in the prompt), which are not bundled here.

Directory layout

internvl3_5_8b_combined_release/
├── README.md
├── LICENSE
├── requirements.txt
├── run_eval.sh
├── predict_scanpath.py                    # image -> scanpath inference
├── evaluate_vllm_unified.py               # evaluation / scoring script
├── configs/
│   ├── internvl3_5_8b_combined.yaml        # free-viewing LoRA SFT config
│   └── internvl3_5_8b_visual_search.yaml   # visual-search LoRA SFT config
├── model/
│   ├── combined_adapter/          # free-viewing scanpath (rank 32)
│   └── visual_search_adapter/     # COCO-Search18 visual search (rank 8)
└── data/
    ├── sample_MIT.json            # 75 entries for the 5 sample MIT images
    ├── images/                    # MIT_0985.jpg .. MIT_0989.jpg
    └── centerbias/
        └── MIT/                   # per-image center-bias priors (IG baseline)

Quick start

Install the dependencies:
```
pip install -r requirements.txt
```
Run the evaluation (needs 1 GPU). The first run downloads the base model from HuggingFace and merges the LoRA adapter into model/combined_adapter_merged/:
```
bash run_eval.sh
```
Results are written to eval_output/.

Predict a scanpath

predict_scanpath.py runs a single image through the model and prints the predicted scanpath — no ground truth or metrics needed.

Free viewing (8 fixations by default, uses model/combined_adapter):

python predict_scanpath.py --image path/to/image.jpg

Visual search (3 fixations by default, uses model/visual_search_adapter) — pass the target object:

python predict_scanpath.py --image path/to/office.jpg --mode search --target laptop

The scanpath is printed both on the model's 0–100 grid and in pixel coordinates. Use --num-fixations N to change the length, --output out.json to save the result, and --save-overlay overlay.png to render the scanpath on the image. The visual-search model was trained on the 18 COCO-Search18 targets: bottle, bowl, car, chair, clock, cup, fork, keyboard, knife, laptop, microwave, mouse, oven, potted plant, sink, stop sign, toilet, tv.

Running evaluations

run_eval.sh is a thin wrapper around evaluate_vllm_unified.py. To customise a run, call the script directly:

python evaluate_vllm_unified.py \
    --base-model OpenGVLab/InternVL3_5-8B-HF \
    --adapter-path model/combined_adapter \
    --val-json data/sample_MIT.json \
    --images-dir data \
    --pkl-dir data/centerbias \
    --output-dir eval_output \
    --metric-mode fast \
    --batch-size 64 --max-num-seqs 32 --max-model-len 4096 \
    --gpu-memory-utilization 0.90 \
    --skip-viz

On the first run the base model is downloaded from HuggingFace and the LoRA adapter is merged into model/combined_adapter_merged/ (reused on later runs). A GPU is required.

Metric modes

Two scoring modes are available via --metric-mode:

fast (default) — scores the ground-truth coordinate of each fixation by probing its digits with per-digit normalisation. Reports per-fixation Information Gain (IG) and log-likelihood (LL). Recommended.
grid — builds the full 100×100 next-fixation probability grid for every transition and normalises over it. Slower (many more forward passes) but also yields AUC and NSS alongside IG/LL, and can dump the grids with --save-grids.

Both modes probe the fixation coordinates digit-by-digit; the digit distribution is renormalised over the ten digit tokens (0–9) so that probability mass on non-digit tokens does not distort the score.

Key options

Flag	Meaning
`--metric-mode {fast,grid}`	Scoring mode (default `fast`).
`--base-model`	HuggingFace base model (`OpenGVLab/InternVL3_5-8B-HF`).
`--adapter-path`	LoRA adapter directory (merged into `*_merged/` on first use).
`--val-json`	Evaluation set in LlamaFactory format (see below).
`--images-dir`	Base directory that image paths in the JSON resolve against.
`--pkl-dir`	Center-bias priors (the IG baseline); omit to fall back to a synthetic Gaussian center bias.
`--output-dir`	Where the results JSON is written.
`--max-samples N`	Evaluate only the first N entries (quick checks).
`--batch-size` / `--max-num-seqs` / `--max-model-len`	vLLM throughput / context knobs.
`--gpu-memory-utilization`	vLLM GPU memory fraction.
`--skip-viz`	Skip per-sample visualisation output.
`--seed`	RNG seed.

Run python evaluate_vllm_unified.py --help for the complete list.

Evaluating on your own data

Pass a --val-json in LlamaFactory format — a list of entries, each with an image and a human/assistant turn pair:

[
  {
    "images": ["images/MIT_0987.jpg"],
    "conversations": [
      {"from": "human", "value": "<image>Analyze this image and predict a human eye movement scanpath ..."},
      {"from": "gpt",   "value": "[(53, 48), (72, 32), (38, 54)]"}
    ]
  }
]
``$

- \text{Coordinates} \text{are} \text{integers} \text{on} \text{a} **0–99 \text{grid}** (\text{the} \text{image} \text{is} \text{treated} \text{as} 100 \times 100),
  \text{written} \text{as} $(x, y)` with `x` = column, `y` = row. The `gpt` turn holds the
  ground-truth scanpath the model is scored against.
- Image paths resolve relative to `--images-dir` (so `images/MIT_0987.jpg` with
  `--images-dir data` reads `data/images/MIT_0987.jpg`).
- Center-bias priors are looked up at `<pkl-dir>/<DATASET>/<index>.pkl` derived
  from the image name `DATASET_index.jpg` (e.g. `MIT_0987.jpg` →
  `data/centerbias/MIT/0987.pkl`). Each pickle is a dict
  `{"centerbias": <2-D log-density array>}`. If a prior is missing, a synthetic
  Gaussian center bias is used instead.