OpenVision Series

February 21, 2026 · View on GitHub

OpenVision (ICCV 2025)

A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning

OpenVision 2 (CVPR 2026)

A Family of Generative Pretrained Visual Encoders for Multimodal Learning

OpenVision 3

A Unified Visual Encoder for Both Understanding and Generation

🌐 OpenVision Project Page • Arxiv Logo Arxiv • 💻 Code • Hugging Face Logo OpenVision Collection

🌐 OpenVision 2 Project Page • Arxiv Logo Arxiv • 💻 Code • Hugging Face Logo OpenVision 2 Collection

🌐 OpenVision 3 Project Page • Arxiv Logo Arxiv • 💻 Code • Hugging Face Logo OpenVision 3 Collection

This repository contains the code for training and fine-tuning vision-language models based on the OpenVision framework. It now supports both the original contrastive + generative training (OpenVision), the simplified caption-only generative training (OpenVision 2), providing efficient and scalable approaches to multimodal learning on TPU infrastructure.

🚀 Recent Updates

January 2026

  • ✨ Released OpenVision 3: a unified visual encoder for both understanding and generation.
    • Please refer to the script for OpenVision 3 usage.
    • The full training code will be released soon.

September 2025

  • Released OpenVision 2: a simplified, generative-only version of OpenVision that removes the text encoder and contrastive loss, keeping only the captioning objective.
  • OpenVision 2 achieves:
    • 1.5–2× faster training
    • ~1.8× lower memory footprint
    • Supports scaling up to 1B+ parameters
    • Maintains or improves performance on multimodal benchmarks (OCR, TextVQA, ChartQA, MME, etc.).

May 2025

  • Released OpenVision models and training code.

🧩 OpenVision 2 at a Glance

  • Architecture: Vision Encoder (ViT) + Text Decoder (no text encoder)
  • Training Objective: Caption-only autoregressive generation
  • Key Optimizations:
    • Dual-stage CLIPA-style training (low → high resolution)
    • Synthetic captions from ReCap-DataComp-1B v2 (LLaMA-3-powered, conditioned on alt-text)
    • Visual token masking (keep ~25–35% tokens) for efficiency
  • Efficiency:
    • ViT-L/14 @224: Training time reduced from 83h → 57h, memory 24.5GB → 13.8GB
    • SoViT-400M/14 @384: Training time 241h → 121h, memory 27.4GB → 14.5GB
    • Enables larger batch size on TPU v4

📦 Core Features (Shared by OpenVision & OpenVision 2)

  • Optimized for Google Cloud TPU training
  • Supports various encoder architectures (ViT models of different sizes)
  • Implements efficient training strategies including model sharding
  • Supports pre-training and multi-stage fine-tuning
  • Compatible with CLIP-style vision-language training

📖 OpenVision (Original)

  • Training Objective: Contrastive (CLIP-style) + Generative (captioning)
  • Highlights:
    • Strong performance across multimodal benchmarks
    • Public release of both code and pretrained weights
    • Serves as the foundation for OpenVision 2

📊 Model Zoo (OpenVision 2)

OpenVision 2 Performance on Multimodal Benchmarks

MethodVision EncoderParamsResTextVQAChartQAOCRMMESEEDSQAGQAPOPE
OpenVisionL/14304M22457.713.9315148769.573.662.986.4
OpenVision 2L/14304M22459.013.7327146069.376.562.687.1
OpenVisionL/14304M33661.215.7339152570.575.163.787.2
OpenVision 2L/14304M33663.014.5357148670.177.563.087.7
OpenVisionSoViT-400M/14400M38462.416.1357149370.472.463.888.0
OpenVision 2SoViT-400M/14400M38464.315.0387147270.774.963.587.5
OpenVision 2H/14632M22460.213.5340147069.375.462.587.2
OpenVision 2H/14632M33663.416.3391147070.676.463.188.4
OpenVision 2H/14632M44865.618.1416149970.675.663.188.7
OpenVision 2g/141.01B22460.213.7338146969.375.062.686.9

Full collection: Hugging Face – OpenVision 2

🔧 How to Load Converted Vision Encoder

Note:
OpenVision2 checkpoints require the custom open_clip version included in this repository.
The upstream open_clip pip package is not compatible.

Example

import torch
# Use the OpenVision2 version of open_clip
from src.convert_upload.open_clip.factory import create_vision_encoder_and_transforms

hf_repo = "UCSC-VLAA/openvision2-vit-large-patch14-224-vision-only"

vision_encoder = create_vision_encoder_and_transforms(
    model_name=f"hf-hub:{hf_repo}"
)

vision_encoder.eval()
dummy_image = torch.ones((1, 3, 224, 224))
with torch.no_grad():
    _, patch_features = vision_encoder(dummy_image)

print("Patch feature shape:", patch_features.shape)

📊 Model Zoo (OpenVision)

Vision Encoder Performance on ImageNet-1K

ModelSizePatch SizeResolutionIN-1K Top-1JAX WeightPyTorch Weight
OpenVision-ViT-Tiny5M1616046.9%AvailableAvailable
OpenVision-ViT-Tiny5M1622449.6%AvailableAvailable
OpenVision-ViT-Tiny5M1638451.5%AvailableAvailable
OpenVision-ViT-Tiny5M816051.9%AvailableAvailable
OpenVision-ViT-Tiny5M822453.5%AvailableAvailable
OpenVision-ViT-Tiny5M838453.9%AvailableAvailable
OpenVision-ViT-Small22M1616063.5%AvailableAvailable
OpenVision-ViT-Small22M1622465.9%AvailableAvailable
OpenVision-ViT-Small22M1638467.1%AvailableAvailable
OpenVision-ViT-Small22M816067.3%AvailableAvailable
OpenVision-ViT-Small22M822468.6%AvailableAvailable
OpenVision-ViT-Small22M838468.5%AvailableAvailable
OpenVision-ViT-Base86M1616072.4%AvailableAvailable
OpenVision-ViT-Base86M1622473.9%AvailableAvailable
OpenVision-ViT-Base86M1638474.5%AvailableAvailable
OpenVision-ViT-Base86M816074.8%AvailableAvailable
OpenVision-ViT-Base86M822475.4%AvailableAvailable
OpenVision-ViT-Base86M838475.6%AvailableAvailable
OpenVision-ViT-Large307M148474.7%AvailableAvailable
OpenVision-ViT-Large307M1422478.5%AvailableAvailable
OpenVision-ViT-Large307M1433678.9%AvailableAvailable
OpenVision-ViT-Large307M884In progressAvailableAvailable
OpenVision-ViT-Large307M8224In progressAvailableAvailable
OpenVision-ViT-Large307M8336In progressAvailableAvailable
OpenVision-SoViT412M148476.2%AvailableAvailable
OpenVision-SoViT412M1422479.7%AvailableAvailable
OpenVision-SoViT412M1438479.9%AvailableAvailable
OpenVision-ViT-Huge632M148477.4%AvailableAvailable
OpenVision-ViT-Huge632M1422480.4%AvailableAvailable

* Results pending

Model Usage

With Our Customized OpenCLIP Tokenizer

import torch
import torch.nn.functional as F
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer

# === Local model loading ===
# Path to the local pretrained weights (.bin file) downloaded from HuggingFace or elsewhere
# ckpt_path = "/Path/to/your/local/ckpt"
# model, preprocess = create_model_from_pretrained(
#     model_name="openvision-vit-large-patch14-224",
#     pretrained=ckpt_path,  
#     device="cuda" if torch.cuda.is_available() else "cpu"
# )

model, preprocess = create_model_from_pretrained('hf-hub:UCSC-VLAA/openvision-vit-large-patch14-224')
tokenizer = get_tokenizer('hf-hub:UCSC-VLAA/openvision-vit-large-patch14-224')

image = Image.open(urlopen(
    'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))
image = preprocess(image).unsqueeze(0)

text = tokenizer(["a diagram", "a dog", "a cat", "a beignet"], context_length=model.context_length)

with torch.no_grad(), torch.cuda.amp.autocast():
    image_features = model.encode_image(image)
    text_features = model.encode_text(text)
    image_features = F.normalize(image_features, dim=-1)
    text_features = F.normalize(text_features, dim=-1)

    text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)

print("Label probs:", text_probs)  # prints: [[0., 0., 0., 1.0]]

Installation

Requirements

# Clone the repository
git clone https://github.com/UCSC-VLAA/OpenVision.git
cd openvision

bash setup.sh  DEVICE=tpu JAX_VERSION=0.4.38

TPU VM Setup

This codebase is optimized for TPU VM instances. To set up a TPU VM:

  1. Create a TPU VM instance on Google Cloud
  2. Install JAX with TPU support
  3. Clone the repository and install dependencies

Training

The training process is divided into two main phases:

  1. Pre-training: Training the model on large-scale datasets with smaller resolution
  2. Fine-tuning: Refining the model on higher resolution images

TPU Pod Training

Setting Up and Using the tpu_command.sh Tool

tpu_command.sh is a convenient tool for synchronizing files between different VMs and managing TPU resources.

Prerequisites

  • Add your SSH key to your Google Cloud project's metadata, and GCP will automatically propagate these SSH keys to all VMs

Basic Usage

  1. In your local terminal, run: . ./tpu_command.sh
  2. Enter tpu to launch the interactive menu
  3. Select the function you need:
    • ssh: Connect to TPU VM
    • sync dir: Synchronize directories to all VMs in the TPU Pod
    • kill job: Terminate running jobs
    • prepare env: Set up the TPU environment
    • check: Check if TPU core is occupied
    • rm tpu logs: Clear TPU logs
    • exit: Exit the tool

Workflow Example

  1. Select the sync dir function, then choose your TPU region, and follow the prompts to upload your code to the Pod
  2. Use prepare_env to set up the environment and synchronize all files
  3. Connect to the VM using ssh
  4. Start a tmux session on the VM
  5. Execute the train.sh script in the tmux session

Training Parameters

The main parameters for training can be adjusted in the scripts/project/openvision/train.sh script:

  • TPU_NAME: Name of your TPU VM instance
  • ZONE: Google Cloud zone where your TPU is located
  • PROJECT_ID: Your Google Cloud project ID
  • EXP_PATH: Google Cloud Storage path for experiment outputs
  • BATCH_FACTOR: Controls the effective batch size
  • PRE_TRAIN_EPOCH/FT_TRAIN_EPOCH: Number of epochs for pre-training and fine-tuning
  • PRE_LR/FT_LR: Learning rates for pre-training and fine-tuning
  • PRE_RES/FT_RES: Image resolutions for pre-training and fine-tuning
  • MODEL/TXT_MODEL: Architecture variants for image and text encoders

Training Commands

To start training with the default settings:

# Run pre-training
bash scripts/project/openvision/train.sh

The script automatically handles:

  1. Pre-training with low resolution (default: 84px/160px)
  2. Fine-tuning with medium resolution (default: 224px)
  3. Optional second fine-tuning with high resolution (default: 384px/336px)

Data Preparation

Prepare your data according to the format expected by the input pipeline. The training script expects the following datasets:

  1. Main training data (e.g., Recap-DataComp-1B dataset)
  2. Evaluation datasets:
    • ImageNet for classification
    • COCO for image-text retrieval
    • Flickr30k for additional retrieval evaluation

Preparing Data Paths

Update the following variables in the training script to point to your datasets:

export IN1K_DATA_DIR=gs://your-bucket/imagenet
export COCO_DATA_DIR=gs://your-bucket/coco
export Flickr_DATA_DIR=gs://your-bucket/flickr30k
export DATACOMP_PATH=gs://your-bucket/datacomp/shards

Customization

Model Architecture

The framework supports different model architectures which can be specified in the training script:

  • Vision models: 'Ti/16', 'S/16', 'B/16', 'L/14', 'So400m/14','H/14' (for Tiny, Small, Base, Large variants)
  • Text models: Same variants are available for text encoders
  • Text decoder: Controlled by the DECODER_NAME parameter

Training Configuration

Training configurations are defined in src/configs/openvision.py. You can:

  1. Modify sharding strategies for distributed training
  2. Change optimization parameters
  3. Adjust data augmentation and tokenization settings
  4. Configure evaluation metrics and checkpointing

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

Acknowledgement

The jax repo is built on big vision, and the pytorch repo is built on OpenCLIP. We've also borrowed some code from TIMM and MAE. Many thanks to the awesome works from the open-source community!

For computation, the OpenVision Project is mainly supported by TPU Research Cloud (TRC) program, and Google Cloud Research Credits program.

OpenVision 3 is also partly supported by the National Center for Transportation Cybersecurity and Resiliency (TraCR) (a U.S. Department of Transportation National University Transportation Center) headquartered at Clemson University, Clemson, South Carolina, USA (USDOT Grant #69A3552344812). Any opinions, findings, conclusions, and recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of TraCR, and the U.S. Government assumes no liability for the contents or use thereof.

Citation

If you find our work useful to your research and applications, please consider citing the paper and staring the repo :)

@article{li2025openvision,
  title   = {OpenVision: A Fully-Open, Cost-Effective Family of Advanced Vision Encoders for Multimodal Learning},
  author  = {Li, Xianhang and Liu, Yanqing and Tu, Haoqin and Zhu, Hongru and Xie, Cihang},
  journal = {arXiv preprint arXiv:2505.04601},
  year    = {2025}
}

@article{liu2025openvision,
  title = {OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning},
  author = {Liu, Yanqing and Li, Xianhang and Zhang, Letian and Wang, Zirui and Zheng, Zeyu and Zhou, Yuyin and Xie, Cihang},
  journal = {arXiv preprint arXiv:2509.01644},
  year = {2025}
}

@article{zhang2026openvision,
  title   = {OpenVision 3: A Family of Unified Visual Encoder for Both Understanding and Generation},
  author  = {Zhang, Letian and Ren, Sucheng and Liu, Yanqing and Li, Xianhang and Wang, Zeyu and Zhou, Yuyin and Yao, Huaxiu and Zheng, Zeyu and Nie, Weili and Liu, Guilin and Yu, Zhiding and Xie, Cihang},
  journal = {arXiv preprint arXiv:2601.15369},
  year    = {2026}
}