Inferens Phi-3-Vision Secara Tempatan
February 1, 2026 ยท View on GitHub
Phi-3-vision-128k-instruct membolehkan Phi-3 bukan sahaja memahami bahasa, tetapi juga melihat dunia secara visual. Melalui Phi-3-vision-128k-instruct, kita boleh menyelesaikan pelbagai masalah visual, seperti OCR, analisis jadual, pengecaman objek, menerangkan gambar dan sebagainya. Kita boleh dengan mudah menyiapkan tugasan yang sebelum ini memerlukan latihan data yang banyak. Berikut adalah teknik dan senario aplikasi yang berkaitan yang dirujuk oleh Phi-3-vision-128k-instruct
0. Persediaan
Sila pastikan perpustakaan Python berikut telah dipasang sebelum digunakan (Python 3.10+ disyorkan)
pip install transformers -U
pip install datasets -U
pip install torch -U
Disyorkan menggunakan CUDA 11.6+ dan memasang flatten
pip install flash-attn --no-build-isolation
Buat Notebook baru. Untuk melengkapkan contoh, disyorkan anda buat kandungan berikut terlebih dahulu.
from PIL import Image
import requests
import torch
from transformers import AutoModelForCausalLM
from transformers import AutoProcessor
model_id = "microsoft/Phi-3-vision-128k-instruct"
kwargs = {}
kwargs['torch_dtype'] = torch.bfloat16
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype="auto").cuda()
user_prompt = '<|user|>\n'
assistant_prompt = '<|assistant|>\n'
prompt_suffix = "<|end|>\n"
1. Menganalisis imej dengan Phi-3-Vision
Kita mahu AI dapat menganalisis kandungan gambar kita dan memberikan penerangan yang berkaitan
prompt = f"{user_prompt}<|image_1|>\nCould you please introduce this stock to me?{prompt_suffix}{assistant_prompt}"
url = "https://g.foolcdn.com/editorial/images/767633/nvidiadatacenterrevenuefy2017tofy2024.png"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
generate_ids = model.generate(**inputs,
max_new_tokens=1000,
eos_token_id=processor.tokenizer.eos_token_id,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(generate_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False)[0]
Kita boleh mendapatkan jawapan yang berkaitan dengan menjalankan skrip berikut dalam Notebook
Certainly! Nvidia Corporation is a global leader in advanced computing and artificial intelligence (AI). The company designs and develops graphics processing units (GPUs), which are specialized hardware accelerators used to process and render images and video. Nvidia's GPUs are widely used in professional visualization, data centers, and gaming. The company also provides software and services to enhance the capabilities of its GPUs. Nvidia's innovative technologies have applications in various industries, including automotive, healthcare, and entertainment. The company's stock is publicly traded and can be found on major stock exchanges.
2. OCR dengan Phi-3-Vision
Selain menganalisis imej, kita juga boleh mengekstrak maklumat daripada imej. Ini adalah proses OCR yang sebelum ini memerlukan kita menulis kod yang kompleks untuk melengkapkannya.
prompt = f"{user_prompt}<|image_1|>\nHelp me get the title and author information of this book?{prompt_suffix}{assistant_prompt}"
url = "https://marketplace.canva.com/EAFPHUaBrFc/1/0/1003w/canva-black-and-white-modern-alone-story-book-cover-QHBKwQnsgzs.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
generate_ids = model.generate(**inputs,
max_new_tokens=1000,
eos_token_id=processor.tokenizer.eos_token_id,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(generate_ids,
skip_special_tokens=False,
clean_up_tokenization_spaces=False)[0]
Keputusannya adalah
The title of the book is "ALONE" and the author is Morgan Maxwell.
3. Perbandingan pelbagai imej
Phi-3 Vision menyokong perbandingan pelbagai imej. Kita boleh menggunakan model ini untuk mencari perbezaan antara imej-imej tersebut.
prompt = f"{user_prompt}<|image_1|>\n<|image_2|>\n What is difference in this two images?{prompt_suffix}{assistant_prompt}"
print(f">>> Prompt\n{prompt}")
url = "https://hinhnen.ibongda.net/upload/wallpaper/doi-bong/2012/11/22/arsenal-wallpaper-free.jpg"
image_1 = Image.open(requests.get(url, stream=True).raw)
url = "https://assets-webp.khelnow.com/d7293de2fa93b29528da214253f1d8d0/news/uploads/2021/07/Arsenal-1024x576.jpg.webp"
image_2 = Image.open(requests.get(url, stream=True).raw)
images = [image_1, image_2]
inputs = processor(prompt, images, return_tensors="pt").to("cuda:0")
generate_ids = model.generate(**inputs,
max_new_tokens=1000,
eos_token_id=processor.tokenizer.eos_token_id,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
Keputusannya adalah
The first image shows a group of soccer players from the Arsenal Football Club posing for a team photo with their trophies, while the second image shows a group of soccer players from the Arsenal Football Club celebrating a victory with a large crowd of fans in the background. The difference between the two images is the context in which the photos were taken, with the first image focusing on the team and their trophies, and the second image capturing a moment of celebration and victory.
Penafian:
Dokumen ini telah diterjemahkan menggunakan perkhidmatan terjemahan AI Co-op Translator. Walaupun kami berusaha untuk ketepatan, sila ambil maklum bahawa terjemahan automatik mungkin mengandungi kesilapan atau ketidaktepatan. Dokumen asal dalam bahasa asalnya harus dianggap sebagai sumber yang sahih. Untuk maklumat penting, terjemahan profesional oleh manusia adalah disyorkan. Kami tidak bertanggungjawab atas sebarang salah faham atau salah tafsir yang timbul daripada penggunaan terjemahan ini.