Inference Phi-3-Vision lokalt
February 1, 2026 · View on GitHub
Phi-3-vision-128k-instruct gør det muligt for Phi-3 ikke kun at forstå sprog, men også at se verden visuelt. Gennem Phi-3-vision-128k-instruct kan vi løse forskellige visuelle opgaver, såsom OCR, tabelanalyse, objektgenkendelse, beskrive billeder osv. Vi kan nemt udføre opgaver, som tidligere krævede omfattende datatræning. Nedenfor er relaterede teknikker og anvendelsesscenarier, som Phi-3-vision-128k-instruct refererer til.
0. Forberedelse
Sørg for, at følgende Python-biblioteker er installeret inden brug (Python 3.10+ anbefales)
pip install transformers -U
pip install datasets -U
pip install torch -U
Det anbefales at bruge CUDA 11.6+ og installere flatten
pip install flash-attn --no-build-isolation
Opret en ny Notebook. For at gennemføre eksemplerne anbefales det, at du først opretter følgende indhold.
from PIL import Image
import requests
import torch
from transformers import AutoModelForCausalLM
from transformers import AutoProcessor
model_id = "microsoft/Phi-3-vision-128k-instruct"
kwargs = {}
kwargs['torch_dtype'] = torch.bfloat16
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, torch_dtype="auto").cuda()
user_prompt = '<|user|>\n'
assistant_prompt = '<|assistant|>\n'
prompt_suffix = "<|end|>\n"
1. Analyser billedet med Phi-3-Vision
Vi ønsker, at AI skal kunne analysere indholdet i vores billeder og give relevante beskrivelser
prompt = f"{user_prompt}<|image_1|>\nCould you please introduce this stock to me?{prompt_suffix}{assistant_prompt}"
url = "https://g.foolcdn.com/editorial/images/767633/nvidiadatacenterrevenuefy2017tofy2024.png"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
generate_ids = model.generate(**inputs,
max_new_tokens=1000,
eos_token_id=processor.tokenizer.eos_token_id,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(generate_ids,
skip_special_tokens=True,
clean_up_tokenization_spaces=False)[0]
Vi kan få de relevante svar ved at køre følgende script i Notebook
Certainly! Nvidia Corporation is a global leader in advanced computing and artificial intelligence (AI). The company designs and develops graphics processing units (GPUs), which are specialized hardware accelerators used to process and render images and video. Nvidia's GPUs are widely used in professional visualization, data centers, and gaming. The company also provides software and services to enhance the capabilities of its GPUs. Nvidia's innovative technologies have applications in various industries, including automotive, healthcare, and entertainment. The company's stock is publicly traded and can be found on major stock exchanges.
2. OCR med Phi-3-Vision
Ud over at analysere billedet kan vi også udtrække information fra billedet. Dette er OCR-processen, som vi tidligere skulle skrive kompleks kode for at gennemføre.
prompt = f"{user_prompt}<|image_1|>\nHelp me get the title and author information of this book?{prompt_suffix}{assistant_prompt}"
url = "https://marketplace.canva.com/EAFPHUaBrFc/1/0/1003w/canva-black-and-white-modern-alone-story-book-cover-QHBKwQnsgzs.jpg"
image = Image.open(requests.get(url, stream=True).raw)
inputs = processor(prompt, image, return_tensors="pt").to("cuda:0")
generate_ids = model.generate(**inputs,
max_new_tokens=1000,
eos_token_id=processor.tokenizer.eos_token_id,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(generate_ids,
skip_special_tokens=False,
clean_up_tokenization_spaces=False)[0]
Resultatet er
The title of the book is "ALONE" and the author is Morgan Maxwell.
3. Sammenligning af flere billeder
Phi-3 Vision understøtter sammenligning af flere billeder. Vi kan bruge denne model til at finde forskellene mellem billederne.
prompt = f"{user_prompt}<|image_1|>\n<|image_2|>\n What is difference in this two images?{prompt_suffix}{assistant_prompt}"
print(f">>> Prompt\n{prompt}")
url = "https://hinhnen.ibongda.net/upload/wallpaper/doi-bong/2012/11/22/arsenal-wallpaper-free.jpg"
image_1 = Image.open(requests.get(url, stream=True).raw)
url = "https://assets-webp.khelnow.com/d7293de2fa93b29528da214253f1d8d0/news/uploads/2021/07/Arsenal-1024x576.jpg.webp"
image_2 = Image.open(requests.get(url, stream=True).raw)
images = [image_1, image_2]
inputs = processor(prompt, images, return_tensors="pt").to("cuda:0")
generate_ids = model.generate(**inputs,
max_new_tokens=1000,
eos_token_id=processor.tokenizer.eos_token_id,
)
generate_ids = generate_ids[:, inputs['input_ids'].shape[1]:]
response = processor.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]
Resultatet er
The first image shows a group of soccer players from the Arsenal Football Club posing for a team photo with their trophies, while the second image shows a group of soccer players from the Arsenal Football Club celebrating a victory with a large crowd of fans in the background. The difference between the two images is the context in which the photos were taken, with the first image focusing on the team and their trophies, and the second image capturing a moment of celebration and victory.
Ansvarsfraskrivelse:
Dette dokument er blevet oversat ved hjælp af AI-oversættelsestjenesten Co-op Translator. Selvom vi bestræber os på nøjagtighed, bedes du være opmærksom på, at automatiserede oversættelser kan indeholde fejl eller unøjagtigheder. Det oprindelige dokument på dets oprindelige sprog bør betragtes som den autoritative kilde. For kritisk information anbefales professionel menneskelig oversættelse. Vi påtager os intet ansvar for misforståelser eller fejltolkninger, der opstår som følge af brugen af denne oversættelse.