Quantizing Phi-3.5 using Apple MLX Framework

MLX is an array framework for machine learning research on Apple silicon, brought to you by Apple machine learning research.

MLX is designed by machine learning researchers for machine learning researchers. The framework is intended to be user-friendly, but still efficient to train and deploy models. The design of the framework itself is also conceptually simple. We intend to make it easy for researchers to extend and improve MLX with the goal of quickly exploring new ideas.

LLMs can be accelerated in Apple Silicon devices through MLX, and models can be run locally very conveniently.

Now Apple MLX Framework supports quantization conversion of Phi-3.5-Instruct(Apple MLX Framework support), Phi-3.5-Vision(MLX-VLM Framework support) support**), and Phi-3.5-MoE(Apple MLX Framework support). Let's try it next:

Phi-3.5-Instruct

python -m mlx_lm.convert --hf-path microsoft/Phi-3.5-mini-instruct -q

Phi-3.5-Vision

python -m mlxv_lm.convert --hf-path microsoft/Phi-3.5-vision-instruct -q

Phi-3.5-MoE

python -m mlx_lm.convert --hf-path microsoft/Phi-3.5-MoE-instruct  -q

🤖 Samples for Phi-3.5 with Apple MLX

Labs	Introduce	Go
🚀 Lab-Introduce Phi-3.5 Instruct	Learn how to use Phi-3.5 Instruct with Apple MLX framework	Go
🚀 Lab-Introduce Phi-3.5 Vision (image)	Learn how to use Phi-3.5 Vision to analyze image with Apple MLX framework	Go
🚀 Lab-Introduce Phi-3.5 Vision (moE)	Learn how to use Phi-3.5 MoE with Apple MLX framework	Go

Resources

1. Learn about Apple MLX Framework https://ml-explore.github.io/mlx/

2. Apple MLX GitHub Rep https://github.com/ml-explore

3. MLX-VLM GitHub Repo https://github.com/Blaizzy/mlx-vlm