AI/ML Benchmark Algorithms & Methodology

November 24, 2025 · View on GitHub

Summary Table

Test Name	Model	Primary Focus	Performance Metric
LLM Inference	TinyLlama-1.1B or Phi-3-mini (4-bit)	CPU matrix operations, memory bandwidth	tokens/sec, time to first token
Image Classification	MobileNetV2 or SqueezeNet	CNN operations, floating-point performance	inference time (ms), throughput (images/sec)
Object Detection	YOLOv8n-nano or YOLO-Fastest	Real-time processing, mixed precision ops	inference latency, FPS capability
Text Embedding Generation	all-MiniLM-L6-v2	Transformer efficiency, vector operations	embeddings/sec, batch processing speed
Speech-to-Text	Whisper-tiny or Vosk small model	Sequence processing, audio feature extraction	Real-time factor (RTF), processing speed

AI/ML benchmarking evaluates the computational capabilities of a device for artificial intelligence and machine learning workloads. These tests measure how effectively the device can execute neural network inference operations, which are increasingly important for modern applications including natural language processing, computer vision, and audio processing.

The AI/ML benchmarks focus on:

Neural network inference performance across different model architectures
Memory bandwidth utilization for model parameters and intermediate tensors
Floating-point and integer operation efficiency for neural operations
Real-time processing capabilities for interactive applications
Power efficiency for mobile AI workloads

Performance Aspects Evaluated

AI/ML benchmark tests evaluate multiple dimensions of neural network performance:

Matrix Operations Performance: Measures the efficiency of matrix multiplications, which form the core of most neural network operations
Memory Bandwidth Utilization: Tests how effectively the system can feed data to neural network operations
Quantized Operation Efficiency: Evaluates performance with reduced precision operations (INT8, INT4) commonly used in mobile AI
Model Architecture Performance: Assesses different neural network architectures (CNNs, Transformers, RNNs)
Batch Processing Efficiency: Tests performance with different batch sizes for various use cases
Latency vs Throughput Trade-offs: Evaluates performance for both real-time and batch processing scenarios

Test Selection Rationale

The selected AI/ML tests were chosen based on their real-world relevance and computational diversity:

Real-world Relevance: Each test represents common AI tasks found in mobile applications, from image recognition to voice assistants
Computational Diversity: Tests cover different neural network architectures (CNNs, Transformers, YOLO) and operation types
Resource Utilization Patterns: Tests vary in their memory, compute, and power requirements to provide comprehensive coverage
Measurable Performance Differences: Tests are designed to show meaningful performance differences across different hardware configurations

2. Test List

AI/ML Test 1: LLM Inference (llama.cpp)

Model Used: TinyLlama-1.1B (~600MB) or Phi-3-mini (3.8B quantized to 4-bit ~2GB) The test uses llama.cpp for efficient LLM inference on CPU. The implementation leverages quantized models to reduce memory requirements while maintaining performance. The test runs a standard prompt to measure text generation performance.

Complexity Analysis: O(n²) for attention mechanism, O(n) for feed-forward operations The computational complexity is dominated by the self-attention mechanism which scales quadratically with sequence length. The space complexity is O(n) for key-value cache storage during generation.

Dataset/Workload Details:

Input: Standard prompt of 100-200 tokens
Generation: 128 new tokens
Model size: 1.1B or 3.8B parameters (quantized to 4-bit)
Quantization: 4-bit GGUF format for memory efficiency
Threading model: Uses llama.cpp's built-in multi-threading

Measurements Captured:

Tokens per second (generation speed)
Time to first token (latency)
Memory usage during inference
Context switching overhead
Power consumption during processing

Hardware Behavior Targeted:

CPU cache efficiency for model parameters
Memory bandwidth for tensor operations
Integer arithmetic for quantized operations
Multi-core scaling for parallel inference

Notes on Deterministic Behavior: Results are deterministic for the same model and input, though generation may vary based on sampling parameters.

AI/ML Test 2: Image Classification (ONNX Runtime)

Model Used: MobileNetV2 or SqueezeNet (~5-10MB) The test implements image classification using ONNX Runtime to ensure cross-platform compatibility. The model processes standard test images to classify them into predefined categories. The implementation uses optimized operators for mobile performance.

Complexity Analysis: O(n) where n is the number of pixels in the input image The complexity is primarily determined by the number of convolution operations, which scales linearly with input size for fixed model architecture. The space complexity is O(n) for intermediate feature maps.

Dataset/Workload Details:

Input: Standard test images (224×224 RGB)
Model: MobileNetV2 or SqueezeNet
Batch size: 1 for latency testing, up to 8 for throughput
Precision: FP16 or INT8 quantized models
Threading model: ONNX Runtime multi-threading

Measurements Captured:

Inference time per image (latency)
Images processed per second (throughput)
Memory usage during inference
Peak power consumption
Accuracy validation against reference

Hardware Behavior Targeted:

Floating-point performance for CNN operations
Memory bandwidth for feature map storage
SIMD instruction utilization for convolutions
Cache efficiency for filter weights

Notes on Deterministic Behavior: Results are deterministic for the same input image and model, with consistent classification outputs.

AI/ML Test 3: Object Detection (Lightweight YOLO)

Model Used: YOLOv8n-nano or YOLO-Fastest (~6-10MB) The test implements real-time object detection using optimized YOLO models. The implementation processes standard test images to detect and classify multiple objects simultaneously. The test measures both detection accuracy and processing speed.

Complexity Analysis: O(n) where n is the number of pixels in the input image YOLO performs detection in a single forward pass, making it more efficient than region proposal-based methods. The space complexity is O(n) for feature maps and detection outputs.

Dataset/Workload Details:

Input: Standard test images (416×416 or 640×640)
Model: YOLOv8n-nano or YOLO-Fastest
Detection classes: 80 COCO classes
Precision: FP16 or INT8 quantized models
Threading model: Optimized for real-time inference

Measurements Captured:

Inference latency (ms)
Frames per second capability
Detection accuracy (mAP)
Memory usage during inference
Power consumption during processing

Hardware Behavior Targeted:

Real-time processing capabilities
Mixed precision operation efficiency
Memory bandwidth for multi-scale feature maps
CPU utilization for detection post-processing

Notes on Deterministic Behavior: Results are deterministic for the same input image and model, with consistent detection outputs.

AI/ML Test 4: Text Embedding Generation

Model Used: all-MiniLM-L6-v2 (~80MB) via ONNX The test generates semantic embeddings for text inputs using a transformer-based model. The implementation processes sentences to create fixed-dimensional vector representations that capture semantic meaning. The test measures both single and batch processing performance.

Complexity Analysis: O(n²) for attention mechanism where n is the sequence length The self-attention mechanism scales quadratically with sequence length. The space complexity is O(n) for attention matrices and intermediate representations.

Dataset/Workload Details:

Input: Standard sentences of varying lengths (10-100 tokens)
Model: all-MiniLM-L6-v2 transformer
Batch size: 1 to 32 for batch processing tests
Precision: FP16 or INT8 quantized models
Threading model: ONNX Runtime multi-threading

Measurements Captured:

Embeddings generated per second
Batch processing efficiency
Memory usage during inference
Vector similarity validation
Processing latency per sentence

Hardware Behavior Targeted:

Transformer architecture efficiency
Vector operation performance
Memory bandwidth for attention computations
Cache efficiency for model parameters

Notes on Deterministic Behavior: Results are deterministic for the same input text, with consistent embedding vectors.

AI/ML Test 5: Speech-to-Text (Whisper Tiny)

Model Used: Whisper-tiny (~75MB) or Vosk small model (~40MB) The test performs automatic speech recognition using transformer-based models. The implementation processes audio clips to transcribe speech to text. The test measures both accuracy and processing speed for real-time applications.

Complexity Analysis: O(n²) for attention mechanism where n is the audio sequence length Audio processing involves converting time-domain signals to spectrograms, followed by transformer processing. The space complexity is O(n) for intermediate representations.

Dataset/Workload Details:

Input: 10-second audio clips at 16kHz
Model: Whisper-tiny or Vosk small model
Audio format: 16-bit PCM, mono
Precision: FP16 or INT8 quantized models
Threading model: Optimized for audio processing pipeline

Measurements Captured:

Real-time factor (RTF) - processing speed vs. audio duration
Word error rate (accuracy validation)
Memory usage during processing
Power consumption during transcription
Latency for streaming scenarios

Hardware Behavior Targeted:

Sequence processing capabilities
Audio feature extraction performance
Transformer attention efficiency
Memory bandwidth for audio processing

Notes on Deterministic Behavior: Results are deterministic for the same audio input, with consistent transcriptions.

3. Scaling & Load Adjustment

Rules for Adapting Workload to Device Tier

The benchmark automatically adjusts workload parameters based on device performance tier to ensure meaningful results across all device classes:

Slow Tier Devices (entry-level smartphones, budget tablets):

LLM Inference: Use TinyLlama-1.1B with 4-bit quantization
Image Classification: MobileNetV2 with 8-bit quantization
Object Detection: YOLOv8n-nano with 8-bit quantization
Text Embedding: all-MiniLM-L6-v2 with 8-bit quantization
Speech-to-Text: Vosk small model (40MB)

Mid Tier Devices (mid-range smartphones, mainstream tablets):

LLM Inference: Phi-3-mini with 4-bit quantization
Image Classification: SqueezeNet with 8-bit quantization
Object Detection: YOLOv8n-nano with 4-bit quantization
Text Embedding: all-MiniLM-L6-v2 with 4-bit quantization
Speech-to-Text: Whisper-tiny with 8-bit quantization

Flagship Tier Devices (high-end smartphones, premium tablets):

LLM Inference: Phi-3-mini with 4-bit quantization, larger context window
Image Classification: SqueezeNet with 4-bit quantization
Object Detection: YOLOv8n-nano with 4-bit quantization, higher resolution
Text Embedding: all-MiniLM-L6-v2 with 4-bit quantization
Speech-to-Text: Whisper-tiny with 4-bit quantization

Model Optimization Strategies

Quantization Levels:

8-bit INT8: For slow tier devices to reduce memory usage
4-bit INT4: For mid and flagship tier devices for optimal performance
FP16: For devices with specialized FP16 compute units

Model Download Strategy:

Models downloaded on-demand rather than bundled with app
"Quick Test" option: 1-2 models for fast benchmark
"Full Suite" option: All 5 models for comprehensive testing
Automatic selection based on available storage space

Dynamic Warm-up Rules

To ensure accurate measurements and account for system optimizations:

Model Loading Warm-up:

Load model into memory before timing begins
Perform 2-3 inference warm-up runs to prime caches
Monitor performance stabilization before starting timed measurements

Memory Pre-allocation:

Pre-allocate all required tensors before timing
Clear system caches before benchmark to ensure consistency
Monitor memory pressure during testing

Iteration Count for Each Benchmark: Each AI/ML benchmark test is executed 3 times for statistical validity, and the results are averaged to produce the final score. The median result is used for scoring to reduce the impact of outliers.

Single-Model Tests:

Each test runs 3 times with warm-up iterations before timing
Results are averaged after removing outliers (values >15% deviation from median)
Minimum run time of 2 seconds per iteration to ensure measurement accuracy

Batch Processing Tests:

Each test runs 3 times with warm-up iterations before timing
Results are averaged after removing outliers (values >15% deviation from median)
Minimum run time of 3 seconds per iteration for stable batch performance

4. Performance Metrics

Inference Performance

Inference performance is the primary metric for all AI/ML benchmark tests, measured from the start of the actual model computation to completion. The system captures:

Inference time: Total time to process a single input
Throughput: Number of inputs processed per second
Latency: Time to first result (important for interactive applications)
Real-time factor: Processing speed relative to input duration (for audio)

Accuracy Validation

To ensure model execution is correct:

Output validation: Compare results against reference outputs
Similarity metrics: Use cosine similarity for embeddings
Word error rate: For speech-to-text accuracy measurement
Top-k accuracy: For classification task validation

Resource Utilization

Resource metrics measure the computational efficiency of AI/ML workloads:

Memory usage: Peak and average memory consumption during inference
Power consumption: Energy usage during processing (where available)
CPU utilization: Core usage during inference operations
Thermal impact: Temperature changes during sustained inference

5. Scoring Conversion

Formula for Converting Raw Performance to Normalized Score

The raw performance metrics are converted to normalized scores using:

$ \text{Normalized} \text{Score} = (\text{Device} \text{Performance} / \text{Baseline} \text{Performance}) \times 10 $

For time-based metrics (lower is better): $ \text{Normalized} \text{Score} = (\text{Baseline} \text{Time} / \text{Device} \text{Time}) \times 100 $

For throughput-based metrics (higher is better): $ \text{Normalized} \text{Score} = (\text{Device} \text{Throughput} / \text{Baseline} \text{Throughput}) \times 100 $

Baseline Reference Device Detail

The baseline device used for normalization is the Google Pixel 6 (2021) with:

CPU: Google Tensor (2×3.0 GHz Cortex-X1 + 2×2.85 GHz Cortex-A78 + 4×2.1 GHz Cortex-A55)
Neural Processing: Google Tensor Processing Unit
Memory: 8GB LPDDR5
Baseline performance values established through extensive testing