magika-go

A revised Go implementation of Google's Magika - an AI-powered file type detection library.

Overview

This library uses the Magika deep learning model to identify file types based on content analysis. Unlike the official implementation, this version:

• Embeds assets at build time: The model and configuration files are compiled directly into the binary via go:embed, eliminating the need for external asset files at runtime.
• Uses the latest model only: This library exclusively uses the standard_v3_3 model, which is the latest and most accurate version. Older models are not supported.

Model

The currently used model is standard_v3_3 and configuration are synced from the upstream Magika repository and embedded at build time. To update to a newer model version, update the parameters in script/sync-magika and run the script or go generate ./....

Installation

Prerequisites

1. ONNX Runtime: Required for model inference

# Download and install ONNX Runtime to a local directory (./third_party)
script/vendor-onnxruntime

2. CGO: Must be enabled for ONNX Runtime bindings

Example

In order to use ONNX, certain flags and library paths must be set. Use script/run to automatically configure the environment variables and go flags to do so:

script/run example/main.go

Usage

package main

import (
    "fmt"
    "strings"

    "github.com/robherley/magika-go/pkg/magika"
)

func main() {
    // Create a scanner - no configuration needed!
    scanner, err := magika.NewScanner()
    if err != nil {
        panic(err)
    }

    // Scan content
    content := "fn main() { println!(\"Hello, world!\"); }"
    result, err := scanner.Scan(strings.NewReader(content), len(content))
    if err != nil {
        panic(err)
    }

    fmt.Printf("Detected: %s (%s)\n", result.Label, result.MimeType)
    // Output: Detected: rust (application/x-rust)
}

Project Structure

├── example/              # Example usage
├── internal/
│   └── onnx/             # ONNX Runtime CGO bindings
├── pkg/
│   └── magika/
│       ├── assets/       # Embedded model and config (via go:embed)
│       ├── scanner.go    # Main API
│       ├── features.go   # Feature extraction
│       └── config.go     # Model configuration
├── script/
│   ├── env               # Environment setup for CGO
│   ├── run               # Run example
│   ├── test              # Run tests
│   ├── sync-magika       # Sync assets from upstream
│   └── vendor-onnxruntime# Download ONNX Runtime
└── third_party/
    └── onnxruntime/      # ONNX Runtime library (gitignored)

Updating Assets

To sync the model and test data from the upstream Magika repository:

# Uses go generate
go generate ./...

# Or directly
script/sync-magika

The sync script pins to a specific commit for reproducibility. Override with MAGIKA_COMMIT environment variable and/or MAGIKA_MODEL if desired.

Testing

# Sync model and test data (gitignored)
script/sync-magika

# Run tests
script/test ./...

License

This project is licensed under the Apache License 2.0 - see the upstream Magika repository for model licensing details.