README.md

September 9, 2025 · View on GitHub

:smiley::heart_eyes::fire::fire: Segmentation-Based Parametric Painting :fire::fire::heart_eyes::smiley:

This repository contains a semantic-based painting optimization algorithm which aims to transform a given input image into a painting. The algorithm takes advantage of modern computer vision techniques, segmentation networks, and a differentiable renderer to generate results.

Project Website

Reference Image 1

Reference Image 2

Reference Image 3

What It Does

Reference Image 4

The algorithm aims to efficiently optimize a collection of stroke parameters to create a painting from a given image input. The method starts with an image and optimizes a parameter vector of strokes. When rendered on a canvas, it recreates the input image in the form of a painting. The approach is designed to efficiently manage any input size or aspect ratio. It divides the canvas into semantic areas using a segmentation network. This provides a higher control over the painting compared to previous optimization and neural methods.

  • Layered Painting & Patch-Based Approach:

    • Implements a coarse-to-fine progression.
    • Uses a layered approach, starting with a rough first coarse painting pass and progressively refines.
    • Uses patches of 128x128 size and batch-optimizes all stroke parameters in each patch.

  • Semantic Segmentation:

    • Provides precision over the granularity of each semantic zone in the artwork.
    • Ensures strokes remain within the designated semantic segment, improving the painting's accuracy.

  • Visual Working Memory:

    • Uses a dynamic attention maps system to focus on areas that need more attention.
    • Yields an organic feel to the painting.

  • Stroke Initialization, Renderer, and Blending:

    • Uses the stroke parameter representation and differentiable renderer.
    • Strokes are parameterized by a 13-dimensional tuple encoding various properties like start, middle, end points, radii, transparency, and RGB color.
    • Strokes are composited into patches with soft blending.

  • Optimization & Loss Functions:

    • Optimizes all stroke parameters in batch for efficiency.
    • Uses both pixel loss and perceptual loss to ensure accurate recreation of the input image.

How To Use

1. Requirements:

  • Python 3.9 or later
  • Install the required packages from requirements.txt using the following command (if using Ubuntu):
pip install -r requirements.txt
  • For Mac OS with Apple Silicon (M1/M2/M3) - OPTIMIZED VERSION:
pip install -r requirements_os.txt

Note: The requirements_os.txt has been updated with all necessary dependencies and is fully optimized for Apple Silicon Macs with MPS (Metal Performance Shaders) support.

  • macOS Setup (Automatic): The code now automatically detects and configures MPS device support. No manual changes needed!

  • Download the renderer and perceptual network here and store them in a folder under the main directory. *Update: also provided in folder model_checkpoints.

🚀 NEW PERFORMANCE OPTIMIZATIONS (2024)

This version includes significant performance and memory optimizations:

  • Mixed Precision Training: Reduces memory usage by ~40%
  • Early Termination: Stops optimization when loss plateaus (20-50% speedup)
  • Aggressive Patch Filtering: Only processes areas that need refinement
  • Memory Management: Automatic cleanup prevents memory accumulation
  • MPS Optimization: Full Apple Silicon support with Metal Performance Shaders
  • Image Resizing: Built-in support for processing smaller images

2. Command:

python main.py [options]

3. Key Arguments:

Core Settings:

  • --image_path: Path to input image (default: 'images/paris2.jpeg')
  • --style: Painting style - 'realistic', 'painterly', 'abstract', 'expressionist' (default: 'expressionist')
  • --canvas_size: Canvas size for patches (default: 128, use 16-32 for faster processing)
  • --save_dir: Directory to save results (default: './results')

🆕 New Optimization Arguments:

  • --use_mixed_precision: Enable mixed precision training (default: True)
  • --image_resize_factor: Resize input image by factor (e.g., 0.33 for 3x smaller, 0.5 for 2x smaller)
  • --upsample: Enable upsampling of small images (default: True, set to False to keep resized images small)
  • --min_image_size: Minimum image size for upsampling (default: 1800, set lower to avoid upsampling after resize)
  • --texturize: Use texturize feature (default: True, set to False for memory savings)
  • --save_animation: Save animation frames (default: True, set to False for memory savings)

Performance Settings:

  • --lr: Learning rate (default: 0.004, try 0.008 for faster convergence)
  • --aspect_ratio_downsample: Automatic downsampling factor for very large images (default: 3)
  • --global_loss: Global loss strategy (default: False)
  • --exp_name: Experiment name (default: 'exp_320')

Advanced Options:

  • --brush_type: Brush type - 'straight' or 'curved' (default: 'curved')
  • --canvas_color: Canvas background - 'black' or 'white' (default: 'black')
  • --use_transparency: Enable stroke transparency (default: False)
  • --stroke_init_mode: Stroke initialization - 'random' or 'grid' (default: 'grid')
  • --renderer_ckpt_path: Path to renderer checkpoint (default: './model_checkpoints/renderer.pkl')

Style Transfer Options:

  • --add_style: Enable CLIP-based style transfer (default: False)
  • --style_prompt: Text prompt for style (default: 'Starry Night by Vincent Van Gogh')
  • --style_transfer: Enable neural style transfer (default: False)
  • --style_img_path: Path to style reference image (default: None)

4. Examples:

Basic usage:

python main.py --image_path images/paris2.jpeg --style painterly

Optimized for memory/speed (recommended for Apple Silicon):

python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 16 \
  --image_resize_factor 0.33 \
  --use_mixed_precision True \
  --texturize False \
  --save_animation False

For maximum memory efficiency:

export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0
python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 8 \
  --image_resize_factor 0.5 \
  --lr 0.008

Ultra-fast testing (10x smaller image):

python main.py \
  --image_path images/paris2.jpeg \
  --style painterly \
  --canvas_size 8 \
  --image_resize_factor 0.1 \
  --upsample False \
  --lr 0.008

⚠️ Upsampling Fix: When using --image_resize_factor, the algorithm now automatically skips upsampling to preserve your intended image size. Use --upsample False to completely disable upsampling, or adjust --min_image_size to control the upsampling threshold.

🛠️ Troubleshooting

Memory Issues:

  • Use smaller --canvas_size (8, 16, 32)
  • Enable --image_resize_factor 0.5 or smaller
  • Set --texturize False and --save_animation False
  • Use export PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 on macOS

Performance Tips:

  • Higher --lr (0.008) for faster convergence
  • Early termination automatically stops when loss plateaus
  • Patch filtering reduces unnecessary computations
  • Mixed precision is enabled by default for optimal performance

Common Issues:

  • "MPS out of memory": Reduce image size or canvas size
  • "Module not found": Install missing packages with pip install <package>
  • Slow performance: Use optimized arguments from examples above

📊 Performance Improvements

Before vs After Optimizations:

MetricBeforeAfterImprovement
Memory Usage18.13 GB (failed)~12 GB (successful)~35% reduction
Level 0 TimeFailed~3.4 minutes✅ Completion
Early TerminationNone20-50% speedupNew feature
Patch ProcessingAll patchesError-filtered only~30-60% reduction
Apple SiliconManual setupAutomatic MPSPlug-and-play

Optimization Features:

  • 🧠 Smart Early Termination: Automatically stops when loss plateaus
  • 🎯 Patch Filtering: Only processes areas that need improvement
  • ⚡ Mixed Precision: Reduces memory usage without quality loss
  • 🍎 Apple Silicon Optimized: Native MPS support for M1/M2/M3 chips
  • 🔧 Memory Management: Automatic cleanup prevents memory leaks
  • 📏 Image Resizing: Process smaller images for faster results

Method Overview

The method uses various techniques and algorithms to produce a painting from an input image. Key components include:

  • Semantic Segmentation: Dividing the canvas into areas of interest.
  • Layered Painting: A coarse-to-fine progression.
  • Visual Working Memory: A dynamic attention maps system that focus on areas that need more attention.
  • Optimization & Loss Functions: Ensuring the painting closely resembles the input image.

Implementation Details:

  • Optimizer: Adam with a learning rate of 0.0002.
  • All painting layers are optimized for 300 iterations.
  • Canvas Background: Black.
  • Segmentation Network: Uses the DETR model with a CNN (ResNet) backbone followed by an encoder-decoder Transformer.

More Results:

Reference Image 5

Reference Image 6

Reference Image 7

References

  • [Huang, et al. 2019. Learning to Paint]
  • [Liu et al. 2021. Paint Transformer]
  • [Zou et al. 2020. Stylized Neural Painting]
  • [DETR: Carion, et al. 2020. End-to-End Object Detection with Transformers]

Citation

@misc{deguevara2023segmentationbased,
      title={Segmentation-Based Parametric Painting}, 
      author={Manuel Ladron de Guevara and Matthew Fisher and Aaron Hertzmann},
      year={2023},
      eprint={2311.14271},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      }

Feedback

For any queries or feedback related to the algorithm, please open an issue on GitHub or contact the authors directly.