Low-Resource Vision Challenges for Foundation Models (CVPR 2024)

Yunhua Zhang, Hazel Doughty, Cees G.M. Snoek

Dataset & Website

• Website
• Dataset

Zero-Shot Transfer Evaluation

Remember to change the data paths in them.

• Circuit Diagram Classification: python zero-shot/circuit.py

• Historic Map Retrieval: python zero-shot/historic_maps.py

• Mechanical Drawing Retrieval: python zero-shot/mechanical_drawings.py

Linear Probe Evaluation

• Circuit Diagram Classification: python linear_probe/circuit.py

Training Our Baselines

Before Training

Generating Data for Data Scarcity

• Using this repository: stable diffusion
• Augmenting circuit diagrams as an example: put the python script img2img_circuit.py into its directory, modify the data path in it, and run it by python img2img_circuit.py --strength 0.3 --outdir /path-to-output/ --n_samples 5 to augment the data, here set the strength as 0.6 for the label-breaking augmentations.
• The augmentations for historic maps and mechanical drawings are the same

Training

First change the data paths in dataloaders and training scripts.

• Circuit Diagram Classification: python scripts/train_circuit.py

• Historic Map Retrieval: python scripts/train_map.py

• Mechanical Drawing Retrieval: python scripst/train_mechanical.py

The optimal hyperparameters listed in parser.add_argument() could be different on different machines.