Cortexbench

CORTEXBENCH is a collection of 17 EAI tasks drawn from 7 existing benchmarks, designed to assess the efficacy of pre-trained visual representations (PVRs) in supporting a variety of EAI applications. Each task includes a downstream policy learning paradigm and evaluation protocol adhering to community standards. By using fixed tasks and learning methods, CORTEXBENCH evaluates the contribution of PVRs and measures progress towards developing an artificial visual cortex for embodied intelligence. This benchmark conducts the largest and most comprehensive study of PVRs to-date.

CORTEXBENCH includes tasks from 7 diverse benchmarks with different combinations of observations, actions, and goals as well as different standard policy learning paradigms:

Benchmark Suite	Observation Space	Action Space	Goal Specification	Policy Learning
Adroit	RGB + proprio.	Continuous	-	IL
Metaworld	RGB + proprio.	Continuous	-	IL
DMControl	RGB + proprio.	Continuous	-	IL
Trifinger	RGB + proprio.	Continuous	Goal Image/Position	IL
ObjectNav	RGB + proprio.	Discrete	Object Category	IL
ImageNav	RGB	Discrete	Goal Image	RL
MobilePick	RGB + proprio.	Continuous	Goal Position	RL

Installation

To install CortexBench, please follow the instructions in ../INSTALLATION.md.

Datasets

Benchmarks from CortexBench use datasets described in DATASETS.md.