DISBench: DeepImageSearch Benchmark

DISBench is the first benchmark for context-aware image retrieval over visual histories. It contains 122 queries across 57 users and 109,467 photos, requiring multi-step reasoning over corpus-level context.

Download

Option A: Hugging Face (Recommended)

huggingface-cli download RUC-NLPIR/DISBench --local-dir DISBench

Option B: Manual Download

python download_images.py --photo-ids-path photo_ids --images-path images

File Structure

DISBench/
├── queries.jsonl              # 122 annotated queries
├── metadata/
│   └── {user_id}.jsonl        # Photo metadata per user
├── images/
│   └── {user_id}/
│       └── {photo_id}.jpg     # Photo files
├── photo_ids/
│   └── {user_id}.txt          # Photo IDs and hashes per user
└── download_images.py         # Image download script

Data Format

queries.jsonl

Each line is a JSON object representing one query:

{
    "query_id": "1",
    "user_id": "10287726@N02",
    "query": "Find photos from the musical performance identified by the blue and white event logo on site, where only the lead singer appears on stage.",
    "answer": ["7759256930", "7759407170", "7759295108", "7759433016"],
    "event_type": "intra-event"
}

Field	Type	Description
query_id	string	Unique query identifier
user_id	string	User whose photo collection to search
query	string	Natural language query (text-only)
answer	list[string]	Ground-truth target photo IDs
event_type	string	"intra-event" or "inter-event"

metadata/{user_id}.jsonl

Each line is a JSON object representing one photo's metadata:

{
    "photo_id": "4517621778",
    "metadata": {
        "taken_time": "2010-04-10 13:52:57",
        "longitude": -1.239802,
        "latitude": 51.754123,
        "accuracy": 16.0,
        "address": "Y, Cherwell Street, St Clement's, East Oxford, Oxford, Oxfordshire, England, OX4 1BQ, United Kingdom",
        "capturedevice": "Panasonic DMC-TZ5"
    }
}

Field	Type	Description
photo_id	string	Unique photo identifier
metadata.taken_time	string	Capture time in YY-MM-DD HH:MM:SS format
metadata.longitude	float	GPS longitude. Missing if unavailable.
metadata.latitude	float	GPS latitude. Missing if unavailable.
metadata.accuracy	float	GPS accuracy level. Missing if unavailable.
metadata.address	string	Reverse-geocoded address. Missing if unavailable.
metadata.capturedevice	string	Camera/device name. Missing if unavailable.

Note: Optional fields (longitude, latitude, accuracy, address, capturedevice) are omitted entirely when unavailable — they will not appear as keys in the JSON object.

images/{user_id}/{photo_id}.jpg

Photo files organized by user. Each user's collection contains approximately 2,000 photos accumulated chronologically from their photosets.

photo_ids/{user_id}.txt

Each line represents one photo ID and its hash on aws storage in the format {photo_id}\t{hash}:

1205732595	c45044fd7b5c9450b2a11adc6b42d

Field	Type	Description
photo_id	string	Unique photo identifier
hash	string	Hashed value of the photo on aws storage

Dataset Statistics

Statistic	Value
Total Queries	122
Intra-Event Queries	57 (46.7%)
Inter-Event Queries	65 (53.3%)
Total Users	57
Total Photos	109,467
Avg. Targets per Query	3.84
Avg. History Span	3.4 years
Query Retention Rate	6.1% (122 / 2,000 candidates)

Data Source

DISBench is constructed from YFCC100M, which preserves a hierarchical structure of users → photosets → photos. All images are publicly shared under Creative Commons licenses. Photoset boundaries are used during construction but are not provided to models during evaluation.

License

The DISBench dataset follows the Creative Commons licensing terms of the underlying YFCC100M data. Please refer to individual image licenses for specific usage terms.