HPT++

:rocket: Contributions

• We refine the knowledge generation process, producing and merging coarse-grained and fine-grained descriptions into multi-granularity descriptions for generating structured graphs with more discriminative semantics.
• We experiment with various methods to model structured information and re-design the relationship-driven attention re-weighting module, enabling re-weighting of attention maps according to relationships between key elements with a predefined ratio.
• To avoid over-fitting in downstream generalization tasks, we incorporate a consistency constraint between prompted and pre-trained text encoders to learn more robust representations. These improvements and comparisons to HPT are validated with extensive experiments.

📊 Results

Base-to-New Generalization

Results reported below show average accuracy for base and new classes across 11 recognition datasets averaged over 3 seeds. Please refer to our paper for more numerical results

Name	Base Accuracy	New Accuracy	Harmonic Mean
CLIP	69.34	74.22	71.70
CoOp	82.69	63.22	71.66
CoCoOp	80.47	71.69	75.83
MaPLe	82.28	75.14	78.55
HPT	84.32	76.86	80.23
HPT++	84.13	77.99	80.95

Cross-Dataset Evaluation

Results reported below show accuracy for the source dataset ImageNet and 4 ImageNet-variant datasets averaged over 3 seeds.

	ImNet	Caltech	Pets	Cars	Flowers	Food	Aircraft	SUN397	DTD	EuroSAT	UCF	Average
CLIP	71.51	93.70	89.14	64.51	68.71	85.30	18.47	64.15	41.92	46.39	66.55	63.88
CoCoOp	71.02	94.43	90.14	65.32	71.88	86.06	22.94	67.36	45.73	45.37	68.21	65.74
MaPLe	70.72	93.53	90.49	65.57	72.23	86.20	24.74	67.01	46.49	48.06	68.69	66.30
HPT	71.72	94.20	92.63	66.33	74.84	86.21	25.68	68.75	50.87	47.36	70.50	67.74
HPT++	71.81	94.02	92.16	65.55	72.43	86.34	28.60	68.78	51.02	50.76	70.53	68.02

Domain Generalization

Results reported below show accuracy for the source dataset ImageNet and the other 10 target datasets averaged over 3 seeds.

	ImageNet	ImageNetV2	ImageNet-S	ImageNet-A	ImageNet-R	Average
CLIP	66.73	60.83	46.15	47.77	73.96	57.17
CoOp	71.51	64.20	47.99	49.71	75.21	59.28
CoCoOp	71.02	64.07	48.75	50.63	76.18	59.90
MaPLe	70.72	64.07	49.15	50.90	76.98	60.26
HPT	71.72	65.25	49.36	50.85	77.38	60.71
HPT++	71.81	65.31	49.28	51.18	77.52	60.82

🗂️ Corpus Preparation

Our Multi-Granularity Knowledge Generation mechanism is implemented on Llama3-8B (Local deployment) and GPT-3.5-turbo (Calling the API), with the main experimental results obtained from Llama3. The generated corpus is available in the directories ./data/corpus (for Llama3) and ./data/corpus_gpt (for GPT-3.5). To customize the generation process, please check the relevant code in ./llms, as instructed below.

Llama3

Please follow the official Llama website instructions to deploy Llama3-8B locally and download the model files to ./llms/meta-llama.

./meta-llama
└── Meta-Llama-3-8B-Instruct
    ├── config.json
    ├── generation_config.json
    ├── gitattributes
    ├── LICENSE
    ├── model-00001-of-00004.safetensors
    ├── model-00002-of-00004.safetensors
    ├── model-00003-of-00004.safetensors
    ├── model-00004-of-00004.safetensors
    ├── model.safetensors.index.json
    ├── README.md
    ├── special_tokens_map.json
    ├── tokenizer_config.json
    ├── tokenizer.json
    └── USE_POLICY.md

Run the following script to generate the corpus under ./data/corpus folder.

python test_llama.py

GPT-3.5

Please enter your API_KEY and ENDPOINT for GPT-3.5-turbo in ./llama/test_gpt.py. Run the following script to generate the corpus under ./data/corpus_gpt folder.

python test_gpt.py

This project defaults to using the corpus generated by Llama3. If you need to switch to the corpus generated by GPT-3.5, please modify the path in the ./trainer/hpt_plus.py file.

🧪 Training and Evaluation

The training and evaluation scripts are consistent with those used in HPT. For detailed instructions on training and evaluation, please refer to RUN.md, ensuring that the script root is updated from ./scripts/hpt to ./scripts/hpt_plus.