README.md

This repository is a collection of resources on multimodal knowledge graph, including datasets, research papers and contests.

Content

• Content
• Multimodal Knowledge Graph Completion
• Multimodal Entity Aligment
• Multimodal Entity Linking
• Multimodal Named Entity Recognition
• Multimodal Event Extraction

Multimodal Knowledge Graph Completion

Title	Venue	Year	Code
Image-embodied Knowledge Representation Learning	IJICA	2016	Github
KG-BERT: BERT for Knowledge Graph Completion	arXiv	2019	Github
Multimodal Data Enhanced Representation Learning	IJCNN	2019	Github
Multi-modal Multi-relational Feature Aggregation Network for Medical Knowledge Representation Learning	ACM Multimedia	2020
Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective	ACM Multimedia	2021	Github
Endowing Language Models with Multimodal Knowledge Graph Representations	arXiv	2022	Github
Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion	TOMM	2022	Github
Contrastive Multi-Modal Knowledge Graph Representation Learning	TKDE	2022
IMKGA-SM: Interpretable Multimodal Knowledge Graph Answer Prediction via Sequence Modeling	arXiv	2022
MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning	Appl. Intell.	2022
MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion	EMNLP	2022	Github
OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport	NIPS	2022	Github
Multimodal Knowledge Learning for Named Entity Disambiguation	EMNLP-findings	2022
Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion	ACM Multimedia	2022
VISTA: Visual-Textual Knowledge Graph Representation Learning	EMNLP-findings	2023	Github
TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio	ACM Multimedia	2023	Github
MPKGAC: Multimodal Product Attribute Completion in E-commerce	WWW (Companion Volume)	2023	Github
Reviving the Context: Camera Trap Species Classification as Link Prediction on Multimodal Knowledge Graphs	CIKM	2023	Github
LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation	AAAI	2024
Zero-Shot Relational Learning for Multimodal Knowledge Graphs	arXiv	2024
MyGO: Discrete Modality Information as Fine-Grained Tokens for Multi-modal Knowledge Graph Completion	arXiv	2024
Un-leashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion	LREC-COLING	2024	Github

Multimodal Entity Aligment

Title	Venue	Year	Code
MMEA: Entity Alignment for Multi-modal Knowledge Graph	KSEM	2020	Github
Multi-modal entity alignment in hyperbolic space	Neurocomputing	2021
Visual Pivoting for (Unsupervised) Entity Alignment	AAAI	2021	Github
Multi-modal Siamese Network for Entity Alignment	KDD	2022	Github
MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph	Neurocomputing	2022
Multi-modal Contrastive Representation Learning for Entity Alignment	COLING	2022	Github
Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment	WWW	2023	Github
Leveraging multimodal features for knowledge graph entity alignment based on dynamic self-attention networks	Expert Systems with Applications	2023	Github
Probing the Impacts of Visual Context in Multimodal Entity Alignment	Data Sci. Eng.	2023
MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid	ACM Multimedia	2023	Github
Revisit and Outstrip Entity Alignment: A Perspective of Generative Models	arXiv	2023
Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment	ISWC	2023	Github
Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment	arXiv	2023	Github
Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths	arXiv	2023	Github

Multimodal Entity Linking

Title	Venue	Year	Code
Multimodal Named Entity Disambiguation for Noisy Social Media Posts	ACL	2018
Building a Multimodal Entity Linking Dataset From Tweets	LREC	2020	Github
Multimodal Entity Linking for Tweets	ECIR	2020	Github
Attention-Based Multimodal Entity Linking with High-Quality Images	DASFAA	2021
Weibo-MEL, Wikidata-MEL and Richpedia-MEL: Multimodal Entity Linking Benchmark Datasets	CCKS	2021	Github
Multimodal Entity Linking: A New Dataset and A Baseline	ACM Multimedia	2021	Github
Visual Entity Linking via Multi-modal Learning	Data Intell.	2022
WikiDiverse: A Multimodal Entity Linking Dataset with Diversified Contextual Topics and Entity Types	ACL	2022	Github
Multimodal Entity Linking with Gated Hierarchical Fusion and Contrastive Training	SIGIR	2022	Github
Multimodal Entity Linking with Mixed Fusion Mechanism	DASFAA	2022	Github
AMELI: Enhancing Multimodal Entity Linking with Fine-Grained Attributes	arXiv	2023
Multi-Grained Multimodal Interaction Network for Entity Linking	KDD	2023	Github
Benchmarking Diverse-Modal Entity Linking with Generative Models	ACL-findings	2023
Generative Multimodal Entity Linking	arXiv	2023	Github
Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities	ICCV	2023	Github
DRIN: Dynamic Relation Interactive Network for Multimodal Entity Linking	ACM Multimedia	2023	Github
MMEL: A Joint Learning Framework for Multi-Mention Entity Linking	UAI	2023	Github
S2abEL: A Dataset for Entity Linking from Scientific Tables	EMNLP	2023	Github
A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking	AAAI	2024	Github
DWE+: Dual-Way Matching Enhanced Framework for Multimodal Entity Linking		2024	Github
Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking	ACL-findings	2024	Github
OVEL: Large Language Model as Memory Manager for Online Video Entity Linking	arXiv	2024

Multimodal Named Entity Recognition

Title	Venue	Year	Code
Multimodal Named Entity Recognition for Short Social Media Posts	NAACL-HLT	2018
Adaptive Co-attention Network for Named Entity Recognition in Tweets	AAAI	2018
Visual Attention Model for Name Tagging in Multimodal Social Media	ACL	2018
Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognition	ICDAR	2019
Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer	ACL	2020	Github
Object-Aware Multimodal Named Entity Recognition in Social Media Posts With Adversarial Learning	IEEE Trans. Multim.	2021
RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER	AAAI	2021	Github
Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance	AAAI	2021	Github
MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts	ICME	2021	Github
PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition	DASFAA	2022
CWI: A multimodal deep learning approach for named entity recognition from social media using character, word and image features	Neural Comput. Appl.	2022
MAF: A General Matching and Alignment Framework for Multimodal Named Entity Recognition	WSDM	2022	Github
Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction	arXiv	2022	Github
ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition	NAACL-HLT	2022	Github
CAT-MNER: Multimodal Named Entity Recognition with Knowledge-Refined Cross-Modal Attention	ICME	2022
Query Prior Matters: A MRC Framework for Multimodal Named Entity Recognition	ACM Multimedia	2022
Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER	ACM Multimedia	2022	Github
Named Entity and Relation Extraction with Multi-Modal Retrieval	EMNLP	2022	Github
Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging	AAAI	2023	Github
A Novel Framework for Multimodal Named Entity Recognition with Multi-level Alignments	arXiv	2023
Grounded Multimodal Named Entity Recognition on Social Media	ACL	2023	Github
MPMRC-MNER: A Unified MRC framework for Multimodal Named Entity Recognition based Multimodal Prompt	CIKM	2023
MGICL: Multi-Grained Interaction Contrastive Learning for Multimodal Named Entity Recognition	CIKM	2023
Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction	ACM Multimedia	2023
Fine-Grained Multimodal Named Entity Recognition and Grounding with a Generative Framework	ACM Multimedia	2023	Github
MCG-MNER: A Multi-Granularity Cross-Modality Generative Framework for Multimodal NER with Instruction	ACM Multimedia	2023	Github
Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge	EMNLP-findings	2023	Github
In-context Learning for Few-shot Multimodal Named Entity Recognition	EMNLP-findings	2023
ESPVR: Entity Spans Position Visual Regions for Multimodal Named Entity Recognition	EMNLP-findings	2023

Multimodal Event Extraction

Title	Venue	Year	Code
Image Enhanced Event Detection in News Articles	AAAI	2020	Github
Cross-media Structured Common Space for Multimedia Event Extraction	ACL	2020
MERL: Multimodal Event Representation Learning in Heterogeneous Embedding Spaces	AAAI	2021
A Chinese Financial Event Extraction Dataset Constructed by Distant Supervision	IJCKG	2021	Github
CLIP-Event: Connecting Text and Images with Event Structures	CVPR	2022	Github
Multimedia Event Extraction From News With a Unified Contrastive Learning Framework	ACM Multimedia	2022	Github