README.md

Overview

Knowledge graph embedding (KGE) is a method for representing symbolic facts in low-dimensional vector spaces, with the goal of projecting relations and entities into a continuous vector space. This approach enhances knowledge reasoning capabilities and facilitates application to downstream tasks.

We introduce DeltaKG (MIT License), a dynamic, PLM-based library for KGEs that equips with numerous baseline models, such as K-Adapter, CaliNet, KnowledgeEditor, MEND, and KGEditor, and supports a variety of datasets, including E-FB15k237, A-FB15k237, E-WN18RR, and A-WN18RR.

Model Architecture

Illustration of KGEditor for a) The external model-based editor, b) The additional parameter-based editor and c) KGEditor.

Installation

Step1 Download the basic code

git clone https://github.com/AnonymousForPapers/DeltaKG.git

Step2 Create a virtual environment using Anaconda and enter it

conda create -n deltakg python=3.8
conda activate deltakg

Step3 Install library

pip install -r requirements.txt

Data & Checkpoints Download

Data

The datasets that we used in our experiments are as follows,

• E-FB15k237

  This dataset is based on FB15k237 and a pre-trained language-model-based KGE. You can download the E-FB15k237 dataset from Google Drive.

For other datasets A-FB15k237, E-WN18RR, and A-WN18RR, you can also download from the above link.

Checkpoints

The checkpoints that we used in our experiments are as follows,

• PT_KGE_E-FB15k237

  This checkpoint is based on FB15k237 and a pre-trained language model. You can download the PT_KGE_E-FB15k237 checkpoint from Google Drive.

For other checkpoints PT_KGE_A-FB15k237, PT_KGE_E-WN18RR, and PT_KGE_A-WN18RR, you can also download from the above link.

The expected structure of files is:

DeltaKG
 |-- checkpoints  # checkpoints for tasks
 |-- datasets  # task data
 |    |-- FB15k237  # dataset's name
 |    |    |-- AddKnowledge  # data for add task, A-FB15k237
 |    |    |    |-- train.jsonl    
 |    |    |    |-- dev.jsonl     
 |    |    |    |-- test.jsonl     
 |    |    |    |-- stable.jsonl  
 |    |    |    |-- relation2text.txt  
 |    |    |    |-- relation.txt 
 |    |    |    |-- entity2textlong.txt   
 |    |    |-- EditKnowledge # data for edit task, E-FB15k237
 |    |    |    |-- ... #  consistent with A-FB15k237
 |    |-- WN18RR  # dataset's name
 |    |    |-- AddKnowledge  # data for add task, A-WN18RR
 |    |    |    |-- train.jsonl    
 |    |    |    |-- dev.jsonl     
 |    |    |    |-- test.jsonl     
 |    |    |    |-- stable.jsonl  
 |    |    |    |-- relation2text.txt  
 |    |    |    |-- relation.txt 
 |    |    |    |-- entity2text.txt   
 |    |    |-- EditKnowledge # data for edit task, E-WN18RR
 |    |    |    |-- ... #  consistent with A-WN18RR
 |-- models  # KGEditor and baselines
 |    |-- CaliNet
 |    |    |-- run.py    
 |    |    |-- ...
 |    |-- K-Adapter
 |    |-- KE  # KnowledgeEditor
 |    |-- KGEditor
 |    |-- MEND
 |-- resource  # image resource
 |-- scripts  # running scripts
 |    |-- CaliNet
 |    |    |-- CaliNet_FB15k237_edit.sh
 |    |    |-- ...
 |    |-- K-Adapter
 |    |-- KE  # KnowledgeEditor
 |    |-- KGEditor
 |    |-- MEND
 |-- src
 |    |-- data       # data process functions
 |    |-- models     # source code of models
 |-- README.md
 |-- requirements.txt
 |-- run.sh  #  script to quick start

How to run

• script

  • The script run.sh has three arguments -m, -d, and -t, which stand for model, dataset, and task.
    • -m: should be the name of a model in models (e.g. KGEditor, MEND, KE);
    • -d: should be either FB15k237 or WN18RR;
    • -t: should be either edit or add.

• Edit Task

  • To train the KGEditor model in the paper on the dataset E-FB15k237, run the command below.

        bash run.sh -m KGEditor -d FB15k237 -t edit
    

  • To train the KGEditor model in the paper on the dataset E-WN18RR, run the command below.

        bash run.sh -m KGEditor -d WN18RR -t edit
    

• Add Task

  • To train the KGEditor model in the paper on the dataset A-FB15k237, run the command below.

        bash run.sh -m KGEditor -d FB15k237 -t add
    

  • To train the KGEditor model in the paper on the dataset A-WN18RR, run the command below.

        bash run.sh -m KGEditor -d WN18RR -t add
    

Experiments

Up to now, baseline models include K-Adapter, CaliNet, KE, MEND, and KGEditor. The results of these models are as follows,

• E-FB15k237

  Model	$Succ@1$	$Succ@3$	$ER_{roc}$	$RK@3$	$RK_{roc}$
  Finetune	0.472	0.746	0.998	0.543	0.977
  Zero-Shot Learning	0.000	0.000	-	1.000	0.000
  K-Adapter	0.329	0.348	0.926	0.001	0.999
  CaliNet	0.328	0.348	0.937	0.353	0.997
  KE	0.702	0.969	0.999	0.912	0.685
  MEND	0.828	0.950	0.954	0.750	0.993
  KGEditor	0.866	0.986	0.999	0.874	0.635

• E-WN18RR

  Model	$Succ@1$	$Succ@3$	$ER_{roc}$	$RK@3$	$RK_{roc}$
  Finetune	0.758	0.863	0.998	0.847	0.746
  Zero-Shot Learning	0.000	0.000	-	1.000	0.000
  K-Adapter	0.638	0.752	0.992	0.009	0.999
  CaliNet	0.538	0.649	0.991	0.446	0.994
  KE	0.599	0.682	0.978	0.935	0.041
  MEND	0.815	0.827	0.948	0.957	0.772
  KGEditor	0.833	0.844	0.991	0.956	0.256

• A-FB15k237

  Model	$Succ@1$	$Succ@3$	$ER_{roc}$	$RK@3$	$RK_{roc}$
  Finetune	0.906	0.976	0.999	0.223	0.997
  Zero-Shot Learning	0.000	0.000	-	1.000	0.000
  K-Adapter	0.871	0.981	0.999	0.000	0.999
  CaliNet	0.714	0.870	0.997	0.034	0.999
  KE	0.648	0.884	0.997	0.926	0.971
  MEND	0.517	0.745	0.991	0.499	0.977
  KGEditor	0.796	0.923	0.998	0.899	0.920

• A-WN18RR

  Model	$Succ@1$	$Succ@3$	$ER_{roc}$	$RK@3$	$RK_{roc}$
  Finetune	0.997	0.999	0.999	0.554	0.996
  Zero-Shot Learning	0.000	0.000	-	1.000	0.000
  K-Adapter	0.898	0.978	0.999	0.002	0.999
  CaliNet	0.832	0.913	0.995	0.511	0.989
  KE	0.986	0.996	0.999	0.975	0.090
  MEND	0.999	1.0	0.999	0.810	0.987
  KGEditor	0.998	1.0	0.999	0.956	0.300

Other KG Representation Open-Source Projects

• OpenKE
• LibKGE
• CogKGE
• PyKEEN
• GraphVite
• Pykg2vec
• PyG
• CogDL
• NeuralKG
• KGxBoard