KnowPrompt:Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction

KnowPrompt is used for relational extraction tasks, injecting latent knowledge contained in relation labels into prompt construction with learnable virtual template words and answer words , and synergistically optimize their representation with structured constraints.

Model description

We take the first step to inject latent knowledge contained in relation labels into prompt construction，the knowledge extraction is then implemented with a Prompt-tuning model。The implementation is as follows：virtual template words around entities initialized using aggregate entity embeddings are used as learnable virtual template words to inject entity knowledge; Meanwhile, we leverage label to compute average embeddings as virtual answers words to inject relationship knowledge. In this structure, entities and relations are mutually constrained and virtual template and answer words should be contextually relevant, so we introduce synergistic optimization to correct virtual template and answer words.

Intended uses & limitations

This model is used for relationship extraction tasks, and the extracted information can be used for more downstream NLP tasks, such as: information retrieval, conversation generation and Q&A. Please refer to the code example for details on how to use it.

The relationship labels in the model training data are limited and can only generalize the relationships in the real world to a certain extent.

Training data

We adopt SemEval as the dataset

Dataset	# Train.	# Val.	# Test.	# Rel.
SemEval	6,507	1,493	2,717	19

Training procedure

Training

The training is divided into two phases, and the first phase performs collaborative optimization of virtual template words and answer words $\mathcal{J}=\mathcal{J}_{[\text {MASK }]}+\lambda \mathcal{J}_{\text {structured }},$

$\lambda$is the hyperparameter for weighing the two loss functions;The second stage optimizes all parameters with a smaller learning rate based on the optimized virtual template words and answer words, using only the loss function $\mathcal{J}_{texttt{[MASK]}}$to finetune the parameters for the language model.The hyperparameters are different for different datasets, as shown in the script file in the source code.Taking SemEval as an example, the hyperparameters are set as follows:

max_epochs=10
max_sequence_length=256
batch_size=16
learning_rate=3e-5
batch_size=16
t_lambda=0.001

Data Evaluation and Results

The results of the comparison with other models in standard settings are shown in the following table.

Methods	Precision
Fine-tuning	87.6
KnowBERT	89.1
MTB	89.5
PTR	89.9
KnowPrompt	90.2 (+0.3)
In low-resource settings，we performed the 8-, 16-, and 32-experiments.K instances of each class are sampled from the initial training and validation sets to form the training and validation sets for the FEW-shot. The results are as follows:
Split	Methods
-----	-----------
k=8	Fine-tuning
	GDPNet
	PTR
	KnowPrompt
k=16	Fine-tuning
	GDPNet
	PTR
	KnowPrompt
k=32	Fine-tuning
	GDPNet
	PTR
	KnowPrompt
As 𝐾 decreases from 32 to 8, the improvement in our KnowPrompt over the other three methods increases gradually.

BibTeX entry and citation info

@inproceedings{DBLP:conf/www/ChenZXDYTHSC22,
author    = {Xiang Chen and
               Ningyu Zhang and
               Xin Xie and
               Shumin Deng and
               Yunzhi Yao and
               Chuanqi Tan and
               Fei Huang and
               Luo Si and
               Huajun Chen},
  editor    = {Fr{\'{e}}d{\'{e}}rique Laforest and
               Rapha{\"{e}}l Troncy and
               Elena Simperl and
               Deepak Agarwal and
               Aristides Gionis and
               Ivan Herman and
               Lionel M{\'{e}}dini},
  title     = {KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization
               for Relation Extraction},
  booktitle = {{WWW} '22: The {ACM} Web Conference 2022, Virtual Event, Lyon, France,
               April 25 - 29, 2022},
  pages     = {2778--2788},
  publisher = {{ACM}},
  year      = {2022},
  url       = {https://doi.org/10.1145/3485447.3511998},
  doi       = {10.1145/3485447.3511998},
  timestamp = {Tue, 26 Apr 2022 16:02:09 +0200},
  biburl    = {https://dblp.org/rec/conf/www/ChenZXDYTHSC22.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```bash
 git clone https://www.modelscope.cn/jeno11/knowprompt_demo.git