A New Entity-relation Joint Extraction Model using Reinforcement

Learning and Its Application Test

Heping Peng

, Zhong Xu

, Wenxiong Mo

, Yong Wang

, Qingdan Huang

, Chengzhu Sun

2, *

and Ting He

Guangzhou Power Supply Bureau of Guangdong Power Grid Co Ltd, Guangzhou, China

College of Computer Science and Technology, Huaqiao University, Xiamen, China

Keywords: Joint Extraction, Entity, Relation, Reinforcement Learning.

Abstract: Extractions of entity and relation are the key part of natural language processing and its application. The

current popular entity extraction methods mainly rely on artificially formulated features and domain

knowledge which cannot achieve simultaneous extraction of entities and their relations, and are largely

affected by noise labeling problems. This paper proposes a new entity-relation extraction model based on

reinforcement learning. This model uses the joint extraction tagging strategy in which the sentences are firstly

input into a joint extractor based on the Long Short-term Memory network for prediction and subsequently

the reinforcement learning algorithm is based on the Policy Gradient for the extraction training. The model is

tested on a public application dataset and the experimental results show the validity of the presented joint

extraction algorithm.

1 INTRODUCTION

There are currently two methods that are widely used

to solve the tasks of entity and relationship extraction.

The traditional pipeline method first extracts the

entities and then identifies the relations between the

pair of entities. These two separations make the task

easy to handle and more flexible. But in fact, these

two tasks have a close relationship. The entity

extracts information to further help the relation

extraction. The quality of the entity extraction

module will affect the relation extraction module. If

the extracted entity pair has no relationship, it will

bring unnecessary information. Noise is generated,

which increases the error rate of relation extraction

(Li, 2014, Ji, 2014). Unlike the pipeline approach, the

joint extraction approach aims to extract both entities

and relations using only one model framework. This

method can effectively extract the semantic

relationship between entities from unstructured text,

and can improve the pipeline-based information

extraction method. However, most existing joint

extraction methods are feature-based structured

systems (Ren 2016, Wu 2016, He 2016), which often

require complex feature engineering and, to some

extent, rely on other NLP toolkits to cause errors to

spread. In order to reduce manual work in feature

extraction, Zheng (Zheng, 2017, Wang, 2017, Bao,

2017) proposed a hybrid neural network model to

extract entities and their relations simultaneously

without any manual features. Although the federated

model can represent entities and relations with shared

parameters in a single model, they also extract entities

and relationships, respectively, and generate other

information.

In this paper, the joint extraction of entities and

relations of unstructured texts is studied in detail. The

policy gradient reinforcement learning algorithm

(Williams, 1992) and Long Short-term Memory

(LSTM) (Hochreiter, 1997, Schmidhuber, 1997) are

used to solve the above problems. This paper

proposed the algorithm model combining

reinforcement learning and deep learning to jointly

extract the entities and relations of public corpus by

applying the joint extraction tagging strategy.

Research on deep reinforcement learning methods

has been widely developed and successfully applied

in fields such as text games (Pascual, 2015,

Gurruchaga, 2015, Ginebra, 2015) and dialogue

generation (Narasimhan, 2015, Kulkarni, 2015,

Barzilay, 2015). The LSTM-based end-to-end model

has been successfully applied to the named entity

recognition tag task (Lample, 2016, Ballesteros,

992

Peng, H., Xu, Z., Mo, W., Wang, Y., Huang, Q., Sun, C. and He, T.

A New Entity-relation Joint Extraction Model using Reinforcement Learning and Its Application Test.

DOI: 10.5220/0011361800003440

In Proceedings of the International Conference on Big Data Economy and Digital Management (BDEDM 2022), pages 992-999

ISBN: 978-989-758-593-7

2016, Subramanian, 2016). The LSTM neural

network model can solve the problem of long-term

sequence dependence, which is very helpful for

sequence modeling tasks. This paper will firstly

introduce joint tagging strategy for entity and relation

extraction and joint extractor based on Long Short-

term Memory network and trainer based on Policy

Gradient reinforcement learning algorithm. The main

contributions of this paper are as follows:(1) A new

model based on reinforcement learning algorithm is

proposed for joint extraction of entity and relation. (2)

The Policy Gradient reinforcement learning

algorithm is applied to the joint extraction problem,

and it can better predict the entities and their relations.

The model scheme based on this paper has achieved

better results than most existing pipelines, joint

learning methods, and provides new ideas for future

research in this field.

2 RELATED WORK

Extraction of entities and relations are two common

tasks in NLP (Zou, 2014, Huang, 2014, Wang, 2014),

for example in Table 1 they are very beneficial for

many NLP tasks such as social media analysis tasks

(Sang, 2012, Xu, 2012). These two tasks are mainly

based on the pipeline method and the joint extraction

method. The traditional method treats the two

subtasks, Named Entity Recognition (NER) (Nadeau,

2007, Sekine, 2007) and Relation Classification (CR)

(Rink, 2010, Harabagiu, 2010), into separate tasks in

a pipelined manner.

Table 1: Examples for the entities and relations extraction

task.

Sentences Entities Relation

Bill Gates and Steve

Ballmer joined

forces at Microsoft

in 1980.

Bill Gates,

Microsoft

Company-

Founder

Bill Gates and Paul

Allen founded the

predecessor of

Microsoft.

Bill Gates,

Microsoft

Company-

Founder

Bill Gates was the

co-founder and

CEO of Microsoft.

Bill Gates,

Microsoft

Company-

Founder

Bill Gates was born

in the US.

Bill Gates,

PlaceofBirth

2.1 Pipeline Method

The classical NER model is a linear statistical model,

such as Hidden Markov Model (HMM) (Luo, 2016,

Huang, 2016, Lin, 2016) and Conditional Random

Fields (CRF) (Passos, 2014, Kumar, 2014, Mccallum,

2014), whose performance depends largely on

manual features of NLP tools and external knowledge

resource extraction. Currently, Recurrent Neuron

Network (RNN) exhibits better performance than

many other neural networks in many sequence-to-

sequence tasks. In the neural network architecture,

NER is considered a continuous marking task. Chiu

and Nichols (Chiu, 2015, Nichols, 2015) proposed a

hybrid model by learning the characteristics of

character and word levels. They independently code

each tag on a linear layer and a log-softmax layer.

Miwa and Bansal (Miwa, 2016, Bansal, 2016)

proposed a coded Bi-directional Long Short-term

Memory (Bi-LSTM) and a separate incremental

neural network structure to jointly decode the tags.

The existing relational classification models

mainly include manual feature-based methods, neural

network-based methods and other valuable methods

(Yu, 2014, Gormley, 2014, Dredze, 2014), Mooney

and Bunescu (Bunescu, 2005, Mooney, 2005) used

distant supervision methods for classification, which

is supervised. The method relies heavily on high

quality label data. Rink (Rink, 2010, Harabagiu,

2010) designed to extract 16 features by using a

number of supervised NLP toolkits and resources

(including part-of-speech tagging POS, English

dictionary Word-Net, etc.). However, this approach

requires a lot of work to design and extract features

and is heavily dependent on other NLP tools. In

recent years, neural network models have been

widely used in relational classification including

convolutional neural networks (Chiu, 2015, Nichols,

2015), long short-term memory networks (Ebrahimi,

2015, Dou, 2015), etc., have achieved good results.

There are other valuable methods. Nguyen (Nguyen,

2009, Moschitt, 2009, Riccardi, 2009) studied the use

of innovative kernels based on syntax and semantic

structure. The synthetic model FCM (Yu, 2014,

Gormley, 2014, Dredze, 2014) studied the

representation of a substructure of an infinite word

statement, and FCM can easily handle arbitrary Type

input and global information.

2.2 Joint Extraction Approach

The entities and relationships extracted based on the

pipeline method ignore the relationship between

these two subtasks, and thus propose a joint

A New Entity-relation Joint Extraction Model using Reinforcement Learning and Its Application Test

993

extraction model. Li and Ji (Li, 2014, Ji, 2014)

proposed a joint model that incrementally predicts

entities and relationships using a structural

perceptron with efficient directed search. Dan and

Yih (Dan, 2007, Yih, 2007) studied the global

reasoning of entity and relationship recognition

through linear programming formulas. Feng and

Zhang (Feng, 2017, Zhang, 2017, Hao, 2017)

combined the intensive learning Q-learning algorithm

with the neural network to extract entities and

relationships. Kate and Mooney (Kate, 2010,

Mooney, 2010) proposed a new method for joint

entities and relation extraction using card pyramid

parsing. Yu and Lam (Yu, 2012, Lam, 2012) jointly

identify entities and extract relationships in an

encyclopedia through a graphical model approach.

Although these methods implement joint extraction,

they cannot achieve high accuracy of entity and

relationship extraction and joint extraction at the

same time.

3 THE JOINT EXTRACTION OF

ENTITIES AND RELATIONS

USING REINFORCEMENT

LEARNING

This paper treats the joint entity and relation

extraction task as a Markov Decision Process (MDP)

and divides a complete joint extraction procedure into

two parts-joint extractor and reinforcement learning

for trainer. For every sentence in a bag, a

reinforcement learning episode will extract the

entities and their relation. Briefly, each sentence will

have its corresponding action which predicted by the

joint extractor and then predict that the bag relation

calculated by the proposed reward function will be

compared to the truly correct bag value. Finally, the

trainer which is the agent by getting rewards trains

the LSTMs network until convergence. Figure 1

shows how the proposed method works.

Figure 1: Overall process of this research.

3.1 Joint Extractor

This paper adopted a novel tagging scheme published

in 2017 from the Association for Computational

Linguistics (ACL) conference for joint extraction

(Zheng, 2017, Wang, 2017, Bao, 2017). Specifically,

tag “O” represents the “Other” bag; the location

information of the word in the entity is marked as

{B(the start of entity), I(the interior of entity), E(the

end of entity), S(single entity)}; the information of

relationship type{1, 2}, Where {1, 2} are respectively

represented as {entity1, entity2}. This strategy

combines the two subtasks in the information

extraction into a sequence labeling problem to further

improve the extraction task. Based on the use of a

joint extraction tagging scheme, this paper only

applies one LSTM network to highlight the role of

reinforcement learning algorithm. The LSTM

network is mainly composed of three gates: forget

gate, input gate and output gate which have the ability

to delete or add information to the cell state. Take the

sentence “Bill Gates and Paul Allen founded the

predecessor of Microsoft” as an example, where

Company-Founder (CF) is the relation of Bill Gates

and Microsoft. Figure 2 displays the novel strategy

combined with LSTM network and LSTM memory

block.

Figure 2: Example for joint extraction tagging scheme and

LSTM block.

The LSTM network is a special extension of

recurrent neural network to avoid long-term

dependency problems. The gate structure is a way to

selectively pass information, usually consisting of a

sigmoid function and a point-by-point product

operation (the output value of the sigmoid layer is 0

to 1, 0 and 1 respectively indicate that the information

has passed and all failed). The detail operation of

LSTM can be defined as follows:

=δ

(

t-1

)

(1)

=δ

(

t-1

)

(2)

=tanh

(

t-1

)

(3)

BDEDM 2022 - The International Conference on Big Data Economy and Digital Management

994

𝑐



𝑓



𝑐



+𝑖



𝑧



(4)

=δ

(

t-1

)

(5)

ℎ



=𝑜



tanh

(

𝑐



)

(6)

𝑇



=𝑊



ℎ



+𝑏



(7)

Where w is the weight and b is the bias. The final

softmax layer computes the confidence vector 𝑦





𝑦



=𝑊



𝑇



+𝑏



(8)

𝑝





(

𝑎



𝑠



,𝜓)

exp (𝑦





)

∑

exp (𝑦





)







(9)

Where 𝑎



is the action predicted by the network,

𝑠



is the representation by the joint extractor

predicted and 𝑁



is the total number of tags. In order

to accelerate the training of the model and improve

the accuracy of the model, this paper pre-trained the

neural network and define the objective function of

the joint extractor using RMSprop proposed by

Hinton (Hinton 2012, Srivastava 2012, Swersky

2012) as follows:

𝐽

(

𝜓

)

= 𝑚𝑎𝑥log(𝑝











||



=𝑦





|𝑠



,𝜓)

(10)

Where |𝑆| is the size of dataset, 𝐿



is the length

of sentence 𝑠



, 𝑦





is the label of word t in the

sentence 𝑠



and 𝑝





is the normalized probabilities

of tags which defined in equation (9).

3.2 Reinforcement Learning for

Trainer

This paper represents the MDP as a tuple (S, A, T, R),

where S={s} is the collection of states, A={a} is the

set of all actions, R(s) is the reward function, and

𝑇(𝑠

‘

|𝑠,𝑎) is the transition function. This paper

introduces these definitions as follows:

States. In order to take advantage of the dataset,

this paper splits the training sentences 𝑆=

{𝑠



,𝑠



,…,𝑠



} into N bags. Define the sentence input

under the current bag as state 𝑠



in MDP.

Actions. This paper uses the policy gradient

algorithm based on round update which can directly

output the action value, while method based on value

function can’t output action values, but state-action

values. Therefore, this paper will adopt the output

value predicted by the LSTM network as the action in

the MDP. Then according to the used tagging strategy,

the total number of actions is 𝑁



=2∗4∗|𝑅|,

where |𝑅| is the size of the predefined relation set.

Rewards. The reward function is chosen to

maximize the final extraction accuracy. First, when

predicting the distribution of each sentence, the

predicted "O" tag is ignored. In the remaining

predicted entity relationship tags, the relationship of

the relationship with the largest probability value is

selected as the current sentence, and then the

probability value is selected by the maximum

likelihood estimation. The biggest as the current bag

prediction relationship, compared with the gold bag.

If they are the same, the reward value of +1 for each

label of the data set is given except “O”, and if it is

different, the reward value of -1 is given. The specific

formula of the reward function is expressed as

follows:

𝑅

(

𝑠



𝐵

)

= 𝛾







𝑟



⎩

⎪

⎨

⎪

⎧

𝛾







𝑟



−𝛾







𝑟



=−1

For example in table 1, the first three statements

can be thought of as a bag. When joint extractor

predicted every word’s label, the label predicted is

“O” like the words “and, joined, in,…” in the first

sentence is ignored. And sampling the probabilities

that the words “Bill”, “Gates”, “Microsoft” are

predicted to “B-CF-1”, “E-PoB-1”, “S-CP-2”

respectively is 0.9, 0.85, 0.7, so the maximum

probability of 0.9 as the relation “Company-Founder”

of the current sentence. Similarly, supposing the

relations of the second sentence and the third is

“Company-Founder”, “PlaceofBirth”. By likelihood

function, the relation of this bag can be calculated as

“Company-Founder”. Due to the gold relation of this

bag is “Company-Founder”, thence the episode

reward will be to set +1.

Transitions. For every episode, a sentence in a bag

will be extracted and immediately next sentence will

be input to joint extractor. One transition includes the

agent being given the state s containing current

information and the future generated. The transition

function 𝑇(𝑠



|𝑠,𝑎) incorporates the reward value

from the agent in state 𝑠 and continue to choose the

next state 𝑠



. The episode stops whenever the model

is convergent.

A New Entity-relation Joint Extraction Model using Reinforcement Learning and Its Application Test

995

Optimization. This paper uses the reinforcement

learning algorithm (Williams 1992) to optimize the

model. For a bag B with n sentences, the expected

total reward will be maximized in the episode. The

reward function of the sentence is 𝑅(𝑠



|𝐵), so the

objective function definition is as follows:

𝐽(𝜃) = 𝐸





,…,



𝑅(𝑠



|𝐵)

(12)

According to the policy gradient algorithm

(Williams 1992), this paper regards 𝑎



as the

predicted label of 𝑠



and update the gradient 𝜃 by

using the likelihood in the following way:

∇𝐽



=∇𝑝

(

𝑎



𝑠



,𝜃)𝑅(𝑠



|𝐵)





(13)

Algorithm 1. Presents the details of complete

joint training process based on MDP framework

ALGORITHM 1: Reinforcement learning for entities

and relations extraction

(

Trainin

hase

)

Initialize the parameters of the LSTM model of joint

extractor with random weights respectively. Pre-train

the LSTM model to predict entities and their relation

given the sentence by joint tagging scheme, where the

parameters are 𝜓.

Input: Episode number L.

𝐁=

{

𝐵



,𝐵



…,𝐵



}

. A LSTM network

model parameterized 𝜓.

Initialize the target network as: 𝜃



=𝜃=𝜓

For episode l=1 to L do

Shuffle B to obtain the bag sequence

𝐁={𝐵



,𝐵



,…,𝐵



}

Foreach 𝐵



∈𝐁 do

Sample the entities and relations for each

sentence in 𝐵



with 𝜃



Compute reward 𝑅

(

𝑠



𝐵

)

for current

sentence

𝑅

(

𝑠



𝐵

)

∑

𝛾







𝑟



end

Update 𝜃 in the model:

∇

𝐽



=∇𝑝

(

𝑎



𝑠



,𝜃



)𝑅(𝑠



|𝐵)





End

4 EXPERIMENT

4.1 Data

This paper adopted ACE2005 which previous studies

has reported on to evaluate the model and use three

common metrics: precision(P), recall(R) and F1-

score(F1). ACE2005 includes three parts: English,

Chinese and Arabic. In order to compare with most

previous work, this paper use the same way (Li 2014,

Ji 2014) with the English dataset to split and

preprocess the data. There are 351 training

documents, 80 validation documents and 80 testing

documents. ACE2005 includes three parts: English,

Chinese and Arabic and defines 7 coarse-grained

relation types. Relation types are “PHYS”(Physical),

“GEN-AFF”(Gen-Affiliation), “ART”(Artifact),

“PART-WHOLE”(Part-Whole), “PER-

SOC”(Person-Social), “ONG-AFF”(Org-Affiliation)

and “METONYMY”(Metonymy). In the process of

extraction, entities and relations can be extracted in a

sentence simultaneously.

4.2 Hyperparameters

This paper employed word2vec to train the word

embeddings and set the dimension of word

embeddings as 300, the number of LSTM units is

fixed at 300 and dropout rate is 0.5. The batch size is

fixed to 160 and episode number is 20. This paper

uses Adam to optimize parameters during the training

procedure. The learning rate is 0.002 and set 𝛾=1

because in this task, the order of sentences in bag

should not influence the predicted result (Zeng 2018,

He 2018, Liu 2018).

Table 2: Entity and relationship extraction results.

Method Entity Relation

Score P(%) R(%) F1(%) P(%) R(%

)

F1(%)

Pipeline (Li

2014, Ji 2014)

83.2 73.6 78.1 67.5 39.4 49.8

Joint

w/Global (Li

2014, Ji 2014)

85.2 76.9 80.8 68.9 41.9 52.1

SPTree (Miwa

2016, Bansal

2016)

82.9 83.9 83.4 57.2 54.0 55.6

RL 86.7 82.1 84.3 69.7 43.4 53.5

4.3 Baselines

The baseline used in this paper is the latest method of

the ACE2005 dataset, including a classic pipeline

model (Li 2014, Ji 2014), a joint feature-based model

called joint w/Global (Li 2014, Ji 2014) and an end-

to-end neural network-based model called SPTree

(Miwa 2016, Bansal 2016). The classic pipeline

method (CRF+ME) trains a linear chain conditional

random field for entity extraction and maximum

BDEDM 2022 - The International Conference on Big Data Economy and Digital Management

996

entropy model for relational extraction. SPTree

proposes a new end-to-end relation extraction model

that represents word sequences and dependent tree

structures through LSTM-RNNs of bidirectional

sequences and bidirectional tree structures. Joint

w/Global developed a number of effective global

features to capture the interdependency among entity

mentions and relations.

4.4 Results

The results of the separate extraction of entities and

relation are shown in Table 2. The combined results

of the joint extraction are shown in Table 3, Where

RL is the method of this paper. The F1 value reached

52.3%, which is the best result compared to the

existing method. It illustrates the effectiveness of the

proposed model in the task of joint extraction of

entities and their relationships.

Table 3: Joint extraction prediction results.

Model P(%) R(%) F1(%)

Pipeline (Li 2014,

Ji 2014)

65.1 38.1 48.1

Joint w/Global (Li

2014, Ji 2014)

65.4 39.8 49.5

SPTree (Miwa

2016, Bansal 2016)

65.8 42.9 51.9

RL 65.6 43.5 52.3

As can be seen from the data in the table 2, SPTree

achieves the highest recall rate for entities and

relations and also is best at the F1 value of the

relations, they are respectively 83.0%, 54.0 and

55.6%. However, the model RL in this paper has the

highest precision in terms of entities and relations and

also is best at the F1 value of the entities, they are

respectively 86.7%, 69.7% and 84.3%. In the joint

extraction of entities and relations from the table 3,

the highest F1 value was also achieved. In order to

facilitate a clear understanding of the indicators

obtained by various models, the histogram 3

characterizes the metrics.

Figure 3: Comparison of histograms of metrics obtained by

each model.

Table 4: Relation of “PlaceofBirth” predicted by the models.

Sentences whose relation type is “PlaceofBirth” RL

PCNN+

Max

PCNN+

ATT

For the two most powerful Americans in Iraq, Gen. George

W. Casey Jr. and Ambassador Zalmay Khalilzad, as for the

Iraqi dignitaries who had gathered here, it was a symbolic

moment: a ceremony on a bluff high above the Tigris River

at which the Americans formally returned the largest of

Saddam Hussein's palace complexes to Iraqi sovereign

control, 31 months after invading troops had seized it for use

as an American base.

PlaceofBirth NA

PlaceofB

irth

It seems inevitable that he's coming back, center fielder

Randy Winn said Wednesday in Los Angeles as the Giants

completed a three-game series with the Dodgers.

PlaceofBirth PlaceofBirth

PlaceofB

irth

The group moved its headquarters to France and then to

Iraq in 1986, when it set up a well-financed military base

under the protection of Saddam Hussein.

PlaceofBirth NA

PlaceofB

irth

Ingrid Rossellini said she was outraged by the conceit, and

by the showing of a widely known scene from the director’s

Rome: Open City in which a German soldier shoots a

character,

ed b

Anna Ma

nani, in the stomach.

PlaceofBirth PlaceofBirth NA

American commanders have described the violence in Iraq

as being caused variously by a mix of foreign terrorists,

Sunni loyalists to Saddam Hussein, Shiite radicals and

criminals.

PlaceofBirth NA

PlaceofB

irth

/通用格式

Pipeline Joint w/Global SPTree RL

Metrics

Method

R(%) F1(%) P(%)

A New Entity-relation Joint Extraction Model using Reinforcement Learning and Its Application Test

997

5 DISCUSSION

Entity and relation extractions are always handled in

an unbalanced corpus, where there is no relation

between entities in most statements. Therefore, the

public corpus-New York Time data set is used for

verification. The New York Times corpus is a distant

supervision dataset created by aligning the freebase

knowledge base with the New York Times corpus. In

order to evaluate how accurately the decision of the

Policy Gradient reinforcement learning algorithm

module is carried out, taking the current classical

mainstream comparison research methods

PCNN+CrossMax (Jiang 2016, Wang 2016, Li 2016)

and PCNN+ATT (Lin 2016, Shen 2016, Liu 2016) to

experiment for relation extraction. The

precision/recall curves of those models in Figure 4. It

can be seen from the model presented in this paper

outperforms other two methods. The maximum value

of F1-Score of this paper’s model can reach out

42.19%, although PCNN+CrossMax got the highest

precision.

Figure 4: Comparison between the PCNN+CrossMax and

PCNN+ATT.

By using the distant supervision as the guide of

reinforcement learning so that we can understand that

the model in this paper can better predict the relation

among the sentence. A real case is shown in table 4.

As can be seen from the table, the results predicted by

the model in this paper are correct, while the other

two models have certain errors

6 CONCLUSIONS

In view of the shortcomings of the pipeline method,

this paper proposes a new model using the

reinforcement learning algorithm, which can realize

the joint extraction of entities and relations at the bag

level by using the combination of the joint labeling

strategy and the special reward function. This model

mainly consists of two modules: one is joint extractor

based on the LSTM network with annotation

extraction strategy, and the other is the Policy

Gradient reinforcement learning algorithm for

training. Experiments show that the proposed model

can complete the joint extraction of entities and

relations at the bag level and achieve better results.

FUNDING

Research works in this paper are supported by the

research and application of the key technology of

electromagnetic transient cloud simulation platform

for extremely large urban distribution network

(080037KK52170012/GZJKJXM20170023).

REFERENCES

B. Pascual; M. Gurruchaga; M.P. Ginebra. A Neural

Network Approach to Context-Sensitive Generation of

Conversational Responses. Transactions of the Royal

Society of Tropical Medicine and Hygiene 2015, 51,

502-504, doi: 10.3115/v1/n15-1020.

Bunescu, R. C; Mooney, R. J. Subsequence kernels for

relation extraction. International Conference on Neural

Information Processing Systems. MIT Press 2005, 1,

171-178, doi: 10.1109/cicsyn.2012.13.

Chiu, J. P. C; Nichols. E. Named Entity Recognition with

Bidirectional LSTM-CNNs. Computer Science 2015,

doi: 10.1109/ebbt.2019.8741631.

Dan, R; Yih, W. T. Global Inference for Entity and Relation

Identification via a Linear Programming Formulation.

Introduction to Statistical Relational Learning 2007, 1,

doi: 10.7551/mitpress/7432.003.0022.

Ebrahimi, J; Dou, D. Chain Based RNN for Relation

Classification. Conference of the North American

Chapter of the Association for Computational

Linguistics: Human Language Technologies 2015, 1,

1244-1249, doi: 10.3115/v1/n15-1133.

Feng, Y; Zhang, H; Hao, W. Joint Extraction of Entities and

Relations Using Reinforcement Learning and Deep

Learning. Comput Intell Neurosci 2017, 2, 1-11, doi:

10.1155/2017/7643065.

Hinton, G; Srivastava, N; Swersky, K. Neural networks for

machine learning lecture 6a overview of mini-batch

gradient descent. Cited on 2012, 14, doi:

10.4135/9781526494924.

Hochreiter, S; Schmidhuber, J. Long short-term memory.

Neural Computation 1997, 9, 1735-1780, doi:

10.4324/9781315174105-4

Jiang, X; Wang, Q; Li, P. Relation extraction with multi-

instance multi-label convolutional neural networks, the

26th International Conference on Computational

BDEDM 2022 - The International Conference on Big Data Economy and Digital Management

998

Linguistics: Technical Papers 2016, 1, 1471-1480, doi:

10.1109/bibm.2018.8621435.

Kate R J, Mooney R J. Joint entity and relation extraction

using card-pyramid parsing. Association for

Computational Linguistics 2010, 1, doi:

10.1109/iccsnt.2011.6182359.

Lample, G; Ballesteros, M; Subramanian, S. Neural

Architectures for Named Entity Recognition. Neural

Architectures for Named Entity Recognition 2016, 1,

260-270, doi: 10.1109/bigcomp.2019.8679233.

Li, Q; Ji, H. Incremental Joint Extraction of Entity

Mentions and Relations. Proceedings of the 52nd

Annual Meeting of the Association for Computational

Linguistics 2014, 1, 402-412, doi: 10.3115/v1/p14-

1038.

Lin, Y; Shen, S; Liu, Z. Neural relation extraction with

selective attention over instances. Proceedings of the

54th Annual Meeting of the Association for

Computational Linguistics 2016, 1, 2124-2133, doi:

10.18653/v1/p16-1200.

Luo, G; Huang, X; Lin, C. Y. Joint Entity Recognition and

Disambiguation. Conference on Empirical Methods in

Natural Language Processing 2016, 1, 879-888, doi:

10.18653/v1/d15-1104.

M, Yu; MR, Gormley; M, Dredze. Factor-based

compositional embedding models. NIPS Workshop on

Learning Semantics 2014, doi 10.18653/v1/d15-1205.

Miwa, M; Bansal, M. End-to-End Relation Extraction using

LSTMs on Sequences and Tree Structures. ACL

2016, 1, 1105-1116, doi: 10.18653/v1/p16-1105.

Nadeau, D; Sekine, S. A survey of named entity recognition

and classification. Lingvisticae Investigationes 2007,

30, 3-26, doi: 10.1075/bct.19.03nad.

Narasimhan, K; Kulkarni, T; Barzilay, R. Language

Understanding for Text-based Games Using Deep

Reinforcement Learning. Computer Science 2015, 40,

1-5, doi: 10.18653/v1/d15-1001.

Nguyen, T. V. T; Moschitti, A; Riccardi, G. Dependency

and Sequential Structures for Relation Extraction.

Conference on Empirical Methods in Natural Language

Processing 2009, 6-7, 1378-1387, doi:

10.3115/1699648.1699684.

Passos, A; Kumar, V; Mccallum, A. Lexicon Infused

Phrase Embeddings for Named Entity Resolution.

Computer Science 2014, doi: 10.3115/v1/w14-1609.

Ren, X; Wu, Z; He, W. CoType: Joint Extraction of Typed

Entities and Relations with Knowledge Bases.

International World Wide Web Conferences Steering

Committee 2016, 1, 1015-1024, doi:

10.1145/3038912.3052708.

Rink, B; Harabagiu, S. UTD: Classifying semantic relations

by combining lexical and semantic resources.

Association for Computational Linguistics 2010, 1,

256-259, doi: 10.1515/9783110920116.xiii.

Sang, J; Xu, C. Right buddy makes the difference:an early

exploration of social relation analysis. ACM

International Conference on Multimedia 2012, 1, 19-

28, doi: 10.1145/2393347.2393358.

Williams, R. J. Simple statistical gradient-following

algorithms for connectionist reinforcement learning.

Machine Learning 1992, 8(3-4), 229-256, doi:

10.1007/978-1-4615-3618-5_2.

Yu, X; Lam, W. Jointly Identifying Entities and Extracting

Relations in Encyclopedia Text via A Graphical Model

Approach. International Conference on Computational

Linguistics DBLP 2012, 1, 1399-1407, doi:

10.1007/978-3-642-40585-3_34.

Zeng, X; He, S; Liu, K. Large scaled relation extraction

with reinforcement learning. Thirty-Second AAAI

Conference on Artificial Intelligence 2018, 1, doi:

10.5220/0006508702330238.

Zheng, S; Wang, F; Bao, H. Joint Extraction of Entities and

Relations Based on a Novel Tagging Scheme.

Proceedings of the Twenty-Seventh International Joint

Conference on Artificial Intelligence 2017, 1, 1227-

1236, doi: 10.24963/ijcai.2018/620.

Zou, L; Huang, R; Wang, H. Natural language question

answering over RDF:a graph data driven approach.

ACM SIGMOD International Conference on

Management of Data 2014, 1, 313-324, doi:

10.1145/2463676.2463725.

A New Entity-relation Joint Extraction Model using Reinforcement Learning and Its Application Test

999