2.1

OOKB entity representation

The methods of OOKB entity representation fall into two categories: (1) using the text description of the OOKB entity. The typical work are references^12-15. (2) using the neighbors of the OOKB entity from auxiliary triples. The typical work are references^8-10. The text descriptions provide rich information for the entity, however, this constraint is too strong that not all KGs contain the information of entity description. On the other hand, we can always extract triples from the text as auxiliary triples to represent the OOKB entity. Thus, our work focuses on the second category. Reference⁸ is the first work to use graph neural network (GNN) to aggregate neighbor information which proves GNN based aggregators perform better than the simple approximation like TransE. The drawback of this work is that treats all neighbor equally. To address that issue, Wang et al.⁹ and Zhao et al.¹⁰ both consider the difference among neighbors. The former work considers in more respects in which the aggregator is aware of redundancy and query relation. It reduces the impact of similar neighbor information and measure the importance of neighbors by the query relation. For example, if asking someone’s nationality, the aggregator need to concern more on neighbors which contain birthplace and live_in other than gender. Besides, reference¹⁶ propose a model that only uses graph structure to predict relations between nodes. However, their work only considers the relations information and can not distinguish entities.

2.2

Using transformer to capture contextual information

The Transformer model has been widely used in recent researches such as pre-trained language mode¹¹, KG embedding^17-18 and machine translation¹⁹. The core of Transformer is self-attention mechanism²⁰, that each token in the input sequence can gather information from other tokens. Reference¹⁷ uses a pre-trained language model Bert to get KG representations. They concatenate the head name, relation name and tail name in one triple as input sequence, then fine tune the parameters of the model on the triple classification task. Each word in the sequence can get information from the other words. Reference¹⁸ only inputs one triple with entity and relation IDs into Transformer per time, and both entity and relation can obtain information from the opposite side. Unlike the traditional training objective of KG-embedding models, Transformer based models use a special task of “masked token prediction” to learn parameters. In the training stage, a head or tail is masked and the model need to predict the true entity. However, the above two mentioned works are not under the OOKB setting. Moreover, they both model the single triple. In this work, we study a more complex scenario which deals with several triples at one time.

3.1

OOKB entity problem definition

A knowledge graph G consists of an entity set ε, a relation set R and a collection of true triples {(h, r,t)}⊆ ε × R × ε. For each triple (h,r,t), we define the reverse relation r^-1 and add (h,r^-1,t) to G. Triple classification and entity prediction are two important tasks in KBC. Unlike the conventional settings of both tasks where all entities in the test procedure have been trained, this work considers a challenging scenario that unseen entities are given in the test procedure. We define the unseen entity as e,e ∉ ε and the unseen entity set as ε_OOKB. The new triple classification and entity prediction tasks are defined as follows.

Task 1 Triple classification with OOKB entity

Given a triple (h, r, t), r ⊆ R, where h ∈ ε_OOKB or t ∈ ε_OOKB, the task is to train a model to classify whether the triple is true or false.

Task 2 Entity prediction with OOKB entity

Given a head entity h,h ∈ ε_OOKB and a relation r,r ∈ R, or given a tail entity t,t ∈ ε_OOKB and a relation r,r ∈ R, the task is to train a model which ranks all tail candidate entities t,t ∈ ε or ranks all head candidate entities h,h ∈ ε.

The main problem of the two tasks is how to represent the OOKB entity. For both tasks, we import the auxiliary triples to construct the representation of OOKB entities, which defined as G_aux = {(h,r,t)|r ∈ R,h ∈ ε_OOKB,t ∈ ε}∩{(h,r,t)|r ∈ R,h ∈ ε,t ∈ ε_OOKB}. Here, we also add (h,r^-1, t) to G_aux. For each OOKB entity e, we can now use the auxiliary triples which contains e to represent it.

3.2

OOKB entity representation

To obtain the OOKB entity representation, we need to construct an aggregation to integrate all neighbor information of this OOKB entity which provided by the auxiliary triples. We define N_t(e) as neighbor triples, N_t(e) = {(e,r,e’)|r ∈ R, e’ ∈ ε}. We also define v_e ∈ ℝ^d as a d-dimensional vector to representation entity e, and define v_r ∈ ℝ^d to represent relation r. According to the exist method^21-22, the aggregation consists of transition function T(v_e) and pooling function P(v_e). These two components are described below in detail.

Transition Function. For each OOKB entity e, its neighbor triples may contain different kinds of relation and entities. The transition function T(v_e) aims to apply the influence of neighbor relation to the representation of the neighbor entity, which can propagate the relation-special information to the OOKB entity. According to the way of relation projection, transition function contains three categories:

Non-relation projection. For an OOKB entity e, its neighbor triple (e, r, e’) ∈ N_t (e) propagates the whole information of the entity e’ to the e:

Distance-based relation projection. For an OOKB entity e, its neighbor triple information depends on the distance between e and e’, the typical examples inspired by TransE and TransH are listed:

where w_r and construct a relation-related hyperplane²³.

NN-based relation projection. For an OOKB entity e, we use a relation-specific matrix M_r to reflect the influence of relation r on entity e’, the examples of the transition function are listed:

Pooling Function. For each OOKB entity e, it often has many neighbor triples. Thus, pooling function is used to summarize all these neighbor information. Typical pooling function P(v_e) contains sum pooling, mean pooling and max pooling, which all satisfy permutation invariance. That means all the input of neighbors are unordered. These three pooling functions are defined by:

where N is the total number of N_t(e) and max is the element-wise max function.

Unlike the above pooling functions that treat all neighbor equally, attention-based pooling assumes the different neighbor contributes the different information. For each neighbor, pooling function assigns a weight to enlarge or reduce its information, and then sum all changed information together.

4.1

LAN

The core of LAN is the attention mechanism. In the aggregator, neighbors should contribute differently to the v_e according to its importance in representing e. LAN donates two different kinds of attention mechanisms: the first one is the logic rule mechanism which captures the attention between the neighbor relation r and the query relation q; the second is the neural network mechanism which captures the attention between the neighbor entity e_i,e_i ∈ N_t(e) and the query relation q. LAN implements the confidence of logic rule r₁ ⇒ r₂ as follows:

The function ‖(x) here is a logic function, which equals 1 when x is true and 0 otherwise. According to this equation, we easily find that the confidence is larger if relation r₁and r₂ often appear as the entity’s neighbor relation at the same time. Thus, for the entity e, LAN calculates all confidences of neighbor relation r and query relation q to measure which neighbor is more important. However, the entity may have many neighbors and some of them provide similar information, which will cause redundancy. The logic rule mechanism is defined as follows:

where j is the j-th neighbor of entity e_i. We can find that if the neighbor relation r can be implied by another neighbor relation r’, its contribution to represent entity e_i will be decreased. This logic rule attention uses the statistical relevance of relations to distinguish the neighbor information.

The neural network mechanism adopts an attention network to measure the relevant of neighbor entity and query relation²⁴, which is defined as:

where u_a and W_a ∈ ℝ^d×2d. z_q ∈ ℝ^d is a relation-aware embedding representation of the query relation. T(v_e) here uses equation (3). To normalize all neighbor attention weights, the softmax function has been used:

Finally, LAN incorporates two attention mechanism together to gather all neighbor information:

4.2

Neighbor-T

As we want to train an aggregation to represent an entity by its neighbor information, we need to take advantages of the graph contextual information. The concept of contextual information first comes from language model. For a word in the text, its contextual information is often a sequence of words before or after it, which implies rich semantic patterns. For an entity in the KG, it also has contextual information. We define its neighbor entity and neighbor relation as graph contextual information. According to this definition, we could find some special graph semantic patterns and use them to construct the entity’s embedding representation.

Transformer model is successfully used to capture contextual information in the language model by its self-attention mechanism. In this work, we conduct Transformer in the KG to capture graph contextual information and aggregate them to represent the entity. We name this aggregation as Neighbor-T. The architecture of Neighbor-T is shown in Figure 2. Firstly, for each entity e_i, we extract its neighbor triple set N_t(e_i). The input sequence consists of two parts: neighbor entity sequence and neighbor relation sequence. In transition layer, equation (3) is used to conduct the influence of the relation on the corresponding entity, and we can get the transitional embedding T(v_ej) of each neighbor. Secondly, we feed transitional embeddings of all neighbors to a Transformer to exchange information by self-attention mechanism. At this stage, each neighbor has a transformed embedding T′(v_ej), which has incorporated the contextual information from the other neighbors. Lastly, we sum all transformed embeddings to obtain the output embedding:

Figure 2.

The architecture of Neighbor-T for entity representation.

Here we also use the prior statistical attention weights to fill the information gap between neighbor relation and query relation.

4.3

Objective and model training

In the aggregation training stage, we can only get all training triples, which means there is no OOKB entity. Thus, for each training triple, we treat its head entity and tail entity as OOKB entities in this sample and extract subgraphs of the head entity and tail entity to construct the model input.

In this work, we propose a three-stage training method to learning parameters since we use Neighbor-T after LAN to construct enhanced representations of OOKB entities. Firstly, we get the pre-trained embeddings of all existing entities by training parameters of LAN. Since the amount of training triple is limited, we need to give Transformer a better initialization. The frequently-used score function is to evaluate the possibility of a triple(h, q, t). The score is larger when the triple is likely to hold. Here, we use TransE to calculate the triple score:

where L1 is the L1 norm. We treat all triples in KG as positive sample and randomly corrupt the head or tail entity by another entity to construct negative samples. Then we use margin-based loss function as our objective:

where [x]₊ = max(0, x), N is total number of positive and negative samples, and γ is the max margin between positive and negative samples.

In the second stage, we train the parameters in Neighbor-T. It is notice that we fix all entity embeddings because they have been trained in LAN. The prediction objective used here is to label the masked entity, which is to predict the true entities of head and tail. For a masked entity e_i, we can obtain its output embedding by Neighbor-T. Then we calculate the probability of each candidate entity and use cross-entropy as the training loss:

Where E ∈ ℝ^n×d is the matrix of the entity embeddings, y_i is the weight vector of the true label distribution. Here we use soft label strategy instead of one-hot label. N is double amount of positive triples because we predict both head and tail entities. This stage is used to make the model learn how to capture contextual information of neighbors.

In the third stage, we sum the output embeddings of LAN and Neighbor-T to get a stronger representation of entities:

We train all parameters in LAN and Neighbor-T by Equation 16 in this stage. This is for finetuning the parameters in the model to adapt two KBC tasks.

5.1

Experimental design

Datasets. Our experiments are all in OOKB settings, that is there are some OOKB entities in the testing. The previous work⁸ has constructed nine datasets from WordNet11 (WN11)²⁵: Head, Tail, Both-1000, 3000, 5000. Head, Tail. Both are the position of OOKB entity, and 1000, 3000, 5000 is the amount of the testing triples which contain OOKB entities. For example, Both-1000 has extracted 1000 triples from the original testing dataset of WN11, and each head and tail entity of the triple are treated as OOKB entities. The original training dataset is split into the new training dataset and auxiliary dataset. The original triples which do not contain OOKB entities are placed in the new training dataset and the original triples which contain one OOKB entity are placed in the auxiliary dataset. The triples with two OOKB entities are discarded. For the validation set, all triples which contains OOKB entities are also discarded to avoid the data leakage. The above nine datasets are used for triple classification task. Wang et al.⁹ construct other ten datasets from Freebase15K (FB15K)⁴ for entity prediction task in the same way: Head, Tail-5, 10, 15, 20, 25, where the number is the percentage of the extracted testing triples. In this task, there is no Both setting because we cannot predict the missing entity by another missing entity and the given relation. We directly use these two groups of datasets and the statistics for them are in Tables 1 and 2.

Table 1.

Statistics of constructed WN11 dataset.

Table 2.

Statistics of constructed FB15K dataset.

Implementation Details. In triple classification task, we can calculate all scores by equation (15). For each relation r, we select a threshold σ_r to classify the true triples if f(h, r,t >= σ_q), and false triple in the other condition. In this task, we use the classification accuracy as the evaluation metrics. The best σ_r is optimized by maximizing classification accuracy on the validation triples. We adopt the same parameter configuration for all nine datasets: the learning rate is 0.001, embedding dimension is 100, batch size is 512, margin is 300, and 64 neighbors are randomly selected for each entity, which are same as reference⁹. Because the scale of our training data is not as large as language model corpus, here we only use one head and one-layer Transformer. We also use dropout strategy to avoid over-fitting the training data. The dropout is set to 0.1.

In entity prediction task, we need to calculate the scores of all candidate entities to determine which is more likely to be the tail entity when giving the head entity and relation, or to be the head entity when giving the tail entity and relation. Here the candidate entities are all existing entities in the training data. After achieving the scores of candidate entities, we rank them in descending order. The evaluation metrics here are mean rank (MR), mean reciprocal rank (MRR) and the proportion of ranks no larger than n (Hits@n, n=1,3,10). All results are under filtered setting⁴, that means any candidate triple already exists in the train, or validation data needs to be removed before ranking. To align with LAN, we also conduct experiments on Head-10 and Tail-10. The LAN parameters are same as them in triple classification task except the margin changes to 1.0. The dropout here is 0.8 because these two datasets are easily over-fitting. We will give the additional analysis about this point in the following.

Our training process is divided into three stages, we train the LAN with 500 epochs and train Neighbor-T with 1000 epochs. In triple classification tasks, we train the LAN+Neighbor-T with 100 epochs; and in entity prediction task, we train it with 500 epochs. The reported results are the testing results performer best in validation data.

6.1

Triple classification with OOKB entity

Table 3 shows the model comparison on nine triple classification datasets. We use three previous models as our baselines. MEAN is the results of reference⁸, we directly list their results from the original paper. Results of LSTM and LAN come from reference⁹. To our knowledge, LAN performs best in all nine datasets before our work is proposed. Thus, we retrain the LAN with their released source code and the same parameter settings at least 5 times and list the best results in LAN(ours). On the basis of LAN(ours), we train Neighbor-T and combine LAN and Neighbor-T to represent entities. The table shows LAN+Neighbor-T achieves the best results in all Tail datasets and Both datasets, which show our model are effective to capture contextual information of both OOKB entities and their neighbors. We find results of Head datasets show a little low results than LAN(proposed), we conjecture it is because the head entities in WN11 are naturally more coarse-grained, which need more information to describe them.

Table 3.

Evaluation accuracy on triple classification.

The results in the first three rows come from the original paper and the results in the last two rows are our implementations.

6.2

Entity prediction with OOKB entity

Table 4 shows the model comparison on two entity prediction datasets: Head-10 and Tail-10. We also list the same three baselines: MEAN from reference⁸ and LSTM and LAN(proposed) from reference⁹. According to the results, we find all results (except Hits@10 in Head-10) of our model are better than LAN(ours). Furthermore, our combined representations of LAN+Neighbor-T achieve the best performance of MRR and Hits@1 in both datasets. Unlike MR is sensitive to the lower positions of the ranking, MRR evaluate the model more stably. Thus, we make sure that the embeddings come from Neighbor-T can enhance the entity representation.

Table 4.

Evaluation results on entity prediction.

Same as triple classification, the results in the first three rows come from the original paper and the results in the last two rows are our implementations.

6.3

The relevance of proportion of OOKB entities and model effectiveness

According to Table 3, we find when using Neighbor-T to enhance entity representations of LAN in different datasets, its improving capacity is different. We illustrate the relevance of proportion of OOKB entities and the improving capacity in all Tail and Both datasets in Figure 3. The figure shows that with the amount of OOKB entities increasing, the relative increasing percentage of the model is growing at the same time. Furthermore, the speed of the result growing could be beyond the linear growth mode in Both datasets. That means, when we meet large amount of OOKB entity, Neighbor-T can help LAN to achieve a stronger representation of the entity to avoid performance decreasing rapidly.

Figure 3.

The relevance of proportion of OOKB entities and model effectiveness.

6.4

The influence of transformer over-fitting

As we have mentioned in the previous section, the Transformer’s performance will influence the final entity embeddings. In our second training stage, we learn Transformer’s parameters by predicting the true entity throughout its neighbor information. Because the Transformer has strong capacity in fitting training data, and the training amount is limited due to OOKB settings, it is easily to cause over-fitting problem. When we use it to construct the representation of OOKB entities, it may fall into some extremely fine-grained contextual semantic patterns. To figure out the impact of Transformer’s performance on final entity embedding, we try different dropouts and evaluate the performance of entity prediction. Figure 4 shows that when dropout increase, the Transformer prediction accuracy will decrease, which means the model reduces its fitting capability at the same time. On the contrary, the results of MRR, Hits@10,3,1 increase. The interpretation is the objective of Transformer is to predict the entity. In the second training stage, if transformer accuracy is too high, it means the embeddings from Neighbor-T will always tend to represent the existing entities. Because the average neighbor number of FB15K is much larger than WN11, it means entities in FB15K have more information and easily to cause over-fitting problem. Thus, we assign dropout to 0.1 in triple classification task and 0.8 in the entity prediction task.

Figure 4.

The influence of transformer’s over-fitting problem. The red line in all four subgraphs is Transformer accuracy, while the blue line is MRR, Hits@10, Hits@3 and Hits@1 respectively.