In recent years, deep learning has shown epoch-making performance in various fields such as image recognition (Tan and Le, 2019), speech recognition (Saon et al., 2017), and natural language processing (Edunov et al., 2018). Deep learning models with many hidden layers appeared every year. In 2015, the residual neural network (ResNet) (He et al., 2016) with more than 100 hidden layers won the ImageNet large scale visual recognition challenge (ILSVRC) (Deng et al., 2009). Figure 1 shows the winner algorithms in ILSVRC from 2010 to 2015. We can see from the figure that as the model descended deeper, the accuracy improved, but the model became excessively large accordingly.

Despite these outstanding performances, the computational cost of the deep learning models due to their complexity and large number of parameters made them difficult to deploy on devices such as smartphones (Han et al., 2015). Therefore, in the field of machine learning, the compression of models is one of the most important factors because it can create a more user-friendly environment for models to be widely used. As a result, research on compressing deep learning models into relatively small models has continued to overcome high computational costs (Buciluă et al., 2006).

Many techniques have been studied and developed to solve the inverse problem of accuracy and computation time in deep learning. Recently, one of the most popular techniques for model compression is knowledge distillation (KD). The basic idea of KD is to compress the model by transferring the knowledge from the pre-trained, large, and complex model, the Teacher model, to the relatively small model, the student model (Hinton et al., 2015).

Hinton et al. (2015) performed model compression using the logit value, which is a pre-activated value in the output layer of the teacher model. Recently, Furlanello et al. (2018) studied how to improve the model’s generalization performance through KD. They set the teacher model and the student model to the same structure. Afterward, the student model built through KD was set as a new teacher model and KD was performed again to increase the generalization performance of the model. Park et al. (2019) proposed relational KD (RKD), which additionally extracts relational information of embedded vectors from data during the training process. They showed that RKD, which adds structural relationships between embedded vectors from data points, outperforms conventional KD.

When a well-performing teacher model is successfully compressed, computers or small devices such as smartphones can learn the features of the data with less computational cost and make fine decisions just as well as before compression. Since the main idea of RKD lies in reducing the structural differences in the vectors between the teacher model and the student model, there will be further improvements if we convey more structural information about the vectors. Perhaps there is a limit to extracting more structural information when comparing a pair of vectors. However, when converting the insight of a pair of vectors to an area-wise approach, we can consider the structural information in the form of a triangle by additionally considering zero vector.

In this paper, we propose area-wise RKD as an extension of RKD. After generating the embedding vectors through the model, we can extract relational knowledge through a function that measures the structural differences between the vectors. The proposed structural relationship information of embedding vectors is the area of a triangle composed of two embedding vectors and a zero vector. A classification model is constructed to reduce the difference in relational knowledge obtained from the teacher model and the student model. By adding the area-wise relational information to the distillation, we achieve improvement in model compression and generalization, two main goals of KD.

We apply the proposed method to build classification models using image and audio data to compare the performance of the models with that of the existing methods. In the experiment using audio data, we compare the proposed method with existing ones in terms of model compression. In the experiment using image data, we set teacher and student models to have an identical structure to check that the proposed area-wise RKD can increase the generalization performance more than the existing methods.

Overall, the main contributions of this paper are:

• We propose area-wise RKD to convey additional structural relational information between vectors when training the model. We achieve improvements in KD for the student model by using the proposed method combined with the existing RKD.

• We propose a loss function that can adjust the ratio of the relational information between the teacher model and the student model.

• We provide extensive experiments comparing the performance of different methods for model compression and generalization on both image and audio data.

The rest of this paper is organized as follows. Section 2 reviews some related works in detail. Section 3 presents the proposed method with a new loss function. Section 4 provides the details of the datasets, the preprocessing process, the models used as teacher and student models, and the experimental results. Finally, in Section 5, we draw conclusions while discussing future research directions.

2.1. Knowledge distillation

In KD, we first train a large and complex deep neural network model, and then when training a small model, we pass the hidden or output layer values of the teacher model to a relatively small model and set those values as the target values. For example, a large and complex model is a single large model with strong regulation, such as an ensemble of multiple trained models or a model with dropout methods. By training large and complex models, we can build models more efficiently for deployment by filtering out redundant information from training data and transferring information to smaller models (Hinton et al., 2015).

In general, the loss function used to train a model reflects as closely as possible whether the data belong to true classes. But during the training process, we also need to consider how good the predictive performance of the model is, i.e., how well it generalizes to make predictions to new data. This requires information about whether the generalization process is going in the right way, but usually, this kind of information is difficult to obtain. However, in the KD process, since the classification probability can be estimated using the value obtained during the training of a large and complex model using data, the generalization method can be transferred to a smaller model using the value (Hinton et al., 2015).

Hinton et al. (2015) proposed a model compression method using logit values, which are pre-activated values in the output layer. They proposed dividing the logit value by the temperature, which is a real number greater than 1, and applying the softmax function to obtain the soft target. They set the temperature values so that the distribution of values in the output layer of the teacher model is properly flattened. The same temperature is applied to the logit values from the student model and the distillation is performed by reducing the soft target difference between the teacher model and the student model. The following is how the soft targets are obtained:

where z_i is the logit, and T is the temperature. If T is set to 1, p_i is equal to the value passed through the softmax function.

Figure 2 shows an example of training results on images labeled 5 on the MNIST data set (LeCun et al., 1998). A neural network model denoted by is trained using the image data and the output layer values. In this case, the probability of classifying the image as 6 is 0.1, which is relatively higher than the rest except for 5. This value provides important information that the image belongs to class 5 but is also similar to class 6. However, this kind of information may become obsolete when using loss functions such as cross-entropy. Thus, by introducing temperature to obtain a soft target, we can obtain a new flattened class distribution of the data, so the values can convey rich information about the class.

2.2. Born again neural networks

Furlanello et al. (2018) proposed born again neural networks (BANs). Rather than viewing KD as a methodology for model compression, they showed that it is possible to improve the generalization performance. In BAN, teacher and student models are set up to have the same structure. At the same time, the classification probability of the teacher model is used as the target value of the student model. Once the student model is trained, the trained model becomes a new teacher model for the next KD process. They showed that student models obtained over multiple generations can improve the performance of model generalization and outperform teacher models. In the image classification task, the goal is to find a neural network function f: X → Y that generalizes well to unseen data when the inputs and labels of the training data are (x, y) ε X × Y (Furlanello et al., 2018). In general, we try to get θ^*by minimizing the loss function L for the weight parameter θ in some space as follows:

In BAN, the student model is trained using the values of the output layer of the teacher model, and the output layer of the student model seeks to obtain optimized parameters. Experiments have shown that student models obtained over multiple generations can improve the performance of model generalization and outperform teacher models. Moreover, a better model was obtained when the average ensemble was applied to the student model.

Figure 3 shows the structure of a BAN, where X is the training data, T and S are the teacher model and the student model, respectively, and f (x) and y are the output layer value and the target value, respectively. According to the algorithm of BAN, S_(t–1) becomes a teacher model of S_t and this process can be performed k times. During the training process, we can optionally introduce the loss function as follows:

The final ensemble model is based on the average of the probability values for each student model to classify the data:

where f^k(x) is the average of the values of the output layer of the student models after training k generation, and θi* is the optimized weight parameter after the training process of the i^th student model.

2.3. Relational knowledge distillation

Park et al. (2019) proposed RKD, a method to distill relational information of values embedded by a model. In order to extract relational information, specific hidden layers are selected from the teacher model and the student model, and the structural information of the values of the hidden layer is defined as a specific function. Afterward, they proposed a method to further utilize the training method in a way to reduce the structural difference between the teacher model and the student model. Then, they showed that RKD can improve the performance of models in the fields of metric learning, image classification, and few-shot learning.

Figure 4 shows how the conventional KD and RKD techniques are applied, where t_i and s_i are embedding vectors obtained after the data x_i passes through the teacher model and the student model, respectively. t_i and s_i are dependent because they are outputs of input x_i. In the conventional method, the knowledge generated from a single data point is transferred individually from the teacher model to the student model. However, in RKD, the knowledge containing relational information generated from more than a single data point is transferred. The two functions of RKD measuring relational information are defined:

where ψ_Distance measures the Euclidean distances between two vectors and μ represents the normalization factor. And ψ_Angle measures the angle formed by the three vectors obtained from the model output.

After obtaining relational information from the teacher and student models, the student model is trained to reduce the information gap using Huber loss:

Huber loss is commonly used for robust regression, and the smaller the δ is, the more robust the model is against outliers. Typically, δ is set between 1 and 2 in robust regression.

Distance-wise distillation loss L_RKD–D and Angle-wise distillation loss L_RKD–A are now defined as:

where l_δ is the Huber loss and δ is set to 1 the same as in RKD.

RKD has been proposed as a new method to extract structural relationship information for output. However, since the distance and angle information are converted through a loss function, we believe that additional information is needed to preserve structural relationships.

In this paper, we propose area-wise RKD as an extension of RKD. The loss function of the area-wise RKD is defined as:

where t_i and s_i are the embedding vectors obtained from teacher and student models, respectively. These vectors can be set within the range of (x_i, x_j)εχ² by creating pairs of input data during training. The function l_δ is to calculate the difference between the relational information of the teacher model and the student model. We use the Huber loss function with δ = 1. The function ψ_Area defines the relational information and is used to calculate the area of the triangle formed by the two embedding vectors and the zero vector after the data is fed into the model as follows:

where θ is the angle between the two vectors t_i and t_j, and μ_t is the average of all area values obtained during training and is defined as:

Then, training is conducted by reducing the size difference of each triangle obtained from the teacher model and the student model. Figure 5 shows an illustration in which the embedding vectors obtained after the training data has passed through the teacher model and the student model form a triangle in the embedding space. In conventional RKD, structural differences in embedding values are learned by reducing distance differences or angular differences, respectively. By adding the area differences to them, it is expected to provide richer information on the structural differences between the teacher model and the student model and build a better classification model.

To construct the student model, the output layer values are used to convey knowledge about the probabilities of the classes using cross-entropy as the loss function. Relational information is additionally conveyed using the embedding value of the flattened layer before the output layer. Relational information can be scaled by multiplying λ. Finally, the loss function is defined as the sum of the loss function for RKD and the loss function for class probabilities and can be used in combination with distance, angle, and area-wise RKD methods:

where f_t(x_i) and f_s(x_i) are obtained from the output layer of the teacher and student models, respectively. Figure 6 shows the structure of the loss function for the student model.

4.1. Datasets

4.2. Feature extraction from the unstructured data

In this study, image data are used for experiments to improve the generalization performance of the model and audio data are used for model compression experiments.

4.3. Experimental procedure and composition

4.4. Results

In this study, we proposed area-wise RKD with a new loss function. This method can reduce the difference in relational information between the teacher model and the student model by obtaining the area information generated by embedding vectors of the data. In addition to the existing distance- and angle-wise RKDs, it is expected to provide richer information on the structural differences between teacher and student models and further improve the performance of the student model.

Existing methods have tried to improve the performance of the student model by transferring information about the distance between embedding vectors or the angle formed by the embedding vectors from the teacher model to the student model. On the other hand, the proposed method can be viewed as considering the information that includes the distance and angle considered in the existing methods at once by trying to convey the information about the area of the triangle created by the embedding vectors. For example, even if the distance between the embedding vectors is the same, it can be seen that a small angle and a large angle between the vectors represent different information.

Conversely, even if the angle between embedding vectors is the same, a short distance and a long distance between vectors represent different information. However, considering the area of a triangle containing both distance information and angle information, the effect of integrating each piece of information to be delivered through the two existing methods can be expected.

To verify the performance of area-wise RKD, we conducted two experiments using audio and image data. In the experiment using audio data, model compression was performed using the proposed method along with the existing methods. The results show that the performance of model compression is best when both area- and angle-wise RKDs are applied together.

In the experiment using image data, we considered BAN to improve the generalization performance of the model by introducing various RKDs. As a result, the performance improvement was the best when angle-wise RKD was applied, and sub-optimal accuracy was shown in the experiment when area-wise and angle-wise RKD were used together.

For future research, we plan to apply the proposed method to data in other domains, such as natural language, video, and time series data. In addition, we could consider applying the proposed method to RNN-based models, Seq2Seq models, and attention-based models.