Time series prediction is a field that has attracted significant attention. Algorithms for time series prediction are widely used in many fields such as weather, electricity, and financial markets. They have been developed by using well-known classic models such as the autoregressive moving average (McLeod and Li, 1983) as well as recurrent neural network (RNN) (Rumelhart et al., 1986; Werbos, 1990; Elman, 1991) and long short-term memory (LSTM) (Hochreiter and Schmidhuber, 1997) to improve their performance. RNN is a sequential model widely used for time series data. Itx takes information from prior inputs to influence current input and output using hidden state recurrently. LSTM is a special structure of RNN. Unlike RNN, LSTM adds memory cells to calculate if the information is important or not. Three gates, input gate, forget gate and output gate, judge the input and decide to retain or forget by some special rules to capture long term dependency of time series data. In addition, as a nonlinear autoregressive exogenous (NARX) model (Chen et al., 2008) using other exogenous variables that can affect the target variable, not just the past value of the target variable, was proposed, the performance of time series prediction has been improved.

The NARX model has been continuously studied, such as an improved hybrid prediction model (Pham et al., 2010) and a new approach to identifying a new class of NARX models (Li et al., 2011). In particular, in the case of RNN, we succeeded in increasing the prediction rate by applying it to the NARX model. However, it was difficult to capture long-term dependency. To overcome this, methods using LSTM, gated recurrent unit algorithms (Cho et al., 2014a) and encoder decoder networks (Cho et al., 2014b; Sutskever et al., 2014) were developed.

Research on introducing an attention mechanism has also been actively conducted, and studies using a new RNN method such as an attention-based encoder decoder network (Liu and Lane, 2016) have also been conducted. Research has also been conducted to introduce attention to the NARX model. A hierarchical attention network that selects the hidden state of the related encoder using two attention mechanisms (Yang et al., 2016), a model that unifies the spatiotemporal feature extraction of exogenous variables and the temporal dynamics modeling of the target variable (Tao et al., 2018), two-stage two-phase RNN (Liu et al., 2019), and various algorithms have been proposed. All these algorithms show improved performance. A dual-stage attention-based RNN (DA-RNN) model (Qin et al., 2017) has also been proposed, which has the advantage of selecting exogenous variables with a great influence on the prediction of target variable and being able to properly capture the long-term dependency of time series. In addition, a model using a new attention-based multi-input LSTM (MILSTM) (Li et al., 2018) using factors with low correlation effectively, followed by a two-dimensional attention-based LSTM (2D-LSTM) (Yu and Kim, 2019) model was also proposed. 2D-LSTM is a model in which the importance of exogenous variables and the importance of time are separately calculated and combined.

In this paper, we propose a two-dimensional attention-based multi-input LSTM (2DA-MILSTM) model that includes the advantages of DA-RNN, 2D-LSTM and MI-LSTM. This model calculates the correlation between the target variable and the exogenous variables, divides the exogenous variables accordingly and then inputs them to MI-LSTM. It also applies two separate attention layers simultaneously using the output value of MI-LSTM and the hidden state of the previous layer. Two kinds of weights are created by applying the attentions to each hidden state and time step. By integrating these two weights, the predicted value can be calculated, and the importance of the hidden state and the time step can be considered to improve the long-term dependency problem and prediction performance. We compare the performance of the proposed model with existing models through prediction experiments on stock price, room temperature and energy data.

The rest of this paper is as follows. In Section 2, we review the existing models, and in Section 3 we explain the detailed parts and configurations of the proposed model in detail. We describe datasets used in the experiment and present results of the experiment in Section 4. Lastly, Section 5 draws the conclusion of this paper.

2.1. Recurrent neural network and long short-term memory

In neural network, RNNs and LSTM are the most popular model for time series including language and speech data (Huang et al., 2015). Figure 1 shows that the RNN model maintains consists of input layer x, hidden layer h, and output layer y. An input layer takes the features at time t, and output layer represents a probability distribution for labels at tie t. As the recurrent layer sores history informations, RNN connects between the previous hidden state and the current hidden state.

Then the values in the hidden and output layers are computed by h(t) = f (Ux(t) + Wh(t − 1))), y(t) = g(Vh(t)), where U,W, and V are the connection weights computed in training, and f (z) and g(z) are sigmoid and softmax activation functions. Softmax (Jang et al., 2016) is used as an activation function for the classification task. It calculates normalized output of probability distribution in the neural network. Defined as $\sigma (z_i)=\text{exp}(z_i)/{\mathrm{\Sigma }}_{j=1}^K\hspace{0.17em}\text{exp}(z_j)$, which takes as input vector z of K classes, normalize output vectors dividing by the sum of all components in exponential scale.

LSTM is also similar to RNN but LSTM captures long term dependencies of times series data better than RNN using three gates, input, forget, and output gate in the LSTM cell. In Figure 2, σ represents logistic sigmoid function, and i, f, o, and c represent the input gate, forget gate, output gate and cell vectors with same size with the hidden vector h.

Then the procedure of LSTM cell using matrix notation is:

For explanation, W_xo represents the input-output gate matrix, W_hi represents the hidden-input gate matrix. These calculations concentrate on which information should be focused to capture short and long term dependencies in time series data as well as determine how much information should be forgotten using the forget gate.

2.2. Multi-input long short-term memory model

The target variable is defined as Y = (y₁, y₂, . . . , y_T )^⊤ ∈ ℝ^T, and the exogenous variables related to the prediction of the target variable are defined as X = (X₁,X₂, . . . ,X_T )^⊤ ∈ ℝ^T×D where T is the time window size and D is the number of exogenous variables. To express the exogenous variables in detail, two notations are used. First, ${\mathbf{X}}_t={(x_t^1,x_t^2,\dots ,x_t^D)}^{\top }\in {ℝ}^D$ denotes the values of all exogenous variables at time t, and ${\mathbf{X}}^k={(x_1^k,x_2^k,\dots ,x_T^k)}^{\top }\in {ℝ}^T$ denotes the values of the kth exogenous variable. The matrix X contains three kinds of exogenous variables: X_p ∈ ℝ^T×P having a positive correlation with the target variable, X_n ∈ ℝ^T×N having a negative correlation with the target variable, index variables X_i ∈ ℝ^T for stock price data if available, where D = P + N + 1.

The Pearson correlation coefficient is used to measure the correlation between the exogenous variables and the target variable, and X_p and X_n are generated using P-largest and N-smallest variables with positive and negative correlation coefficients, respectively. In our experiment, {4, 6, 10, 15} is a set of possible values for P and N, and 15, the largest value, is finally selected.

We define “Self”, “Index”, “Positive”, and “Negative” based on Y, X_i, X_p, and X_n values, respectively, by using LSTM with the hidden state size of r. The definitions are:

• Self:

  $$\tilde{\mathbf{Y}}=\mathbf{LSTM}(\mathbf{Y}).$$

• Index:

  $$\tilde{\mathbf{I}}=\mathbf{LSTM}({\mathbf{X}}_{\mathbf{i}}),$$

  where Ỹ,Ĩ ∈ ℝ^T×r.

• Positive:

  $$\tilde{\mathbf{P}}=\frac{1}{P}\sum _{p=1}^P\text{LSTM\hspace{0.17em}}\left({\mathbf{X}}_{\text{p}}^p\right).$$

• Negative:

  $$\tilde{\mathbf{N}}=\frac{1}{N}\sum _{n=1}^N\text{LSTM\hspace{0.17em}}\left({\mathbf{X}}_{\text{n}}^n\right),$$

  where P̃, Ñ ∈ ℝ^T×r.

The MI-LSTM is a model that can extract valuable information from small correlation factors and discard negative noises using extra input gates controlled by convincing factors. This model deviates from the existing LSTM methods and shows that a model having multi-inputs to the LSTM can improve prediction performance. As defined above, MI-LSTM uses four input values: the past value of the target variable, exogenous variables with positive and negative correlation, and an index exogenous variable. The input values are expressed as Ỹ, P̃, Ñ,Ĩ, and the values after passing through MI-LSTM can be expressed as

and an output value of ${\tilde{\mathbf{Y}}}^{\prime }={({\tilde{\mathbf{Y}}}_1^{\prime },{\tilde{\mathbf{Y}}}_2^{\prime },\dots ,{\tilde{\mathbf{Y}}}_T^{\prime })}^{\top }\in {ℝ}^{T\times p}$ is obtained.

Figure 3 is the structure of MI-LSTM. The following attention is applied to Ỹ′ to give weights:

where the matrix W_b ∈ ℝ^p×p, bias b_b ∈ ℝ^p, and vector v_b ∈ ℝ^p are the parameters to be learned, ỹ ∈ ℝ^p is the attention output, W ∈ ℝ^1×p and b ∈ ℝ are also parameters to be learned, and ReLU (rectified linear unit) is an activation function. ReLU was introduced by Nair and Hinton (2010) to make the deep learning model train better. Marked as ReLU(x) = max(0, x), unlike the activation functions before such as sigmoidal activation, it gets fewer vanishing gradient problems which prohibit neurons to learn weights in the propagation steps.

2.3. Dual-stage attention-based recurrent neural network

DA-RNN consists of two steps. In the first step it extracts the hidden state of the previous encoder and the relevant exogenous variables in each step by using the input attention. Then, in the second step the model selects the output value of the related decoder using the temporal attention mechanism. This model can select the most relevant exogenous variables as well as adequately capture the long-term dependency of the time series. Figure 4 and Figure 5 show the structure of DA-RNN.

The n exogenous variables are represented by matrix X = (x₁, x₂, . . . , x_n)^⊤ = (x₁, x₂, . . . , x_T ) ∈ ℝ^n×T, where T is the time window size, ${\mathbf{x}}^k={(x_1^k,x_2^k,\dots ,x_T^k)}^{\top }\in {ℝ}^T$ represents the kth exogenous variable, and ${\mathbf{x}}_t={(x_t^1,x_t^2,\dots ,x_t^n)}^{\top }\in {ℝ}^n$ represents the vectors of n exogenous variables at time t. DA-RNN creates a new input vector by multiplying this by the weight obtained through the following input attention mechanism. It inputs the new input vectors back to the LSTM and makes predictions by obtaining weights through a temporal attention mechanism.

where h_t−1 ∈ ℝ^m and s_t−1 ∈ ℝ^m are the previous hidden state and the cell state in the encoder LSTM unit, respectively, and v_e ∈ ℝ^T, W_e ∈ ℝ^T×2m, and U_e ∈ ℝ^T×T are parameters to be learned, and m is the size of the hidden state of the encoder. As in the above equations, a new input value is created by multiplying the weight, and after inputting it into LSTM as shown in the following equations, it is finally input into the temporal attention mechanism of the decoder.

where v_d ∈ ℝ^m, W_d ∈ ℝ^m×2p, and U_d ∈ ℝ^m×m are parameters to be learned, p is the size of the decoder’s hidden state, f₁ and f₂ are LSTMs, and v_y ∈ ℝ^p, W_y ∈ ℝ^p×(p+m), b_w ∈ ℝ^p and b_v ∈ ℝ are also parameters to be learned.

3.1. Two-dimensional attention-based multi-input LSTM

In this section we present the details of the two-dimensional attention-based multi-input LSTM (2DA-MILSTM) model, a new model that includes the advantages of the MI-LSTM and DA-RNN models described in Section 2 and 2D-LSTM. Figure 6 is the structure of the 2DA-MILSTM model.

The exogenous variables are divided using the value of the correlation coefficient, and the output value after inputting it to MI-LSTM is expressed as Z = (z₁, z₂, . . . , z_T )^⊤ ∈ ℝ^T×u, where u is the number of MI-LSTM hidden units, and ${\mathbf{z}}_t={(z_t^1,z_t^2,\dots ,z_t^u)}^{\top }\in {ℝ}^u$ represents the u hidden unit input vector at time t. The proposed model uses H-attention and T-attention for Z, which can be calculated as:

where F is the function we want to learn.

In this model, as in the existing MI-LSTM model, “Self”, “Positive”, and “Negative” are used as input values of MI-LSTM. However, unlike in the existing model, three factors excluding index variable are input. This is because there is no index exogenous variable in the data we consider in this paper. Therefore, we use the Ỹ,P̃, Ñ ∈ ℝ^T×r as the input of the three factors, and Ỹ_t,P̃_t, Ñ_t ∈ ℝ^r as the corresponding input vector for t = 1, 2, . . . , T. Figure 7 shows the structure of the modified MI-LSTM.

The forget gate and the output gate of the MI-LSTM remain the same when compared to the original LSTM as shown in the following equations:

where f_t, o_t ∈ ℝ^u are the forget gate and output gate, respectively, h_t−1 ∈ ℝ^u is the hidden state of the previous time step and u is the number of the MI-LSTM hidden units. W_f,W_o ∈ ℝ^u×(u+r) are the weights of the forget gate and the output gate, respectively, and b_f, b_o ∈ ℝ^u are the biases.

All input gates are determined by the “Self” variable and the previous hidden state to control the auxiliary factors as in the following equations:

where i_t, i_pt, i_nt ∈ ℝ^u are the input gates of three factors, W_i,W_ip,W_in ∈ ℝ^u×(u+r) are the weights, and b_i, b_ip, b_in ∈ ℝ^u are the biases. The cell states of three factors C̃_t, C̃_pt, C̃_nt ∈ ℝ^u are obtained as:

where W_c,W_cp,W_cn ∈ ℝ^u×(u+r) are the weights, and b_c, b_cp, b_cn ∈ ℝ^u are the biases. In LSTM, the element-wise multiplication of C̃_t and i_t is the final cell state input at time t:

where l_t, l_pt, l_nt ∈ ℝ^u are the cell inputs of the factors, and ⊙ is the element-wise multiplication. Then, the sum of the weights is performed for the cell input as:

where α_t, α_pt, α_nt ∈ ℝ are the attention weights, and L_t is the input of the final cell state at time t. The attention weights are determined by the cell state input itself and the cell state in the previous step as shown in the following equations:

where u_t, u_pt, u_nt ∈ ℝ are values that appear in the middle of the calculation of the attention weights, and W_a ∈ ℝ^u×u and b_a, b_ap, b_an ∈ ℝ are parameters to be learned. Also, C_t−1 ∈ ℝ^u is the state of the cell at time t−1. [u_t, u_pt, u_nt]^⊤ is the attention score and is input to the softmax layer. When the update vector of the cell state L_t is stabilized, the remaining MI-LSTM units proceed in the same manner as the original LSTM.

where C_t, h_t ∈ ℝ^u are the cell state and hidden state at time t, respectively.

The structure of MI-LSTM is the same as the above equations. In this paper, the equations (3.1)–(3.18) are expressed as F₂ to summarize as:

It is also intended to be expressed simply as:

and we get an output value Z = (Z₁,Z₂, . . . ,Z_T ) ∈ ℝ^T×u for each time step.

3.2. H-attention

Considering the kth output value ${\mathbf{z}}^k=(z_1^k,z_2^k,\dots ,z_T^k)\in {ℝ}^T$, we can construct the attention model by referring to the previous hidden state ${\mathbf{h}}_{t-1}^{\prime }$ and the cell state ${\mathbf{C}}_{t-1}^{\prime }$ in the LSTM unit. By constructing an attention model, an attention weight ${\alpha }_t^k$ can be obtained. We focus on the output value in the MILSTM structure and consider whether the output value contains accurate information and sufficiently explains the information by looking at the importance at time step when using it. Therefore, we introduce an H-attention mechanism that can select which value is of high importance among the output values from MI-LSTM as:

where v_e ∈ ℝ^T, W_e ∈ ℝ^T×2m, and U_e ∈ ℝ^T×T are parameters to be learned, and ${\alpha }_t^k$ is the attention weight that measures the importance of the kth hidden unit at time t. In addition, the softmax function is applied to $u_t^k$, so that the weight values sum to 1.

A new input vector can be extracted by multiplying the weight derived through the H-attention mechanism by the input value.

Using this new input vector, the hidden state at time t can be updated as:

where f_lstm is the LSTM function.

3.3. T-attention

In order to predict the output ŷ_T+1, another LSTM-based RNN is used to introduce an attention that can select the importance of time step. Considering the input series ${\mathbf{z}}_t={(z_t^1,z_t^2,\dots ,z_t^u)}^{\top }\in {ℝ}^u$ at time t, the attention weight of each hidden state is calculated based on the previous hidden state d_t−1 ∈ ℝ^u and the cell state ${\mathbf{C}}_{t-1}^{″}\in {ℝ}^u$ of the LSTM unit, and the attention model can be constructed as:

where $[{\mathbf{d}}_{t-1};{\mathbf{C}}_{t-1}^{″}]\in {ℝ}^u$ is a concatenation of the previous hidden state and the cell state of the LSTM unit, and v_d ∈ ℝ^m,W_d ∈ ℝ^m×2u, and U_d ∈ ℝ^m×m are parameters to be learned. The attention weight ${\beta }_t^i$ denotes the importance of the ith hidden state for prediction. The attention mechanism computes the context vector c_t as the weighted average of all new input vectors as:

Note that the context vector c_t is distinct at each time step.

Once we get the context vector, we can obtain ỹ_t−1 as:

where [y_t−1; c_t−1] ∈ ℝ^m+1 is a concatenation of the decoder input y_t−1 and the calculated context vector c_t−1. The parameters w̃ ∈ ℝ^m+1 and b̃ ∈ ℝ map the connected part to the input size of the T-attention.

The newly calculated ỹ_t−1 can be used to update the decoder hidden state at time t as:

where f₂ is the LSTM function. Then d_t can be updated as:

where [d_t−1; ỹ_t−1] ∈ ℝ^u+1 is a concatenation of the previous hidden state d_t−1 and the input value ỹ_t−1 of the LSTM of the T-attention. ${\mathbf{W}}_f^{\prime },{\mathbf{W}}_i^{\prime },{\mathbf{W}}_o^{\prime },{\mathbf{W}}_c^{\prime }\in {ℝ}^{u\times (u+1)}$ and ${\mathbf{b}}_f^{\prime },{\mathbf{b}}_i^{\prime },{\mathbf{b}}_o^{\prime },{\mathbf{b}}_c^{\prime }\in {ℝ}^u$ are parameters to be learned. σ and ⊙ are logistic sigmoid functions and element multiplications, respectively.

We conduct an experiment using two types of data sets to check whether the proposed model has good performance in stock price data as well as in time series data in other fields. A stock price prediction experiment is conducted using KOSPI 200 data. A general prediction experiment on time series data in various fields other than stock prices is conducted using temperature data and energy data.

4.1. Data description

• KOSPI 200: This is a data set where, among the listed companies on KOSPI, 200 stocks considered to represent KOSPI due to their large market capitalizations and large trading volume are selected, and used to predict the stock index. This is daily data measured for a total of 2158 days from August 23, 2011 to May 29, 2020. In our experiment, the KOSPI 200 index is used as the target variable, and the stock prices for 165 companies are used as exogenous variables (Table 1). These companies are selected so that there are no missing values in the data for companies such as Samsung Electronics, SK Hynix, Samsung Biologics, Naver, Hyundai Motor Company, and LG Chem.

• SML 2010 (Zamora-Martinez et al., 2014): This is a public dataset used for indoor temperature prediction as data collected from a monitor system installed in a general house. Data was sampled every minute and calculated every 15 minutes on average. It consists of approximately 40 days (before and after 43 days) of monitoring data. We use 4137 data. In the experiment, 16 exogenous variables are used as shown in Table 2, and data having missing values and variables such as date and time are removed. Among the indoor temperatures, the temperature of dinning room is used as the target variable.

• Appliances energy (Candanedo et al., 2017): This data has been used to predict appliance energy use in low energy buildings. Data were measured at intervals of 10 minutes over a period of 4.5 months. The total size of data is 19734, and as shown in Table 3; 27 exogenous variables are used (excluding date) and appliances, energy consumption, are used as the target variable. For time series data, there should be stationarity before make any forecast. So we preprocessed the three time series data with min max scaling using Scikit-learn packages (Pedregosa et al., 2011).

4.2. Design of experiment

The parameters of our model include the LSTM dimension r, the MI-LSTM dimension u, time window size T, the number of exogenous variables with positive correlation P and the number of exogenous variables with negative correlation N, the size of hidden states of the LSTM for H-attention, q, and the size of hidden states of the LSTM for T-Attention, m.

In our experiment, in the case of r and m, we selected 128 as a result of conducting experiments with 64 and 128. In addition, P and N are set to be the same and selected among {4,6,10,15}; in addition, the time window size T is selected among {10,16,32}. Lastly, 0.8, 0.1, and 0.1 are used as the ratio of training, validation, and test datasets.

We consider three different metrics to evaluate the performance of the models in time series prediction experiments. Assuming y_t is the target at time t and ŷ_t is the predicted value at time t,

• Mean absolute percentage error (MAPE): a value obtained by taking the average of the absolute percentage of the residual

  $$\text{MAPE}=\frac{1}{N}\sum _{i=1}^N\left|\frac{y_t^i-{\widehat{y}}_t^i}{y_t^i}\right|.$$

• Root mean square error (RMSE): a value obtained by taking the root of the average of the squares of the residual

  $$\text{RMSE}=\sqrt{\frac{1}{N}\sum _{i=1}^N{\left({\widehat{y}}_t^i-y_t^i\right)}^2}.$$

• Mean absolute error (MAE): average of all absolute errors

  $$\text{MAE}=\frac{1}{N}\sum _{i=1}^N\left|{\widehat{y}}_t^i-y_t^i\right|.$$

Note that the MSE, RMSE, and MAE are calculated using normalized data due to huge difference in amount among different time series data.

To compare the performance of 2DA-MILSTM, we conduct a comparative experiment with two models, DA-RNN and MI-LSTM. Also, not based on neural network model, we conduct the classical time series model in statistics field, ARIMA. Designed by Hyndman and Khandakar, auto.arima in forecast package for R (Hyndman and Benítez, 2016), automatically determines the orer of ARIMA model using step-wise algorithm for forecasting. The experiment was conducted using KOSPI 200 data (stock price data), SML 2010 data (room temperature data), and appliances energy data (energy data).

4.3. Results

The time series prediction results of 2DA-MILSTM and three existing methods over the three datasets are shown in Table 4. In Table 4, we observe that the MAPE, RMSE and MAE of ARIMA are larger than those of the other neural net-based methods. There are two interesting exceptions for such observation, one of which is MAPE for KOSPI 200 dataset. ARIMA outperforms the other methods for the data in terms of MAPE. The possible reasons are (1) The stock price by nature reveals higher volatility as its level goes up, (2) The MAPE puts a heavier penalty on negative errors than on positive errors. These reasons are because of the high volatility at high level of stock price it is difficult to predict it accurately and there are possibly more positive errors at high level. Subsequently, it can be small even though a model does not predict well at high level since the MAPE puts a lighter penalty on positive errors.

We can also see that the proposed model outperforms the other methods for all three datasets in terms of all three criteria, except the case decsribed above. This suggests that it might be beneficial to consider the correlation between exogenous variables and target variable by introducing MI-LSTM. We can consider the importance of exogenous variables using the input attention; however, DA-RNN just downgrades the importance of variables with negative correlation with the target variable. With integration of the MI-LSTM as well as two attention mechanisms, the proposed 2DA-MILSTM achieves the best MSE, RMSE and MAE across three datasets since it uses the MI-LSTM to consider the correlation of exogenous variables effectively as well as employs H- and T-attention mechanisms to capture relevant hidden features across all time steps as well as potential output information.

For visual comparison, we show the prediction results of the three neural net-based methods over certain ranges of the three datasets in Figures 8–16.

Figures 8–10 show that our model makes better predictions for KOSPI 200 data than the other two models. In particular, in the case of DA-RNN, it follows the overall trend, but we can some time shift across all time period. In the case of MI-LSTM, the prediction is accurate until time point of 70, but after that time, the result deteriorates. However, we can see that 2DA-MILSTM predicts almost exactly the actual value except slight overestimating and shifting after time point of 115.

Figures 11–13 show that our model predicts SML 2010 data better than the other two models. In the case of both DA-RNN and MI-LSTM, we can see that the prediction before time point of 60 is not accurate (the prediction is accurate only between 60 and 75 for DA-RNN). Using MI-LSTM, in addition to the range of 60–75, the prediction is pretty accurate after time point of 105. However, we can see that 2DA-MILSTM predicts very accurately over all the time period except between 70 and 95.

Figures 14–16 show that our model makes better predictions for appliances energy data than the other two models. In the case of this data, all three methods do not predict accurately between time point of 20 and 40. DA-RNN predicts slightly better than the other two models for the time period. However, it overestimates before time point of 20. We can also see that MI-LSTM underestimates between 40 and 110. However, 2DA-MILSTM predicts very accurately before 20 and between 40 and 110.

In this paper, we proposed a two-dimensional attention-based MI-LSTM (2DA-MILSTM) motivated by the advantages of MI-LSTM, which effectively uses exogenous variables with low correlation, and DA-RNN and 2D-ALSTM, in which an attention mechanism is introduced twice. The proposed method was applied to KOSPI 200, SML 2010 and appliance energy datasets. The results show that our proposed 2DA-MILSTM can outperform existing methods for time series prediction.

The proposed model, 2DA-MILSTM, can selectively use output value information and temporal information by introducing H-attention and T-attention. H-attention can better capture potential information of the output value, and T-attention can capture temporal information. 2DA-MILSTM, which uses the two attention mechanisms selects the output value with accurate information as well as captures temporal information appropriately to improve predictive performance. Better forecasting depends on capturing long- and short-term dependencies. A longer forecasted time series increases variability. While MI-LSTM capturing the dependencies, T-attention and H-attention make the output values of MI-LSTM stable considering temporal and potential values. This means that this task can control the increasing variability of time series data because the two attention mechanism weights control both temporal and potential information.

The stock price has high volatility at high level. Therefore, if one uses the log-return of the stock price rather than the original scale, then the prediction at high level might be more accurate; consequently, the performance of the model can be improved. However, in the literature of NARX field, there are several papers where the stock prices are used as the original scale, and two of which are the motivation for our research, DA-RNN and MI-LSTM. Therefore, we believe that using the same (original) scale for the stock price makes it easier for readers to compare our proposed model with the two models.

As a follow-up study, visualization method for H-attention and T-attention weights could be considered to check where the two attention mechanisms concentrate among exogenous variables, time stamp and logits from MI-LSTM each. In addition, non-linear relation between exogenous variables and target variable should be considered. When applying non-linear relations, it can be expected that the proposed 2DA-MILSTM model receives additional information between exogenous variables and the target variables. Also, various ranges of the hyperparameters should be experimented with while considering the relationship between hyperparameters.