Smart grid appears as a key term in the context of optimizing energy efficiency. One of the integral facets of the smart grid is advanced metering infrastructure (AMI), which enables two-way communication meters and the availability of electricity consumption data at a higher frequency. By 2021, UK energy suppliers will have installed 50 million smart meters, and 22.5 million smart meter installations are planned in South Korea.

Therefore, various studies have analyzed AMI data in relation to energy consumption in households. Krishna et al. (2015) proposed an anomaly detection method based on principal component analysis (PCA) and density-based spatial clustering of applications with noise to verify the integrity of the smart meter measurements. Glasgo et al. (2017) used direct metering methods, non-intrusive load monitoring methods, and statistical methods to inform estimates of device-level energy consumption in the United States, and Chalmers et al. (2019) examined electricity usage, behaviors and changes in habits from smart meter records.

Various clustering methods have been applied to characterize electricity consumption data and interpret the load pattern. Kwac et al. (2014) proposed an electricity customer segmentation methodology that classified households according to extracted features, and Xu et al. (2015) applied the hierarchical K-means method to a large-scale AMI dataset. Blakely et al. (2019) proposed a spectral clustering approach using a voltage time series produced from AMI equipment to validate and correct customer electrical phase labels. Shin et al. (2011) forecasted load pattern based on the clustering and classification algorithms of temporal data-mining techniques. Further analyses in this field can be found in Chicco et al. (2004) and Romero et al. (2011). However, most existing studies simply apply conventional clustering methods to the electricity consumption data. Therefore, the non-stationary characteristic of the AMI data has not been considered.

In this paper, we develop a data-adaptive clustering method that may reflect this characteristic of the AMI data and also provide a new perspective on electricity consumption compared to the existing clustering methods. The proposed method is based on the functional PCA (FPCA), which is commonly used for data reduction in functional data (Van Der Linde, 2008; Silverman and Ramsay, 1997). Although concepts and methods in functional data analysis may be robust for serial dependence, they have been developed for independent observations. However, time series data such as economy and energy consumptions, does not support this assumption. To solve this problem, Brillinger (2001) first suggested frequency domain PCA (FDPCA), which considers the correlation in time. FDPCA is also called a dynamic PCA (DPCA), and has been studied in various fields (Salvador et al., 2003; Mina and Verde, 2007).

However, FDPCA is based on the spectral representation of a stationary signal. Stationarity, which means statistical characteristics such as mean and variance do not change over time, is a very important assumption in time series analysis. However, in real data analysis, the stationarity assumption is rarely satisfied (Hamilton, 1989; Azadeh et al., 2010).

Figure 1, which depicts the load pattern from randomly selected households in South Korea from January, 2012 to October, 2014, illustrates the non-stationary characteristic of the data.

To alleviate the stationarity assumption, Ombao and Ho (2006) proposed a time-dependent FDPCA for the locally stationary data. They formed a small block around each time point, and therefore, the multi-channel signal was almost stationary in that time block, which defined the time-varying spectral density matrix. Their method extracted time-varying spectral features of a multi-channel signal. Thus, we expect that the time-dependent FDPCA is a proper starting point to cluster the non-stationary time series data.

In this paper, we propose a time-dependent FDPCA based clustering method for AMI data that is non-stationary time series data. The clustering results are further applied to validate the reform of the progressive electricity tariff system in South Korea. The South Korean government has planned to apply differentiated tariff rates by season and hour, which involves charging higher rates during peak seasons/hours and lower rates during non-peak seasons/hours, and was scheduled to start in July, 2021. We apply the reformed system to each cluster group, and analyze the effect of the reform on household energy usage.

The remainder of this paper is organized as follows: In Section 2, we describe the proposed time-dependent frequency domain principal component clustering method and also provide a practical algorithm. Simulation results are presented in Section 3, and real AMI data analysis are illustrated in Section 4. Finally, the concluding remarks are presented in Section 5.

Here, we first briefly review the FDPCA and time-dependent FDPCA, and then explain the proposed clustering method based on the time-dependent FDPCA.

2.1. Time-dependent frequency domain principal component analysis (FDPCA)

Suppose that we have p-dimensional stationary time series X(t) = [X₁(t), …, X_p(t)]′ with zero mean. The corresponding spectral representation is

where A(ω) is a time-invariant transfer function matrix and Z(ω) is a stochastic process with orthonormal increments. The corresponding spectral density matrix of X(t) is defined to be

where ${\overline{\mathbf{A}(\omega )}}^{\prime }$ denotes the transpose of the complex conjugate of A(ω), with −∞ < ω < ∞.

We now intend to approximate X(t) using a q-variate process J(t) = [J₁(t), …, J_q(t)]′ (q ⩽ p), defined by

where c(u) is the p × q valued filter and absolutely summable.

Then, we can reconstruct X(t) from J(t) by defining

where b(u) is the p × q filter and absolutely summable.

To obtain X̂(t), minimize the mean square approximation error

Then, the minimum solution is

where V₁(ω), …, V_q(ω) are eigenfunctions of the spectral density matrix, f(ω), and correspond to the eigenvalues, v₁(ω) > … > v_q(ω), which are arranged in decreasing magnitude (for a detailed description of the FDPCA, see Theorem 9.3.1 in Brillinger (2001) and Chapter 7 in Shumway et al. (2000)).

As mentioned above, the stationarity assumption is crucial in the FDPCA. Ombao and Ho (2006) divided the time series into blocks to apply FDPCA to the locally stationary process. Therefore, the time series within the block appears stationary. Then, define X(t) as

where T is the length of X(t) and its transfer function matrix, A, now depends on time. The time-varying spectral density matrix is defined as

where u ∈ [0, 1] denotes the rescaled time point and ω ∈ (−π, π).

Ombao and Ho (2006) conducted the FDPCA in the neighborhood of a particular rescaled time u^* ∈ [0, 1]. Under the locally stationary process, the spectrum f(u^*, ω) is stable and can be estimated by using the time series around t^* = [u^*T], where [.] denotes the greatest integer function. Let M = 2_J be the number of observations in a neighborhood for some positive integer J. Then, {X(t^* − (M/2) + 1, ) …, X(t^* + M/2)} become time blocks and Fourier coefficients are obtained as

where t′ = t^* − M/2 and ω_k = 2πk/M, k = −M/2 + 1, …, M/2. Then, the periodogram matrix and its smoothed version by the rectangular kernel are computed as

where H(u^*) = 2h(u^*) + 1 and h(u^*) is a positive integer.

Finally, the time-varying eigenvalues and eigenvectors are obtained by applying eigenanalysis to the matrix f̂(u^*, ω_k). The first q eigenvalues, ξ̂¹(u^*, ω_k), …, ξ̂^q(u^*, ω_k), are interpreted as the time-varying spectrum. The corresponding eigenvectors are denoted as Ξ̂¹(u^*, ω_k), …, Ξ̂^q(u^*, ω_k).

Compared to the eigenvalues, v₁(ω) > … > v_q(ω), and the corresponding eigenfunctions, V₁(ω), …, V_q(ω) obtained from the FDPCA, the time-dependent FDPCA provides time-varying eigen-values and eigenfunctions.

Therefore, the dimensions of the eigenvalues and eigenvectors in the time-dependent FDPCA are higher than those of the FDPCA. The eigenvalue, ξ̂ ^j(u^*, ω_k), in the time-dependent FDPCA can be represented by a time-frequency map, and the p×q dimensional matrices are obtained for each u^* and ω_k in the eigenfunction, Ξ̂^j(u^*, ω_k) for j = 1, …, q.

2.2. Clustering

To cluster the data based on the time-varying FDPCA, we define weighted time-varying eigenvectors as follows. For each time point, u^*, and frequency, ω_k, the jth eigenfunction, Ξ̂^j(u^*, ω_k), is multiplied by the corresponding eigenvalue at the same time point and frequency, ξ̂ ^j(u^*, ω_k). Then, we compute the sum of all frequencies. That is, the p-dimensional weighted time-varying eigenvector, W^j(u^*), is defined as,

Now, we perform the conventional K-means clustering (Likas et al., 2003) to the first weighted time-varying eigenvector, W¹(u^*), for u^* ∈ [0, 1].

2.3. Algorithm

Here, we present a practical algorithm of the proposed method.

Here, we conduct simulation studies to confirm the performance of the proposed method.

3.1. Simulation setting

We generate 50 curves from K different processes, {$X_j^{(k)}(t)$}, for k = 1, …, K. Each curve length is 100 non-stationary time series, and [$X_{(1)}^1(t),\dots ,X_{50}^{(1)}(t),\dots ,X_1^{(K)}(t),\dots ,X_{50}^{(K)}(t)$] are input curves for each clustering method.

The simulation settings are similar to that of Fryzlewicz and Ombao (2009) with small modifications.

1. Case 1 – Nonstationary autoregressive processes with abruptly changing parameters

where ε_j(t) ~ N(0, 1), and the parameters are defined as follows:

• (a) Two groups (K = 2) with

  $$\begin{array}{ll}{\phi }_1^{(1)}(t)=\{\begin{array}{ll}0.6,\hfill & t=1,\dots ,50,\hfill \\ -0.6,\hfill & t=51,\dots ,100,\hfill \end{array}\hfill & {\phi }_1^{(2)}(t)=\{\begin{array}{ll}-0.9,\hfill & t=1,\dots ,50,\hfill \\ 0.9,\hfill & t=51,\dots ,100,\hfill \end{array}\hfill \\ {\phi }_2^{(1)}(t)=-0.6,\hfill & {\phi }_2^{(2)}(t)=-\mathrm{0.81.}\hfill \end{array}$$

• (b) Two groups (K = 2) with

  $$\begin{array}{ll}{\phi }_1^{(1)}(t)=\{\begin{array}{ll}0.6,\hfill & t=1,\dots ,33,\hfill \\ -0.6,\hfill & t=34,\dots ,66,\hfill \\ 0.6,\hfill & t=67,\dots ,100,\hfill \end{array}\hfill & {\phi }_1^{(2)}(t)=\{\begin{array}{ll}-0.9,\hfill & t=1,\dots ,33,\hfill \\ 0.9,\hfill & t=34,\dots ,66,\hfill \\ -0.9,\hfill & t=67,\dots ,100,\hfill \end{array}\hfill \\ {\phi }_2^{(1)}(t)=-0.6,\hfill & {\phi }_2^{(2)}(t)=-\mathrm{0.81.}\hfill \end{array}$$

• (c) Three groups (K = 3) with

  $$\begin{array}{l}{\phi }_1^{(1)}(t)=\{\begin{array}{ll}0.9,\hfill & t=1,\dots ,50,\hfill \\ -0.9,\hfill & t=51,\dots ,100,\hfill \end{array}\\ {\phi }_1^{(2)}(t)=\{\begin{array}{ll}-0.9,\hfill & t=1,\dots ,50,\hfill \\ 0.9,\hfill & t=51,\dots ,100,\hfill \end{array}\\ {\phi }_1^{(3)}(t)=\{\begin{array}{ll}0.3,\hfill & t=1,\dots ,50,\hfill \\ -0.3,\hfill & t=51,\dots ,100,\hfill \end{array}\\ {\phi }_2^{(1)}(t)=-0.9,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }{\phi }_2^{(2)}(t)=-0.9,\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }\text{and\hspace{0.28em}}\mathrm{\hspace{0.17em} }\mathrm{\hspace{0.17em} }{\phi }_2^{(3)}(t)=-\mathrm{0.3.}\end{array}$$

2. Case 2 – Nonstationary autoregressive processes with sinusoidal waves

For j = 1, …, 50, curves are generated from the following two processes,

where ε_j(t) ~ N(0, σ²), and the parameters are

• (a) ${\phi }_1(t)=\{\begin{array}{ll}-0.9,\hfill & t=1,\dots ,33,\hfill \\ 0.9,\hfill & t=34,\dots ,66,\hfill \\ -0.9\hfill & t=67,\dots ,100,\hfill \end{array}$ and φ₂(t) = −0.9.

• (b) ${\phi }_1(t)=\{\begin{array}{ll}-0.9,\hfill & t=1,\dots ,25,\hfill \\ 0.9,\hfill & t=26,\dots ,50,\hfill \\ -0.9\hfill & t=51,\dots ,75,\hfill \\ 0.9,\hfill & t=76,\dots ,100,\hfill \end{array}$ and φ₂(t) = −0.9.

The sample curves generated from Case 1-(a) and Case 2-(a) are shown in Figure 2.

3.2. Results

In each case, 100 Monte Carlo simulations are conducted, and the average performances are presented in the following tables. As an evaluation measure, the correct classification rate (CCR) and the adjusted Rand index (aRand) of Hubert and Arabie (1985) are considered. aRand measures the correspondence between two partitions on how object pairs are classified in the contingency table. A larger aRand value indicates a higher similarity between two partitions.

For comparison, we consider two conventional clustering methods along with the proposed time-dependent FDPC (TFDPC) clustering method. We apply the K-means clustering and K-centres functional clustering method (kCFC) of Chiou and Li (2007) to the simulation datasets. kCFC is also based on the FPCA and can be applied to longitudinal data such as AMI data. The main difference between their method and ours is that we consider the FDPCA, which is more suitable for the time series data that is correlated in time, while kCFC is based on the time domain PCA

From the results, we observe that the proposed TFDPC clustering method works best in Case 1. However, in Case 1-(c), all three methods works poorly and there is no significant difference between the methods.

In Case 2 with small noise variance, σ² = 1.5², the CCR of all three methods are close to 0.9, and K-means performs well in both (a) and (b). As the noise variance increases, the performance of all methods deteriorate, but the proposed method does not deteriorate much and works best.

From the simulation results, we conclude that the conventional methods also works well in well-separated cases, but the proposed one works best in some non-stationary autoregressive processes.

4.1. Clustering result

The proposed time-varying clustering method is applied in order to assign the load patterns to different clusters. Hourly AMI data were measured for 668 residential customers in Seoul, South Korea from January 1, 2012 to October 31, 2014. The data was sourced from the Korea Electric Power Corporation. For fast computation, we converted hourly data into a three-day averaged time series. Therefore, the number of time points in each AMI dataset is T = 345.

We first performed the augmented Dickey–Fuller test to confirm the non-stationarity of the data (Said and Dickey, 1984). We find that the null hypothesis cannot be rejected for 189 out of the total 668 customers, implying that some datasets in the AMI time series are non-stationary

Before applying the method, we first normalized each time series. For each AMI dataset, all its measurements are normalized in the range of [0, 1] by using its maximum value as the reference power (Xu et al., 2015). The randomly selected three AMI datasets are presented in Figure 3. The top row represents raw AMI data and the bottom row shows the corresponding normalized data. After normalization, the values are distributed from 0 to 1, although the patterns are retained.

We now perform the TFDPC clustering method as follows: We first segment the data into 10 blocks, each with 93 observations and the number of shift is set to 28. Figure 4 plots the first two eigenvalues, ξ̂¹(u, ω) and ξ̂²(u, ω). The center frequencies are brighter than other frequencies in all time points. Further, in the first time-varying spectrum, a relatively higher spectral power is observed periodically.

Now, we compute the first weighted time-varying eigenvector, W¹(u), as defined in (2.6) and perform K-means clustering with K = 3. Figure 5 shows the first weighted time-varying eigenvector and re-ordered eigenvector according to the K-means clustering result. All 668 households are clustered in three groups: 201, 320, and 147 households. We observe that the first and the third group display different bright patterns, while the second group contains households that have relatively constant spectral power over time.

Based on the clustering results in Figure 5, we plot the average AMI data for each group in Figure 6. The proposed method clearly classifies households according to the pattern of energy usage. Cluster 1 contains 201 households, and the average AMI time series in this group has relatively high usage during the winter season. By contrast, cluster 3, which contains 147 households, shows a high usage pattern during the summer season and relatively low energy usage in the winter season. Cluster 2 contains 320 households, of which energy usage is high during both the summer and winter, although relatively higher during the summer season.

For comparison, we also apply conventional clustering methods to the normalized AMI data. We apply K-means to the normalized AMI data, and the clustering results are presented in the left plot in Figure 7. Even when we normalize the AMI time series, the K-means method simply classifies the data according to the amount of usage. Therefore, all cluster groups show similar patterns, but with different magnitudes. We also consider kCFC of Chiou and Li (2007). The second plot in Figure 7 shows the kCFC results, and all three groups have relatively higher AMI values during the summer season.

4.2. Application of the electricity price system

The policy of differentiating tariff rates by season and hour involves charging higher rates during peak seasons/hours and lower rates during non-peak seasons/hours. Higher rates are applied in the summer and winter season, and during peak hours. Spring and autumn, as well as sub-peak and off-peak hours are subject to lower tariff rates. Table 3 presents the electricity price system before tariff reform, and Table 4 lists the electricity tariff rates after tariff reform.

Figure 8 presents the box-plot of the electricity prices for each cluster group before tariff reform, after tariff reform, and the differences. As the charges depend on the season, we plot the results according to the season. We observe three outcomes:

• in summer, the charge on cluster 1, which is the high usage group during the winter season, decreases after tariff reform,

• in winter, the mean charge on cluster 3, the lowest usage group during the winter, decreases after tariff reform, while it increases for the other cluster groups,

• in spring and fall, most households’ electricity bills increase.

Therefore, we expect that the proposed clustering results can be used to validate the reform of the progressive electricity tariff system in South Korea.

In this paper, we propose a new clustering method, the TFDPC clustering method. The proposed method is based on the time-varying FDPCA, and we apply the K-means clustering method to the time-varying eigenanalysis results. We applied the method to residential energy usage and obtained meaningful clustering groups compared to the existing methods. The proposed method was further applied to the electricity price system in South Korea to validate the reform of the progressive electricity tariff system.

From a practical viewpoint, we believe that the proposed method can be extended as follows,

• The clustering results can be applied to characterize residential energy use if more detailed information on residential energy usage is provided. Then, the electricity tariff system can be more precisely customized.

• Other clustering methods can also be applied to the time-varying eigenvector. Comparing these results can offer deeper insights.