Much work has been done in regression diagnostics since Cook’s distance (Cook, 1977) was introduced forty years ago. The concept of regression diagnostics, considered in the classical linear model, has been extended to the Box-Cox transformation model (Box and Cox, 1964), ridge regression model (Hoerl and Kennard, 1970), and nonparametric regression models such as spline smoothing model, local polynomial regression, semiparametric model, and varying coefficient model. All regression diagnostic results in various regression models are done under the assumption of “large n, small p”, i.e., the number of unknown parameters which are estimated is less than the number of samples in the data.

High-dimensional data (small n, large p) are very popular in the areas of information technology, bioinformatics, astronomy, and finance. Classical statistical inferences such as the least squares estimation in the linear model cannot be used in high-dimensional data. Recently, many methodological and computational advances have allowed high-dimensional data to be efficiently analyzed; in addition, least absolute shrinkage and selection operator (LASSO), as introduced by Tibshirani (1996), remains an important statistical tools for high-dimensional data.

In this paper, we study diagnostic issues in the LASSO regression model for high-dimensional data. Kim et al. (2015) recently derived an approximate version of Cook’s distance in the LASSO regression; however, it is based on the “large n, small p” assumption. Cook’s distance in the LASSO model suggested by Kim et al. (2015) cannot be directly used in high-dimensional data because the covariance estimator of the LASSO estimator, which is necessary in defining a version of Cook’s distance, is not easily derived in the high-dimensional data. Further, under high dimensional setup, we are more interested in influential observations on the variable selection rather than the influence on estimators. Studies on the diagnostic measures for the LASSO model in the high-dimensional data are relatively few. Among them, Zhao et al. (2013) proposed an influence measure for marginal correlations between the response and all the predictors, and Jang and Anderson-Cook (2017) suggested influence plots for LASSO. In this paper, we focus the influence of one or few observations on the variable selection by the LASSO using the deletion method. The influence on the variable selection in the classical model via the least squares was studied by Bae et al. (2017), and the selection of a smoothing parameter in the robust LASSO was done by Kim and Lee (2017).

In high-dimensional data, the influence of one or few observations on some estimators could be more serious and important because the number of observations is small compared to the “large n, small p” setup. We investigate that a variable selection result based on the LASSO regression can be significantly different if one or few observations are deleted. LASSO estimates often do not have a analytic form; therefore, we assume the design matrix is orthogonal.

This paper is organized as follows. In Section 2, the difference between LASSO estimates based on the full samples and the partial samples after deleting some observations, respectively, is derived under simple setup of the design. Numerical studies based on artificial data sets are done in Section 3, and an illustrative example based on a real data set is given in Section 4. Finally, concluding remarks are given in Section 5.

2.1. LASSO estimator based on partial samples

Consider a simple linear regression model with no intercept, i.e.,

where ∑ x_i = 0, $\sum x_i^2=1$, and ∑ y_i = 0. Then, it can be shown in Tibshirani (1996), for example, that the LASSO estimator of β under model (2.1) is

where sgn(x) denotes the sign of x and β̂= ∑ x_iy_i is the least squares estimator (LSE) of β. Now, let K = {i₁, i₂, . . . , i_k} be an index set of size k, and let β̂_L(K)(λ) be a LASSO estimator of β based on (n – k) observations after deleting k observations in K. Then, we have the following result.

Proposition 2

Without loss of generality, we assume that β̂ > 0. Then, β̂_L(λ) = (β̂ – λ)₊. Now, we further assume that β̂ ≥ ∑_i∈K x_iy_i ≥ 0. Then, it is easy to show that

To see the relationship between β̂_L(λ) and β̂_L(K)(λ) in terms of λ (seeFigure 1), we consider for the single case deletion i.e., K = {i}. First, if λ ≥ β̂, then β̂_L(λ) = β̂_L(i)(λ) = 0. Second, if λ ≤ β̂ – x_iy_i, then${\widehat{\beta }}_{L(i)}(\lambda )=({\widehat{\beta }}_L(\lambda )-x_i{y}_i)/{\sum }_{j\ne i}{x}_j^2$. Finally, if β̂ – x_iy_i < λ < β̂, then β̂_L(λ) = β̂ – λ while β̂_L(i)(λ) = 0, i.e., if β̂ – x_iy_i < λ < β̂, then the deletion of the i^th observation results in not selecting the covariate. Therefore, if λ is chosen in this range, the i^th observation is said to be influential on the feature selection.

3.1. Influence on coefficient estimates

Consider a simple linear regression model

where ∑ x_i = 0, $\sum x_i^2=1$, and ∑ y_i = 0. We generate n = 20 random numbers by the following steps.

• Step 1: Generate x_i from N(0, 1)

• Step 2: Generate ɛ_i from N(0, 0.1²)

• Step 3: Let Y_i = 0.1x_i + ɛ_i

• Step 4: Do centering and scaling to meet Σ x_i = 0, $\sum x_i^2=1$, and ∑ y_i = 0.

The LSE based on the artificial data is β̂ = 0.346. When the 20^th observation is deleted, β̂ –x₂₀y₂₀ = 0.179. Therefore, if 0.179 < λ < 0.346, β̂_L(λ) = 0.346 – λ; however, β̂_L(20)(λ) = 0.

3.2. Influence on variable selections

In this numerical study, we investigate three important aspects of the influence of the i^th observation on the variable selection in the “small n, large p” setup. First, we want to see that the number of selected variables via the LASSO is very sensitive to the deletion of one observation. The number of selected variables has very important implication in the sense of the prediction and the estimation of the degrees of freedom. Second, we want to see the variable selection performance of the LASSO in the outlier model (a model with one outlying observation) when the tuning parameter is given so that the LASSO selects the true number of variables. Third, we want to see how correctly the LASSO selects variables in the outlier model when the tuning parameter is estimated by the cross-validation. To do this, we generate random numbers from the model given as follows.

Consider a linear regression model

where β = (5, 5, 5, 5, 0, . . . , 0)^t is a 100-dimensional vector. We generate n = 20 random numbers by the following steps.

• Step 1: Generate each X_j, j = 1, . . . , 100 from N(5, 1).

• Step 2: Generate ɛ_i from N(0, 0.1²).

• Step 3: Let $Y_i={\mathbf{x}}_i^{\prime }\mathit{\beta }+{\varepsilon }_i$.

• Simulation (I) - Sensitivity of LASSO to a single observation

  In this simulation study, we want to see that the number of selected variables via the LASSO is very sensitive to the deletion of one observation. Table 1 shows the LASSO estimators based on the full samples (i.e., n = 20) selected variables 1, 2, 3, and 4, which are true nonzero variables. However, the LASSO estimators based on 19 observations after deleting the i^th (i = 1, 2, . . . , 20) observation show different selection results. For example, if we delete the 2^nd observation, the LASSO selected 10 variables; in addition, the LASSO selected just one variable if we deleted the 16^th observation.

• Simulation (II) - Sensitivity of LASSO to a single outlier (λ : given)

  Here, we want to see the variable selection performance of the LASSO in the outlier model (a model with one outlying observation) when the tuning parameter is given so that the LASSO selects the 4 variables, i.e., true number of variables. We considered three outlier models, where each model contains outlier with y₁ + c, c = 10, 30, 50. Table 2 shows the average proportion of the number of correctly selected variables among 4 variables selected by the LASSO out of 100 replications. Table 2 also shows the shows that the proportion of selecting a true model is about 0.5 when we delete the 1st observation (outlier); however, the proportion is less than 0.3 either when we use the full data or when we delete the other observation than the outlier.

• Simulation (III) - Sensitivity of LASSO to a single outlier (λ : estimated)

  Finally, we want to see the variable selection performance of the LASSO in the outlier model when the tuning parameter is estimated by the cross-validation, for example. To do this we compute the average proportion of the number of the selected variables by LASSO out of 100 replications. If the number of selected variables less than four, we assumed that 4 variables are selected by the LASSO. Table 3 shows the average proportion of the number of correctly selected variables. We considered three outlier models, where each model contains an outlier with y₁ + c, c = 10, 30, 50. Table 3 also indicates the proportion of selecting true model is above 0.5 when we delete the 1st observation (outlier), but the proportion is less than 0.5 either when we use the full data or when we delete other observation than the outlier.

As an illustrative example for the influence of observations on the variable selection in LASSO, we use the brain aging data of Lu et al. (2004). This data set contains measurements of p = 403 genes and n = 30 human brain samples, and the response is the age of each human. We fit this data set by the LASSO based on the original sample (n = 30), and fit based on n = 29 observations after deleting the i^th (i = 1, . . . , 30) observation to see the influence of the i^th observation on the variable selection by the LASSO.

Table 4 shows that the LASSO selects 25 variables among 403 variables. If we delete one observation, then most cases result in selecting around 25 variables except the cases of deleting the 8^th, 9^th, and 19^th observation, respectively. We may conclude that those three observations are quite influential as far as the number of variable selection is concerned.

One or few observations could be very influential on estimators in “small p, large n” case, and this phenomenon becomes more serious in “small n, large p” case. In this paper, we investigate the influence of observations on the LASSO estimates and the selected variables by the LASSO. Also, we derived analytic expression for the influence of the i^th observation on the LASSO estimates in the simple linear regression. Simulation results show that the influence of an outlier is more serious in the high dimensional case than in the low dimensional case.

For further studies, it will be worth studying the basic building blocks which affect variable selection results. Despite difficulties, it is also necessary to modify the LASSO model to a robust LASSO that is not sensitive to outliers.

This work was supported by a 2-year Research Grant of Pusan National University.