A structured model with both single-index and varying coefficients is a powerful tool in modeling high dimensional data. It has been widely used because the single-index can overcome the curse of dimensionality and varying coefficients can allow nonlinear interaction effects in the model. For high dimensional index vectors, variable selection becomes an important question in the model building process. In this paper, we propose an efficient estimation and a variable selection method based on a smoothing spline approach in a partially linear single-index-coefficient regression model. We also propose an efficient algorithm for simultaneously estimating the coefficient functions in a data-adaptive lower-dimensional approximation space and selecting significant variables in the index with the adaptive LASSO penalty. The empirical performance of the proposed method is illustrated with simulated and real data examples.
In a generalized linear model, the regression function
where
Allowing a linear association between the response and covariates in SICM yields a partially linear single-index-coefficient model (PLSICM),
The semiparametric model (
For the estimation of index parameters and unknown coefficient function in PLSICM, various estimation methods have been proposed such as local linear method, kernel method, B-splines, empirical likelihood method, and penalized splines (Xia and Li, 1999; Xue and Wang, 2012; Huang, 2012; Yang
In this paper, we propose a simple estimation method based on a smoothing splines approach for selecting variables in the index and simultaneously estimating unknown nonparametric functions, regression parameters in the partial terms, and index parameters. Smoothing splines have advantages over other nonparametric estimation methods because they can avoid the problem of choice and the placements of knots.
The paper is organized as follows. Section 2 presents a smoothing splines technique of a data-adaptive lower-dimensional approximation in a penalized likelihood method in an ordinary nonparametric setting so as to speed up the computation of function estimates without any loss of performance and extend it to PLSICM. We propose a simple and efficient method for estimating and selecting index parameters based on the penalization approach. Simulated and real data examples are illustrated to evaluate the performance of the proposed method in Section 3 and Section 4 respectively. Performance comparisons are made with different penalties and other estimation methods. The paper is concluded with a discussion in Section 5.
Suppose that the data (
where
For given
For estimation of
For fixed
where {
Inserting (
Selecting appropriate smoothing parameters in nonparametric function estimation is important because they determine the performance of the function estimates. Kim and Gu (2004) suggested the following modification to the generalized cross-validation (GCV) score,
where
For given the current estimates of
Then we have
Therefore, we derive the following penalized least squares functional for
which is to be minimized, where
For the penalty in (
Given
The estimator of
Step 1. Start with an initial estimator of
Step 2. Given
Step 3. Given
Step 4. Repeat step 2 and 3 until convergence. The final estimates of
If there are more than one unknown functions to estimate, a Gauss-Seidel type algorithm (backfitting algorithm) estimates each of the coefficient functions iteratively. Note that the classical smoothing splines on the product domain are calculated via smoothing spline ANOVA decomposition. However, a similar decomposition cannot be used to obtain varying-coefficient function estimates in our model due to the association between predictors
Bayesian confidence intervals for a minimizer of the penalized likelihood functional were first derived by Wahba (1983) from the Bayes model of a penalized likelihood estimator. Consider
where
A simulation study was conducted to evaluate the performance of the proposed estimators. The following criteria are considered to assess the performance of the selection of index covariates; IZ represents the average number of nonzero index parameters that are incorrectly selected as zero; CZ is the average number of zero index parameters that are correctly selected. The biases and standard deviations of the estimates of
In each example, we carried out 200 simulations and the sample size in each simulation is set to
and
A simple example of the SICM is the partially linear single-index model, which can be written as
where
In this example, we consider the following SICM
where
This example is the same as the Example 2, but with
This example is the same as the Example 2, but with an additional 20 noise covariates, so that
Table 1 showed that the adaptive LASSO performed well in estimating and selecting the significant variables of
We demonstrated the proposed method to the body fat dataset. The data contain 252 observations with 14 variables, in which the response variable is the percentage of body fat determined by the underwater weighting technique. The covariates include age, weight, height, and 10 body circumference measurements (neck, chest, abdomen, hip, thigh, knee, ankle, biceps, forearm, and wrist). The dataset is available from the website (http://lib.stat.cmu.edu/datasets/bodyfat). After excluding 6 outliers similar to Peng and Huang (2011), we adopt several structured models, including PLSICM to identify the association between the percentage of body fat and other covariates, by selecting the index parameters. We considered the PLSIM, SIM, and PLSICM as,
where
Table 3 showed the estimation results of the body fat data for each model. For comparison, the results of Feng and Xue (2015) (CP), Peng and Huang (2011) (SIM-SCAD), and a linear model with SCAD penalty (LM-SCAD) were also presented. Age was found to have a nonzero constant effect on the percentage of body fat. Among ten body circumference variables, neck, abdomen, hip, and wrist were selected by the proposed method. All models found that abdomen was the most important measurement for the prediction of the percentage of body fat. Previous results showed that the wrist was more important than hip and thigh circumferences; however, our models showed that hip was more important than the wrist to predict the percentage of body fat. Figure 1 showed the coefficient function estimates and its 95% Bayesian confidence intervals of the PLSICM. The estimated coefficient function of index of circumferences
and the fitted model showed consistent results to PLSICM with an additional result that the height was reversely related to the percentage of body fat at the cost of the reduced multiple
In this paper we proposed a simple nonparametric estimation method that employs smoothing splines to estimate varying-coefficient functions and select index parameters by shrinkage methods in PLSICM. This was based on the availability of reliable information that some predictors are linearly associated with the response; however, a single-index, a possible linear combination of predictors, is related to the response to different degrees according to the other predictors. The application of the splines method overcomes drawbacks suffered by high dimensional kernels. We therefore suggest a simple method based on a lower dimensional approximation in smoothing splines computation to estimate varying-coefficient functions as well as to select index parameters simultaneously using existing techniques. Simulation results have shown that the proposed method outperformed previously proposed methods. Future studies should investigate the parallel nonparametric estimation methods in partially linear single-index varying coefficient mixed effect models in the framework of smoothing spline regression. Figure 1: Varying coefficient function estimates in partially linear single-index-coefficient model of body fat data.
This research was supported by 2017 Research Grant from Kangwon National University (No.52017 0503) and Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2015R1D1A3A01019998).
Bias, SD, and model selection results (IZ, CZ) for the estimates of
Method | IZ | CZ | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
Bias | SD | Bias | SD | Bias | SD | |||||
Example 1 | 100 | SS-LASSO | 0.0272 | 0.1257 | 0.0221 | 0.0932 | 0.0431 | 0.2005 | 0.000 | 1.495 |
SS-SCAD | 0.0289 | 0.1226 | 0.0158 | 0.0894 | 0.0407 | 0.1991 | 0.030 | 1.840 | ||
SS-ALASSO | 0.0044 | 0.0903 | 0.0228 | 0.0753 | 0.0258 | 0.1430 | 0.060 | 4.900 | ||
200 | SS-LASSO | 0.0023 | 0.0501 | 0.0079 | 0.0355 | 0.0241 | 0.1689 | 0.000 | 2.015 | |
SS-SCAD | 0.0049 | 0.0471 | 0.0039 | 0.0307 | 0.0216 | 0.1640 | 0.000 | 2.740 | ||
SS-ALASSO | 0.0017 | 0.0551 | 0.0159 | 0.0620 | 0.0310 | 0.1920 | 0.035 | 4.935 | ||
300 | SS-LASSO | −0.0014 | 0.0080 | 0.0028 | 0.0073 | 0.0004 | 0.0101 | 0.000 | 2.375 | |
SS-SCAD | 0.0005 | 0.0076 | 0.0003 | 0.0100 | −0.0005 | 0.0098 | 0.000 | 2.850 | ||
SS-ALASSO | −0.0029 | 0.0104 | 0.0058 | 0.0146 | 0.0005 | 0.0107 | 0.000 | 5.000 | ||
Example 2 | 100 | SS-LASSO | 0.0097 | 0.0880 | 0.0172 | 0.0586 | 0.0143 | 0.0934 | 0.005 | 2.430 |
SS-SCAD | 0.0236 | 0.0924 | 0.0286 | 0.0952 | 0.0163 | 0.1213 | 0.035 | 3.095 | ||
SS-ALASSO | 0.0225 | 0.1396 | 0.0439 | 0.1136 | 0.0187 | 0.1061 | 0.135 | 4.805 | ||
200 | SS-LASSO | 0.0012 | 0.0452 | 0.0026 | 0.0086 | 0.0021 | 0.0121 | 0.000 | 3.480 | |
SS-SCAD | 0.0069 | 0.0542 | −0.0014 | 0.0095 | 0.0018 | 0.0181 | 0.000 | 3.885 | ||
SS-ALASSO | −0.0029 | 0.0062 | 0.0044 | 0.0094 | 0.0013 | 0.0047 | 0.000 | 5.000 | ||
300 | SS-LASSO | −0.0015 | 0.0028 | 0.0024 | 0.0043 | 0.0005 | 0.0030 | 0.000 | 3.355 | |
SS-SCAD | −0.0001 | 0.0024 | 0.0005 | 0.0039 | −0.0001 | 0.0029 | 0.000 | 3.920 | ||
SS-ALASSO | −0.0020 | 0.0031 | 0.0037 | 0.0053 | 0.0003 | 0.0030 | 0.000 | 5.000 | ||
Example 3 | 100 | SS-LASSO | 0.0834 | 0.1481 | 0.0635 | 0.1317 | 0.0342 | 0.1920 | 0.035 | 0.965 |
SS-SCAD | 0.0962 | 0.1526 | 0.0515 | 0.1352 | 0.0365 | 0.1908 | 0.045 | 1.335 | ||
SS-ALASSO | 0.0645 | 0.1617 | 0.0653 | 0.1389 | 0.0225 | 0.1665 | 0.195 | 3.850 | ||
200 | SS-LASSO | 0.0213 | 0.1112 | 0.0301 | 0.0916 | 0.0070 | 0.0912 | 0.000 | 0.780 | |
SS-SCAD | 0.0255 | 0.1113 | 0.0261 | 0.0906 | 0.0025 | 0.0795 | 0.010 | 1.135 | ||
SS-ALASSO | 0.0100 | 0.0913 | 0.0228 | 0.0749 | 0.0021 | 0.0734 | 0.031 | 4.740 | ||
300 | SS-LASSO | −0.0010 | 0.0025 | 0.0087 | 0.0413 | 0.0047 | 0.0327 | 0.000 | 0.750 | |
SS-SCAD | 0.0012 | 0.0246 | 0.0063 | 0.0403 | 0.0038 | 0.0324 | 0.000 | 1.060 | ||
SS-ALASSO | −0.0048 | 0.0271 | 0.0120 | 0.0463 | 0.0024 | 0.0323 | 0.000 | 4.920 | ||
Example 4 | 100 | SS-LASSO | −0.0020 | 0.0110 | 0.0081 | 0.0166 | −0.0005 | 0.0102 | 0.000 | 2.225 |
SS-SCAD | 0.0009 | 0.0080 | 0.0032 | 0.0145 | −0.0023 | 0.0094 | 0.000 | 3.420 | ||
SS-ALASSO | −0.0079 | 0.0164 | 0.0159 | 0.0272 | 0.0013 | 0.0119 | 0.000 | 5.000 | ||
200 | SS-LASSO | −0.0027 | 0.0054 | 0.0040 | 0.0080 | 0.0015 | 0.0059 | 0.000 | 2.890 | |
SS-SCAD | −0.0003 | 0.0042 | 0.0009 | 0.0063 | 0.0002 | 0.0055 | 0.000 | 3.552 | ||
SS-ALASSO | −0.0009 | 0.0552 | 0.0093 | 0.0242 | 0.0050 | 0.0495 | 0.005 | 5.000 | ||
300 | SS-LASSO | −0.0015 | 0.0034 | 0.0025 | 0.0047 | 0.0005 | 0.0035 | 0.000 | 3.453 | |
SS-SCAD | 0.0003 | 0.0028 | 0.00005 | 0.0041 | −0.0004 | 0.0033 | 0.000 | 3.760 | ||
SS-ALASSO | −0.0037 | 0.0068 | 0.0068 | 0.0109 | 0.0008 | 0.0042 | 0.015 | 5.000 |
SD = standard deviation; IZ = the average number of nonzero index parameters that are incorrectly selected as zero; CZ = the average number of zero index parameters that are correctly selected; SS = smoothing spline; LASSO = least absolute shrinkage and selection operator; SCAD = smoothly clipped absolute deviation; ALASSO = adaptive LASSO.
Bias and SD of
Method | RASE_{1} | RASE_{2} | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
Bias | SD | Bias | SD | Bias | SD | |||||
Example 1 | 100 | SS-LASSO | −0.0018 | 0.0263 | −0.0037 | 0.0177 | 0.0017 | 0.0163 | 0.0360 | - |
SS-SCAD | −0.0020 | 0.0225 | −0.0029 | 0.0162 | 0.0004 | 0.0150 | 0.0376 | - | ||
SS-ALASSO | −0.0011 | 0.0200 | −0.0030 | 0.0156 | 0.0018 | 0.0137 | 0.0331 | - | ||
200 | SS-LASSO | 0.00006 | 0.0099 | −0.0021 | 0.1017 | 0.0001 | 0.0090 | 0.0279 | - | |
SS-SCAD | 0.0003 | 0.0099 | −0.0022 | 0.0103 | 0.00004 | 0.0090 | 0.0239 | - | ||
SS-ALASSO | 0.0001 | 0.0099 | −0.0030 | 0.0115 | −0.0003 | 0.0090 | 0.0295 | - | ||
300 | SS-LASSO | −0.0003 | 0.0075 | 0.0013 | 0.0073 | −0.00002 | 0.0077 | 0.0213 | - | |
SS-SCAD | −0.0003 | 0.0075 | 0.0013 | 0.0074 | −0.0002 | 0.0077 | 0.0192 | - | ||
SS-ALASSO | −0.0005 | 0.0093 | 0.0016 | 0.0073 | −0.00006 | 0.0077 | 0.0192 | - | ||
Example 2 | 100 | SS-LASSO | 0.0041 | 0.0342 | 0.0055 | 0.0260 | −0.0015 | 0.0306 | 0.0341 | 0.0520 |
SS-SCAD | 0.0009 | 0.0539 | 0.0129 | 0.0484 | 0.0011 | 0.0457 | 0.0329 | 0.0474 | ||
SS-ALASSO | 0.0022 | 0.0479 | 0.0082 | 0.0375 | 0.0054 | 0.0411 | 0.0310 | 0.0424 | ||
200 | SS-LASSO | −0.0016 | 0.0110 | −0.0009 | 0.0105 | −0.0004 | 0.0120 | 0.0003 | 0.0005 | |
SS-SCAD | −0.5110 | 0.0145 | −0.0004 | 0.0110 | −0.0001 | 0.0118 | 0.0003 | 0.0005 | ||
SS-ALASSO | −0.0013 | 0.0100 | −0.0003 | 0.0099 | −0.0008 | 0.0098 | 0.0003 | 0.0004 | ||
300 | SS-LASSO | −0.0001 | 0.0086 | −0.00004 | 0.0087 | −0.0005 | 0.0083 | 0.0152 | 0.0241 | |
SS-SCAD | −0.0001 | 0.0086 | −0.0009 | 0.0086 | −0.0004 | 0.0083 | 0.0151 | 0.0227 | ||
SS-ALASSO | −0.0001 | 0.0085 | 0.0004 | 0.0086 | −0.0003 | 0.0084 | 0.0151 | 0.0233 | ||
Example 3 | 100 | SS-LASSO | 0.0139 | 0.1885 | 0.0488 | 0.1700 | 0.0062 | 0.1965 | 0.3651 | 0.6765 |
SS-SCAD | 0.0146 | 0.1808 | 0.0486 | 0.1822 | 0.0079 | 0.2018 | 0.3770 | 0.7419 | ||
SS-ALASSO | 0.0223 | 0.1850 | 0.0435 | 0.1695 | −0.0058 | 0.1861 | 0.2818 | 0.4043 | ||
200 | SS-LASSO | −0.0027 | 0.1008 | 0.0133 | 0.1030 | −0.0141 | 0.1033 | 0.1966 | 0.3609 | |
SS-SCAD | −0.0033 | 0.1013 | 0.0122 | 0.1030 | −0.0122 | 0.1025 | 0.1962 | 0.3695 | ||
SS-ALASSO | −0.0019 | 0.0977 | 0.0163 | 0.1026 | −0.0192 | 0.1031 | 0.1387 | 0.2197 | ||
300 | SS-LASSO | 0.0061 | 0.0880 | 0.0037 | 0.0877 | −0.0048 | 0.0835 | 0.1375 | 0.2291 | |
SS-SCAD | 0.0063 | 0.0881 | 0.0038 | 0.0877 | −0.0046 | 0.0837 | 0.1347 | 0.2307 | ||
SS-ALASSO | 0.0038 | 0.0858 | 0.0073 | 0.0864 | −0.0011 | 0.0835 | 0.1101 | 0.1472 | ||
Example 4 | 100 | SS-LASSO | −0.0012 | 0.0168 | 0.0025 | 0.0182 | −0.0025 | 0.0202 | 0.0537 | 0.1086 |
SS-SCAD | −0.0004 | 0.0170 | 0.0043 | 0.0169 | −0.0020 | 0.0196 | 0.0366 | 0.0575 | ||
SS-ALASSO | −0.0008 | 0.0158 | 0.0057 | 0.0156 | 0.0023 | 0.0197 | 0.0313 | 0.0428 | ||
200 | SS-LASSO | −0.0008 | 0.0105 | −0.0010 | 0.0103 | −0.0027 | 0.0105 | 0.0286 | 0.0617 | |
SS-SCAD | −0.0008 | 0.0107 | −0.0008 | 0.0104 | −0.0025 | 0.0107 | 0.0222 | 0.0408 | ||
SS-ALASSO | −0.0001 | 0.0107 | −0.00006 | 0.0102 | −0.0003 | 0.0122 | 0.0180 | 0.0322 | ||
300 | SS-LASSO | −0.0005 | 0.0092 | 0.0009 | 0.0091 | 0.0004 | 0.0080 | 0.0183 | 0.0325 | |
SS-SCAD | −0.0004 | 0.0091 | 0.0007 | 0.0090 | 0.0001 | 0.0080 | 0.0173 | 0.0272 | ||
SS-ALASSO | −0.0006 | 0.0091 | −0.0010 | 0.0087 | −0.00005 | 0.0079 | 0.0155 | 0.0246 |
SD = standard deviation; RASE = square root of the average squared error; SS = smoothing spline; LASSO = least absolute shrinkage and selection operator; SCAD = smoothly clipped absolute deviation; ALASSO = adaptive LASSO.
Results for the body fat data
Method | SS-ALASSO | CP | SIM-SCAD | LM-SCAD | |||
---|---|---|---|---|---|---|---|
PLSIM | SIM | PLSICM | PLSICM2 | ||||
Age | 0.0124 | 0.0000 | 0.0199 | 0.0095 | 0.0099 | 0.0149 | 0.0489 |
Weight | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1457 |
Height | 0.0000 | 0.0000 | 0.0000 | −0.0366 | 0.0000 | 0.0000 | −0.0395 |
Neck | −0.1197 | −0.1828 | −0.1120 | −0.1504 | −0.0968 | −0.1691 | −0.1408 |
Chest | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | −0.0943 |
Abdomen | 0.9663 | 0.9614 | 0.9804 | 0.9659 | 0.9689 | 0.9606 | 0.7663 |
Hip | −0.2128 | −0.1516 | 0.1276 | −0.2075 | 0.0000 | 0.0000 | −0.3638 |
Thigh | 0.0000 | 0.0924 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.1461 |
Knee | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
Ankle | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
Biceps | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 |
Forearm | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0000 | 0.0413 |
Wrist | −0.0807 | −0.1032 | −0.0900 | −0.0346 | −0.2278 | −0.2202 | −0.1186 |
0.6797 | 0.6818 | 0.6874 | 0.6815 | 0.6691 | 0.6738 | 0.6148 |
SS = smoothing spline; ALASSO = adaptive least absolute shrinkage and selection operator; SIM = single-index model; SCAD = smoothly clipped absolute deviation; LM-SCAD = linear model with SCAD; PLSIM = partially linear SIM; PLSICM = partially linear single.-index-coefficient model.