Expectile regression with adaptive elastic net penalty

Introduction

Expectile regression has emerged as a powerful statistical modeling approach with applications in econometrics, medicine, and high-dimensional data analysis. Unlike traditional mean regression, expectile regression provides a comprehensive understanding of the conditional distribution of a response variable, making it particularly valuable in scenarios where asymmetric relationships exist [1] . Due to its flexibility, expectile regression has been widely applied in risk analysis, survival models, and predictive analytics. Despite its advantages, a key challenge in expectile regression is the accurate selection of relevant covariates, particularly in high-dimensional settings and in the presence of multicollinearity. Conventional penalized regression techniques, such as SCAD and Elastic Net, have been introduced to address this issue [2,3] . However, these methods may suffer from instability in variable selection, leading to suboptimal predictive accuracy and model interpretability. For instance, SCAD tends to yield inconsistent selection results in highdimensional datasets, whereas Elastic Net balances variable selection and regularization but may retain unnecessary covariates, reducing model sparsity.

To overcome these limitations, we propose Expectile Regression with Adaptive Elastic Net (ER-AdEN), a novel penalization technique that integrates the strengths of Lasso and Ridge penalties while incorporating adaptive weighting to enhance variable selection and model stability. The adaptive weights dynamically adjust penalty contributions based on initial coefficient estimates, allowing ER-AdEN to achieve superior sparsity control and robustness against multicollinearity [4] . This approach not only improves the accuracy of variable selection but also ensures more stable and interpretable models, particularly in high-dimensional settings. To evaluate the performance of the proposed method, we conduct an extensive Monte Carlo simulation study and apply ER-AdEN to a real-world diabetes dataset. The effectiveness of ER-AdEN is benchmarked against ER-SCAD and ER-Elastic Net, using key evaluation metrics such as Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), and Expected Prediction Error (EPE). Our results demonstrate that ER-AdEN consistently outperforms existing methods in terms of predictive accuracy, sparsity control, and computational efficiency. This study contributes to the advancement of expectile regression modeling by providing a more efficient and stable approach for variable selection, particularly in highdimensional data applications. The findings have significant implications for medical analytics, financial risk modeling, and predictive modeling in complex datasets, offering a robust tool for researchers and practitioners dealing with highdimensional and multicollinear data.

Methods

Linear regression with the adaptive elastic net penalty

The Adaptive Elastic Net penalty is an advanced approach for simultaneous shrinkage and covariate selection in regression models [5] . This method extends the Elastic Net by introducing adaptive weights, which enhance its ability to select relevant variables, especially in the presence of multicollinearity. The Adaptive Elastic Net estimates are obtained by solving the following minimization problem:

Where, Y is the n × 1 vector of centered outcomes. X is the 𝑛 × 𝑝 matrix of standardized covariates. β is the p × 1 vector of parameters to be estimated. 𝜆 1 and 𝜆 2 are the regularization parameters for the Lasso and Ridge components, respectively. 𝑤 𝑗 * are adaptive weights, typically defined as 𝑤 𝑗 * = 1 |𝛽 ̂𝑗| 𝛾 , where 𝛽 ̂𝑗 are initial estimates, and γ > 0 is a tuning parameter. Compared to the Lasso, which is symmetric, continuous, and non-decreasing, and the Reciprocal Lasso, which is noncontinuous and divergent at zero, the Adaptive Elastic Net offers a balance between variable selection and stability [6] . By combining the sparsity-inducing property of Lasso with the regularization effect of Ridge, it ensures a more robust selection process [3] . The flexibility and adaptability of the Adaptive Elastic Net make it particularly effective in handling high-dimensional datasets and collinear variables [5] , offering superior performance in both accuracy and model interpretability. This approach has been increasingly adopted in various fields, including econometrics, medicine, and machine learning.

Expectile regression with the adaptive elastic net penalty

In large datasets, selecting active covariates and constructing a stable, interpretable model are critical for effective regression analysis. To achieve these objectives in Expectile Regression (ER), we propose ER with the Adaptive Elastic Net (AdEN) penalty, which addresses multicollinearity and enhances variable selection [5] . The optimization problem for ER with AdEN is defined as:

Is the asymmetric squared loss function for expectile regression [1] . 𝜆 1 and 𝜆 2 are regularization parameters for the Ridge and adaptive Lasso components, respectively. 𝑤 𝑗 * are adaptive weights, typically defined as 𝑤 𝑗 * = 1 |𝛽 ̂𝑗| 𝛾 , where 𝛽 ̂𝑗 are initial estimates, and γ > 0 is a tuning parameter. This formulation provides a flexible framework to balance variable selection and multicollinearity handling, making it particularly suitable for high-dimensional data. While solving the ER-AdEN optimization problem involves challenges, such as balancing the Ridge and adaptive Lasso terms, modern convex optimization techniques and coordinate descent algorithms enable efficient solutions [7] . Unlike rLasso, the AdEN penalty is continuous and convex, ensuring convergence to a global minimum and providing more stable solutions [8] . This makes the Adaptive Elastic Net a robust alternative for expectile regression with large, complex datasets.

Computational strategy for expectile regression with adaptive elastic net

The optimization problem for penalized expectile regression with the Adaptive Elastic Net (AdEN) penalty is formulated as:

Where, 𝜂 and 𝜉 are positive slack variables introduced to handle the asymmetric loss function for expectile regression [1,9] . 𝜆 1 and 𝜆 2 are the regularization parameters for Ridge and adaptive Lasso penalties. 𝑤 𝑗 * are adaptive weights, typically defined as

, where 𝛽 ̂𝑗 are initial estimates, and γ > 0 is a tuning parameter. This penalized expectile regression problem can be transformed into a quadratic programming problem. By introducing positive slack variables η and ξ, the optimization problem becomes convex and tractable. The resulting objective function combines the smooth quadratic loss with adaptive elastic penalties for efficient estimation [10] .

Algorithm

To solve the optimization problem, we utilize the following steps: 1. Start with initial estimates 𝛽 ̂𝑗 for calculating the adaptive weights 𝑤 𝑗 * .

2. The penalized objective function is minimized iteratively for each parameter 𝛽 𝑗 , updating the estimates while keeping the remaining parameters fixed. At each step, the adaptive weights are updated to reflect the current estimates. 3. To enhance convergence and avoid local minima, we apply the SAA algorithm. This technique iteratively refines the parameter estimates by simulating the annealing process, allowing the algorithm to explore a broader solution space. 4. Repeat the updates until the objective function stabilizes within a predefined tolerance level.

Advantages of the ER-AdEN computational strategy. The problem remains convex, ensuring convergence to a global minimum. The combination of coordinate descent and SAA ensures computational efficiency, even for high-dimensional datasets. Adaptive weights allow for dynamic adjustment of penalties, improving variable selection and model stability. This computational strategy provides a robust and scalable framework for implementing expectile regression with the Adaptive Elastic Net penalty, suitable for large and complex datasets [11] .

Simulation Studies

In this section, we highlight the effectiveness of the Adaptive Elastic Net model for expectile regression (ER-AdEN) through Monte Carlo experiments. The computations are implemented using R programming, leveraging a modified version of the "HQREG" package to solve the models.

Example 1: Low-Dimensional Case (p<n)

The data is generated from the following linear model:

Where, 𝑥 is sampled from a multivariate normal distribution 𝑁 𝑝 (0, Σ), with Σ 𝑖,𝑗 = 𝜌 |𝑖-𝑗| and 𝜌 = 0.4. True coefficients are set as: β = (2.5, -1.8,0,0,1.2,0,0,0) ⊤ is a scaling parameter controlling the signal-to-noise ratio. Two types of random errors (ϵ) are considered: 𝑁(0,1): Gaussian errors. 𝜒 2 (4): Asymmetric errors. This data generation process follows standard practices commonly adopted in statistical simulation studies for evaluating regression models under controlled conditions. Signal-to-noise scaling parameter: σ=2.5. Sample size: n=1000. Number of repetitions: k=400. The simulation process is designed as follows. First, a dataset comprising 1200 observations is generated, with the initial 900 observations used as in-sample data and the remaining 300 reserved for out-of-sample evaluation. The (ER-AdEN) model is then estimated using the optimal λ, determined via 10-fold cross-validation. For comparison, additional models are also estimated, including the elastic-net penalized least squares regression (LSR-EN), the elastic-net penalized Huber loss regression (HR-EN), and the conventional expectile regression (ER). The performance of these models is evaluated based on three key aspects: variable selection accuracy, coefficient estimation precision, and prediction ability. Finally, the entire process is repeated 400 times, and the results, including the average and standard deviation, are reported to ensure robust and reliable comparisons across methods.

In this section, we fix α=0.9 when selecting the tuning parameter λ. The selection of λ is based on 10-fold crossvalidation, using the 1SE (Standard Error) criteria to determine the optimal λ for all models. Simulation results for Gaussian 𝜖~𝑁(0,1) and Chi-squared (ϵ∼χ2(4) errors are provided in Tables 2 and3, respectively. 3 highlights the performance under Chi-squared errors. ER-AdEN demonstrates a clear advantage in terms of predictive accuracy (RMSE, MAE, EPE) and variable selection metrics, particularly under asymmetric error distributions. The ER-AdEN model provides superior performance across all metrics under both symmetric (Gaussian) and asymmetric (Chi-squared) error distributions. It outperforms ER-SCAD and ER-Elastic Net, in variable selection, prediction accuracy, and computational efficiency, making it a reliable choice for expectile regression tasks.

Example 2: High-Dimensional Case (p>n)

For the high-dimensional case (p>n), we use the same model as in the low-dimensional case. However, there are two key differences: The number of predictors is set to p=100, while the sample size is reduced to n=80. Three different proportions of active variables are considered. The first 15 predictors are active, with the coefficients of the first five variables set to (3, 1.5, 2, 2.5, 1). The remaining coefficients are zeros (𝛽 𝑗 = 0 𝑓𝑜𝑟 𝑗 = 16, … ,100). The first 10 predictors are active, with coefficients similar to Situation 1. The remaining coefficients are zeros (𝛽 𝑗 = 0 𝑓𝑜𝑟 𝑗 = 11, … ,100 ). Only the first five predictors are active, with the same coefficients as above. The rest of the coefficients are zeros (𝛽 𝑗 = 0 𝑓𝑜𝑟 𝑗 = 6, … ,100). The simulation process follows the same steps as the low-dimensional case with the following adjustments. The first 60 observations are used as in-sample data, while the remaining 20 are used for out-of-sample evaluation. The ER model is excluded from this analysis as it is not suitable for high-dimensional settings (p>n). The following tables present the optimal tuning parameter λ and the performance metrics for the models considered. The performance metrics for the simulation study, including RMSE, MAE, and EPE, are summarized in Tables 5 and6. These tables highlight the comparative performance of ER-AdEN against alternative methods under various expectile levels (θ = 0.1, 0.5, 0.9). ER-AdEN offers additional flexibility through its regularization properties. Compared to ER-SCAD and ER-Elastic Net, ER-AdEN consistently achieves lower RMSE and MAE values while maintaining fewer false positives (FP), highlighting its efficiency and reliability in both Gaussian and Chi-squared error distributions. Its robust performance in high-dimensional settings establishes it as a reliable choice for scenarios where p>n As depicted in Figure 2, the MAE results further confirm the improved predictive performance of ER-AdEN across various expectile levels (θ =0.1, 0.5, 0.9). ER-AdEN demonstrates lower MAE compared to ER-SCAD and ER-Elastic Net, indicating its stability and reliability for accurate predictions. Figure 3 highlights the trends in EPE across expectile levels (θ=0.1, 0.5, 0.9), emphasizing the stability of ER-AdEN at higher expectile levels (θ=0.7, 0.9). The consistent performance of ER-AdEN compared to ER-SCAD and ER-Elastic Net demonstrates its robustness in handling diverse data distributions and complex modeling scenarios.

Real data analysis

Medical data analysis plays a critical role in understanding and managing diseases like diabetes. One of the major challenges in this field is identifying the most relevant features from a large set of clinical and demographic variables. To address this problem, we propose the ER-AdEN model, which integrates the (ER-AdEN) penalty into expectile regression for effective variable selection and predictive modeling. We demonstrate the efficacy of the ER-AdEN model using a publicly available diabetes dataset from the UCI Machine Learning Repository. The dataset includes observations from 768 individuals and features a diverse set of demographic, clinical, and biochemical parameters: age, body mass index (BMI), blood pressure, insulin levels, glucose concentration, skin thickness, diabetes pedigree function, and pregnancy count. The outcome variable is the level of glycated hemoglobin (HbA1c), a critical indicator of blood sugar control. For model evaluation, the data were partitioned into training and testing sets in a 3:2 ratio, with 460 observations used for training and 308 for testing. To evaluate the performance of the proposed method, we compare ER-AdEN against two alternative approaches: ER-SCAD and ER-Elastic Net. Three expectile levels (θ ) 0.1, 0.5, and 0.9 are considered to capture different aspects of the conditional distribution of HbA1c. The 5-fold cross-validation results, summarized in Table 4, show that the ER-AdEN method consistently outperforms ER-SCAD and ER-Elastic Net in terms of variable selection accuracy and predictive performance. These findings, along with insights from simulation studies, provide strong evidence supporting the effectiveness of the Adaptive Elastic Net penalty in expectile regression for medical data analysis.

Conclusions

This study introduced a novel penalized expectile regression method, ER-AdEN, which integrates the Adaptive Elastic Net penalty to improve variable selection and prediction accuracy in expectile regression models. The ER-AdEN approach effectively balances the strengths of ridge and lasso penalties, allowing for precise identification of relevant predictors while maintaining computational efficiency. Comprehensive simulation studies and real-world analysis of the diabetes dataset demonstrated that ER-AdEN consistently outperformed alternative methods, including ER-SCAD and ER-Elastic Net, in terms of variable selection accuracy and predictive performance. These results underline the potential of ER-AdEN as a robust tool for regression analysis, particularly in medical applications such as diabetes management and prediction.

References

1. Newey WK, Powell JL. Asymmetric least squares estimation and testing. Econometrica. 1987;55(4):819847. DOI: 10.2307/1911031.
2. Fan J, Li R. Variable selection via nonconcave penalized likelihood and its oracle properties. J Am Stat Assoc. 2001;96(456):1348-1360. DOI: 10.1198/016214501753382273.
3. Zou H, Hastie T. Regularization and variable selection via the elastic net. JR Stat Soc Ser B Stat Methodol. 2005;67(2):301-320. DOI: 10.1111/j.1467-9868.2005.00503.x.
4. AL-Sabbah SA, Raheem SH. Use Bayesian adaptive lasso for Tobit regression with real data. Int J Agric Stat Sci. 2021;17(Suppl 1):2169-2181. DocID: https://connectjournals.com/03899.2021.17.2169.
5. Zou H, Zhang HH. On the adaptive elastic-net with a diverging number of parameters. Ann Stat. 2009;37(4):1733-1751. DOI: 10.1214/08-AOS625.
6. Tibshirani R. Regression shrinkage and selection via the lasso. JR Stat Soc Ser B Stat Methodol. 1996;58(1):267288. DOI: 10.1111/j.2517-6161.1996.tb02080.x.
7. Friedman J, Hastie T, Tibshirani R. Pathwise coordinate optimization. Ann Appl Stat. 2007;1(2):302-332. DOI: 10.1214/07-AOAS131.
8. Song S, Liang H. A new penalty for sparse highdimensional regression. Biometrika. 2015;102(3):639652. DOI: 10.1093/biomet/asv029.
9. Raheem SH, Alhusseini FH H, Alshaybawee T. Modelling volatility in financial time series using ARCH models. Int J Innov Creat Chang. 2022;12(7):248-261.
10. Econometric Society. Econometrica: Journal of the Econometric Society. Econometrica, University of Chicago, 1955, p. 23.