Optimized variable selection via the penalized exponential loss function

Introduction

Variable selection is considered one of the fundamental issues in applied statistics and machine learning, as it aims to identify the most influential set of variables in explaining the statistical behavior of models, thereby achieving a balance between estimation accuracy and model simplicity. Penalized regression methods such as Lasso (Tibshirani, 1996) [1] and Elastic Net (Zou & Hastie, 2005) [2] have represented pioneering tools in this regard due to their ability to perform shrinkage and simultaneous variable selection. However, these traditional approaches can be strongly affected by the presence of outliers or non-ideal data distributions, which weakens their predictive performance and practical efficiency. To address this limitation, recent years have witnessed a growing tendency toward developing robust methods, by employing alternative loss functions such as the Huber or Tukey loss functions (Huber, 1981;Maronna et al., 2019) [4, 3] , which reduce the influence of atypical values and maintain estimation stability. In this context, exponential loss functions have attracted increasing attention due to their flexibility and adaptability to complex data structures (Li & Zhu, 2008) [4] . Building on these developments, this research proposes a new framework for robust variable selection under the name Penalized Exponential Loss Function (PELF), which combines the strength of penalized models in achieving regularization with the robustness property provided by exponential losses.

This approach is expected to improve the predictive performance of models and reduce estimation sensitivity to outliers, making it a promising alternative in practical applications, particularly in medical, economic, and environmental fields where contaminated or non-ideal data are frequently encountered.

The Proposed Method (PELF)

This research introduces the Penalized Exponential Loss Function (PELF) as a novel framework for robust variable selection in high-dimensional regression models. The main motivation behind this proposal is to overcome the limitations of traditional penalized regression approaches, which often exhibit sensitivity to outliers and irregular noise distributions. Unlike classical loss functions such as the quadratic loss, the exponential loss function provides a nonlinear dampening mechanism that significantly reduces the influence of extreme deviations in the response variable.

Formally, we start by defining the exponential loss function as follows:

where λ lambda denotes the residual term and γ gamma is a tuning parameter that controls the degree of robustness. Larger values of γ gamma lead to smoother penalization of residuals, while smaller values yield stronger robustness against outliers.

The PELF framework combines this robust loss with a penalty function applied to the regression coefficients. The optimization problem can be expressed as:

Where,  n: is the number of observations,  𝑦𝑖 :is the response variable for observation iii,  𝑥 𝑖 𝑇 𝛽: represents the linear predictor for the i-th observation,  𝛽 𝑗: The element j of the vector of coefficients β, i.e., the regression  λ =regularization parameter. This formulation integrates the robustness of the exponential loss with the sparsity-inducing property of the L1 penalty (Lasso), thereby enhancing the model's ability to perform reliable variable selection in the presence of contaminated or noisy data.

Why Classical Lasso is Not Robust

The classical Lasso estimator (Tibshirani, 1996) [1] is defined as:

Although Lasso is highly effective for variable selection, it remains sensitive to outliers because the quadratic loss function amplifies the effect of extreme residuals. As a result, even a small proportion of contaminated observations can substantially distort coefficient estimates and weaken predictive performance.

Robustness through PELF

The proposed PELF method addresses this limitation by replacing the quadratic loss in Lasso with the exponential loss function. This modification ensures that large residuals contribute less to the objective function, thereby providing natural robustness against outliers. At the same time, the L1 penalty preserves the ability to shrink irrelevant coefficients toward zero, maintaining efficient variable selection.

The final objective function of the PELF estimator can be written as:

This formulation yields several advantages  Robustness Outliers have limited impact due to the dampening effect of the exponential loss.  Sparsity non-informative variables are excluded by the L1 penalty, leading to simpler models.  Balance of accuracy and interpretability the method maintains predictive performance while ensuring that the selected model remains interpretable and resistant to noise.

Thus, the PELF approach provides a strong balance between robust estimation and efficient feature selection, making it particularly suitable for real world applications where data contamination and irregular noise are common.

Simulation study and Real data

This section presents the experimental evaluation of the proposed Penalized Exponential Loss Function (PELF) method in comparison with classical Lasso (Tibshirani, 1996) [

. The assessment focuses on the ability of the methods to achieve accurate variable selection and reliable prediction in the presence of outliers or non-ideal data. Custom R code was developed to implement the PELF method and generate solution paths, and the same framework was applied to Lasso to ensure a fair comparison. The evaluation relies on key criteria, including:  Mean Squared Error (MSE): to measure the predictive accuracy of the models.  Average number of zero coefficients (Ave0's): to assess the efficiency of variable selection.  Prediction Error (P.E.): which indicates the closeness of the estimated values to the true responses.

Simulation Experiments

Multiple simulation experiments were conducted to evaluate the performance of PELF relative to Lasso. The study considered different sample sizes (N=50, 100, 200, 300), with each scenario repeated (1000) times to ensure the stability and reliability of the results. The simulated datasets included outliers and noise to reflect realistic data conditions.

Simulation 1

The data were simulated from the following true linear model: 𝑌 = X T 𝛽 + 0.5 𝜀, (3.1) R = 1000 datasets were generated with sample sizes of (N=50, 100, 200, 300) observations, each including 15 predictors. The coefficients vector was set as

In this section, we conduct simulation studies to evaluate the finite-sample performance of our estimator. We choose N=50, 100, 200, 300, p = 15, and

where the (i, j) th element of,Ω 2 is 𝜌 |i-j | , 𝜌 = 0.5.

 Regarding the Mean Squared Error (MSE):

The MSE values obtained from the PELF-Lasso method were consistently lower than those of the traditional Lasso, indicating higher predictive accuracy. For instance, when the sample size reached 300, the MSE decreased from 0.571 in Lasso to 0.501 in PELF-Lasso. This demonstrates the capability of the exponential squared loss function to enhance estimation precision and reduce prediction errors.  Concerning the Average Number of Zero Coefficients (Ave. 0's): This measure represents the number of variables excluded from the model, that is, those whose coefficients were estimated as zero. The PELF-Lasso method excluded a greater number of variables compared to the traditional Lasso, indicating better efficiency in identifying only the significant predictors and eliminating irrelevant ones. This highlights its superiority in achieving a more parsimonious and accurate model in terms of variable selection.

 Effect of Sample Size (n): As the sample size increased, both methods showed a gradual decline in MSE, which is expected as larger samples generally yield more accurate estimates. However, the improvement rate in PELF-Lasso was more pronounced, suggesting greater stability and robustness, even with larger datasets.

Overall Conclusion

The findings confirm that the PELF-Lasso method demonstrates superior performance over the traditional Lasso method when applied to uncontaminated data, both in terms of predictive accuracy and variable selection efficiency. This underscores the effectiveness of employing the exponential squared loss function in achieving an optimal balance between statistical precision and model simplicity. Table 2 presents the results of the first example with a contamination level of 0.05, meaning that 5% of the data include outlier observations. The table compares the performance of the Lasso and PELF-Lasso methods in terms of Mean Squared Error (MSE) and the average number of zero coefficients (Ave. 0's) for different sample sizes (50, 100, 200, and 300).

The results clearly show that the PELF-Lasso method continues to outperform the traditional Lasso even in the presence of slight contamination in the data.

 Mean Squared Error (MSE):

The PELF-Lasso consistently achieves lower MSE values than the Lasso across all sample sizes. For example, when the sample size reached 300, the MSE decreased from 0.370951 in Lasso to 0.311514 in PELF-Lasso. This indicates that the proposed method is more stable and accurate under the influence of outliers, effectively reducing their negative impact on model estimation.

 Average Number of Zero Coefficients (Ave. 0's):

The PELF-Lasso method maintains its ability to select the most relevant variables and exclude the insignificant ones. It produced a higher number of zero coefficients compared to Lasso, especially for larger samples, demonstrating its efficiency in balancing model simplicity and accuracy even under noisy or partially contaminated data.  Effect of Sample Size (n): As the sample size increases, both methods exhibit a gradual reduction in MSE, which is expected due to improved estimation accuracy with more data. However, the decrease in MSE is more pronounced for the PELF-Lasso, showing that the proposed method benefits more effectively from larger samples and demonstrates higher stability and robustness.

Overall Conclusion

The results confirm that the PELF-Lasso method exhibits greater robustness than the traditional Lasso in the presence of mild contamination. It achieves lower prediction errors and better variable selection accuracy, highlighting the effectiveness of the exponential squared loss function in enhancing model stability and mitigating the influence of outliers. The results in Table 3 show that the PELF-Lasso method continues to outperform the traditional Lasso under Design 2 with a 5% contamination level.

Overall, the PELF-Lasso achieves lower MSE values across all sample sizes, indicating better predictive accuracy and robustness against outliers. It also produces a higher number of zero coefficients, showing improved ability to eliminate irrelevant variables and retain only the significant ones.

As the sample size increases, both methods improve, but the PELF-Lasso demonstrates more stable and efficient performance, confirming its effectiveness in handling mildly contaminated data.

Real data

Thalassemia is a hereditary blood disorder characterized by defective hemoglobin synthesis, inherited from the patient's ancestors. Thalassemia is classified into two main types: alpha and beta. The severity of alpha and beta thalassemia depends on the number of genes missing between the four alpha globin genes and the two beta globin genes. In 2013, approximately 208 million people were affected by thalassemia, of whom 4.7 million were severe. Males and females show similar rates of infection, and diagnosis is typically made through blood tests, which include a complete blood count, specialized hemoglobin tests, and genetic testing. Iron overload is a major complication in patients receiving chronic blood transfusions. Three decades have passed since allogeneic hematopoietic stem cell transplantation (HSCT) was performed to treat thalassemia. This procedure has now become a common treatment for the ultimate cure of thalassemia major, with more than 4,000 HSCTs performed worldwide. To confirm the effectiveness of our proposed approach, PELF-Lasso, we will examine actual data from thalassemia patients. R code will be used to compare PELF-Lasso with existing methods (Lasso). The RMSE for various estimation method at sample size N=110, showing that the PELF-Lasso method achieved lowest error.

From the above table, we notice the comparison of the two statistical models, Lasso, PELF-Lasso in a specific design (Design) the sample size N=350 and the results of comparison using MSE. We note that MSE-PELF-Lasso is the least, meaning it is the best model Table 1 shows that the PELF-Lasso method achieved lower performance compared to the other methods studied. We conclude from Figure 4.1 that the PELF-Lasso method has better performance in predicting actual results. Figure1, The RMSE for various estimation method at sample size N=110, showing that the PELF-Lasso method achieved lowest error.

Discussion

In this article, we propose a rigorous variable selection procedure via penalized regression. We examine the sample characteristics and robustness of the proposed estimators. Through theoretical and simulation results, we demonstrate the advantages of our proposed method. We also show that our proposed method can make a significant difference in the analysis of real data. More specifically, we show that our estimator has the highest breakdown point for the finite sample, and that the effect functions are limited for extreme values in either the response domain or the covariate domain. Huber PJ. Robust Statistics. New York: Wiley; 1981. 5. This formulation yields several advantages: 6. Robustness-Outliers have limited impact due to the dampening effect of the exponential loss. 7. Sparsity-Non-informative variables are excluded by the L1 penalty, leading to simpler models. 8. Balance of accuracy and interpretability-The method maintains predictive performance while ensuring that the selected model remains interpretable and resistant to noise. 9. Thus, the PELF approach provides a strong balance between robust estimation and efficient feature selection, making it particularly suitable for real-world applications where data contamination and irregular noise are common.

References