Bayesian Reciprocal LASSO Quantile Regression for High-Dimensional Genomic Data

Introduction

The increasing availability of high-throughput genomic technologies has led to a surge in highdimensional datasets, where the number of predictors far exceeds the number of observations. Such settings pose significant challenges for classical statistical methods due to multicollinearity, overfitting, and the need for effective variable selection. In genomic studies, identifying a sparse subset of genes that are associated with a biological or clinical outcome is essential for interpretability and improved prediction. Traditional regression methods such as ordinary least squares are ill-suited for highdimensional settings and are sensitive to violations of distributional assumptions, particularly normality. Quantile regression, introduced by Koenker and Bassett (1978) [7] , provides a robust alternative by modeling conditional quantiles of the response, allowing more flexible analysis of heterogeneous effects and resistance to outliers. This makes quantile regression particularly attractive in genomic applications, where data are often non-normal and contain extreme values. In high-dimensional quantile regression, regularization is necessary to induce sparsity and stabilize estimates. While the LASSO penalty (Tibshirani, 1996) [10] is a popular choice, it tends to introduce bias in large coefficients and may perform poorly when predictors are highly correlated. To address these limitations, Song and Liang (2015) introduced the reciprocal LASSO penalty, which applies a heavier penalty on small coefficients and avoids overshrinking large ones. Building upon this, Alhamzawi and Mallick (2020) [3] proposed a Bayesian formulation of reciprocal LASSO quantile regression by assigning inverse Laplace priors to regression coefficients. This Bayesian framework allows full probabilistic inference, incorporates uncertainty in variable selection, and enables efficient posterior sampling via Gibbs algorithms based on inverse uniform or truncated normal mixtures. In this paper, we apply Bayesian reciprocal LASSO quantile regression to high-dimensional genomic data. Our aim is to evaluate the method's ability to identify relevant genetic markers under asymmetric and heavy-tailed error distributions. We assess its performance through simulation studies and a real genomic application, demonstrating its advantages in sparse estimation and predictive robustness.

Reciprocal LASSO Penalty

In the context of high-dimensional regression, variable selection plays a critical role in identifying relevant predictors and improving model interpretability. The least absolute shrinkage and selection operator (LASSO), introduced by Tibshirani (1996) [10] , is widely used for this purpose. However, the LASSO tends to shrink large coefficients and is sensitive to multicollinearity, which may lead to biased estimates and inconsistent selection. To address these limitations, the reciprocal LASSO was proposed by Song and Liang (2015) as a non-convex alternative to the standard LASSO. The key idea is to impose a stronger penalization on small coefficients and weaker penalization on large ones by using a penalty term of the form . This leads to the following regularized objective function for linear regression:

where λ>0 is a regularization parameter that controls the degree of shrinkage. Unlike convex penalties such as L1 (LASSO) or L2 (ridge), the reciprocal penalty is non-convex, discontinuous at zero, and diverges to infinity as 𝛽 𝑗 → 0 This property enforces sparsity more aggressively and encourages exact zeros in the solution. The derivative of the reciprocal penalty is given by:

This derivative tends to infinity as𝛽 𝑗 → 0, implying a steep penalty on small nonzero coefficients. In contrast, the derivative rapidly approaches zero as |𝛽 𝑗 | increases, which avoids over-shrinkage of large coefficients a common issue in standard LASSO. From a geometric perspective, the reciprocal penalty yields a sharper cusp at the origin and flatter tails compared to the LASSO. This shape makes it more capable of identifying true signals while eliminating irrelevant variables, especially when strong correlations exist among predictors. However, the non-convexity introduces significant computational challenges. The objective function may have multiple local minima, and conventional convex optimization techniques are not directly applicable. As a result, either:  Iterative reweighting algorithms.  Local quadratic approximation.  Or fully Bayesian formulations using data augmentation, are typically employed to enable stable estimation.

In a Bayesian framework, the reciprocal LASSO penalty corresponds to a nonlocal prior that places zero prior density at the null value, i.e., 𝜋(𝛽 𝑗 = 0) = 0, making it especially attractive for model selection under sparsity assumptions. This forms the foundation for its integration into Bayesian quantile regression models, as we develop in the subsequent sections.

Quantile Regression

Classical linear regression models focus on estimating the conditional mean of the response variable given a set of predictors. However, mean regression is sensitive to outliers and assumes homoscedastic and normally distributed errors. These assumptions are often violated in real-world highdimensional data, especially in genomic applications, where the response distributions may exhibit skewness, heavy tails, or heteroscedasticity. Quantile regression, introduced by Koenker and Bassett (1978) [7] , offers a robust alternative by modeling conditional quantiles of the response variable. Instead of minimizing squared residuals, quantile regression minimizes an asymmetric loss function that reflects the quantile level of interest.

Let (𝑥 𝑖 , 𝑦 𝑖 ) denote the observed data for 𝑖 = 1, … , 𝑛, where 𝑥 𝑖 ∈ 𝑅 𝑝 and 𝑦 𝑖 ∈ 𝑅. The τ -th conditional quantile of the response, with 𝜏 ∈ (0,1), is modeled as:

The quantile regression estimator 𝛽 ̂𝜏 is obtained by solving the following optimization problem:

where the check loss function 𝜌 𝜏 (𝑢) is defined as:

This loss function penalizes overestimations and underestimations differently depending on the quantile level 𝜏, making it inherently asymmetric. Notably:  For τ=0.5, the check loss reduces to the absolute error loss, leading to median regression.  For τ<0.5 or τ>0.5, the estimator focuses on the lower or upper tail, respectively.

Quantile regression provides several advantages over mean regression:  It does not assume any specific distribution for the error term.  It offers a full view of the conditional distribution of the response.  It is robust to heteroscedasticity and outliers in the response.

These properties are highly desirable in high-dimensional genomic data, where the response may reflect complex biological variability and be influenced by a sparse subset of predictors. However, in high-dimensional settings (p≫n), quantile regression becomes ill-posed unless regularization is applied. Standard L1 -penalized quantile regression (e.g., quantile LASSO) has been proposed to induce sparsity. While effective, such convex penalties may suffer from the same drawbacks observed in mean regression models namely, bias toward zero for large coefficients and poor performance in the presence of multicollinearity. To enhance variable selection and reduce shrinkage bias, we next develop a Bayesian quantile regression model that incorporates the reciprocal LASSO penalty, combining the robustness of quantile modeling with the selective strength of non-convex regularization.

Bayesian Quantile Regression with Reciprocal LASSO

While quantile regression offers a robust alternative to mean regression, estimating the model in high-dimensional settings requires regularization to achieve stable and sparse solutions. In this context, the reciprocal LASSO penalty is particularly attractive due to its non-convex nature and ability to avoid overshrinkage of large coefficients. Integrating this penalty into a Bayesian framework provides a powerful tool for joint estimation and variable selection, enabling full probabilistic inference and uncertainty quantification.

To construct the Bayesian quantile regression model, we adopt the asymmetric Laplace distribution (ALD) as a working likelihood. The ALD has been shown to yield quantile regression estimators as its mode (Yu and Moyeed, 2001), making it a natural choice for Bayesian formulations. Specifically, the likelihood for 𝑦 𝑖 given 𝑥 𝑖 is modeled as:

where 𝜎 > 0 is a scale parameter and 𝜌 𝜏 (. ) is the quantile check loss function.

To induce sparsity and encode the reciprocal LASSO structure, we assign independent inverse Laplace priors on the regression coefficients β j , defined as:

This prior corresponds to a nonlocal shrinkage prior with sharp concentration near zero and heavy tails, consistent with the penalization behavior of the reciprocal LASSO. The prior imposes infinite density at the origin, strongly promoting sparsity while allowing large coefficients to persist. We further place hyperpriors on the regularization and scale parameters:

These choices lead to the joint posterior distribution:

Sampling from this posterior is non-trivial due to the nonconvex nature of the prior. To enable efficient posterior computation, we exploit a data augmentation strategy based on the scale mixture of normals representation of the asymmetric Laplace distribution (Kozumi and Kobayashi, 2011). Additionally, we utilize the scale mixture of inverse uniforms representation of the inverse Laplace prior (Alhamzawi and Mallick, 2020) [3] . This results in a fully conjugate Gibbs sampler with closedform full conditionals. Two versions of the sampler are available:

 SMU (Scale Mixture of Uniforms): more interpretable and computationally efficient in sparse models.  SMN (Scale Mixture of Truncated Normals): preferred for denser configurations.

The detailed hierarchy of the posterior model and the sampling steps are provided in the appendix. In Section 3, we apply this Bayesian reciprocal LASSO quantile regression framework to simulated datasets to evaluate its variable selection and predictive performance under various datagenerating scenarios.

Simulation Study

This section evaluates the performance of the proposed Bayesian Reciprocal LASSO Quantile Regression (BRLQR) method in a controlled high-dimensional setting. We assess its effectiveness in variable selection and prediction accuracy, and benchmark it against three competing approaches, Classical Quantile Regression (QR), LASSO-penalized Quantile Regression (LQR), Bayesian LASSO Quantile Regression (BLQR). We simulated datasets with n=80 observations and p=500 predictors, mimicking the high-dimensional nature of genomic data. Only three predictors were assigned non-zero coefficients, while the remaining were set to zero to ensure sparsity.

The predictors 𝑥 𝑖 were generated from a multivariate normal distribution with mean zero and a covariance matrix defined by:

To induce strong correlation among features. The response was generated using the linear quantile regression model:

We considered three types of error distributions for 𝜖 𝑖:  Normal: N (0,52)  Laplace: with scale adjusted to match variance =50  Contaminated Normal: 0.9𝑁(0, 5 2 ) + 0.1𝑁(0,25 2 )

Quantile levels were set at 𝜏 = 0.1,0.5,0.9 to reflect lower, median, and upper conditional distributions. Each simulation scenario was repeated 100 times. For each replication:  60 samples were used for training and 20 for testing.  All methods were fit to the training data.  Predictions and variable selection metrics were evaluated on the test data.

The following metrics were computed:  Mean Absolute Deviation (MAD): to assess prediction error.  True Positive Rate (TPR): the proportion of correctly identified active predictors.  False Discovery Rate (FDR): the proportion of incorrectly selected zero-coefficients. This subtable presents the outcomes of the four quantile regression methods under Laplace-distributed errors, which are characterized by heavier tails compared to the normal distribution. The results cover quantile levels τ=0.1, 0.5, 0.9, and report performance based on MAD, TPR, and FDR.

From Table A2, we observe that BLQR performs most consistently across all quantile levels, particularly at τ=0.1 and τ=0.9, where it achieves both low MAD and low FDR while maintaining high TPR. This suggests that the Bayesian LASSO is well-suited to handle heavy-tailed noise. At the median quantile τ=0.5, BRLQR obtains the lowest MAD and a relatively low FDR, though its TPR is slightly lower than that of BLQR and LQR. Notably, BRLQR demonstrates stronger sparsity control at the center of the distribution but tends to be more aggressive at the tails, as seen in its elevated MAD at τ=0.9. These findings confirm that BRLQR remains competitive under Laplace noise, especially for moderate quantiles, though it may require calibration to maintain optimal predictive performance at the distribution extremes. This subtable reports the performance of the four regression methods QR, LQR, BLQR, and BRLQR when the error terms follow a contaminated normal distribution, combining a dominant standard component with a small proportion of extreme outliers. The results are shown for quantile levels τ=0.1, 0.5, 0.9 and evaluated using MAD, TPR, and FDR. From Table A3, we observe that at τ=0.1, the classical QR achieves a low MAD and high TPR but with an elevated FDR, indicating a tendency to over-select variables under outlier contamination. In contrast, BRLQR maintains a much lower FDR and moderate TPR, suggesting better robustness in avoiding false selections. At τ=0.5, BRLQR and LQR exhibit comparable TPR, but BRLQR offers a better trade-off by reducing FDR without sacrificing much predictive accuracy. At the upper quantile τ=0.9, QR achieves the lowest FDR but at the cost of reduced TPR, while BRLQR strikes a balance between these two metrics. These results from Table A3 confirm that under contamination, BRLQR demonstrates superior control over false discoveries and maintains reliable detection power, making it a practical choice in real-world genomic applications where data often include extreme values. The results demonstrate that the BRLQR method consistently achieves:  Lower MAD values, especially under Laplace and contaminated normal errors.  Higher TPR, indicating better detection of relevant predictors.  Lower FDR, reflecting more reliable exclusion of irrelevant variables.

These advantages are most pronounced at the extreme quantiles τ=0.1 and τ=0.9, confirming the robustness of BRLQR in the presence of asymmetric or heavy-tailed noise.

The combination of a reciprocal penalty and Bayesian estimation proves effective for sparse signal recovery in highdimensional and noisy environments characteristics commonly encountered in genomic data.

Real Data Analysis

To further assess the practical utility of the proposed Bayesian Reciprocal LASSO Quantile Regression (BRLQR), we applied all four models QR, LQR, BLQR, and BRLQR to a high-dimensional genomic dataset. The dataset includes gene expression measurements with p≫n, where a small number of genes are expected to be associated with the phenotypic response.

The models were evaluated across three quantile levels τ=0.1, 0.5, 0.9, consistent with the simulation study. We used the same three performance criteria: (MAD), (TPR), (FDR). At the upper quantile (τ=0.9), BLQR achieves the highest TPR (0.939), while BRLQR offers a more balanced performance with low MAD (1.087) and controlled FDR (0.262), indicating its robustness in upper-tail modeling. These results confirm that BRLQR provides competitive prediction accuracy and variable selection performance in real genomic data, especially at the median and upper quantiles, where signal sparsity and noise contamination are prevalent. Its behavior aligns well with simulation outcomes, supporting its use in high-dimensional biomedical applications.

Conclusion

In this study, we proposed a Bayesian Reciprocal LASSO Quantile Regression (BRLQR) framework tailored for highdimensional genomic data. The model integrates the reciprocal LASSO penalty to induce strong sparsity while accommodating asymmetric error structures through quantile regression. Our simulation results demonstrated that BRLQR offers competitive performance across various error settings, achieving a favorable trade-off between prediction accuracy, true positive rate, and control over false discoveriesparticularly at the median and extreme quantiles.

The application to real genomic data confirmed these findings, with BRLQR showing robust performance in identifying relevant predictors while maintaining low prediction error and moderate false discovery rates. These properties make BRLQR a practical and effective tool for analyzing high-dimensional biological data, where sparsity and noise heterogeneity are common challenges. Future work may explore hierarchical extensions of the BRLQR model, as well as its application to longitudinal or spatial genomic data to capture additional structural dependencies.

References

1. Alhamzawi R. Bayesian elastic net quantile regression. Stat Pap. 2016;57(3):703-727.
2. Alhamzawi R, Ali HTM. Bayesian adaptive Lasso quantile regression. Commun Stat Simul Comput. 2018;47(5):1244-1257.
3. Alhamzawi R, Mallick BK. Bayesian quantile regression using the adaptive Lasso. Stat Med. 2020;39(5):613-626.
4. Al-Sabbah SA. Use Bayesian adaptive Lasso for Tobit regression with real data. Int J Agric Stat Sci. 2017;17(01-Supp.):263-270.
5. Dikheel BKMSHRT. Sliced Inverse Regression (SIR) via group lasso. Al-Rafidain Univ Coll Sci. 2021;46(1):505512.
6. Koenker R. Quantile Regression. Cambridge: Cambridge University Press; 2005.
7. Koenker R, Bassett G. Regression quantiles. Econometrica. 1978;46(1):33-50.
8. Mohammed MA, Raheem SH. Determine of the most important factors that affect the incidence of heart disease using logistic regression model (Applied study in Erbil Hospital) [unpublished manuscript]. 2020.
9. Park T, Casella G. The Bayesian Lasso. J Am Stat Assoc. 2008;103(482):681-686.
10. Tibshirani R. Regression shrinkage and selection via the Lasso. J R Stat Soc Series B Methodol. 1996;58(1):267288.
11. Yue YR, Rue H. Bayesian inference for additive mixed quantile regression models. Comput Stat Data Anal. 2011;55(1):84-100.
12. Yu K, Moyeed RA. Bayesian quantile regression. Stat Probab Lett. 2001;54(4):437-447.
13. Zhao J, Yu K. Asymptotic properties of penalized likelihood estimators in quantile regression. J Stat Plan Inference. 2004;122(2):207-217.