Principal component analysis as a preprocessing technique for multiple linear regression models; dealing with superfluous variables

Introduction

Most data analyst often deal with large-scale multidimensional data set with many variables which makes the calculations inconvenient. Often, several of these variables are highly correlated or convey overlapping information. They would want to reduce the number of variables but are not sure which variables to remove without losing information. Some data may include redundant or irrelevant features which will require reducing the dimensionality of the data while preserving its essential structure. Such data are called superfluous data as they contribute little to the overall explanatory power of the dataset. These redundant or superfluous variables do not contribute meaningful insight but instead increase data complexity, computational cost, and the likelihood of multicollinearity in predictive modeling. As a result, the accuracy, interpretability, and efficiency of data analysis or building predictive models may be compromised. There is the need for a statistical technique that can identify and reduce these redundant variables while preserving the key information within the data. Superfluous variables do not enhance the analytical outcome but may distort statistical results or obscure meaningful patterns within the data. The inclusion of highly correlated variables can increase the dimensionality of data unnecessarily, complicate model interpretation, and introduce multicollinearity in regression analyses (Mundfrom & Kay, 2018) [12] . In regression or classification models, multicollinearity caused by these redundant variables can make it difficult to determine the true effect of each predictor. Therefore, identifying and removing such variables is essential for maintaining the accuracy, reliability, and efficiency of data-driven decision-making. To address this problem, Principal Component Analysis (PCA) is employed as a dimensionality-reduction technique that transforms a set of correlated variables into a smaller number of uncorrelated components known as principal components (Syad, 2024). These components capture most of the variance present in the original variables while eliminating redundancy. Consequently, PCA simplifies complex datasets, enhances interpretability, and provides a clearer understanding of the underlying structure of the data. In this regard, the technique plays a crucial role in identifying and eliminating superfluous variables, thereby improving the quality and efficiency of modern statistical and data analyses (Fan et al., 2021) [3] . Principal component analysis (PCA) plays a vital role especially in information extraction and data simplification. It helps to eliminate noisy and redundant features, focusing the model on the most important patterns and improving performance. Principal component analysis is a widely used technique in predictive modelling like multiple linear regression. It is designed to simplify complex datasets, enhance interpretability and improve the performance of various predictive models. As datasets grow in size and complexity, especially in fields such as finance, healthcare, image processing, and natural language processing, the need for dimensionality reduction becomes increasingly critical. Principal component analysis addresses this need by reducing the number of variables (features) in a dataset while retaining the most meaningful information. When dealing with a lot of variables, they tend to move together in groups as more than one variable might be measuring the same driving principle of the dependent variable. Hence, we often have some sort of redundant information which we can remove and. Principal component analysis identifies this information. Recent studies have continued to demonstrate the usefulness of PCA as a robust method for dimensionality reduction and data interpretation across a wide range of fields. Yin and Zhou (2024) [16] revisited PCA's theoretical foundations in high-dimensional spaces, explaining how data normalization and covariance structures affect the interpretation of eigenvalues and eigenvectors. Ustuner (2024) [15] applied a randomized version of PCA (R-PCA) for hyperspectral image classification. By reducing high-dimensional image data into a smaller set of features while maintaining classification accuracy, the research by Ustuner (2024) [15] highlighted PCA's importance in remote sensing and machine learning. Li et al. (2024) [9] explored PCA in big data environments, demonstrating that weighted PCA methods can efficiently compress large multidimensional datasets without losing essential information. The principal component analysis cannot only help us to balance the trade-off between the number of variables and information loss but also to get a better sense for highdimensional data and develop a deeper understanding for the driving forces in a data set and thus, it is an important technique. Hence, this research is aimed at demonstrating how PCA simplifies complex data structures, improves computational efficiency, and enhances the reliability of statistical models. The primary goal is to use PCA is to identify the axes in the dataset that contain the most information, allowing for more efficient data analysis and improved performance in multiple linear regression model. The study will identify superfluous variables using correlation matrix, apply Principal Component Analysis (PCA) to the dataset and obtain the standard deviations of the principal components which are used to calculate the variance explained by each component, demonstrate how PCA transforms correlated variables into a smaller number of uncorrelated principal components that retain most of the original information and illustrate how PCA improves the accuracy, interpretability and efficiency of building a multiple linear regression model. The R Software (R Development Core Team) was used to implement the analysis using a dataset collected through personal interview of 115 people.

Materials and Methodology 2.1 Data Collection

The study population in this research are peoples from different areas of the Bayelsa State in Nigeria. The personal interview method of data collection (Hamed Taherdoost, 2021) [7] was used in this research. The researcher interviewed 115 persons from whom the information is being collected and considered for analysis. The respondents were required to fill their information such as gender, age, weight, height, number of children, status of smoking, status of drinking, region of residence, level of education, income earned, expenses on medicals, type of health facility, nature of job and body mass index (BMI). The attributes of the data are shown on Table 1. The body mass index based on the weight and height of the person.

Multiple Linear Regression Model

Multiple Linear Regression (MLR) is a statistical technique used to model the linear relationship between a dependent variable and two or more independent (or predictor) variables (Kaya Uyanık & Güler, 2013) [8] . The dependent variable represents the outcome of interest, while the independent variables represent the predictors or explanatory factors. Regression models are used to describe how changes in the independent variables influence the dependent variable, and they also serve as tools for prediction, forecasting, and decision-making. The multiple linear regression model specification is given as

Each coefficient 𝛽 𝑖 represents the expected change in Y due to a one unit change in 𝑋 𝑖 , while holding other variables constant. The R-squared and Adjusted R-squared measures of how well the model fits the data. R-squared indicates the proportion of the variance in the dependent variable explained by the independent variables. Multicollinearity occurs when two or more independent variables in a regression model are highly correlated.

Principal Component Analysis

The main use of principal component analysis is to balance the number of variables and the information loss from removing variables (Mishra et al., 2017). To reduce the information loss, we need to determine the importance or amount of information of a variable in our data set. For this, we can use the variance of a variable as a higher variability which indicates a signal while a lower variability indicates noise.

The principal component analysis works by linearly combining the variables in our data set and then transforming the newly derived variables to maximise their variance (that is the importance of representing the dependent variable). The linear combination of the variables is represented in a covariance matrix while the principal components are determined by the Eigenvectors and Eigenvalues of the covariance matrix. The principal component analysis thereby gives a measure of how variables relate to each other (by covariance matrix), the directions in which the data is dispersed (by Eigenvectors) and the relative importance of each direction (by Eigenvalues).

Analysis, Results and Findings 3.1 Multiple Linear Regression of the medical expenses on all independent variables

The Multiple Linear Regression method was used to explain the relationship between several independent variables and the dependent variable, namely "medical expenses". The coefficients of parameters of multiple linear regression are shown on Table 2. The Multiple R-squared value is given as 0.1789. It can be seen that the R-squared indicates the model's ability to explain variations in the data is low. This indicates that only 17.89% of the variance in the dependent variable was explained by the independent variables. The intercept of 𝛽 0 = -0.00006 indicates a negative relationship for the medical expenses when the independent variables are centered around its mean values.

Identifying Superfluous variables using Correlation matrix

A correlation analysis is performed to understand the relationship between the variables in the dataset. Superfluous data often manifested where multiple predictor variables are highly correlated. The lowest correlation (0.01) which is between the height and the level of education. This may be due to the fact that there is no relationship between the height of a person and the level of education attained. The highest correlation (0.94) is between the BMI and the weight of the respondent. Also, another highly correlated variables are status of drinking and status of smoking (0.91), region and level of education (0.70) and height and BMI (0.77). Some of these highly correlated variables or irrelevant variables are considered as superfluous or data with little predictive power.

In some cases of the highly correlated variables, one of the two variables can be eliminated from the model without losing much information. One of the variables may contain information that is already captured by other variables in the dataset. For instance, the variable BMI and weight or BMI and height are highly correlated with each other, one might be considered superfluous as it doesn't add unique information. Again, the BMI was calculated using the height and weight.

The superfluous variable will have little or no relationship with the dependent variable (medical expenses) or do not influence the model's output significantly. Again, including superfluous variables can cause overfitting of the data and thereby increasing the complexity of a model.

Performing Principal Component Analysis (PCA)

We perform principal component analysis by applying it to the set of independent variables to create the principal components. The output in R is given in Table 3. Table 3 shows the PCA and the total variances. The standard deviations of the principal components are used to calculate the variance explained by each component using a cumulative variance threshold which involves setting a specific threshold for the total proportion of variance to be explained and selecting the minimum number of components required to reach that threshold. In this case, we choose 90%.

The first seven components have a cumulative explained variance of 90%.

Figure 1 shows a scree plot is a line plot of the eigenvalues (which represent the amount of variance) against the corresponding principal component number. A scree plot is a line plot of the eigenvalues (which represent the amount of variance) against the corresponding principal component number. PC2 is influenced by three variables smoking (-0.4117), region (0.4263) and drinking (-0.4178). This suggests a contrast between region and smoking and drinking. All other PC's are explained in similar way.

Performing Multiple Linear Regression using the principal components

Multiple Linear Regression model was performed using the principal components as independent variables and the dependent variable from the dataset. The output is given in Table 4.

References

1. Ahmed SS, Yisa EM, Jibrin M, Yahaya G. Principal component analysis for investigating the relationship between semester results and academic performance of students in a polytechnic in Niger State, Nigeria. J Appl Sci Environ Manage. 2023;27(10):2355-2359.
2. Akbar TA, Javed A, Ullah S, Ullah W, Pervez A, Akbar RA, et al. Principal component analysis (PCA)-GIS modeling for groundwater and associated health risks in Abbottabad, Pakistan. Sustainability. 2022;14(21):14572.
3. Fan C, Chen M, Wang X, Wang J, Huang B. A review on data preprocessing techniques toward efficient and reliable knowledge discovery from building operational data. Front Energy Res. 2021;9:652801.
4. Gao J, Hu W, Chen Y. Revisiting PCA for time series reduction in temporal dimension. arXiv. 2024.
5. Greenacre M, Blasius J, Nenadić O. Correspondence analysis and related methods. Boca Raton: Chapman & Hall/CRC; 2022.
6. Guo L, Yi Q. An exploration of the application of principal component analysis in big data processing. Appl Math Nonlinear Sci. 2024;9(1):1-24.
7. Taherdoost H. Data collection methods and tools for research: a step-by-step guide to choose data collection technique for academic and business research projects. Int J Acad Res Manag. 2021;10(1):10-38.
8. Kaya Uyanık G, Güler N. A study on multiple linear regression analysis. Procedia Soc Behav Sci. 2013;106:234-240.
9. Li G, Wang X, Liu H, Zhang Q. An exploration of the application of principal component analysis in big data processing. Appl Math Nonlinear Sci. 2024;9(4):664675.
10. Li H, Tan Y, Zeng D, Su D, Qiao S. Attitude-predictive control of large-diameter shield tunneling: PCA-SVR machine learning algorithm application in a case study of the Zhuhai Xingye Express Tunnel. Appl Sci. 2025;15(4):1880.
11. Mishra S, Sarkar U, Taraphder S, Datta S, Swain D, Saikhom R, et al. Principal component analysis. Int. J Livest Res. 2017;1.
12. Mundfrom D, Kay L. The effect of multicollinearity on prediction in regression models. Gen Linear Model J. 2018;44:24-28.
13. R Development Core Team. R: a language and environment for statistical computing. Vienna: R Foundation for Statistical Computing; 2008.
14. Hamina S. Data reduction using principal component analysis: theoretical underpinnings and practical applications in public health. J Contemp Med Educ. 2024;14(1):1-14.
15. Ustuner M. Randomized principal component analysis for hyperspectral image classification. arXiv. 2024.
16. Yin Y, Zhou W. High-dimensional PCA revisited: insights from general spiked models and data normalization effects. arXiv. 2024.