Tracing the historical, theoretical, and mathematical foundations of factor analysis and principal component analysis

Introduction

Factor Analysis (FA) and Principal Component Analysis (PCA) are fundamental methods in multivariate statistics, essential for data reduction and dimensionality reduction across various disciplines [1] . Despite their shared objective of simplifying complex datasets, FA and PCA serve distinct purposes due to differences in their theoretical underpinnings and application domains. Originating in psychology, FA is primarily aimed at uncovering latent variables that influence observed measurements, particularly in cognitive and psychological assessments [2] . In contrast, PCA, developed within mathematical statistics, focuses on transforming observed variables into principal components to capture the maximum variance without making assumptions about latent constructs [3,4] . The distinct roles of FA and PCA have spurred numerous comparative studies aimed at clarifying their applications and limitations. Velicer and Jackson [5] underscored that FA, rooted in latent variable modeling, is more suitable for theoretical research where identifying underlying factors is essential, whereas PCA excels in exploratory data simplification without latent assumptions. Widaman [6] observed that PCA often inflates loadings, particularly in smaller samples, making FA preferable in scenarios requiring precise latent factor estimation. Several studies, including Park, Dailey [7] , noted that PCA's computational efficiency contributes to its misuse in fields requiring latent structure identification, like communication research, where EFA is more appropriate. Brown [8] and Jolliffe [1] emphasized that PCA maximizes total variance through orthogonal components, while FA models shared variance, distinguishing the methods in terms of objectives.

De Winter and Dodou [9] showed through simulations that PCA tends to yield more stable loadings, whereas FA is more accurate for correlated factors. Daniel [10] and Starkweather [11] reviewed methodological differences, advising PCA for data reduction and FA for latent structure modeling, particularly in social sciences. Di Franco [12] highlighted that PCA is often incorrectly used for latent modeling due to its simplicity, risking misinterpretation. Meanwhile, Dobriban [13] discussed Parallel Analysis as a method for choosing factors in both techniques, cautioning that PA may underestimate factors in high-dimensional data. Travaglini [14] and Rao [15] examined theoretical distinctions, noting that FA is more efficient in estimating latent variables in large samples, whereas PCA simplifies data more efficiently when underlying structures are not hypothesized. Mabel and Olayemi [16] and emphasized that FA's assumptions align it better with fields focused on latent modeling, while PCA's empirical focus suits applications like image processing. Finally, Bai and Ng [18] and Allee, Do [19] further supported FA's role in fields requiring structural modeling and PCA's utility in data-centric fields, concluding that method choice should depend on research objectives. Despite extensive research comparing FA and PCA, there is limited comprehensive analysis that traces their evolution and clarifies their respective roles from historical, theoretical, and mathematical perspectives. This paper aims to address this gap by providing a detailed review of FA and PCA's historical development, theoretical foundations, and mathematical structures. By exploring these aspects, the study seeks to offer a clearer understanding of each method's unique role in multivariate analysis and provide guidelines for their appropriate use in research. Understanding the distinctions between FA and PCA is essential for researchers across disciplines who rely on multivariate analysis for data interpretation and decisionmaking. As dimensionality reduction becomes increasingly crucial in handling large datasets, particularly in fields like genomics, image processing, and social sciences, this study provides timely guidance on selecting the appropriate technique based on analytical objectives [ 14 , 15 ] . Highlighting the theoretical and mathematical differences between FA and PCA not only contributes to methodological rigor but also helps prevent common analytical errors, such as the inappropriate use of PCA for latent structure analysis. This research serves as a foundational resource for scholars and practitioners aiming to apply FA and PCA accurately, thereby enhancing the reliability and interpretability of their findings in complex data environments. This paper is organized as follows: Section I reviews the historical origins of FA and PCA, detailing the development of Spearman's Two-Factor Theory and PCA's mathematical foundation. Section II examines conceptual differences, focusing on FA's latent model assumptions versus PCA's variance-maximizing framework. Section III provides a comparative analysis of mathematical structures, including eigenvalue decomposition in PCA and factor loading models in FA. Section IV explores FA's reliance on psychological theories, contrasting it with PCA's empirical, data-driven roots. Section V summarizes the relevance of each method's distinct objectives and applications across disciplines. Section VI concludes.

Historical development of factor analysis (FA) and Principal Component Analysis (PCA) 2.1 Origins and Early Development of Factor Analysis

FA has its roots in early 20th-century psychometrics, primarily through Charles Spearman's work. Spearman's influential 1904 paper introduced the Two-Factor Theory as a means of explaining cognitive abilities through two main components: a general intelligence factor (g) and specific factors unique to each task [2] . This theory was revolutionary for its time, providing a framework for investigating underlying constructs that shape observed behavior [20] . The essence of Spearman's approach was his tetrad difference method, which analyzed correlations between test scores to identify shared factors. If a single common factor existed, the tetrad difference among scores would be zero [21] . Despite its initial limitations due to reliance on correlation matrices, Spearman's model laid the foundation for FA as a statistical tool in psychological research, especially in understanding intelligence [21,22] . Building on Spearman's model, Louis Thurstone further advanced FA in the 1930s by introducing multiple-factor analysis. Thurstone's approach challenged Spearman's singular focus on a general intelligence factor, proposing instead that intelligence comprised multiple distinct abilities, or primary mental abilities. This shift allowed FA to analyze multiple dimensions within cognitive abilities, broadening its scope across various disciplines [23] . The development of rotational techniques like Varimax in the 1950s by Kaiser further enhanced interpretability, allowing FA to present factors more independently [24] . These methodological advancements and the rise of digital computing expanded FA's applications, enabling it to handle larger datasets across psychology, marketing, and health sciences [25] . With the increasing availability of digital computing in the 1970s, FA's utility grew. This era marked the introduction of Confirmatory Factor Analysis (CFA) which enabled researchers to test specific hypotheses about factor structures rather than rely solely on exploratory methods. Developed by Jöreskog [26] , CFA formalized FA's role in structural modeling, allowing researchers to validate pre-defined factor models. This marked a major theoretical progression, making FA a vital tool for hypothesis testing within structural equation modeling (SEM) frameworks [27] . Additionally, advancements like Maximum Likelihood Estimation (MLE) improved FA's accuracy by optimizing the factor model's fit to data, solidifying FA's relevance in areas where understanding latent structures is paramount [ 16 ] . Today, FA remains essential in psychology, behavioral research, and SEM. The method is widely used to examine relationships between observed and latent variables, enabling researchers to test complex structural models that go beyond surface correlations [28,29] . FA's focus on uncovering latent constructs distinguishes it from other methods, especially PCA, which evolved separately within the realm of mathematical statistics.

Historical Background of Principal Component Analysis (PCA)

While FA developed within psychometrics, PCA emerged from mathematical statistics. Introduced by Karl Pearson in 1901, PCA was designed to reduce data dimensionality without assumptions about latent factors. Pearson's aim was to represent high-dimensional data on lower-dimensional planes by preserving variance, laying a foundation for data simplification and dimensionality reduction [1] . This approach, which minimized the distance between data points and a bestfitting linear plane, focused on maximizing variance rather than revealing underlying constructs, thus distinguishing PCA from FA in both purpose and application [20] . Harold Hotelling expanded upon Pearson's work in 1933, formalizing PCA's mathematical structure through eigenvalue decomposition of the covariance matrix. This technique transformed correlated variables into uncorrelated principal components, each capturing successive portions of the total variance. Hotelling's method provided a mathematically robust way to analyze complex datasets, which further established PCA as a core method for data reduction [ 30 , 31 ] . Unlike FA, PCA's design does not assume that data arise from a latent factor model, making it ideal for applications, where reducing dimensions while retaining maximal data variance is the goal [32,33] . The utility of PCA expanded in the 1960s and 1970s with the growth of digital computation, allowing its application to larger, more complex datasets. Fields such as meteorology, oceanography, genetics, and neuroscience adopted PCA as a tool for simplifying high-dimensional data. In particular, PCA became valuable in image processing, where reducing data dimensions is critical for pattern recognition and noise minimization [4,34] . Unlike FA, which seeks to uncover latent structures, PCA focuses solely on capturing and maximizing variance, making it a core technique for data preprocessing in machine learning and other computational fields [33] . Hence, the historical evolution of FA and PCA illustrates their distinct objectives and methodologies. FA emerged from psychometric theory, driven by the need to understand latent constructs through statistical analysis, while PCA was developed as a data-centric tool for maximizing variance without assuming underlying structures. FA's expansion into Confirmatory Factor Analysis (CFA) and structural modeling further established it as a theory-driven tool for social sciences, whereas PCA's foundational role in computational applications highlights its empirical orientation. These distinct trajectories underscore the different purposes and applications of each method, shaping their respective roles in modern statistical analysis [8,35] .

Conceptual differences between Factor Analysis (FA) and Principal Component Analysis (PCA) 3.1 Purpose and Philosophical Orientation

The conceptual distinction between FA and PCA reflects their unique origins, purposes, and assumptions. FA originated in psychology and social sciences with the aim of identifying latent variables unobserved constructs that explain the correlations among observed data [21,36] . FA's primary objective is thus to uncover these latent structures, which influence the relationships between variables. This theorydriven approach aligns with psychometric principles, where observed behaviors are analyzed to infer hidden constructs, such as cognitive abilities or personality traits [20,25] . In contrast, PCA emerged from mathematical and statistical needs, aiming to reduce data dimensionality by transforming correlated variables into a set of uncorrelated components. Introduced by Pearson (1901), PCA is designed to maximize the variance explained by each component without making assumptions about latent structures [4] . This variance-focused approach is particularly valuable in fields like machine learning, image processing, and genetics, where simplifying high-dimensional data is essential [34 . Unlike FA, PCA does not infer underlying constructs but treats data as a multivariate system optimized for variance capture [37] .

Role of latent variables and factor model

A central distinction between FA and PCA lies in their treatment of latent variables. In FA, each observed variable is modeled as a function of a smaller set of latent factors unobserved, theoretically inferred constructs that explain shared variance. This relationship is represented through a factor loading matrix, where loadings indicate the degree to which each latent factor influences an observed variable. Such a structure allows FA to separate shared (common) variance from unique (specific) variance, making it particularly suitable for studies seeking to identify underlying constructs within complex datasets [22,37] . This model is essential in fields like psychology and education, where understanding latent variables is crucial [20] . PCA, however, does not assume the presence of latent variables. Instead, it transforms observed variables into new, orthogonal components, each representing a linear combination of the original data. These components, or principal components, are derived to capture as much variance as possible, with no implied latent constructs [ 3 ] . Thus, PCA operates through eigenvalue decomposition of the covariance matrix, focusing on variance maximization rather than factor structure. This model-free approach makes PCA ideal for data simplification rather than for uncovering latent constructs, distinguishing it from FA's theory-driven objectives [4,33] .

Assumptions and Model-Based Differences

The model assumptions in FA and PCA further underscores their conceptual divergence. FA operates under a model that distinguishes between common and unique variances, where each observed variable is expressed as a linear combination of common and unique factors [21] . This model is governed by specific assumptions, such as the expectation that common factors explain the correlations among observed variables, while unique factors account for variance specific to each variable. This covariance-based model allows FA to focus on revealing latent structures that reflect shared variance [38] . In contrast, PCA is model-free, meaning it does not assume latent factors. Rather than decomposing observed variables into common and unique components, PCA aims solely to transform data into uncorrelated components ordered by the variance they explain [39] . This entails performing eigenvalue decomposition on the covariance matrix, where principal components are ranked by their explained variance. Unlike FA, PCA treats all variance as meaningful, making no distinction between shared and unique variances. This difference in approach makes PCA a key tool for dimensionality reduction rather than latent variable identification, as it optimizes for total variance rather than correlation structure [1,32] .

Interpretability and Rotational Methods

The interpretability of FA and PCA results also varies significantly due to the different approaches to factor/component rotation. In FA, rotation is essential to improve interpretability, as it enables researchers to obtain a simplified structure that aligns with theoretical constructs. Orthogonal rotation methods, like Varimax, maximize variance for each factor by aligning factor loadings closely with certain variables. This approach enhances clarity, making FA particularly effective for exploring theoretical constructs in social sciences [8,24,40] . Rotation is less common in PCA; when applied, it is typically to simplify component interpretation without inferring latent structures. Since PCA components are orthogonal by default, rotation only minimally alters component orientation. For example, Promax or Direct Oblimin rotations can yield oblique components if desired, though this is generally less common in PCA compared to FA (Jolliffe & Cadima, 2016).

This minimal reliance on rotation reflects PCA's focus on maximizing variance capture over interpretability, aligning it more with empirical data processing than theoretical analysis [8, 32] .

Practical Applications and Use Cases

The applications of FA and PCA diverge based on their conceptual frameworks and underlying assumptions. FA's focus on latent structure identification makes it invaluable in fields like psychology, where researchers aim to understand constructs like cognitive abilities or personality traits. FA is widely applied in scale development, validation, and exploring complex constructs. Its ability to distinguish common from unique variances provides insights into latent factors influencing observed measures, enhancing its utility in research involving theoretical constructs [25,38] . Conversely, PCA is primarily applied in fields where data simplification and noise reduction are essential. Common applications include image processing, genetics, and machine learning, where high-dimensional datasets require dimensionality reduction without losing critical information. For instance, in image compression, PCA reduces image size by retaining the most significant principal components, thus preserving essential features while discarding less relevant data. In machine learning, PCA preprocesses data for clustering or classification, transforming high-dimensional data into lower dimensions to optimize computational efficiency and model performance [33,35] . Based on the above discussion, it can be concluded that FA and PCA differ conceptually in fundamental ways, each suited to unique applications and research goals. FA's origin in psychometrics and focus on latent variable identification makes it invaluable for exploring unobserved constructs in psychological studies. contrast, PCA's mathematical foundation in data reduction aligns it with fields requiring simplification of high-dimensional data across computational and natural sciences. FA's reliance on a factor model contrast with PCA's data-centric approach, where variance is maximized for dimensionality reduction without inferring latent structures. These distinctions underscore the roles each method plays in statistical analysis and their value across different fields of study.

Mathematical differences between Factor Analysis (FA) and Principal Component Analysis (PCA)

Factor Analysis (FA) and Principal Component Analysis (PCA) both serve the purpose of data reduction, yet they are based on distinct mathematical frameworks that influence their applications, interpretations, and limitations. Understanding these differences is essential for researchers who wish to apply these methods appropriately in multivariate analyses. This section delves into the mathematical distinctions between FA and PCA, drawing on historical foundations and theoretical models to illustrate each method's unique approach to data decomposition.

Mathematical Model of Factor Analysis

FA is based on a model that decomposes observed variables into shared (common) and unique (specific) components. This model assumes that each observable variable X can be expressed as a linear combination of a smaller set of latent factors (common factors) and unique variances that are specific to each variable [20,21] . Mathematically, this relationship is given by:

Where X is an 𝑛 × 1 vector of observed variables, Λ represents the factor loading matrix of dimensions 𝑛 × 𝑚 (where m is the number of common factors), F denotes the vector of latent factors, and ϵ represents the unique variances [ 4 ] . The factor loading matrix Λ indicates the strength of association between each latent factor and the observed variables. The latent factors F are assumed to be uncorrelated with the specific errors ϵ, which makes FA suitable for isolating latent structures within the data [ 22 ] . The covariance structure in FA is a fundamental component, expressed by:

Where Φ is the covariance matrix of the common factors, often simplified as an identity matrix when factors are standardized and uncorrelated, and Ψ represents the covariance matrix of unique variances, typically assumed to be diagonal, indicating that unique factors are uncorrelated with one another [25] . This decomposition is aimed at capturing the shared variance among observed variables through the common factors, aligning FA with a model-based, theory-driven approach that seeks to uncover latent variables underlying observed behaviors or measurements [21,22] .

Mathematical foundation of principal component analysis

PCA, in contrast, is not based on a latent variable model but instead transforms observed variables into a set of uncorrelated components that maximize the total variance captured in the data. The primary goal of PCA is dimensionality reduction, achieved by decomposing the covariance matrix Σ through eigenvalue decomposition or singular value decomposition (SVD) [3,41] . This decomposition can be expressed as:

Where P is a matrix, whose columns are the eigenvectors of Σ, and D is a diagonal matrix of eigenvalues associated with each principal component. The eigenvalues represent the amount of variance captured by each principal component, allowing PCA to transform the original variables into orthogonal (uncorrelated) components that successively capture the maximum remaining variance [4,33] .

Each principal component 𝒁 𝒊 in PCA is a linear combination of the original variables 𝑿 𝒋 , represented by:

Where 𝑝 𝑖𝑗 are the weights (loadings) of the original variables 𝑿 𝑗 in the 𝑖 -𝑡ℎ principal component [ 3 ] . Unlike FA, PCA does not separate shared and unique variances; instead, it considers all variance in the data as meaningful, capturing maximum variance through orthogonal components and allowing for data simplification in high-dimensional datasets [1, 32] .

Variance structure and Eigenvalue interpretation

In FA, the variance in each observed variable is divided into common variance (shared among variables) and unique variance (specific to individual variables). The commonality of each observed variable 𝑋 𝑖 in FA, or the proportion of variance explained by the common factors, is calculated as the sum of squared loadings:

The unique variance, which FA isolates, represents the portion of each variable's variance not accounted for by the latent factors, a distinction that PCA does not make (Jolliffe, 1986). By contrast, in PCA, the eigenvalues λ j of the covariance matrix represent the variance explained by each principal component. The proportion of variance captured by each component 𝑍 𝑖 is calculated as:

This variance-based approach in PCA ranks components by explanatory power, allowing for the selection of a subset of components that together capture most of the variance, typically useful for reducing dimensionality in datasets where the latent structure is not a concern [ 32 , 33 ] .

Dimensionality Reduction and Selection Criteria

The methods differ significantly in how they determine the number of factors or components to retain. In FA, the number of factors is usually selected based on theoretical or empirical criteria, such as the Kaiser criterion (eigenvalues greater than one) or Scree plot analysis, often aligned with the studies theoretical constructs [38 . In PCA, components are retained to capture a high cumulative percentage of total variance (e.g., 80-90%), reflecting its emphasis on data preservation rather than latent structure modeling [4,37] .

Rotation Techniques and Interpretability

Rotation enhances the interpretability of FA by redistributing variance across factors, aligning factor loadings with specific observed variables. Orthogonal rotations (e.g., Varimax) maintain uncorrelated factors, while oblique rotations (e.g., Promax) allow correlated factors, enhancing the theoretical interpretability of factor structures [8,40] . PCA, however, typically does not rely on rotation, as its principal components are orthogonal by construction, and rotation may alter PCA's objective of maximizing variance capture. When applied in PCA, rotations like Promax or Oblimin are used solely to simplify component interpretation rather than to suggest any latent structure [1] .

Component Scores and Estimation Techniques

In FA, factor scores represent the estimated values of latent factors for each individual in the dataset, computed postanalysis through methods such as Bartlett's method or regression-based scoring. These scores provide indirect measures of the latent constructs that influence observed variables [42] . PCA component scores, on the other hand, are derived directly from the data by applying eigenvectors to the observed variables, making PCA a more straightforward tool for obtaining dimensionality-reduced representations of the data [30,39] . Hence, FA and PCA differ fundamentally in their mathematical models, variance structures, dimensionality selection criteria, and interpretative methods. FA's latent factor model, aimed at partitioning variance into common and unique components, aligns it with theory-driven analyses that seek to identify latent constructs. PCA, however, uses a model-free approach that maximizes total variance capture through orthogonal transformations, making it suitable for empirical data simplification. These mathematical distinctions reinforce FA's role in theory-based research and PCA's utility in data-centric dimensionality reduction, underscoring the complementary applications of each method in multivariate analysis [1,33,37] .

FA's Basis on two-factor theory and contrast with PCA's Data-Centric Approach

Factor Analysis (FA) and Principal Component Analysis (PCA) diverge fundamentally in their theoretical underpinnings. FA is deeply rooted in psychology and is driven by the goal of uncovering latent variables, an approach first formalized by Charles Spearman's Two-Factor Theory [ 2 ] . By contrast, PCA, developed from a mathematical perspective, is a data-centric method intended to optimize variance representation, without presupposing any underlying latent constructs [3] . This section explores the foundations of Spearman's Two-Factor Theory, the mathematical basis of PCA, and how these distinct origins shape the purposes and applications of each method.

Origins of factor analysis: Spearman's two-factor theory

Charles Spearman introduced the foundational Two-Factor Theory in 1904, marking the origin of FA as a statistical tool for uncovering latent variables in psychological data. Spearman proposed that cognitive abilities could be explained through a single general factor, g, which is common to all tasks, and multiple specific factors unique to individual tasks (Spearman, 1904). This model aimed to represent correlations among observed variables through underlying latent constructs, emphasizing the role of shared variance in explaining behavioral patterns [20,21] .

Mathematically, Spearman's model can be represented as:

Where Y denotes the observed variable, λ is the loading that reflects the influence of the general factor g on Y, and ϵ represents the specific factor, unique to each task. This formula encapsulates the essence of FA: decomposing observed variables into shared (common) and unique components, with the common factor g capturing the variance shared across multiple tasks [21,22] . Spearman further refined this model by introducing the concept of the tetrad difference. If a single common factor underlies multiple observed variables, the difference in tetrads (groups of four variables) should be zero, indicating a rank-1 matrix structure. This mathematical condition, represented as:

Where R represents correlations, laid the groundwork for identifying latent factors in FA (Spearman, 1904). The tetrad difference allowed researchers to confirm whether a common factor was sufficient to explain correlations among observed variables, solidifying FA as a theory-driven method for exploring latent structures.

The Development of FA beyond two-factor theory

Spearman's model of a single general factor was expanded in the 1930s by Louis Thurstone, who introduced multiple-factor analysis to capture diverse cognitive abilities [43] . Thurstone proposed that intelligence could not be encapsulated by a single factor but rather by a set of primary mental abilities. This shift transformed FA, allowing for multiple factors to be extracted from data, thereby broadening its applicability across various domains [25,37] . Thurstone's work also introduced rotational methods, such as Varimax, which facilitated clearer interpretations by aligning factor loadings with specific observed variables. These methodological advancements underscored FA's flexibility in revealing complex latent structures and reinforced its alignment with psychology's focus on identifying unobserved constructs [24,40] . The later development of Confirmatory Factor Analysis (CFA) by Jöreskog in the 1970s further solidified FA's role in hypothesis-driven research by allowing predefined factor models to be tested against data, making FA an essential tool for validating theoretical constructs in fields like psychology and social sciences [27] .

PCA's Empirical and Data-Centric Basis

PCA, unlike FA, was developed from a mathematical perspective focused on data simplification rather than latent variable discovery. Introduced by Karl Pearson in 1901, PCA aimed to reduce data dimensionality by transforming correlated variables into orthogonal principal components that maximize variance without assuming any underlying latent structure [ 3 ] . This data-centric approach to PCA prioritizes variance capture over interpretative models of latent constructs, making it ideal for empirical applications in highdimensional data scenarios [4] .

In PCA, each principal component 𝒁 𝒊 is derived as a linear combination of observed variables 𝑿 𝒊 :

Where 𝑝 𝑖𝑗 represents the loading of each observed variable 𝑋 𝑗 on the 𝑖 -𝑡ℎ principal component [ 3 ] . This construction differs fundamentally from FA's model-based approach, as PCA does not attempt to account for shared or unique variances but instead maximizes the variance captured by each successive component. This framework is particularly valuable for fields like genomics, machine learning, and image processing, where preserving total variance is essential for effective dimensionality reduction [ 1 , 32 ] .

Core Differences: Theory-Driven versus data-driven approaches

The core distinction between FA and PCA lies in their underlying objectives. FA's model-based approach, rooted in psychological theories, aims to uncover latent variables that explain correlations among observed variables. This makes FA inherently suitable for research that seeks to validate theoretical constructs, as it distinguishes between common and unique variances in data [42] . By contrast, PCA's datadriven approach is purely empirical, focusing on transforming data into orthogonal components to capture the maximum amount of total variance, making no assumption of latent factors or underlying constructs [3,39] . While FA applies a factor model to explain observed behaviors, PCA operates through eigenvalue decomposition of the covariance matrix, emphasizing data simplification without theoretical implications. This difference reflects FA's alignment with psychology and social sciences, where latent constructs are central, whereas PCA aligns with fields that prioritize empirical data handling over latent structure modeling, such as computational sciences and engineering [1,33] .

Applications and Interpretations of FA and PCA

The theoretical foundations of FA have influenced its applications primarily within psychology, social sciences, and market research, where understanding latent constructs like cognitive abilities and personality traits is paramount [21,22] . These constructs, often unobservable, are inferred from observed variables, positioning FA as a powerful tool for developing, validating, and analyzing theoretical constructs.

In contrast, PCA is favored in fields that require data reduction without latent assumptions. For instance, in bioinformatics and genomics, PCA simplifies complex datasets by retaining only the principal components that capture the majority of the variance. This approach allows PCA to reduce data dimensionality efficiently while preserving critical information for pattern recognition, clustering, and classification in high-dimensional data environments [4,33] . As such, PCA is particularly suitable for machine learning, image processing, and other computational fields where data-centric methods are essential for managing and analyzing large datasets. Based on the above discussion it can be concluded that the foundational differences between FA and PCA underscore their unique applications and theoretical orientations. FA, grounded in Spearman's Two-Factor Theory and extended through multiple-factor models and rotational methods, is inherently theory-driven, aiming to identify and measure latent variables that explain observed behaviors. PCA, rooted in Pearson's empirical approach to variance maximization, prioritizes dimensionality reduction without underlying theoretical assumptions, making it optimal for data simplification in high-dimensional contexts. These distinct orientations continue to shape FA and PCA's applications across fields. FA's alignment with psychology and social sciences makes it indispensable for theory-based research, while PCA's empirical orientation serves computational fields requiring efficient data handling. Together, these methods illustrate the contrasting but complementary roles that theory and data-centric approaches play in multivariate analysis, with FA focusing on latent constructs and PCA on efficient data representation [35,37] .

Conclusion

This article has examined the historical, theoretical, and mathematical distinctions between Factor Analysis (FA) and Principal Component Analysis (PCA), two fundamental techniques in multivariate analysis that, though both aimed at dimensionality reduction, serve distinct purposes. Emerging from psychology, FA, grounded in Spearman's Two-Factor Theory, was developed to uncover latent structures within observable variables, especially in psychological and social sciences. Conversely, PCA, developed by Pearson and Hotelling, is a mathematical method focused on maximizing data variance without assuming underlying constructs, making it ideal for empirical data simplification. The theoretical distinctions between FA and PCA underscore FA's model-based approach, which partitions variance into common and unique components to reveal latent variables. This approach aligns FA with theory-driven research. PCA, on the other hand, employs a data-centric strategy by transforming observed variables into orthogonal components to maximize total variance captured, making it essential for fields that prioritize empirical data reduction, like genomics and machine learning. Mathematically, FA relies on a factor loading matrix to separate shared and specific variances, supporting latent variable discovery. PCA, in contrast, uses eigenvalue decomposition to maximize total variance through uncorrelated components, focusing on dimensionality reduction without latent interpretation. These distinctions in model structure, rotation techniques, and scoring highlight FA's interpretive focus and PCA's empirical utility.

In conclusion, FA and PCA offer complementary solutions for multivariate analysis. FA's alignment with latent structure modeling makes it essential for theory-based studies, while PCA's empirical orientation supports data-centric research.

Recognizing their distinct theoretical and mathematical foundations enables researchers to select the appropriate method, ensuring analytical rigor and alignment with research objectives across disciplines.

Limitations and Future Research Scope

While this article provides a comprehensive theoretical comparison of Factor Analysis (FA) and Principal Component Analysis (PCA), its conclusions are based primarily on conceptual distinctions rather than extensive empirical validation. Future research should apply these methods across diverse datasets and domains to empirically test and refine these theoretical insights. Additionally, expanding research on hybrid techniques that combine FA and PCA, integrating these methods within machine learning frameworks, and establishing clear guidelines for method selection in specific research fields could further strengthen the utility of FA and PCA in multivariate analysis.

Funding and/or Conflicts of Interests/Competing Interests

The author declares that there are no conflicts of interest or competing interests related to this study. No financial support was received for the conduct of this research or the preparation of this article.

Data availability statements

I have not analyzed or generated any datasets, because my work proceeded within a theoretical and mathematical approach.

References

1. Jolliffe IT. Principal component analysis for special types of data. Springer, 2002.
2. Spearman C. 'General intelligence,' objectively determined and measured. Am J Psychol. 1904;15(2):201-93.
3. Pearson K. LIII. On lines and planes of closest fit to systems of points in space. Lond Edinb Dubl Philos Mag J Sci. 1901;2(11):559-572.
4. Jolliffe IT, Cadima J. Principal component analysis: a review and recent developments. Philos Trans R Soc A Math Phys Eng Sci. 2016;374(2065):20150202.
5. Velicer WF, Jackson DN. Component analysis versus common factor analysis: Some issues in selecting an appropriate procedure. Multivariate Behav Res. 1990;25(1):1-28.
6. Widaman KF. Common factor analysis versus principal component analysis: Differential bias in representing model parameters? Multivariate Behav Res. 1993;28(3):263-311.
7. Park HS, Dailey R, Lemus D. The use of exploratory factor analysis and principal components analysis in communication research. Hum Commun Res. 2002;28(4):562-577.
8. Brown JD. Statistics Corner. Questions and answers about language testing statistics: Choosing the right number of components or factors in PCA and EFA. Shiken JALT Test Eval SIG Newsl. 2009;13(2):19-23.
9. Winter DJ, Dodou D. Common factor analysis versus principal component analysis: A comparison of loadings by means of simulations. Commun Stat Simul Comput. 2014;10(03610918.2013):862274.
10. Daniel LG. Common factor analysis or components analysis: An update on an old debate, 1990.
11. Starkweather J. Principal components analysis vs. factor analysis and appropriate alternatives [Internet], 2010, Available from: http://it.unt.edu/benchmarks
12. Di Franco G. Differences between factor analysis and principal component analysis. Marradi A, Di Franco G, Editors, 2013, p. 113-132.
13. Dobriban E. Permutation methods for factor analysis and PCA, 2020.
14. Travaglini G. Principal components and factor analysis. A comparative study, 2011.
15. Rao CR. Principal component and factor analyses. Handb Stat. 1996;14:489-505.
16. Mabel OA, Olayemi OS. A comparison of principal component analysis, maximum likelihood and the principal axis in factor analysis. Am J Math Stat. 2020;10(2):44-54.
17. Osborne JW, Costello AB. Sample size and subject-toitem ratio in principal components analysis. Pract Assess Res Eval. 2019;9(1):11.
18. Bai J, Ng S. Principal components and regularized estimation of factor models. ARXIV Prepr ARXIV:170808137. 2017.
19. Allee KD, Do C, Raymundo FG. Principal component analysis and factor analysis in accounting research. J Financ Report. 2022;7(2):1-39.
20. Bartholomew DJ, Knott M, Moustaki I. Latent variable models and factor analysis: A unified approach. John Wiley & Sons, 2011.
21. Mulaik SA. A brief history of the philosophical foundations of exploratory factor analysis. Multivariate Behav Res. 1987;22(3):267-305.
22. Harman HH. Modern factor analysis. Univ Chicago Press, 1976.
23. Cudeck R, MacCallum RC. Factor analysis at 100: Historical developments and future directions. Routledge, 2007.
24. Tabachnick BG, Fidell LS, Ullman JB. Using multivariate statistics. Pearson, 2013.
25. Akhtar-Danesh N. A comparison between major factor extraction and factor rotation techniques in Qmethodology. Open J Appl Sci. 2017;7(4):147-56.
26. Jöreskog KG. A general approach to confirmatory maximum likelihood factor analysis. Psychometrika. 1969;34(2):183-202.
27. Browne MW. An overview of analytic rotation in exploratory factor analysis. Multivariate Behav Res. 2001;36(1):111-50.
28. Ferrando PJ, Lorenzo-Seva U. Program FACTOR at 10: Origins, development, and future directions. Psicothema. 2017;29(2):236-240.
29. Cattell R. A note on correlation clusters and cluster search methods. Psychometrika. 1944;9(3):169-184.
30. Jeffers JN. Two case studies in the application of principal component analysis. J R Stat Soc C Appl Stat. 1967;16(3):225-236.
31. Anderson TW. Asymptotic theory for principal component analysis. Ann Math Stat. 1963;34(1):122-148.
32. Stewart GW. On the early history of the singular value decomposition. SIAM Rev. 1993;35(4):551-566.
33. Shlens J. A tutorial on principal component analysis. arXiv Prepr arXiv:14041100, 2014.
34. López M, Ramírez J, Górriz JM, Álvarez I, SalasGonzalez D, Segovia F, et al. Principal component analysis-based techniques and supervised classification schemes for the early detection of Alzheimer's disease. Neurocomputing. 2011;74(8):1260-1271.
35. Jolliffe I. A 50-year personal journey through time with principal component analysis. J Multivar Anal. 2022;188:104820.
36. Steiger JH. Dispelling some myths about factor indeterminacy. Multivariate Behav Res. 1996;31(4):539550.
37. Howard MC. A review of exploratory factor analysis decisions and overview of current practices: What we are doing and how can we improve? Int J Hum Comput Interact. 2016;32(1):51-62.
38. Gorsuch RL. Common factor analysis versus component analysis: Some well-and little-known facts. Multivariate Behav Res. 1990;25(1):33-39.
39. Hotelling H. Analysis of a complex of statistical variables with principal components. J Educ Psychol. 1933;24:498520.
40. Kaiser HF. The varimax criterion for analytic rotation in factor analysis. Psychometrika. 1958;23(3):187-200.
41. Hotelling H. Analysis of a complex of statistical variables into principal components. J Educ Psychol. 1933;24(6):417.
42. Thurstone LL. The vectors of mind: Multiple-factor analysis for the isolation of primary traits. Chicago (IL): Univ Chicago Press, 1935.
43. Thurstone LL. Multiple factor analysis. Psychol Rev. 1931;38(5):406.