Comparative analysis of time series forecasting: Traditional Statistical Approaches vs. Machine Learning Methods

Introduction

Time series forecasting techniques have become essential tools across a wide range of applications, including the analysis and prediction of technical, physical, and economic data. They are crucial for making significant decisions with far-reaching impacts, necessitating a comprehensive evaluation of their overall effectiveness. Accurate error estimation of predictors is vital not only as an indicator of system reliability and accuracy but also for selecting the best forecasting method and parameter configuration. Many researchers have proposed various methods to measure the accuracy of forecast algorithms for univariate time series data. However, these methods often have limitations, lack precise definitions, or are unbounded, potentially leading to incorrect conclusions. Significant research has been conducted on the potential of artificial intelligence (AI), specifically machine learning (ML) methods and neural networks (NNs), to enhance time series forecasts. AI has made considerable contributions to forecasting, with numerous publications introducing new ML algorithms, methodologies, and accuracy improvements. The primary objective of both ML and statistical techniques is to minimize a loss function, typically the sum of squared errors, to improve forecasting accuracy. They differ in their approaches: statistical methods use linear processes, while ML methods employ non-linear algorithms. ML techniques are computationally more complex and depend heavily on computer science, situating them at the intersection of statistics and computer science. It is crucial to objectively assess the overall performance of ML approaches in forecasting, as this has not been thoroughly done, raising concerns about their effectiveness in advancing forecasting res8earch and accuracy (Muhammed Irshad M et al, 2024) [23] . Numerous studies have employed NN techniques, comparing their accuracy with traditional statistical methods. Neural networks were first used for forecasting in 1964, but significant progress was made only after the introduction of the backpropagation algorithm about 20 years later (Zhao, 2009) [24 ] . Ahmed et al. (2010) [2] reviewed numerous studies, some dating back to 1995, and noted that the results were mixed. Similarly, Adya and Collopy (1998) [1] analyzed 48 NN experiments and found inconsistent results compared to statistical approaches. The first large-scale study using 3003 time series was the M3 Competition, published in 2000. Makridakis (2000) [14] utilized an Automated Artificial Neural Network (AANN) method, which performed below the most accurate statistical methods in the competition. Eleven years later, Crone et al. (2011) [6] published the results of a specialized NN competition using a subset of the M3 monthly data, comparing 22 NN and computational intelligence (CI) methods alongside 11 statistical methods. The primary goal of this research is to empirically determine whether ML techniques outperform statistical methods and how their www.extensionjournal.com advantages can be leveraged to improve forecasting accuracy. Previous studies concluded that no ML approach surpassed the Theta method. This research aims to establish empirically the conditions under which ML methods may be superior to statistical approaches and how their benefits can be utilized to enhance forecasting accuracy.

Statistical and ML methods

To evaluate the performance of machine learning (ML) algorithms against classical statistical approaches, we considered two accurate statistical methods: Simple Exponential Smoothing (SES) and Autoregressive Integrated Moving Average (ARIMA). SES is designed to forecast series without a trend, while ARIMA is employed to understand the data and predict future trends. In contrast, the machine learning approaches utilized in this evaluation include the Multi-Layer Perceptron (MLP) and Bayesian Neural Network (BNN).

Statistical Methods

Simple Exponential Smoothing (SES)

Exponential smoothing is a time series forecasting method originally designed for univariate data but can be adapted to handle data with a systematic trend or seasonal component. As a powerful alternative to the well-known Box-Jenkins ARIMA family of models, this technique produces forecasts by calculating weighted averages of past observations, with weights that decrease exponentially over time. Essentially, more recent observations carry more weight.

The forecast for the next period ) is determined by assigning a weight (α) to the most recent observation ( ) and a weight of 1 -α to the most recent forecast ( ). A smaller value of 𝛼 (e.g., 0.1) results in greater smoothing, while a larger value of 𝛼 (e.g., 0.9) provides less smoothing. Alternatively, one can select 𝛼 from a range of values (e.g., 𝛼 = 0.1, 0.2, …, 0.9) and choose the one that minimizes the Mean Squared Error (MSE).

Autoregressive Integrated Moving Average (ARIMA)

The ARIMA model is one of the most significant and commonly utilized models in time series analysis. Its widespread use is largely due to its robust statistical characteristics and the well-established Box-Jenkins methodology for model construction. The ARIMA (p,d,q) model, where p, d, and q represent the orders of the autoregressive, differencing, and moving average components, respectively can be formulated as follows:

The backshift operator B is defined as = , where, represents the value of the price series at time t. The differencing operator Δ is expressed as . The polynomials in φ(B) and θ(B) have degrees p and q, respectively. Additionally, the disturbance term at time t, denoted as , is a random variable characterized by a mean of zero and a constant variance of

Machine Learning Methods 2.2.1 Multi-Layer Perceptron (MLP)

The multilayer perceptron (MLP), as shown in Figure 1, is a network of interconnected neurons or nodes that enables nonlinear mapping between an input layer and an output layer within a model. The connections between these nodes are represented by weights, and the output signals are generated through simple nonlinear activation functions based on the sum of the node's inputs. These weights scale the output from one node and pass it as input to nodes in the next layer of the network. This directional flow of information is what characterizes the multilayer perceptron as a "feed-forward neural network." In the MLP structure, the input layer receives the inputs, and the output layer produces the outputs. Typically, an MLP consists of an output layer and at least one hidden layer. Each node in the MLP is fully connected to the nodes in the adjacent layers. Research has shown that with appropriate adjustments to the weights and activation functions, a multilayer perceptron can approximate any smooth, measurable function between the input and output vectors (Hornik et al., 1989) [10] . The MLP is a widely used neural network architecture for both classification and regression tasks in modern applications. The formulation of the MLP is as follows:

In this context, represents the weight vector associated with the hidden node, while denote the weights for the output node. The network's output is denoted by .The input vector 𝑥̀ is derived by augmenting the original input vector x with a constant term, such that 𝑥̀

=

The function g captures the output of the hidden nodes, typically defined by a squashing function. The Multi-Layer Perceptron (MLP) is highly parameterized, with its complexity influenced by the choice of the number of hidden nodes NH (Van Dijk et al., 2002) [21] .

Bayesian Neural Network (BNN)

A Bayesian neural network (BNN) is a neural network constructed according to a Bayesian probabilistic formulation (MacKay, 1992), thereby linking BNNs to the Bayesian parameter estimation principle in classical statistics and the regularization concept found in methodologies such as ridge regression. The methodology of BNN shares similarities with the Multilayer Perceptron (MLP) approach, as it optimizes network parameters using Bayesian principles, wherein weights are estimated under predefined error distributions. BNN finds extensive applications across various domains, including engineering, finance, and economics. Its fundamental objective lies in treating network parameters or weights as random variables, governed by predefined distributions that promote models of moderate complexity or those generating smooth fits. Posterior distributions of weights are evaluated after data observation, enabling computation of network predictions. In the given formulation, 𝐸𝐷 represents the summation of the square errors in the network outputs, 𝐸𝑊 denotes the summation of the squares of the network parameters, specifically the weights, and 𝛼 stands for the regularization parameter.

In the Bayesian technique, a common practice for selecting the prior involves utilizing a normal density distribution. This choice assigns augmented significance to smaller network parameter values.

Where L denotes the number of parameters (weights). The posterior is then given by P (6)

Let D denote the gathered dataset. The probability density of the dataset given the parameters can be computed as follows, assuming normally distributed errors:

By replacing the density expressions in both the prior and the data's probability density function into the posterior, where M denotes the number of training data points, we can gain a deeper insight into the relationship between these variables.

Where c is normalizing constant. The regularization constant is also determined using Bayesian principals, from

To achieve optimal weights and parameters, it is essential to maximize p(𝑤/𝐷, 𝖺) and p(𝖺/𝐷) correspondingly. The expression for p(𝐷/𝖺) involves a quadratic approximation of J concerning the weights, followed by the integration of these weights.

Data Pre-processing

A data mining approach known as data pre-processing involves transforming raw data into a more comprehensible format. Real-world data often exhibits inadequacies, inconsistencies, and deficiencies in specific patterns or trends, leading to numerous inaccuracies. In the evolution of time series collection techniques, achieving stationarity in both mean and variance is deemed crucial. Within the realm of machine learning (ML), differing perspectives exist regarding the necessity of pre-processing. Some studies assert that ML methods are inherently capable of effectively modeling various data patterns and can thus be applied directly to the original data. Conversely, other research contends that without adequate pre-processing, ML algorithms may exhibit instability and yield subpar results.

The effectiveness of subsequent forecasting endeavors can be greatly influenced by pre-processing the time series data.

Modifying the time series data through pre-processing techniques simplifies the task of prediction modeling. Three primary types of pre-processing methods include trend removal, log or power transformations, and seasonal adjustments. MLP (Multi-Layer Perceptron) models are often unable to accurately capture seasonality, despite claims to the contrary. The activation function of MLP models tends to exhibit instability, which can be mitigated by detrending the series, thereby achieving greater consensus regarding the trend.

Original data:

No pre-processing is applied.

Transforming the data 3.2.1 Log Transformation

Log transformation is applied to reduce the variability in the data. Among various transformation techniques employed to mitigate skewness in data and approximate it to normality, log transformation stands out as a prominent method. When the original data conforms to a log-normal distribution or closely resembles one, the application of log transformation results in a dataset that conforms to a normal or nearly normal distribution.

Box-Cox Transformation

In the context where t represents the time period and 𝜆 denotes a parameter, with w signifying the modified variable and y indicating the target variable, it is noted that when the parameter value equals one, the dataset inherently exhibits a normal distribution, thereby obviating the necessity Box-Cox transformation.

Data description

The study utilized the monthly wholesale price data of potatoes in the Delhi market as a case study, sourced from the Ministry of Consumer Affairs webpage for India, spanning from January 2010 to July 2020. The original pricing series was partitioned into two subsets: a training set consisting of 115 observations, and a testing set comprising data from the preceding year for validation purposes. Descriptive statistics of the three series employed in the analysis are provided in Table 1, while Figure 2 depicts the time plots of the respective series.

Accuracy measures

Two measures of accuracy are used first is SMAPE and MASE

Symmetric Mean Absolute Percentage Error (SMAPE) sMAPE =

In this scientific framework, let k indicate the forecasting horizon, denote the actual observations, and represent the forecast produced by the model at time t. It is important to note that the symmetric Mean Absolute Percentage Error (sMAPE) imposes heavier penalties for large positive errors than for negative ones.

Mean Absolute Scaled Error (MASE)

MASE =

In time series analysis, where n denotes the number of historical observations and m indicates the frequency of the time series, it is important to note that the Mean Absolute Scaled Error (MASE) is characterized by its scale independence, among other properties.

Results and Discussion

Today's forecasting heavily relies on machine learning techniques, which are expected to yield higher overall accuracy. Table 2 presents 12-month estimates generated using a range of time series models, showcasing the forecasting performance of these models and benchmark models against two evaluation criteria for a 12-month forecast horizon, as detailed in Table 3. Figure 4 illustrates that statistical methods outperform ML approaches. Specifically, MLP achieves a 2.68% lower SMAPE value compared to ARIMA, which is somewhat unexpected. On the other hand, BNN performs favorably in comparison to statistical methods.

To enhance the accuracy of MLP, the most effective approach is preprocessing the data, which involves ensuring data stability and reducing noise points. While there are several preprocessing techniques available, we have applied Log and Box-Cox transformations to the same dataset, resulting in improved sMAPE results when compared to preprocessing alone. However, this transformation does not lead to an increase in MASE accuracy for MLP methods.

Conclusions

While it may be disappointing that machine learning models exhibit lower forecasting accuracy compared to statistical methods, researchers hold a strong optimism regarding their vast potential in forecasting applications. Clearly, there is a need for additional efforts to develop these systems, a common requirement for new techniques, including complex forecasting methods that have significantly improved accuracy over time. This study's key contribution is highlighting that traditional statistical techniques outperform machine learning techniques, underscoring the urgency of identifying underlying causes and developing solutions. Data pre-processing plays a critical role in enhancing the performance of ML algorithms, and for better results, it is preferable to employ log pre-processing methods rather than ML methods, simplifying the data.

At this juncture, several empirically verifiable suggestions/speculations can be put forth as potential ways forward for ML methods:  Pre-processing data before applying ML techniques, leading to simpler models, reduced computation requirements, and faster learning.  Grouping series into homogeneous categories and constructing ML algorithms optimized for each.  Guarding against over-fitting, as it remains uncertain whether ML models can effectively distinguish noise from data patterns.  Incorporating uncertainty estimation in point forecasts and providing data for creating confidence intervals around these projections.

Acknowledgement

Competing interestsThe authors have no potential conflicts to report that are important to the article's content.

References

1. Adya M, Collopy F. How effective are neural networks at forecasting and prediction? A review and evaluation. J Forecast. 1998;17(5-6):481-495.
2. Ahmed NK, Atiya AF, Gayar NE, El-Shishiny H. An empirical comparison of machine learning models for time series forecasting. Economet Rev. 2010;29(56):594-621.
3. Assimakopoulos V, Nikolopoulos K. The theta model: A decomposition approach to forecasting. Int J Forecast. 2000;16(4):521-530.
4. Bergmeir C, Benítez JM. On the use of cross-validation for time series predictor evaluation. Inform Sci. 2012;191:192-213.
5. Crone SF, Hibon M, Nikolopoulos K. Advances in forecasting with neural networks? Empirical evidence from the NN3 competition on time series prediction. Int J Forecast. 2011;27(3):635-660.
6. Gardner Jr ES. Exponential smoothing: The state of the art-Part II. Int J Forecast. 2006;22(4):637-666.
7. Goodwin P, Lawton R. On the asymmetry of the symmetric MAPE. Int J Forecast. 1999;15(4):405-408.
8. Green KC, Armstrong JS. Simple versus complex forecasting: The evidence. J Bus Res. 2015;68(8):16781685. www.extensionjournal.com
9. Hornik K, Stinchombe M, White H. Multi-layer feedforward networks are universal approximators. Neural Networks. 1989;2:359-366.
10. Hyndman RJ, Koehler AB. Another look at measures of forecast accuracy. Int J Forecast. 2006;22(4):679-688.
11. MacKay DJ. Bayesian interpolation. Neural Comput. 1992;4(3):415-447.
12. Ismail MM, Nicholson A, Abu-Mostafa Y. Financial markets, very noisy information processing. Proc IEEE. 1998;86(11):2184-2195.
13. Makridakis S, Hibon M. The M3-Competition: results, conclusions, and implications. Int J Forecast. 2000;16(4):451-476.
14. Makridakis S. The forthcoming Artificial Intelligence (AI) revolution: Its impact on society and firms. Futures. 2017;90:46-60.
15. Makridakis S, Spiliotis E, Assimakopoulos V. Statistical and Machine Learning forecasting methods: Concerns and ways forward. PLOS One. 2018;13(3).
16. Leshno M, Lin VY, Pinkus A, Schocken S. Multilayer feedforward networks with a nonpolynomial activation function can approximate any function. Neural Networks. 1993;6(6):861-867.
17. Lippmann R. An introduction to computing with neural nets. IEEE ASSP Mag. 1987;4(2):4-22.
18. Sharda R, Patil RB. Connectionist approach to time series prediction: An empirical test. J Intell Manuf. 1992;3(5):317-323.
19. Dijk VD, Terasvirta T, Franses PH. Smooth transition autoregressive models: A survey of recent developments. Economet Rev. 2002;21:1-47.
20. Zhang GP, Qi M. Neural network forecasting for seasonal and trend time series. Eur J Oper Res. 2005;160(2):501-514.
21. Muhammed Irshad M, Kader Ali Sarkar, Digvijay Singh Dhakre, Debasis Bhattacharya. Forecasting of monthly cardamom price using long memory time series modelling technique. Int J Stat Appl Math. 2024;9(6):121-126. DOI:10.22271/maths.2024.v9.i6b.1906.
22. Zhao L. Neural networks in business time series forecasting: Benefits and problems. Rev Bus Inf Syst. 2009;13(3).