Regression Neural Networks Advantage over Classical Regression Analysis

Full Text

Introduction Regression Analysis The current realities of the modern economy lead to many risky challenges for companies in various industries. They are accompanied by drastic changes that negatively affect the company’s per-formance and its sustainability as a system. These circumstances require organizations to make operational management decisions, their system flexibility, which can adapt to the influence of random factors influencing it both internally and externally. However, in order for an enterprise to carry out competent manage-ment, there is a need to predict such situations, which leads to the search for tools for predicting and promptly responding to emerging negative situations [1; 2]. The fundamental criterion for competent management in organizations is the predictability of the system, in this case the organization is this system. This criterion is necessary for the pre-mature neutralization of negative consequences from the influence of various external and internal factors. For example, a number of key employees leave the organization; after such an event, the business must immediately respond and take various actions to eliminate the consequences. The best option would be to implement a department-function diagram, so that if several units suddenly leave the department, other participants in the process can take over their functions. Alternatively, another example, it is necessary to monitor the market for the inculcation of innovations to increase competitiveness. After receiving statistics about the market needs, the business needs to make a conclusion about what product the company needs to make. Recent events related to the pandemic have also hit the market hard, making it clear that systems are needed to offer management solutions that can keep costs to a minimum. All these and many other factors negatively affect organizations, in connection with which there was a need to introduce models that can warn in advance about such incidents, and offer ways to eliminate them. As a rule, one of the best solutions will be to carry out statistical monitoring and data analysis through mathematical modeling of production pro-cesses. For which one of the most effective methods will be the using regression analysis, which allows you to predict the economic impact based on retrospective data. The English statistician F. Galton [1] first introduced the term “regression”. Regression analysis, derived from the Latin word “regression” meaning, “moving backwards”, is a comprehensive method for studying the interactions between various indicators. The key idea is to analyze the dependence of the effective indicator on a set of factor variables (for example, 1, 2, ), with each of the variables making a certain contribution to the function Y=f (xi), which describes their combined impact on the result under study. In the regression analysis process, the key is to establish exactly how these factor features affect the outcome measure and how changes in these features may affect the outcome. This study is based on the assumption that the performance indicator dynamics directly depends on changes in factor characteristics. While other variables that may also have an effect on outcome are treated as constants or averages to focus on the main change factors. Thus, regression analysis not only reveals relationships between variables, but also helps to understand how specific changes in some variables can systematically influence changes in others, providing valuable data for making informed decisions in various areas of scientific research. In the statistical analysis arsenal, there are two main types of regression models, each of which serves to study the interactions between variables and their influence on the result under study. A linear multiple regression model focused on studying the influence of one key factor attribute on a performance indicator, representing the relation-ship as a simple linear function. This means that changes in the performance indicator described as a direct consequence of changes in that factor. On the other hand, a nonlinear multiple regression model includes several factor features, whose impact on the performance indicator can be described using various mathematical functions such as exponential, logarithmic, polynomial or power functions. These models make it possible to reflect more complex and multi-level relationships, where each factor characteristic can make its own unique contribution, which can enhance or weaken the influence of other factors. It is especially important that both linear and nonlinear models can demonstrate both direct and inverse relationships between factor characteristics and the performance indicator. This means that an increase or decrease in one or more factor characteristics can lead to a corresponding increase or decrease in the effective indicator. Thus, under-standing the structure of relationships between variables through these models opens up prospects for in-depth analysis and forecasting in various fields of scientific research. Mathematical description of regression [3]: The essence of regression is as follows: at the input we have a vector X, we need to predict all values of Y(x) see formula . (1) The error function is the standard deviation, see formula (2) This problem is solved by taking the derivative of SE equal to zero, see formula (3) This will find a minimum, and since SE is a quadratic function, its minimum always exists. Therefore, it is not difficult to find β using formula (4) This completes the description of linear regression. In reality, linearity in the data is not always present, which leads to the need to introduce nonlinearity. Solving this problem on a large data set is a bad solution because it is not always possible. In such cases, it may be useful to train a neural network model to solve a regression problem, which has in its arsenal various methods based on gradient descent that provide predictors that are more accurate. Regression Neural Network A regression neural network is a type of artificial neural net-work specifically designed for analyzing and solving regression problems [4; 5]. These networks gracefully process continuous variables to produce quantitative predictions, unlike their classification cousins that define categories. Due to their ability to accurately predict numerical values, regression neural networks find their application in a wide range of problems. They play a key role in predicting asset prices, assessing economic performance, time series analysis and many other areas where the accuracy and reliability of forecasts is critical. These networks are becoming an indispens-able tool in industries that need to extract deep quantitative insights from complex data, providing professionals with valuable insights for informed decisions. In this context, regression neural net-works do not just process information, they uncover hidden patterns and trends, turning raw data into strategic knowledge that allows for the develop-ment of optimal strategic decisions. Let’s look at the key aspects of regression neural networks [6-8]: Error optimization. The core of regression models is formed by loss functions such as mean square error (MSE) [9] and mean absolute error (MAE) [10], which allow us to estimate deviations between predicted and true values. In this case, the main task is to regularly reduce deviations, allowing you to learn from errors and correct predictions. Architectural flexibility. The diversity of input data and unique task requirements dictate the design of regression networks, which can take the form of multilayer perceptrons (MLP) [11], con-volutional neural networks (CNN) [12], or recurrent neural networks (RNN) [13]. 1. Results Predicate finalization. The final chord in a regression network melody is its output layer, usually containing a single neuron with linear activation or none at all. This allows the network to express continuous variables, thereby ensuring accuracy and consistency in the output. Powerful optimization algorithms, including stochastic gradient descent (SGD), Adam and RMSprop, support the training efficiency of regression neural networks. These methods help reduce error on the training data, providing a fast and accurate approximation to the best possible prediction results. Each of these elements plays a critical role in the functioning of a regression neural network, making it not only a tool for com-pleting tasks, but also a platform for innovation and discovery in applications ranging from financial forecasting to medical research [2]. The approach provided in this work visualizes the performance of two different models - a linear regression model and a multilayer perceptron (MLP) neural network - using two key indicators: mean square error (MSE) and R-square (R²) [14]. These metrics are important for understanding how well each model predicts home prices in California’s open housing dataset (Table 1, 2). Table 1 Data on housing characteristics in California MedInc House Age AveRooms AveBedrms Popul AveOccup Latitude Longitude MedValue 8.3252 41.0 6.984127 1.023810 322.0 2.555556 37.88 -122.23 4.526 8.3014 21.0 6.238137 0.971880 2401.0 2.109842 37.86 -122.22 3.585 7.2574 52.0 8.288136 1.073446 496.0 2.802260 37.85 -122.24 3.521 5.6431 52.0 5.817352 1.073059 558.0 2.547945 37.85 -122.25 3.413 S o u r c e: by V.D. Saushkin Table 2 Data on home prices in California House_id Price 1 4.526 2 3.585 3 3.521 4 3.413 S o u r c e: by V.D. Saushkin Let’s look at what each graph represents and the meaning of the results. Learning curve. Learning curve is a graph showing changes in the performance of a neural model (Figure 1). Mean Square Error (MSE). MSE is a measure of the root mean square difference between estimated values (predictions) and actual values. Essentially, it captures the deviation of forecasts from true values, with an emphasis on more severe penalties for more severe errors due to squaring each term. Figure 1. Learning curve S o u r c e: by V.D. Saushkin Interpretation of the graph (Figure 2): The graph titled “MSE Comparison” for each model shows a bar indicating the MSE value obtained after testing each model. Lower MSE values are better because they indicate that the model’s pre-dictions are closer to the actual data. If one band is significantly lower than the other is, it indicates that the model fits the test data better. Data on the MSE indicators of linear regres-sion and neural network: Linear Regression - MSE: 0.55, Neural Network - MSE: 0.26, which is two times less (Figure 2). R-squared (R²), also known as the coefficient of determination, measures the proportion of variance in a dependent variable that can be pre-dicted from the independent variables. It indicates the degree of fit and is therefore a measure of how well unseen patterns can be predicted by the model, with a value ranging from 0 to 1. An R² of 1 indicates that the regression predictions fit the data perfectly. Interpretation of the graph (Figure 3): The graph titled “R² Comparison” displays bars for each model’s R² score. Higher R² values are preferred because they indicate that the model is accounting for a greater proportion of the variance in the dependent variable. A high R² means that the model explains most of the variability in the target variable based on its input characteristics. Data on R² indicators of linear regression and neural network: Linear Regression - R²: 0.57, Neural Network - R²: 0.80. Precision-Recall Curve (PR) [15-17] is a graph showing the relationship between correctly pre-dicted positive observations - accuracy, and the proportion of correctly predicted actual positive observations. The values of this curve vary in the same way as for ROC. The graphs of these curves are shown in Figure 4. These graphs show that the performance of the neural network is higher than the performance of reression analysis, for NN the indicator is equal 0.95, while for LR is equal 0.9. Figure 2. Diagram of the root mean square error between linear regression and neural network S o u r c e: by V.D. Saushkin Figure 3. R-squared (R²) diagram between linear regression and neural network S o u r c e: by V.D. Saushkin Figure 4. AUC-ROC and AUC-PR S o u r c e: by V.D. Saushkin Conclusion Linear Regression vs. Neural Network. Comparing the two methods on each graph gives you an idea of which model works better on the dataset in its current configuration. If the neural network bars are higher on the R² graph and lower on the MSE graph, it means that the neural net-work has a higher prediction accuracy than the linear regression model. It is very important to note that MLP may require tuning (e.g., adjusting the number of hidden layers, neurons, or iterations) to optimize its performance. Conversely, if linear regression is superior to or very close to MLP, it may indicate that the dataset is linearly separable or that the neural network needs additional functions and more complex architectures to exploit its capabilities. This information helps decide which model to deploy based on the tradeoffs between complexity, performance, and computational cost. For example, if the improvement in MSE and R² is small for a neural network compared to a linear regression, you may prefer the simpler and more interpretable linear model, especially in contexts where model interpretability is critical. Finally, these visual comparisons guide further exploration and tuning, providing an easy way to communicate model performance to stakeholders and make informed decisions about improving or deploying the model.

About the authors

Olga A. Saltykova

RUDN University

Author for correspondence.
Email: saltykova-oa@rudn.ru
ORCID iD: 0000-0002-3880-6662
SPIN-code: 3969-6707

PhD in Physical and Mathematical Sciences, Associate Professor of the Department of Mechanics and Control Processes, Academy of Engineering

6 Miklukho-Maklaya St, Moscow, 117198, Russian Federation

Vyacheslav D. Saushkin

RUDN University

Email: kingrailag@gmail.com
ORCID iD: 0009-0007-2812-184X
SPIN-code: 1525-5653

Graduate student of the Department of Mechanics of Control Processes, Academy of Engineering

6 Miklukho-Maklaya St, Moscow, 117198, Russian Federation

References

Supplementary files