Construction and modeling of the operation of elements of computing technology on fast neurons

Full Text

1. Introduction The need to increase the speed of artificial neural networks (ANN) containing a huge number of neurons leads to the construction of “fast neurons”, which is achieved by a special implementation of activation functions [1-3] and their hardware implementation. This has led to the expansion of research in the field of creating and studying new activation functions and their practical use. Currently, great efforts are aimed at accelerating the operation of activation functions for the construction of “fast” neurons and neural networks based on them. A comparison of various typical activation functions (AFs) in the ANN is performed in the works [4-6]. The issues of searching for methods to reduce the computational complexity of implementing AFs are discussed in [7-9]. Of interest is the creation of elements of computer technology (CT) using complete logical bases (complete systems of logical functions). These are, for example, ∧, ∨, (conjunction, disjunction, negation); ∧, (conjunction, negation); ∨, (disjunction, negation). The devices are implemented by covering them with neurons and neural networks. Interest in this approach is not weakening and is currently growing with the increasing interest in artificial intelligence technologies. It is worth noting the emergence of a class of specialized processors that serve to accelerate the work of neurons and the training algorithms of ANNs as a whole [10]. In this paper, we propose to implement typical elements of the VT on “fast” neurons obtained by replacing known activation functions with new functions that are characterized by a higher implementation speed. Previously, the authors proposed a new activation function called the sparabola, created a complete logical basis and algorithms for setting up such neurons and neural networks [11, 12]. In this paper, the main focus is on the construction of fast neurons using new activation functions s-parabola, Sparabola-ReLU and ReLU-Sparabola, which are combinations of parabolas and linear functions. Based on them, fast VT elements are proposed: “XOR”, trigger, half adder, adder. Structure of the article In section 2. Materials and Methods in paragraph 2.1. The basic requirements for activation functions are presented. Section 2.2. Standard and new activation functions propose new functions that can be used in ANNs configured using the backpropagation method. The settings and quality parameters of neurons that form the logical bases “AND”-“OR”-“NOT” and the bases “AND-NOT”, “OR-NOT” are shown. The settings are linked to the basic neuron schemes (one neuron, two neurons, constructor) for configuring activation functions. In section 3. Results in paragraph 3.1. show the settings of the activation functions on the logical basis “AND”-“OR”-“NOT” by the method of back propagation of the error. In relation to the basic circuits of neuron connection, comparative estimates of the speed of tuning and implementation of functions are given. Similar studies are performed in section 3.2. for tuning activation functions to the logical bases “AND-NOT”, “OR-NOT”. To expand the studies in section 3.3., new activation functions are additionally tuned to the XOR function. Section 3.4. considers the construction of VT elements on logical bases. The results of covering the circuits of the RS trigger of the half adder and adder with logical elements on neurons and the “XOR” function are shown. In section 4. Discussion discusses the obtained results in comparison with the state of the world level of development of the subject area. The prospects of the proposed approach are associated with the use of fast activation functions in multilayer and convolutional ANNs. It is noted that significant acceleration can be achieved by switching to hardware implementation of activation functions in the form of bit-parallel circuits and the use of CORDIC family algorithms. In section 5. Conclusion summarizes the results obtained and provides suggestions for further research development. 2. The materials and methods 2.1. Basic requirements for activation functions. Complete logical basis on traditional and fast neurons The requirements for activation functions are quite contradictory, but at the same time, the following features can be highlighted. 1. If neural networks are trained using the backpropagation method using the gradient descent process, then the layers in the model and the activation functions must be differentiable. Some activation functions, for example, linear or hyperbolic tangent, are differentiable over the entire range of admissible values. 2. The requirements for the activation function of an ANN are determined by a special theorem on completeness [13], according to which the function must be twice differentiable and continuous. The derivative of the activation function must be defined on the entire abscissa axis. To be used in a neuron, the function must be monotonically increasing or decreasing, have parameters that can be adjusted during the training process. 3. It is desirable that the output of the activation function be symmetrical with respect to zero, so that the gradients do not shift in a certain direction. Such a case corresponds, for example, to the sigmoid rational. 4. For logical problems, the excitation values of the output layer neurons must belong to the range [0, 1]. This corresponds, for example, to the sigmoid function and the unit jump (step function). However, the unit jump function does not meet the requirements of differentiability. It is not differentiable at point 0, and its derivative is 0 at all other points. Gradient descent methods do not work for such a function. This can create problems with training, since the numerical gradients calculated near the point where the derivative does not exist may be incorrect. 5. Since activation functions must be calculated repeatedly in deep networks, their calculation must be inexpensive in computational terms. This fact requires a revision of the implementation methods and, possibly, the creation of new functions, which we will call fast-acting. 2.2. Typical and new activation functions Table 1 presents activation functions that have found wide application in neural networks. The approach proposed in this paper is based on the idea. 1. Construction of fast neuron models based on the activation function of the “s-parabola” type and their application in individual ANNs and classifier committees. The “s-parabola” function has a structure in which the upper part (the first quarter) is the upper branch of the parabola, and the lower part is a mirror image of the lower part of the parabola relative to the ordinate axis (the third quarter). The graph and formula of the proposed activation function are presented in Table 2 . 2. Combining different types of activation functions to achieve an effective solution to the problem. The prospects and advantage of the s-parabola are associated with the simplicity of calculating the function, which ensures the speed of implementation. The s-parabola can be used as an activation function of the ANN, since it satisfies the established requirements of twice differentiability. A similar function can be used in multilayer direct propagation ANNs to solve problems of recognizing rigid objects and predicting time processes. Table 2 presents some new neuronal activation functions. Let us consider the construction of a complete logical basis on fast neurons. The implementation of the functions “AND”, “OR”, “NOT”, “AND-NOT”, “OR-NOT” is carried out on the basis of basic circuits with one and two neurons Table 3 . 2.3. Performance evaluation of activation functions Let be the bit depth of the numbers being processed, then we can estimate the complexity of executing the activation functions (Table 3). Table 3 Activation functions sorted in order of decreasing complexity Function Computational complexity Rating (for = 8) SiLu 2((log )2 + log + 1) + 840 Sigmoid 2(1 + (log )2 + log ) 832 Hyperbolic tangent 2((log )2 + log ) + 2 + 1 785 Sigmoid-rational (1 + log ) 200 S-parabola 2 2 128 ReLU-Sparabola , + 2 2 (8, 136), average 72 Sparabola-ReLU + 2 2, (136, 8), average 72 ReLU + 1 9 For ReLU-Sparabola, Sparabola-ReLU the complexity of the implementation depends on which part of the function is executed (left or right). 2.4. Structural diagrams of neurons and their constructs Variants of neuron schemes are presented in Fig. 1. w12 y x w22 a) b) c) d) Single neuron circuits Circuits with two neurons Figure 1. Basic neuron schemes for setting up activation functions 3. Results The activation functions are configured using the backpropagation method for the logical functions “OR”, “AND”, “NOT” (Table 4-6). The calculation of speed characteristics was performed on a personal computer with the following parameters: processor: Intel Core i5-6600K @ 3.50 GHz; RAM: 32 GB. 3.1. Activation function settings on the logical basis “AND”-“OR”-“NOT” Table 4 Comparative characteristics of the setting for the “OR” function on one neuron Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights 1 = 1, 2 = 1, = -0.5 Setup results 1 = 1.385, 2 = 1.351, = -0.516 1 = 4.963, 2 = 5.071, = 3.812 1 = 0.475, 2 = 0.483, = -0.004 1 = 15.685, 2 = 15.712, = -6.967 0.1 0.1 2.0 - 0.5 -0.5 0.25 - - - - 0.5 Average deviation 0.124 0.189 0.181 0.009 Training time, ms 648 7307 27072 1483 Time to implement 10000000 data table cycles, ms 621 645 578 1040 Table 5 Comparative characteristics of the setting for the “AND” function on one neuron Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights 1 = 1, 2 = 1, = -1.5 Setup results 1 = 1.268, 2 = 1.359, = -1.756 1 = 0.516, 2 = 0.405, = 0.098 1 = 0.874, 2 = 0.874, = -0.514 1 = 16.338, 2 = 16.338, = -24.899 0.1 1.75 0.1 - 0.5 -1.0 -0.3 - - - - 0.5 Average deviation 0.123 0.274 0.098 0.010 Training time, ms 5942 602 34332 3159 Time to implement 10000000 data table cycles, ms 621 636 562 1053 Table 6 Comparative characteristics of the setting for the “NOT” function on one neuron Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights = -1, = 0 Setup results = -3.994, = 6.246 = -3.993, = 6.246 = -0.999, = 0.999 = -18.328, = 8.688 0.5 0.5 0.5 - -1.5 -1.5 0 - - - - 0.5 Average deviation 0.001 0.001 0.0001 0.010 Training time, ms 219 218 218 2719 Time to implement 10000000 data table cycles, ms 220 219 183 2843 3.2. Settings of activation functions for logical bases “AND-NOT”, “OR-NOT” Table 7-12 presents the results of the implementation of the alternative full basis “AND-NOT” and “OR-NOT” using one and two neurons. Table 7 Characteristics of the setting for the “AND-NOT” function on one neuron Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights - 1 = 1, 2 = 1, = -1.5 Setup results 1 = -1.040, 2 = -1.035, = 0.515 1 = -0.585, 2 = -0.546, = 1.871 1 = -0.510, 2 = -0.562, = 3.177 1 = -16.373, 2 = -16.405, = 24.852 0.9 1 4 - 1.75 -0.75 -2 - - - - 0.5 Average deviation 0.286 0.186 0.220 0.010 Training time, ms 1198 5064 8801 3223 Time to implement 10000000 data table cycles, ms 692 655 581 1043 Table 8 Characteristics of setting for the “AND-NOT” function on two neurons (Fig. 1(d)) Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights 11 = -7, 21 = -7, 12 = -7, 22 = -4, 12∗ = -11 01 = -2.6, 02 = 10 - Setup results 11 = -13.998, 21 = -12.525, 12 = -1.530, 22 = -1.461, 12∗ = -0.326, 01 = 1.289, 02 = 1.353 11 = -6.892, 21 = -6.691, 12 = 0.364, 22 = 0.341, 12∗ = 0.193, 01 = 7.415, 02 = 0.469 11 = -12.926, 21 = -12.057, 12 = -1.470, 22 = -1.431, 12∗ = -0.297, 01 = 1.583, 02 = 1.463 11 = -0.073, 21 = -0.593, 12 = -4.991, 22 = -4.559, 12∗ = 3.812, 01 = 0.559, 02 = 5.303 0.5 0.5 0.5 - 0.01 0.01 0.01 - - - - 1.0 Average deviation 0.009 0.006 0.009 0.039 Training time, ms 538 450 506 10311 Time to implement 10000000 data table cycles, ms 39200 32600 36700 4303800 Table 9 Results of the implementation of the “AND-NOT” function on two neurons (Fig. 1(c)) 1 2 S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid 0 0 1.072 1.310 1.621 0.987 1 1 0.107 0.142 0.066 0.010 0 1 0.818 0.722 0.940 0.985 1 0 0.844 0.593 0.940 0.985 Table 10 Comparative characteristics of setting for the “OR-NOT” function on one neuron Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights - 1 = 1, 2 = 1, = -0.5 - 1 = 1, 2 = 1, = -0.5 Setup results 1 = -0.135, 2 = -0.135, = 0.158 1 = -0.325, 2 = -0.420, = 0.870 1 = -0.448, 2 = -0.515, = 1.583 1 = -16.226, 2 = -16.238, = 7.139 4.0 2.0 3.0 - -0.25 -1.0 -0.75 - - - - 0.5 Average deviation 0.296 0.231 0.215 0.010 Training time, ms 5800 2232 6155 1597 Time to implement 10000000 data table cycles, ms 647 677 614 1120 Table 11 Characteristics of setting for the “OR-NOT” function on two neurons (Fig. 1(d)) Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights 11 = -7, 21 = -7, 12 = -7, 22 = -4, 12∗ = -11 01 = -2.6, 02 = 10 - Setup results 11 = -12.977, 21 = -12.665, 12 = 0.511, 22 = 0.496, 12∗ = 0.327, 01 = 1.307, 02 = 0.603 11 = -6.932, 21 = -6.747, 12 = -1.395, 22 = -1.375, 12∗ = -0.204, 01 = 7.709, 02 = 1.558 11 = -12.441, 21 = -12.075, 12 = 0.466, 22 = 0.449, 12∗ = 0.301, 01 = 1.530, 02 = 0.526 11 = 0.508, 21 = 0.536, 12 = -9.564, 22 = -9.557, 12∗ = -0.360, 01 = -0.464, 02 = 3.900 0.5 0.5 0.5 - 0.01 0.01 0.01 - - - - 0.5 Average deviation 0.010 0.007 0.010 0.037 Training time, ms 377 819 279 18819 Time to implement 10000000 data table cycles, ms 1536 1488 1456 2512 Table 12 Results of the implementation of the “OR-NOT” function on two neurons (Fig. 1(c)) 1 2 S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid 0 0 0.852 0.681 0.874 0.983 1 1 -0.240 -0.236 -0.452 0.008 0 1 0.178 0.209 0.271 0.009 1 0 0.107 0.218 0.279 0.009 The tables show that when implementing a logical basis on two neurons using parabolic activation functions, the tuning results exceed the results of implementation on a sigmoid in accuracy and speed. Sigmoid is worst tuned to the “AND-NOT” function. When implemented on one neuron, sigmoid works more accurately, although slower. 3.3. Tuning neurons to the “XOR” function To expand the research, new activation functions were additionally configured using the backpropagation method on the “XOR” function using the ANN constructed according to the scheme in Fig. 1(d). The results are shown in Table 13. Table 13 Characteristics of setting for the “XOR” function on two neurons (Fig. 1(d)) Characteristics of training S-parabola ReLU-Sparabola Sparabola-ReLU Sigmoid Starting weights 11 = -7, 21 = -7, 12 = -7, 22 = -4, 12∗ = -11 01 = -2.6, 02 = 10 Setup results 11 = -14.176, 21 = -13.494, 12 = -2.025, 22 = -1.963, 12∗ = -0.633, 01 = 1.375, 02 = 0.748 11 = -6.872, 21 = -6.686, 12 = 1.733, 22 = 1.691, 12∗ = 0.392, 01 = 7.547, 02 = -1.079 11 = -13.496, 21 = -12.694, 12 = -1.946, 22 = -1.876, 12∗ = -0.586, 01 = 1.557, 02 = 0.918 11 = -6.999, 21 = -6.999, 12 = -6.275, 22 = -6.154, 12∗ = -14.208, 01 = 2.613, 02 = 9.382 0.5 0.5 0.5 - 0.01 0.01 0.01 - - - - 1.0 Average deviation 0.009 0.010 0.008 0.035 Training time, ms 603 542 550 23681 Time to implement 10000000 data table cycles, ms 1944 1511 1474 2489 Comment 0 0 1 1 Prohibition 0 1 1 0 Record “1” 1 0 0 1 Record “0” 1 1 × × Storage a) RS trigger on “AND-NOT” logic circuits b) Truth table Figure 2. Trigger on “AND-NOT” elements Comment 0 0 × × Storage 0 1 1 0 Record “1” 1 0 0 1 Record “0” 1 1 0 0 Prohibition a) RS trigger on “OR-NOT” logic circuits b) Truth table Figure 3. Trigger on “OR-NOT” elements It is evident that the main trends identified in the compilation of logical bases are also preserved in the implementation of a more complex “XOR” function, which is widely used in computer elements. 3.4. Construction of CD elements on logical bases The CD circuits are implemented by covering them with neurons and neural networks configured on the logical bases “AND”-“OR”-“NOT”, “AND-NOT”, “OR-NOT” or with the sequential formation of more complex elements. Fig. 2 and Fig. 3 show the results of such coverage of the RS-trigger circuits with logical elements on the neurons “AND-NOT” and “OR-NOT”. Fig. 4a shows the circuit diagrams of some basic elements of the CD, implemented in the logical basis of “OR-NOT”, and Fig. 4b shows with the addition of the “single jump” element. Fig. 5a shows the implementation of the RS trigger on two “AND-NOT” neurons, and Fig. 5b adds a “single jump” type element to the outputs, which allows us to obtain clear values of “1” and “0” at the outputs. The results of the RS-trigger simulation are presented in Tables 16-17. Let us consider the construction of a neural network for the implementation of a half adder. A single-bit adder (half adder), designated as SM/2, does not have an input carry, since it is the least significant bit of a multi-bit adder. A half adder can be built on the basis of XOR logic circuits and the AND circuit (Fig. 6-7). The logic of SM/2 operation is determined by the truth table. Based on two half adders and an “OR” circuit, a full one-bit adder SM can be constructed as shown in Fig.8-9. a) using the outputs of “AND-NOT” neurons b) adding neurons with the activation function “single jump” Figure 4. RS trigger in which the “AND-NOT” element is implemented on one neuron Table 14 Results of the RS trigger operation, in which the “AND-NOT” element is implemented on one neuron 1 2 1 2 0 0 Prohibition 1 1 Storage 0 1 2.713 -0.702 1 0 0.074 2.638 Table 15 Results of the RS trigger operation, in which the “AND-NOT” element is implemented on one neuron with the addition of neurons with the “single jump” activation function 1 2 1 2 0 0 Prohibition 1 1 Storage 0 1 1 0 1 0 0 1 Table 16 Results of the RS trigger operation, in which the “AND-NOT” element is implemented on two neurons 1 2 1 2 0 0 Prohibition 1 1 Storage 0 1 0.999 -0.059 1 0 -0.061 0.992 Table 17 Results of the RS trigger operation, in which the “AND-NOT” element is implemented on two neurons with the addition of neurons with the “single jump” activation function 1 2 1 2 0 0 Prohibition 1 1 Storage 0 1 1 0 1 0 0 1 y1 y2 a) using “AND-NOT” neuron outputs b) adding neurons with the activation function “single jump” Figure 5. RS trigger in which the “AND-NOT” element is implemented on two neurons 1 2 0 0 0 0 0 1 1 0 1 0 1 0 1 1 0 1 a) Half adder circuit (SM/2) b) Truth table Figure 6. Logical diagram of the half adder (SM/2) y x2 x2 Pi a) base model b) model with additional neurons (with the “single jump” function) Figure 7. Models of a half-adder (SM/2) on neurons 1 2 -1 0 0 0 0 0 0 0 1 1 0 0 1 1 0 1 1 0 0 1 0 1 0 1 0 1 1 1 0 0 1 1 1 1 1 1 a) Full single-bit adder (SM) b) SM operation logic Figure 8. Logic diagram of the sub adder (SM) Figure 9. Model of a full adder (SM) on neurons 4. Discussion Comparison of the obtained results with existing works in the field of implementation of activation functions shows that of interest is not only the decomposition of a complex activation function, for example, a sigmoid, into simpler functions in order to speed up its implementation [1, 2], but also a direct replacement of the function with other functions, for example, an s-parabola and its variations, which allow a more efficient implementation in time. In this case, the effect of acceleration is ensured by several times with some loss in the accuracy of the implementation of thresholds [0, 1]. Undoubtedly, of interest is also some slight complication of the VT schemes due to the addition of neurons with the traditional single-jump function to the final cascades, as shown in paragraph 3.4. Another aspect of acceleration is related to the efficient hardware implementation of activation functions. The special organization of “fast” neurons and neural networks is the subject of, for example, the works [14-16]. Great hopes are placed on bit-parallel circuits for calculating functions included in activation functions. We note a series of works aimed at increasing the speed of solving various problems of comparison, establishing correspondence between streams and performing arithmetic operations on bit vectors, including simultaneous processing of matrix columns [17-20]. In this case, the expected success is associated with the transition to CORDIC algorithms [21-23], which allow regulating the speed and accuracy of calculations. This direction will be developed in subsequent works by the authors. 5. Conclusions Summing up, we can recapitulate. 1. The composition of activation functions based on the parabola has been expanded, including S-parabola, ReLU-Sparabola and Sparabola-ReLU. 2. Activation functions have been constructed that implement complete logical bases “OR-ANDNOT” and “AND-NOT, OR-NOT” on neurons and neural networks with new activation functions. Compared to implementations based on the “Sigmoid” function, in the general case, the implementation is accelerated by 1.6-1.8 times with a small possible loss in accuracy and setup time. Setup time is not a decisive factor here, since after setup, the neurons are further used without changing the established coefficients, which are fixed. 3. Typical elements of computing equipment have been implemented on them: triggers, half adders, adders. 4. It is planned to implement the s-parabola activation function in multilayer ANNs, including convolutional neural networks of the YOLO class. 5. The possibilities and applications of new activation functions in convolutional neural networks will be investigated. 5. To speed up the calculation of activation functions, bit-parallel calculation schemes will be proposed using the CORDIC family of algorithms.

About the authors

Mikhail V. Khachumov

RUDN University; Federal Research Center “Computer Science and Control” of Russian Academy of Sciences

Email: khmike@inbox.ru
ORCID iD: 0000-0001-5117-384X
Scopus Author ID: 55570238100

Candidate of Physical and Mathematical Sciences, Senior Researcher

6 Miklukho-Maklaya str., Moscow, 117198, Russian Federation; 9 60-let Octyabrya prosp., Moscow, 117312, Russian Federation

Yuliya G. Emelyanova

Ailamazyan Program Systems Institute of RAS

Email: yuliya.emelyanowa2015@yandex.ru
ORCID iD: 0000-0001-7735-6820
Scopus Author ID: 57202835704

Candidate of Technical Sciences, Senior Researcher

4a Peter the Great str., Pereslavl-Zalessky, Yaroslavl Region, 152021, Russian Federation

Vyacheslav M. Khachumov

RUDN University; Federal Research Center “Computer Science and Control” of Russian Academy of Sciences; Ailamazyan Program Systems Institute of RAS

Author for correspondence.
Email: vmh48@mail.ru
ORCID iD: 0000-0001-9577-1438
Scopus Author ID: 56042383100

Doctor of Technical Sciences, Chief Researcher

6 Miklukho-Maklaya str., Moscow, 117198, Russian Federation; 9 60-let Octyabrya prosp., Moscow, 117312, Russian Federation; 4a Peter the Great str., Pereslavl-Zalessky, Yaroslavl Region, 152021, Russian Federation

References

Supplementary files