JP7316073B2 - Arithmetic unit of neural network - Google Patents

Description

The present invention relates to a computation device for neural networks.

2. Description of the Related Art Conventionally, a plurality of activation functions may be used in neural network computing devices (see, for example, Patent Document 1). A sigmoid function, a Softmax function, a ReLU (Rectified Linear Unit) function, an identity function, or the like is used as the activation function, for example. In order to reduce the amount of calculation, new activation functions are devised every day.

JP 2018-92560 A

In conventional neural network computing devices, in order to speed up processing, hardware dedicated to programs related to activation functions is sometimes designed and the activation functions are implemented in hardware. In this case, there is a problem that it becomes difficult to deal with the new activation function.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an arithmetic device for a neural network in which an activation function is implemented in hardware to speed up processing and the activation function to be applied can be easily changed.

The present disclosure employs the following technical means to solve the above problems. The symbols in parentheses described in the claims and this section are an example showing the correspondence relationship with the specific means described in the embodiment described later as one aspect, and limit the technical scope of the present invention. not something to do.

In order to achieve the above object, the present invention provides a neural network computing device (116) comprising an activation function circuit (208) for performing computations using the activation functions for a plurality of types of activation functions. and a register (214) for storing designation of the activation function type in association with the thread.

According to the present disclosure, since the activation function is made into hardware to speed up the processing and the type of the activation function is specified by the register, the activation function to be applied can be easily changed.

1 is a block diagram showing the configuration of an arithmetic device according to an embodiment of the present invention; FIG. FIG. 4 is a diagram for explaining threads processed by the arithmetic device according to the embodiment of the present invention; 1 is a block diagram showing the configuration of a dataflow processor according to an embodiment of the present invention; FIG. 3 is a block diagram showing the configuration of a PE (Processing Element) according to the embodiment of the present invention; FIG. (a) is a diagram showing assembler code specifying a Parametric ReLU function as an activation function according to an embodiment of the present invention; (b) A diagram showing an assembler code specifying a Hyperbolic Tangent function as an activation function according to an embodiment of the present invention. (a) A diagram showing a graph of a Parametric ReLU function and its differential function as an activation function according to the embodiment of the present invention. (b) A diagram showing a graph of a Hyperbolic Tangent function and its differential function as an activation function according to the embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In addition, the embodiment described below shows an example of the case of carrying out the present invention, and does not limit the present invention to the specific configuration described below. In carrying out the present invention, a specific configuration according to the embodiment may be appropriately employed.

[Configuration of computing device 10]
FIG. 1 is a block diagram showing the configuration of an arithmetic device 10 according to an embodiment of the invention. The arithmetic device 10 has a data flow processor 100 , a host CPU (Central Processing Unit) 200 , a system bus 300 , a ROM (Read Only Memory) 400 , a RAM (Random Access Memory) 500 , an external interface 600 and an event handler 700 .

FIG. 2 is a diagram for explaining threads processed by the arithmetic device 10 according to the embodiment of the present invention. As shown in FIG. 2, the program code has a graph structure in which data and processing are separated. This graph structure holds task parallelism and graph parallelism of the program. The arithmetic unit 10 processes a large number of threads obtained by dividing the graph-structured program.

For example, when the arithmetic device 10 performs convolution or pooling processing in a convolutional neural network (CNN), the data flow processor 100 is heavily loaded. The heavy load on the data flow processor 100 can be reduced by providing the arithmetic unit 10 with an activation function module 208 and an activation function differentiation module 210, which will be described later, and implementing the activation functions in hardware.

Returning to FIG. 1, the description continues. The data flow processor 100 is connected to an event handler 700 that generates interrupt processing, and supports interrupts generated by the event handler 700 . A specific configuration of the data flow processor 100 will be described later with reference to FIG.

The host CPU 200 is the central processing unit (CPU) of the computing device 10 . The host CPU 200 can exchange information with each unit of the arithmetic device 10 via the system bus 300 . The host CPU 200 implements the functions of each section of the arithmetic device 10 by executing programs stored in the ROM 400 or the RAM 500 .

The ROM 400 is a read-only non-volatile memory. The RAM 500 is a read/write volatile memory. The ROM 400 and the RAM 500 store programs for the host CPU 200 to implement the functions of the arithmetic unit 10 . The ROM 400 and RAM 500 may store the calculation results of the data flow processor 100 .

The external interface 600 is an input/output device through which the arithmetic device 10 inputs information from the outside or outputs information to the outside. The external interface 600 is, for example, a camera, an ultrasonic sensor, or the like.

[Configuration of Data Flow Processor 100]
FIG. 3 is a block diagram showing the configuration of the dataflow processor 100. As shown in FIG. The dataflow processor 100 has a command unit 102 , a memory subsystem 104 , a thread scheduler 112 and an execution core 114 .

The command unit 102 is an instruction section that instructs execution of functions of the data flow processor 100 . The command unit 102 may communicate with a config interface, which is a user-configurable configuration interface. The command unit 102 may have the function of a command buffer that temporarily stores commands such as data read instructions, write instructions, and erase instructions.

Memory subsystem 104 is a system of memory interposed between RAM 500 and execution core 114 . Memory subsystem 104 may communicate with a system bus interface or a ROM interface. Memory subsystem 104 includes arbiter 106 , L1 cache 108 and L2 cache 110 .

The arbiter 106 is placed between the PE 116 and the L1 cache 108, and is a coordinating device that adjusts according to a predetermined rule using priority or the like when a plurality of data access requests conflict. L1 cache 108 is memory that can be read and written faster than RAM 500 . The L2 cache 110 is memory that can be read and written faster than the RAM 500 below the L1 cache.

The thread scheduler 112 is a thread scheduler that allocates multiple threads to the execution core 114 according to the priority of processing.

The execution core 114 is an execution unit that executes thread processing. The execution core 114 has a plurality of processing elements (PE) 116 . The PE 116 may be a scalar operator that performs scalar operations or a vector operator that performs vector operations. In this embodiment, at least one PE 116 is a vector operator.

[Configuration of PE 116]
FIG. 4 is a block diagram showing the configuration of PE 116 according to the embodiment of the present invention. The PE 116 has an instruction decoder/controller 202 , a general-purpose register file 204 , an arithmetic unit 206 and a setting register 212 .

The instruction decoder/control unit 202 is an instruction decoder/control unit that decodes instructions and controls each unit. The instruction decoder/control unit 202 designates the registers used for the operation to the general-purpose register file 204, designates the operation thread to the setting register 212, and designates the module to be used for the operation to the operation unit 206. FIG.

The arithmetic unit 206 is an arithmetic section that generates output data based on input data. As described above, the arithmetic unit 206 uses the module specified by the instruction decoder/controller 202 to compute the value of the register specified by the instruction decoder/controller 202 . In this embodiment, arithmetic unit 206 performs vector arithmetic. The arithmetic unit 206 has an activation function module 208 and an activation function differentiation module 210 .

The activation function module 208 is an activation function circuit that applies activation functions to operations, and is configured by hardware. The activation function module 208 has multiple activation functions of different types.

Activation functions are e.g. identity function, step function, linear function, soft plus function, sigmoid function, hard sigmoid function, Hyperbolic Tangent function, soft sine function, ReLU function, Leaky ReLU function, Parametric ReLU function, Thresholded ReLU function , eLU (Exponential Linear Units) function, SeLU (Scaled Exponential Linear Unit) function, Softmax function, Gumbel-Softmax function, Swish function, and the like. In FIG. 4, the activation function module 208 has three activation functions, activation function A, activation function B, and activation function C, as an example.

The activation function differentiation module 210 is an activation function differentiation function circuit that applies differentiation of the activation function to computation, and is configured by hardware. The activation function differentiation module 210 has a plurality of activation function differentiation functions of different types. The differentiation functions of the plurality of activation functions that the activation function differentiation module 210 applies to the calculation are, for example, functions obtained by differentiating the plurality of activation functions stored in the activation function module 208 . In FIG. 4, the activation function differentiation module 210 has an activation function A differentiation function, an activation function B differentiation function, and an activation function C differentiation function.

In the present embodiment, an example in which the arithmetic unit 206 has the activation function module 208 and the activation function differentiation module 210 was given, but the arithmetic unit 206 may have only the activation function module 208 .

The setting register 212 is a register that stores data for designating the type of activation function and designating the coefficient of the activation function. As the setting registers 212, the number of setting registers 212 corresponding to the number of threads corresponding to the processor is provided. The setting register 212 has a register 214 that stores data designating the type of activation function. In this embodiment, the register value specifying activation function A is "1", the register value specifying activation function B is "2", and the register value specifying activation function C is "3". The activation function corresponding to the register values specified for a thread is called the "default activation function" of that thread.

Configuration registers 212 may include registers 216 that store data specifying coefficients of the activation function. In this embodiment, the setting register 212 designates the type and coefficient of the activation function.

The instruction decoder/control unit 202 designates the type of activation function and its coefficient with an instruction code to the arithmetic unit 206 . The designation of the type of activation function is determined by the bit that designates the type of activation function in the instruction code. When the bit specifying the type of activation function in the instruction code is "0", the corresponding activation function (that is, the default activation function) is executed according to the value of the register that specifies the type of activation function. . Here, "0" is a predetermined value different from the register values corresponding to the activation functions AC. On the other hand, when the bit specifying the type of activation function of the instruction code is "1" to "3", the arithmetic unit 206 executes one of the activation functions A to C corresponding to the specified value. .

Similarly, the designation of the coefficients of the activation function is determined by the bits of the instruction code that designate the coefficients of the activation function. When the bit specifying the type of activation function in the instruction code is "0", the corresponding activation function coefficient (i.e., the default activation function coefficient ). On the other hand, when the bit specifying the type of activation function in the instruction code is "1" or more, the instruction code specifies the coefficient of the activation function as necessary, and the arithmetic unit 206 Executes the activation function using the value obtained.

The types of activation functions and designation of coefficients described above will be described with reference to FIGS. 5 and 6. FIG. FIG. 5(a) is a diagram showing assembler code specifying a Parametric ReLU function as an activation function according to an embodiment of the present invention. FIG. 6A is a diagram showing graphs of a Parametric ReLU function and its differential function as activation functions according to the embodiment of the present invention.

Assembler code (1) 302 describes data specifying the type of activation function. In this embodiment, the assembler code (1) 302 has "1" as an argument specifying the type of activation function, and specifies the Parametric ReLU function associated with "1".

The assembler code (2) 304 describes setting data (actfunc#sel) specifying the activation function specified in the assembler code (1) 302 in the register R1 specifying the type of activation function. .

Assembler code (3) 306 specifies the coefficients of the activation function as arguments and describes applying the activation function to the operation. In this embodiment, the assembler code (3) 306 specifies the coefficient "1" of the Parametric ReLU function with the argument "1" and describes applying the Parametric ReLU function to the calculation.

The assembler code (4) 308 specifies the coefficient of the differential function of the activation function as an argument, and describes applying the differential function of the activation function to the calculation. In this embodiment, the assembler code (4) 308 specifies the coefficient of the differential function of the Parametric ReLU function with the argument "1", and describes applying the differential function of the Parametric ReLU to the calculation.

When the Parametric ReLU function applied to the calculation by the assembler code shown in FIG. 5(a) is graphed, it becomes a graph of the Parametric ReLU function with coefficient a=1 and its differential function as shown in FIG. 6(a).

Next, the assembler code for changing the activation function applied to the operation from the Parametric ReLU function and its differential function to the Hyperbolic Tangent function and its differential function will be described. FIG. 5(b) is a diagram showing an assembler code that specifies a Hyperbolic Tangent function as an activation function according to the embodiment of the present invention, and FIG. It is a figure which shows the graph of a tangent function and its derivative function.

As shown in FIG. 5B, in the assembler code (1) 402 in this embodiment, the value specifying the type of activation function is changed from "1" to "2". This change changes the designation of the activation function and its derivative function from the Parametric ReLU function associated with the value "1" to the Hyperbolic Tangent function associated with the value "2".

Assembler code (2) 304 to assembler code (4) 308 are the same assembler code as assembler code (2) 404 to assembler code (4) 408 . In this way, the default activation function applied to the operation can be changed only by designating the function type by the assembler code (1) 402 without changing other assembler codes.

Note that the Hyperbolic Tangent function is a function without coefficients and does not require specification of coefficients. Therefore, even if the assembler code (3) 406 specifies the coefficient of the function with the argument "1" and the assembler code (4) 408 specifies the coefficient of the function with the argument "1", both are ignored.

Graphing the activation function applied by the assembler code shown in FIG. 5(b) results in a graph of the Hyperbolic Tangent function without coefficients and its derivative function as shown in FIG. 6(b).

In this way, the neural network arithmetic device 10 includes an arithmetic unit 206 having an activation function circuit, and a setting register 212 that stores designation of the activation function type and coefficient in association with the thread. Activation functions can be implemented in hardware to speed up processing, and activation functions to be applied can be easily changed.

Further, since at least one activation function circuit of the arithmetic device 10 of the neural network performs vector arithmetic, a plurality of data can be processed in parallel.

In addition, since the arithmetic unit 10 of the neural network has a plurality of types of activation function circuits and their differential function circuits as the arithmetic unit 206, it is possible to switch and use a plurality of activation functions and their differentials with one instruction. .

In addition, since the type of activation function and coefficients are specified by the value of the setting register 212 or the argument of the instruction code, it is possible to deal with different activation functions and their coefficients simply by changing the default value of the setting register 212. Therefore, rewriting of the ROM code to change the default activation function can be eliminated or reduced.

In addition, since the specification of the type of activation function and its coefficients can be specified by the instruction code over the specification by the setting register 212, a thread that does not want to receive the specification of the activation function by the setting register 212 can use the instruction code. An activation function and its coefficients can be specified. Even if the specification by the instruction code can be given priority over the specification by the setting register 212, if a predetermined value is specified as an argument of the instruction code, the activation function by the setting register 212 and its coefficient are specification can be adopted.

[Modification]
Although the data flow processor 100 is used in the embodiment of the present invention, a conventional processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) may be used in place of the data flow processor 100. good.

10... Arithmetic device, 100... Data flow type processor,
200... Host CPU, 300... System bus, 400... ROM,
500...RAM, 600...external interface,
700 event handler, 102 command unit,
104... memory subsystem, 106... arbiter,
108 L1 cache, 110 L2 cache,
112... thread scheduler, 114... execution core, 116... PE,
202... Instruction decoder/control unit, 204... General-purpose register file,
206 operation unit, 208 activation function module,
210 ... differentiation module of activation function, 212 ... setting register,
214... register for designating the type of activation function,
216 ... register for specifying the coefficient of the activation function

Claims (7)

A calculation unit (206) having an activation function circuit (208) for performing calculations using the activation functions for a plurality of types of activation functions;
a register (214) for storing designation of the type of activation function in association with a thread;
with
When the type of the activation function is specified in the instruction code of the thread sent to the operation unit, the operation unit is activated by the instruction code specified in the instruction code regardless of the specification in the register. performing an operation using the activation function;
A computing unit (116) of a neural network. When a predetermined value is specified as the specification of the type of the activation function in the instruction code of the thread sent to the arithmetic unit, the arithmetic unit associates the thread with the register. 2. A computation device for a neural network according to claim 1, wherein computation is performed using the activation function designated by . 3. The neural network operation device according to claim 1 , further comprising a register (216) for storing the designation of the coefficient of the activation function in association with the thread. When a predetermined value is specified as a coefficient of the activation function in the instruction code of the thread sent to the arithmetic unit, the arithmetic unit associates the thread with the register. 4. The neural network operation device according to claim 3 , wherein the operation is performed using the coefficients of the activation function designated by . A calculation unit (206) having an activation function circuit (208) for performing calculations using the activation functions for a plurality of types of activation functions;
a register (214) for storing designation of the type of activation function in association with a thread;
a register (216) for storing designation of coefficients of the activation function in association with threads;
with
When the coefficient of the activation function is specified in the instruction code of the thread sent to the operation unit, the operation unit receives the coefficient specified in the instruction code regardless of the specification in the register. A computation device for a neural network , which performs computation using the coefficients of the activation function. 6. The neural network operation device according to claim 1, wherein at least one of the plurality of types of activation function circuits performs vector operations on a plurality of data in parallel. The arithmetic unit further comprises a differential function circuit (210) for a plurality of types of activation functions for computing respective differential functions of the plurality of types of activation functions,
7. The neural network computing device according to claim 1, wherein computation is performed using said activation function and a differential function of said activation function.

Images