JP7253468B2 - Neural network processor, neural network processing method, and program - Google Patents

Description

The present invention relates to neural network technology.

In recent years, various techniques using CNN (Convolutional Neural Network), which is one of neural network techniques, have been developed (see, for example, Patent Document 1). Among CNNs, a technique using a DCNN (Deep Convolutional Neural Network) with many intermediate layers is attracting particular attention because it has achieved results in various fields.

JP 2015-197702 A

DCNN achieves high recognition performance in various tasks such as general object recognition and semantic segmentation. On the other hand, the DCNN requires an extremely large number of parameters to execute processing, and therefore the hardware scale becomes large when the DCNN is implemented by hardware. In particular, the scale of hardware for implementing a memory for storing and holding parameters (e.g., data of weighting coefficients) increases, making it difficult to reduce the cost of implementing a DCNN in hardware.

Therefore, in view of the above problems, it is an object of the present invention to realize a neural network processor, a neural network data processing method, and a program that execute high-performance neural network processing while suppressing an increase in hardware scale. and

In order to solve the above problems, a first invention is a neural network processor for executing neural network processing including feature extraction layer processing and determination layer processing, comprising: a read-only memory; An access memory, a control unit, and an inner product processing unit are provided.

The read-only memory stores data for processing of the feature extraction layer.

The random access memory can read and write data, and stores and holds data for the processing of the decision layer.

The controller controls the read-only memory and the random access memory.

The inner product processing unit reads data for processing the feature extraction layer from the read-only memory as the first data, executes processing of the feature extraction layer using the first data, and reads data for processing the determination layer from the random access memory. is read out as the second data, and the second data is used to execute the processing of the decision layer.

In this neural network processor, the processing data (e.g., weighting coefficient data) of the feature extraction layer, which is determined once the neural network model is determined, is implemented as hardware in a read-only memory (ROM) that requires a small hardware scale. , and data for processing of the decision layer (for example, weighting coefficient data) whose data may be changed is held in a rewritable random access memory (RAM). Then, in this neural network processor, processing by the neural network is executed according to the above state. In other words, in this neural network processor, processing data (for example, weighting coefficient data) of the feature extraction layer, which is not changed, is held in a read-only memory that requires a small hardware scale, and it is possible to change it. Processing data (e.g., weight coefficient data) for a certain decision layer is held in random access memory and neural network processing is executed, so high-performance neural network processing can be executed while suppressing an increase in hardware scale. be able to.

A second invention is the first invention, wherein the read-only memory executes the processing of the feature extraction layer and header data including data for specifying the read time of the data for the processing of the feature extraction layer. The data for processing the feature extraction layer is stored and held so that the data for processing the feature extraction layer can be output in order of necessity.

Then, when the control unit outputs a read command signal to the read-only memory, the read-only memory stores the data for the processing of the feature extraction layer in the order required when executing the processing of the feature extraction layer. Output.

In this neural network processor, the weighting coefficient data determined according to the network configuration of the feature extraction layer are read out in the order required when executing the prediction processing (in the processing order during the prediction processing). stored in memory. Therefore, in this neural network processor, when executing prediction processing, only a read command signal (for example, a single-shot trigger signal) is input to the read-only memory, and the inner product processing unit does not need to execute prediction processing. Weighting factor data for processing the feature extraction layer can be obtained in the order of In other words, in this neural network processor, simply by inputting a readout command signal (for example, a single trigger signal) into the read-only memory, the necessary weighting coefficient data for processing the feature extraction layers can be obtained in the order of processing. , there is no need to perform complex addressing for the ROM as in the prior art.

A third invention is the second invention, wherein the control unit reads the header data stored in the read-only memory, and based on the header data, reads the header data for the processing of the feature extraction layer from the read-only memory. The time at which the data output processing is completed is specified, and control is performed so that the data for the processing of the decision layer is stored and retained in the random access memory at a time before the specified time.

In this neural network processor, the header data stored in the read-only memory can specify the time to read the data (for example, weighting coefficient data) for the processing of the feature extraction layer. Therefore, in this neural network processor, the data for processing of the determination layer (weighting factor data) is read out based on the specified time for reading out the data for processing of the feature extraction layer (for example, weighting factor data). can be previously transferred from the ROM to the random access memory. As a result, in this neural network processor, as soon as processing of the feature extraction layer is completed, data for processing of the judgment layer (for example, weight coefficient data) can be acquired (read out) from the random access memory. can be That is, in this neural network processor, as soon as the processing of the feature extraction layer is completed, the processing of the determination layer can be executed. As a result, this neural network processor can speed up the processing for the neural network (the time until the processing of the decision layer is completed can be shortened).

A fourth invention is the invention according to any one of the first to third inventions, further comprising a decompression unit that decompresses the compressed data.

The read-only memory stores data for processing of the feature extraction layer as compressed data.

The decompression unit decompresses the compressed data stored in the read memory.

The inner product processing unit uses the data decompressed by the decompression unit to process the feature extraction layer.

In this neural network processor, the data of the feature extraction layer is stored in the read-only memory as compressed data, so the memory capacity required for the read-only memory is further reduced. Therefore, in this neural network processor, the hardware scale of the read-only memory can be further reduced. As a result, the hardware scale of this neural network processor can be further reduced.

A fifth invention is a neural network processor for executing neural network processing including feature extraction layer processing and determination layer processing,
a read-only memory that stores and retains data for processing of the feature extraction layer;
a random access memory capable of reading and writing data and storing and holding data for processing of the decision layer;
a control unit that controls the read-only memory and the random access memory;
is a neural network processing method executed using a neural network processor comprising A neural network processing method includes a first step and a second step.

A first step reads data for processing the feature extraction layer from the read-only memory as the first data, and executes the processing of the feature extraction layer using the first data.

A second step reads out data for the processing of the decision layer from the random access memory as the second data, and executes the processing of the decision layer using the second data.

As a result, it is possible to realize a neural network processing method that has the same effect as the first invention.

A sixth invention is a program for causing a computer to execute the neural network processing method of the fifth invention.

Thus, it is possible to realize a program for causing a computer to execute the neural network processing method that has the same effect as the first invention.

According to the present invention, it is possible to realize a neural network processor, a neural network data processing method, and a program that perform high-performance neural network processing while suppressing an increase in hardware scale.

1 is a schematic configuration diagram of a neural network processor 100 according to a first embodiment; FIG. 4 is a timing chart of processing executed by the neural network processor 100; 4 is a flowchart of processing executed by the neural network processor 100; FIG. 2 is a schematic configuration diagram of a neural network processor 100A according to a first modified example of the first embodiment; FIG. 4 is a diagram showing a CPU bus configuration;

[First embodiment]
A first embodiment will be described below with reference to the drawings.

<1.1: Configuration of Neural Network Processor>
FIG. 1 is a schematic configuration diagram of a neural network processor 100 according to the first embodiment.

As shown in FIG. 1, the neural network processor 100 includes a first interface unit IF1, a control unit 1, a first ROM (Read Only Memory) 2, a second ROM 3, a quantization processing unit 4, and a first RAM ( Random Access Memory) 5, a second RAM 6, an inner product processing section 7, and a bus B1. As shown in FIG. 1, each functional unit of the neural network processor 100 is connected by a bus B1, and necessary data, commands, etc. can be input/output via the bus B1. Some or all of the above functional units may be connected directly, if necessary, instead of being connected via a bus.

The first interface unit IF1 inputs data Din to be processed from the outside, and outputs data including the result of processing by the neural network processor 100 to the outside as data Dout. The first interface unit IF1 outputs (writes) the input data Din to the first RAM 5 via the bus B1, and transfers data including the processing result of the neural network processor 100 from the first RAM 5 via the bus B1. Get (read).

The control unit 1 performs overall control of the neural network processor 100, control of each functional unit, and processing necessary for neural network processing. The control unit CPU1 is realized by a CPU (Central Processing Unit) or a CPU core.

The first ROM 2 is a read-only memory, and stores and holds data (for example, parameter data, neural network weighting coefficient data) used for the processing of the feature extraction layer. The first ROM 2 is connected to the bus B1 and also to the inner product processing section 7, as shown in FIG. The first ROM 2 may be connected to the inner product processing section 7 via the bus B1. When the first ROM 2 receives the trigger signal Sig_trg from the control unit 1, it outputs, for example, header data to the control unit 1 via the bus B1, and the header data and the data used for the processing of the feature extraction layer are inner products. It is continuously output to the processing unit 7 .

A second ROM 3 is a read-only memory and is connected to the bus B1. The second ROM 3 stores and holds data (for example, parameter data, neural network weighting coefficient data) used for the processing of the decision layer. The second ROM 3, in accordance with a command from the control unit 1, transfers data used in the processing of the judgment layer (for example, parameter data, neural network weighting coefficient data) to the second RAM 6 via the bus B1.

The quantization processing unit 4 is connected so as to be able to transmit and receive data to and from the first RAM. The quantization processing unit 4 performs quantization processing on feature map data that is input to a feature extraction layer (for example, a convolution layer of a DCNN (Deep Convolution Neural Network)). Also, the quantization processing unit 4 performs quantization processing on input data of a decision layer (for example, a fully connected layer of DCNN). The quantization processing unit 4 reads data to be quantized from a predetermined area of the first RAM 5 and writes data after quantization to a predetermined area of the first RAM 5 . The quantization processing unit 4 may be directly connected to the first RAM 5 as shown in FIG. 1, or may be connected via a bus (for example, bus B1). .

The first RAM 5 is a RAM (Random Access Memory) for storing and holding data necessary for executing neural network processing. The first RAM 5 is connected to the bus B1 and also to the quantization processing section 4, as shown in FIG.

The second RAM 6 is a random access memory. The second RAM 6 is connected to the bus B1 and also to the inner product processing section 7, as shown in FIG. The second RAM 6 stores and holds data transferred from the second ROM 3 and used for the processing of the decision layer (for example, parameter data, neural network weighting coefficient data). The second RAM 6 outputs to the inner product processing unit 7 the stored data used for the processing of the judgment layer at a predetermined timing according to the command from the control unit 1 .

The inner product processing unit 7 includes a second interface unit 71 and an inner product calculation processing unit 72, as shown in FIG. The inner product processing unit 7 is connected to a first ROM 2, a first RAM 5, a second RAM 6, and a bus B1, as shown in FIG. The inner product processing unit 7 receives a control signal Ctr1 from the control unit 1 via, for example, the bus B1, and executes predetermined processing according to the received control signal Ctr1.

The second interface section 71 is an interface with the first ROM 2 , the first RAM 5 and the second RAM 6 .
(1) When the processing of the feature extraction layer is executed, the second interface unit 71 reads the data (for example, parameter, weight coefficient) D_wij_feature for the processing of the feature extraction layer from the first ROM 2, and stores the read data as It is output to the inner product calculation processing unit 72 as data D_wij. The second interface unit 71 also reads the quantized data D_Qin from a predetermined area of the first RAM 5 and outputs the read data to the inner product calculation processing unit 72 as data D_Qin.
(2) When the process of the judgment layer is executed, the second interface unit 71 reads the data (for example, parameter, weight coefficient) D_wij_deci for the process of the judgment layer from the second RAM 6, and converts the read data to the data D_wij , and is output to the inner product calculation processing unit 72 . The second interface unit 71 also reads the quantized data D_Qin from a predetermined area of the first RAM 5 and outputs the read data to the inner product calculation processing unit 72 as data D_Qin.

The inner product calculation processing unit 72 receives data D_Qin and data D_wij output from the second interface unit 71 . The inner product calculation processing unit 72 executes inner product calculation processing using data D_Qin and data D_wij, and obtains data of the inner product calculation processing result as data Do. Then, the inner product calculation processing unit 72 outputs the acquired data Do to the first RAM 5 .

<1.2: Operation of Neural Network Processor>
The operation of the neural network processor 100 configured as described above will be described below.

FIG. 2 is a timing chart of processing executed by the neural network processor 100. As shown in FIG.

FIG. 3 is a flow chart of processing executed by the neural network processor 100 .

The operation of the neural network processor 100 will be described below with reference to FIGS. 1 to 3. FIG.

In general, a CNN includes an input layer, a convolution layer, and a fully connected layer. For example, image data is input as input data Din to the first interface unit IF1 of the neural network processor 100, image recognition processing is performed by CNN, and the image recognition processing result is output to the outside as output data Dout.

In CNN, in the processing of the convolutional layer or the processing of the fully connected layer, weighting operation processing is performed on the input data, and an activation function (for example, a ramp function (ReLU: Rectified Linear Unit) is applied to the processing result. , sigmoid function, Softmax function, etc.) to obtain the output of the convolutional layer or the fully connected layer.

For convenience of explanation, a case (one example) where CNN processing is executed in the neural network processor 100 will be explained below. The following description assumes that the convolutional layers of the CNN consist of N layers (N: natural number).

(Step S1):
In step S1, the neural network processor 100 executes input processing of data to be processed (for example, image data to be processed when performing image recognition processing). Specifically, in the neural network processor 100, data Din to be processed is acquired (input) from the outside by the first interface unit IF1. The data Din is transferred from the first interface section IF1 to the first RAM 5 via the bus B1, and the data Din is stored in a predetermined area of the first RAM5.

Further, the neural network processor 100 executes quantization processing. Specifically, the quantization processing unit 4 reads the data Din from a predetermined area of the first RAM 5, performs quantization processing on the read data, and acquires the quantized data as data D_Qin. . Then, the quantization processing unit 4 writes the acquired quantized data D_Qin in a predetermined area of the first RAM 5 .

Note that the quantization processing may include, for example, dynamic range adjustment, binarization processing, and offset adjustment processing in addition to data quantization processing.

(Step S2, Step S3):
In step S2, the control section 1 transmits a trigger signal Sig_trig to the first ROM 2 via the bus B1.

Upon receiving the trigger signal Sig_trig from the control unit 1, the first ROM 2 reads the header data D_head stored in the first ROM 2, and sends the header data D_head to the control unit 1 and the inner product processing unit 7 via the bus B1. is output to the second interface unit 71 (step S3).

The header data D_head includes, for example, data such as the number of layers constituting the feature extraction layer, the data length of each layer, the number of data, and the like.

(Steps S41 to S45):
In steps S41 to S45, the process for each layer of the feature extraction layer is repeatedly executed for the number of feature extraction layers.

After outputting the header data D_head, the first ROM 2 sequentially reads the data of the feature extraction layers from the data of the first layer to the data of the Nth layer of the feature extraction layers, and uses the read data as data D_wij_feature. Output to the processing unit 7 (weighting factor data acquisition process, step S42).

Specifically, (1) weighting coefficient data of the first layer of the feature extraction layer, (2) weighting coefficient data of the second layer of the feature extraction layer, . The weighting factor data are read from the first ROM 2 in the above order and output to the inner product processing section 7 as data D_wij_feature.

In step S43, an inner product calculation process is executed. Specifically, the inner product calculation process is executed as follows.

The second interface unit 71 of the inner product processing unit 7 acquires the weight coefficient data of the first layer of the feature extraction layer output from the first ROM 2 as data D_wij_feature, and uses the data as data D_wij to perform the inner product calculation processing unit 72. output to

Further, the second interface unit 71 of the inner product processing unit 7 reads the data after quantization of the data input to the first layer of the feature extraction layer from the first RAM 5 and acquires it as data D_Qin. The second interface unit 71 then outputs the acquired data D_Qin to the inner product calculation processing unit 72 .

なお、内積演算処理部７２は、行列演算処理を行う場合、上記内積演算処理を繰り返し実行することで、行列演算処理結果を取得する。例えば、内積演算処理部７２は、２つの行列Ａ、Ｂの積（行列の積）を求める演算を行う場合、行列Ａのｉ行目の１行に含まれる要素からなる行ベクトルと、行列Ｂのｊ列目の１列に含まれる要素からなる列ベクトルとに対して、上記と同様に内積演算処理を実行することで、行列Ａ、Ｂの積により取得される行列のｉ行ｊ列目の要素の値（データ）を取得することができる。 The inner product calculation processing unit 72 uses data D_Qin and data D_wij to perform inner product calculation processing. For example, when data D_Qin is n-dimensional vector data (n: natural number) and data D_wij (weighting factor data) is n-dimensional vector data, expressed as follows:
D_Qin=[ _{x1x2x3 ... xn ]}
D_wij=[w ₁ w ₂ w ₃ . . . _wn ]
The inner product calculation processing unit 72 acquires the inner product calculation result by executing the process of calculating the inner product of two n-dimensional vectors, that is, the process corresponding to the following formula.

When performing matrix calculation processing, the inner product calculation processing unit 72 acquires a matrix calculation processing result by repeatedly executing the inner product calculation processing. For example, when the inner product operation processing unit 72 performs an operation to obtain the product of two matrices A and B (matrix product), the row vector consisting of the elements contained in the i-th row of the matrix A and the matrix B By performing inner product operation processing in the same manner as above on a column vector consisting of elements contained in the first column of the j-th column of the matrix A and B, the i-th row j-th column of the matrix obtained by the product of You can get the value (data) of the element of .

By processing in this manner, the inner product calculation processing unit 72 can acquire the processing result of any matrix calculation.

In the inner product operation processing unit 72, as described above, the processing of the first layer of the feature extraction layer (for example, the CNN convolution processing (weighted addition processing for data)) is executed, and the first layer of the feature extraction layer is Acquire the data corresponding to the output of each synapse.

It should be noted that weighting coefficient data matching the order of the processing of the first layer of the feature extraction layer is input as data D_wij to the inner product calculation processing unit 72 in that order. Therefore, in the inner product operation processing unit 72, the data D_wij and the corresponding quantized data (data to be processed (data read from the first RAM 5)) are input to the inner product operation processing unit 72. By repeatedly executing the inner product calculation process in order, data corresponding to the output of each synapse in the first layer of the feature extraction layer can be obtained.

In the inner product calculation processing unit 72, after acquiring data corresponding to the output of each synapse in the first layer of the feature extraction layer (this data is referred to as "data D0") through the above process, the acquired data is: Execute the processing by the activation function and acquire the data Do. That is, the inner product calculation processing unit 72
Do=f_act(D0)
f_act(): activation function (e.g. ReLU function, Softmax function, sigmoid function, etc.)
Data Do is acquired by executing the process corresponding to .

In step S<b>44 , the inner product calculation processing unit 72 outputs the data Do acquired as described above to the first RAM 5 and writes the data Do in a predetermined area of the first RAM 5 .

As described above, the neural network processor 100 executes the processing of the first layer of the feature extraction layer. In the timing chart of FIG. 2, between time t11 and time t12, data (weighting coefficient data) D_wij_feature (weighting coefficient data) for processing the first layer of the feature extraction layer is transferred from the first ROM 2 (in FIG. 2, the first layer of the feature extraction layer data for the processing of Df_1st_L) are read out in the order of processing, and the inner product calculation processing is executed using the read data (weighting coefficient data) D_wij_feature and the quantized data D_Qin to be processed. be done. Then, in the neural network processor 100, as described above, the processing corresponding to the activation function is executed to obtain the data Do.

Then, in the neural network processor 100, the processing of the second layer, the third layer, . S45, processing between times t12 and t2 in FIG. 2).

Then, the data Do acquired by the above process (data acquired by the process of the feature extraction layer) is output (written) to the first RAM 5 .

(Step S51):
In step S51, a process of specifying the data reading end time of the feature extraction layer is executed. Specifically, the control unit 1 reads the header data D_header from the first ROM 2 (time t1 to t11 in FIG. 2), and from the data included in the header data D_header, transfers the feature extraction layer from the first ROM 2 to the inner product processing unit 7. , the time T1 required to transfer the data (weighting coefficient data) for the processing of .

Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｔｉｍｅ
＝Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｎｕｍ×Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ａｃｃｕｒａｃｙ
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｔｉｍｅ
＝Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｎｕｍ×Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ａｃｃｕｒａｃｙ
Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｔｉｍｅ：第ｋ層の実数演算に必要な処理時間
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｔｉｍｅ：第ｋ層の整数演算に必要な処理時間
Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ｎｕｍ：第ｋ層の実数演算対象のデータ数
Ｌ（ｋ）．Ｒｅａｌ＿Ｄ＿ａｃｃｕｒａｃｙ：第ｋ層の実数演算の計算精度
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ｎｕｍ：第ｋ層の整数演算対象のデータ数
Ｌ（ｋ）．Ｉｎｔ＿Ｄ＿ａｃｃｕｒａｃｙ：第ｋ層の整数演算の計算精度
そして、制御部１は、上記により取得した時間Ｔ１と、特徴抽出層の処理用のデータ（重み係数データ）の転送が開始される時刻ｔ１１とから、特徴抽出層の処理用のデータ（重み係数データ）の転送が完了する時刻ｔ２を、
ｔ２＝ｔ１１＋Ｔ１
により算出する。これにより、制御部１は、特徴抽出層の処理用のデータ（重み係数データ）の転送が完了する時刻ｔ２を特定する。 For example, in the header data D_header acquired from the first ROM 2,
(1) the number of layers N (N: natural number) included in the feature extraction layer;
(2) The time required for real number calculation in the processing of the k-th layer (k: natural number, 1 ≤ k ≤ N) included in the feature extraction layer, and the k-th layer (k: natural number, 1 ≤ N) included in the feature extraction layer k ≤ N) calculation accuracy of real number arithmetic in the process,
(3) The time required for integer arithmetic in the processing of the k-th layer (k: natural number, 1 ≤ k ≤ N) included in the feature extraction layer, and the k-th layer (k: natural number, 1 ≤ N) included in the feature extraction layer k ≤ N) calculation accuracy of integer arithmetic in the process,
is included. Based on the above data, the control unit 1, for example, executes a process corresponding to the following formula to determine the time T1 required to transfer the data for processing the feature extraction layer (weighting coefficient data). calculate.

L(k). Real_D_time
=L(k). Real_D_num*L(k). Real_D_accuracy
L(k). Int_D_time
=L(k). Int_D_num*L(k). Int_D_accuracy
L(k). Real_D_time: Processing time required for real number calculation in the k-th layer L(k). Int_D_time: Processing time required for k-th layer integer arithmetic L(k). Real_D_num: Number of real number operation target data in the k-th layer L(k). Real_D_accuracy: Calculation accuracy of real number operation in the k-th layer L(k). Int_D_num: number of data for integer operation in the k-th layer L(k). Int_D_accuracy: Calculation accuracy of k-th layer integer arithmetic Then, from the time T1 obtained above and the time t11 at which the transfer of the data for the processing of the feature extraction layer (weighting coefficient data) is started, the control unit 1 determines: The time t2 at which the transfer of data (weighting coefficient data) for processing of the feature extraction layer is completed is
t2=t11+T1
Calculated by Thereby, the control unit 1 specifies the time t2 at which the transfer of the data (weighting coefficient data) for the processing of the feature extraction layer is completed.

(Step S52):
In step S52, a process of transferring the weighting factor data of the determination layer is executed. Specifically, the control unit 1 acquires the time T2 required to transfer the weighting factor data D_wij_deci of the judgment layer stored in the second ROM 3 to the second RAM 6 . Then, at a time before time t20 (=t2-T2), the control unit 1 outputs a command to the second ROM 3 and the second RAM 6 to transfer the weighting coefficient data D_wij_deci of the judgment layer stored and held in the second ROM 3 to the second RAM 6. Then, at a time before time t20 (=t2-T2), data transfer processing of the weighting coefficient data D_wij_deci of the decision layer is started.

As a result, the process of transferring the weighting coefficient data D_wij_deci of the determination layer from the second ROM 3 to the second RAM 6 can be completed by the time t2 when the data reading process of the feature extraction layer is completed.

In the neural network processor 100, by performing the processing of steps S51 and S52 in parallel with the processing of the feature extraction layer (steps S41 to S45), immediately after the time t2 when the processing of the feature extraction layer is completed, , the processing of the decision layer can be performed.

(Steps S6, S7):
In steps S6 and S7, processing of the judgment layer is executed (processing at times t2 to t3 in FIG. 2).

By the time t2 (the start time of the processing of the judgment layer), the processing of the feature extraction layer is completed, and the data for the processing of the judgment layer (weight coefficient data for the processing of the judgment layer) is transferred from the second ROM 3 to the second ROM 3 . 2 The data transfer process to the RAM 6 has been completed. Therefore, in step S<b>6 , the second interface section 71 of the inner product processing section 7 acquires (reads) the determination layer processing data (weight coefficient data for determination layer processing) D_wij_deci from the second RAM 6 . Then, the second interface unit 71 outputs the acquired data D_wij_deci to the inner product calculation processing unit 72 as data D_wij.

Further, the second interface unit 71 of the inner product processing unit 7 reads out the quantized data of the data input to the determination layer from the first RAM 5 and acquires it as data D_Qin. The second interface unit 71 then outputs the acquired data D_Qin to the inner product calculation processing unit 72 .

なお、内積演算処理部７２は、行列演算処理を行う場合、上記内積演算処理を繰り返し実行することで、行列演算処理結果を取得する。例えば、内積演算処理部７２は、２つの行列Ａ、Ｂの積（行列の積）を求める演算を行う場合、行列Ａのｉ行目の１行に含まれる要素からなる行ベクトルと、行列Ｂのｊ列目の１列に含まれる要素からなる列ベクトルとに対して、上記と同様に内積演算処理を実行することで、行列Ａ、Ｂの積により取得される行列のｉ行ｊ列目の要素の値（データ）を取得することができる。 The inner product calculation processing unit 72 uses data D_Qin and data D_wij to perform inner product calculation processing. For example, when data D_Qin is n-dimensional vector data (n: natural number) and data D_wij (weighting factor data) is n-dimensional vector data, expressed as follows:
D_Qin=[ _{x1x2x3 ... xn ]}
D_wij=[w ₁ w ₂ w ₃ . . . _wn ]
The inner product calculation processing unit 72 acquires the inner product calculation result by executing the process of calculating the inner product of two n-dimensional vectors, that is, the process corresponding to the following formula.

When performing matrix calculation processing, the inner product calculation processing unit 72 acquires a matrix calculation processing result by repeatedly executing the inner product calculation processing. For example, when the inner product operation processing unit 72 performs an operation to obtain the product of two matrices A and B (matrix product), the row vector consisting of the elements contained in the i-th row of the matrix A and the matrix B By performing inner product operation processing in the same manner as above on a column vector consisting of elements contained in the first column of the j-th column of the matrix A and B, the i-th row j-th column of the matrix obtained by the product of You can get the value (data) of the element of .

By processing in this manner, the inner product calculation processing unit 72 can acquire the processing result of any matrix calculation.

In the inner product operation processing unit 72, as described above, the determination layer processing (for example, out mapping processing for adjusting the number of dimensions of data (Out Mapping), fully connected layer processing (corresponding to vector inner product processing), Softmax layer processing, etc.) is executed, and data corresponding to the output result of the judgment layer processing is acquired as data Do. Then, the acquired data Do is output from the inner product calculation processing unit 72 to the first RAM 5 and written in a predetermined area of the first RAM 5 .

It should be noted that in the processing of the judgment layer as well, as in the processing of the feature extraction layer, processing corresponding to the activation function may be executed as necessary.

The data of the processing result of the judgment layer acquired by the above processing is read from the first RAM 5 by the first interface unit IF1, and is transmitted to the outside as the output data Dout, for example.

As described above, in the neural network processor 100, the processing data (weighting coefficient data) of the feature extraction layer, which is determined when the neural network model is determined, is implemented as hardware in a small ROM (this ROM). In the case of the embodiment, the processing data (weighting coefficient data) of the judgment layer, which is written in the first ROM 2) and may be changed, is stored in the rewritable RAM (the second RAM 6 in the case of the present embodiment). do. Then, in the neural network processor 100, processing by the neural network is executed according to the above state. In other words, in the neural network processor 100, the processing data (weighting coefficient data) of the feature extraction layer that is not changed is held in a ROM that requires a small hardware scale, and the decision layer that may be changed is stored. Since processing data (weight coefficient data) is held in RAM and neural network processing is executed, high-performance neural network processing can be executed while suppressing an increase in hardware scale.

Further, in the neural network processor 100, the weighting factor data determined according to the network configuration of the feature extraction layer are read out in the order required when executing the prediction process (in the order of the process during the prediction process). It is stored in a ROM (first ROM 2 in this embodiment). Therefore, in the neural network processor 100, when the prediction process is executed, the inner product processing unit 7 can predict Weighting factor data for feature extraction layer processing can be obtained in the order required when performing the processing. In other words, in the neural network processor 100, only by inputting a single trigger signal into the first ROM, the weighting coefficient data for processing the necessary feature extraction layers can be obtained in the order of processing. There is no need to perform complex addressing for In the above embodiment, a single trigger signal is used to continuously transfer the data necessary for the processing of the feature extraction layer from the first ROM 2 to the inner product processing unit 7 in the order of processing. However, it is not limited to this. For example, in the neural network processor 100, by designating one address of the first ROM 2, the data necessary for the processing of the feature extraction layer are continuously transferred from the first ROM 2 to the inner product processing unit in the order of processing. 7 may be transferred.

Further, in the neural network processor 100, as described above, the header data stored in the first ROM 2 can specify the time to read the data (weighting coefficient data) for the processing of the feature extraction layer. Therefore, in the neural network processor 100, the data (weighting factor data) for processing the determination layer is transferred from the second ROM 3 to the second RAM 6 based on the specified time for reading the data (weighting factor data) for the process of the feature extraction layer. can be forwarded in advance to As a result, in the neural network processor 100, immediately after the processing of the feature extraction layer is completed, the data (weight coefficient data) for the processing of the determination layer can be obtained (read out) from the second RAM 6. can be done. In other words, in the neural network processor 100, when the processing of the feature extraction layer is completed, the processing of the determination layer can be immediately executed. As a result, the neural network processor 100 can speed up the processing for the neural network (the time until the processing of the judgment layer is completed can be shortened).

In the above description, the case where the data (weight coefficient data) for the processing of the judgment layer is stored and held in the second ROM 3 has been described, but the data for the processing of the judgment layer (weighting coefficient data) is not limited to this. The weighting factor data) may be input to the neural network processor 100 from the outside. In this case, the neural network processor 100 performs processing as follows. That is, data (weighting coefficient data) for the processing of the decision layer is input from the outside to the first interface unit IF1 as data Din. Then, in the neural network processor 100, as described above, based on the time to read out the data (weighting coefficient data) for processing the specified feature extraction layer, the data for processing the determination layer (weighting coefficient data) data) is transferred from the first interface unit IF1 to the second RAM 6 in advance. As a result, in the neural network processor 100, even when data (weighting coefficient data) for processing the decision layer is input from the outside, the neural network processor 100 immediately after the processing of the feature extraction layer is completed. , the data (weighting factor data) for the processing of the judgment layer can be obtained (read out) from the second RAM 6 .

In other words, in the neural network processor 100, even when data (weighting coefficient data) for processing the judgment layer is input from the outside, the processing of the judgment layer can be executed immediately after the processing of the feature extraction layer is completed. state. As a result, the neural network processor 100 can speed up the processing for the neural network (the time until the processing of the judgment layer is completed can be shortened).

In addition, the memory capacity of the second RAM 6 can be reduced because the second RAM 6 only needs to store and hold data necessary for the determination process (process of the determination layer).
Further, in the neural network processor 100,
(1) Execute the first feature extraction layer processing (for example, the processing of the 1st to Nth layers),
(2) Execute the first determination layer process,
(3) Execute the second feature extraction layer processing (for example, the processing of the 1st to Nth layers),
(4) Execute the process of the second decision layer;
In this case, the data necessary for executing the first judgment layer process is stored in the second RAM 6 before the process (2), and the first judgment layer process is executed before the process (4). The data required for is stored in the second RAM 6 .

At this time, in the second RAM 6, among the data necessary for executing the second judgment layer processing (the next judgment layer processing), only the data that is different from the data necessary for executing the first judgment layer processing is updated. You should do it. Thereby, in the neural network processor 100, the memory capacity of the second RAM 6 can be reduced, and the amount of data transfer (the amount of data to be updated) of the second RAM 6 can be reduced.

<<First Modification>>
Next, the 1st modification of 1st Embodiment is demonstrated. Parts similar to those of the above-described embodiment are given the same reference numerals, and detailed description thereof is omitted.

FIG. 4 is a schematic configuration diagram of a neural network processor 100A according to a first modification of the first embodiment.

The neural network processor 100A of this modified example has a configuration in which an extension unit 8 is added to the neural network processor 100 of the first embodiment. Otherwise, the neural network processor 100A of this modification is the same as the neural network processor 100 of the first embodiment.

Compressed data is written in the first ROM 2 in the neural network processor 100A of this modification.

The decompression unit 8 reads the compressed data D1 stored in the first ROM 2, decompresses the compressed data D1, and obtains data D_wij_feature after decompression. That is, the decompression unit 8 acquires the data D_wij_feature output from the first ROM 2 in the first embodiment by executing the decompression process described above. The decompression unit 8 then outputs the decompressed data D_wij_feature to the inner product processing unit 7 .

In the neural network processor 100A, the data D_wij_feature output from the decompression unit 8 is used instead of the data D_wij_feature output from the first ROM of the first embodiment, and the processing in the inner product processing unit 7 is executed. The processing in the inner product processing unit 7 is the same as in the first embodiment.

In the neural network processor 100A of this modified example, since the data of the feature extraction layer are stored in the first ROM 2 as compressed data, the memory capacity required in the first ROM 2 is further reduced compared to the first embodiment. less. Therefore, in the neural network processor 100A of this modified example, the hardware scale of the first ROM 2 can be further reduced. As a result, the neural network processor 100A of this modification can further reduce the hardware scale compared to the neural network processor 100 of the first embodiment.

[Other embodiments]
Some or all of the functional units of the neural network processors 100 and 100A may be implemented by microcode or by predetermined hardware together with microcode.

Further, in the neural network processors 100 and 100A, as disclosed in Prior Art Document A below, a neural network based on a Binarized-DCNN (DCNN: Deep Convolution Neural Network) (hereinafter referred to as "BNN") is implemented. can be In this case, the weight coefficient data of the feature extraction layer realized by BNN may be stored in the ROM (eg, the first ROM 2) as uncompressed data or compressed data.
(Prior art document A):
Ryuji Kamiya et al., “Acceleration of identification calculation and model compression by Binarized-DCNN” IEICE Technical Report 116(366), 47-52, 2016-12-15 The Institute of Electronics, Information and Communication Engineers In addition, in neural network processors 100 and 100A, A neural network (referred to as a “multivalued neural network”) having a multivalued basis matrix that is a binary basis matrix obtained by vector decomposition of BNN may be implemented. In this case, the weight coefficient data of the feature extraction layer realized by the multivalued neural network may be stored in the ROM (eg, the first ROM 2) as uncompressed data or compressed data.

In the above embodiments (including modifications), the neural network processors 100 and 100A are configured to have the first RAM 5 and the second RAM 6, but the configuration is not limited to this. In the neural network processors 100 and 100A, for example, the first RAM 5 and the second RAM 6 may be realized by one RAM.

Various data transfers and signal transmission/reception in the above embodiments (including modifications) are not limited to the forms described above. , signal transmission/reception may be performed, or data transfer and signal transmission/reception may be performed via the bus.

In addition, in the above embodiments (including modifications), the number of ROMs and RAMs has been described as an example, but the present invention is not limited to this, and the number and arrangement of ROMs and RAMs may be other than those described above. may Also, the ROM and RAM may be provided outside the neural network processors 100 and 100A.

Each block (each functional unit) of the neural network processors 100 and 100A described in the above embodiments may be individually integrated into one chip by a semiconductor device such as LSI, or may be integrated into one chip so as to include part or all of them. may be changed. Also, each block (each functional unit) of the neural network processor 100 described in the above embodiment may be implemented by a plurality of semiconductor devices such as LSIs.

Although LSI is used here, it may also be called IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Further, the method of circuit integration is not limited to LSI, and may be implemented by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after the LSI is manufactured, or a reconfigurable processor that can reconfigure the connections and settings of the circuit cells inside the LSI may be used.

Also, part or all of the processing of each functional block in each of the above embodiments may be implemented by a program. Part or all of the processing of each functional block in each of the above embodiments is performed by a central processing unit (CPU) in a computer. A program for performing each process is stored in a storage device such as a hard disk or ROM, and is read from the ROM or RAM and executed.

Further, each process of the above embodiments may be realized by hardware, or may be realized by software (including cases where it is realized together with an OS (operating system), middleware, or a predetermined library). Furthermore, it may be realized by mixed processing of software and hardware.

For example, when the functional units of the above embodiments (including modifications) are realized by software, the hardware configuration shown in FIG. Each functional unit may be realized by software processing using a hardware configuration connected by a Bus.

Also, the execution order of the processing methods in the above embodiments is not necessarily limited to the description of the above embodiments, and the execution order can be changed without departing from the gist of the invention.

A computer program that causes a computer to execute the method described above and a computer-readable recording medium that records the program are included in the scope of the present invention. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories. .

The computer program is not limited to being recorded on the recording medium, and may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, or the like.

In addition, the word "department" may be a concept including "circuitry". A circulatory may be implemented in whole or in part by hardware, software, or a mixture of hardware and software.

The functionality of the elements disclosed herein may be effected by any general purpose processor, special purpose processor, integrated circuit, ASIC (" may be implemented using circuitry or processing circuitry including "application specific integrated circuits"), conventional circuitry and/or combinations thereof. A processor is referred to as processing circuitry or circuitry when it includes transistors and other circuitry therein. In this disclosure, a circuit arrangement, unit or means is hardware that performs or is programmed to perform the recited function. The hardware may be any hardware disclosed herein or otherwise known that is programmed or configured to perform the functions recited. When the hardware is a processor, which may be viewed as a type of circuitry, the circuitry, means or unit is a combination of hardware and software, the software used to configure the hardware and/or the processor. be.

The specific configuration of the present invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the gist of the invention.

100, 100A neural network processor 1 control unit 2 first ROM
3 Second ROM
5 1st RAM
6 Second RAM
7 inner product processing unit 8 decompression unit

Claims (6)

A neural network processor for executing neural network processing including feature extraction layer processing and determination layer processing,
a read-only memory that stores and holds data for processing of the feature extraction layer;
a random access memory capable of reading and writing data and storing and holding data for processing of the decision layer;
a control unit that controls the read-only memory and the random access memory;
Data for processing the feature extraction layer is read from the read-only memory as first data, and the processing of the feature extraction layer is executed using the first data, and the determination layer is processed from the random access memory. an inner product processing unit that reads data for as second data and executes the processing of the determination layer using the second data;
Neural network processor with The read-only memory is
Header data including data for specifying readout time of data for processing of the feature extraction layer;
storing data for processing the feature extraction layer so that the data for processing the feature extraction layer can be output in the order required when executing the processing for the feature extraction layer;
When the control unit outputs a read command signal to the read-only memory, the read-only memory stores the data for the processing of the feature extraction layer as required when executing the processing of the feature extraction layer. output in the order that
2. The neural network processor according to claim 1. The control unit
reading the header data stored in the read-only memory, and based on the header data, identifying a time at which output processing of data for processing of the feature extraction layer is completed from the read-only memory;
controlling so that the data for processing of the decision layer is stored and retained in the random access memory at a time before the specified time;
3. The neural network processor according to claim 2. further comprising a decompression unit that decompresses the compressed data;
The read-only memory is
Data for processing of the feature extraction layer is stored as compressed data,
The extension is
decompressing the compressed data stored in the read memory;
The inner product processing unit
using the data decompressed by the decompression unit to process the feature extraction layer;
4. The neural network processor according to claim 1. A neural network processor for executing neural network processing including feature extraction layer processing and determination layer processing,
a read-only memory that stores and holds data for processing of the feature extraction layer;
a random access memory capable of reading and writing data and storing and holding data for processing of the decision layer;
a control unit that controls the read-only memory and the random access memory;
A neural network processing method executed using a neural network processor comprising
a first step of reading data for processing the feature extraction layer from the read-only memory as first data, and performing processing of the feature extraction layer using the first data;
a second step of reading data for processing of the decision layer from the random access memory as second data and performing the processing of the decision layer using the second data;
A processing method for a neural network comprising: A program for causing a computer to execute the neural network processing method according to claim 5 .

Images