JP7675480B2 - Calculation device, recognition device and control device - Google Patents

Description

The present invention relates to a calculation device that performs calculations based on input data and a calculation method thereof. The present invention also relates to a recognition device that uses the calculation device to recognize input data and a control device that performs control according to the input data. In particular, the present invention relates to a technology for collecting external information using sensors such as cameras and LIDAR (Light Detection and Ranging) and detecting the type of object and its coordinates from that information.

In recent years, traffic accidents have become a social issue, and there is an increasing demand for safety when traveling by vehicle. To meet these demands, various technologies for autonomous driving and driving assistance have been proposed. Among these technologies, object recognition and behavior prediction methods using CNN (Convolutional Neural Network), a type of DNN (Deep Neural Network), are known to have high recognition performance, and their application to autonomous driving is progressing.

CNN is a neural network that takes image data, which is information about the outside world, as input and is composed of multiple convolutional layers connected in cascade. Here, a convolutional layer is a series of operations that consists of product-sum operations and activation function operations, multiplying the pixels in the input data by the corresponding weight parameters, accumulating the results a certain number of times to create output data, and then performing activation function operations to output the results.

By performing a convolutional layer operation on image data, the type of a specific object in the input image data and the coordinates of the object are output. First, the specific configuration is explained. In the recognition device described in this patent, the first layer performs a product-sum operation on the input image data and the weight parameters of the convolution operation of the first layer to output the convolution operation result. The jth convolutional layer of multiple neural networks is called the jth layer, and the jth layer (an integer that satisfies 1≦j≦L) outputs the operation result of the jth convolutional layer from the output data of the j-1th layer and the weight parameters of the convolution operation of the jth layer. If the final layer is the Lth layer, it takes the output data of the L-1th layer, which is the layer before the Lth layer, and the weight parameters of the convolution operation of the Lth layer as input, and outputs the type of object and the coordinates of the object. Each convolutional layer performs a convolution operation using the input data and weight parameters, then performs an activation function operation and outputs the result.

When implementing a computationally intensive process such as CNN, which mainly involves multiply-and-accumulate operations, in an in-vehicle ECU (Electronic Control Unit), it is necessary to achieve a throughput equivalent to the processing speed of the external information acquisition device and a latency equal to or less than the vehicle control cycle. Conventionally, when implementing CNN, it is common to use a GPU (Graphic Processing Unit) or FPGA (Field Programmable Logic Array) designed for edge devices. However, in-vehicle devices have only limited computing performance due to restrictions on the conditions of use, implementation costs, and device size that can be installed, and issues arise in which the required computing performance cannot be met. Therefore, in order to meet the required computing performance, it is common to achieve the target processing speed by improving the device itself or by using a computing unit specialized for CNN operations.

Next, we will describe the conventional technology when CNN is implemented in hardware as a computing device. External input is acquired using an external information acquisition device such as a camera or LIDAR. The acquired information is stored in memory. The computing device is composed of a memory, a model storage unit, a parameter storage unit, and multiple convolution calculation units, and outputs recognition results such as the type of object and its coordinates.

The external world information stored in the memory is sent to the convolution calculation unit. The model storage unit also stores data that has been previously trained, and saves the trained data in the parameter storage unit. The parameter storage unit selects weight parameters for each layer, the number of channels for each layer, and the number of filters for each layer from the trained data it receives, and sends them to the convolution calculation units from the 1st layer to the Lth layer. The first layer of the convolution calculation unit takes as input data from the memory, the weight parameters for the 1st layer, the number of channels for the 1st layer, and the number of filters for the 1st layer, and outputs the calculation result to the 2nd layer. The convolution calculation units are also connected in series. The jth layer, which is the jth layer, takes as input the output of the convolution calculation unit for the j-1th layer, the weight parameters for the jth layer, the number of channels for the jth layer, and the number of filters for the jth layer, and outputs the calculation result to the j+1th layer.

The convolution calculation unit performs a convolution calculation based on the input data sent from the memory and the weight parameters, number of channels, and number of filters sent from the parameter storage unit, then performs an activation function calculation and outputs the calculation result to the next layer.

However, in the case of in-vehicle computing devices, where the computing performance of the computing unit is limited, there is an issue that the computing device cannot be fully utilized due to a mismatch between the computing unit of the computing device and the number of filters and channels used for the convolution calculation, resulting in a decrease in processing speed.

In response to this, the method of Patent Document 1 has been proposed as a method and device for efficiently executing CNN calculations within limited calculation performance in CNN-related calculations. The configuration of Patent Document 1 is shown below. First, the calculation device described in Patent Document 1 is composed of a divider, a calculator, and a generator. Next, the connections in Patent Document 1 are shown.

The output from the divider is input to the calculator. The calculator outputs to the generator, and the generator accepts input from the calculator.

Next, the operation of the arithmetic device proposed in Patent Document 1 is shown. First, a divider divides the weight parameters of a selected layer of a convolutional neural network in at least one of the depth dimension and the dimension of the number of kernels. Then, an arithmetic parameter array containing a plurality of arithmetic parameters is obtained. Next, each arithmetic parameter in the arithmetic parameter array is used in the arithmetic unit to execute the arithmetic of the selected layer, thereby obtaining a partial arithmetic result array. Finally, a generator generates the output of the layer based on the partial arithmetic result array. As a result, the divider divides the weight parameters according to the arithmetic unit of the arithmetic unit, and then transmits them to the arithmetic unit. This improves the operating efficiency or utilization rate of the arithmetic unit, and avoids hardware limitations due to the size of the parameters.

JP 2019-82996 A

In Patent Document 1, the weight parameters input to the calculator in advance are divided in at least one direction, either in the depth direction of the CNN calculation or in the dimension of the number of kernels, and the divided data is transferred from the divider to the calculator to perform the convolution calculation. Then, for a calculator of a limited size based on the results of this calculation, the weight parameters divided into a size that can be calculated by the calculator are used to perform the convolution calculation.

However, in Patent Document 1, a division operation is required in advance to obtain a partial array of weight parameters using a divider outside the arithmetic unit, and then a calculation is required in a generator to recombine the partial array calculation of the divided weight parameters. Furthermore, in the convolution calculation, it is necessary to maintain the consistency of the calculation before and after the division. For this reason, there are restrictions on the method of dividing the weight parameters and the size of the weight parameters after division, which poses the problem that the arithmetic unit cannot be fully utilized.

To solve the above problem, in the present invention, filter information is deleted based on the sensitivity and calculation direction (execution order) of the convolution calculation.

More specifically, in a computing device that performs CNN computation based on input data, the computing device includes a model storage unit that stores a model used in the CNN computation, a CNN computation unit that performs the CNN computation by executing a convolution computation for each of a plurality of convolution layers using the model, a weight parameter acquisition unit that acquires weight parameters of a convolution filter used in the convolution computation from the model storage unit, a channel information acquisition unit that acquires channel information for each of the plurality of convolution layers from the model stored in the model storage unit, a filter information acquisition unit that acquires filter information for each of the plurality of convolution layers from the model storage unit, a computation allocation unit that assigns a combination of weight information and computation required for the CNN computation and transmits it to the CNN computation unit, and a deletion index determination unit that deletes a part of the filter information used in the convolution computation based on the maximum number of computations possible for the CNN computation unit, the execution order of the CNN computation, the channel information for each convolution layer of the CNN computation unit acquired from the channel information acquisition unit, the weight information acquired from the weight parameter acquisition unit, and the filter information for each of the plurality of convolution layers acquired from the filter information acquisition unit.

A typical implementation of the present invention can improve the utilization rate of the computing unit. Problems, configurations, and effects other than those described above will become clear from the explanation of the following embodiment.

FIG. 2 is a functional block diagram of a recognition device which is a computing device in the first embodiment. FIG. 11 is an explanatory diagram for explaining deletion channel index information in the first embodiment. FIG. 11 is an explanatory diagram for explaining deletion filter index information in the first embodiment. FIG. 2 is a diagram illustrating an internal configuration of a CNN calculation unit in the first embodiment. FIG. 13 is a diagram illustrating an internal configuration of a deletion index determination unit in the first embodiment. FIG. 2 is a diagram illustrating an internal configuration of a convolution calculation unit in the first embodiment. 10 is a flowchart showing a process flow of a deletion filter determination unit in the first embodiment. 11 is a flowchart showing a process flow of a deletion channel determination unit in the first embodiment. 11 is a flowchart showing a process flow of a computation allocation unit in the first embodiment. 11A and 11B are diagrams illustrating an example of a stride and a removable filter in the first embodiment. FIG. 11 is an explanatory diagram illustrating the state of a deletion filter in the first embodiment; FIG. 11 is a functional block diagram of a recognition device which is a computing device in a second embodiment. FIG. 11 is a diagram illustrating an internal configuration of a deletion index determination unit in the second embodiment. FIG. 11 is an explanatory diagram showing the storage state of a filter when a calculation is performed by a calculation unit in the second embodiment. FIG. 11 is a functional block diagram of a control device according to a third embodiment.

Figure 1 is a functional block diagram of a recognition device 1000 in the first embodiment. The recognition device 1000 is a type of computing device, and is a device that recognizes the situation of the outside world using outside world information from a camera or LIDAR.

The recognition device 1000 in FIG. 1 is connected to an external world information acquisition device 101. It is composed of a CNN calculation unit 102, a model storage unit 104, a channel information acquisition unit 105, a weight parameter acquisition unit 106, a filter information acquisition unit 107, a deletion index determination unit 108, and a calculation allocation unit 109. As a result, the recognition device 1000 outputs a recognition result 103.

Here, each component will be described. First, the external world information acquisition device 101 acquires external world information and transmits the acquired information to the CNN calculation unit 102. This external world information acquisition device 101 can be realized by a sensor such as a camera or LIDAR. Furthermore, the channel information acquisition unit 105 receives model information 110 which is output from the model storage unit 104. Furthermore, the weight parameter acquisition unit 106 receives model information 110 which is output from the model storage unit 104.

The filter information acquisition unit 107 also receives model information 110, which is an output from the model storage unit 104. The deletion index determination unit 108 also receives channel information 111 indicating the number of channels in each layer from the channel information acquisition unit 105. The deletion index determination unit 108 also receives weight parameter information 112 indicating the weight parameters of each layer, which is an output from the weight parameter acquisition unit 106. The deletion index determination unit 108 also receives filter information 113 indicating the number of filters in each layer, which is an output from the filter information acquisition unit 107.

The computation allocation unit 109 also receives model information 110, which is the output of the model storage unit 104. Furthermore, the computation allocation unit 109 receives deletion filter index information 114 for each layer, which is the output of the deletion index determination unit. Furthermore, the computation allocation unit 109 also receives deletion channel index information 116 for each layer.

The CNN calculation unit 102 also receives input data 117, which is external world information from the external world information acquisition device 101, a CNN calculation control signal 115, which is the output of the calculation allocation unit 109, channel information 111, and filter information 113. The CNN calculation unit 102 then uses these to output the recognition result 103.

Next, the operation of the recognition device 1000 and the signal flow will be described. First, external world information is transmitted from the external world information acquisition device 101 to the CNN calculation unit 102. Next, the CNN calculation unit 102 performs calculations based on the model information 110 of the model storage unit 104 and the CNN calculation control signal 115, channel information 111, and filter information 113 of the calculation allocation unit 109. Note that in FIG. 1, the arrows indicating the input channel information 111 and filter information 113 are omitted. Then, the CNN calculation unit 102 uses these to calculate and output the recognition result 103.

Next, the channel information acquisition unit 105 extracts the number of channels for each layer from the model information in the model storage unit 104, and outputs channel information 111 to the deletion index determination unit 108. In addition, the weight parameter acquisition unit 106 extracts weight parameters for each layer from the model information 110 received from the model storage unit 104, and outputs weight parameter information 112 to the deletion index determination unit 108.

The filter information acquisition unit 107 also acquires the number of filters for each layer from the model information 110 in the model storage unit 104, and outputs the filter information 113 for each layer including this to the deletion index determination unit 108.

The deletion index determination unit 108 also calculates the indexes of the filters and channels for which calculations are to be deleted from the channel information 111, weight parameter information 112, and filter information 113. The deletion index determination unit 108 then outputs deletion filter index information 114 and deletion channel index information 116, which include these indexes, to the calculation allocation unit 109.

The computation allocation unit 109 also determines the computation order and identifies filters and channels to be deleted from the model based on the deleted filter index information 114, deleted channel index information 116 for each layer, and model information 110. Here, the computation allocation unit 109 only needs to identify at least one of a filter and a channel as the target for deletion. In particular, it is desirable to select a filter as the target for deletion. Then, the computation allocation unit 109 transmits a CNN computation control signal 115 including the identified filter and channel to the CNN computation unit 102.

The deletion channel index information 116 will be described with reference to FIG. 2. The deletion filter index information 114 will be described with reference to FIG. 3. The operation of the deletion index determination unit 108 will be described in detail with reference to FIG. 5. The recognition device of this embodiment can also be realized by a so-called computer. In this case, the functions of each unit are executed by a processing device such as a CPU in accordance with a program. The program is stored in a storage medium. Each unit can also be realized by dedicated hardware such as an FPGA (Field Programmable Gate Array) or a dedicated circuit.

Figure 2 is an explanatory diagram for explaining the deletion channel index information 116. First, there are L layers of CNN in this embodiment, and each layer is connected in a cascade manner from the 1st layer to the Lth layer. Also, there are channels 251 in each layer, and the channel 251 in one layer is connected to all the channels 251 in the next layer. Here, a different index is given to the channel in each layer, and the deletion channel index information 116 specifies the channel index to be deleted for each layer. Hereinafter, this specified index number will be referred to as the deletion channel index.

Next, FIG. 3 is an explanatory diagram for explaining the deletion filter index information 114. First, in CNN, filters 351 exist in each of the 1st to Lth layers. In FIG. 3, for a certain jth layer (1≦j≦L), a 3×3 filter is shown as an example. Note that the number of filters is merely an example and is not limited to this.

In this embodiment, the same index number 352 is assigned to all filters in one layer for each layer. This deletion filter index information 114 specifies the filter index 352 to be deleted for each layer. Hereinafter, this specified index number will be referred to as the deletion filter index.

Figure 4 is a diagram showing the internal configuration of the CNN calculation unit 102. In Figure 4, the CNN calculation unit 102 has a memory 201, a parameter storage unit 202, and a convolution calculation unit 203. The CNN calculation unit 102 also handles input data 117, CNN calculation control signal 115, model information 110, CNN calculation control signal 204 for one layer, weight parameters for all layers 205, the number of filters in all layers 206, the number of channels in all layers 207, individual weight parameters 208 for one layer, that is, the weight parameters for each layer, the number of individual filters in each layer 209, and the number of individual channels in each layer 210. The CNN calculation unit 102 then uses these to output the recognition result 103.

Next, the connections of the CNN calculation unit 102 shown in FIG. 4 will be described. Input data 117 is input to the memory 201. This input data 117 is the external world information described above. In addition, the parameter storage unit 202 holds the model information 110.

The first-layer convolution operation unit 203-1 receives the following signals or information:
An individual CNN operation control signal 204-1, i.e., a first-layer CNN operation control signal, branched off from the CNN operation control signal 115
Saved data 211 (input data 117) saved in memory 201
Individual weighting parameters 208-1 branched from the weighting parameters 205 output from the parameter storage unit 202
The number of individual filters 209-1 branched from the number of filters 206 output from the parameter storage unit 202
The number of individual channels 210-1 branched from the number of channels output from the parameter storage unit 202
Thereafter, the following signals or information are input to the second-layer and subsequent convolution operation units, that is, the j-th layer convolution operation unit 203-j.
The CNN operation control signal 204-j of the jth layer is branched from the CNN operation control signal 115.
Output data 212-j of the convolution operation unit of the j-1th layer
Individual weight parameters 208-j branched from the weight parameters 205 output from the parameter storage unit 202
The number of individual filters 209-j branched from the number of filters 206 output from the parameter storage unit 202
The number of individual channels 210-j branched from the number of channels output from the parameter storage unit 202
The following signals or information are input to the convolution operation unit 203-L in the final layer, the Lth layer:
An individual CNN operation control signal 204-L branched off from the CNN operation control signal 115
Output data 212-L of the L-1th layer convolution operation unit 203-j
Individual weight parameters 208-L branched from the weight parameters 205 output from the parameter storage unit 202
The number of individual filters 209-L branched from the number of filters 206 output from the parameter storage unit 202
The number of individual channels 210-L branched from the number of channels output from the parameter storage unit 202
Then, the Lth layer convolution operation unit 203-L outputs the recognition result 103 obtained by using these.

Next, the operation of the CNN calculation unit 102 will be described in detail. First, the CNN calculation unit 102 performs a convolution calculation based on the input data 117 and the model information 110, and outputs the recognition result 103. At this time, each convolution calculation unit 203 receives a CNN calculation control signal 204 for each layer branched from the CNN calculation control signal 115, and skips the calculation of some filters and channels. This skipping of calculations is an example of the deletion of the present application. Here, the deletion of the present application also includes passive processing such as omitting or ignoring calculations. Here, the deleted filters and channels, that is, the skipped filters and channels, are the same until the model information 110 is changed due to an external model update. Note that, as described above, the target to be skipped may be either the filter or the channel. As a result, in this embodiment, in other words, in this application, at least one of the filter information and the channel information used for convolution is deleted.

The memory 201 also stores input data 117. Then, saved data 211, which is the input data 117, is output to the convolution calculation unit 203. Next, the convolution calculation unit 203 performs a convolution calculation based on the input saved data 211 (output data 212) and the individual weighting parameters 208. As a result of this convolution calculation, the number of calculations, the number of output data, and the allocation of data to the calculation units are determined by the CNN calculation control signal 204, which is a signal for controlling the calculation order and data reading, the number of individual filters 209, and the number of individual channels 210. The convolution calculation unit 203 outputs output data 212, which is the calculation result determined in this way. This is repeated from the first layer to the Lth layer.

The model information 110 received by the parameter storage unit 202 is divided into weight parameters 205 for each layer, the number of filters 206 for each layer, and the number of channels 207 for each layer, and output to the convolution calculation unit 203.

This concludes the explanation of the CNN calculation unit 102. Next, we will explain the deletion index determination unit 108. Figure 5 is a diagram showing the internal configuration of the deletion index determination unit 108.

In FIG. 5, the deletion index determination unit 108 receives channel information 111, weight parameter information 112, and filter information 113 as inputs. Then, based on these inputs, the deletion index determination unit 108 outputs deletion channel index information 116 and deletion filter index information 114. To this end, the deletion index determination unit 108 has a target processing speed storage unit 301, a maximum number of operations storage unit 302, an operation speed analysis unit 303, a sensitivity information analysis unit 304, a deletion channel determination unit 305, a deletion filter determination unit 306, a deletion channel index output unit 307, and a deletion filter index output unit.

Next, the connection relationship of the deletion index determination unit 108 is shown using FIG. 5. The calculation speed analysis unit 303 receives as input the target processing speed 310 which is the output of the target processing speed storage unit 301, the maximum number of calculations that can be calculated per calculation unit 315 which is the output of the maximum number of calculations storage unit 302, the channel information 111, and the filter information 113.

The sensitivity information analysis unit 304 receives as input the channel information 111, the weight parameter information 112, and the filter information 113. The removal channel determination unit 305 receives as input the number of channels to be removed 311, which is the result of the calculation speed analysis unit 303, and the sensitivity information 313, which is the result of the sensitivity information analysis unit 304.

The deletion filter determination unit 306 receives as input the number of deletion filters 312 output from the calculation speed analysis unit 303 and the sensitivity information 316 output from the sensitivity information analysis unit 304. The deletion channel index output unit 307 receives as input the priority information 317 output from the deletion channel determination unit 305 and the number of operations 315 output from the maximum number of operations storage unit 302, and outputs deletion channel index information 116 for each layer.

Furthermore, the deletion filter index output unit 308 receives as input the output 318 of the deletion filter determination unit 306 and the number of operations 315, which is the output of the maximum number of operations storage unit 302, and outputs the deletion filter index information 114 for each layer.

Next, the operation of the deletion index determination unit 108 will be described with reference to FIG. 5. First, the calculation speed analysis unit 303 estimates the time required for the convolution calculation for each layer and the difference between the target processing speed and the time required for the convolution calculation, based on the input channel information 111, filter information 113, and the maximum number of calculations that can be performed per calculation unit 315.

The sensitivity information analysis unit 304 also estimates sensitivity information for the recognition accuracy of all channels in each layer and sensitivity information for the recognition accuracy of all filters in each layer from the channel information 111, weight parameter information 112, and filter information 113.

The channel to be removed determination unit 305 also receives difference information between the calculation speed of each layer and the target processing speed input from the calculation speed analysis unit 303, sensitivity information 313 to the recognition accuracy for each layer output from the sensitivity information analysis unit 304, and sensitivity information 313 of the channels within the layer. Based on these, the channel to be removed determination unit 305 sets the priority of the channels to be removed in order from the channel with the lowest sensitivity to recognition accuracy so that the difference from the target processing speed becomes 0. Note that the operation of the channel to be removed determination unit 305 will be described in detail later with reference to FIG. 8.

The deletion filter determination unit 306 also prioritizes the filters to be deleted for each layer based on the difference information between the processing speed of each layer and the target processing speed from the calculation speed analysis unit 303 and the filter sensitivity information 316.

The deleted channel index output unit 307 also receives priority information 317 of the channels to be deleted for each layer determined by the deleted channel determination unit 305, and information on the maximum number of operations that the calculator can perform output from the maximum number of operations storage unit 302. Based on these, the deleted channel index output unit 307 sets an index number so that the number of remaining channels is a multiple of the maximum number of operations that the calculator can perform or a divisor other than 1, and outputs this.

Here, the deletion channel index output unit 307 performs processing according to the following (Equation 1), where Mc is the maximum number of channels that can be calculated, C is the number of channels in a certain jth layer (1≦j≦L), and Cd is the number of channels to be deleted from the jth layer. In other words, the deletion channel index output unit 307 sets Cd that satisfies (Equation 1), and sets the calculation order of the Cd channels in ascending order of priority information to the deletion channel index information 116 of the jth layer.

The deletion filter index output unit 308 sets index information based on the priority of the filters to be deleted set by the deletion filter determination unit 306 so that the number of remaining filters is a multiple of the maximum number of operations that the calculator can perform or a divisor other than 1. The deletion filter index output unit 308 then outputs this. Note that the determination of the filter priority by the deletion filter determination unit 306 will be described later with reference to FIG. 7.

Here, the deletion filter index output unit 308 performs processing according to the following (Equation 2) where Mf is the maximum number of filters that can be calculated, F is the number of filters in a certain jth layer, and Fd is the number of filters to be deleted from the jth layer. In other words, the deletion filter index output unit 308 sets Fd that satisfies (Equation 2), and sets the deletion filter index information 114 for the jth layer for the calculation order of the Fd filters in ascending order of priority information.

Next, the convolution calculation unit 203 will be described. Figure 6 is a diagram showing the internal configuration of the convolution calculation unit 203 in this embodiment.

First, the configuration of the convolution calculation unit 203 will be explained. Here, the jth layer will be explained as an example. The convolution calculation unit 203 receives as input the CNN calculation control signal 204, the number of individual channels 210, the number of individual filters 209, the output data of the previous layer 212-j, and the individual weighting parameters 208. The convolution calculation unit 203 then uses these to output the output data 212-j, which is the result of the calculation. For this purpose, it has a calculation control unit 401, an input data temporary storage unit 402, a calculation unit 403, and an output data temporary storage unit 404.

The following shows the connections within the convolution calculation unit 203. The calculation control unit 401 receives as input the CNN calculation control signal 204, the number of individual channels 210, and the number of individual filters 209. The input data temporary storage unit 402 receives as input the output data 212-j of the previous layer, the individual weighting parameters 208, and a control signal 410 output from the calculation control unit 401 that controls the output order and output timing of the stored data.

The calculation unit 403 also receives as input a calculation stop control signal 412 that controls the calculation order, the start of calculation, and the stop of calculation, as well as input data 411 required for the calculation. The output data temporary storage unit 404 also receives as input a calculation result 413 of the calculation unit 403 and a control signal 414 that controls the output timing of the stored data, and outputs output data 212-j+1, which is the calculation result.

The operation of the convolution calculation unit 203 is explained below. First, the output data 212-j of the previous layer and the individual weight parameters 208 are stored in the input data temporary storage unit 402. Then, the calculation control unit 401 outputs the data output timing and information on the channels and filters for which output is to be skipped, in accordance with the order in which the calculations are to be performed, based on the information of the CNN calculation control signal 204, the number of individual channels 210, and the number of individual filters 209. In addition, the calculation control unit 401 outputs a calculation stop control signal 412 to the calculation unit 403, which controls the number of calculations, the calculation start timing, and the end timing.

Then, the output data temporary storage unit 404 holds a control signal 414 that controls the timing of outputting the data after the calculation, allowing it to be used in other parts. The calculation unit 403 also performs a convolution calculation and activation function processing on the input data, and outputs the calculation result 413 to the output data temporary storage unit 404. The output data temporary storage unit 404 sequentially outputs the calculation result, output data 212-j+1, in accordance with the control signal 414 from the calculation control unit 401. As for the CNN calculation control signal 204, the number of individual channels 210, and the number of individual filters 209, the calculation control unit 401 continues to hold the data unless the model information 110 is updated.

Next, the operation of the deletion filter determination unit 306 will be described in detail. Figure 7 is a flowchart showing the operation of the deletion filter determination unit 306 to set the priority of the filter to be deleted.

First, the deletion filter determination unit 306 starts operation when the number of channels to be deleted 311, which is the output of the calculation speed analysis unit 303, and the sensitivity information 313, which is the output of the sensitivity information analysis unit 304, are input (step S1001).

Next, the deletion filter determination unit 306 extracts n layers (n is an integer that satisfies n>=0) with processing speeds equal to or lower than the target processing speed based on the output of the calculation speed analysis unit 303. The deletion filter determination unit 306 then sorts the layers in order of slowest processing speed (step S1002).

Next, when layers with a processing speed equal to or lower are arranged, the deletion filter determination unit 306 sets i as a parameter representing the currently targeted layer, and sets i=n (step S1003).

Next, the deletion filter determination unit 306 determines whether i=0 (step S1004). If the result is that i=0, the process proceeds to step S1006. If i is not 0, the process proceeds to step S1005.

The deletion filter determination unit 306 also sets the deletion priority for each filter size, starting from the lowest sensitivity based on the sensitivity information to recognition accuracy (step S1005). For example, in the case of a filter consisting of 3 x 3 = 9 weight parameters, the priority is set in order of the nine weight parameters that will cause the least degradation in recognition accuracy when deleted. Note that the number of filters is not limited to this example. Also, the criterion for deletion is not limited to the amount of degradation, and may be any criterion that satisfies a predetermined rule. In addition to the degradation in recognition accuracy, the amount of degradation in calculation accuracy may also be used.

The deletion filter determination unit 306 also decrements i by one and returns to step S1004 (step S1007).

Finally, the deletion filter determination unit 306 outputs the deletion priority information of the filters calculated individually for each n layer (step S1006).

This concludes the explanation of the operation of setting the priority of filters to be deleted in the deletion filter determination unit 306.

Next, the operation of the deletion channel determination unit 305 will be described. Figure 8 is a flowchart showing the operation of the deletion channel determination unit 305.

First, the deletion channel determination unit 305 starts operation when the number of deletion channels 311, which is the output of the calculation speed analysis unit 303, and the sensitivity information 313, which is the output of the sensitivity information analysis unit 304, are input (step S2001).

Next, the deletion channel determination unit 305 extracts n layers (n is an integer that satisfies n>=0) whose processing speed is below the target based on the number of deletion channels 311, which is the output of the calculation speed analysis unit, and sorts them in order of slowest processing speed (step S2002).

Then, the deletion channel determination unit 305 sets i as a parameter representing the layer currently being targeted when arranging layers below the processing speed, and sets i=n (step S2003).

Then, the deletion channel determination unit 305 determines whether i=0 (step S2004). If the result is that i=0, the process proceeds to step S2006. If i is not 0, the process proceeds to step S2005.

The channel to be deleted determination unit 305 also calculates the number of channels that need to be deleted based on the sensitivity information 313 that is the output of the sensitivity information analysis unit 304 for the channels of the i layer and the number of channels to be deleted 311 that is the output of the calculation speed analysis unit 303. It is desirable to calculate the number of channels that need to be deleted until the processing speed of the i layer exceeds the target processing speed. The channel to be deleted determination unit 305 then sets deletion priorities in order of decreasing sensitivity to recognition accuracy (step S2005).

Finally, the deletion channel determination unit 305 outputs the deletion priority information of the channels calculated individually for each n layer (step S2006).

Next, the operation of the computation allocation unit 109 will be described. Figure 9 is a flowchart showing the operation of the computation allocation unit 109. The processing flow will be described below.

First, the computation allocation unit 109 starts operation when the deletion channel index information 116 and deletion filter index information 114 are input from the deletion index determination unit 108 (step S3001). Next, the computation allocation unit 109 stores the input deletion channel index information (step S3002).

Next, the computation allocation unit 109 stores the deletion filter index information and the deletion priority (step S3002). Next, the computation allocation unit 109 determines whether the filter to be deleted is at either end as viewed from the stride direction of the deletion filter index information (step S3004). The stride direction and both ends will be described later with reference to FIG. 10. If it is determined that the filter is at either end, the process proceeds to step S3007. If it is determined that the filter is not at either end, the process proceeds to step S3005.

The operation allocation unit 109 also determines whether changing the stride direction is possible in the implementation (step S3005). For this reason, it is desirable for the operation allocation unit 109 to make a determination based on implementation constraints. If the result of this determination is that the stride direction can be changed, the process proceeds to step S3009. If the stride direction cannot be changed, the process proceeds to step S3006.

Next, the computation allocation unit 109 changes the priority of the deletion filter index to the next highest priority after both ends (step S3006).
The deletion channel index information is output (step S3007).

Then, the computation allocation unit 109 outputs the deletion filter index information and the stride direction (step S3008).

If it is determined in step S3006 that this is not possible, the computation allocation unit 109 changes the stride direction by 90 degrees (step S3009). Then, the process proceeds to step S3007.

Next, the stride and the filters that can be removed will be described. FIG. 10 is a diagram for explaining an example of the stride and the filters that can be removed. In the convolution operation, the convolution operation is performed while the convolution filter 803 and the filter 805 move in the stride directions 802 and 804 on the intermediate data 801 in FIG. 10(a) and (b). Here, the filter is 3×3, but the size of the filter is not limited to this.

The two ends of the filter refer to the filters shown in gray in the convolution filter 803 in FIG. 10(a) and the filter 805 in FIG. 10(b). Here, the stride direction 802 of the convolution filter 803 is left and right, so the filters at both ends are left and right. Also, the stride direction 804 of the filter 805 is up and down, so the filters at both ends are up and down.

Next, FIG. 11 shows an example of deleting a 3×3 filter. Here, an example of actual deletion when the maximum number of operable operations is 8 is shown. The maximum number of operations is 8 as an example here, but is not limited to this and may be 2 ⁿ (n≧1). In this way, when the maximum number of operable operations is 8, the number of filters to be deleted is one. Therefore, one filter is deleted from the gray area shown in FIG. 11. The deleted filter 913 is the filter after deletion with a horizontal stride 912. Also, the deleted filter 914 is the filter after deletion with a vertical stride 915. This concludes the explanation of the first embodiment.

Next, a second embodiment will be described. The second embodiment is an example in which the recognition device 1000 of the first embodiment is applied to recognition of the outside world while a vehicle is traveling. For this reason, in the second embodiment, the processing speed of the convolution calculation is changed according to the traveling speed and the speed required for the calculation processing, such as when traveling on a road with no speed limit, when traveling on a normal highway, or when traveling in an urban area.

The processing speed required by the ECU differs when driving on normal highways or urban areas, and when driving on roads with no speed limit. The width of the road surface and surrounding conditions also differ. Therefore, this embodiment shows an example of responding to the increase in required processing speed by observing changes in driving speed and increasing the number of calculations that perform multiple calculations in one computing unit.

In addition, in the second embodiment, driving speed information is acquired, and when the average speed V per unit time exceeds the upper driving speed limit X determined in advance at the time of design, the target processing speed is changed and the number of deletion indexes is increased. This is an embodiment for increasing the number of calculations and improving the processing speed. Note that parts common to the first embodiment are given the same reference numerals in the figures and their explanations are omitted.

Fig. 12 is a functional block diagram of the recognition device 1000, which is a calculation device in the second embodiment. Here, differences from the first embodiment will be described with reference to Fig. 12. The recognition device 1000 in the second embodiment has a driving speed acquisition unit 901 added. In addition, the present embodiment has a deletion index determination unit 903, which is configured differently from the first embodiment. Next, the connection relationship of the recognition device 1000 in the second embodiment will be described. First, the driving speed acquisition unit 901 outputs the driving speed to the deletion index determination unit 903. The deletion index determination unit 108 receives outputs from the driving speed acquisition unit 901, the channel information acquisition unit 105, the weight parameter acquisition unit 106, and the filter information acquisition unit 107.

Next, differences between the operation of the recognition device 1000 of the second embodiment and the first embodiment will be described. First, the vehicle speed acquisition unit 901 monitors the vehicle speed, and continuously outputs the current speed to the deletion index determination unit 108.

The recognition device 1000 of this embodiment can also be realized by a so-called computer. In this case, the functions of each part are executed by a processing device such as a CPU according to a program. The program is stored in a storage medium. Each part can also be realized by dedicated hardware such as an FPGA (Field Programmable Gate Array) or a dedicated circuit.

Figure 13 is a diagram showing the internal configuration of the deletion index determination unit 903 in the second embodiment. Here, the differences in the configuration of the deletion index determination unit 903 from the deletion index determination unit 108 will be described with reference to Figure 13.

The deletion index determination unit 903 further includes a target processing speed determination unit 902 and a calculation speed analysis unit 905 in addition to the deletion index determination unit 108 of the first embodiment. Here, the target processing speed determination unit 902 receives as an additional input the running speed information output from the running speed acquisition unit 901. The calculation speed analysis unit 905 receives as an additional input the output 916 from the maximum number of operations storage unit 302 and the target processing speed 310, which is the output from the target processing speed determination unit, and outputs the maximum number of operations 906.

Next, the connection relationship of the deletion index determination unit 903 in the second embodiment will be described. First, travel speed information is input to the target processing speed determination unit 902. In addition, the output 916 of the maximum number of operations storage unit 302, the target processing speed 310 which is the output of the target processing speed determination unit 902, the channel information 111, and the filter information 113 are input to the calculation speed analysis unit 905. Then, the calculation speed analysis unit 905 outputs the number of channels to be deleted 311, the number of filters to be deleted 312, and the changed maximum number of operations 906. In addition, the changed maximum number of operations 906 is input to the deletion channel index output unit 307 and the deletion filter index output unit 308.

The operation of the deletion index determination unit 903 in the second embodiment will be described below. The target processing speed determination unit 902 calculates the running speed V per unit time from the input running speed information. Here, if V<X is satisfied, the target processing speed determination unit 902 maintains the existing target processing speed G. On the other hand, if V>X is satisfied, the target processing speed determination unit 902 changes the target processing speed G to 2G and transmits the changed target processing speed to the calculation speed analysis unit 905.

Furthermore, when the target processing speed is increased in the calculation speed analysis unit 905, the target processing speed determination unit 902 changes the maximum operable number ²ⁿ to 2n ^-1 . When the target processing speed is decreased, the target processing speed determination unit 902 returns it to ²ⁿ . Next, the target processing speed determination unit 902 determines the number of filter channels to be deleted based on the changed maximum operable number. Then, the target processing speed determination unit 902 transmits the changed maximum number of operations 906 to the deleted channel index output unit 307 and the deleted filter index output unit 308. Other operations are the same as those in FIG. 5 and will not be described.

Next, FIG. 14 is an explanatory diagram of the storage state of the filter when it is calculated by the calculator. FIG. 14(a) shows an example before the maximum number of calculations is reduced, and FIG. 14(b) shows an example after the maximum number of calculations is reduced. These are also configured based on the calculator 151 and weight parameters 152.

Now, the case where the maximum number of operations is 8 (n=3) and the filter size is 9 will be described with reference to FIG. 14(a). When the maximum number of operations is 8, one index is deleted from the filter and the weight parameter 152 is set in the calculator 151 for calculation. Also, an example where the driving speed increases and the maximum number of operations becomes 4 will be described with reference to FIG. 14(b). When the maximum number of operations is 4, 5 is deleted from the filter and set in the calculator 151. In this case, compared to FIG. 14(a), twice as many weight parameters 152 can be input to the calculator 151. This concludes the description of the second embodiment.

The third embodiment is an embodiment in which the recognition device 1000 of the first and second embodiments is applied to a control device 2000. FIG. 15 is a functional block diagram of the control device 2000 in the third embodiment. The control device 2000 is implemented as, for example, an ECU.

In FIG. 3, the control device 2000 has a recognition device 1000 and a control signal generation unit 2001. The recognition result 103 output from the recognition device 1000, which executes the processing of the first and second embodiments, is transmitted to the control signal generation unit 2001. Next, the control signal generation unit 2001 generates a control signal 2002 according to the recognition result, and controls the control target 3000 based on this.

Here, when the control device 2000 is executed by an ECU, the control target 3000 is a vehicle. In this case, automatic driving and driving assistance of the vehicle can be realized based on the processing described in each embodiment.

This concludes the explanation of each embodiment, but various modifications and application examples are envisioned for each embodiment. For example, in each embodiment, the recognition device 1000 has been used as an example, but a calculation device that performs calculations not limited to recognition also falls within the scope of each embodiment.

In addition, according to each embodiment, when performing calculations using general image data, the utilization rate of the calculator is about 50%, and by using the calculation deletion mechanism according to the present invention, it is expected that the utilization rate of the calculator can be improved to nearly 100%. In other words, the utilization rate can be improved.

Furthermore, in this embodiment, when performing calculations in a convolutional neural network, some of the filters and channels used in the convolution calculation are deleted. For this reason, deletion index information, which is a number indicating the position or order of the filters and channels that differ for each layer, is provided to the calculation control unit. This allows the reading of some of the input data for the convolution calculation and some of the weight parameters to be skipped in accordance with the calculation unit of the calculator, and part of the convolution calculation is deleted. This allows the calculator to be used efficiently in a device with limited calculation performance.

101 External world information acquisition device 102 CNN calculation unit 104 Model storage unit 105 Channel information acquisition unit 106 Weight parameter acquisition unit 107 Filter information acquisition unit 108 Delete index determination unit 109 Calculation allocation unit 111 Channel information 112 Weight parameter information 113 Filter information 114 Delete filter index information 115 CNN calculation control signal 116 Delete channel index information 117 Input data 202 Parameter storage unit 203 Convolution calculation unit 204 Branched CNN calculation control signal 208 Individual weight parameter 209 Number of individual filters 210 Number of individual channels 211 Saved data 301 Target processing speed storage unit 302 Maximum number of calculations storage unit 303 Calculation speed analysis unit 304 Sensitivity information analysis unit 305 Delete channel determination unit 306 Delete filter determination unit 401 Calculation control unit 412 Calculation stop control signal 801 Intermediate data 802 Stride direction 803 Convolution filter 901 Travel speed acquisition unit 902 Target processing speed determination unit

Claims (10)

In a computing device that performs CNN calculations based on input data,
A model storage unit for storing a model used in the CNN calculation;
A CNN calculation unit that performs the CNN calculation by executing a convolution calculation for each of a plurality of convolution layers using the model;
a weight parameter acquisition unit that acquires weight parameters of a convolution filter used in the convolution operation from the model storage unit;
a channel information acquisition unit that acquires channel information for each of the plurality of convolution layers from the model stored in the model storage unit;
a filter information acquisition unit that acquires filter information for each of the plurality of convolution layers from the model storage unit;
An operation allocation unit that allocates a combination of weight information and operations required for the CNN operation and transmits the combination to the CNN operation unit;
a deletion index determination unit that deletes a portion of filter information used for the convolution calculations, based on the maximum number of calculations possible for the CNN calculation unit, an execution order of the CNN calculations, channel information for each convolution layer of the CNN calculation unit acquired from the channel information acquisition unit, weight information acquired from the weight parameter acquisition unit, and filter information for each of the plurality of convolution layers acquired from the filter information acquisition unit. 2. The computing device according to claim 1,
The deletion index determination unit is a calculation device that further deletes a part of the channel information used in the convolution calculation. 2. The computing device according to claim 1,
The deletion index determination unit is a calculation device that determines filter information to be deleted in response to a change in the weight information. 3. The computing device according to claim 2,
The deletion index determination unit is a calculation device that determines channel information to be deleted in response to a change in the weight information. 2. The computing device according to claim 1,
The filter indicated by the filter information is composed of a plurality of weight parameters,
The deletion index determination unit is a calculation device that determines a weight parameter that satisfies a predetermined rule from among the plurality of parameters. 6. The computing device according to claim 5,
The deletion index determination unit is a calculation device that determines a weight parameter to be deleted based on an amount of deterioration in calculation accuracy as the predetermined rule. 2. The computing device according to claim 1,
The deletion index determination unit is a calculation device that determines filter information to be deleted in accordance with a processing speed. 8. The computing device according to claim 7,
The deletion index determination unit is a calculation device that determines filter information to be deleted based on sensitivity information indicating sensitivities of the plurality of convolution layers and the size of the filter. The computing device according to any one of claims 1 to 8,
As the input data, external environment information acquired from an external environment acquisition device is used,
A recognition device that recognizes an external situation using the external world information. The recognition device according to claim 9,
A control device characterized in that the result of the calculation in the recognition device is output as a control signal for an object in accordance with the recognized external situation.

Images