JP7622351B2 - Arithmetic device, arithmetic method, and computer program - Google Patents

Description

This disclosure relates to a computing device, a computing method, and a computer program that use a neural network.

Conventionally, neural network configurations have been proposed that perform recognition according to the results of learning based on the content of data (see, for example, non-patent document 1).

Andrew G. Howard et. al., “MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications,” arXiv:1704.04861, 2017.

The arithmetic device according to the present disclosure has an arithmetic processing unit that executes a predetermined process by inputting predetermined data to a neural network having a plurality of convolution layers arranged in succession from the input side to the output side, and each of the plurality of convolution layers executes a convolution operation using a kernel with a different dilate size determined for each convolution layer for at least one of the plurality of dimensions including the channel direction and the spatial direction, for each feature included in a predetermined range of a feature map having a plurality of dimensions including the channel direction and the spatial direction, which is output from the immediately preceding layer and input to the convolution layer.

The computation method according to the present disclosure is a computation method in which a computer executes a predetermined process by inputting predetermined data to a neural network having multiple convolution layers arranged in succession from the input side to the output side, and each of the multiple convolution layers executes a convolution computation using a kernel with a different dilate size determined for each convolution layer for at least one of the multiple dimensions including the channel direction and the spatial direction, for each feature included in a predetermined range of a feature map having multiple dimensions including the channel direction and the spatial direction, which is output from the immediately preceding layer and input to the convolution layer.

The computer program disclosed herein is a computer program that causes a computer to execute a predetermined process by inputting predetermined data to a neural network having multiple convolutional layers arranged in succession from the input side to the output side, and each of the multiple convolutional layers executes a convolution operation using a kernel with a different dilate size determined for each convolutional layer for at least one of the multiple dimensions including the channel direction and the spatial direction, for each feature included in a predetermined range of a feature map having multiple dimensions including the channel direction and the spatial direction, which is output from the immediately preceding layer and input to the convolutional layer.

FIG. 1 is a schematic diagram showing a schematic configuration of a computer. FIG. 2 is a functional block diagram of a processor. FIG. 1 is a schematic diagram showing the structure of a neural network. FIG. 2 is a schematic diagram showing an example of a structure of a feature map. 1A is a schematic diagram showing an example of the structure of a kernel having a dilate size of 1; FIG. 1B is a schematic diagram showing an example of the structure of a kernel having a dilate size of 2; and FIG. 1C is a schematic diagram showing an example of the structure of a kernel having a dilate size of 4. 13 is a processing flowchart of a calculation processing unit. 1A-1C are schematic diagrams showing different structures of neural networks. FIG. 13 is a schematic diagram showing yet another different structure of a neural network. FIG. 13 is a schematic diagram showing yet another different structure of a neural network. 1A is a schematic diagram showing another example of the structure of a kernel having a dilate size of 1; FIG. 1B is a schematic diagram showing another example of the structure of a kernel having a dilate size of 2; and FIG. 1C is a schematic diagram showing another example of the structure of a kernel having a dilate size of 4. FIG. 1 is a schematic diagram illustrating a configuration of a camera equipped with a computing device according to an embodiment. 1 is a process flowchart of a camera equipped with a computing device according to an embodiment. FIG. 2 is a schematic diagram showing a structure of a neural network according to an embodiment. FIG. 11 is a schematic diagram illustrating an example of processing using a neural network. FIG. 11 is a schematic diagram illustrating another example of processing using a neural network.

The calculation device, calculation method, and computer program are described in detail below with reference to the drawings. However, it should be understood that the present invention is not limited to the drawings or the embodiments described below.

Figure 1 is a schematic diagram showing the general configuration of a computer according to one embodiment of the present disclosure, and Figure 2 is a functional block diagram of a processor.

In this embodiment, a computer 1, which is an example of a computing device, executes a predetermined process by inputting predetermined data into a neural network. To this end, the computer 1 includes an input/output interface 11, a memory 12, and a processor 13.

The input/output interface 11 has an interface circuit for accepting data to be processed by the computer 1 or for outputting data processed by the computer 1. The input/output interface 11 includes, for example, a communication interface circuit for connecting the computer 1 to a communication network, or a peripheral device interface circuit for connecting the computer 1 to various peripheral devices such as a keyboard and a display. The input/output interface 11 accepts the input of specified data and outputs the processing results.

The memory 12 is an example of a storage unit, and includes at least one of a semiconductor memory, a magnetic disk device, and an optical disk device. The memory 12 stores a driver program, an operating system program, an application program, data, and the like used in processing by the processor 13. For example, the memory 12 stores a computer program for implementing the computational processing of the present disclosure as an application program. The memory 12 also stores a set of parameters (number of layers, layer configuration, kernel, weighting coefficient, etc.) for defining a neural network as data. Various programs may be installed into the memory 12 from a computer-readable portable recording medium using a known setup program or the like. Examples of computer-readable portable recording media include a CD-ROM (Compact Disc Read-Only Memory) and a DVD-ROM (DVD Read-Only Memory).

The processor 13 includes one or more processors and their peripheral circuits. The processor 13 is a processing circuit that controls the overall operation of the computer 1, and is, for example, a CPU (Central Processing Unit). The processor 13 may have a GPU (Graphics Processing Unit) that can process a large amount of routine calculations in parallel. The processor 13 controls the operation of the input/output interface 11, etc., so that various processes of the computer 1 are executed by appropriate means based on the programs, etc. stored in the memory 12. The processor 13 executes processes based on the programs (operating system programs, driver programs, application programs, etc.) stored in the memory 12. The processor 13 can also execute multiple programs (application programs, etc.) in parallel.

As shown in FIG. 2, the processor 13 has an arithmetic processing unit 131 for executing calculations by the neural network NN1. The arithmetic processing unit 131 is a functional module implemented by a program executed on a processor possessed by the processor 13. Alternatively, the arithmetic processing unit 131 may be implemented in the computer 1 as an independent integrated circuit, microprocessor, or firmware.

The arithmetic processing unit 131 executes a predetermined process by inputting predetermined data to the neural network NN1. The predetermined process is, for example, object detection that identifies the type and position of an object shown in an image, region segmentation that performs class classification on a pixel-by-pixel basis in an image, etc.

The neural network NN1 has an upstream layer UL1, a middle layer ML1, and a downstream layer DL1.

Figure 3 is a schematic diagram showing the structure of neural network NN1, Figure 4 is a schematic diagram showing an example of the structure of a feature map, and Figure 5 is a schematic diagram showing an example of the structure of a kernel (or filter).

The neural network NN1 is, for example, a neural network having a CNN (Convolutional Neural Network) type architecture. The neural network NN1 accepts, for example, image data as input, executes a predetermined calculation, and detects objects shown in the image data. The neural network NN1 is trained in advance according to the backpropagation method using multiple images showing the object to be detected, and information indicating the position, type, etc. of the object in each image as training data.

In this case, the neural network NN1 performs a convolution operation on the input image data in the upstream layer UL1, and outputs a feature map with the same resolution as the image data. The upstream layer UL1 then performs a pooling process on the feature map, generates a feature map with a reduced resolution, and outputs the feature map to the middle layer ML1. The middle layer ML1 performs a convolution operation using multiple convolution layers, and outputs a feature map with the same resolution as the input feature map. The downstream layer DL1 performs a pooling process on the input feature map, and generates a feature map with a reduced resolution. Multiple bounding boxes are output from the finally obtained feature maps of various resolutions, and NMS (Non-Maximum Suppression) processing is performed on the multiple bounding boxes to detect objects.

The neural network NN1 may have a different upstream layer, a middle layer, and a downstream layer further downstream of the above-mentioned upstream layer UL1, middle layer ML1, and downstream layer DL1. In this case, the different upstream layer may include the downstream layer DL1 and perform processing such as object detection.

The neural network NN1 may also be trained for semantic segmentation. In this case, the upstream layer UL1 performs convolution and pooling on the input image data in the same manner as in the case of object detection to generate a feature map with reduced resolution, and outputs it to the middle layer ML1. The middle layer ML1 performs convolution using multiple convolution layers, and outputs a feature map with the same resolution as the input feature map to the downstream layer DL1. The downstream layer DL1 performs pooling on the input feature map in the same manner as the upstream layer UL1 to generate a feature map with reduced resolution. A transposed convolution is performed on the feature map output from the neural network NN1 to generate a feature map with increased resolution, and each pixel of the image is classified into multiple classes, and semantic segmentation is performed.

The upstream layer UL1 has an input layer that accepts input of image data, one or more convolution layers that perform convolution operations on the input image data, and outputs a feature map 100 having multiple dimensions. Furthermore, the upstream layer UL1 may have a pooling layer, and the resolution of the obtained feature map may be reduced by the pooling layer before being output to the middle layer ML1.

Figure 4 shows the structure of the feature map 100 output from the upstream layer UL1. The feature map 100 has a structure in which multiple features are arranged in multiple dimensions. In the feature map 100, the multiple features are arranged in spatial directions that include at least the H (height) direction and the W (width) direction, and are further arranged in the C (channel, depth) direction.

In this embodiment, the nodes shown in FIG. 3 correspond to the channels of the feature map 100. That is, the feature map 100 has a number of channels according to the number of nodes included in the final layer L10, which is located at the most output side of the upstream layer UL1. For the sake of simplicity, FIG. 3 illustrates eight of the nodes (N101-N108) included in the final layer L10. Note that the number of nodes and the number of features to be convolved are not limited to this embodiment.

The middle layer ML1 has multiple convolutional layers L11-L13 arranged in succession from the input side to the output side. The convolutional layers L11-L13 perform a convolutional operation for at least one of the multiple dimensions of the feature map. In this embodiment, the convolutional layers L11-L13 perform a convolutional operation for at least the C direction and output the result of the operation.

Each of the multiple convolution layers performs a convolution operation on individual features included in a predetermined range of a feature map having multiple dimensions that is output from the immediately preceding layer and input to the convolution layer. That is, each convolution layer has the same number of nodes as the number of channels of the input feature map, of which eight corresponding to N101-N108 are shown in FIG. 3. Nodes N111-N118 included in the convolution layer L11 perform a multiply-and-accumulate operation between the features output by nodes N101-N108 included in the final layer L10 of the upstream layer UL1 and the corresponding weighting coefficients of the kernel. For the convolution operation in each convolution layer, a kernel with a different dilate size is used for at least one of the multiple dimensions, which is determined for each convolution layer. In this embodiment, a kernel with a different dilate size is used for each convolution layer for the C direction.

In FIG. 3, a line connecting a node in an input layer (e.g., convolutional layer L11) to a node in an output layer (e.g., convolutional layer L12) indicates that the features output by the node in the input layer are used in the calculation of the node in the output layer (hereinafter also referred to as "connected").

The convolutional layers L11-L13 perform convolution operations on individual features that fall within a specified range in the C direction of the feature map. In Figure 3, the midstream layer and the specified range are indicated by a double border.

Nodes N111-N118 in convolutional layer L11 perform convolution operations using a kernel with a dilate size of 1 on the individual features output from nodes N101-N108 of the feature map output from the final layer L10 of the immediately preceding upstream layer UL1.

Figure 5(a) is a schematic diagram showing an example of the structure of kernel K11 with a dilate size of 1. In Figures 5(a)-(c), the hatched squares in each kernel indicate that a weighting factor has been set. Also, the unshaded squares in each kernel indicate that no weighting factor has been set (or that 0 has been set as the weighting factor). In other words, only the weighting factor corresponding to the hatched squares is used as the input for the convolution calculation. In kernel K11 shown in Figure 5(a), weighting factor W112 is set at the position corresponding to the target node, and weighting factors W111 and W113 are set at positions that are 1 away from the target node.

The convolution calculation in the convolution layer L11 is performed by sequentially changing the target node (stride = 1) and applying the kernel K11 to each feature included in a predetermined range of the feature map output from the final layer L10 of the immediately preceding upstream layer UL1. For example, node N112 adds the product of the feature output by node N101 and the weighting factor W111, the product of the feature output by node N102 and the weighting factor W112, and the product of the feature output by node N103 and the weighting factor W113, and outputs the result. Note that in this specification, the stride refers to the interval between the positions at which the kernel is applied, based on the feature map input to the midstream layer.

In the convolution layer L12, a convolution operation is performed using the kernel K12 with a dilate size of 2 shown in FIG. 5(b), and in the convolution layer L13, a convolution operation is performed using the kernel K13 with a dilate size of 4 shown in FIG. 5(c). In the kernel K12 shown in FIG. 5(b), a weighting factor W122 is set at a position corresponding to the target node, and weighting factors W121 and W123 are set at positions where the distance from the target node is 2. In the kernel K12, a weighting factor is not set at a position where the distance from the target node is 1. In the kernel K13 shown in FIG. 5(c), a weighting factor W132 is set at a position corresponding to the target node, and weighting factors W131 and W133 are set at positions where the distance from the target node is 4. In the kernel K13, a weighting factor is not set at positions where the distance from the target node is 1, 2, or 3.

In this way, in the middle layer ML1, convolutional calculations are performed using kernels with different dilate sizes for each convolutional layer by multiple convolutional layers L11-L13 arranged consecutively from the input side to the output side.

Each feature output from nodes N131-N138 falling within a predetermined range of the feature map output from the most output convolutional layer L13 is combined with a portion of features falling within a predetermined range of the feature map input to the most input convolutional layer L11. The portion of features refers to features falling within a range separated by the sum of the dilate sizes of the kernels used in each of the convolutional layers L11-L13 (1+2+4=7).

In FIG. 3, for example, node N135 in convolutional layer L13 is connected to nodes N121 and N125 in convolutional layer L12.

In convolutional layer L12, node N121 is connected to nodes N111 and N113 of convolutional layer L11. Node N125 is connected to nodes N113, N114, and N117 of convolutional layer L11. Therefore, node N135 of convolutional layer L13 is connected to nodes N111, N113, N114, and N117 of convolutional layer L11 via nodes N121 and N125 of convolutional layer L12.

In the convolutional layer L11, node N111 is connected to node N101 and node N102 in the final layer L10 of the upstream layer UL1. Node N113 is connected to node N102, node N103, and node N104 in the final layer L10 of the upstream layer UL1. Node N115 is connected to node N104, node N105, and node N106 in the final layer L10 of the upstream layer UL1. Node N117 is connected to node N106, node N107, and node N108 in the final layer L10 of the upstream layer UL1. Therefore, node 135 of convolutional layer L13 is connected to nodes N101-N108 of the final layer L10 of the upstream layer UL1 via nodes N121 and N125 of convolutional layer L12 and nodes N111, N113, N114, and N117 of convolutional layer L11.

In this way, each feature output from nodes N131-N138 of convolutional layer L13 is combined with all C-direction features output from nodes N101-N108 of the final layer L10 of the upstream layer UL1.

The feature map output from the convolution layer L13 is input to the downstream layer DL1. The downstream layer DL1 has an output layer at the output side that outputs the calculation results of the neural network NN1. By using a softmax function as an activation function, the output layer can identify the most likely type of detection target type for a region of interest on an image, for example. In addition, by using a sigmoid function as an activation function, the output layer can output a confidence level for each type. The downstream layer DL1 may include a transposed convolution layer that performs a transposed convolution calculation, a pooling layer, etc., on the input side of the output layer.

Figure 6 is a processing flowchart of the calculation processing unit 131. The calculation processing unit 131 inputs predetermined data to the neural network NN1 and executes the following processing.

First, the calculation processing unit 131 causes the upstream layer UL1 of the neural network NN1 to execute a predetermined calculation on predetermined data and output a feature map 100 (step S1). The feature map 100 includes features corresponding to a predetermined range that is the target of the convolution calculation in the convolution layers L11-L13.

Next, the calculation processing unit 131 executes a convolution calculation on the unprocessed layer closest to the input side among the multiple convolution layers L11-L13 included in the middle layer ML1 of the neural network NN1 (step S2). In the convolution calculation in step S2, a kernel with a dilate size preset for each convolution layer is used. In this embodiment, a kernel with a dilate size of 1 is used in the convolution layer L11, a dilate size of 2 is used in the convolution layer L12, and a dilate size of 4 is used in the convolution layer L13.

Next, the calculation processing unit 131 determines whether the convolution calculations by all convolution layers L11-L13 included in the middle layer ML1 have been completed (step S3).

If it is determined that the convolution calculations by all convolution layers have not been completed (step S3: N), the calculation processing unit 131 executes the process of step S2. A dilate size different from the dilate size associated with the processed convolution layer is associated with the unprocessed convolution layer. Therefore, by repeatedly executing the process of step S2 in this manner, convolution calculations using kernels with different dilate sizes are executed in each of the multiple convolution layers.

If it is determined that the convolution calculations by all the convolution layers have been completed (step S3: Y), the calculation processing unit 131 inputs the output of the middle layer ML1 to the downstream layer DL1 and causes the downstream layer DL1 to execute a predetermined calculation (step S4). The calculation processing unit 131 obtains the calculation result output from the final layer of the downstream layer DL1 and ends the calculation by the neural network NN1.

The computational load when convolution calculations using a 3 × 3 kernel (k _H = k _W = 3) are performed by MobileNets described in Non-Patent Document 1 on a feature map having 256 channels (C _in = C _out = 256) with H × W features per channel can be calculated using the following Equation 1.

{(k _H ×k _W ×C _in )+(1×1×C _in ×C _out )}×H×W = (2304+65536)×H×W = 67840×H×W
(Equation 1)

On the other hand, when the computing device of the present disclosure performs a calculation using the same kernel for the same feature map, the computational load is calculated using the following formula 2.

C _in ×log ₂ C _in ×k _H ×k _W ×H×W = 256×8×3×3×H×W = 18432×H×W (Formula 2)

By performing calculations using the neural network NN1 in this way, the calculation device can reduce the amount of calculations required for a feature map that has a large number of features in the dimension to be calculated.

The dilate sizes of the kernels used in the convolution layers L11-L13 are powers of 2, such as 1 (=2 ⁰ ), 2 (=2 ¹ ), and 4 (=2 ² ). Setting the dilate sizes in this way enables convolution operations that combine more features using fewer convolution layers.

The dilate size of the kernel used in the convolutional layer may be a power of a natural number equal to or greater than 2. For example, a kernel whose dilate size is a power of 3, such as 1 (=3 ⁰ ), 3 (=3 ¹ ), or 9 (=3 ² ), may be used. Furthermore, the dilate size of the kernel is not limited to a power of a natural number equal to or greater than 2, and may be set appropriately so that a required range of features can be combined by a predetermined number of convolutional layers.

In the present disclosure, the structure of the neural network is not limited to the structure shown in FIG. 3. For example, the dilate sizes used in the convolution calculations in multiple convolution layers included in the middle layer of the neural network do not have to be set in ascending order, such as 1, 2, 4. FIG. 7 is a schematic diagram illustrating a different structure of the neural network.

The neural network NN2 shown in FIG. 7 has an upstream layer UL2, a middle layer ML2, and a downstream layer DL2.

The upstream layer UL2 is similar to the upstream layer UL1 in the neural network NN1, and the downstream layer DL2 is similar to the downstream layer DL1, so detailed explanations are omitted.

The middle layer ML2 has multiple convolutional layers L21-L23 arranged in succession from the input side to the output side. The nodes N211-N218 included in the convolutional layer L21 perform convolutional operations using a kernel with a dilate size of 4 on the individual features output from the nodes N201-N208 of the feature map output from the final layer L20 of the upstream layer UL2, which is the immediately preceding layer. The nodes N221-N228 included in the convolutional layer L22 perform convolutional operations using a kernel with a dilate size of 1 on the individual features output from the nodes N211-N218 of the feature map output from the immediately preceding layer, the convolutional layer L21. The nodes N231-N238 included in the convolutional layer L23 perform convolutional operations using a kernel with a dilate size of 2 on the individual features output from the nodes N221-N228 of the feature map output from the immediately preceding layer, the convolutional layer L22.

In this way, when performing convolution operations using kernels with different dilate sizes in multiple consecutive convolution layers, the dilate sizes do not need to be changed in ascending or descending order from the input side to the output side, and the order can be arbitrary.

Each feature output from nodes N231-N238 of the output-most convolutional layer L23 is combined with all C-direction features input from nodes N201-N208 of the final layer L20 of the upstream layer UL2 to nodes N211-N218 of the convolutional layer L21.

In addition, multiple convolutional layers included in the middle layers of the neural network may perform convolution operations with a stride of 2 or more. Figure 8 is a schematic diagram illustrating yet another different structure of a neural network.

The neural network NN3 shown in FIG. 8 has an upstream layer UL3, a middle layer ML3, and a downstream layer DL3.

The upstream layer UL3 is similar to the upstream layer UL1 in the neural network NN1, so a detailed description is omitted.

The middle layer ML3 has multiple convolutional layers L31-L33 arranged in succession from the input side to the output side. Nodes N311-N318 included in the convolutional layer L31 perform convolution operations using a kernel with a dilate size of 4 on the individual features output from nodes N301-N308 of the feature map output from the final layer L30 of the upstream layer UL3, which is the immediately preceding layer.

In convolution layer L32, a convolution operation is performed using a kernel with a dilate size of 1. The stride of the convolution operation at this time is 2. In other words, when a convolution operation is performed at node N321, no convolution operation is performed at the adjacent node N322. Also, a convolution operation is performed at node N323, which is located at a distance of 2 from node N321.

In convolution layer L33, a convolution operation is performed using a kernel with a dilate size of 2. At this time, the stride of the convolution operation is 4. That is, a convolution operation is performed at nodes N331 and N335, and a convolution operation is not performed at nodes N332-N334 and nodes N336-N338.

Each feature output from nodes N331 and N335 of the output-most convolutional layer L33 is combined with all C-direction features input from nodes N301-N308 of the final layer L30 of the upstream layer UL3 to nodes N311-N318 of the convolutional layer L31.

Layer L34 included in downstream layer DL3 performs a transposed convolution operation using the features output from nodes N331 and N335.

In this way, in the neural network NN3, the middle layer ML3 includes convolution layers L32 and L33 that perform convolution operations with a stride of 2 or more. Therefore, in the neural network NN3, the amount of calculations on the output side is reduced compared to convolution layers that perform convolution operations with a stride of 2 or more.

Although an example has been described in which the strides are set to 1, 2, and 4 in the convolutional layers L31-L33, the present disclosure is not limited to this. In other words, it is sufficient that a convolutional layer that performs convolutional operations with a stride of 2 or more is placed at any position in the middle layer ML3. Note that placing a convolutional layer that performs convolutional operations with a stride of 2 or more in a position close to the input side is effective in reducing the amount of calculations.

In addition, individual features that fall within a predetermined range and are the subject of the convolution operation in the midstream layer may be combined with features that do not fall within the predetermined range. Figure 9 is a schematic diagram illustrating yet another different structure of a neural network.

The neural network NN4 shown in FIG. 9 has an upstream layer UL4, a middle layer ML4, and a downstream layer DL4. The downstream layer DL4 is similar to the downstream layer DL1 in the neural network NN1, so a detailed description is omitted.

In addition to nodes N401-N408 corresponding to the specified range that is the subject of the convolution operation in the middle layer ML4, the upstream layer UL4 has nodes N400 and N409 outside the specified range.

The middle layer ML4 has multiple convolutional layers L41-L43 arranged in succession from the input side to the output side. In addition to nodes N411-N418 included in the specified range, convolutional layer L41 has nodes N410 and N419 outside the specified range. In addition to nodes N421-N428 included in the specified range, convolutional layer L42 has nodes N420 and N429 outside the specified range. In addition to nodes N431-N438 included in the specified range, convolutional layer L43 has nodes N430 and N439 outside the specified range.

Node N410 outside the specified range in convolutional layer L41 is connected to node N400 outside the specified range and node N401 included in the specified range in the final layer L40 of the upstream layer UL4. Node N420 outside the specified range in convolutional layer L42 is connected to node N410 outside the specified range and node N411 included in the specified range in convolutional layer L41. Node N430 outside the specified range in convolutional layer L43 is connected to node N420 outside the specified range and node N421 included in the specified range in convolutional layer L42.

Node N419 outside the specified range in convolutional layer L41 is connected to node N408 included in the specified range and to node N409 outside the specified range in the final layer L40 of the upstream layer UL4. Node N429 outside the specified range in convolutional layer L42 is connected to node N418 included in the specified range and to node N419 outside the specified range in convolutional layer L41. Node N439 outside the specified range in convolutional layer L43 is connected to node N428 included in the specified range and to node N429 outside the specified range in convolutional layer L42.

In this way, in the neural network NN4, nodes included in the specified range are connected to nodes outside the specified range. Therefore, in the neural network NN4, the features output by the calculations at the nodes included in the specified range can be used for the calculations at the nodes outside the specified range.

The convolution calculation in the middle layer ML1 may also be performed in a dimension other than the C direction, for example, the HW direction. In the convolution calculation performed in the HW direction (two dimensions), a two-dimensional kernel is used. Figure 10 is a schematic diagram showing another example of the kernel structure. In Figures 10(a)-(c), it is shown that a weighting factor is set in the hatched squares in each kernel. In Figures 10(a)-(c), no weighting factor is set in the unshaded squares in each kernel.

Figure 10 shows examples of kernels used in convolution calculations performed in the HW direction, and Figures 10(a), (b), and (c) are examples of the structures of kernels K21, K22, and K23 with die size of 1, 2, and 4, respectively. Note that the two dimensions convolved using the kernels shown in Figures 10(a)-(c) are not limited to the H direction and W direction, and may be, for example, the H direction and C direction, or the W direction and C direction.

In addition, in the computation device disclosed herein, the computation processing unit 131 may perform convolution in three or more dimensions using a kernel (not shown) having a three or more dimensional structure.

Next, an example in which a computing device according to one embodiment is mounted on a camera will be described. FIG. 11 is a schematic diagram showing the configuration of a camera equipped with a computing device according to one embodiment.

The arithmetic device according to this embodiment is a microcontroller 4 mounted on a camera 2. The camera 2 is an example of an imaging device, and includes an image sensor 3, a microcontroller 4, and a lens 5. The image sensor 3 is an example of an imaging element, and converts light from a subject collected by the lens 5 into image data. The microcontroller 4 is an integrated circuit that integrates a processor, a memory, and an input/output interface. The microcontroller 4 performs calculations based on the image data input from the image sensor 3, and outputs a signal indicating an area in the image data to be the target of autofocus. The microcontroller 4 also operates as a focus control unit, and outputs a focus control signal that controls the lens 5 to focus on a predetermined area represented in the image data. The lens 5 is an example of an optical system, and collects light from a subject on the image sensor 3. The lens 5 has an autofocus mechanism 51, and the autofocus mechanism 51 focuses the lens 5 on the subject based on the focus control signal output from the microcontroller 4.

Fig. 12 is a processing flowchart of a camera equipped with a computing device according to one embodiment. Camera 2 starts the processing of Fig. 12 in response to detection of an action instructing execution of autofocus, such as a user half-pressing a shutter button (not shown). Camera 2 may repeatedly execute the processing of Fig. 12 while the action instructing execution of autofocus is being detected.

First, the image sensor 3 transmits image data corresponding to the collected light to the microcontroller 4 (step S10).

Next, the microcontroller 4 inputs the received image data to the neural network and executes a calculation to identify an area corresponding to the object to be identified contained in the image data (step S20). The calculation process by the neural network is the same as the flowchart of the calculation by the neural network shown in FIG. 6.

Then, the microcontroller 4 generates a focus control signal for focusing on the area corresponding to the object identified by the calculation, and transmits it to the autofocus mechanism 51. The autofocus mechanism 51 changes the position of the focusing lens included in the lens 5 based on the focus control signal received from the microcontroller 4 (step S30). Whether or not the specified area of the image is in focus is determined by a method such as a contrast AF method or a phase difference AF method.

The microcontroller 4 has a processing unit 431 having a neural network NN4 instead of the processing unit 131 having the neural network NN1. Figure 13 is a schematic diagram showing the structure of the neural network NN5 of the processing device mounted on the camera 2.

The neural network NN5 has an upstream layer UL5, a middle layer ML5, and a downstream layer DL5.

The upstream layer UL5 differs from the upstream layer UL1 in the neural network NN1 in that it has a first upstream layer UL5A and a second upstream layer UL5B, each of which performs a specific calculation, but is otherwise similar.

The middle layer ML5 has a first middle layer ML5A and a second middle layer ML5B, each of which performs a specific operation. The middle layer ML5 also has a number of convolutional layers L51-L52 arranged in succession from the input side to the output side. The nodes N511-N514 included in the convolutional layer L51 perform convolutional operations using a kernel with a dilate size of 1 on the individual features output from the nodes N501-N504 of the feature map output from the final layer L50 of the upstream layer UL5, which is the immediately preceding layer. The nodes N515-N518 included in the convolutional layer L51 also perform convolutional operations using a kernel with a dilate size of 1 on the individual features output from the nodes N505-N508 of the feature map output from the final layer L50 of the upstream layer UL5. Note that the kernel used for the convolution operation of nodes N515-N518 may have different parameters than the kernel used for the convolution operation of nodes N511-N514.

Similarly, in convolutional layer L52, a convolution operation is performed using a kernel with a dilute size of 2.

In this way, in the middle layer ML5, convolutional calculations are performed using kernels with different dilate sizes for each convolutional layer by multiple convolutional layers L51-L52 arranged consecutively from the input side to the output side.

The features output from the nodes N521-N524 in the feature map output from the most output convolutional layer L52 are combined with some of the features included in the specified range of the feature map input to the most input convolutional layer L51. The "some features" are features included in a range separated by the sum of the dilate sizes of the kernels used in each of the convolutional layers L51-L52 (1+2=3). In other words, all of the features input from the nodes N501-N504 in the final layer of the final layer L50 of the upstream layer UL5 to the nodes N511-N514 in the convolutional layer L51 are combined with the features output from the nodes N525-N528 in the convolutional layer L52.

In the neural network NN5, node N524 in convolutional layer L52 is connected to node N516 in convolutional layer L51, which has a distance of 2 outside the specified range. Therefore, features input to node N516 from nodes N505-N507 are also connected to node N524.

Downstream layer DL5 differs from downstream layer DL1 in neural network NN1 in that it has a first downstream layer DL5A and a second downstream layer DL5B, each of which performs a specific calculation, but is otherwise similar.

Figure 14 is a schematic diagram illustrating an example of processing using a neural network.

The first upstream layer UL5A, the first middle layer ML5A, and the first downstream layer DL5A of the neural network NN5 can detect a rectangular area including a predetermined identification object (e.g., a cow) by learning the teacher data. The first upstream layer UL5A, the first middle layer ML5A, and the first downstream layer DL5A may have an architecture similar to that of an SSD (Single Shot MultiBox Detector), for example. In this case, the first upstream layer UL5A performs a convolution operation and outputs a feature map with the same resolution as the resolution of the image PIC output by the image sensor 3. Then, the first upstream layer UL5A performs a pooling process on the feature map, generates a feature map with a reduced resolution, and outputs it to the first middle layer ML5A. The first middle layer ML4A performs a convolution operation and outputs a feature map with the same resolution as the resolution of the input feature map. The first downstream layer DL5A performs a pooling process and generates a feature map with a reduced resolution. The microcontroller 4 detects multiple bounding boxes from the feature maps of various resolutions finally obtained, and performs NMS processing on the multiple bounding boxes. By processing in this way, the neural network NN5 detects a rectangular area BB1 that includes the object to be identified from the image PIC.

The points included in rectangular area BB1 are likely to be the points that the user of camera 2 is paying attention to. Therefore, autofocus mechanism 51 controls lens 5 so as to focus on the points included in rectangular area BB1. Here, rectangular area BB1 includes point P1 that corresponds to the object to be identified and point P2 that does not. If autofocus mechanism 51 is controlled to focus on point P2 that does not correspond to the object to be identified, there is a high possibility that the autofocus control will not be in line with the user's intentions.

On the other hand, the second upstream layer UL5B, the second middle layer ML5B, and the second downstream layer DL5B can divide a region having a predetermined characteristic (e.g., corresponding to an object to be identified) by learning the teacher data. The second upstream layer UL5B, the second middle layer ML5B, and the second downstream layer DL5B may have an architecture similar to that of U-Net, for example. In this case, the second upstream layer UL5B performs a convolution operation and a pooling process, and outputs a feature map having a lower resolution than the resolution of the image PIC output by the image sensor 3. The second middle layer ML5B performs a convolution operation, and outputs a feature map having the same resolution as the resolution of the input feature map to the second downstream layer DL5B. The second downstream layer DL5B performs a transposed convolution operation. The second downstream layer DL5B may also perform a transposed convolution operation. The microcontroller 4 classifies the pixels of the image PIC into multiple classes. By processing in this way, the neural network NN5 divides the area of the image PIC into areas R1 and R2 corresponding to the class of objects to be identified, and area R3 corresponding to a class other than the objects to be identified.

The microcontroller 4 installed in the camera 2 can reduce the amount of calculation required to execute the above-mentioned processing by using the neural network NN5.

The microcontroller 4 of this embodiment can perform more efficient calculations using the neural network NN5. Figure 15 is a schematic diagram illustrating another example of processing using the neural network NN5.

In the neural network NN5 of this embodiment, some of the nodes included in the first middle layer ML5A and the second middle layer ML5B are connected. The above-mentioned learning is performed on such a neural network NN5. In the trained neural network NN5, the first middle layer ML5A is connected to the second middle layer ML5B, and the features on the first middle layer ML5A side that performs object detection and the features on the second middle layer ML5B side that performs region segmentation are mixed together. The mixing ratio is greater the closer to the boundary between the first middle layer ML5A and the second middle layer ML5B.

By inputting image PIC into such neural network NN5, a region that is the intersection of rectangular region BB1 and regions R1 and R2 corresponding to the object to be identified is identified, and region R4 that contains only the object to be identified is identified. Region R4 does not contain any points that do not correspond to the object to be identified, so by controlling autofocus mechanism 51 to focus on point P3 included in region R4, it is possible to perform autofocus control in accordance with the user's intentions. The computation device of the present disclosure can perform convolution computations that are suitable for such autofocus control with a small amount of computation.

It will be appreciated by those skilled in the art that various changes, substitutions and modifications may be made thereto without departing from the spirit and scope of this disclosure.

1 Computer 131 Processing unit NN1 Neural network

Claims (11)

The neural network has a plurality of convolution layers arranged in succession from an input side to an output side, and a calculation processing unit executes a predetermined process by inputting predetermined data to the neural network,
Each of the plurality of convolution layers performs a convolution operation using kernels with different dilate sizes determined for each of the convolution layers in the channel direction for all features in the channel direction of a feature map having a plurality of dimensions including a channel direction and a spatial direction, the feature map being output from a previous layer and input to the convolution layer;
The dilate size, which differs for each of the plurality of convolutional layers, is a power of n (n is a natural number equal to or greater than 2) and is set so that all of the features of the feature map input to the most input convolutional layer are combined with individual features of the feature map output from the most output convolutional layer.
Calculation device. The computing device according to claim 1, wherein all the features of the feature map output from the most output convolutional layer are combined with features included in a range separated by the sum of the dilate sizes of the kernels used in each of the plurality of convolutional layers of the feature map input to the most input convolutional layer. 3. The computing device according to claim 1, wherein each feature included in a predetermined range of the feature map input to the most input convolutional layer is combined with a feature included in a range spaced apart by a sum of the dilate sizes of kernels used in each of the plurality of convolutional layers in the predetermined range of the feature map output from the most output convolutional layer. The arithmetic device according to any one of claims 1 to 3, wherein one of the plurality of convolution layers executes a convolution operation with a stride of 2 or more. The arithmetic device according to claim 4, wherein one of the plurality of convolution layers performs convolution operations with a stride based on the dilate size. The arithmetic device according to claim 4 or 5, wherein one of the plurality of convolution layers performs convolution operations with a stride twice the dilate size. The arithmetic device according to claim 6, wherein one of the plurality of convolution layers performs a convolution operation with a stride of 4 when the dilate size is 2. The computing device according to any one of claims 2 to 7, wherein the neural network further comprises a transposed convolution layer connected to the plurality of convolution layers and performing a transposed convolution operation. an imaging element that outputs image data according to the light collected by the optical system;
9. A computing device according to claim 1, wherein the computing unit of the computing device executes a process of identifying an area corresponding to an object to be identified among image areas represented in the image data by inputting the image data as the predetermined data to the neural network;
a focus control unit that controls the optical system so as to focus on an area corresponding to the identified object. A computing method in which a computer executes a predetermined process by inputting predetermined data into a neural network having a plurality of convolution layers arranged successively from an input side to an output side, comprising the steps of:
Each of the plurality of convolution layers performs a convolution operation using kernels with different dilate sizes determined for each of the convolution layers in the channel direction for all features in the channel direction of a feature map having a plurality of dimensions including a channel direction and a spatial direction, the feature map being output from a previous layer and input to the convolution layer;
The dilate size, which differs for each of the plurality of convolutional layers, is a power of n (n is a natural number equal to or greater than 2) and is set so that all of the features of the feature map input to the most input convolutional layer are combined with individual features of the feature map output from the most output convolutional layer.
Calculation method. A computer program for causing a computer to execute a predetermined process by inputting predetermined data into a neural network having a plurality of convolution layers arranged successively from an input side to an output side,
Each of the plurality of convolution layers performs a convolution operation using kernels with different dilate sizes determined for each of the convolution layers in the channel direction for all features in the channel direction of a feature map having a plurality of dimensions including a channel direction and a spatial direction, the feature map being output from a previous layer and input to the convolution layer;
The dilate size, which differs for each of the plurality of convolutional layers, is a power of n (n is a natural number equal to or greater than 2) and is set so that all of the features of the feature map input to the most input convolutional layer are combined with individual features of the feature map output from the most output convolutional layer.
Computer program.

Images