JP6945987B2 - Arithmetic circuit, its control method and program - Google Patents

Description

The present invention relates to an arithmetic circuit used for pattern recognition and the like, a control method thereof, and a program.

The neural network method is widely applied to image processing devices such as pattern recognition devices. Among neural networks, a calculation method called Convolutional Neural Networks (hereinafter abbreviated as CNN) is attracting attention as a method that enables robust pattern recognition with respect to fluctuations in the recognition target. For example, Non-Patent Document 1 discloses various application examples and implementation examples of a convolutional neural network (CNN). For CNN processing, various network configurations have been proposed according to the signal to be recognized, the recognition function to be realized, and the like. Here, the configuration of the convolutional neural network shows a configuration expressed by the connection relationship of the convolution operation, such as the number of layers and the number of feature faces in the layer.

FIG. 15 is a network configuration diagram showing an example of a simple CNN process. The input layer 1501 corresponds to raster-scanned image data of a predetermined size when CNN processing is performed on the image data. The feature planes 1503a to 1503c indicate the feature planes of the first layer 1508. The feature surface is a data surface corresponding to the processing result of a predetermined feature extraction operation (convolution operation and non-linear processing). Since the feature surface corresponds to the feature extraction result for recognizing a predetermined object in the upper layer and is the processing result for the raster-scanned image data, the processing result is also represented by the surface.

The feature planes 1503a to 1503c are calculated by the convolution calculation and the non-linear processing corresponding to the input layer 1501. For example, the feature surface 1503a is calculated by the convolution operation of the two-dimensional filter kernel 15021a schematically shown and the non-linear conversion of the operation result.

For example, the convolution operation in which the filter kernel (filter coefficient matrix) size is volumeSize × lowSize is processed by the product-sum operation as shown below.

ここで、「ｉｎｐｕｔ（ｘ，ｙ）」は座標（ｘ、ｙ）での参照画素値を示し、「ｏｕｔｐｕｔ（ｘ，ｙ）」は座標（ｘ、ｙ）での演算結果を示す。また、「ｗｅｉｇｈｔ（ｃｏｌｕｍｎ，ｒｏｗ）」は座標（ｘ＋ｃｏｌｕｍｎ、ｙ＋ｒｏｗ）での重み係数を示し、「ｃｏｌｕｍｎＳｉｚｅ」及び「ｒｏｗＳｉｚｅ」はコンボリューションカーネルサイズを示す。

Here, "input (x, y)" indicates a reference pixel value in coordinates (x, y), and "output (x, y)" indicates a calculation result in coordinates (x, y). Further, "weight (column, low)" indicates a weighting coefficient in coordinates (x + volume, y + low), and "columnSize" and "lowSize" indicate a convolution kernel size.

In the CNN process, the product-sum operation is repeated while scanning a plurality of filter kernels on a pixel-by-pixel basis, and the final product-sum result is non-linearly converted to calculate the characteristic surface. Since the feature surface 1503a is calculated from one image data in the previous layer, the number of connections is 1. There is only one filter kernel 15021a for calculating the feature plane 1503a. Here, the filter kernel 15021b and the filter kernel 15021c are filter kernels used when calculating the feature planes 1503b and 1503c, respectively. The filter kernel may be abbreviated as a filter or a kernel.

FIG. 16 is a diagram illustrating an example in the case of calculating the feature surface 1505a in the CNN process. The feature plane 1505a is calculated from the three feature planes 1503a to c of the previous layer 1508 and is coupled to the feature planes 1503a to c. When calculating the data of the feature surface 1505a, first, a filter calculation (convolution calculation) using the kernel 15041a schematically shown is performed on the feature surface 1503a, and the result is held in the cumulative adder 1601. Similarly, the kernels 15042a and 15043a are subjected to convolution operations on the feature surfaces 1503b and 1503c, respectively, and the results are cumulatively added to the cumulative adder 1601. After the convolution operation using the three types of kernels is completed, the non-linear conversion process 1602 using the logistic function and the hyperbolic tangent function (tanh function) is performed.

The feature surface 1505a is calculated by performing the above processing while scanning the entire image pixel by pixel. Similarly, the feature plane 1505b is calculated for the three feature planes of the previous layer 1508 by using the three convolution operations shown in kernel 15041b, kernel 15042b, and kernel 15043b. Further, the feature plane 1507 is calculated for each of the feature planes 1505a to 1505 of the previous layer 1509 by using two convolution operations shown in kernel 15061 and kernel 15062.

It is assumed that each convolution coefficient is determined in advance by learning using a general method such as perceptron learning or backpropagation learning. For example, in object detection, pattern recognition, etc., a large size convolution kernel of 10 × 10 or more may be used.

As described above, since a large number of large kernel-sized convolutional operations are repeated in the CNN process, a huge number of product-sum operations are required. In order to support various recognition tasks with common hardware, it is required to efficiently process various networks with a high degree of parallelism.

Patent Document 1 proposes a device for increasing the speed by preparing a plurality of multiply-accumulate arithmetic units and processing convolution operations corresponding to a plurality of receptive field positions (pixel positions of feature planes to be calculated) in parallel. Further, Patent Document 2 proposes a CNN processing device having a configuration in which an arithmetic unit is assigned to a convolutional kernel.

JP 2010-134697 US2012 / 0303932

Yann LeCun, Koray Kavacvuoglu and Clement Farabet: Convolutional Networks and Applications in Vision, Proc. International Symposium on Circuits and Systems (ISCAS'10), IEEE, 2010,

However, in Patent Document 1, a plurality of receptive fields are processed in parallel by paying attention to one characteristic aspect to be calculated, but when the parallel processing cannot be performed efficiently depending on the kernel size of the convolution, the area to be processed, and the like. There is. For example, when the kernel size is small, the transfer time of the data input to the product-sum calculation unit becomes a bottleneck, and the processing efficiency of the product-sum calculation may decrease.

The present invention has been made in view of the above problems, and avoids a decrease in processing efficiency of the product-sum operation by sequentially performing filter operations on a part of the reference data held in the holding unit and a different filter. It is an object of the present invention to provide an arithmetic circuit. Another object of the present invention is to provide a control method and a program of the arithmetic circuit.

In order to solve the above problems, according to the present invention, an operation of connecting to a storage device that stores reference data and coefficient data of a filter operation and calculating a plurality of calculated feature surfaces by a filter operation on one reference feature surface. In the circuit, at least one arithmetic unit that executes the filter calculation of the reference data and the coefficient data, a reference data holding means that holds a plurality of reference data transferred from the storage device, and transfer from the storage device. and coefficient data holding means for holding a plurality of coefficient data, the each of the at least one computing unit, and one reference data among said plurality of reference data held in the reference data holding means, said coefficient data the operation of a single coefficient data of the plurality of coefficient data stored in the storage means, to execute the operation with each of the same one reference data different one coefficient data so as to sequentially execute a plurality of It has a control means for executing the calculation of the calculated feature plane while sequentially switching the calculated feature plane for each partial region of a predetermined size in the region of the calculated feature plane to be calculated.

According to the present invention, it is possible to avoid a decrease in the processing efficiency of the product-sum calculation by sequentially performing the filter calculation with a filter different from a part of the reference data held in the holding unit.

It is a block diagram which shows the structure of the arithmetic circuit of 1st Embodiment. It is a figure which conceptually explains the arithmetic processing of the arithmetic circuit of 1st Embodiment. It is a figure explaining the basic concept of a convolution operation. It is a figure which shows the structure of the control part of the arithmetic circuit. It is a figure explaining an example of a parallel convolution operation. It is a figure explaining the structure of the parallel product sum arithmetic unit of the arithmetic circuit. It is a figure explaining the structure of the shift register of an arithmetic circuit. It is a time chart explaining the operation of 1st Embodiment. It is a time chart explaining the operation of 1st Embodiment. It is a figure explaining the structure of the image processing apparatus provided with the parallel arithmetic circuit. It is a flowchart explaining operation of an image processing apparatus. It is a time chart explaining the operation of the 2nd Embodiment. It is a block diagram which shows the structure of the arithmetic circuit of 2nd Embodiment. It is a figure explaining the characteristic structure and operation of the 2nd Embodiment. It is a network configuration diagram which shows the example of CNN processing. It is a figure explaining the convolution operation in CNN processing.

(First Embodiment)
First, the first embodiment of the present invention will be described. FIG. 1 is a diagram illustrating a configuration of an arithmetic circuit according to a first embodiment of the present invention.

Prior to the description of FIG. 1, as an example of various arithmetic processes performed by the arithmetic circuit of the present embodiment, the basic concept of convolution arithmetic by this arithmetic circuit will be described with reference to FIG. FIG. 3 is an example of calculating the feature surface 306 from the reference feature surface 302 by the convolution calculation. Here, the concept of calculating the feature plane data 305 at three positions consecutive in the vertical direction of the feature plane 306 in parallel will be described. The basic idea is the same for the case where the positions of the characteristic surface 306 that are continuous in the horizontal direction are calculated in parallel. The size of the convolution kernel (filter kernel) is a coefficient of 3 rows and 1 column for explanation. The reference data required to calculate the data 305 of the feature surface 306 in parallel is the data 301 of the reference feature surface 302.

The shift register 303 and the shift register 307 of FIG. 3 hold reference data 301 and coefficient data of the convolution kernel, respectively. The shift register 303 supplies data at different reference positions to the plurality of multiply-accumulate calculators 304 in parallel, and the shift register 307 sequentially supplies coefficient data common to the plurality of product-sum calculators 304. The shift register 303 and the shift register 307 operate sequentially in synchronization with a clock (not shown), and the outputs thereof are calculated in parallel by the parallel multiply-accumulate calculator 304. Here, focusing on the characteristic surface data o1 to be calculated, o1 = i1 × w1 is calculated at the first clock, and o1 = o1 + i2 × w2 at the second clock, and o1 = o1 + i3 × w3 at the third clock are calculated. As a result, a desired convolution result (i1 × w1 + i2 × w2 + i3 × w3) can be obtained with 3 clocks. When the convolution kernel is two-dimensional, the two-dimensional convolution operation is realized by repeating and accumulating the above processing column by column while changing the reference data and the coefficient data.

By performing the convolution calculation with reference to the calculated feature surface 306 in this way, the data of the feature surface 306 at the position corresponding to the degree of parallelism of the multiply-accumulate calculator 304 is calculated in parallel with the number of clocks according to the size of the filter kernel. can do.

In this embodiment, a parallel convolution calculation method based on such a characteristic surface to be calculated will be described as an example. The calculation method of the present embodiment has a feature that there is no causal relationship between the size of the filter kernel and the degree of parallelism of the product-sum calculation unit 304. That is, the convolution operation can be processed with various degrees of parallelism.

The arithmetic circuit shown in FIG. 1 is a portion corresponding to the arithmetic circuit 1002 in the image processing apparatus shown in FIG. The arithmetic circuit shown in FIG. 1 sequentially calculates feature planes from lower layers according to the hierarchical connection relationship of a plurality of data groups as shown in FIG. The RAM 101 is a memory for storing the data of the previous layer to be calculated and the data of the calculation result. The RAM 101 is the same as the RAM 101 of FIG.

The control unit 102 controls control related to data transfer, control related to processing order of characteristic surfaces, and the like. FIG. 4 is a diagram illustrating a more detailed configuration of the control unit 102. The sequence control unit 1201 inputs and outputs various control signals 1204 that control the operation of FIG. 1 according to the information set in the register group 1202. Similarly, the sequence control unit 1201 outputs a control signal 1206 for controlling the memory control unit 1205. The sequence control unit 1201 is composed of a sequencer including a binary counter, a Johnson counter, and the like. The register group 1202 is composed of a plurality of register sets, and records, for example, information on a feature surface to be referred to, information on a feature surface to be calculated, information on a kernel, information on a processing order of feature surfaces, and the like. A predetermined value of the register group 1202 is written in advance from the CPU 1007 via the bridge 1004 and the image bus 1003.

The reference data shift register 106 is a data supply unit that supplies reference data to the parallel multiply-accumulate calculator 107. The reference data shift register 106 supplies the reference data (feature surface data of the previous layer required for the convolution calculation) buffered in the reference data buffer 105 to the parallel multiply-accumulate calculator 107 in parallel at a predetermined timing. The coefficient data shift register 104 is a data supply unit that supplies coefficient data to the parallel multiply-accumulate calculator 107, and sequentially supplies parameter data (weighting coefficient) required for the convolution calculation to the parallel multiply-accumulate calculator 107.

The parallel product-sum calculation unit 107 shall include m (m is 1 or more) product-sum calculation units. The parallel multiply-accumulate unit 107 operates in parallel with the same clock. FIG. 6 is a diagram showing a schematic configuration of the parallel product-sum calculator 107. The data 6011 to 601 m are the output data of the reference data shift register 106, and are different reference data supplied to each multiplier 6031 to 603 m. The data 602 is the output data of the coefficient data shift register 104, and is the data commonly supplied to each multiplier 6031 to 603 m. The cumulative adders 6041 to 604 m accumulate the multiplication results during the convolution kernel calculation period. The clear signal 605 is used to clear the built-in latches of the cumulative adders 6041 to 604 m when a predetermined convolution operation unit is completed. The latch enable signal (Latch Enable signal) 606 updates the cumulative addition value with the signal. It is assumed that a signal synchronized with a clock signal (not shown) is connected to the Latch Enable signal.

The coefficient data holding units 1031 to 103n temporarily store the coefficient data required for the arithmetic processing from the coefficient data (parameter data) stored in the RAM 101. The coefficient data holding units 1031 to 103n are composed of a cache and a prefetch buffer. The coefficient data holding units 1031 to 103n have n holding units (n is 1 or more) and hold weighting coefficients corresponding to n types of convolution kernels. In the present embodiment, the coefficient data is stored in the RAM 101, but the data is not limited to the RAM 101 and may be stored in another storage unit or storage device. For example, the coefficient data may be stored in a ROM or the like (not shown). As the operation result extraction unit, the result shift register 108 extracts the operation result at each end of the convolution operation.

In the present embodiment, a plurality of types of convolution kernels are stored in the coefficient data buffers 1031 to 103n, and the convolution kernels are sequentially switched and supplied to the parallel multiply-accumulate arithmetic unit 107 to process different convolution operations for the same reference data. .. That is, the data of different feature planes are calculated in order.

The non-linear conversion processing unit 109 performs non-linear conversion processing such as a sigmoid function on the data output from the result shift register 108. The output result of the nonlinear conversion processing unit 109 is stored in the RAM 101 via the control unit 102, and is held in the RAM 101 as reference data in the next layer. By referring to the feature plane which is the calculation result of the previous layer stored in the RAM 101, the multi-layer network can be sequentially processed.

The coefficient data shift register 104, the reference data shift register 106, and the result shift register 108 are shift registers with a data load function. The reference data buffer 105 and the coefficient data buffers 1031 to 103n are composed of a plurality of registers having the same bit width as the reference data shift register 106 and the coefficient data shift register 104, respectively. The result shift register 108 is composed of a plurality of registers having the same number of bits as the effective bits of the cumulative adder output of the parallel multiply-accumulate calculator 107. FIG. 7 shows a configuration example of these shift registers.

FIG. 7 describes an example when the number of registers is 4. The multi-bit flip-flops 701a to 701 latch the data of a predetermined bit in synchronization with the CLOCK signal. For the selectors 702a to c, OUTx (x: 0 to 2) is selected when the selection signal (Load signal) is 0, and INx (x: 1 to 3) is selected when the selection signal (Load signal) is 1. That is, the shift operation and the load operation are selected according to the load signal. The Enalbe signal is an enable signal for data transition. When it is 1, the data is latched at the rising edge of the CLOCK signal, and when it is 0, the latched data is held as it is (state transition does not occur).

The Load2 / Load4 / Load5 signals in FIG. 1 are the Load signals of the coefficient data shift register 104, the reference data shift register 106, and the result shift register 108, respectively. The Enable1 / Enable2 / Enable3 signals in FIG. 1 are the Enable signals of the coefficient data shift register 104, the reference data shift register 106, and the result shift register 108, respectively. After loading the initial data, the coefficient data shift register 104 executes a clock number shift operation having the same horizontal convolution kernel size. Weight coefficient data is sequentially supplied to the parallel multiply-accumulate calculator 107 according to the shift operation. The OUTn signals in FIG. 7 of the shift registers 405a and 405b are commonly connected to all parallel multiply-accumulate units 107.

Similarly, the reference data shift register 106 is loaded with initial data from the reference data buffer 105. After that, the same clock number shift operation as the horizontal convolution kernel size is executed, and a plurality of reference data (FIGS. 7 OUT1 to OUTn signals) are simultaneously supplied to the parallel multiply-accumulate unit 107.

The coefficient data shift register 104 and the reference data shift register 106 operate in synchronization. The parallel product-sum calculator 107 executes the product-sum operation according to the data supplied from the coefficient data shift register 104 and the reference data shift register 106. The cumulative sum obtained here is loaded into the result shift register 108 after the calculation of all the convolution kernels corresponding to the target feature planes is completed, and is sent to the nonlinear conversion processing unit 109 at a predetermined timing. As shown in FIG. 6, the parallel multiply-accumulate unit 107 is assumed to have m of the same circuits operating at the same clock. The result shift register 108 is composed of flip-flops capable of holding m product-sum operation outputs.

The output of the parallel multiply-accumulate unit 107 connects only a predetermined effective bit to the result shift register 108. The non-linear conversion processing unit 109 can be configured by a look-up table or the like. The data converted here is stored at a predetermined address of the RAM 101. The storage address here is also controlled according to the control unit 102.

FIG. 5 is a diagram schematically illustrating a specific example of parallel processing by the arithmetic circuit of the present embodiment. The reference data surface 802 and the calculated data surface 804 of FIG. 5 are represented using raster-scanned data coordinates, respectively. On the reference data surface 802, the calculation results (input (x, y), x: horizontal position, y: vertical direction) of the previous layer in which each data (minimum one box schematically shown) is stored in the RAM 101 in the raster scan order. Position). It is assumed that the calculated data surface 804 indicates the calculation result (output (x, y), x: horizontal position, y: vertical position) in which each data is raster-scanned.

The calculation range 803 indicates the position of the data obtained by the parallel multiply-accumulate operation unit 107 (when m = 4), and the reference range 801 is the calculation when the kernel size of the convolution operation is 3 × 3. The range of reference data for range 803. The control unit sequentially transfers the data of each line in the reference range 801 to the reference data register buffer, and the parallel multiply-accumulate arithmetic unit realizes the convolution operation along with the shift operation of the reference data.

Here, the basic operation of the convolution arithmetic processing will be described. The convolution calculation according to the present embodiment calculates the data of the m pixel positions continuous in the horizontal direction of the calculation feature plane in parallel. Each of the coefficient data buffers 1031-103n consists of at least more registers than the horizontal size of the convolution kernel. For example, in the case of data in which the weighting coefficient is represented by 8 bits, it is composed of a plurality of registers having a width of 8 bits. For example, when the horizontal convolution kernel size is "11", the number of the registers is "11".

Actually, it is composed of the number of registers of the assumed maximum convolution size. The control unit 102 loads a plurality of types of coefficients required for the product-sum calculation process into the register in advance, and selects and uses each of the calculation feature planes. The reference data buffer 105 is used to temporarily hold a part of the reference data stored in the RAM 101.

When the reference data is data represented by 8 bits, the reference data buffer 105 is composed of a plurality of 8-bit wide registers. The reference data buffer 105 is composed of a predetermined number or more of registers. This predetermined number is the number of reference data required for each of a plurality of arithmetic units to be processed in parallel to execute one unit of convolution operation. This predetermined number is, for example, (“the number of arithmetic units to be processed in parallel” + “convolution kernel size in the same direction as parallel processing” -1) × “convolution kernel in the direction parallel to parallel processing”. Calculated by "size".

Further, here, in order to pipeline the reading of the reference data and the convolution operation, the reference data buffer may be composed of a double buffer composed of registers having twice the above size. The reference data buffer outputs a plurality of data to be loaded into the reference data shift register 106 in parallel under the control of the control unit 102.

FIG. 2 is a diagram conceptually explaining the operation mode of the arithmetic circuit of FIG.

FIG. 2A shows an example of calculating a network having a one-to-one coupling relationship using a multiply-accumulate calculator 202 that operates in four parallels. Here, the convolution calculation of the 4-pixel position 203 of the feature surface 206 to be calculated is processed in parallel. The product-sum calculator 202 proceeds with the calculation in the order of raster scan in parallel processing units while referring to the necessary reference data 201 determined by the content of the convolution operation via the data buffer 205.

FIG. 2B shows an example in which a network having a one-to-two coupling relationship is processed by the arithmetic processing unit having the configuration shown in FIG. 2A. The feature planes 208 and 209 are processed in order using the four-parallel multiply-accumulate calculator 202. On the feature surface 208, the calculation areas 207 and 210 are calculated in this order. That is, the feature planes 208 and 209 are sequentially calculated by the plane sequential processing in the parallel processing unit. In this case, the reference data referred to by the feature planes 208 and 209 is common, but since the feature planes are processed in order of the planes, the same reference data 201 is transferred to the data buffer 205 for each processing of the feature planes 208 and 209. Will be.

In FIG. 2C, different feature planes 208 and 209 are sequentially calculated in parallel processing units using a four-parallel multiply-accumulate calculator 202. That is, the calculation areas 207, 211, 210, and 212 are processed in this order. In this case, for example, the reference data 201 on the reference feature surface 204 required for the calculation of the calculation areas 207 and 211 of the two feature surfaces 208 and 209 is held in the data buffer 205 and reused. Generally, the reference feature surface is stored in a low-speed, large-capacity memory, and the data buffer 205 is composed of a high-speed, small-capacity memory, registers, and the like. As shown in FIG. 2C, when processing is sequentially performed in parallel processing units across a plurality of feature planes, the data at the time of calculating the target area of the feature planes 208 and 209 can be shared via the data buffer 205. Therefore, as shown in FIG. 2B, the number of transfer of reference data to the data buffer 205 can be halved as compared with the case of processing in surface order. When the read transfer speed of the reference data is not taken into consideration, the processing times of FIGS. 2 (B) and 2 (C) can be regarded as equivalent, but when the transfer speed is slow, the data transfer time of FIG. 2 (B) is shown in FIG. The processing time is regulated, and the processing time may increase as compared with FIG. 2C. This becomes noticeable when the degree of parallelism of the arithmetic unit is high and the size of the convolution kernel is small.

As shown in this example, the best performance according to the number of parallel computing units can be obtained by switching the processing order as shown in FIGS. 2 (A) and 2 (C) according to the configuration and operating conditions of the CNN network. Can be pulled out.

In the present embodiment, it is proposed to optimize the processing order of the arithmetic circuit according to the configuration of the CNN network and the operating conditions, and select an appropriate operating mode.

Next, the operation mode of the arithmetic circuit of the present embodiment will be described in more detail with reference to FIGS. 8 and 9. As shown in FIG. 2A, FIG. 8 shows an outline of a time chart in the case where each feature plane is sequentially calculated in a plane sequence. As shown in FIG. 2C, FIG. 9 shows an outline of a time chart in a case where two characteristic planes are switched in convolution calculation units for calculation. 8 and 9 are time charts of operation modes in which the processing order of arithmetic processing is different. The operation mode can be changed by setting the register group 1202 in the control unit 102.

First, with reference to FIG. 8, an example (corresponding to the processing of FIG. 2A) in which one characteristic surface is subjected to raster scan processing in parallel processing units will be described. All the signals shown in FIG. 8 perform a synchronous operation based on a clock signal (not shown). FIG. 8 shows a part of the timing at the start of the feature surface processing. FIG. 8 shows a case where the kernel size is 5 × 5.

It is assumed that necessary coefficient data is loaded in the coefficient data buffers 1031 to 103n before the start of the arithmetic processing of the feature surface. The sel signal is a signal for selecting the output of the coefficient data buffers 1031 to 103n, and is used to select a desired coefficient from the coefficients corresponding to a plurality of convolution kernels. Here, the sel signal is always 0 during the operation of arithmetically processing one characteristic surface.

Further, it is assumed that the reference data buffer 105 is loaded with all the reference data necessary for arithmetic processing in the interval 402, which is the kernel vertical arithmetic interval 1.

The control unit 102 first activates the Load3 signal in order to start loading the necessary reference data in the section 403, which is the next kernel vertical calculation section 2. Here, it is assumed that the Road3 signal is in the enabled state when the signal level is 1. It is assumed that the reference data required for the kernel vertical section 1 has already been stored in the reference data buffer 105. Here, it is assumed that the reference data buffer is composed of a double buffer, and data reference and data load can be processed at the same time.

The control unit 102 takes out the reference data from the RAM 101 and sets it in the reference data buffer 105 at the same time when the Load3 signal is activated. The number of data to be set is determined from the size of the convolution kernel and the degree of parallelism. For example, when the kernel size of the convolution operation is 5 × 5, and the degree of parallelism of the operation unit is 20, 20 + 5-1 = 24 pieces of data are set. * The CLR signal is a signal for initializing the cumulative adders 6041 to 604 m of the parallel multiply-accumulate calculator 107, and when the signal is 0, the register built in the cumulative adder is initialized to 0. ..

The control unit 102 sets the * CLR signal to 0 before starting the convolution calculation of the new feature plane position. Since the reference data buffer 105 has a double buffer configuration, it outputs the data used in the kernel vertical calculation section 1 (section 402) and stores the data to be used in the kernel vertical calculation section 2 (section 403). .. After that, the reference data buffer 105 is controlled to read and write data as a double buffer according to a toggle signal (not shown).

The Load2 signal is a signal for instructing the initialization of the coefficient data shift register 104. When the signal is 1 and the Enable1 signal is valid (signal level 1), a plurality of weight coefficient data held in the coefficient data buffer 1031 are collectively loaded into the coefficient data shift register 104.

The Enable1 signal is a signal that controls the data transition of the shift register. Since the Enable1 signal is always set to 1 during the operation of the arithmetic unit, when the Load2 signal is 1, the output is latched according to the clock signal, and when the Load2 signal is 0, the output is shifted according to the clock signal. Continue processing.

The sequence control unit 1201 of the control unit 102 activates the Load2 signal when counting the number of clocks corresponding to the horizontal size of the convolution kernel, and stops the shift operation. At the same time, the sequence control unit 1201 collectively loads the weighting coefficient data held in the coefficient data buffer 1031 into the coefficient data shift register 104.

That is, the weighting coefficients are collectively loaded in the horizontal unit of the convolution kernel, and the loaded coefficients are shifted out according to the operating clock. Here, in the case of FIG. 8, the Ser signal is always 0x00, and the coefficient data buffers 1031 to 1031n sequentially output the coefficients of a specific kernel. That is, one characteristic surface is calculated with the same kernel.

The Load4 signal is a signal for instructing the initialization of the reference data shift register 106. When the signal is 1 and the Enable2 signal is valid (signal level 1), the reference data held in the reference data buffer 105 is collectively loaded into the reference data shift register 106. The data stored in the reference data buffer 105 outputs necessary reference data in horizontal processing units according to a timing signal (not shown). The data output by the reference data buffer 105 outputs different reference data corresponding to each kernel horizontal operation section (section 401).

The Enable2 signal is a signal that controls the data transition of the shift register, but is always set to 1 during operation. Therefore, when the Load4 signal is 1, the output of the reference data buffer 105 is latched according to the clock signal, and when the Load4 signal is 0, the shift process is continued according to the clock signal.

The sequence control unit 1201 of the control unit 102 activates the Load4 signal when counting the number of clocks according to the horizontal size of the convolution kernel, stops the shift operation, and at the same time, collectively loads the reference data held in the reference data buffer 105. do.

That is, the necessary reference data is collectively loaded in units of one column of the convolution kernel, and the loaded reference data is shifted according to the operating clock. As described above, the control unit 102 controls the Load4 signal at the same timing as the Load2 signal.

Since the parallel multiply-accumulate unit 107 continues the product-sum operation in synchronization with the clock, the convolution kernel size is simultaneously applied to a plurality of points on the feature surface calculated according to the shift operation of the shift registers 104 and 106. Executes the product-sum operation process.

Specifically, during the shift operation period of the shift register 104 and the shift register 106 (kernel horizontal calculation section 401 in FIG. 8), the product-sum calculation for one column of the convolution kernel is performed.

By repeating the operation for each column in the horizontal direction while exchanging the weighting coefficient and the reference data, the two-dimensional convolution operation result corresponding to the number of parallelisms is calculated (the kernel vertical operation section 1 (section) in FIG. 8). 402)).

In this way, the control unit 102 controls each signal according to the kernel size and the degree of parallelism, so that the product-sum calculation process and the data (reference data) required for the product-sum calculation process are supplied from the RAM 101 in parallel. Let me.

The Load5 signal is a signal for loading the result of the parallel product-sum calculator in parallel to the result shift register 108, and the control unit 102 finishes the product-sum calculation of the parallel processing unit of the target feature surface, and then the Load5 signal and Enable3. Output 1 to the signal. As a result, when the Load5 signal is 1 and the Enable3 signal is 1, the shift register 108 collectively loads the outputs of the parallel multiply-accumulate calculator 107. The control unit 102 activates the signal of Enable 3 during the shift operation of the shift registers 104 and 105, and shifts out the calculation result held in the result shift register 108. The shifted-out calculation result is converted by the nonlinear conversion processing unit 109, and then stored by the control unit 102 at a predetermined address of the RAM 101 according to information such as a calculation result storage destination pointer written in the register group 1202.

In the arithmetic circuit of the present embodiment, reading of reference data to the RAM 101 and writing of the arithmetic result can be processed at high speed by parallel processing during the product-sum calculation processing period. However, depending on the relationship between the degree of parallelism and the convolution kernel, access to the RAM 101 may not be completely pipelined during the multiply-accumulate operation period. For example, if the degree of parallelism is high and the convolution kernel is small, the transfer of reference data by Load 3 may not be in time. In that case, the control unit 102 gives priority to the completion of access to the RAM 101, and delays the start of the product-sum calculation process by controlling the Enable1 / Enable2 / Enable3 signal and the Latch Enable signal of the cumulative adder.

FIG. 9 is a time chart in which two feature planes are sequentially processed in parallel arithmetic units. That is, it corresponds to FIG. 2 (C).

Here, only the difference from FIG. 8 will be described. In FIG. 9, the Ser signal and the Load3 signal are different from those in FIG. FIG. 9 shows an example in which processing is performed while switching between the two characteristic surfaces in kernel operation units. Prior to the start of processing of the feature surface, the control unit 102 stores the weighting coefficients required for the calculation of the feature surface in the coefficient data buffer 1031 and the coefficient data buffer 1032, respectively. Further, it is assumed that the reference data buffer 105 is already loaded with the reference data commonly used in the kernel vertical calculation section 1 (section 502) and the kernel vertical calculation section 2 (section 503).

In the kernel vertical calculation section 1 (section 502), the convolution calculation is executed by the parallel multiply-accumulate calculator 107 using the coefficient data selected at sel = 0x00. On the other hand, in the kernel vertical calculation section 2 (section 503), the convolution calculation is executed using the coefficient data selected at sel = 0x01. In these two sections, the common reference data stored in the reference data buffer 105 is referred to, and the reference data output by the reference data buffer 105 is in horizontal processing units according to a timing signal (not shown) as in the case of FIG. Output the required reference data. At that time, the same reference data is repeatedly output in the kernel vertical direction calculation section 1 (section 502) and the kernel vertical direction calculation section 2 (section 503) in the horizontal processing unit. Therefore, the time allowed for loading the reference data commonly used in the kernel vertical calculation section 3 (section 504) and the kernel vertical calculation section 4 (not shown) is the section 505, which corresponds to the case of FIG. 8 (corresponding to the section 405). ) Is twice as long.

In the operation of FIG. 9, the reference data is shared, the coefficient data is exchanged, and the data of the calculation area 207 on the feature surface 208 and the calculation area 211 on the feature surface 209 of FIG. 2C are sequentially calculated. Further, in the kernel vertical direction calculation section 3 (section 504), the coefficients are exchanged again to calculate the data in the calculation area 210 of the feature plane 208. While reusing the reference data in this way, the processing order of the feature planes calculated by exchanging the coefficients is controlled.

As is clear from FIG. 8, in the case of FIG. 9, by sharing the reference data between the two kernel vertical calculation sections (section 502, section 503), the number of data transfers to the reference data buffer (= transfer rate). ) Can be halved. As a result, it is possible to reduce the case where the data transfer regulates the processing time when the transfer of the reference data takes a long time, or when the kernel size is small and the kernel calculation interval is short.

For example, it is assumed that the parallel degree of the parallel product-sum calculation unit 107 is 20 and the kernel size is 5, and the parallel calculation unit completes the product-sum operation for one weighting coefficient in one cycle. Assuming that the weighting coefficient is 1 byte and the data transfer cycle is 4 bytes / cycle, the processing cycle per convolution processing is as follows in the operation mode of FIG.

The processing cycle of the parallel computing unit is 5 × 5 = 25 cycles.

Parallel computing The processing cycle required to transfer reference data required for processing unit operations is (20 + 5-1) x 5/4 = 30 cycles.

In this case, the data transfer controls the processing time, and the performance of the parallel computing unit is not fully utilized.

On the other hand, in the operation mode of FIG. 9, since the reference data is shared, the processing cycle is as follows.

The processing cycle of two parallel arithmetic processing units is 25 × 2 = 50 cycles.

The processing cycle required for transferring the reference data required for the operation of the parallel arithmetic processing unit is 30 cycles, and the arithmetic processing regulates the processing time, and the performance of the parallel arithmetic unit is utilized.

FIG. 10 shows the configuration of an image processing apparatus including the arithmetic circuit 1002 of the present embodiment. This image processing device has a function of detecting a specific object from input image data by pattern recognition processing. The image input module 1000 of FIG. 10 is composed of an optical system, a photoelectric conversion device such as a CCD or CMOS sensor, a driver circuit for controlling the sensor, an AD converter, a signal processing circuit for controlling various image corrections, a frame buffer, and the like.

The RAM (Random Access Memory) 101 is a memory used as a calculation work buffer of the calculation circuit 1002. Data corresponding to the characteristic surface of the CNN is stored in the RAM 101.

The DMAC (Direct Memory Access Controller) 1006 controls data transfer between each processing unit on the image bus 1003 and the CPU bus 1010. The bridge 1004 provides a bridge function between the image bus 1003 and the CPU bus 1010.

The pre-processing module 1005 performs various pre-processing for effectively performing the pattern recognition processing by the CNN processing. The preprocessing module 1005 is hardware that processes image data conversion processing such as color conversion processing / contrast correction processing.

The CPU 1007 controls the operation of the entire device by executing a control program. The ROM (Read Only Memory) 1008 stores instructions and parameter data that define the operation of the CPU 1007. The RAM 1009 is a memory required for the operation of the CPU 1007. The CPU 1007 can also access the RAM 101 on the image bus 1003 via the bridge 1004.

FIG. 11 is a flowchart illustrating the operation of the image processing apparatus of the present embodiment. Hereinafter, it is assumed that the flowchart is realized by the CPU 1007 executing the control program. In step S1101, the CPU 1007 executes various initialization processes prior to the start of the recognition process. The CPU 1007 transfers the weighting coefficient necessary for the operation of the arithmetic circuit from the ROM 1008 to the RAM 101, and sets various registers for defining the operation of the arithmetic circuit 1002, that is, the network configuration of the CNN. Specifically, the CPU 1007 sets predetermined values in a plurality of registers existing in the control unit 102 of the arithmetic circuit 1002. Similarly, the CPU 1007 writes a value necessary for operation to a register such as the preprocessing module 1005.

Next, in step S1102, the processing order for calculating each feature plane is determined.

As described with reference to FIGS. 8 and 9, the processing order of the feature planes is determined according to conditions such as the network structure of the CNN, the data transfer performance from the RAM 101 to the arithmetic units, and the number of arithmetic units operating in parallel. For example, when calculating a plurality of feature planes with respect to the feature planes of the lower hierarchy, the processing order is determined based on the transfer cycle and the calculation cycle. The transfer cycle is a cycle (transfer time) required for reading reference data required for a convolution operation that processes the positions of a plurality of calculated feature planes in parallel, and the operation cycle is a processing cycle required for the convolution operation. The processing order is determined based on the transfer cycle and the calculation cycle.

That is, in step S1102, it is determined whether to sequentially process each feature surface for each feature surface in a surface sequence based on the operating conditions, or to sequentially process each feature surface in a processing unit of an arithmetic unit across the feature surfaces.

After the initialization process of step S1101 and the process order determination of step S1102 are completed, a series of object recognition operations are started.

First, in step S1103, the image input module 1000 converts the signal output by the image sensor into digital data and stores it in a frame buffer (built in the image input unit 1000) for each frame.

When the storage in the frame buffer is completed, in step S1104, the preprocessing module 1005 starts the image conversion process based on the predetermined signal. The preprocessing module 1005 extracts luminance data from the image data on the frame buffer and performs contrast correction processing.

The luminance data is extracted by generating the luminance data from the RGB image data by a general linear conversion process. The contrast correction method also applies a generally known contrast correction process to enhance the contrast. The preprocessing module 1005 stores the luminance data after the contrast correction processing in the RAM 101 as a detection image.

When the preprocessing for one frame of image data is completed, the preprocessing module 1005 enables a completion signal (not shown). In step S1105, the arithmetic circuit 1002 activates the arithmetic circuit 1002 based on the completion signal, and starts the object detection process based on the CNN. In step S1106, when the calculation of the characteristic surface of the final layer is completed, the arithmetic circuit 1002 generates a completion interrupt to the CPU 1007. In step S1107, when the CPU 1007 receives the processing end interruption of the arithmetic circuit 1002, it analyzes the characteristic surface of the final layer and determines the position and attributes of the object in the image. When the analysis process of step S1107 is completed, the process for the image of the next frame is continued in step S1108.

As described above, in the present embodiment, the order of the feature planes to be processed by controlling the reference data and the coefficient data supplied to the parallel multiply-accumulate calculator 107 according to the operating conditions is changed for each convolution kernel. As a result, the reusability of the reference data can be improved and the memory access bottleneck can be eliminated.

According to the present embodiment, by controlling the processing order of the feature planes based on the configuration of the CNN network, the transfer cycle of the reference data, and the calculation cycle, it is possible to efficiently process various networks with simple control. can.

In the present embodiment, the case where the processing is performed in the processing order straddling the two characteristic surfaces has been described, but the present invention is not limited to this, and a configuration in which processing is performed while switching more characteristic surfaces may be used.

Further, in the present embodiment, an example of a configuration having a plurality of coefficient data buffers is shown. In this case, the influence of the load time of the coefficient data can be reduced, but this is not the case.

Further, in the present embodiment, the case where the two-dimensional convolution operation is processed by the parallel arithmetic unit has been described, but the present invention is not limited to the convolution operation. In the embodiment, an example of CNN processing for two-dimensional image data has been described, but it can also be applied to CNN processing for one-dimensional data such as audio data and three-dimensional data including changes in the time direction.

Further, in the present embodiment, the case of CNN processing has been described, but the present invention is not limited to this, and can be applied to other hierarchical processing such as Restricted Boltzmann Machines and Recurrent Neural Network.

Further, in the present embodiment, as shown in FIG. 5, a case where data of a plurality of feature planes arranged in the horizontal direction is processed in parallel has been described, but the present invention is not limited to this, and feature plane data continuous in the vertical direction is used. It may be configured to process in parallel.

Further, in the present embodiment, it has been described that a plurality of multiply-accumulate calculators process in parallel, but the present invention is not limited to this, and the feature plane data may be calculated using one product-sum calculator. good.

(Second Embodiment)
In the first embodiment, the configuration for processing while switching the feature planes to be processed in the convolution kernel unit has been described, but in the present embodiment, the configuration for processing while switching the feature planes to be processed in the multiply-accumulate operation unit will be described.

FIG. 13 is a diagram showing the configuration of the arithmetic circuit of the present embodiment. Here, only the difference from the first embodiment will be described. In FIG. 13, a product-sum state holding unit 110 is newly added to the configuration of FIG. The product-sum sum state holding unit 110 has a function of holding the state of the product-sum operation. FIG. 14 is a diagram illustrating the configuration and operation of the parallel product-sum calculation unit 107 including the product-sum state holding unit 110.

As shown in FIG. 14A, a case where the three feature planes 1405 to 1407 are sequentially processed in arithmetic processing units will be described. The filter kernels 1402 to 1404 are convolution kernel matrices required for calculating the feature planes 1405 to 1407, respectively.

FIG. 14B is a diagram illustrating an example of a coefficient data shift register 1408, a reference data shift register 1409, a parallel product-sum calculator 1410, and a product-sum state holding unit 110. The coefficient data shift register 1408 supplies the weighting coefficients to the multiply-accumulate calculator 1410 in order, similarly to the coefficient data shift register 104 of FIG. The reference data shift register 1409 supplies the reference data to the multiply-accumulate calculator 1410 in the same manner as the reference data shift register 106 of FIG. The multiply-accumulate calculator 1410 shows here one product-sum calculator among the parallel multiply-accumulate calculators. The product-sum calculator 1410 includes a multiplier and an adder. The product-sum state holding unit 11014111 indicates one product-sum state holding unit among the plurality of product-sum state holding units. The cumulative sum shift register 1412 comprises three shift registers. The selector-1413 selects one of the outputs of the cumulative sum shift register 1412.

FIG. 14C is a diagram illustrating the operation of the configuration shown in FIG. 14B. Here, a case where the three feature planes 1405 to 1407 are sequentially processed in the multiply-accumulate operation unit will be described. The coefficient data shift register 1408 stores the coefficient data 1402 to 1404 in the order of interleaving for each coefficient. Here, a filter kernel having a kernel size of 3 × 3 will be described. It is assumed that the data strings A1 to A3, the data strings B1 to B3, and the data strings C1 to C3 are each one data string of the filter kernels 1402 to 1404. In this case, the selector 1413 is set to feed back the MA3 output of the cumulative sum shift register 1412.

The output 1414 of the coefficient data shift register outputs in order in which the coefficients of different kernels are interleaved. In the first embodiment, it is necessary to control the Ser signal and select the kernel coefficient used for the calculation, but in the case of the present embodiment, it is not necessary. When storing the coefficient data in the coefficient data buffers 103 to 103n, it is only necessary to store the coefficient data of different kernels in the interleaved order. The control unit 102 stores the coefficient data in an interleaved state when the coefficient data is stored in the coefficient data buffer 1031.

The output 1415 of the reference data shift register shifts and outputs the reference data in order. Here, the shift clock of the reference data shift register is 1/3 of the shift clock of the coefficient data shift register. The first three clocks process the multiply-accumulate operation of three different filter kernels on the reference data D1, and the multiplier output 1406 is the same as the cumulative add output 1417. Next, the outputs 1418 to 1420 of the cumulative sum shift register are sequentially changed as shown in FIG. 14 (C). By feeding back MA3 of the cumulative sum shift register 1412 to the multiply-accumulate calculator 1410, it is possible to hold the product-sum calculation results of the three states. The arrow indicated by the dotted line in FIG. 14C indicates the state of the product-sum operation with respect to the convolution kernel 1402. The product-sum calculation loop via the cumulative sum shift register 1412 outputs the convolution calculation result to the output of the cumulative sum shift register 1412 after 9 cycles (output 1420). Similarly, the convolution calculation results for the convolution kernels 1403 and 1404 are sequentially output as the output of the cumulative sum shift register 1412.

FIG. 12 is a diagram showing an example of operation timing of the present embodiment. The basic operation is the same as that of the first embodiment. In this actual embodiment, data corresponding to the horizontal direction of parallelism and the kernel size is read from the RAM 101 during the valid period of the Laod3 signal. In addition, the product-sum operation for each row of three different feature planes is calculated in the kernel horizontal direction calculation section, and the convolution calculation of the three feature faces is executed in the kernel vertical direction calculation section. The calculation result is output through the nonlinear conversion processing unit 109 in the order in which the results of the three feature planes are interleaved.

As described above, in the present embodiment, different convolution operations are sequentially processed in the product-sum operation unit. Therefore, the reference data can be shared and reused in the product-sum operation unit. That is, the number of reference data to be transferred from the RAM 101 to the coefficient data shift register when calculating the three characteristic planes may be one. Therefore, as in the case of the first embodiment, it is possible to reduce the possibility that the data transfer from the RAM 101 to the coefficient data buffers 1031 to n limits the processing time.

Further, in the first embodiment, since the convolution kernel operation unit is processed across the feature planes, the reference data of the size required for the kernel operation is required as the reference data to be stored in the reference data buffer. On the other hand, in the present embodiment, since the processing is performed across the feature planes in the product-sum calculation unit, the reference data of the size required for the product-sum calculation unit may be used. That is, only "the number of arithmetic units to be processed in parallel" + "convolution kernel size in the same direction as the direction of parallel processing" -1 is sufficient.

Further, in the present embodiment, when changing the number of feature planes to be processed across the feature planes, only the data set in the coefficient data shift register 1408, the setting of the selector 1413, and the shift clock of the reference data shift register 1409 are modified. Is fine. For example, when processing two feature planes straddling the product-sum operation unit, the coefficient data An1402 and Bn1403 are interleaved and stored in the coefficient data shift register. The shift clock of the reference data shift register is set to 1/2 times, and the MA2 output of the cumulative sum shift register 1412 is set to be fed back to the adder of the multiply-accumulate calculator 1410. By adding such a simple configuration, the degree of freedom regarding the processing order of the feature planes can be increased.

In the second embodiment, the case where the processing order is controlled when the number of feature planes to be calculated is 1 to 3 has been described, but the present invention is not limited to this. By increasing the number of cumulative sum shift registers 1412, the processing order can be controlled for more calculated feature planes.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 RAM
102 Control unit 103 Coefficient data buffer 104 Coefficient data shift register 105 Reference data buffer 106 Reference data shift register 107 Parallel product-sum calculator 108 Result shift register 109 Non-linear conversion processing unit

Claims (16)

It is an arithmetic circuit that connects to a storage device that stores reference data and coefficient data of filter calculation and calculates a plurality of calculated feature planes by filter calculation for one reference feature plane.
At least one arithmetic unit that executes the filter operation of the reference data and the coefficient data, and
A reference data holding means for holding a plurality of reference data transferred from the storage device, and
A coefficient data holding means for holding a plurality of coefficient data transferred from the storage device, and
In each of the at least one arithmetic unit , one of the reference data of the plurality of reference data held by the reference data holding means and one of the plurality of coefficient data held by the coefficient data holding means . The calculation with the coefficient data is executed so as to sequentially execute the calculation with one reference data of the same and one coefficient data different from each other, and the calculation of a plurality of calculation feature planes is performed on the calculation feature plane to be calculated. A control means for executing while sequentially switching the calculated feature planes for each partial region of a predetermined size in the region.
An arithmetic circuit characterized by having. A reference data supply means that loads reference data from the reference data holding means and supplies the reference data to the arithmetic unit.
It further has a coefficient data supply means for loading coefficient data from the coefficient data holding means and supplying the coefficient data to the arithmetic unit.
Wherein, while is supplied with the same one reference data to one of said arithmetic unit from the reference data supply means, said coefficient data supplying means different plurality of coefficient data to the one of the arithmetic units from The arithmetic circuit according to claim 1, wherein the arithmetic circuits are supplied one by one. The reference data holding means has at least a first buffer and a second buffer, and the first buffer holds the reference data used for the filter calculation and supplies the held reference data to the calculator. The arithmetic circuit according to claim 1 or 2, wherein the second buffer holds reference data transferred from the storage device. The calculation circuit according to any one of claims 1 to 3, wherein the filter calculation is a convolution calculation of the reference data and the coefficient data. Any one of claims 1 to 4, further comprising a shift register that holds the output data of the arithmetic unit and a conversion means that performs non-linear conversion processing on the output data of the shift register. The arithmetic circuit described in. The arithmetic circuit according to claim 5, wherein the control means stores the output data of the shift register or the output data of the conversion means in the storage device. It is an arithmetic circuit that connects to a storage device that stores reference data and coefficient data of filter calculation and calculates a plurality of calculated feature planes by filter calculation for one reference feature plane.
At least one arithmetic unit that executes the filter operation of the reference data and the coefficient data, and
A reference data holding means for holding a plurality of reference data transferred from the storage device, and
A coefficient data holding means for holding a plurality of coefficient data transferred from the storage device, and
A reference data supply means that loads reference data from the reference data holding means and supplies the loaded reference data to the arithmetic unit.
A coefficient data supply means that loads the plurality of coefficient data from the coefficient data holding means and supplies the loaded coefficient data to the arithmetic unit.
One of the reference data of the plurality of reference data supplied from the reference data supply means and one of the plurality of coefficient data supplied from the coefficient data supply means to each of the at least one arithmetic unit . The calculation with the coefficient data is executed so as to sequentially execute the calculation with one reference data of the same and one coefficient data different from each other, and the calculation of a plurality of calculation feature planes is performed on the calculation feature plane to be calculated. A control means for executing while sequentially switching the calculated feature planes for each partial region of a predetermined size in the region.
An arithmetic circuit characterized by having. The calculation circuit according to claim 7, wherein the filter calculation is a convolution calculation of the reference data and the coefficient data. The number of reference data held in the reference data holding means is the number of reference data required for the arithmetic unit to perform one unit of the convolution operation, and is the number of the arithmetic units and the filter operation. The arithmetic circuit according to claim 4 or 8, wherein the calculation circuit is determined based on the size of the filter of the above. The arithmetic circuit according to claim 2 or 7, wherein the reference data supply means and the coefficient data supply means are shift registers having a data load function. The arithmetic circuit according to any one of claims 1 to 10, wherein the filter arithmetic is an arithmetic represented by a hierarchical connection relationship of a plurality of data groups of a convolutional neural network. The arithmetic circuit has a plurality of arithmetic units that process the filter operations in parallel, and the control means controls parallel processing by the plurality of arithmetic units based on the hierarchical coupling relationship. The arithmetic circuit according to claim 11. An image processing apparatus having the arithmetic circuit according to any one of claims 1 to 12 and processing image data as the reference data. The image processing apparatus according to claim 13, wherein the arithmetic circuit performs arithmetic processing for pattern recognition. It is a control method of an arithmetic circuit that connects to a storage device that stores reference data and coefficient data of a filter calculation and calculates a plurality of calculated feature planes by a filter calculation for one reference feature plane.
A calculation step in which at least one arithmetic unit executes the filter operation of the reference data of the filter operation and the coefficient data of the filter operation.
A reference data holding step in which the reference data holding means holds a predetermined number of reference data transferred from the storage device, and
A coefficient data holding step in which the coefficient data holding means holds the coefficient data of the first filter and the coefficient data of the second filter transferred from the storage device, and
Control means, wherein at least each one of the computing units, and one reference data among said plurality of reference data held in the reference data holding means, the coefficient of the plurality of coefficient data stored in the data storing means the operation of a single coefficient data, a manner is performed as to successively perform the operation between each the same one reference data different one coefficient data, the calculation of the plurality of calculated feature plane, the calculation target of the A control process in which the calculated feature planes are sequentially switched and executed for each partial region of a predetermined size in the region of the calculated feature planes.
A method characterized by having. It is a control program of an arithmetic circuit that connects to a storage device that stores reference data and coefficient data of filter calculation and calculates a plurality of calculated feature planes by filter calculation for one reference feature plane.
An operation step of causing at least one arithmetic unit to execute the filter operation of the reference data of the filter operation and the coefficient data of the filter operation.
A reference data holding step of causing the reference data holding means to hold a predetermined number of reference data transferred from the storage device, and a reference data holding step.
A coefficient data holding step for holding the coefficient data of the first filter and the coefficient data of the second filter transferred from the storage device in the coefficient data holding means, and a coefficient data holding step.
In each of the at least one arithmetic unit , one of the reference data of the plurality of reference data held by the reference data holding means and one of the plurality of coefficient data held by the coefficient data holding means . The calculation with the coefficient data is executed so as to sequentially execute the calculation with one reference data of the same and one coefficient data different from each other, and the calculation of a plurality of calculation feature planes is performed on the calculation feature plane to be calculated. A control step to execute while sequentially switching the calculated feature planes for each partial region of a predetermined size in the region, and
A program characterized by having a computer execute.

Images