JP7549841B2 - Image processing device, image recognition device, image processing program, and image recognition program - Google Patents

Description

The present invention relates to an image processing device, an image recognition device, an image processing program, and an image recognition program, for example, for image recognition of a learned object.

As demand for self-driving technology grows, research is being actively conducted into detecting pedestrians and vehicles using image recognition.
One such technique uses the GMM-MRCoHOG feature described in Patent Document 1. This technique maps the frequency distribution of co-occurrence of luminance gradient directions between different resolutions of the same image into a feature space as a continuous value, and boasts a highly robust discrimination ability compared to image recognition techniques using prior art Histograms of Oriented Gradients (HOG) features, Co-occurrence Histograms of Oriented Gradients (CoHOG) features, Multiple Resolution Co-occurrence. Histograms of Oriented Gradients (MRCoHOG) features, and the like.

More specifically, the HOG feature, CoHOG feature, and MRCoHOG feature all represent the frequency distribution of the intensity gradient direction of each pixel in a histogram by voting the intensity gradient direction of each pixel into a bin corresponding to a quantized direction (generally eight directions). In the HOG feature, the intensity gradient direction of each pixel is voted, in the CoHOG feature, the co-occurrence of intensity gradient directions of two pixels is voted, and in the MRCoHOG feature, the co-occurrence of intensity gradient directions at pixels across different resolutions is voted.
In contrast to these, the GMM-MRCoHOG feature uses a Gaussian Mixture Model (GMM) to arrange the luminance gradient direction in a state space as continuous values, thereby expressing the frequency distribution of co-occurrences by a multi-peak probability density function. While the first three form a state space with fixed bins, the GMM-MRCoHOG feature can form a state space autonomously.

Incidentally, in image recognition using GMM-MRCoHOG features, an image to be recognized is divided into a plurality of blocks, and a different basis function optimized for each block is used, which poses a problem of requiring a large amount of computational resources, such as memory capacity and CPU computing power.
In particular, when implementing the technology in a field programmable gate array (FPGA) or the like, where the computing resources are limited, saving computing resources is extremely important.

JP 2018-124963 A

The present invention aims to save computational resources in image recognition using brightness gradient orientation.

(1) The invention described in claim 1 provides an image processing apparatus comprising: an image acquisition means for acquiring an image for image recognition learning; a division means for dividing the acquired image into a plurality of blocks; a frequency distribution acquisition means for acquiring a frequency distribution of co-occurrence of luminance gradient directions for each of the divided blocks; a unification means for integrating the frequency distributions acquired for each of the acquired blocks into a single frequency distribution; and a basis function generation means for generating basis functions serving as a standard for image recognition based on the unified frequency distribution.
(2) In the invention as set forth in claim 2, there is provided the image processing apparatus as set forth in claim 1, characterized in that the unifying means performs the unification by superimposing the frequency distributions in the plurality of blocks.
(3) The invention described in claim 3 provides the image processing device described in claim 2, characterized in that the unifying means generates samples based on the frequency distribution for each of the acquired blocks, and adds up the generated samples across the multiple blocks, thereby superimposing the frequency distributions in the multiple blocks.
(4) In the invention described in claim 4, there is provided the image processing device described in claim 1, claim 2, or claim 3, characterized in that the frequency distribution acquisition means acquires a frequency distribution of co-occurrence of luminance gradient directions between different resolutions of the same image.
(5) The invention described in claim 5 provides an image processing device described in any one of claims 1 to 4, characterized in that the image acquisition means acquires a plurality of images, and the unification means unifies the frequency distribution for each block of the plurality of images into a single frequency distribution.
(6) The invention described in claim 6 provides an image processing device according to any one of claims 1 to 5, characterized in that the image acquisition means acquires a recognition target image including a recognition target and a non-recognition target image not including a recognition target, and the frequency distribution acquisition means acquires a frequency distribution in a corresponding block based on a difference in frequency distribution of a luminance gradient direction in the corresponding block between the recognition target image and the non-recognition target image.
(7) In the invention described in claim 7, there is provided an image processing device described in any one of claims 1 to 6, characterized in that the basis functions are probability density functions based on a Gaussian mixture model, and a determination means is provided for determining an appropriate number of mixtures based on a balance between the likelihood and the number of mixtures.
(8) In the invention described in claim 8, there is provided an image processing device described in any one of claims 1 to 7, characterized in that a weighting is set for the multiple blocks when integrating the frequency distributions, and the unifying means integrates the frequency distributions for each of the multiple blocks in accordance with the weighting.
(9) The invention described in claim 9 provides an image recognition device comprising: a basis function acquisition means for acquiring a basis function described in any one of claims 1 to 8; an image acquisition means for acquiring an image related to image recognition; a division means for dividing the acquired image into blocks; a feature acquisition means for applying the acquired basis function to each of the divided blocks and acquiring a feature for the basis function; and a determination means for determining whether or not a predetermined image recognition target is included in the acquired image using the feature acquired from each of the blocks.
(10) The invention described in claim 10 provides an image processing program for implementing on a computer an image acquisition function for acquiring an image for image recognition learning, a division function for dividing the acquired image into a plurality of blocks, a frequency distribution acquisition function for acquiring a frequency distribution of co-occurrence of luminance gradient directions for each of the divided blocks, a unification function for integrating the frequency distributions for each of the acquired blocks into a single frequency distribution, and a basis function generation function for generating basis functions serving as a standard for image recognition based on the unified frequency distribution.
(11) The invention described in claim 11 provides an image recognition program that realizes on a computer a basis function acquisition function for acquiring a basis function described in any one of claims 1 to 8, an image acquisition function for acquiring an image related to image recognition, a division function for dividing the acquired image into blocks, a feature acquisition function for applying the acquired basis function to each of the divided blocks and acquiring features for the basis function, and a judgment function for determining whether or not a predetermined image recognition target is included in the acquired image using the feature acquired from each of the blocks.

By using a unified basis function for multiple blocks, computational resources can be saved.

FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. FIG. 13 is a diagram for explaining a process of generating basis functions. FIG. 13 is a diagram for explaining a process of generating basis functions. FIG. 13 is a diagram for explaining a process of generating basis functions. FIG. 13 is a diagram for explaining mapping onto a feature space. FIG. 1 is a diagram for explaining the Akaike information criterion. FIG. 2 is a diagram showing various mathematical expressions. 11 is a flowchart for explaining a basis function generation process. FIG. 1 is a diagram for explaining an image recognition method. 13 is a graph showing the results of image recognition.

(1) Overview of the embodiment By adopting a unified GMM-MRCoHOG feature for the state space of all blocks, the basis functions are unified across each block. This allows for a significant reduction in the use of computational resources, in preparation for hardware implementation.
Specifically, the frequency distribution of the brightness gradient direction of each positive and negative image used for learning in each block is approximated by a probability density distribution using kernel density estimation.
Next, focusing on characteristic parts of the positive and negative images, the cumulative distribution function is calculated using a measure based on the JS information amount. A certain number of samples are then generated in a common feature space from the cumulative distribution functions of all blocks using the inverse function method, and approximated with a mixed Gaussian distribution using the EM algorithm. During the approximation, an appropriate number of mixtures is automatically determined using the Akaike information criterion.

(2) Details of the embodiment FIG. 1 is a diagram showing an example of a hardware configuration of an image processing device 8. As shown in FIG.
The image processing device 8 is configured by connecting a CPU 81, a ROM 82, a RAM 83, a storage device 84, a storage medium drive device 85, an input unit 86, and an output unit 87 via a bus line.
The CPU 81 is a central processing unit that operates according to an image processing program stored in the storage device 84, and performs processing to generate basis functions used for image recognition from learning images.

The ROM 82 is a read-only memory, and stores basic programs and parameters for operating the CPU 81 .
The RAM 83 is a readable and writable memory, and provides a working memory when the CPU 81 performs image processing.

The storage device 84 is configured using a large-capacity storage medium such as a hard disk, and stores image processing programs, learning images (learning image data), and the like.
The image processing program is a program that causes the CPU 81 to perform image processing functions.

The storage medium drive device 85 is a device that drives an external storage medium such as a semiconductor storage device or a hard disk.
The CPU 81 can read the learning image data from the storage medium.
The input unit 86 includes input devices such as a keyboard and a mouse for receiving input from an operator, reads various programs and data, and receives operations from the operator.
The output unit 87 includes output devices such as a display and a printer for presenting various information to an operator, and outputs an image processing operation screen and image processing results.

In addition, the image processing device 8 is equipped with a communication control unit for connecting to a communication network and an interface for connecting to external devices, and can also download learning image data from an external server.

The image processing device 8 executes an image processing program to generate basis functions used in image recognition using GMM-MRCoHOG features in accordance with the steps shown in each of Figures 2 to 4. This will be described below.
Image recognition using GMM-MRCoHOG features is a technology developed by the present inventors and boasts extremely high recognition accuracy.

In this embodiment, as shown in FIGS. 2(a) and 2(d), a basis function is created using the JS (Jensen-Shannon) divergence between a positive image 10 in which the image recognition target (a pedestrian) is captured in various poses, and a negative image 20 in which various background scenery is captured excluding the pedestrian.
The inventors of the present application have found that use of the JS divergence enables more robust image recognition.

In the figure, one positive image 10 and one negative image 20 are shown, but the image processing device 8 creates basis functions by learning from approximately 20,000 positive images 10 and negative images 20.
In this way, the image processing device 8 is equipped with an image acquisition means for acquiring a plurality of images for image recognition learning, each of which is composed of a recognition target image (positive image 10) containing the recognition target and a non-recognition target image (negative image 20) not containing the recognition target.

First, the image processing device 8 divides the positive image 10 into a plurality of identical square blocks 11a, 11b, 11c, .... In this example, the image is divided into a total of 18 blocks 11, three in the horizontal direction and six in the vertical direction, in accordance with the shape of a pedestrian.
Thus, the image processing device 8 includes a division means for dividing the image into a plurality of blocks.
Then, the image processing device 8 maps the frequency distribution of co-occurrence of the luminance gradient direction of each pixel in each block 11 into the feature space 13a, 13b, 13c, . . . for each block 11 shown in FIG.

The luminance gradient direction is, for example, the direction from low luminance to high luminance at the position of the pixel in question, and will hereinafter be abbreviated to gradient direction.
Furthermore, the space onto which the gradient direction is mapped and the space derived therefrom (such as the space obtained by subsequent sampling) are spaces into which image features are extracted, and therefore will be referred to as feature spaces.
When there is no need to distinguish between the blocks 11a, 11b, 11c, . . . and the feature spaces 13a, 13b, 13c, . . . , they will be simply referred to as the blocks 11 and the feature spaces 13, respectively, and the same applies to other similar components.

FIG. 5 is a diagram for explaining mapping of gradient directions onto the feature space 13. In FIG.
As shown in FIG. 5A, the image processing device 8 converts the resolution of the positive image 10 to generate a high-resolution image 15, a medium-resolution image 16, and a low-resolution image 17, each having a different image size, from the positive image 10.
If the resolution of the positive image 10 is appropriate, the positive image 10 is used as is as the high-resolution image 15. The image processing device 8 performs the following processing for each block of the images at each resolution.

First, the image processing device 8 calculates the gradient direction for each pixel of the high-resolution image 15, the medium-resolution image 16, and the low-resolution image 17. The angle of the gradient direction is a continuous value between 0° and 360°. It is also possible to use quantized values, such as 36 directions.

After calculating the gradient direction, the image processing device 8 obtains in block 11a the co-occurrence of the gradient directions of a reference pixel (hereinafter, the pixel of interest) and a pixel located at a distance from the reference pixel (hereinafter, the offset pixel) as follows:

First, as shown in FIG. 5(b), the image processing device 8 sets a pixel of interest 5 in the high-resolution image 15, and focuses on offset pixels 1a to 1d that are an offset distance 1 from the pixel of interest 5 in the high-resolution image 15 (i.e., adjacent in high resolution).
The distance of n pixels is referred to as the offset distance n.

Then, the image processing device 8 obtains the co-occurrence (combination of gradient directions) of each gradient direction between the pixel of interest 5 and offset pixels 1a to 1d, and plots the corresponding points as data points 51, 51, ... in the feature space 13a for block 11a shown in Figure 5(c).

For example, in FIG. 5(b), when plotting the co-occurrence of the pixel of interest 5 and the offset pixel 1a, if the gradient direction of the pixel of interest 5 is 26° and the gradient direction of the offset pixel 1a is 135°, the image processing device 8 plots a data point 51 at a position in the feature space 13a where the horizontal axis is 26° and the vertical axis is 135°.
Similarly, the image processing device 8 obtains the co-occurrence between the pixel of interest 5 and the offset pixels 1b to 1d and plots them in the feature space 13a. Note that the reason why the co-occurrence with the pixels above and to the left of the pixel of interest 5 is not obtained is because the co-occurrence is obtained while moving the pixel of interest 5 sequentially to the right, and therefore these have already been obtained and plotted in the previous stage.

Next, the image processing device 8 similarly obtains the co-occurrence of gradient directions with the pixel of interest 5 for offset pixels 2a to 2d of the medium-resolution image 16, which are located at an offset distance of 2, and plots them in the feature space 13a.Furthermore, the image processing device 8 similarly obtains the co-occurrence of gradient directions with the pixel of interest 5 for offset pixels 3a to 3d of the low-resolution image 15, which are located at an offset distance of 3, and plots them in the feature space 13a.

In this way, the image processing device 8 plots the co-occurrence of gradient directions with offset pixels at offset distances 1 to 3 from high resolution to low resolution, and then performs similar processing while sequentially moving the pixel of interest 5 within block 11a (also moving the pixel of interest 5 to the medium resolution image 16 and the low resolution image 17) to complete the feature space 13a for block 11a.

Although the movement of the pixel of interest 5 is performed within the block 11a, the offset pixel is selected even if it exceeds the block 11a. Also, an appropriate method is used for processing an edge of the block 11a that does not have an adjacent offset pixel.
In the same manner, the image processing device 8 plots data points in the feature spaces 13b, 13c, ... for the blocks 11b, 11c, .... In this way, it is possible to obtain a feature space 13 for each block 11, in which the frequency distribution of co-occurrence of gradient directions is represented by the density of data points.

In this way, the image processing device 8 includes a frequency distribution acquisition means for acquiring the frequency distribution of co-occurrence of brightness gradient directions between different resolutions of the same image for each divided block, and maps the co-occurrence distribution of gradient directions between multiple images with different resolutions into the feature space 13 as continuous values.
Returning to FIG. 2, the image processing device 8 also creates feature spaces 23a, 23b, 23c, . . . corresponding to the blocks 11a, 11b, 11c, .

Next, as shown in Fig. 2(c), the image processing device 8 generates a probability density function fp(x) for each block 11 from the positive data (data points on the feature space 13) plotted in the feature space 13. In the figure, the highs and lows of density are represented diagrammatically by contour lines.
Similarly, the image processing device 8 generates a probability density function fn(x) from the negative data (data points on the feature space 23) plotted in the feature space 23, as shown in FIG. 2(f).

fp(x) and fn(x) are expressed by equations (1) and (2), respectively, with the Gaussian function shown in equation (3) in FIG. 7A being the kernel density function.
n is the number of data. Xi(p) and Xi(n) are positive data and negative data, respectively, and are two-dimensional vector quantities. x is a point in the feature space and is a two-dimensional vector quantity.

Although the figures show superscript characters, they will be written in normal characters to avoid misconversion of character codes. The same applies to other equations. Also, vector quantities are written in bold, but they will also be written in normal characters to prevent misconversion.
h is a bandwidth, which is a parameter that defines the amount of spread of the distribution, etc. An appropriate value is set for this.

fp(x) and fn(x) represent the probability of co-occurrence of gradient directions in the positive image 10 and the negative image 20, respectively.
In this way, the image processing device 8 votes the co-occurrence of the gradient directions of positive data and negative data as continuous values in the feature space, and approximates the voted data points to a probability density function by kernel density function estimation.

Next, the image processing device 8 combines feature spaces (such as feature space 13a and feature space 23a) in which the positions of blocks on the positive image 10 and the negative image 20 correspond, as shown in Figure 3(a), and generates a JS information amount 33 for each pair from each fp(x) and fn(x), as shown in Figure 3(b).
In this way, the image processing device 8 generates the JS information amount 33a from the pair of feature space 13a and feature space 23a, generates the JS information amount 33b from the pair of feature space 13b and feature space 23b, and so on, generating the JS information amount 33 for each block 11.

The JS information amount is expressed as J(fp(x):fn(x)) in equation (5) of FIG.
J(fp(x):fn(x)) is defined using equation (6) so that the KL (Kullback-Leibler) information amount shown in equation (4) has symmetry.
The JS divergence constitutes a metric space that measures the distance between two probability distributions, and by using this, the similarity between fp(x) and fn(x) for x can be calculated.

The absolute value of J(fp(x):fn(x)) increases as the shapes of fp(x) and fn(x) differ (are less similar). Therefore, the JS information amount can express the characteristic parts of either fp(x) or fn(x).
Areas where there is a difference between fp(x) and fn(x), where the occurrence probability is biased towards one side, are useful information (areas where there is no bias are difficult to determine whether they are pedestrians or background), and the image processing device 8 extracts information on areas where the difference is large using the JS information amount.
In this way, the frequency distribution acquisition means included in the image processing device 8 acquires the frequency distribution in a block based on the difference in the frequency distribution of the luminance gradient direction in the corresponding block between the recognition target image and the non-recognition target image.

Next, as shown in FIG. 3( c), the image processing device 8 samples J(fp(x):fn(x)) for each block 11 using an inverse function method that uses the respective cumulative distribution functions, thereby generating samples (points in the feature space) for each block 11.
In this manner, feature spaces 35a, 35b, . . . in which samples are generated for each of the blocks 11a, 11b, .
In this way, the image processor 8 generates samples based on the frequency distribution for each block.

Since J(fp(x):fn(x)) is biased toward either positive or negative data, sampling using the inverse function method can generate a large number of samples that are concentrated in areas where the occurrence probability is biased.
When J(fp(x):fn(x)) is calculated using kernel density estimation, the number of Gaussian distributions in the underlying equation (3) depends on the number of data, resulting in a very large number of parameters. However, this can be reduced by sampling.

In the above explanation, the feature space 13 is generated for each block from a single positive image 10, but the image processing device 8 creates the feature space 13 for each block by superimposing the feature spaces 13 obtained from a large number of learning positive images 10 for each block.
For example, the feature space 13a1 created from the first positive image 10 and the feature space 13a2 created from the second positive image 10 are added together to create the feature space 13a, and similarly, the feature spaces 13b1, 13b2, ... are added together to create the feature space 13b. The same is true for the negative image 20.

As shown in FIG. 4(a), the image processing device 8 generates a feature space 35 based on sampling data for each block 11, and then integrates these feature spaces 35a, 35b, ... by adding together all the samples as shown in FIG. 4(b), thereby generating a unified feature space 36 in which the frequency distribution in the gradient direction is represented by the density of the samples.
In this way, the image processing device 8 is equipped with a unification means that integrates and unifies the frequency distributions for each block into a single frequency distribution by superimposing the frequency distributions for each block, and the unification means superimposes the frequency distributions in multiple blocks by adding up the generated samples across multiple blocks.
Furthermore, since the image processing device 8 integrates the frequency distributions for a large number of learning images into one, the unifying means unifies the frequency distributions for each block of a plurality of images into one frequency distribution.

As a modified example, a weighting may be set for each block 11, and the samples in the feature space 35 corresponding to that block 11 may be added according to that weighting.
For example, for blocks 11 with low weights, one addition is made per sample, for blocks 11 with high weights, three additions are made per sample, and so on.
This allows the weighting of blocks 11 with low importance (such as blocks at the four corners where pedestrians are less likely to be captured) to be set low, and the weighting of blocks 11 with high importance to be set high.
In this modified example, weighting is set for a plurality of blocks when integrating frequency distributions, and the unifying means included in the image processing device 8 integrates the frequency distributions for each of the plurality of blocks in accordance with the weighting.

After generating the unified feature space 36 in this manner, the image processing device 8 determines the number of mixtures using c-AIC (described later) as shown in FIG. 4( c), and further generates a state space by GMM using the EM algorithm (a method of searching for a formula for GMM by repeating the E step and the M step), and sets this as the basis function 37.
Here, the state space refers to a space in which the boundaries and arrangement of features are determined by a histogram, GMM, or the like.
In this manner, the image processing device 8 includes a basis function generating means for generating basis functions that serve as a reference for image recognition based on a unified frequency distribution.
Conventionally, a GMM was generated for each feature space 35a, 35b, ..., and a basis function was generated for each block 11. In contrast, the image processing device 8 of the present embodiment is novel in that it generates a basis function 37 common to all blocks 11 from the feature space 36.

The GMM is a model that approximates an arbitrary distribution by linearly superimposing Gaussian distributions, and is expressed by Equation (11), where k is the mixture number (the number of Gaussian distributions to be superimposed), N is the probability density function of the k-th Gaussian distribution with mean μk and variance-covariance Σk, and θ is the parameter of the mixed normal distribution with the mixture number k.
αj is a mixing coefficient representing the weight of the Gaussian distributions to be overlapped, and is a positive real number whose sum is 1 when added.
The GMM is normalized so that its integration becomes 1, and the distribution of samples in the feature space 36 can be approximated by a multi-modal probability density function p(x|θ) using the GMM.

In GMM, when the mixture number k is specified, the target distribution is clustered into k clusters and a Gaussian distribution is placed on top of them.
Thus, in order to construct an optimal state space using a GMM, it is necessary to determine the number of mixtures. However, if the number of mixtures is increased too much, the generalization ability of the model decreases and the calculation cost increases, which is a problem.
Therefore, before generating the GMM, the image processing device 8 automatically determined the number of mixtures using a measure based on the Akaike Information Criterion (AIC).

FIG. 6 is a diagram for explaining the Akaike information criterion.
The Akaike's information criterion includes AIC (Akai's Information Criterion) and c-AIC (correction of AIC) which uses AIC.
Here, AIC is a standard for evaluating the quality of a statistical model, and the more excellent the generalization ability of a model, the smaller its value.
On the other hand, c-AIC is a version of AIC that can be applied to a small number of samples.
As shown in Fig. 6(a), AIC is expressed as the sum of the monotonically decreasing degree of fit of the model and the monotonically increasing number of parameters. A model with the smallest AIC value has a good balance between penalty and model complexity and is a model with excellent generalization ability.

In this embodiment, the AIC is defined by equation (7) in FIG.
n is the number of samples, k is the number of mixtures, p is the occurrence probability of sample xi from the GMM, and θk (hat omitted) is a parameter of the GMM configured with the number of mixtures k.
tk is expressed by the following equation (8): where d is the number of dimensions of the sample data.

However, AIC assumes a large sample size and tends to overestimate the number of parameters when the number of samples is small.
Therefore, in this embodiment, the number of mixtures is determined according to c-AIC expressed by equation (9), which highly evaluates the simplicity of the model when the number of samples is small.

In equation (9), the first term is constructed by the negative log-likelihood, which is assumed to monotonically decrease as the model becomes more complex.
The second term is a penalty term depending on the number of parameters, and increases monotonically.
In this embodiment, the c-AIC values were calculated for several mixture numbers and curve approximation was performed, and the mixture number was determined from the approximated value obtained.
By using a curve approximation, the number of mixtures with the lowest c-AIC can be determined without being affected by the variability of the training data.

When an appropriate number of mixtures was searched for in this way, the result was as shown in FIG.
The horizontal axis of this graph represents the number of mixtures, and the vertical axis represents the negative log-likelihood. The smaller the negative log-likelihood (i.e., the larger the likelihood), the better the model.
The searched values plotted on the graph are the calculated values of c-AIC, and the estimated values are the points on the approximation curve obtained from the searched values.
As shown in the graph, when the number of mixtures is about 15 or more, the negative log-likelihood is almost constant, and it seems possible to reduce the number of mixtures to about 15.
In this way, the image processing device 8 generates the basis functions using a probability density function based on a Gaussian mixture model, and includes a determination means for determining an appropriate number of mixtures based on a balance between the likelihood and the number of mixtures.

FIG. 8 is a flowchart for explaining the basis function generation process performed by the image processing device 8.
The CPU 81 reads one positive image 10 from the storage device 84 and inputs and stores it in the RAM 83 (step 10).
Next, the CPU 81 divides the positive image 10 into blocks 11, plots the co-occurrence of gradient directions for each block in the feature space 13, and stores the plot in the RAM 83 (step 15).
The CPU 81 performs the above program processing on the positive images 10 as many times as necessary to generate the basis functions.
Next, the CPU 81 generates fp(x) for each block 11 by convolving the multiple feature spaces 13 stored in the RAM 83 for each block 11, and stores the fp(x) in the RAM 83 (step 25).

Next, the CPU 81 inputs the necessary number of negative images 20 stored in the storage device 84 into the RAM 83 (step 35) and plots them in the feature space 23 (step 40), and then generates fn(x) for each block 11 and stores them in the RAM 83 (step 50).

Next, the CPU 81 generates a JS information amount for each block 11 using fp(x) and fn(x) stored in the RAM 83, and stores the amount in the RAM 83 (step 55).
Next, the CPU 81 performs sampling based on the JS information amount stored in the RAM 83, generates a feature space 35 based on the samples for each block 11, and stores the sampled feature space 35 in the RAM 83 (step 60).

Next, the CPU 81 integrates the feature spaces 35 for each block 11 stored in the RAM 83 by adding them together, thereby generating a unified feature space 36 and storing it in the RAM 83 (step 65).
Next, the CPU 81 determines the mixture number for the feature space 36 stored in the RAM 83 using c-AIC, and further generates a GMM based on the mixture number using the EM algorithm (step 70).
Then, the CPU 81 stores the GMM in the RAM 83 and sets it as the basis function 37 to be used for image recognition (step 75).

Next, a method for extracting image features using the basis functions 37 will be described.
Although not shown in the figure, the image recognition device 9 has a hardware configuration similar to that of the image processing device 8, and is equipped with a memory device 94 that stores an image recognition program, an image to be recognized, and basis functions 37 generated by the image processing device 8, a CPU 91 that performs image recognition in accordance with the image recognition program, and a RAM 93 that provides working memory for the CPU 91.

The image recognition device 9 calculates the burden rate of the image with respect to the basis function 37 as a feature amount of the image, as follows.
The burden rate λj is expressed by equation (10) in FIG. 7(d), where z is a latent parameter (a k-dimensional vector quantity in which the j-th component is 1 and the others are 0).
The contribution rate λ j represents the probability that the distribution of data points x is generated from the j-th Gaussian distribution.
When the calculation is performed for each z, a k-dimensional vector by λj is obtained, which is used as a feature by the image recognition device 9. If the data point x is not similar to either the positive image 10 or the negative image 20, it approaches a 0 vector.
Based on this principle, the image recognition device 9 extracts features from an image in the following manner.

FIG. 9 is a diagram for explaining the image recognition method.
The following process is performed by the CPU 91 in accordance with the image recognition program.
As shown in FIG. 9A, the image recognition device 9 reads an image 40, which is the object of image recognition, into a RAM 93, and sets a rectangular area thereon using a discrimination filter 41a.
In this manner, the image recognition device 9 includes an image acquisition means for acquiring an image related to image recognition.
Then, the image recognition device 9 divides the image extracted by the discrimination filter 41a into, for example, 3×6 blocks 11a, 11b, . . . like the positive image 10 and the negative image 20.
Thus, the image recognition device 9 includes a division means for dividing the acquired image into blocks.

The image recognition device 9 then plots the co-occurrence of gradient directions across high, medium and low resolutions for each block 11 in a feature space 13 .
Then, the image recognition device 9 reads out the basis function 37 from the storage device 94, and calculates the burden rate of each block 11 with respect to the basis function 37 by using the formula (10) in FIG.
In this way, the image recognition device 9 is equipped with a basis function acquisition means for acquiring the basis function generated by the image processing device 8, and a feature acquisition means for applying the basis function to each divided block and acquiring features for the basis function.

The image recognition device 9 uses the feature amount based on the load ratio calculated in this manner to determine whether or not a pedestrian is captured in the image in the discrimination filter 41 , and stores the determination result in the RAM 93 .
This can be achieved in various ways. For example, a judgment may be made for each block 11 and then a comprehensive judgment may be made, or the burden rates of each block 11 may be integrated and judged as a whole.

The determination can be performed, for example, by inputting normalized features into a classifier such as an SVM (support vector machine) or AdaBoost.
After the image recognition device 9 has determined the image in the discrimination filter 41a in this manner, it shifts the discrimination filter 41 by one block while scanning the image 40, and performs the same determination.
In this manner, the image recognition device 9 includes a determination means for determining whether or not a predetermined image recognition target is included in an image using the feature amount acquired from each block.

The image recognition device 9 creates a state space by an optimal GMM that is unified for all blocks 11, and applies the same basis function 37 to each block 11, so that the state spaces of the blocks 11 are compatible with each other.
This makes it possible for the block 11 to inherit the feature values once calculated even if the discrimination filter 41 is moved, and eliminates the need to recalculate the feature values each time the discrimination filter 41 in the image is slid.

For example, the discrimination filter 41a and discrimination filter 41b in FIG. 9A share the block 11A.
Conventionally, a basis function is set for each block 11 for each discrimination filter 41, so that the feature amount is calculated again even for the same block 11A, as shown in the upper diagram of FIG. 9B.

In contrast, the image recognition device 9 uses the same basis function 37, so as shown in the lower diagram of FIG. 9(b), the feature value of block 11A becomes the same in discrimination filters 41a and 41b, and the feature value calculated earlier can be inherited. This allows for a significant saving in calculation resources.

As described above, in conventional methods, because different basis functions are used for each block, when calculating the features of adjacent rectangular regions, the feature calculations must be performed again for all regions, even though there are overlapping regions, resulting in high calculation costs. However, by adopting a common basis function 37, these problems can be solved.

Furthermore, in the conventional method that does not use GMM, as shown in the upper diagram of Fig. 9(c), for example, pairs of gradient directions quantized into eight directions are voted for in a two-dimensional histogram set for each block 11. Since the bins of the histogram are set regardless of the recognition target, voting is also performed for bins in which image features do not appear.
In contrast, in the method of generating a state space by GMM, as shown in the lower diagram of Fig. 9(c), a region with high probability density is formed autonomously in the region where image features appear, so there is no need to process regions where features do not appear, such as region 45. This makes it possible to reduce calculation costs.

Furthermore, conventionally, a basis function is set for each block 11. For example, if a basis function is generated for a 3×6 block 11, the discrimination filter must also be 3×6.
In contrast to this, in the method of this embodiment, since the basis function 37 is common, it may be possible to design the discrimination filter by modifying it in block units, for example to 3 × 5. This allows the coupling between the discrimination filter generating system and the discrimination filter using system to be looser than in the past.

FIG. 10 is a graph showing the results of image recognition using the basis functions 37 generated by the image processing device 8.
10(a), (b), and (c) are ROC (Receiver Operating Characteristic) curves based on the c-AIC value when the mixture number k is 45, 32, and 15, respectively, where the horizontal axis indicates the false positive rate and the vertical axis indicates the true positive rate.

The thick line shows the GMM-MRCoHOG features obtained by the image recognition device 9 with the basis function unified to basis function 37, the dashed line shows the conventional GMM-MRCoHOG features using different basis functions for each block 11, and the thin line shows the MRCoHOG features using a histogram. The closer the curve is to the upper left corner, the better the accuracy.

As shown in the graph, when k=45, the recognition accuracy is slightly lower when the GMM-MRCoHOG features with unified basis functions are used than when the conventional GMM-MRCoHOG features are used, but it is still higher in recognition accuracy than when the conventional MRCoHOG features are used, and is fully practical.
When k=32 and 15, the recognition accuracy is slightly lower when the GMM-MRCoHOG features with a unified basis function are used than when k=45. However, the recognition accuracy is higher than when the conventional MRCoHOG features are used, and the recognition accuracy is sufficient for practical use.

Although the present embodiment has been described above, various modifications are possible.
For example, in this embodiment, positive and negative images are used, but the basis functions can also be created using only positive images.
In addition, in this embodiment, the JS divergence is used as a metric between probability distributions, but other metrics can also be used.

As described above, according to this embodiment, the amount of memory used can be significantly reduced by unifying the basis functions used in each block into one.
In addition, since the basis functions are common to each block, when calculating the features of adjacent rectangular regions, the features that have already been calculated can be used, making it possible to reduce calculation costs.
Although there is concern that the accuracy may decrease due to the unification of basis functions, it is possible to maintain accuracy while suppressing the use of computational resources by automatically determining the number of mixtures using a measure based on the Akaike Information Criterion. If you want to improve accuracy further, you can simply increase the number of mixtures.
This allows the device to be installed in devices that do not have high-performance arithmetic processing, such as FPGAs, small computers, or general-purpose computing on graphics processing units (GPGPUs), while maintaining high discrimination capabilities.

5 Pixel of interest 8 Image processing device 10 Positive image 11 Block 13, 23, 35, 36 Feature space 15 High resolution image 16 Medium resolution image 17 Low resolution image 20 Negative image 33 JS information amount 37 Basis function 40 Image 41 Discrimination filter 45 Region 81 CPU
82 ROM
83 RAM
84 Storage device 85 Storage medium drive device 86 Input unit 87 Output unit

Claims (11)

An image acquisition means for acquiring images for image recognition training;
A partitioning means for partitioning the acquired image into a plurality of blocks;
a frequency distribution acquiring means for acquiring a frequency distribution of co-occurrence of luminance gradient directions for each of the divided blocks;
a unifying means for unifying the acquired frequency distributions for each block into a single frequency distribution;
a basis function generating means for generating a basis function serving as a criterion for image recognition based on the unified frequency distribution;
13. An image processing device comprising: the unifying means performs the unification by superimposing the frequency distributions in the plurality of blocks.
2. The image processing device according to claim 1, the unifying means generates a sample based on the acquired frequency distribution for each block, and adds up the generated samples across the plurality of blocks, thereby superimposing the frequency distributions in the plurality of blocks.
3. The image processing device according to claim 2. the frequency distribution acquisition means acquires a frequency distribution of co-occurrence of brightness gradient directions between different resolutions of the same image;
4. The image processing device according to claim 1, 2 or 3. The image acquisition means acquires a plurality of images;
the unifying means unifies frequency distributions for each block of the plurality of images into a single frequency distribution.
5. The image processing device according to claim 1, wherein the image processing device further comprises: a first input unit; The image acquisition means acquires a recognition target image including a recognition target and a non-recognition target image including no recognition target,
the frequency distribution acquisition means acquires a frequency distribution in a corresponding block based on a difference between a frequency distribution of a luminance gradient direction in the corresponding block of the recognition target image and the non-recognition target image,
6. An image processing device according to any one of claims 1 to 5. The basis function is a probability density function based on a Gaussian mixture model, and a determination means is provided for determining an appropriate number of mixtures based on a balance between likelihood and number of mixtures.
7. The image processing device according to claim 1, wherein the image processing device further comprises: a first input section; a weighting factor is set for each of the plurality of blocks when integrating the frequency distributions, and the unifying means integrates the frequency distributions of each of the plurality of blocks in accordance with the weighting factor;
8. The image processing device according to claim 1, wherein the image processing device further comprises: A basis function acquisition means for acquiring the basis function according to any one of claims 1 to 8;
An image acquisition means for acquiring an image related to image recognition;
A partitioning means for partitioning the acquired image into blocks;
a feature acquisition means for applying the acquired basis function to each of the divided blocks and acquiring a feature for the basis function;
a determining means for determining whether or not a predetermined image recognition target is included in the acquired image by using the feature amount acquired from each block;
An image recognition device comprising: An image acquisition function to acquire images for image recognition training;
A segmentation function for segmenting the acquired image into a plurality of blocks;
a frequency distribution acquisition function for acquiring a frequency distribution of co-occurrence of luminance gradient directions for each of the divided blocks;
A unification function for unifying the frequency distributions for each block obtained by integrating them into a single frequency distribution;
a basis function generating function for generating a basis function serving as a criterion for image recognition based on the unified frequency distribution;
An image processing program that realizes this on a computer. A basis function acquisition function for acquiring the basis function according to any one of claims 1 to 8;
An image acquisition function for acquiring an image related to image recognition;
A segmentation function for segmenting the acquired image into blocks;
a feature acquisition function that applies the acquired basis function to each of the divided blocks and acquires a feature for the basis function;
a determination function for determining whether or not a predetermined image recognition target is included in the acquired image by using the feature amount acquired from each block; and
An image recognition program that realizes this on a computer.

Images