JP7626437B2 - Image Processing Device - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

The template matching technique is a method for searching for a partial image similar to a smaller query image from a larger input image. In the template matching technique, the query image is slid from the top left to the bottom right of the input image, and the similarity between the query image and the image of the part that overlaps the query image is calculated.

On the other hand, in the field of general object detection, a technique has been proposed or put into practical use that uses a convolutional neural network (CNN) to detect specific classes such as people and cars from an image. For example, Patent Document 1 proposes a method of identifying all local regions in an image of interest using a CNN while sliding the local region in small steps. In addition, a deep CNN has been proposed or put into practical use that simultaneously detects regions that are likely to be objects in an image and identifies the classes of the regions found. YOLO (You Look Only Once) and SSD (Single Shot MultiBox Detector) are examples of such deep CNNs.

JP 2020-160921 A

The above-mentioned template matching method is difficult to speed up, and a scale for evaluating the similarity between images must be determined in advance. By using the technology described in Patent Document 1 and deep CNNs such as YOLO and SSD, it is possible to speed up the process up to object detection. However, it is necessary to separately check whether the detected object is similar to the object in the query image. For example, when checking whether a specific person in the query image exists in an input image containing many people, it is necessary to calculate the similarity between the image of each person detected using the technology described in Patent Document 1 or a deep CNN such as YOLO or SSD and the image of the person in the query image to check whether the person images are images of the same person. Therefore, a scale for evaluating the similarity between images is still required, and it is difficult to speed up the process up to obtaining the final result.

The object of the present invention is to provide an image processing device that solves the above-mentioned problems, namely, the difficulty of increasing speed and the need to determine in advance a scale for evaluating the similarity between images.

An image processing device according to one aspect of the present invention includes:
An image processing device that detects a partial image similar to a query image from an input image,
a neural network that has been machine-learned using training data so as to output an estimated position of the partial image similar to the query image in an input image when a query image and an input image having a partial image similar to the query image are input in association with each other;
a processing means for inputting an input image and a query image in association with each other to the trained neural network, and calculating a position of a partial image in the input image that is similar to the query image;
The device is configured to include:

Further, an image processing method according to an aspect of the present invention includes:
An image processing method for detecting a partial image similar to a query image from an input image, comprising:
performing machine learning of a neural network using training data so as to output an estimated position of the partial image similar to the query image in the input image when a query image and an input image having a partial image similar to the query image are input in association with each other;
An input image and a query image are input to the trained neural network in association with each other, and a position of a partial image in the input image that is similar to the query image is calculated.
It is structured as follows.

In addition, a program according to one embodiment of the present invention includes:
A computer detects a partial image similar to a query image from an input image.
A process of performing machine learning of a neural network using training data so as to output an estimated position of a partial image similar to the query image in an input image when a query image and an input image having a partial image similar to the query image are input in association with each other;
A process of inputting an input image and a query image in association with each other to the trained neural network, and calculating a position of a partial image in the input image that is similar to the query image;
The present invention is configured to cause the

By having the above-mentioned configuration, the present invention can quickly search for partial images similar to a query image from within an input image, and also eliminates the need to predetermine a scale for evaluating the similarity between images.

1 is a block diagram of an image processing apparatus according to a first embodiment of the present invention. FIG. 11 is a block diagram of an image processing apparatus according to a second embodiment of the present invention. FIG. 11 is a schematic diagram showing an overview of machine learning processing of a neural network in a second embodiment of the present invention. 10 is a flowchart showing an example of the operation of a processing unit in an image processing device according to a second embodiment of the present invention. FIG. 11 is a diagram illustrating an example of the configuration of a neural network in an image processing device according to a second embodiment of the present invention. FIG. 11 is a schematic diagram showing the connection relationship between a first convolution layer, a second convolution layer, and a third convolution layer of a neural network in an image processing device according to a second embodiment of the present invention.

Next, an embodiment of the present invention will be described in detail with reference to the drawings.
[First embodiment]
First, an image processing apparatus according to a first embodiment of the present invention will be described. In this embodiment, an overview of the image processing apparatus of the present invention will be described.

Figure 1 is a block diagram of an image processing device 10 according to a first embodiment of the present invention. The image processing device 10 according to this embodiment is configured to detect a partial image similar to a query image from an input image. The image processing device 10 includes a trained neural network 11 and a processing means 12.

The neural network 11 undergoes machine learning processing using training data so that when a query image and an input image having a partial image similar to the query image are input in association with each other, the neural network 11 outputs an estimated position in the input image of the partial image similar to the query image.

The processing means 12 inputs the input image and the query image to the neural network 11 in association with each other, and calculates the position in the input image of a partial image that is similar to the query image.

In this manner, in the image processing device 10 according to this embodiment, when a query image and an input image having a partial image similar to the query image are input in association with each other, the neural network 11 performs machine learning processing using training data so as to output an estimated position in the input image of the partial image similar to the query image. Then, the processing means 12 inputs the input image and the query image in association with each other to the neural network 11, and calculates the position in the input image of the partial image similar to the query image. Therefore, according to the image processing device 10, it is possible to quickly search for a partial image similar to the query image from within the input image, and it is also not necessary to determine in advance a scale for evaluating the similarity between images.

Second Embodiment
Next, an image processing apparatus according to a second embodiment of the present invention will be described. In this embodiment, the image processing apparatus of the present invention will be specifically described.

Figure 2 is a block diagram of an image processing device 20 according to a second embodiment of the present invention. The image processing device 20 according to this embodiment is configured to detect whether or not a specific person appearing in a query image is present in an input image containing many people. The image processing device 20 includes an image server 21, a communication interface (I/F) unit 22, an operation input unit 23, a screen display unit 24, a memory unit 25, and a calculation processing unit 26.

The camera 27 is an imaging device installed on the street or indoors where many people pass by for the purpose of crime prevention, surveillance, etc. The camera 27 may capture the same or different imaging areas from a fixed location with a fixed imaging direction. Alternatively, the camera 27 may be mounted on a moving body such as a car and capture the same or different imaging areas while moving. The number of cameras 27 is arbitrary. There may be one camera 27, or there may be two or more cameras 27 that capture different imaging areas. The camera 27 may be, for example, a color camera equipped with a CCD (Charge-Coupled Device) image sensor or a CMOS (Complementary MOS) image sensor with a pixel capacity of several million pixels, and that captures images continuously at a constant frame rate.

The image server 21 is connected to the camera 27 by wire or wirelessly, and stores video (video) consisting of a time series of images captured by the camera 27 for a certain period of time. The image server 21 stores images captured by the camera 27 in units of frame images, for example. The image server 21 also stores each frame image in association with a camera ID that uniquely identifies the camera 27 that captured the frame image and the capture time.

The communication I/F unit 22 is composed of a data communication circuit, and is configured to perform data communication with an external device wirelessly or via a wired connection. The operation input unit 23 is composed of devices such as a keyboard and a mouse, and is configured to detect the operation of an operator and output the operation to the calculation processing unit 26. The screen display unit 24 is composed of devices such as an LCD (Liquid Crystal Display), and is configured to display various information on the screen in response to instructions from the calculation processing unit 26.

The storage unit 25 is composed of storage devices such as a hard disk and memory, and is configured to store processing information and programs 251 required for various processes in the arithmetic processing unit 26. The programs 251 are programs that are loaded into the arithmetic processing unit 26 and executed to realize various processing units, and are loaded in advance from an external device or recording medium (not shown) via a data input/output function such as the communication I/F unit 22 and stored in the storage unit 25. The main processing information stored in the storage unit 25 includes an image DB (database) 252 and a neural network 253.

The image DB 252 is a database that stores images from the camera 27 acquired through the image server 21. The neural network 253 is configured in the same manner as the neural network 11 described with reference to FIG. 1.

The calculation processing unit 26 has a processor such as an MPU and its peripheral circuits, and is configured to read and execute a program 251 from the storage unit 25, thereby implementing various processing functions through cooperation between the above hardware and the program 251. The main processing functions implemented by the calculation processing unit 26 are an image acquisition means 261, a learning means 262, and a processing means 263.

The image acquisition means 261 is configured to acquire images captured by the camera 27 from the image server 21 and store them in the image DB 252.

The learning means 262 is a means for performing machine learning on the neural network 253. The learning means 262 performs machine learning processing on the neural network 253 using training data so that, when a query image and an input image having a partial image similar to the query image are input in association with each other, the learning means 262 outputs an estimated position in the input image of the partial image similar to the query image.

The processing means 263 is configured similarly to the processing means 12 described with reference to FIG. 1.

Next, the operation of the image processing device 20 will be described. The operation of the image processing device 20 is broadly divided into an image collection operation, a learning operation, and a search operation. The image collection operation is performed by the image acquisition means 261. The learning operation is performed by the learning means 262. The search operation is performed by the processing means 263. Each operation will be described below.

<Image collection operation>
The image acquisition means 261 periodically acquires the latest images from the image server 21 that stores images captured by the camera 27, and executes a process of adding the images to the image DB 252. The above-mentioned process by the image acquisition means 261 is executed, for example, before the learning operation and the search operation are started, and is executed continuously during the learning operation and the search operation.

<Learning Operation>
The learning means 262 has a function of generating training data to be used for learning, and a function of machine learning the neural network 253 using the generated training data.

In generating the teacher data, the learning means 262 creates a large number of teacher data using, for example, a part of the image stored in the image DB 252. One teacher data is composed of one frame image (hereinafter referred to as a large image) of the camera 27 in which one or more people are captured, one image (hereinafter referred to as a small image) in which one person is captured, and rectangular information representing the position on the large image of a person that matches the person captured in the small image. Here, the size (number of pixels) of the large image is a fixed value of H pixels in height and W pixels in width. Also, the size (number of pixels) of the small image is a fixed value of h pixels in height and w pixels in width. h<H, w<W. The rectangular information may be, for example, the coordinate values (pixel positions) of the four vertices of the circumscribing rectangle of the matching person. The learning means 262 may create the teacher data as described above by interactive processing via the operation input unit 23 and the screen display unit 24. Also, the learning means 262 may create the teacher data using various annotation tools. In addition to creating training data from images stored in image DB252, training data created elsewhere may also be used.

In the machine learning of the neural network 253, the learning means 262 inputs the large image and the small image constituting the teacher data to the neural network 253 in association with each other as shown in the schematic diagram of FIG. 3 ((1) of FIG. 3). When the large image and the small image are input in association with each other, the neural network 253 outputs the estimated position in the large image of a partial image similar to the small image and its probability (score) ((2) of FIG. 3). When the actual output is obtained from the neural network 253, the learning means 262 compares it with the rectangular information constituting the teacher data and evaluates how different the actual output is by using an error function ((3) of FIG. 3). Then, the learning means 262 updates the weight value of the neural network 253 in a direction that reduces the error function, for example, by using the error backpropagation method ((4) of FIG. 3). The learning means 262 repeats the above process using a lot of teacher data, for example, until the error function becomes sufficiently small.

<Search operation>
FIG. 4 is a flowchart showing an example of the operation of the processing unit 263. Referring to FIG. 4, the processing unit 263 first acquires a query image (step S1). For example, the processing unit 263 may acquire a partial image cut out with specified rectangular information from an image uniquely determined by a camera ID and an image capture time specified by an operator or an external device among images stored in the image DB 252 as a query image. Alternatively, the processing unit 263 may directly acquire a query image from an operator through the operation input unit 23 or the screen display unit 24. Alternatively, the processing unit 263 may acquire a query image from an external device through the communication I/F unit 22. Here, the size of the query image may be the same as the size of the small image constituting the teacher data.

Next, the processing means 263 acquires an input image (step S2). For example, the processing means 263 may acquire as an input image, from among the images stored in the image DB 252, an image that is uniquely determined by a camera ID and an image capture time specified by an operator or an external device. Alternatively, the processing means 263 may acquire the input image directly from the operator via the operation input unit 23 or the screen display unit 24. Alternatively, the processing means 263 may acquire the input image from an external device via the communication I/F unit 22. Here, the size of the input image may be the same as the size of the large image that constitutes the teacher data.

Next, the processing means 263 uses the trained neural network 253 to calculate the position in the input image of the partial image similar to the query image (step S3). Specifically, the processing means 263 first inputs the query image and the input image to the trained neural network 253 in association with each other, and obtains one or more pairs of the estimated position in the input image of the partial image similar to the query image and the score output from the neural network 253. Next, the processing means 263 determines whether the maximum score of the one or more pairs is equal to or greater than a predetermined threshold. Next, if the maximum score is not equal to or greater than the threshold, the processing means 263 generates a search result indicating that there is no partial image similar to the query image in the input image. Furthermore, if the maximum score is equal to or greater than the threshold, the processing means 263 generates the estimated position and score related to the maximum score as the search result.

Next, the processing means 263 displays the generated search results on the screen display unit 24 and/or outputs them to an external device via the communication I/F unit 22 (step S4).

Next, we will explain an example configuration of the neural network 253.

Figure 5 is a diagram showing an example of the configuration of a neural network 253. The neural network 253 in this example includes a first input layer 31, a second input layer 32, a first convolution layer 33, a second convolution layer 34, a third convolution layer 35, an intermediate layer 36, and an output layer 37.

The first input layer 31 and the second input layer 32 constitute the input layer of the neural network 253. Of these, the first input layer 31 is a layer that propagates pixel information of the input image (large image) directly to the first convolution layer 33, and is composed of neurons whose number corresponds to the size of the input image. The second input layer 32 is a layer that propagates pixel information of the query image (small image) directly to the second convolution layer 34, and is composed of neurons whose number corresponds to the size of the query image.

The first convolution layer 33 and the second convolution layer 34 constitute the first intermediate layer of the neural network 253. Of these, the first convolution layer 33 is a layer that extracts features (feature map) of the input image input to the first input layer 31. The first convolution layer 33 is composed of a number of neurons according to the size of the input image and the size of the local region (local receptive field) where convolution is performed. The size of the local region where convolution is performed may be, for example, 3×3 neurons, but is not limited to this. In addition, the output value of each neuron of the first convolution layer 33 is calculated based on the result of convolution calculation of the output values of multiple neurons in the corresponding local region of the first input layer 31 and 3×3 weights (weight matrix) for the first convolution layer 33, and the bias and activation function for the first convolution layer 33.

The second convolutional layer 34 is a layer that extracts features (feature map) of the query image input to the second input layer 32. The second convolutional layer 34 is composed of a number of neurons according to the size of the query image and the size of the local region in which convolution is performed. The size of the local region in which convolution is performed may be, for example, 3×3 neurons, but is not limited to this. The output value of each neuron in the second convolutional layer 34 is calculated based on the result of a convolution operation of the output values of multiple neurons in the corresponding local region of the second input layer 32 with 3×3 weights (weight matrix) for the second convolutional layer 34, and the bias and activation function for the second convolutional layer 34.

The third convolution layer 35 is a layer that extracts features that combine the features of the input image extracted by the first convolution layer 33 and the features of the query image extracted by the second convolution layer 34. Specifically, the third convolution layer 35 is composed of a number of neurons according to the size of the first convolution layer 33 and the size of the local region of the first convolution layer 33 where convolution is performed. The size of the local region where convolution is performed may be, for example, 3 × 3 neurons, but is not limited to this. In addition, the second convolution layer 34 and the third convolution layer 35 are fully connected. Therefore, the output value of each neuron in the third convolution layer 35 is calculated based on the result of a convolution operation between the output values of multiple neurons in the corresponding local region in the first convolution layer 33 and 3 × 3 weights (first weight matrix) for the third convolution layer 35, the result of a convolution operation between the output values of all neurons in the second convolution layer 34 and 3 × 2 weights (second weight matrix) for the third convolution layer 35, and the bias and activation function for the third convolution layer 35.

Figure 6 is a schematic diagram showing the connection relationship between the first convolution layer 33, the second convolution layer 34, and the third convolution layer 35. For convenience of explanation, the size of the first convolution layer 33 is 9 neurons vertically x 6 neurons horizontally, the size of the second convolution layer 34 is 3 neurons vertically x 2 neurons horizontally, and the size of the third convolution layer 35 is 7 neurons vertically x 4 neurons horizontally. Then, for example, the output of a 3 x 3 neuron whose upper left vertex is the second neuron 352 from the upper left corner of the first convolution layer 33, and the output of all the neurons of the second convolution layer 34, that is, the 3 x 2 neurons, are input to the second neuron 351 from the upper left corner of the third convolution layer 35. Neuron 351 calculates the result of a convolution operation between the output of the 3×3 neuron in the first convolution layer 33 and 3×3 weights (first weight matrix), and the result of a convolution operation between the output of the 3×2 neuron in the second convolution layer 34 and 3×2 weights (second weight matrix). Neuron 351 also calculates an output value of neuron 351 based on the sum of the two convolution operation results and a predetermined bias and activation function.

Here, all the weights mentioned above can be positive or negative.

Referring again to FIG. 5, the third convolution layer 35 is followed by other intermediate layers 36, and the final stage is an output layer 37. That is, the output value of each neuron of the third convolution layer 35 becomes the input value of the intermediate layer 36. The intermediate layer 36 and the output layer 37 function to estimate and output the position of a partial image similar to the query image in the input image based on a feature obtained by combining the features of the input image extracted by the third convolution layer 35 and the features of the query image. The intermediate/output layer 38 composed of the intermediate layer 36 and the output layer 37 may be composed of a deep CNN (Deep CNN) that simultaneously performs, for example, detection to find an object-like region in the image and classification to determine the class of the found region. Examples of such deep CNN include YOLO and SSD. When YOLO or SSD is used, the output value of each neuron of the third convolution layer 35 is treated as an input value of the input layer of YOLO or SSD.

The above is an example of the configuration of the neural network 253. However, the neural network 253 is not limited to the example of the configuration described above. For example, a fourth convolution layer may be used instead of the first convolution layer 33, the second convolution layer 34, and the third convolution layer 35. This fourth convolution layer is composed of, for example, a number of neurons according to the size of the first input layer 31 and the size of the local area (e.g., 3 × 3 neurons) to be convolved. In addition, the second input layer 32 and the fourth convolution layer are fully connected. Therefore, the output value of each neuron of the fourth convolution layer is calculated based on the result of convolution calculation of the output values of multiple neurons of the corresponding local area of the first input layer 31 and the weights for the local area for the fourth convolution layer, the result of convolution calculation of the output values of all neurons of the second input layer 32 and the weights for the neurons of the second input layer 32 for the fourth convolution layer, and the bias and activation function for the fourth convolution layer. Alternatively, at least one of the first convolution layer 33, the second convolution layer 34, and the third convolution layer 35 of the neural network 253 may be composed of two or more convolution layers.

As described above, in the image processing device 20 according to this embodiment, when a query image and an input image having a partial image similar to the query image are input in association with each other, the neural network 253 performs machine learning processing using training data by the learning means 262 so as to output an estimated position in the input image of the partial image similar to the query image. Then, the processing means 263 inputs the input image and the query image in association with each other to the trained neural network 253, and calculates the position in the input image of the partial image similar to the query image. Therefore, according to the image processing device 20, it is possible to quickly search for a partial image similar to the query image from within the input image, and it is also not necessary to determine in advance a scale for evaluating the similarity between images.

In addition, the neural network 253 in this embodiment has a convolution layer (the first convolution layer 33, the second convolution layer 34, and the third convolution layer 35, or the fourth convolution layer) that extracts a composite feature that combines the features of the input image input to the first input layer 31 and the features input to the second input layer 32. Therefore, the neural network 253 can extract features for each local region of the input image in terms of similarity with the query image. As a result, improved accuracy can be expected compared to a neural network that does not have a convolution layer that extracts such composite features.

Although the present invention has been described above with reference to the above-mentioned embodiments, the present invention is not limited to the above-mentioned embodiments. Various modifications that can be understood by a person skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. For example, in the above-mentioned embodiment, the present invention is applied to an image processing device that detects a partial image similar to a query image in which a specific person appears from an input image in which multiple people appear, but is not limited to this. For example, the present invention may be applied to an image processing device that detects a partial image similar to a query image in which a specific object other than a person appears from an input image in which multiple objects other than people (e.g., a car, a boat, a dog, etc.) appear.

The present invention can be applied to the field of detecting specific objects from images, and in particular to the field of detecting partial images from a large image that resemble a smaller image.

10 Image processing device 11 Neural network 12 Processing means 20 Image processing device 21 Image server 22 Communication I/F unit 23 Operation input unit 24 Screen display unit 25 Memory unit 26 Arithmetic processing unit 31 First input layer 32 Second input layer 33 First convolution layer 34 Second convolution layer 35 Third convolution layer 36 Intermediate layer 37 Output layer 38 Intermediate/output layer 251 Program 252 Image DB
253 Neural network 261 Image acquisition means 262 Learning means 263 Processing means 351, 352 Neuron

Claims (7)

An image processing device that detects a partial image similar to a query image from an input image,
a neural network that has been machine-learned using training data so as to output an estimated position of the partial image similar to the query image in an input image when a query image and an input image having a partial image similar to the query image are input in association with each other;
a processing means for inputting an input image and a query image in association with each other to the trained neural network, and calculating a position of a partial image in the input image that is similar to the query image;
Equipped with
The neural network includes a first input layer to which the input image is input, a second input layer to which the query image is input, a convolution layer that extracts a composite feature obtained by combining a feature of the input image input to the first input layer with a feature input to the second input layer, and an intermediate layer and an output layer that calculate and output the estimated position based on the composite feature extracted by the convolution layer.
Image processing device. Each neuron of the convolution layer is configured to input an output value of a neuron in a local region of the first input layer and an output value of all neurons of the second input layer.
The image processing device according to claim 1 . The intermediate layer and the output layer are composed of a deep convolutional neural network that simultaneously detects object-like regions in an image and classifies the detected regions.
3. The image processing device according to claim 1 or 2 . The intermediate layer and the output layer are configured by YOLO (You Only Look Once).
4. An image processing device according to claim 1 . The intermediate layer and the output layer are composed of SSDs (Single Shot Multibox Detectors).
4. An image processing device according to claim 1 . An image processing method for detecting a partial image similar to a query image from an input image, comprising:
performing machine learning of a neural network including a first input layer to which an input image is input, a second input layer to which the query image is input, a convolution layer that extracts a composite feature obtained by combining a feature of the input image input to the first input layer with a feature input to the second input layer, and an intermediate layer and an output layer that calculate and output the estimated position based on the composite feature extracted by the convolution layer, when a query image and an input image having a partial image similar to the query image are input in association with each other, using teacher data to output an estimated position of the partial image similar to the query image in the input image ;
An input image and a query image are input to the trained neural network in association with each other, and a position of a partial image in the input image that is similar to the query image is calculated.
Image processing methods. A computer detects a partial image similar to a query image from an input image.
a process of performing machine learning on a neural network, the process including: a first input layer to which an input image is input using teacher data; a second input layer to which the query image is input; a convolution layer that extracts a composite feature obtained by combining a feature of the input image input to the first input layer with a feature input to the second input layer; and an intermediate layer and an output layer that calculate and output the estimated position based on the composite feature extracted by the convolution layer, so as to output an estimated position in the input image of a partial image similar to the query image when the query image and the input image having the partial image similar to the query image are input in association with each other;
A process of inputting an input image and a query image in association with each other to the trained neural network, and calculating a position of a partial image in the input image that is similar to the query image;
A program to carry out the above.

Images