JP7718268B2 - Image processing device, image processing method, and image processing program - Google Patents

Description

This disclosure relates to an image processing device, an image processing method, and an image processing program, and in particular to technology for reducing the amount of calculation required for convolution processing.

In recent years, as computer processing power has improved, the processing time required for multi-layered neural networks (DNNs: Deep Neural Networks) has decreased, leading to widespread use of DNNs. As a result, there is a growing demand for DNNs to process ever larger amounts of data, one example of which is the field of video recognition.

Since video data is three-dimensional data with a time axis added to two-dimensional images, 3DCNN (3-Dimensional CNN), an extension of 2DCNN (2-Dimensional Convolutional Neural Network), is known as a DNN for processing video data. Because video data contains more data than still image data, 3DCNN requires a large amount of calculation, which leads to problems such as high memory usage and long calculation times.

As shown in Figure 11, in a 2DCNN, the number of weighting factors W by which the input value X is multiplied is equal to the square of the kernel size of the kernel used in the convolution process ("3" in Figure 11) multiplied by the number of input channels ("2" in Figure 11) and the number of output channels ("2" in Figure 11). Therefore, the number of multiplications in the convolution process is equal to the number of weighting factors W multiplied by the square of the value obtained by dividing the size of each input channel (input size, "6" in Figure 11) by the stride ("1" in Figure 11).

Here, Stride is the interval at which the kernel is multiplied against the input. For example, if Stride is 1, the kernel is multiplied against the input, shifting it by 1. If Stride is 2, the kernel is multiplied against the input, shifting it by 2.

In summary, the number of multiplications in the convolution process of 2DCNN is
(kernel size) ² × (number of input channels) × (number of output channels) × (input channel size/stride) ² … (1)
Similarly, the number of multiplications in the convolution process of 3DCNN is
(kernel size) ³ × (number of input channels) × (number of output channels) × (input channel size/stride) ² × (number of input frames/stride) ... (2)
Dividing equation (2) by equation (1) gives
(kernel size) ³ × (number of input frames/stride) ... (3)
For example, when the kernel size is 3, the number of input frames is 16, and the stride is 1, the number of multiplications in the convolution process for 3DCNN is 3 ² ×(16/1) ² =48 times that of 2DCNN.

To address this issue, the amount of calculation required by the 3DCNN can be reduced by lowering the resolution of the input image, but this inevitably leads to a decrease in recognition accuracy when performing video image recognition, etc.

One method for reducing the amount of computation required for 3DCNN without reducing the resolution of the input image focuses on the characteristics of activation functions such as ReLU (Rectified Linear Unit).

In other words, because activation functions such as ReLU output zero when the input value is negative, neural networks using such activation functions can be said to have sparse activation, in the sense that neurons that output positive values are sparse.

By focusing on this sparsity, the amount of computation required by the neural network can be reduced by omitting convolution processing in zero regions where there are no neurons that output positive values, and performing convolution processing only in non-zero regions where neurons output positive values.

Specifically, as shown in Figure 12, non-zero region 1201 of the output of layer n-1 of neural network Y is used as the input to layer n, and convolution processing is performed only on non-zero region 1201, outputting the calculation result. Non-zero region 1202 of the output of layer n becomes the input to layer n+1.

Therefore, the convolution processing of the nth layer of neural network Y only needs to be performed on the non-zero region 1201 of the output of the n-1th layer, and similarly, the convolution processing of the n+1th layer only needs to be performed on the non-zero region 1202 of the output of the nth layer, thereby reducing the number of times the convolution processing is performed.

This conventional technology can be applied not only to 3DCNNs but also to a wide variety of neural networks (see, for example, Non-Patent Document 1).

Mengye Ren, Andrei Pokrovsky, Bin Yang, Raquel Urtasun, "SBNet: Sparse Blocks Network for Fast Inference", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8711-8720. Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", arXiv:1506.02640v5 [cs.CV] 9 May 2016.

When considering applying this type of conventional technology to the field of image processing, the image data input to the first layer of a neural network is often non-zero overall, making it impossible to reduce the convolution process as is. For this reason, the above conventional technology uses non-zero regions that are defined in advance based on the characteristics of the image.

For example, when detecting LiDAR (Light Detection and Ranging) objects, a map mask 1301 of roads and sidewalks corresponding to the input data 13 can be prepared, as shown in Figure 13. In this way, the background areas outside the map mask 1301 in the input data 13 can be treated as zero areas and excluded from interrupt processing, thereby reducing the amount of calculation required by the neural network.

However, when detecting vehicles, the range in which vehicles move is limited to roads, so areas other than roads can be excluded as zero areas. However, if the range in which an image object of interest in an image can move extends across the entire image, it is not possible to define a mask (zero area) as in the above-mentioned conventional technology.

For the same reason, convolution processing cannot be effectively reduced even when the range in which an image object can move is not the entire image, but rather is widespread within the image.

The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, image processing method, and image processing program that can reduce the amount of calculation required in image processing using a neural network, even when the movement range of an image object cannot be specified in advance.

In order to achieve the above object, an image processing device according to one embodiment of the present disclosure is an image processing device that performs image processing using a CNN that receives as input a feature space constituted by feature sequences extracted from an input image, and includes an image object region extraction unit that extracts only a portion of a predetermined image object in the input image as an image object region, and a feature sequence extraction unit that extracts a feature sequence for each image object region, and the CNN is characterized in that it treats regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to these non-zero regions, and makes them the subject of convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing.

In this case, the image object region may be a rectangular region that includes the portion of the image object, and the non-zero region may be associated with the image object region depending on the position and size of the rectangular region in the input image.

Alternatively, the image object may be a person, and the CNN may perform image processing to recognize the person's behavior.

Alternatively, the image object may be a person or an object, and the CNN may perform image processing to recognize the scene in the input image.
4. The image processing device according to claim 1, wherein the image processing device is a computer.

Furthermore, the specified image object may be an image object that is the subject of image processing using a CNN.

The feature sequence extraction unit may also extract feature sequences using a convolutional neural network, a multilayer perceptron, a neural network other than a convolutional neural network or a multilayer perceptron, principal component analysis, or a predefined feature calculation method.

An image processing device according to another embodiment of the present disclosure is an image processing device that performs image processing using a CNN that receives as input a feature space constituted by feature sequences extracted from an input image, and includes an image object region extraction unit that extracts image object regions that include at least a portion of a predetermined image object in the input image, and a feature sequence extraction unit that extracts a feature sequence for each image object region, wherein the CNN treats regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them the subject of convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing, and the CNN is an SBnet.

An image processing device according to another aspect of the present disclosure performs image processing using a CNN that receives as input a feature space composed of feature sequences extracted from an input video sequence. The image processing device includes: an image object region extraction unit that extracts image object regions in the input video sequence that include at least a portion of a predetermined image object and that are common to frames that include the image object; and a feature sequence extraction unit that extracts a feature sequence for each image object region. The CNN performs sparse convolution processing by treating regions in the feature space that correspond to the image object regions as non-zero regions, assigning the feature sequences extracted for each image object region to the non-zero regions, and making them the target of convolution processing, and excluding zero regions other than the non-zero regions from the target of convolution processing.

In this case, the feature array extraction unit may extract a feature array for each image object region for each frame.

The image object region extraction unit may also extract a frame-specific region that includes at least a portion of a specified image object for each frame that constitutes the input video, and the image object region may be a sum region of frame-specific regions that include at least a portion of an image object that is common between frames.

The image object region extraction unit may also extract a rectangular frame-specific region that includes at least a portion of a specified image object for each frame that constitutes the input video, and the image object region may be the smallest rectangular region among rectangular regions that include a union of frame-specific regions that include at least a portion of an image object that is common between frames.

The image object region extraction unit may also include a stillness determination unit that determines whether each image object is still, extracts a frame-specific region that includes at least a portion of a specified image object for each frame constituting the input video, obtains an extraction likelihood for each frame-specific region, and determines the frame-specific region with the highest extraction likelihood, among the frame-specific regions related to a common image object, as the image object region related to that image object.

In addition, for multiple image object regions that relate to a common image object and that relate to a stationary image object, the feature sequence extraction unit may extract feature sequences related to any one of the image object regions, thereby extracting feature sequences related to the other image object regions.

In addition, the image object region extraction unit may obtain, for each frame region, the class to which the image object related to that frame region belongs, and the stillness determination unit may determine whether the image object is still based on the class to which the image object belongs.

The stillness determination unit may also determine that an image object is still when frame-specific regions relating to the common image object are located at the same position within a frame.

Alternatively, the image object may be a person, and the CNN may perform image processing to recognize the person's behavior.

Alternatively, the image object may be a person or object, and the CNN may perform image processing to recognize scenes in the input video.

Furthermore, the specified image object may be an image object that is the subject of image processing using a CNN.

Furthermore, when a still image object identical to a still image object included in one input image is included in another input video, the CNN may set the area in the feature space corresponding to the image object area relating to the same still image object included in the other input image as a zero area and exclude it from the target of convolution processing, and the image processing result for the still image object included in the one input video, among the image processing results by the CNN for the one input video, may be the image processing result for the same still image object included in the other input image.

The CNN may also be a 3D CNN.

CNN may also be SBnet.

The feature sequence extraction unit may also extract feature sequences using a convolutional neural network, a multilayer perceptron, a neural network other than a CNN or multilayer perceptron, principal component analysis, or a predefined feature calculation method.

An image processing method according to one embodiment of the present disclosure is an image processing method executed by an image processing device that executes image processing using a CNN that receives as input a feature space constituted by feature sequences extracted from an input image, and includes an image object region extraction step of extracting only a portion of a predetermined image object in the input image as an image object region, and a feature sequence extraction step of extracting a feature sequence for each image object region, wherein the CNN defines regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to these non-zero regions, and makes them the subject of convolution processing, and executes sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing.

An image processing method according to another aspect of the present disclosure is an image processing method executed by an image processing device that performs video processing using a CNN that receives as input a feature space constituted by feature sequences extracted from input video, and includes an image object region extraction step that extracts image object regions in the input video that include at least a portion of a predetermined image object and are common to frames that include the image object, and a feature sequence extraction step that extracts a feature sequence for each image object region, wherein the CNN treats regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them the target of convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from the target of convolution processing.

An image processing program according to one embodiment of the present disclosure is an image processing program executed by a computer that executes image processing using a CNN that receives as input a feature space constituted by feature sequences extracted from an input image, and causes the computer to execute an image object region extraction step of extracting only a portion of a predetermined image object in the input image as an image object region, and a feature sequence extraction step of extracting a feature sequence for each image object region, wherein the CNN sets regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to these non-zero regions, and makes them the subject of convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing.

An image processing program according to another embodiment of the present disclosure is an image processing program executed by a computer that executes video processing using a CNN that receives as input a feature space constituted by feature sequences extracted from input video, and causes the computer to execute an image object region extraction step of extracting image object regions in the input video that include at least a portion of a predetermined image object and that are common to frames that include the image object, and a feature sequence extraction step of extracting a feature sequence for each image object region, wherein the CNN sets regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them the subject of convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing.

In this way, non-zero regions in feature space are set using image object regions that contain at least a portion of a specified image object in the input image, and convolution processing is performed, while zero regions other than these non-zero regions are excluded from the convolution processing.This means that even if it is not possible to set non-zero regions in advance due to the characteristics of the image object, sparse convolution processing can be performed, thereby reducing the amount of calculation required by the CNN.

1 is a diagram illustrating an example of an operating environment of an image processing device 1 according to a first embodiment of the present disclosure. 1A is a block diagram showing the main hardware configuration of the image processing device 1, and FIG. 1B is a block diagram showing the functional configuration of the image processing device 1. FIG. 10 is a schematic diagram illustrating the process of extracting image object regions r1 and r2 from an image 100, and then extracting feature arrays v1 and v2 to generate a feature space S. FIG. 1A is a diagram illustrating a binary mask that sets non-zero and zero regions in the feature space S, and FIG. 1B is a diagram illustrating an index list for simplifying the binary mask in FIG. 1A and inputting it into SBnet. 1 is a flowchart illustrating processing for generating a feature space S to be input to a CNN by an image processing apparatus 1 according to a first embodiment. FIG. 10 is a block diagram showing a functional configuration of an image processing device 1 according to a second embodiment. This is a schematic diagram that explains the process from extracting frame-by-frame regions from a moving image 700 to extracting an image object region by determining whether the image object is a stationary object or tracking a moving image object. FIG. 10 is a schematic diagram illustrating the process of extracting a feature array for each image object region for each frame and generating a feature space S for each frame. 10 is a flowchart illustrating processing for generating a feature space S to be input to a 3D CNN by an image processing apparatus 1 according to a second embodiment. 10A and 10B are schematic diagrams illustrating a process for reducing the amount of calculation for obtaining spatiotemporal features of a stationary object according to a modified example of the present disclosure. FIG. 1 is a diagram illustrating the amount of calculation required for a typical CNN. FIG. 10 is a diagram illustrating a configuration for reducing the amount of calculation used in SBnet. 10A and 10B are diagrams illustrating an example of an application in which the amount of calculation required to detect a moving object is reduced using SBnet.

Hereinafter, embodiments of an image processing device, an image processing method, and an image processing program according to the present disclosure will be described with reference to the drawings.
[1] First Embodiment An image processing device according to this embodiment processes individual frames that make up still images and moving images.
(1-1) Operating Environment of Image Processing Device As shown in Fig. 1, an image processing device 1 according to this embodiment is connected to a video camera 110 via a communication network 120. The video camera 110 is capable of capturing moving images made up of a plurality of frames, as well as still images. Hereinafter, still images and individual frames constituting moving images will be collectively referred to simply as "images."

The image processing device 1 acquires an image of a person s1 and an object (a "chair" in the example of Figure 1) s2 from the video camera 110. Figure 1 shows an image 100 acquired by the image processing device 1 from the video camera 110. The image 100 includes an image object o1 of the person s1 and an image object o2 of the object s2.

The image processing device 1 performs image processing such as identifying a person s1 or an object s2, identifying the skeleton and behavior of the person s1, and identifying the type of scene in the image. The image processing device 1 performs this type of image processing using a convolutional neural network (CNN).

As its name suggests, CNN performs image processing such as identifying people, objects, human skeletons and behavior, and scenes depicted in images by repeating convolution processing. This convolution processing tends to require a huge amount of calculation as the image resolution increases.
(1-2) Hardware configuration of image processing device 1 The image processing device 1 is a so-called computer, and as shown in FIG. 2( a), it comprises a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, a HDD (Hard Disk Drive) 204, and a NIC (Network Interface Card) 205 connected via an internal bus 206.

When the image processing device 1 is reset, for example by being powered on, the CPU 201 reads and starts up a boot program from the ROM 202, and executes the OS (Operating System) and image processing program 207 read from the HDD 204, using the RAM 203 as a working storage area.

The HDD 204 can store application programs such as the OS and the image processing program 207, as well as data such as images 100 captured using the video camera 110 and the results of image processing by the image processing program 207.

The image processing device 1 may be a single computer, or may use multiple computers for distributed processing. It goes without saying that the number of CPUs 201 is not limited to one, and a multi-processor configuration may also be used. The hardware configuration of the image processing device 1 is not limited as long as it is a computer capable of executing the image processing program 207 according to this embodiment.

The communication network 120 may be a LAN (Local Area Network), the Internet, or a combination thereof, or may be wired communication, wireless communication, or a combination thereof. Furthermore, it may be serial communication such as USB (Universal Serial Bus), and the type of communication network is not limited as long as it can transmit and receive the image 100.
(1-3) Functional Configuration of Image Processing Device 1 By using the hardware configuration described above and executing an image processing program, the image processing device 1 realizes the functions shown in FIG. 2(b).
(1-3-1) Image object region extraction unit 211
When the image object region extraction unit 211 receives the image 100 captured using the video camera 110 , it extracts an image object region for cutting out (cropping) each image object included in the image 100 .

In the example of Figure 3, image 100 includes image object (person) o1 and image object (object) o2. The image object region extraction unit 211 extracts image object region r1 for cropping image object o1 and image object region r2 for cropping image object o2.

To extract an image object region, for example, YOLO (You Only Look Once) can be used (see Non-Patent Document 2). Using YOLO, it is possible to obtain the coordinate values (x and y coordinates of the upper left vertex on the image) and size (width w and height h of the bounding box) of a rectangular bounding box (BB) that surrounds an image object contained in image 100, and this bounding box can then be used as the image object region of that image object.

In addition, using YOLO, it is possible to obtain, for each bounding box, a confidence score indicating the likelihood of detecting that bounding box, and a class probability indicating which class the image object enclosed by the bounding box is classified into.

Note that an image object region may be extracted by detecting only a part of an image object, such as the face portion of an image object of a person included in image 100 .
(1-3-2) Characteristic sequence extraction unit 212
The feature sequence extraction unit 212 extracts feature sequences for each of the image object regions r1 and r2. These feature sequences are ultimately used as input data for the CNN 214 that performs image processing.

In conventional technology, feature sequences are extracted from the entire image 100, or from only a predetermined region on the image 100, and then input to the CNN 214. In contrast, in this embodiment, feature sequences are extracted only from image object regions, which may differ for each image 100, and then input to the CNN 214 via the feature space generation unit 213.

In the example of Figure 3, feature array v1 is extracted for image object region r1, where h1 and w1 are the height and width of image object region r1 in image 100, expressed in number of pixels, respectively. Also, c is the number of input channels. In other words, CNN 214 performs c types of convolution processing using feature array v1 as input.

Similarly, feature array v2 is extracted for image object region r2. h2 and w2 in feature array v2 are the height and width of image object region r2 in image 100, and c is the number of input channels.

The method for extracting feature sequences for image object regions may be the same as the method for extracting feature sequences for image object regions in conventional technology.

The feature sequence extraction unit 212 may extract feature sequences using a CNN other than CNN 214, or may extract feature sequences using a multi-layer perceptron (MLP). A multi-layer perceptron is a forward propagation neural network with at least three node layers.

Feature sequences may also be extracted using neural networks other than CNN and multilayer perceptrons. Furthermore, feature sequences may also be extracted using principal component analysis, or other predefined feature calculation methods may also be used to extract feature sequences.

Principal component analysis is a method of synthesizing a small number of uncorrelated variables called principal components that best represent the overall variability from a large number of correlated variables using multivariate analysis (a statistical data analysis method). For example, a principal component synthesized from a large number of pixel values that make up an image object region (in a color image, consisting of gradation values for each color component) can be used as a feature array.
(1-3-3) Feature space generation unit 213
The feature space generation unit 213 generates a feature space S to be input to the CNN 214 using information (in the YOLO example, coordinate values and size) that identifies the image object regions r1 and r2 on the image 100 extracted by the image object region extraction unit 211 and the feature arrays for each of the image object regions r1 and r2 extracted by the feature array extraction unit 212.

Specifically, in the feature space S input to the CNN 214, the regions corresponding to the image object regions r1 and r2 are set as non-zero regions, and the feature arrays v1 and v2 associated with the image object regions r1 and r2 are assigned to these non-zero regions.

This non-zero region is the same region to which the extracted feature arrays for the regions corresponding to image object regions r1 and r2 are assigned when extracting feature arrays for the entire image 100 to generate feature space S.

In Figure 11, which was used to explain the prior art, the non-zero regions 1101 and 1102 are fixed in advance, whereas in this embodiment, the non-zero regions can vary depending on the image object regions r1 and r2 in the image 100. Therefore, even if the image objects o1 and o2 can be located in an unspecified range within the image 100, the size of the non-zero regions can be minimized within the range in which the CNN 214 can perform image processing.

The feature space generation unit 213 generates information for identifying non-zero regions within the feature space S. In the example of Figure 3, h and w in the feature space S are the height and width of the image 100, respectively. Also, c is the number of input channels.

Since feature arrays v1 and v2 share the same number of input channels c as feature space S, by specifying heights h1 and h2 and widths w1 and w2, the range of non-zero regions in feature space S can be identified and feature arrays v1 and v2 can be assigned within feature space S.

Non-zero regions may be identified using a binary mask that assigns "1" to pixels in non-zero regions of image 100 and "0" to pixels in zero regions.

When the CNN 214 uses SBnet (see Non-Patent Document 1), the feature space generation unit 213 converts the binary mask into an index list to identify non-zero regions. For example, in the example of Figure 4(a), "1" is assigned to pixels in the non-zero regions (shaded areas) of the binary mask 401, and "0" is assigned to pixels in the zero regions (non-shaded areas).

In contrast, in Figure 4(a), the pixel with an x coordinate of "3" and a y coordinate of "0" is in the non-zero area, to which a value of "1" is assigned. In the example of Figure 4(b), the index list 402 has a column with an x coordinate of "3" and a y coordinate of "0." In other words, the coordinate values of pixels belonging to the non-zero area are entered in the index list 402.

The feature space generation unit 213 may identify non-zero and zero regions in the feature space S in this manner.

A feature array is assigned to each x and y coordinate in the non-zero region of the feature space S. The number of elements in the feature array is equal to the number of input channels c.
(1-3-4) CNN214
The CNN 214 performs sparse convolution processing, which convolves only non-zero regions of the feature space S and does not convolve zero regions. Therefore, the amount of calculation can be reduced by the amount of convolution processing for the zero regions.

Furthermore, when the image object that is the subject of the convolution process of CNN214 can move over a wide range on the image, conventional technology is forced to widen the non-zero region, which is less effective at reducing the amount of calculation required for the convolution process.

On the other hand, in this embodiment, the non-zero area can be narrowed down to the range occupied by the image object on the image, so even if the range over which the image object can move on the image is wide, the amount of calculation required for the convolution process can be reduced effectively.
(1-4) Processing of Image Processing Device 1 The image processing device 1 executes the image processing program 207 with the aim of reducing the amount of calculation required for image processing by the CNN 214. That is, as shown in Fig. 5, first, when an image 100 is acquired by receiving it from the video camera 110 or reading it from the HDD 204 (S501), the image object region extraction unit 211 extracts an image object region for each image object (S502).

Next, the feature array extraction unit 212 extracts feature arrays for each image object region in accordance with the processing content of the CNN 214 (S503).

The feature space generation unit 213 sets non-zero and zero regions in the feature space S according to the image object region (S504) and assigns the non-zero region feature array (S505). Then, image processing is performed on the CNN 124 using the assigned feature array as input to the feature space S.

After the image processing is performed by the CNN 214, for example, the processing results are stored in the HDD 204, and the processing ends.
[2] Second Embodiment Next, a second embodiment of the present disclosure will be described. The image processing device 1 according to this embodiment is a device that performs moving image processing using a 3DCNN, and while it has a configuration that is generally common to the image forming processing device 1 according to the first embodiment, it differs in the way in which the image object region is set. The following description will mainly focus on the differences.

In this specification, common reference numerals are used to designate components and the like that are common between the embodiments and the modified examples.
(2-1) Functional configuration of image processing device 1 As shown in FIG. 6, the image processing device 1 has the functions of a frame-by-frame area extraction unit 601, a stationary object determination unit 602, a tracking unit 603, an image object area extraction unit 604, a feature array extraction unit 605, a feature space generation unit 606, and a 3DCNN 607.
(2-1-1) Frame-by-frame region extraction unit 601
The frame region extraction unit 601 extracts a frame region as a region for cropping an image object for each frame that constitutes a moving image.

In the example of Figure 7, frames t1 to tn that make up the video 700 each contain image objects (people) o1(t1) to o1(tn) and image objects (objects) o2(t1) to o2(tn).

The frame-specific region extraction unit 601 extracts frame-specific regions r1(t1) to r1(tn) for cropping image objects (people) o1(t1) to o1(tn) from frames t1 to tn, and frame-specific regions r2(t1) to r2(tn) for cropping image objects (objects) o2(t1) to o2(tn).

Similar to the image object region extraction unit 211 in the first embodiment described above, the frame-specific region extraction unit 601 may also use YOLO to extract frame-specific regions r1(t1) to r1(tn) and r2(t1) to r2(tn).

Note that frame-specific regions r1(t1) to r1(tn) may be extracted by targeting only a portion of an image object, such as the face portion of a person's image object included in each of frames t1 to tn.
(2-1-2) Stationary object determination unit 602
The stationary object determination unit 602 determines whether an image object in a moving image is a stationary object or a moving object.

As mentioned above, when using YOLO to detect a bounding box surrounding an image object, it is also possible to obtain a class probability that indicates which class the image object falls into.

If this class probability is used to determine that the image object belongs to a class of stationary objects, such as a chair, the stationary object determination unit 602 determines that the image object is a stationary object.

On the other hand, if the class probability indicates that the image object belongs to the moving object class, the stationary object determination unit 602 determines that the image object is not a stationary object. Which classes belong to stationary objects and which classes belong to moving objects are set in advance.

In addition, for frame regions related to image objects o1(t1) to o1(tn) that are common between frames, such as frame region r1(t1) to r1(tn), and frame regions related to image objects o2(t1) to o2(tn) that are common between frames, such as frame region r2(t1) to r2(tn), if the frame regions related to the common image objects between frames are in the same position, the image objects may be determined to be stationary.

Furthermore, if the shooting environment using the video camera 110 is constant, it is also possible to determine whether an image object is stationary by calculating the difference between frames.

Whether or not frame-specific regions are in the same position can be determined based on criteria such as whether specific vertices of the frame-specific regions are within a specified range of each other, or whether the overlapping area of the frame-specific regions accounts for a specified percentage or more of the area of the frame-specific regions.

In the example of Figure 7, the frame-specific regions r2(t1) to r2(tn) are all classified into the "chair" class, and the "chair" class belongs to stationary objects, so the image objects o1(t1) to o2(tn) are determined to be stationary objects.
(2-1-3) Tracking unit 603
When an image object in a moving image is a moving object, the tracking unit 603 tracks the image object to identify the correspondence between frame-by-frame regions. An image object determined not to be a stationary object by the stationary object determination unit 602 is a moving object, and therefore becomes a target for tracking by the tracking unit 603.

As mentioned above, when using YOLO to detect a bounding box surrounding an image object, it is also possible to obtain a class probability that indicates which class the image object falls into.

If this class probability can be used to uniquely determine a combination of bounding boxes (frame-specific regions) that share a common class between frames, the image object enclosed by that bounding box can be tracked.

Other conventional techniques for adding image objects contained in video may be used to track the image objects and determine the correspondence between frame-specific regions between frames.

In the example of Figure 7, the frame regions r2(t1) to r2(tn) are all classified into the "person" class, which belongs to moving objects, and there is only one frame region classified into the "person" class in each frame. Therefore, the frame regions r2(t1) to r2(tn) are identified as corresponding frame regions between frames.
(2-1-4) Image object region extraction unit 604
The image object region extraction unit 604 extracts an image object region including a frame-specific region related to a common image object as a common image object region between frames, thereby extracting an image object region for cropping the image object, and in particular, utilizing the image object region as a non-zero region.

In the example of Figure 7, the smallest rectangular area among the rectangular areas including the frame-specific areas r1(t1) to r1(tn) related to the image objects o1(t1) to o1(tn) is extracted as image object area r1.

Furthermore, as mentioned above, YOLO can obtain a confidence score for each bounding box that indicates the likelihood of detecting that bounding box.

In the example of Fig. 7, this reliability score is used to select the frame-specific region r2(tk) using the bounding box with the highest reliability among the frame-specific regions r2(t1) to r2(tn) related to a common stationary object (chair), and the region r2(tk) is set as the image object region r2. Needless to say, t1 ≤ tk ≤ tn.
(2-1-5) Characteristic sequence extraction unit 605
The characteristic sequence extraction unit 605 extracts a characteristic sequence for each image object region for each frame, similar to the characteristic sequence extraction unit 212 according to the first embodiment.

In the example of Figure 8, feature array v1 is extracted for image object region r1, and h1 and w1 are the height and width of image object region r1 in each frame of video 700, expressed in number of pixels. Also, c is the number of input channels. In other words, the 3DCNN 214 performs c types of convolution processing using feature array v1 as input.

Image objects o1(t1) to o1(tn) are moving objects (people) and can be in different positions and sizes in each frame. For this reason, the image object region r1 is larger in size (height h1 and width w1) than the individual frame regions r1(t1) to r1(tn).

Similarly, feature array v2 is extracted for image object region r2. h2 and w2 in feature array v2 are the height and width of image object region r2 in each frame of video 700, and c is the number of input channels.

In addition, for image object areas related to stationary image objects, a feature array may be extracted for the image object area included in one of multiple frames, and when extracting feature arrays for image object areas included in other frames, the previously extracted feature array may be reused.
(2-1-6) Feature space generation unit 606
Similar to the feature space generation unit 213 according to the first embodiment, the feature space generation unit 213 generates a feature space S for the entire video 700 to be input to the 3D CNN 607, using information identifying the image object regions r1 and r2 on the image 100 extracted by the image object region extraction unit 604 for each frame constituting the video 700, and the feature arrays for each image object region r1 and r2 extracted by the feature array extraction unit 605.

In the example of Figure 8, a feature space S is generated for each frame, similar to the feature space S in the first embodiment described above. The feature spaces S generated for each of frames r1 to tn are input to the 3DCNN 607.

Note that if an image object appears and disappears partway through the video 700, the feature array assigned to the non-zero region for frames that do not contain that image object may be a zero array. A zero array is an array in which all array components are zero.

Alternatively, for frames that do not include the image object, the region in the feature space S corresponding to the image object region may be set to zero.
(2-1-7)3DCNN607
Similar to the CNN 214 according to the first embodiment, the 3D CNN 607 performs sparse convolution processing, which performs convolution processing only on non-zero regions of the feature space S for the entire video 700, and does not perform convolution processing on zero regions. Therefore, the amount of calculation can be reduced by the amount of convolution processing on the zero regions.

Furthermore, in the prior art, the entire range in which the image object can move within the video 700 must be considered a non-zero region, whereas in the present embodiment, the non-zero region can be narrowed down to the range in which the image object actually moved within the video 700, which is highly effective in reducing the amount of calculation required for the convolution process.
9, the image processing device 1 acquires the moving image 700 by receiving it from the video camera 110 or reading it from the HDD 204 (S901). As described above, since it is necessary to extract the image object area by extracting the frame area, the image processing device 1 acquires the entire moving image 700 in a lump, rather than sequentially acquiring the frames that make up the moving image 700, in parallel with the shooting by the video camera 110.

In the image processing device 1, the frame-by-frame region extraction unit 601 detects a frame-by-frame region for each frame constituting the moving image 700 (S902), and then the stationary object determination unit 602 determines whether the image object related to the frame-by-frame region is a stationary object (S903).

Furthermore, for frame-specific areas related to image objects that are determined to be moving rather than stationary, the tracking unit 603 performs tracking processing of the moving object (S904).

An image object area for cropping the image object is extracted from the frame-specific area in which the image object is common (S905). In this case, as described above, the method for extracting image objects differs between frame-specific areas related to stationary objects and frame-specific areas related to moving objects. Furthermore, a feature sequence is extracted for each frame and for each image object area (S906).

Then, non-zero and zero regions in the feature space are set using the image object region for each image object (S907), and feature arrays are assigned to the non-zero regions (S908).

The 3DCNN 607 executes image processing using the feature space S in which the feature arrays are assigned to the non-zero regions as an input (S909). The image processing results by the 3DCNN 607 may be stored in the HDD 204, for example.
[3] Modifications The present disclosure has been described above based on the embodiments, but it goes without saying that the present disclosure is not limited to the above-described embodiments, and the following modifications can be implemented.
(3-1) In the above second embodiment, an example was given in which image processing is performed once using the 3DCNN 607. However, it goes without saying that the present disclosure is not limited to this, and image processing may be performed multiple times using the 3DCNN 607. In such a case, the following may be performed.

When the video camera 110 is fixed, stationary objects remain stationary even in the captured image. By focusing on this point, the amount of calculation required by the 3DCNN 607 can be further reduced.

That is, as shown in Figure 10, first, image processing is performed by the 3DCNN 607 on a stationary object between time T and T + ΔT to calculate spatiotemporal features.

After that, if a stationary object that remains stationary between times T' and T'+ΔT is the same as the stationary object for which spatiotemporal features were previously calculated, then a feature array is not calculated for each image object region for that stationary object. Therefore, the non-zero region for that stationary object is not set in feature space S, nor is a feature array assigned to that non-zero region; instead, image processing is first performed by the 3DCNN 607, and the calculated spatiotemporal features are reused.

Whether or not the object is the same as the still object whose spatiotemporal features were previously calculated may be determined, for example, by calculating the difference between an image containing the earlier still object and an image containing the later still object, or by other methods.

In this way, the amount of calculation required for image processing by the 3DCNN 607 can be reduced, and the amount of calculation required to generate the feature space S can also be reduced.
(3-2) In the above embodiment, an example was described in which an image object region was extracted using YOLO. However, it goes without saying that the present disclosure is not limited to this, and the following method may be used instead.

For example, an image object contained in image 100 may be extracted, and the rectangular area having the vertex with the smallest x-coordinate and y-coordinate in that image object and the vertex with the largest x-coordinate and y-coordinate in that image object at diagonal positions may be used as the image object area.

Image object areas may also be extracted using methods other than those described above.

Regardless of the method used to extract image object regions, applying this disclosure can reduce the amount of computation required by the CNN, as in the above embodiment.

Furthermore, in order to reduce the amount of computation required for the CNN, it is desirable to extract the image object region so that it is as small as possible. Therefore, if only the image object is included in the image object region, the amount of computation required for the CNN can be reduced to the greatest extent.
(3-3) In the above embodiment, a case where a moving image is processed using a 3D CNN has been described as an example, but it goes without saying that the present disclosure is not limited to this, and instead of a 3D CNN, a moving image may be processed using a CNN other than a 3D CNN or a neural network other than a CNN. Even when a moving image is processed using a neural network other than a 3D CNN, the same effect as above can be obtained by applying the present disclosure.
(3-4) The present disclosure may be an image processing method realized by processing by the above-described image processing device 1. Furthermore, the present disclosure may be an image processing program realized by executing these methods by a processor such as a CPU, or may be a digital signal consisting of this image processing program.
The present disclosure may also be the image processing program or the digital signal recorded on a computer-readable recording medium. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, BDs (Blu-ray (registered trademark) Discs), and semiconductor memories. The present disclosure may also be the digital signal recorded on these recording media.
Furthermore, the present disclosure may also be such that the image processing program or the digital signal is transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, or the like.
The present disclosure may also provide a computer system having a microprocessor and a memory, wherein the memory stores the image processing program, and the microprocessor operates in accordance with the image processing program.
The image processing program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network or the like.
(3-5) The above-described embodiments and modifications may be combined with each other.
Furthermore, all the numbers used above are merely examples to specifically explain the present disclosure, and the present disclosure is not limited to the numbers used as examples.
The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software.
The order in which the steps are performed is merely an example for specifically explaining the present disclosure, and other orders may be used. Also, some of the steps may be performed simultaneously (in parallel) with other steps.
Furthermore, various modifications of the present embodiment that are within the scope of what would occur to a person skilled in the art are also included in the present disclosure, as long as they do not deviate from the gist of the present disclosure.

The image processing device, image processing method, and image processing program disclosed herein are useful as technologies for reducing the amount of calculation required for convolution processing.

1. Image processing device 100. Image (still image)
110...................................................Video camera 120...................................................Communication network 207...................................................Image processing program 211, 604...................................Image object region extraction unit 212, 605...................................Feature sequence extraction unit 213, 606...................................Feature space generation unit 214...................................................CNN (Convolutional Neural Network)
401...................................................Binary mask 402...................................................................Index list 601...................................................................Frame-by-frame region extraction unit 602...........................................................Stationary object determination unit 603...................................................................Tracking unit 607...................................................................3DCNN (3-dimensional CNN)
700...................................................Moving images 1201, 1202...Non-zero region 1301...................................................Map mask o1, o2...................................................Image objects o1(t1) to o1(tn)...Image objects o2(t1) to o2(tn)...Image objects r1, r2...................................................Image object region r1(t1) to r1(tn)...Frame-specific region r2(t1) to r2(tn)...Frame-specific region v1, v2...................................................Feature array S...................................................Feature space

Claims (26)

An image processing device that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image,
an image object region extraction unit that extracts only a portion of a predetermined image object in an input image as an image object region;
a feature sequence extraction unit that extracts a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
An image processing device characterized by performing sparse convolution processing in which zero regions other than non-zero regions are excluded from the convolution processing. The image object region is a rectangular region including the part of the image object,
2. The image processing apparatus according to claim 1, wherein the non-zero regions are associated with image object regions according to the positions and sizes of rectangular regions in the input image. The image object is a person,
3. The image processing device according to claim 1, wherein the CNN performs image processing to recognize human behavior. The image object is a person or object,
4. The image processing device according to claim 1, wherein the CNN performs image processing to recognize a scene in an input image. 5. The image processing device according to claim 1, wherein the predetermined image object is an image object that is to be subjected to image processing using a CNN. 6. The image processing device according to claim 1, wherein the feature sequence extraction unit extracts the feature sequence by any one of a convolutional neural network, a multilayer perceptron, a neural network other than a convolutional neural network or a multilayer perceptron, principal component analysis, and a predefined feature calculation method. An image processing device that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image,
an image object region extraction unit that extracts an image object region including at least a part of a predetermined image object from an input image;
a feature sequence extraction unit that extracts a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
For zero regions other than non-zero regions, sparse convolution processing is performed, excluding them from the convolution processing target .
The image processing device, wherein the CNN is SBnet. An image processing device that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input video,
an image object region extraction unit that extracts an image object region that includes at least a part of a predetermined image object in an input video sequence and that is common to frames that include the image object;
a feature sequence extraction unit that extracts a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
An image processing device characterized by performing sparse convolution processing in which zero regions other than non-zero regions are excluded from the convolution processing. 9. The image processing apparatus according to claim 8, wherein the feature sequence extracting unit extracts a feature sequence for each image object region for each frame. The image object region extraction unit
extracting a frame-by-frame region including at least a portion of a predetermined image object for each frame constituting the input video;
10. The image processing apparatus according to claim 8, wherein the image object region is a sum of frame regions including at least a part of an image object that is common between frames. The image object region extraction unit
extracting a rectangular frame-by-frame region including at least a portion of a predetermined image object for each frame constituting the input video;
10. The image processing device according to claim 8, wherein the image object area is the smallest rectangular area among rectangular areas including a sum area of frame-specific areas that includes at least a part of an image object that is common between frames. The image object region extraction unit
a stillness determination unit that determines whether each image object is still;
extracting a frame-by-frame region including at least a part of a predetermined image object for each frame constituting the input video sequence, and obtaining an extraction likelihood for each frame-by-frame region;
10. The image processing apparatus according to claim 8, wherein the frame region having the highest extraction likelihood among the frame regions relating to a common image object is set as the image object region relating to the image object. The image processing device according to claim 12, characterized in that, for a plurality of image object regions relating to a common image object and also relating to a stationary image object, the feature array extraction unit extracts the feature array relating to any one of the image object regions, thereby extracting the feature array relating to the other image object regions. the image object region extraction unit acquires, for each frame region, a class to which the image object related to the frame region belongs;
14. The image processing device according to claim 12, wherein the stillness determination unit determines whether or not an image object is still based on a class to which the image object belongs. 14. The image processing device according to claim 12, wherein the stillness determination unit determines that an image object is still when frame-specific regions relating to the common image object are located at a common position within a frame. The image object is a person,
16. The image processing device according to claim 8, wherein the CNN performs image processing to recognize human behavior. The image object is a person or object,
17. The image processing device according to claim 8, wherein the CNN performs image processing for recognizing scenes in an input video image. 18. The image processing device according to claim 8, wherein the predetermined image object is an image object that is to be subjected to image processing using a CNN. When a still image object included in one input image is included in another input video,
The CNN excludes, from the target of the convolution process, a region in the feature space corresponding to an image object region relating to the same still object image object included in the other input image as a zero region;
16. The image processing device according to claim 12, wherein, among the image processing results by the CNN for the one input video, the image processing result for a still image object included in the one input video is used as the image processing result for the same still image object included in the other input image. 20. The image processing device according to claim 8, wherein the CNN is a 3D CNN. 21. The image processing device according to claim 8, wherein the CNN is SBnet. 22. The image processing device according to claim 8, wherein the feature sequence extraction unit extracts the feature sequence by any one of a convolutional neural network, a multilayer perceptron, a neural network other than a CNN or a multilayer perceptron, principal component analysis, and a predefined feature calculation method. An image processing method executed by an image processing device that executes image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image,
an image object region extraction step of extracting only a part of a predetermined image object from the input image as an image object region;
a feature sequence extraction step of extracting a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
An image processing method characterized in that sparse convolution processing is performed in which zero regions other than non-zero regions are excluded from the convolution processing. An image processing method executed by an image processing device that executes video processing using a CNN that receives as input a feature space configured by a feature array extracted from an input video, the method comprising:
an image object region extraction step of extracting an image object region that includes at least a part of a predetermined image object in the input video sequence and that is common to frames that include the image object;
a feature sequence extraction step of extracting a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
An image processing method characterized in that sparse convolution processing is performed in which zero regions other than non-zero regions are excluded from the convolution processing. An image processing program executed by a computer that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image,
an image object region extraction step of extracting only a part of a predetermined image object from the input image as an image object region;
a feature sequence extraction step of extracting a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
An image processing program characterized by performing sparse convolution processing in which zero regions other than non-zero regions are excluded from the convolution processing. An image processing program executed by a computer that performs video processing using a CNN that receives as input a feature space configured by a feature array extracted from an input video,
an image object region extraction step of extracting an image object region that includes at least a part of a predetermined image object in the input video sequence and that is common to frames that include the image object;
a feature sequence extraction step of extracting a feature sequence for each image object region;
CNN uses the following feature space:
A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and
An image processing program characterized by performing sparse convolution processing in which zero regions other than non-zero regions are excluded from the convolution processing.