JP7749489B2 - Video analysis device and video analysis method - Google Patents

Description

The embodiments relate to a video analysis device and a video analysis method for analyzing videos and images.

In image recognition using neural networks, a frame of a certain size (number of pixels) called a detection window is used to detect specific information from within an image. For example, if you try to use a detection window to detect information in an image obtained by digital sampling from a video frame, even if the image's angle of view (captured area) is the same, if the resolution is different, the information contained in detection windows of the same size will differ (for example, a lower resolution will pick up information from a wider area). Usually, the size of the detection window is balanced by enlarging or reducing the image.

Patent No. 6706788 Patent No. 6867117

However, when using a neural network to detect objects of similar size from multiple images with different resolutions, differences in the information within the detection window make it difficult to perform stable learning, and there is also the problem that features are lost when the image size is reduced to fit the detection window.

The problem to be solved by the present invention is to provide a video analysis device and method for analyzing a plurality of image data with different resolutions.

In one embodiment, the video analysis device switches the size of the detection window and model parameters of the neural network depending on the resolution of the image data obtained by digitally sampling the video frame , thereby providing diversity and detecting the target object from various types of image data.

FIG. 1 is a diagram showing the configuration of a video analysis device according to the first embodiment. FIG. 2 is a schematic diagram showing an example of a data flow for analysis processing by the video analysis device according to the first embodiment. FIG. 3 is a flowchart showing the analysis process performed by the video analysis device according to the first embodiment. FIG. 4 is a schematic diagram showing an example of video sampling data to be analyzed by the video analysis device according to the embodiment. FIG. 5 is a flowchart showing the analysis process performed by the video analysis device according to the second embodiment. FIG. 6 shows an example of table data provided in the video analysis device according to the second embodiment. FIG. 7 is a flowchart showing an analysis process performed by the video analysis device according to the third embodiment. FIG. 8 is a flowchart showing the analysis process performed by the video analysis device according to the fourth embodiment.

Hereinafter, embodiments will be described with reference to the drawings.
(First embodiment)
For example, when images are acquired by digital sampling from video footage of the same scene, the resolution of the images may differ depending on the digital sampling. In this embodiment, an example is shown in which a neural network is used to detect an object (referred to as a detection object) of roughly the same size from multiple image data sets that have the same or similar angle of view (photographing range) but different resolutions.

Figure 1 is a diagram showing the configuration of a video analysis device according to an embodiment.

The video analysis device 1 is a device that detects a detection target from input video or images and outputs the result to the outside, and may be equipped with a computer such as a CPU and memory, and digital signal processing means such as a DSP (Digital Signal Processor).

The video input unit 11 inputs digital image data from the outside into the video analysis device 1 and outputs the image data. The input image data may be any image data, such as a still image of video data. The video input unit 11 extracts meta information (number of pixels or resolution) from the input image data and outputs the meta information to the field of view acquisition unit 12 as field of view information.
In addition, when outputting image data, the video input unit 11 may, for example, use a sample of the image data on which analysis is to be performed and adjust the angle of view (shooting range) of the image data to be output so that it matches the sample.

The field of view acquisition unit 12 acquires field of view information from the video input unit 11.

The detection NN selector 13 selects a neural network NN for detecting the detection target from the image data based on the field of view information input from the field of view acquisition unit 12. The neural network NN is, for example, a Deep Neural Network (DNN) and may also include a Convolutional Neural Network (CNN). The neural network NN may also include any neural network such as a Recurrent Neural Network (RNN) or a Long Term Short Memory (LTSM). These various neural networks are common technologies, and therefore will not be described here.

The neural network NN has a detection window size as a parameter. The detection window indicates the area on the image data for detecting the detection target from the image data, and the detection window size indicates the number of pixels included in the detection window.

The storage unit 14 is a memory that stores various information such as a table 141 and NN model parameters 142.

Table 141 stores the detection window size and NN model of the associated neural network NN for each piece of field-of-view information for image data input from the video input unit 11. In other words, it indicates that the neural network NN is changed to the associated NN model for each piece of field-of-view information for the image data. For example, data TB1 indicates that when the resolution of the image data input from the video input unit 11 is 720 [number of pixels] x 480 [number of pixels], the image data is processed using a neural network NN with the NN model set to detection NN1 and the detection window size of the detection NN1 set to 16 x 16. Table 141 may be set at the time of shipment, or may be acquired by the video analysis device 1 by downloading it from a server or the like via the Internet. Table 141 may also be set in the storage unit 14 in any other manner.

NN model parameters 142 are parameters used in the neural network NN, and are parameters of the neural network NN corresponding to the NN model used in table 141. For example, the model parameters of NN1 for detecting data TB1 are MP1, the model parameters of NN2 for detecting data TB2 are MP2, and the model parameters of NN2 for detecting data TB2 are MP3. The NN model parameters 142 are assumed to have been obtained by learning using data of the detection window size set in table 141.

The image recognition unit 15 runs the neural network NN selected by the detection NN selection unit 13 on the image data captured by the video input unit 11.

The feature detection unit 151 performs feature detection using a neural network (NN) as part of the functions of the image recognition unit 15. The feature detection unit 151 may be, for example, a CNN.

The region calculation unit 152 performs image recognition to detect specific objects in the image data captured by the video input unit 11. The region calculation unit 152 may use a general object recognition method or a neural network.

The result output unit 16 outputs the results of analysis by the image recognition unit 15 to an external device such as a monitor (not shown).

Figure 2 is a schematic diagram showing an example of the data flow used for analysis processing by a video analysis device according to an embodiment. The video analysis device 1 of this embodiment uses an apple as the analysis object AO and detects defects such as worm holes on the apple. The defects such as worm holes are referred to as the detection object DO.

Let's say that the video analysis device 1 obtains low-resolution image data SD1 by digitally sampling the photographic data of the apple, which is the analysis target AO.

When low-resolution image data SD1 is input, the field of view acquisition unit 12 acquires field of view information and inputs it to the detection NN selection unit 13. The detection NN selection unit 13 acquires information from the memory unit 14 based on the resolution of the input field of view information, sets the detection window size DW1 to 5x7, and sets the detection NN model to a low-resolution DNN (detection neural network NN1). The image recognition unit 15 analyzes the image data SD1 using the neural network NN1. Here, it is assumed that the neural network NN1 has been trained using data of the detection window size DW1 that includes the detection target DO. The detection target DO is detected from the image data SD1 through the analysis by the image recognition unit 15.

Similarly, when the video analysis device 1 obtains high-resolution image data SD2 by digitally sampling photographic data of an apple, which is the analysis object AO, the image recognition unit 15 analyzes the image data SD2 using a detection window size DW2 of 25 x 37 and a high-resolution DNN (detection neural network NN2) as the detection NN model. The image data SD2 has the same angle of view as the image data SD1, and as shown in Figure 2, the size of the apple, which is the analysis object AO, relative to the size of the image data SD is similar in the image data SD1 and SD2. The neural network NN2 is trained using data with a detection window size DW2 that includes the detection object DO, and the detection object DO is detected from the image data SD2 through analysis by the image recognition unit 15.

Note that, unless otherwise specified, image data SD1 or SD2, detection window size DW1 or DW2, and neural network NN1 or NN2 will be referred to as image data SD, detection window size DW, and neural network NN, respectively. Also, in this embodiment, the video analysis device 1 is shown with three pieces of data each in table 141 and NN model parameters 142 in Figure 1, but may have more than three pieces of data.

Figure 3 is a flowchart showing the analysis process performed by the video analysis device according to the embodiment.

In the video analysis device 1, the video input unit 11 obtains image data SD (still image) including an analysis object AO (corresponding to the apple in Figure 2), and the field of view acquisition unit 12 acquires field of view information for the image data SD (step 101). Note that the field of view acquisition unit 12 may also acquire field of view information via a different route, for example, through user settings, without using a still image.

The detection NN selector 13 uses information from table 141 (a table showing the relationship between field of view, detection window size, and NN model) in the storage unit 14 to select the NN model to use that is linked to the field of view information acquired in step S101 (step S102). To explain this more specifically using the example of TB1 in Figure 1, the detection NN selector 13 selects the detection window size DW "16x16" and the detection NN model "detection NN1" that are linked to the resolution information "720x480" acquired in step S101.

The detection NN selector 13 then transmits the selected NN information and detection window size to the image recognition unit 15, which then performs a detection process for the target object (detection target DO) from the image data output by the video input unit (step 103). More specifically, in step 103, rather than using the entire image SD as input to the NN of the image recognition unit 15, input is made to the NN in units of detection windows DW. Because the detection window DW is smaller than the size of the image SD, the image recognition unit 15, for example, first determines the detection window position (say, range 1) on the image SD and performs NN calculations and analysis on range 1. After analysis of range 1 is complete, another detection window position (say, range 2) is determined and NN calculations are performed on range 2, and so on. Subsequently, the detection window position may be shifted slightly to perform analysis on the entire image.

If the target is detected as a result of the NN calculation, the result output unit 16 may output the result to a monitor (not shown) or the like (step 104). The output method in step 104 may be to display some kind of image on the screen, or to store the result as a log in a file such as the memory unit 14.

By following the above procedure, it is possible to detect a detection target DO of a consistent size in each image SD for multiple image data SDs with the same angle of view but different resolutions.

Figure 4 is a schematic diagram showing an example of video sampling data analyzed by a video analysis device according to an embodiment. The detection range DA can be determined based on the type of input data (video, image, etc.) to the video input unit 11 and the characteristics of the detection target DO, etc.

FIG. 4A shows an example in which multiple detection targets DO11 and DO12 are present in image data SD1. The video analysis device 1 can detect the detection targets DO11 and DO12 by executing the process of FIG. 3 while moving the detection window DW over the entire image data SD1. For example, in a product quality check, a captured video of the product can be input into the video analysis device 1 to detect how many abnormalities exist in the product being analyzed. Furthermore, in a non-defective product inspection, if even one abnormality is detected, the product is deemed to have failed, and the result output unit 16 or the like may output the result and terminate the recognition process in the image recognition unit 15. Figures 4B and subsequent figures will be described in the following embodiments.
Second Embodiment
In this embodiment, an example of detecting the detection object DO in the case of FIG. 4B will be described.

Figure 4(b) shows an example where one or more detection targets DO exist in a known specific area on the image data SD2. As in Figure 4(b), when the area DA in which the target exists within the image data SD2 is determined, the video analysis device 1 may set the known specific area as the detection range DA and perform recognition only within that detection range DA.

Figure 5 is a flowchart showing the analysis process performed by the video analysis device according to the second embodiment.

The field of view acquisition unit 12 acquires field of view information of the input image data SD2 and inputs it to the detection NN selection unit 13 (step S201). The detection NN selection unit 13 acquires the detection target DO and detection range DA (step S202). The detection target DO and detection range DA may be set by the user using, for example, a keyboard connected to the video analysis device 1, or an image such as that shown in Figure 4(b) may be displayed on a monitor (not shown) connected to the video analysis device 1, allowing the user to set the detection target DO and detection range DA.

The detection NN selection unit 13 selects a detection window size and NN model based on the input field of view information, detection target DO, and detection range DA, and sets them in the image recognition unit 15 (step S202).

Figure 6 shows an example of table data provided in the video analysis device according to the second embodiment.

Table 1411 is data stored in the memory unit 14, and in addition to the contents of Table 141 in Figure 1, the "detection target" and "detection range" are linked to the resolution of the input image. For example, data TB11 indicates that when the resolution of image data input from the video input unit 11 is 720 x 480, the model of the neural network NN used for analysis is "detection NN1," the detection window size of the detection NN1 is "16 x 16" (e.g., DW21 in Figure 4(b)), the detection target DO is "abnormality in part X" (e.g., DO21 in Figure 4(b)), and the detection range DA is "360 x 100 + 320 x 120" (e.g., DA21 in Figure 4(b)). The type of abnormality, such as a scratch, of the detection object DO21 has been determined in advance as "abnormality in part X," and the detection range DA21 is a square with two diagonal corners at coordinates (x1, y1) = (360, 100) and (x2, y2) = (320x120). In this embodiment, the detection NN1 is trained using image data of the "16x16" detection window DW21, which includes the "abnormality in part X" of the detection object DO21.

For example, when the resolution of the image data input from the video input unit 11 is 720x480, data TB14 indicates that the neural network NN model used for analysis is "Detection NN1," the detection window size of the detection NN1 is "16x16" (e.g., DW22 in Figure 4(b)), the detection target DO is "Y section abnormality" (e.g., DO22 in Figure 4(b)), and the detection range DA is "180x240+240x240" (e.g., DA22 in Figure 4(b)). In this embodiment, the detection NN1 is trained using data TB11 as well as data TB14. Note that the size of the detection window DW may be changed depending on the type of detection target DO, as in the examples of DW21 and DW22 in Figure 4(b). In this case, the neural network NN may be trained using each detection window DW size to create a different neural network NN. For example, in Figure 4(b), when the characteristics of the detection target DO21 (for example, an abnormality in part X) and the detection target DO21 (for example, an abnormality in part Y) are significantly different, using dedicated NNs that have been trained separately for each may result in higher recognition accuracy for the detection target DO.

Returning to FIG. 5, the detection NN selector 13 selects the NN model to use from the information in table 1411 stored in the memory unit 14 or the like (step S203). The image recognition unit 15 executes a detection process for the detection target DO using the NN selected in step S203 for the range of the detection window DW in the detection range DA (step S204). If a detection target DO is detected in step S204 (for example, DW21 in FIG. 4(b)), the result output unit 16 may store information such as the position of the detected detection target DO in the memory unit 14 or the like (Yes in step S205, step S206). Note that instead of step S206, the process may proceed to step S208, where the result is output to the result output unit 16.

If the detection target DO is not detected in step S204 (for example, the example of DA23 in Figure 4(b)), the process moves to the next detection range DA and repeats the same process from step S203 (No in step S205, No in step S207). Once processing for all detection ranges DA has been completed, the information on the detection target DO stored in step S206 may be output to the result output unit 16 (Yes in step S207, step S208).

The above procedure enables image recognition for multiple detection ranges DA. In this embodiment, the recognition range of the detection target DO is not the entire image, but is limited to a pre-set range (detection range DA), so processing can be completed in a short time.
(Third embodiment)
4C, an example is shown in which a detection target DO is detected in the area of a certain object (analysis target AO) shown in a video. For example, in food inspection, when an abnormality in the analysis target AO shown in a video (image data SD) is to be inspected, the video analysis device 1 of this embodiment first detects the analysis target AO, and then detects the detection target DO in the area of the analysis target AO.

Figure 7 is a flowchart showing the analysis process performed by the video analysis device according to the third embodiment. The following explanation uses Figure 4(c) as an example.

The field of view acquisition unit 12 acquires field of view information for the input image data SD3 and inputs it to the detection NN selection unit 13 (step S301). At the same time, in the image recognition unit 15, the area calculation unit 152 performs a detection process for the analysis target AO on the image data SD3 (step S302). In step S302, the image recognition unit 15 may have information about the analysis target AO (e.g., an apple) preset therein. Alternatively, the user may enter information about the analysis target AO into the image recognition unit 15 using, for example, a keyboard connected to the video analysis device 1.

The area calculation unit 152 detects the analysis target AO from the input image data SD3 and identifies its area. For example, in an example of apple quality control, if there are multiple apples in the image data SD3, the display area of those apples is calculated and obtained. A general object recognition method is used for this identification. The display area may be the shape of the analysis target AO itself, or it may be expressed as a rectangle containing the analysis target AO, or it may be expressed in some other format. The image recognition unit 15 sets the calculated display area as the detection range DA. In the example of Figure 4(c), in step S302, analysis target AO3 and analysis target AO31 are detected as the analysis targets AO.

The detection NN selection unit 13 selects a detection window size and NN model linked, for example, to table 1411 in FIG. 6 from the input information including the angle of view information and detection target DO, and sets these in the image recognition unit 15 (step S303). The image recognition unit 15 performs detection processing of the detection target DO using the selected detection NN for the detection range DA (corresponding to the display area of the analysis target AO) provided by the area calculation unit 152 in the input video (step S304). The results of the processing in step S304 may be output to a monitor, etc. (step S305). The procedures from steps S303 to S305 may be performed for all analysis target AO detected in step S302.

According to the video analysis device 1 of this embodiment, the recognition range of the detection target DO is not the entire video, but the display area of the analysis target AO acquired in advance is set as the detection range DA, thereby enabling the detection target DO to be detected in a short time.
(Fourth embodiment)
In this embodiment, an example is shown in Fig. 4(d) in which detection targets DO scattered within image data SD are detected. For example, a phenomenon called film grain may occur in video film (each frame of the film), which is analog data. In this embodiment, the video analysis device 1 randomly determines the position of a detection window in the image data SD, for example, and performs film grain detection processing on the image data within the detection window.

Figure 8 is a flowchart showing the analysis process performed by the video analysis device according to the fourth embodiment. The following explanation uses Figure 4(d) as an example.

The field of view acquisition unit 12 acquires field of view information of the input image data SD4 and inputs it to the detection NN selection unit 13 (step S401). The detection NN selection unit 13 acquires information on the detection target DO (equivalent to film grain) (step S402). The information on the detection target DO may be set in the detection NN selection unit 13 by the user via, for example, a keyboard connected to the video analysis device 1.

The detection NN selection unit 13 selects the associated detection window size and NN model from, for example, table 1411 in FIG. 6 according to the acquired field of view information and detection target DO, and sets them in the image recognition unit 15 (step S403). The neural network NN selected in step S403 has previously learned film grain using data on the detection window size that includes the detection target "film grain" in table 1411 in FIG. 6, for example.

In addition, in table 1411 of FIG. 6, further items may be linked to the detection target "film grain." For example, possible items include the "filming era" of the footage for which film grain is to be detected, and the "video type" (live action, animation, etc.). Other possible items include the "digitization process" (e.g., whether the data is digitized from analog data or data obtained by further sampling the digitized data) used to extract image data from the footage, the "data source" (e.g., Blu-ray, DVD, 4K/2K Blu-ray), and the "film size" (e.g., 8mm, 16mm, 24mm, 35mm). When these items are taken into consideration, it is possible to set a detection window size and NN model for the neural network NN for each item. Furthermore, different NN models may be set depending on the combination of items, and the linked NN model is trained using data that matches the combined items.

The image recognition unit 15 determines the position of the detection window DW4 (step S404). In step S204, the position of the detection window DW4 may be determined randomly at any position in the image data SD4, or a position in the image data SD4 determined according to a certain rule may be determined as the position of the detection window DW4.

For the data of the detection window DW4 whose position was selected in step S404, the NN selected in step S403 performs a detection process for the detection target DO (step S405), similar to step S103 in FIG. 3. If the detection target DO is detected (Yes in step S406), the result output unit 16 may output information such as the position of the detected detection target DO to an external monitor or the like (step S407). On the other hand, if the detection target DO is not detected (No in step S406), the process returns to step S404, where the next position for the detection window DW4 is determined and the same process is repeated. When returning to step S404, it is desirable that the image recognition unit 15 determine the position of the detection window DW4 so that it does not overlap with the range of the detection window DW4 determined previously.

The above procedure makes it possible to detect film grain that occurs in image data SD obtained by digitally sampling video film.

At least one of the embodiments described above can provide a video analysis device and a video analysis method for analyzing multiple image data with different resolutions.

Note that the names, definitions, and types of condition parameters, their corresponding options, values, evaluation indicators, etc. displayed on the analysis screens shown in the drawings are shown as examples in this embodiment and are not limited to those shown in this embodiment.

While several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be embodied in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are within the scope and spirit of the invention, and are also included in the inventions and their equivalents as set forth in the claims. Furthermore, the scope of the present invention also includes cases in which each component in the claims is expressed separately, as a combination of multiple components, or as a combination of these. Furthermore, multiple embodiments may be combined, and examples composed of such combinations are also within the scope of the invention.

In addition, to clarify the explanation, the drawings may show the width, thickness, shape, etc. of each part more schematically than in the actual embodiment. In block diagrams, data and signals may be exchanged between unconnected blocks, or even between connected blocks in directions not indicated by arrows. The processes shown in the flowcharts may be realized by hardware such as an IC chip or a digital signal processor (DSP), software (e.g., a program) running on a computer including a microcomputer, or a combination of hardware and software. The device of the present invention is also applicable when the claims are expressed as control logic, a program containing instructions for a computer to execute, or a computer-readable recording medium containing the instructions. Furthermore, the names and terminology used are not intended to be limiting; other expressions that have substantially the same content and intent are also included in the present invention.

1...Video analysis device, 11...Video input unit, 12...Viewing angle acquisition unit, 13...Detection NN selection unit, 14...Memory unit, 15...Image recognition unit, 16...Result output unit, 141...Table, 142...NN model parameters, 151...Feature detection unit, 152...Area calculation unit.

Claims (6)

a video input unit capable of selectively inputting a plurality of image data having different resolutions;
an image recognition unit that has a plurality of different detection neural networks (NNWs) prepared, and that, when the image data is input, detects a desired window area in the image data, and detects a feature amount of the image data of the detected window area using one of the detection neural networks;
a field of view acquisition unit that receives field of view information corresponding to the resolution of the image data input to the video input unit from the video input unit or from an external device;
a storage unit that stores a table that contains a plurality of sets of information on the angle of view of the image data and information on a detection window size and a detection neural network model that correspond to the angle of view;
a selection unit that identifies the detection window size and the detection neural network model that correspond to each other and are stored in the storage unit based on the field of view information provided by the field of view acquisition unit, provides information on the identified detection window size and the detection neural network model to the image recognition unit, and selects one of a plurality of sets of detection neural networks to cause the image recognition unit to detect feature quantities of image data;
A video analysis device equipped with : In the storage unit,
First information for the first detection neural network, including the number of pixels (720 × 480) and the number of pixels (16 × 16) as the detection window size;
Second information for the second detection neural network, including the number of pixels (1920 x 1080) and the number of pixels (32 x 32) as the detection window size;
For the third detection neural network, third information is prepared, which is the number of pixels (3840 × 2160) and the number of pixels (64 × 64) as the detection window size,
moreover
The first information includes information 1a to which a first detection range is added and information 1b to which information on a second detection range is added,
The second information includes second-a information to which the first detection range is added and second-b information to which information on the second detection range is added,
The third information includes third-a information to which the first detection range is added and third-b information to which information on the second detection range is added,
the image recognition unit operates the first detection neural network in response to the first information, the second detection neural network in response to the second information, and the third detection neural network in response to the third information;
Furthermore, depending on the setting,
In the detection window size of the number of pixels (16 x 16), operations based on the information 1a and the information 1b are selectively available;
In the detection window size of the number of pixels (32 × 32), operations based on the information 2a and the information 2b are selectively available;
In the detection window size of the number of pixels (64 x 64), operations based on the information 3a and the information 3b are selectively available.
The video analysis device according to claim 1. 3. The video analysis device according to claim 1, wherein the image data is image data acquired from a video film . The video analysis device of claim 3, wherein when the image recognition unit detects film grains present in multiple locations in the image data acquired from the video film, the image recognition unit randomly determines the position of the detection window that sets the detection window size of the table within the detection range and detects abnormalities in the image data within the detection window . 4. The video analysis device of claim 3, wherein film grain detection is performed according to the specified "video type" of the video film, or the "digitization process" for extracting digital image data from the video film, or the "film size" of the video film, or the "data source." A video analysis method using the video analysis device according to claim 1.