JP7632692B2 - Apparatus, method and program for detecting video of interest - Google Patents

Description

One aspect of the present invention relates to an attention video detection device, method, and program for detecting attention video including an object of interest, for example, from video data.

In recent years, with the widespread use of high-definition cameras, various technologies have been proposed to analyze human behavior based on captured video. For example, there are technologies that detect criminal behavior based on surveillance camera footage and dangerous behavior at construction sites.

In order to accurately detect human behavior, it is generally necessary to observe a large amount of video. In such cases, manual observation is time-consuming and labor-intensive, so research is being conducted on technologies that use algorithms to automatically detect specific behaviors. In addition, research is being actively conducted on the robustness of person tracking and face recognition technologies to detect the movements of multiple people individually (see, for example, non-patent document 1).

Z. Xiao et al. “Improving Transferability of Adversarial Patches on Face Recognition with Generative Models.” CVPR2021.

Incidentally, for example, to accurately detect the movement of a person, it is necessary to consider the interaction between the person and the surrounding objects that the person is working on, that is, the correlation between their movements. However, the technology described in Non-Patent Document 1 detects the movement of a specific person or object that is the target of monitoring on its own. As a result, there are cases where the behavior of a person cannot be detected with high accuracy.

For example, scenes involving interactions between a person and objects, such as climbing a ladder or hammering a nail with a hammer, have unique potential dangers associated with surrounding objects, such as falling off a ladder or accidentally hitting a hand with the hammer. However, detecting only the movements of people as the monitoring target cannot monitor the potential dangers of the people's actions.

This invention has been made in light of the above circumstances, and aims to provide technology that makes it possible to accurately detect the image of interest required to estimate the movement of an object of interest.

In order to solve the above problem, one aspect of an attention video detection device or method according to the present invention includes a first processing unit or process for acquiring video data captured by a camera, a second processing unit or process for detecting object detection information including information representing an object and a position of the object in the video frame from a video frame of the video data, a third processing unit or process for extracting depth information representing a distance distribution from the camera of an image depicted in the video frame from the video frame of the video data, and detecting distance detection information by calculating a distance of the object from the camera based on the extracted depth information , and The system is equipped with a fourth processing unit or process for detecting a specific image area including a first object that is the target of attention and a second object that is the work target of the first object based on the object detection information and the distance detection information, and including a combination of the first object and the second object, in which the difference in distance from the camera between the first object and the second object is less than a predetermined value, and information representing the difference in distance; and a fifth processing unit or process for detecting, as the image of interest, a specific image area in which the combination of the first object and the second object included in the specific image area and the difference in distance satisfy a predetermined detection condition.

According to one aspect of the present invention, for example, a specific image area including a first object designated as a target of attention and a surrounding second object to be worked on by the first object, and including a combination of the first object and the second object, the difference in distance between the first object and the second object from a camera being equal to or less than a predetermined value, and information representing the difference in distance, is detected as the image of attention. As a result, when analyzing the movement of the first object based on the image of attention, it is possible to estimate the movement of the first object with high accuracy by taking into account the interaction in the depth direction between the first object and the surrounding second object to be worked on.

In other words, according to one aspect of the present invention, a technology can be provided that enables accurate detection of the image of interest required to estimate the movement of an object of interest.

FIG. 1 is a block diagram showing an example of the hardware configuration of an interest video detection device according to an embodiment of the present invention, together with the configuration of the peripheral parts. FIG. 2 is a block diagram showing an example of the software configuration of the interest video detection device according to the embodiment of the present invention. FIG. 3 is a flowchart showing an example of a processing procedure and processing contents of an attention video detection process executed by the control unit of the attention video detection device shown in FIG. FIG. 4 is a flowchart showing an example of the processing procedure and processing contents of the object detection processing of the interest video detection processing shown in FIG. FIG. 5 is a flowchart showing an example of the processing procedure and processing contents of the distance information detection processing in the interest video detection processing shown in FIG. FIG. 6A is a flowchart showing the processing procedure and processing contents of the first half of the specific section determination processing of the interest video detection processing shown in FIG. FIG. 6B is a flowchart showing the processing procedure and processing contents of the latter half of the specific section determination processing of the interest video detection processing shown in FIG. FIG. 7 is a flowchart showing an example of a processing procedure and processing contents of the attention video detection processing shown in FIG. FIG. 8 is a diagram showing an example of an attention video detection rule stored in the attention video detection related information storage unit shown in FIG. 3. As shown in FIG.

Below, an embodiment of the present invention is described with reference to the drawings.

[One embodiment]
(Configuration example)
FIG. 1 is a block diagram showing an example of the hardware configuration of an attention video detection apparatus according to an embodiment of the present invention together with its peripheral devices, and FIG. 2 is a block diagram showing an example of the software configuration of the attention video detection apparatus.

The image detection device BD of interest consists of an information processing device such as a server computer or a personal computer, and a camera CM and a terminal MT are connected to the image detection device BD via a signal cable or a network not shown.

The camera CM is installed, for example, on a ceiling or wall surface capable of capturing an image of the area to be monitored, captures the entire body of a person to be monitored who is in the area to be monitored together with the surrounding area, and transmits the time-series image data to the image detection device BD.

The video data captured by the camera CM may be sent directly from the camera CM to the attention video detection device BD, or may be temporarily stored in a video database (not shown) and then sent to the attention video detection device BD. The number of cameras CM is not limited to one, and may be multiple.

The terminal MT is used, for example, by a system administrator or a monitor who monitors the behavior of the monitored person, and is configured as an information processing terminal such as a personal computer. The terminal MT has a function of receiving, for example, video information of interest detected by the video information of interest detection device BD, and analyzing the behavior of the monitored person based on the received video information of interest. Note that the function of analyzing the behavior of the monitored person based on the video of interest may be provided by the video information of interest detection device BD.

In addition, when the attention video detection device BD is equipped with a machine learning model, the terminal MT may have a function of inputting learning data necessary for learning the machine learning model into the attention video detection device BD.

The image detection device of interest BD is equipped with a control unit 1 using hardware processors such as a central processing unit (CPU) and a graphics processing unit (GPU), and is connected to the control unit 1 via a bus 5 with a storage unit having a program storage unit 2 and a data storage unit 3, and an input/output interface (hereinafter the interface is abbreviated as I/F) unit 4. The control unit 1 may be configured using a programmable logic device (PLD) or a field programmable gate array (FPGA), etc.

The input/output I/F unit 4 has a communication interface function and transmits and receives video data and various input/output data between the camera CM and the terminal MT via a signal cable or a network.

The program storage unit 2 is configured by combining a non-volatile memory such as a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various control processes according to one embodiment, in addition to middleware such as an operating system (OS). Hereinafter, the OS and each application program will be collectively referred to as a program.

The data storage unit 3 is, for example, a combination of a non-volatile memory such as an SSD, which can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), as a storage medium, and includes a video data storage unit 31, a specific section video determination related information storage unit 32, and a focus video detection related information storage unit 33 as the main storage units required to implement one embodiment.

The video data storage unit 31 is used to temporarily store the time-series video data transmitted from the camera CM for subsequent video of interest detection processing.

The specific section video assessment related information storage unit 32 stores specific section video assessment related information including the names of the designated area and designated object that have been designated in advance as monitoring targets.

For example, when a surveillance camera CM is fixedly installed in a monitored area, the designated area is automatically set by the terminal MT or the attention video detection device BD using a method such as semantic segmentation based on the video of the camera CM. The designated area may also be set manually by an administrator or a monitor on the terminal MT.

The designated area is expressed by a shape such as a rectangle, polygon, or circle. Furthermore, the designated area is not limited to one, and multiple designated areas can be specified. For example, if the shape is rectangular, multiple rectangles are specified as detection target areas within a video frame captured by a single fixed camera CM. Note that the designated area does not necessarily have to be set. In this case, the entire video frame is considered to be the designated area.

On the other hand, the designated object is the name of the object to be detected, such as a person, ladder, hammer, safety belt, etc. Note that the designated object does not necessarily have to be specified. In this case, all objects are considered to be detection targets.

The attention video detection related information storage unit 33 stores detection rules necessary for detecting attention video. The detection rules define rules for determining whether or not a video of a specific section detected by the specific section determination processing unit 14 described later is attention video in which an interaction between a person and an object is likely to occur, and multiple detection rules are stored in advance.

The control unit 1 includes, as processing functions necessary for implementing one embodiment, a video data acquisition processing unit 11, an object detection processing unit 12, a distance information detection processing unit 13, a specific section determination processing unit 14, and a focus video detection processing unit 15. All of the above processing units 11 to 15 are realized by causing the hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2.

The video data acquisition processing unit 11 acquires time-series video data output from the camera CM via the input/output I/F unit 4, and performs processing to temporarily store each of the acquired video data in the video data storage unit 31.

The object detection processing unit 12 includes a machine learning model for object detection constructed using a neural network. The object detection processing unit 12 reads the video data from the video data storage unit 31 frame by frame, and inputs the read video frames into the machine learning model for object detection to detect the class and position of the object appearing in the video frame as object detection information. An example of this object detection process will be described in the operation example.

The distance information detection processing unit 13 includes a machine learning model for detecting distance information that is constructed using a neural network. The distance information detection processing unit 13 reads the video data from the video data storage unit 31 frame by frame, and inputs the read video frames into the machine learning model for detecting distance information to detect the distance from the camera CM of the image shown in the video frame in pixel units. An example of this distance information detection process will also be described in the operation example.

The specific section determination processing unit 14 receives object detection information and distance detection information from the object detection processing unit 12 and the distance information detection processing unit 13, respectively. Then, by comparing this detection information with the specific section video determination related information stored in the specific section video determination related information storage unit 32, a process is performed to detect a video of a specific section in which a specified object is captured within a specified area. An example of the specific section detection process will be explained in more detail in the operation example.

The attention video detection processing unit 15 receives the specific section video information detected by the specific section determination processing unit 14. The received specific section video information is then compared with the detection rules stored in the attention video detection related information storage unit 33 to determine whether the specific section video information is attention video, and if so, performs processing to output the attention video from the input/output I/F unit 4 to the terminal MT. An example of attention video determination processing will also be described in more detail in the operation example.

The machine learning model for object detection and the machine learning model for distance information detection are, for example, configured by a convolutional neural network, but the type of neural network can be appropriately selected and used. Also, in the specific section determination processing unit 14 and the attention video detection processing unit 15, the specific section determination process and the attention video detection process may each be performed using a machine learning model.

(Example of operation)
Next, an example of the operation of the interest video detecting device BD configured as above will be described.
In the following description, it is assumed that the machine learning model for object detection and the machine learning model for distance information detection have been trained in advance.

Figure 3 is a flowchart showing an example of the overall processing procedure and processing content of the attention video detection process performed by the control unit 1 of the attention video detection device BD.

(1) Acquisition of Video Data Under the control of the video data acquisition processing unit 11, the control unit 1 of the target video detection device BD acquires time-series video data obtained by photographing the area to be monitored by the camera CM via the input/output I/F unit 4. Then, the acquired video data is stored in the video data storage unit 31 sequentially for each frame.

(2) Object Detection Once a certain period of video data has been acquired, the control unit 1 of the target video detection device BD, under the control of the object detection processing unit 12, performs the following process in step S1 to detect information representing the object class and position for each frame from the video data.

Figure 4 is a flowchart showing an example of the processing procedure and processing content of the object detection processing performed by the object detection processing unit 12.

That is, in step S11, the object detection processing unit 12 reads the video data frame by frame from the video data storage unit 31. The read video frames are then input to a machine learning model for object detection, and information representing the class and position of each object shown in the video frame is detected by the learning model.

For example, if "person", "ladder", and "safety belt" have been specified in advance as objects to be detected, then in step S12, the object detection learning model of the object detection processing unit 12 performs a detection process for each of the above objects from the video frame, and outputs class information represented as a binary vector in which a detected object is "1" and an undetected object is "0". For example, if a "person" and a "ladder" are detected in a current video frame, the class information (1, 1, 0) is output. Note that the vector length depends on the number of object classes.

On the other hand, the position of an object is represented by bounding box information. The bounding box information is represented by (x_min, y_min, x_max, y_max). This indicates the coordinates of the upper left corner of the rectangle (x_min, y_min) and the lower right corner (x_max, y_max) when the coordinates of the upper left corner of the video frame are (0, 0). The learning model for object detection of the object detection processing unit 12 outputs bounding box information for each object included in the above class information in step S13.

The object detection processing unit 12 outputs object detection information consisting of the class information and bounding box information output from the learning model for object detection to the specific section determination processing unit 14 in step S14.

In order to maintain the chronological consistency of a person, a person ID for identifying the person may be generated using a known person tracking technique, and the person ID may be included in the object detection information. Person tracking techniques are described in detail in, for example, K. Zhou et al. "Omni-Scale Feature Learning for Person Re-Identification." ICCV2019. Also, known techniques can be used to detect objects from video frames.

(3) Extraction of Distance Information In parallel with the above object detection process, the control unit 1 of the target video detection device BD, under the control of the distance information detection processing unit 13, executes a process in step S2 to extract the distance from the camera CM of the image captured in the above video frame in pixel units as follows.

Figure 5 is a flowchart showing an example of the processing procedure and processing content of the distance information detection processing performed by the distance information detection processing unit 13.

That is, the distance information detection processing unit 13 first reads the video data frame by frame from the video data storage unit 31 in step S21. The read video frames are then input to a machine learning model for distance information detection, and this learning model detects and outputs information representing the distance distribution of each pixel from the camera CM of the image captured in the video frames.

For example, the distance distribution is represented by a depth frame D. Depth frame D contains the corresponding depth information d(u, v) for each pixel p(u, v) in video frame I. Here, p and d are functions that return the pixel value and depth value corresponding to each pixel position on the video frame, and u and v represent the pixel position on the video frame.

In step S22, the machine learning model for distance information detection extracts a depth frame D corresponding to the input video data, and outputs the extracted depth frame D to the specific section determination processing unit 14 as distance detection information in step S23.

If a depth camera or the like is used as the camera CM, the depth frame may be acquired from the depth camera. In this case, however, the depth frame acquired from the depth camera needs to be aligned with the video frame in time and space.

(4) Determination of specific section The control unit 1 of the interest video detection device BD then, under the control of the specific section determination processing unit 14, in step S3, performs a process of determining a specific section in which a specified object is captured in a specified area in the video frame based on the above object detection information and distance detection information, as follows.

Figures 6A and 6B are flowcharts showing an example of the processing procedure and processing content of the specific section determination processing performed by the specific section determination processing unit 14.

6A, the specific section determination processing unit 14 first receives the object detection information and the distance detection information from the object detection processing unit 12 and the distance information detection processing unit 13. In addition, in step S312, the specific section video determination related information is read from the specific section video determination related information storage unit 32.

In step S313, the specific section determination processing unit 14 then compares the object detection information with the specific section video determination related information and determines whether any object is captured in the designated area of the detection target defined in the specific section video determination related information. If any object is captured in the designated area, the specified area is extracted as object area information.

Next, in step S314, the specific section determination processing unit 14 determines whether or not there is object area information in which a person is included among the plurality of detected object area information, and ends the process if there is no corresponding area information. On the other hand, if object area information in which a person is included is found, the specific section determination processing unit 14 next determines in step S315 whether or not there is object area information in which an object other than a person is included among the object area information in which the person is included, and ends the process if there is no corresponding object area information.

On the other hand, suppose that object region information containing an object is detected. In this case, in step S316, the specific section determination processing unit 14 extracts designated object region information including an object that matches the object specified as the detection target by the specific section video determination related information.

By performing the series of processes in steps S311 to S316 described above, designated object area information is detected in which the designated area contains at least one person area containing a person, and one or more object areas containing a designated object that may be the subject of work for the person.

(4-2) Detection of specific section video focusing on distance Next, in step S321 of FIG. 6B, the specific section determination processing unit 14 extracts distance detection information corresponding to the specified object region information from the distance detection information previously received in step S311. That is, distance detection information corresponding to each of the person region in which a person is photographed and the object region in which a specified object is photographed within the specified region is extracted. Here, the distance detection information corresponding to the person region and the object region is included as depth information in the depth frame corresponding to the bounding box of each region. That is, the distance detection information corresponding to each of the person region and the object region is a set of depth information d(u, v) included in the depth frame corresponding to the bounding box (x_min, y_min, x_max, y_max).

The specific section determination processing unit 14 then calculates the average distance for each of the person region and object region in step S322. This average distance is calculated as the average of the set of depth information d(u, v) included in the depth frame corresponding to the bounding box (x_min, y_min, x_max, y_max).

In step S323, the specific section determination processing unit 14 then generates a list H of N person regions in the specified object region information and a list O of M object regions other than person regions. In step S324, the specific section determination processing unit 14 initializes i of the person region h_i included in the list H to i=0, and in step S325, initializes j of the object region o_j included in the list O to j=0. In step S326, the specific section determination processing unit 14 calculates the difference E between the average distance of the person region h_0 included in the list H and the average distance of the object region o_0 included in the list O.

In step S327, the specific section determination processing unit 14 then compares the calculated average distance difference E with a threshold value T to determine whether the average distance difference E is equal to or less than the threshold value T. If the result of this determination is that the average distance difference E is equal to or less than the threshold value T, in step S328, the pair of the person area h_0 and the object area o_0 and the average distance difference E are added to the specific section video information. Note that if the average distance difference E exceeds the threshold value T, the above addition is not performed and the process proceeds to step S329.

In step S329, the specific section determination processing unit 14 then increments j of the object region information o_j (j+1) to select the next object region information o_1, and returns to step S326 to calculate the difference E between the average distance of the object region o_1 and the average distance of the person region h_0. Then, in step S327, it is determined whether the calculated difference E is equal to or less than a threshold value T, and if the difference E is equal to or less than the threshold value T, the pair of the person region information h_0 and object region information o_1 and the average distance difference E are added to the specific section video information, and if the difference E exceeds the threshold value T, they are not added.

In the same manner, in steps S326 to S330, the specific section determination processing unit 14 repeatedly calculates the average distance difference E between all M object regions o_j (j=2, ..., M) included in the object region information list O and the person region h_0, and compares it with the threshold value T. Then, if a pair of an object region o_j (j=2, ..., M) and a person region h_0 for which the difference E is equal to or smaller than the threshold value T is found, this pair is added to the specific section video information together with the average distance difference E.

The specific section determination processing unit 14 then increments the value of i of the person area h_i (i+1) in step S331 to select the next person area h_1. Then, the process returns to step S325 to initialize j of the object area o_j, and calculates the difference E between the average distance of the person area h_1 and the average distance of the object area o_0 in step S326, and determines whether the difference E is equal to or less than the threshold value T in step S327. If the difference E is equal to or less than the threshold value T, the pair of the person area h_1 and the object area o_0 and the difference E in the average distance are added to the specific section video information.

The specific section determination processing unit 14 thereafter repeats the process of calculating the average distance difference E for each combination of the person region h_1 and all M object regions o_j (j=1, ..., M) and comparing it with the threshold value T. Then, when a combination of the person region h_1 and the object region o_j (j=1, ..., M) for which the average distance difference E is equal to or less than the threshold value T is found, this combination and the average distance difference E are added to the specific section video information.

That is, in steps S325 to S332, the specific section determination processing unit 14 repeatedly executes the process of calculating the average distance difference E for all combinations of all N person regions h_i (i=0,...,N) and all M object regions o_j (j=0,...,M) and comparing it with a threshold value T for each combination. Then, when a combination of person region h_i (i=0,...,N) and object region o_j (j=0,...,M) for which the average distance difference E is equal to or less than the threshold value T is found, this pair is added to the specific section video information together with the average distance difference E.

In the above series of processes, the difference E in the average distance between the person area h_i (i=0,...,N) and the object area o_j (j=0,...,M) is used to determine whether or not to add it to the specific section video information, because a smaller difference in the average distance indicates a higher possibility of interaction occurring between a person and an object.

Finally, in step S333, the specific section determination processing unit 14 outputs the specific section video information obtained by the above series of processes to the attention video detection processing unit 15.

(5) Detection of Attention Video Next, in step S4, the control section 1 of attention video detection device BD, under the control of attention video detection processing section 15, executes a process of detecting an attention video from specific section video information as follows.

Figure 7 is a flowchart showing an example of the processing procedure and processing content of the attention video detection processing performed by the attention video detection processing unit 15.

That is, the attention video detection processing unit 15 first receives specific section video information from the specific section determination processing unit 14 in step S41. Next, in step S42, attention video detection processing unit 15 reads attention video detection related information from the attention video detection related information storage unit 33. Then, in step S43, the attention video detection processing unit 15 compares the specific section video information with the detection rule defined by the attention video detection related information, and determines whether or not there is specific section video information that matches the detection rule.

For example, suppose that detection rules such as those shown in FIG. 8 are defined in the attention video detection related information. For each specific section video included in the specific section video information, the attention video detection processing unit 15 determines whether or not the combination of the person area and the object area and the difference in the average distance correspond to any of the above detection rules. This determination also takes into account the distance on the video frame between the person area and the object area. Note that the distance on the video frame between the person area and the object area is expressed as the Euclidean distance between the center point of the bounding box of the person area and the center point of the bounding box of the object area.

If the attention video detection processing unit 15 finds a specific section video that matches any of the above detection rules as a result of the above judgment, it outputs the specific section video as an attention video from the input/output I/F unit 4 to the terminal MT in step S44. On the other hand, the specific section video that does not match any of the above detection rules is discarded.

(Action and Effects)
As described above, in one embodiment, for each frame of video data acquired from the camera CM, the class and position of an object captured in the video frame are detected as object detection information, and the distance distribution from the camera CM of the image captured in the video frame is detected as distance detection information. Then, from the object detection information, all object area information including a person and an object to be worked on, which are included in a designated area defined as a detection target by the specific section video related information, is detected, and further, based on the distance detection information, all specific section videos including a pair of a person and an object in which the difference in the average distance between the person and the object is equal to or less than a threshold are detected from each object area information. Then, from each detected specific section video, a specific section video that corresponds to a detection rule predefined by the attention video detection related information is detected, and the detected specific section video is output as an attention video.

Therefore, according to one embodiment, a video that includes a person designated as a detection target and a surrounding object that the person is working on, and in which the distance between the person and the object is within a threshold, is detected as a video of interest. As a result, when analyzing the behavior of a person based on the video of interest, it is possible to estimate the behavior of the person with high accuracy by taking into account the interaction between the person and the surrounding object that is the subject of the person's work.

[Other embodiments]
In the above embodiment, an example was described in which an image area including a pair of a person and a surrounding object that is a work target, in which the difference in distance between the person and the surrounding object is equal to or less than a threshold, is set as the image of interest. However, the detection target is not limited to a combination of a person and an object, and it is also possible to detect, for example, an image including a combination of a person to be monitored and another person who works with the person, or a combination of an animal or a robot and a surrounding object related to the animal or robot, as the image of interest.

In addition, in the above embodiment, an example was given of a case where an image in which the difference in distance between a person and surrounding objects is below a threshold value is set as the image of interest, but an image that satisfies the above condition may be set as the image of interest if the image continues for a predetermined period of time or longer.

In the above embodiment, the functions of the attention video detection device BD are provided in an information processing device such as a server computer or a personal computer that is provided independently of the camera CM and the terminal MT. However, the present invention is not limited to this, and all or part of the functions of the attention video detection device BD may be provided in the camera CM and the terminal MT.
Furthermore, the type, function, processing procedure, processing contents, etc. of the target image detecting device BD can be modified in various ways without departing from the spirit and scope of the present invention.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, in implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

In short, this invention is not limited to the above-described embodiment as it is, and in the implementation stage, the components can be modified and embodied without departing from the gist of the invention. Furthermore, various inventions can be formed by appropriate combinations of multiple components disclosed in the above-described embodiment. For example, some components may be deleted from all of the components shown in the embodiment. Furthermore, components from different embodiments may be appropriately combined.

BD...interesting video detection device CM...camera MT...terminal 1...control section 2...program storage section 3...data storage section 4...input/output I/F section 5...bus 11...video data acquisition processing section 12...object detection processing section 13...distance information detection processing section 14...specific section determination processing section 15...interesting video detection processing section 31...video data storage section 32...specific section video determination related information storage section 33...interesting video detection related information storage section

Claims (5)

A first processing unit that acquires video data captured by a camera;
a second processing unit that detects object detection information from a video frame of the video data, the object detection information including information representing an object and a position of the object in the video frame;
a third processing unit that extracts depth information representing a distance distribution from the camera of an image of the object depicted in the video frame from the video frame of the video data, and obtains distance detection information by calculating a distance of the object from the camera based on the extracted depth information;
a fourth processing unit that detects a specific image area including a first object that is an attention target and a second object that is a work target of the first object based on the object detection information and the distance detection information, the specific image area including a combination of the first object and the second object, the difference in distance between the first object and the second object from the camera being equal to or smaller than a predetermined value, and information indicating the difference in distance;
a fifth processing unit that detects, as an attention video , a combination of the first object and the second object contained in the specific video area and the specific video area in which the difference in distance between the first object and the second object satisfies a preset detection condition. The video attention detection device according to claim 1, wherein the second processing unit outputs, as the object detection information, class information representing the presence or absence of the object and bounding box information representing an area in the video frame in which the object exists. The attention video detection device according to claim 1 or 2, wherein the first object is a person, and the second object is an object that is the subject of the person's work. An attention video detection method performed by an information processing device,
acquiring image data captured by a camera;
detecting object detection information from a video frame of the video data, the object detection information including information representing an object and a position of the object in the video frame;
extracting depth information representing a distance distribution from the camera of an image of the object depicted in the video frame from the video frame of the video data, and calculating a distance of the object from the camera based on the extracted depth information to obtain distance detection information;
a step of detecting a specific image area including a first object to be focused on and a second object to be worked on by the first object based on the object detection information and the distance detection information , the specific image area including a combination of the first object and the second object, the difference in distance between the first object and the second object from the camera being equal to or smaller than a predetermined value, and information representing the difference in distance;
A method for detecting an image of interest comprising: a step of detecting, as an image of interest , a combination of the first object and the second object contained in the specific image area, and the specific image area in which the difference in distance between the first object and the second object satisfies a predetermined detection condition. 4. A program for causing a processor included in the video detection device of interest according to claim 1 to execute a process performed by at least one of the first processing unit to the fifth processing unit included in the video detection device of interest.

Images