JP7582833B2 - Situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device - Google Patents

Description

The present invention relates to a situation detection system, a situation detection method, a situation detection program, a trained model generation system, and a situation detection device.

For example, Patent Document 1 discloses an information processing device as a conventional technology for performing various detections from video images of a drive recorder or the like using artificial intelligence and deep learning. This information processing device includes an acquisition unit, a traffic light area recognition unit, a speed information acquisition unit, an acceleration information acquisition unit, and a judgment unit. The acquisition unit acquires a captured image captured by a camera mounted on the vehicle. The traffic light area recognition unit recognizes a red light area indicating a red light of a traffic light from the captured image acquired by the acquisition unit. The speed information acquisition unit acquires speed information indicating the speed of the vehicle. The acceleration information acquisition unit acquires acceleration information indicating the acceleration of the vehicle. The judgment unit judges whether or not the vehicle is running a red light based on the speed information or acceleration information, the red light area, and red light running identification information that is predetermined to identify that the vehicle is running a red light. In this case, the red light running identification information is created in advance using a machine learning method that uses an SVM (Support Vector Machine).

JP 2018-072940 A

However, the information processing device described in Patent Document 1 above has room for further improvement, for example, in terms of detecting more detailed situations.

The present invention has been made in consideration of the above circumstances, and aims to provide a situation detection system, a situation detection method, a situation detection program, a trained model generation system, and a situation detection device that can perform appropriate situation detection.

In order to achieve the above object, the situation detection system according to the present invention includes a first learning dataset including video data representing a video captured from a moving object and teacher data regarding object detection features representing an object detected in the video data, a second learning dataset including measurement data representing measurement values corresponding to the state of the moving object measured in conjunction with the capture of the video and the object detection features, teacher data regarding moving object behavior detection features representing the movement of the moving object accompanying a change in the measurement data over time, and teacher data regarding object behavior detection features representing the movement of the object accompanying a change in the object detection features over time, and a first learning dataset including measurement data representing a measurement value corresponding to the state of the moving object measured in conjunction with the capture of the video data and teacher data regarding object behavior detection features representing the movement of the object accompanying a change in the object detection features over time. a pre-processing unit that acquires a third learning dataset including the motion detection feature and the object motion detection feature, and training data regarding a related action feature that represents a behavior that occurs in relation to the motion of the moving body and the motion of the object, and a fourth learning dataset including the related action feature and training data regarding related action type data that represents a type of the behavior; an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection feature from the video image data using the multiple first learning datasets acquired by the pre-processing unit; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the plurality of third training data sets acquired by the preprocessing unit; a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the plurality of third training data sets acquired by the preprocessing unit; and a trained type identification model that identifies the type of the behavior from the related behavior feature using the plurality of fourth training data sets acquired by the preprocessing unit. a type identification model generation unit that generates a type identification model by a type identification model generation unit that generates a type identification model by a type identification model generation unit, a detection target input unit that inputs the video data and the measurement data to be detected, and a type identification unit that uses the trained object detection model generated by the object detection model generation unit, the trained behavior detection model generated by the behavior detection model generation unit, the trained related behavior detection model generated by the related behavior detection model generation unit, and the trained type identification model generated by the type identification model generation unit to identify the type of behavior from the video data and the measurement data input by the detection target input unit.

In addition, in the above situation detection system, the second learning dataset includes, in addition to the measurement data and the object detection feature, distance data representing a distance between the moving object and an object surrounding the moving object measured in conjunction with the shooting of the video, teacher data regarding the moving object behavior detection feature, and in addition to the teacher data regarding the object behavior detection feature, teacher data regarding a distance change feature representing a change in the distance associated with a change in the distance data over time; the third learning dataset includes the moving object behavior detection feature, the object behavior detection feature, and the distance change feature, and teacher data regarding the related behavior feature; and the behavior detection model generation unit generates, by learning, the learned behavior detection model that extracts the moving object behavior detection feature, the object behavior detection feature, and the distance change feature from the measurement data, the object detection feature, and the distance data using the multiple second learning datasets acquired by the preprocessing unit. The related behavior detection model generation unit generates a learned related behavior detection model by learning, which extracts the related behavior feature from the mobile object behavior detection feature, the object behavior detection feature, and the distance change feature, using the third learning data sets acquired by the preprocessing unit, and the detection target input unit inputs the video data, the measurement data, and the distance data to be detected, and the type identification unit uses the learned object detection model generated by the object detection model generation unit, the learned behavior detection model generated by the behavior detection model generation unit, the learned related behavior detection model generated by the related behavior detection model generation unit, and the learned type identification model generated by the type identification model generation unit to identify the type of behavior from the video data, the measurement data, and the distance data input by the detection target input unit.

In order to achieve the above object, the situation detection method according to the present invention includes a step of acquiring a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data, a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured in conjunction with the capture of the video data and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object accompanying a time change in the measurement data and teacher data related to object behavior detection features representing the movement of the object accompanying a time change in the object detection features, a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in conjunction with the movement of the moving object and the movement of the object, and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior, and using the acquired multiple first learning datasets to detect the object from the video data. The method includes a step of generating, by learning, a trained object detection model that extracts features from the measurement data and the object detection features using the acquired multiple second training data sets, a step of generating, by learning, a trained behavior detection model that extracts the mobile object behavior detection features and the object behavior detection features from the measurement data and the object detection features using the acquired multiple third training data sets, a step of generating, by learning, a trained related behavior detection model that extracts the related behavior features from the mobile object behavior detection features and the object behavior detection features using the acquired multiple fourth training data sets, a step of generating, by learning, a trained type identification model that identifies the type of the behavior from the related behavior features using the acquired multiple fourth training data sets, a step of inputting the video data and the measurement data to be detected, and a step of identifying the type of the behavior from the input video data and the measurement data using the generated trained object detection model, the trained behavior detection model, the trained related behavior detection model, and the trained type identification model.

In order to achieve the above object, the situation detection program of the present invention acquires a first learning dataset consisting of video data representing a video captured from a moving object and teacher data on object detection features representing an object detected in the video data, a second learning dataset consisting of measurement data representing a measurement value corresponding to the state of the moving object measured in conjunction with the capture of the video and the object detection features, teacher data on moving object behavior detection features representing the movement of the moving object accompanying a time change in the measurement data and teacher data on object behavior detection features representing the movement of the object accompanying a time change in the object detection features, a third learning dataset consisting of the moving object behavior detection features and the object behavior detection features, and teacher data on related behavior features representing behaviors occurring in conjunction with the movement of the moving object and the movement of the object, and a fourth learning dataset consisting of the related behavior features and teacher data on related behavior type data representing a type of the behavior, and uses the acquired multiple first learning datasets to acquire the video data. the computer executes each of the following processes: generating, by learning, a trained object detection model that extracts the object detection feature from the measurement data, using the acquired multiple second training datasets, generating, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature, using the acquired multiple third training datasets, generating, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature, using the acquired multiple fourth training datasets, generating, by learning, a trained type identification model that identifies the type of the behavior from the related behavior feature, inputting the video data and the measurement data to be detected, and using the generated trained object detection model, the trained behavior detection model, the trained related behavior detection model, and the trained type identification model to identify the type of the behavior from the input video data and the measurement data.

In order to achieve the above object, the trained model generation system of the present invention includes a first learning dataset including video data representing a video captured from a moving object and training data on object detection features representing an object detected in the video data, a second learning dataset including measurement data representing measurement values corresponding to the state of the moving object measured in conjunction with the capture of the video and the object detection features, training data on moving object behavior detection features representing the movement of the moving object accompanying a change in the measurement data over time, and training data on object behavior detection features representing the movement of the object accompanying a change in the object detection features over time, a third learning dataset including the moving object behavior detection features and the object behavior detection features, and training data on related behavior features representing behaviors occurring in conjunction with the movement of the moving object and the movement of the object, and a preprocessing unit that acquires a fourth learning dataset including the related behavior features and training data on related behavior type data representing the type of the behavior. and an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection feature from the video image data using the multiple first training data sets acquired by the preprocessing unit; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the multiple second training data sets acquired by the preprocessing unit; a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the multiple third training data sets acquired by the preprocessing unit; and a type identification model generation unit that generates, by learning, a trained type identification model that identifies the type of the behavior from the related behavior feature using the multiple fourth training data sets acquired by the preprocessing unit.

In order to achieve the above object, the situation detection device according to the present invention is characterized by comprising: a detection target input unit that inputs video data representing a video captured from a moving object and measurement data representing a measurement value corresponding to the state of the moving object measured in conjunction with the capture of the video; a trained object detection model that has been trained to extract from the video data an object detection feature that represents an object detected in the video data; a trained behavior detection model that has been trained to extract from the measurement data and the object detection feature a moving object behavior detection feature that represents the movement of the moving object accompanying a time change in the measurement data and an object behavior detection feature that represents the movement of the object accompanying a time change in the object detection feature; a trained relational behavior detection model that has been trained to extract from the moving object behavior detection feature and the object behavior detection feature a relational behavior feature that represents a behavior occurring in relation between the movement of the moving object and the movement of the object; and a trained type identification model that has been trained to identify the type of the behavior from the relational behavior feature.

In order to achieve the above object, the situation detection system according to the present invention includes an object detection model generation unit that generates, by learning, a trained object detection model that extracts object detection features representing objects detected in video data representing video images captured from a moving object, from the video data representing the video images; a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts, from measurement data representing measurement values corresponding to the state of the moving object measured in the course of capturing the video images and the object detection features, mobile object behavior detection features representing the movement of the moving object associated with a change in the measurement data over time and object behavior detection features representing the movement of the object associated with a change in the object detection features over time; and a behavior detection model generation unit that extracts, from the mobile object behavior detection features and object behavior detection features, a relationship behavior feature that represents a behavior occurring in relation to the movement of the moving object and the movement of the object. the related behavior detection model generation unit that generates, by learning, a learned related behavior detection model for identifying the type of the behavior from the related behavior feature; a type identification model generation unit that generates, by learning, a learned type identification model for identifying the type of the behavior from the related behavior feature; a detection target input unit that inputs the video data and the measurement data to be detected; and a type identification unit that identifies the type of the behavior from the video data and the measurement data input by the detection target input unit using the learned object detection model generated by the object detection model generation unit, the learned behavior detection model generated by the behavior detection model generation unit, the learned related behavior detection model generated by the related behavior detection model generation unit, and the learned type identification model generated by the type identification model generation unit.

The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the present invention have the effect of being able to perform appropriate situation detection.

FIG. 1 is a block diagram showing a schematic configuration of a situation detection system according to a first embodiment. FIG. 2 is a schematic diagram illustrating object detection in the situation detection system according to the first embodiment. FIG. 3 is a schematic diagram illustrating object detection in the situation detection system according to the first embodiment. FIG. 4 is a schematic diagram illustrating behavior detection in the situation detection system according to the first embodiment. FIG. 5 is a schematic diagram illustrating related behavior detection in the situation detection system according to the first embodiment. FIG. 6 is a schematic diagram showing the processing of the learning phase and the usage phase performed by the processing circuit of the situation detection system according to the first embodiment. FIG. 7 is a block diagram illustrating a schematic configuration of the type identification unit of the situation detection system according to the first embodiment. FIG. 8 is a schematic diagram illustrating an example of related behavior detection in the situation detection system according to the first embodiment. FIG. 9 is a schematic diagram illustrating an example of related behavior detection in the situation detection system according to the first embodiment. FIG. 10 is a flowchart illustrating an example of processing performed by the processing circuit of the situation detection system according to the embodiment. FIG. 11 is a block diagram showing a schematic configuration of a situation detection system according to a modified example. FIG. 12 is a block diagram showing a schematic configuration of a situation detection system according to a modified example. FIG. 13 is a block diagram illustrating a schematic configuration of a type identification unit of a situation detection system according to the second embodiment. FIG. 14 is a flowchart showing an example of processing performed by a processing circuit of a situation detection system according to a modified example.

Below, an embodiment of the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to this embodiment. Furthermore, the components in the following embodiment include those that are easily replaceable by a person skilled in the art, or those that are substantially the same.

[Embodiment 1]
1 is a system for detecting a moving object V and its surrounding situation using images. The moving object V that is the subject of situation detection by the situation detection system 1 is typically a vehicle traveling on a road surface, and may be, for example, any of a private car, a rental car, a sharing car, a ride-sharing car, a bus, a taxi, a truck, a transport vehicle, a work vehicle, and the like.

Here, examples of detection techniques using images include object detection and action detection.

As shown in Figure 2, object detection is the detection of the type and position of an object from still images that make up a moving image. More specifically, object detection is the detection of the static position and type of an object at a certain moment from still images. Objects detected by object detection include, for example, vehicles, pedestrians, bicycles, obstacles, street lights, signs, utility poles, signs, stop lines, etc. Figure 2 shows an example of a case where a "bicycle" and a "stop line" are detected as objects by object detection in a still image, and the positions of the detected "bicycle" and "stop line" are indicated by rectangular frames. In addition, this object detection can also detect changes in the position of an object by comparing multiple still images that make up a moving image, as shown in Figure 3. As an example, Figure 3 shows a case where a "bicycle" and a "stop line" are detected as objects by object detection in multiple still images that make up a moving image, and a change in the position of the "bicycle" from right to left is detected.

On the other hand, as shown in FIG. 4, behavior detection is to detect the movement (behavior) of an object alone from the change over time in the position, state, etc. of an object detected by object detection from multiple still images that compose a moving image. In other words, behavior detection is to detect the movement (behavior) of an object with time change. In this case, for example, a detected object that is stationary at its position (in other words, not moving), such as a "stop line," is also included in the concept of the movement of the object as an example of the movement (behavior) of the object. In behavior detection, for example, even if multiple objects are detected from still images that compose a moving image, the movement of each object alone is detected. FIG. 4 shows, as an example, a case where two "bicycles" are detected by object detection in multiple still images that compose a moving image, and then the movement of the two "bicycles" alone, "each of them running unsteadily," is detected by behavior detection.

This behavior detection detects the state or change of an object alone (for example, the unsteady movement of a "bicycle") and can predict danger to, for example, the driver of the moving body V, but it cannot associate multiple detected objects with each other to detect the meaning of their behavior. For this reason, for example, in the example of Figure 4, both of the two unsteady-moving "bicycles" will be judged to be dangerous regardless of their relationship to other objects.

In contrast, the situation detection system 1 of this embodiment uses relational action detection in addition to object detection and action detection to detect relationships between moving bodies V and objects detected from video images, and correlates these movements to detect the meaning (type) of their mutual behavior, thereby achieving more detailed situation detection.

As used herein, the relational behavior detection is, as shown in FIG. 5, a detection of behaviors (behavior) that arise due to the relation between multiple objects, and the relationship between objects, from the movement of the object alone and the movement of the moving body V detected by behavior detection from the multiple still images that make up the moving image. In other words, the relational behavior detection is a detection of behaviors that arise due to the relation between the movements of multiple objects, including the moving body V, that are associated with each other. The behavior that arises due to the relation between the movements of multiple objects can also be rephrased as, for example, "an event that occurs in response to a change in the relationship between multiple objects." As an example, FIG. 5 shows a case where two "bicycles" are detected by object detection in the multiple still images that make up the moving image, and the movement of the two "bicycles" alone, that is, "each of them is running unsteadily," is detected by behavior detection. FIG. 5 also shows a case where the relational behavior detection detects behaviors in which the "bicycle" on the front side that is running unsteadily is heading in the direction of travel of the vehicle (moving body V) that is about to turn left, while the "bicycle" on the back side is heading in a direction different from the direction of travel of the vehicle.

The situation detection system 1 of this embodiment detects behavior (movement) that occurs as a result of the related movements of multiple objects, including the moving body V, using the above-mentioned related behavior detection, and identifies the meaning (type) of the behavior. As a result, for example, in the example of FIG. 5, the situation detection system 1 can determine that the "bicycle" in the foreground heading in the direction of travel of the vehicle is dangerous, while determining that the "bicycle" in the background, which is swaying but heading in a different direction from the vehicle, is not dangerous.

As shown in FIG. 6, the situation detection system 1 of this embodiment has a learning phase in which a process for generating a trained model M is performed, and a usage phase in which the trained model M is used to actually perform the above-mentioned situation detection. The situation detection system 1 is realized by various computer devices, such as a personal computer, a workstation, or a tablet terminal. Each component of the situation detection system will be described in detail below with reference to FIGS. 1, 6, and 7.

Specifically, the situation detection system 1 includes an input device 10, an output device 20, a memory circuit 30, and a processing circuit 40. The input device 10, the output device 20, the memory circuit 30, and the processing circuit 40 are connected to each other so that they can communicate with each other via a network. The situation detection system 1 of this embodiment performs both the processes in the learning phase and the processes in the use phase. Here, the situation detection system 1 is configured by aggregating into a single system a trained model generation system that executes the processes in the learning phase and a situation detection device that executes the processes in the use phase.

The input device 10 is a device that performs various inputs to the situation detection system 1. The input device 10 is realized, for example, by an operation input device that accepts various operation inputs from a user, a data input device that accepts data (information) input from other devices outside the situation detection system 1, etc. The operation input device is realized, for example, by a mouse, a keyboard, a trackball, a switch, a button, a joystick, a touchpad, a touch screen, a non-contact input circuit, a voice input circuit, etc. The data input device is realized, for example, by a communication interface that transmits and receives various data to and from devices via communication, whether wired or wireless, a hard disk drive (HDD), a solid state drive (SSD), a flexible disk (FD), a magneto-optical disk (Magneto-Optical disk), a CD-ROM, a DVD, a USB memory, an SD card memory, a flash memory, etc.

In this embodiment, the input device 10 receives at least video data and measurement data from various devices.

The video data is data that represents a video image of the surroundings of the moving body V captured from the moving body V. The video image of the surroundings of the moving body V is captured, for example, by a camera 50 mounted on the moving body V, and is input from the camera 50 to the input device 10 as video image data.

Measurement data is data representing measurement values corresponding to the state of the moving body V measured in conjunction with the capture of a moving image. The measurement values are typically measured in the moving body V in synchronization with the capture of a moving image, and represent the state of the moving body V at the time of capture. The measurement values are, for example, physical quantities such as the speed of the moving body V, the acceleration/deceleration of the moving body V, the rotation speed of the power source for running the moving body V, and the amount of change in direction of the moving body V. The measurement values may also include the steering angle of the moving body V, the amount of operation of the accelerator pedal (amount of accelerator depression), the amount of operation of the brake pedal (amount of brake depression), and the like. The measurement values corresponding to the state of the moving body V are measured, for example, by a measuring device 51 such as a sensor or detector mounted on the moving body V, and are input from the measuring device 51 to the input device 10 as measurement data.

The camera 50 and the measuring device 51 may constitute, for example, on-board equipment such as a drive recorder mounted on the moving body V.

The output device 20 is a device that performs various outputs from the situation detection system 1. The output device 20 is realized, for example, by a display that outputs and displays various image information, a speaker that outputs sound information, a data output device that outputs data (information) to other devices outside the situation detection system 1, etc. The data output device is realized, for example, by a communication interface that transmits and receives various data between devices via communication, whether wired or wireless, a recording medium interface that writes various data to a recording medium similar to the above, etc.

Note that the data input device of the input device 10 and the data output device of the output device 20 may share some or all of their configurations.

The memory circuit 30 is a circuit that stores various data. The memory circuit 30 is realized, for example, by a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, a hard disk, an optical disk, or the like. The memory circuit 30 stores, for example, a program for the situation detection system 1 to realize various functions. The programs stored in the memory circuit 30 include a program for making the input device 10 function, a program for making the output device 20 function, a program for making the processing circuit 40 function, and the like. The memory circuit 30 also stores various data such as raw data D0 input via the input device 10, data necessary for various processes in the processing circuit 40, a learning data set D3 used for learning the learned model M, the learned model M, and type identification result data D5 output via the output device 20. The various data are read out from the memory circuit 30 by the processing circuit 40 or the like as necessary. The memory circuit 30 may be realized by a cloud server or the like connected to the situation detection system 1 via a network.

The processing circuit 40 is a circuit that realizes various processing functions in the situation detection system 1. The processing circuit 40 is realized, for example, by a processor. The processor refers to circuits such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array). The processing circuit 40 realizes each processing function, for example, by executing a program read from the memory circuit 30.

The above describes an outline of the overall configuration of the situation detection system 1 according to this embodiment. With this configuration, the processing circuit 40 according to this embodiment has a function for performing various processes to generate the trained model M in the learning phase. In addition, the processing circuit 40 according to this embodiment has a function for performing various processes using the trained model M in the usage phase.

Specifically, in order to realize the various processing functions described above, the processing circuitry 40 of this embodiment is functionally configured to include a preprocessing unit 41, a model generating unit 42, a detection target input unit 43, a type identifying unit 44, and an output unit 45. The processing circuitry 40 realizes each of the processing functions of the preprocessing unit 41, the model generating unit 42, the detection target input unit 43, the type identifying unit 44, and the output unit 45, for example, by executing a program read from the memory circuitry 30.

The preprocessing unit 41 is a part that has the function of performing various preprocessing operations on data for training the trained model M during the learning phase. In this embodiment, the preprocessing unit 41 can perform a process of acquiring a learning dataset D3 consisting of input data D1, which is an explanatory variable, and teacher data D2, which is a target variable.

The learning dataset D3 acquired by the preprocessing unit 41 is a teacher dataset used when generating the trained model M by machine learning. The learning dataset D3 is configured by linking together, as a set, input data D1 that serves as explanatory variables and teacher data D2 that serves as a target variable corresponding to the input data D1. More specifically, the learning dataset D3 is configured from the input data D1 quantified as an explanatory variable and the teacher data D2 quantified as a target variable.

The pre-processing unit 41 can, for example, perform various pre-processing operations on raw data D0 input from other devices outside the situation detection system 1, and create and acquire a learning dataset D3 by linking the input data D1 and the teacher data D2 corresponding to the input data D1 as a set.

The raw data D0 to be preprocessed by the preprocessing unit 41 may be input from another device outside the situation detection system 1 via a data input device constituting the input device 10, or may be input by a user's operation via an operation input device constituting the input device 10. In this embodiment, the raw data D0 includes at least the above-mentioned video image data and measurement data.

The preprocessing performed by the preprocessing unit 41 on the raw data D0 includes, for example, a process of collecting and combining the raw data D0, a process of extracting the input data D1 from the raw data D0 and quantifying it as an explanatory variable, a process of annotating the raw data D0 (a process of adding metadata to the data to make it valid as teacher data D2) to create teacher data D2 and quantifying it as a target variable, and a process of linking and combining the quantified input data D1 and the quantified teacher data D2. For example, the preprocessing unit 41 may perform preprocessing on the raw data D0 each time the raw data D0 is input, or may perform preprocessing on the raw data D0 at an appropriate timing in response to a user's operation via an operation input device constituting the input device 10.

The preprocessing unit 41 may also acquire a pre-created learning data set D3 from another device outside the situation detection system 1, for example, via a data input device constituting the input device 10. Here, the other device outside the situation detection system 1 is, for example, a learning data providing system that stores a large amount of input data D1, teacher data D2, learning data set D3, etc. in a database.

In this embodiment, the training dataset D3 acquired by the preprocessing unit 41 includes a first training dataset D31, a second training dataset D32, a third training dataset D33, and a fourth training dataset D34. That is, the preprocessing unit 41 in this embodiment acquires the first training dataset D31, the second training dataset D32, the third training dataset D33, and the fourth training dataset D34 as the training dataset D3.

The first learning dataset D31 is a teacher dataset used when generating the learned object detection model M1 by machine learning as the learned model M. The learned object detection model M1 is a learned model M used to extract object detection features from video data. Here, the object detection feature is a feature representing an object detected in video data, and represents the static position and type of an object at a certain moment in a still image constituting the video. The process of extracting the object detection feature from video data using the learned object detection model M1 corresponds to the above-mentioned process related to object detection. The first learning dataset D31 is configured by linking "video data" and "teacher data related to object detection features corresponding to the video data" as a set. In other words, the first learning dataset D31 consists of the video data quantified as input data D1 (explanatory variable) and the teacher data related to the object detection feature quantified as teacher data D2 (target variable).

The second learning dataset D32 is a teacher dataset used to generate the learned behavior detection model M2 by machine learning as the learned model M. The learned behavior detection model M2 is a learned model M used to extract moving object behavior detection features and object behavior detection features from measurement data and object detection features. Here, the moving object behavior detection feature is a feature that represents the movement (behavior) of the moving object V accompanying the time change of the measurement data, and represents the characteristics of the movement of the moving object alone accompanying the time change. The object behavior detection feature is a feature that represents the movement (behavior) of the object accompanying the time change of the object detection feature, in other words, the movement of the object in the still image that constitutes the moving image, and represents the characteristics of the movement of the object alone accompanying the time change. The process of extracting the moving object behavior detection feature and the object behavior detection feature from the measurement data and object detection feature using the learned behavior detection model M2 corresponds to the above-mentioned process related to behavior detection. The second learning dataset D32 is configured by linking together as a set "measurement data and object detection features" and "teacher data on mobile object behavior detection features corresponding to the measurement data and teacher data on object behavior detection features corresponding to the object detection features." In other words, the second learning dataset D32 is composed of the measurement data and the object detection features quantified as input data D1 (explanatory variables), and the teacher data on the mobile object behavior detection features and the teacher data on the object behavior detection features quantified as teacher data D2 (target variable).

Here, the learned behavior detection model M2 is described as a learned model M that extracts both moving object behavior detection features and object behavior detection features from measurement data and object detection features, but is not limited to this, and may be composed of, for example, multiple sub-models. That is, the learned behavior detection model M2 may be composed of a moving object behavior detection sub-model that extracts moving object behavior detection features from measurement data, and an object behavior detection sub-model that extracts object behavior detection features from object detection features. In this case, the second learning dataset D32 may be composed of a moving object behavior detection learning sub-dataset consisting of measurement data and teacher data regarding moving object behavior detection features that represent the movement of a moving object V accompanying a time change in the measurement data, and an object behavior detection sub-dataset consisting of object detection features and teacher data regarding object behavior detection features that represent the movement of an object accompanying a time change in the object detection features.

The third learning dataset D33 is a teacher dataset used to generate the learned relational behavior detection model M3 by machine learning as the learned model M. The learned relational behavior detection model M3 is a learned model M used to extract relational behavior features from moving body behavior detection features and object behavior detection features. Here, the relational behavior feature is a feature that represents the behavior (behavior) that occurs in relation between the movement of the moving body V and the movement of the object in the still image that constitutes the moving image, and represents the characteristics of the behavior that occurs due to the change in the relationship associated with the time change of each of the multiple objects including the moving body V. In other words, the relational behavior feature can also be said to be a feature that represents how the relationship between the movements of the multiple objects including the moving body V changes over time. The process of extracting the relational behavior feature from the moving body behavior detection feature and the object behavior detection feature using the learned relational behavior detection model M3 corresponds to the process related to the above-mentioned relational behavior detection. The third learning dataset D33 is configured by linking together a set of "mobile object behavior detection features and object behavior detection features" and "teacher data on related behavior features corresponding to the mobile object behavior detection features and the object behavior detection features." In other words, the third learning dataset D33 is composed of the mobile object behavior detection features and object behavior detection features quantified as input data D1 (explanatory variables) and the teacher data on the related behavior features quantified as teacher data D2 (target variable).

The fourth learning dataset D34 is a teacher dataset used when generating the learned type identification model M4 by machine learning as the learned model M. The learned type identification model M4 is a learned model M used to identify the type of behavior from the related behavior feature. Here, the type of behavior represents the type and meaning of the behavior (behavior) that occurs in relation to the movement of the moving body V and the movement of an object in a still image that constitutes a moving image, and represents the type and meaning of the characteristics of the behavior represented by the related behavior feature. The process of identifying the type of behavior from the related behavior feature using the learned type identification model M4 corresponds to the process of identifying the type of behavior that occurs in relation to the movement of the moving body V and the movement of an object in a still image that constitutes a moving image in the above-mentioned related behavior detection. The fourth learning dataset D34 is configured by linking the "related behavior feature" and "teacher data related to the related behavior type data that represents the type of behavior corresponding to the related behavior feature" as a set. In other words, the fourth learning dataset D34 consists of the relevant behavior feature amount quantified as input data D1 (explanatory variable) and training data regarding the relevant behavior type data quantified as training data D2 (objective variable).

The preprocessing unit 41 stores the acquired multiple learning data sets D3 (first learning data set D31, second learning data set D32, third learning data set D33, and fourth learning data set D34) in the memory circuit 30.

The model generation unit 42 is a part having a function capable of executing a process of generating a trained model M by machine learning in the learning phase, which identifies the type of behavior that occurs in relation to the movement of the moving body V and the movement of the object from the video image data and the measurement data. The model generation unit 42 of this embodiment is capable of executing a process of generating a trained model M by machine learning using multiple training data sets D3 acquired by the preprocessing unit 41. For example, the model generation unit 42 may train and generate the trained model M each time a new training data set D3 is acquired, or may train and generate the trained model M at an appropriate timing in response to a user's operation via an operation input device that constitutes the input device 10.

The model generation unit 42 generates a trained model M by performing machine learning based on various machine learning algorithms AL using multiple learning datasets D3 as teacher datasets. Examples of the machine learning algorithms AL used include well-known algorithms such as deep learning, neural network, logistic regression, ensemble learning, support vector machine, random forest, and naive bays. The model generation unit 42 performs machine learning of the trained model M using the input data D1 of the training dataset D3 as an explanatory variable and the teacher data D2 as an objective variable. As a result of the machine learning, the model generation unit 42 generates a trained model M that has been subjected to machine learning to identify the type of the above behavior from the video image data and the measurement data.

The trained model M is realized, for example, by a neural network. As the neural network, for example, a neural network having any configuration such as a convolution neural network (CNN), a recurrent neural network (RNN), or a long short-term memory (LSTM) network can be adopted. In this case, the model generation unit 42 performs machine learning using multiple training data sets D3 to learn learning weighting coefficients and the like used as weights in the neural network, and generates the trained model M.

More specifically, the model generation unit 42 of this embodiment is conceptually configured to include an object detection model generation unit 42a, a behavior detection model generation unit 42b, a related behavior detection model generation unit 42c, and a type-specific model generation unit 42d.

The object detection model generation unit 42a is a part that has a function of executing a process of generating a learned object detection model M1 by machine learning from the learned model M. The object detection model generation unit 42a is capable of executing a process of generating a learned object detection model M1 by machine learning using a plurality of first learning datasets D31 acquired by the preprocessing unit 41. The object detection model generation unit 42a performs machine learning of the learned object detection model M1 using "video image data" from the first learning dataset D31 as input data D1 (explanatory variable) and "teacher data related to object detection features" as teacher data D2 (target variable). As a result of the machine learning, the object detection model generation unit 42a can generate a learned object detection model M1 that has been subjected to machine learning for extracting object detection features from video image data.

The trained object detection model M1 generated by the object detection model generation unit 42a is a model in which the input is "video data" and the output is "object detection features." In other words, the trained object detection model M1 is functionalized to accept input of video data and output object detection features from the video data. More specifically, the trained object detection model M1 performs a calculation based on the learning weighting coefficients in the neural network for video data input to the input layer of the neural network, and causes the computer to function to output object detection features from the output layer of the neural network.

The behavior detection model generation unit 42b is a part having a function capable of executing a process of generating a learned behavior detection model M2 by machine learning from the learned model M. The behavior detection model generation unit 42b can execute a process of generating a learned behavior detection model M2 by machine learning using a plurality of second learning datasets D32 acquired by the preprocessing unit 41. The behavior detection model generation unit 42b performs machine learning of the learned behavior detection model M2 using the "measurement data and object detection feature amount" of the second learning dataset D32 as input data D1 (explanatory variables) and the "teacher data regarding the moving object behavior detection feature amount and the teacher data regarding the object behavior detection feature amount" as teacher data D2 (target variable). The behavior detection model generation unit 42b can generate a learned behavior detection model M2 that has been subjected to machine learning for extracting the moving object behavior detection feature amount and the object behavior detection feature amount from the measurement data and the object detection feature amount as a result of the machine learning.

The learned behavior detection model M2 generated by the behavior detection model generation unit 42b is a model whose input is "measurement data and object detection features" and whose output is "mobile object behavior detection features and object behavior detection features." That is, the learned behavior detection model M2 is functionalized to accept the input of measurement data and object detection features and output the mobile object behavior detection features and object behavior detection features from the measurement data and object detection features. More specifically, the learned behavior detection model M2 performs a calculation based on the learning weighting coefficients in the neural network for the measurement data and object detection features input to the input layer of the neural network, and causes the computer to function to output the mobile object behavior detection features and object behavior detection features from the output layer of the neural network.

In addition, as described above, when the learned behavior detection model M2 is configured to be divided into multiple sub-models (mobile behavior detection sub-model, object behavior detection sub-model), the behavior detection model generation unit 42b may generate the multiple sub-models individually by machine learning as the learned behavior detection model M2.

The related behavior detection model generation unit 42c is a part having a function capable of executing a process of generating a learned related behavior detection model M3 by machine learning from the learned model M. The related behavior detection model generation unit 42c is capable of executing a process of generating a learned related behavior detection model M3 by machine learning using a plurality of third learning datasets D33 acquired by the preprocessing unit 41. The related behavior detection model generation unit 42c performs machine learning of the learned related behavior detection model M3 using the "mobile object behavior detection feature and object behavior detection feature" from the third learning dataset D33 as input data D1 (explanatory variables) and the "teacher data related to the related behavior feature" as teacher data D2 (target variable). The related behavior detection model generation unit 42c can generate a learned related behavior detection model M3 that has been subjected to machine learning to extract related behavior features from the mobile object behavior detection feature and the object behavior detection feature as a result of the machine learning.

The learned relational behavior detection model M3 generated by the relational behavior detection model generation unit 42c is a model in which the input is "mobile object behavior detection feature and object behavior detection feature" and the output is "relevant behavior feature." That is, the learned relational behavior detection model M3 is functionalized to receive the input of the mobile object behavior detection feature and the object behavior detection feature, and output the relevant behavior feature from the mobile object behavior detection feature and the object behavior detection feature. More specifically, the learned relational behavior detection model M3 performs a calculation based on the learning weighting coefficient in the neural network for the mobile object behavior detection feature and the object behavior detection feature input to the input layer of the neural network, and causes the computer to function to output the relevant behavior feature from the output layer of the neural network.

The type-specific model generation unit 42d is a part having a function capable of executing a process of generating a learned type-specific model M4 by machine learning from the learned model M. The type-specific model generation unit 42d is capable of executing a process of generating a learned type-specific model M4 by machine learning using a plurality of fourth learning data sets D34 acquired by the preprocessing unit 41. The type-specific model generation unit 42d performs machine learning of the learned type-specific model M4 using the "related behavior feature amount" of the fourth learning data set D34 as input data D1 (explanatory variable) and the "teacher data related to related behavior type data" as teacher data D2 (target variable) from the fourth learning data set D34. The type-specific model generation unit 42d can generate a learned type-specific model M4 that has been subjected to machine learning to identify the type of the above-mentioned behavior from the related behavior feature amount as a result of the machine learning.

The learned type-specific model M4 generated by the type-specific model generation unit 42d is a model in which the input is a "related behavior feature" and the output is a "quantified value of the type of behavior that occurs when the movement of the moving body V and the movement of the object are related." In other words, the learned type-specific model M4 is functionalized to accept the input of the related behavior feature and output a quantified value of the type of the behavior from the related behavior feature. More specifically, the learned type-specific model M4 performs a calculation based on the learning weighting coefficient in the neural network for the related behavior feature input to the input layer of the neural network, and causes the computer to function to output a quantified value of the type of the behavior from the output layer of the neural network.

The model generation unit 42 stores the trained models M (trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, trained type identification model M4) generated as described above in the memory circuit 30. At this time, if a previously generated trained model M has already been stored in the memory circuit 30, the model generation unit 42 replaces the stored trained model M with the newly generated trained model M.

The detection target input unit 43 is a part that has a function capable of executing a process to input moving image data and measurement data to be detected during the use phase. Here, the moving image data and measurement data to be detected may be referred to as "detection target data (input data) D4." Furthermore, among the "detection target data D4," the moving image data may be referred to as "moving image data D41," and the measurement data may be referred to as "measurement data D42."

The detection target data D4 may be input from another device outside the situation detection system 1 via a data input device constituting the input device 10, or may be input by a user's operation via an operation input device constituting the input device 10. The detection target input unit 43 of this embodiment can execute a process of inputting the detection target data D4 (movie image data D41, measurement data D42) received via the input device 10 from the camera 50, the measurement device 51, etc. to the type identification unit 44. The detection target input unit 43 may input the detection target data D4 in real time in accordance with the movement of the moving body V, for example, or may input the detection target data D4 after the fact at an appropriate timing after the movement of the moving body V has ended. The detection target input unit 43 may temporarily store the input detection target data D4 in the memory circuit 30.

The type identification unit 44 is a part having a function of executing a process of identifying the type of behavior occurring in the use phase in relation to the movement of the moving body V and the movement of an object in a still image constituting the moving image, using the learned model M. The type identification unit 44 of this embodiment can execute a process of identifying the type of the behavior from the detection target data D4 that is the detection target input by the detection target input unit 43, using the learned model M generated by the model generation unit 42. More specifically, the type identification unit 44 can execute a process of identifying the type of the behavior from the moving image data D41 and the measurement data D42 input by the detection target input unit 43, using the learned object detection model M1 generated by the object detection model generation unit 42a, the learned behavior detection model M2 generated by the behavior detection model generation unit 42b, the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42c, and the learned type identification model M4 generated by the type identification model generation unit 42d.

Specifically, as shown in FIG. 7, the type identification unit 44 of this embodiment is functionally conceptually configured to include an object detection feature extractor 44a, an integration unit 44b, a behavior detection feature extractor 44c, a related behavior feature extractor 44d, and a classifier 44e.

The object detection feature extractor 44a is a part having a function capable of executing a process of extracting an object detection feature D61 from video data D41 using the trained object detection model M1 generated by the object detection model generation unit 42a. The object detection feature extractor 44a inputs the "video data D41" of the detection target data D4 input by the detection target input unit 43 as input data to the trained object detection model M1, and outputs the "object detection feature D61" from the trained object detection model M1 in response to the input data. As a result, the object detection feature extractor 44a extracts the object detection feature D61 representing an object detected in the video data D41 from the video data D41. The process of extracting the object detection feature D61 from the video data D41 by the object detection feature extractor 44a corresponds to the above-mentioned process related to object detection. The object detection feature extractor 44a stores the output object detection feature D61 in the memory circuit 30 and also inputs it to the next stage integration unit 44b.

The integration unit 44b is a part that has a function capable of executing a process of integrating the object detection feature D61 and the measurement data D42. The integration unit 44b integrates the "object detection feature D61" input by the object detection feature extractor 44a and the "measurement data D42" of the detection target data D4 input by the detection target input unit 43 so as to be a combination of the same time in a chronological order, stores it in the memory circuit 30, and inputs it to the behavior detection feature extractor 44c in the next stage.

The behavior detection feature extractor 44c is a part having a function capable of executing a process of extracting a moving object behavior detection feature D62 and an object behavior detection feature D63 from the measurement data D42 and the object detection feature D61 using the learned behavior detection model M2 generated by the behavior detection model generation unit 42b. The behavior detection feature extractor 44c inputs the "measurement data D42 and object detection feature D61" at the same time in time series input by the integration unit 44b as input data to the learned behavior detection model M2, and outputs the "moving object behavior detection feature D62 and object behavior detection feature D63" from the learned behavior detection model M2 in response to this. As a result, the behavior detection feature extractor 44c extracts a mobile object behavior detection feature D62 representing the movement of the mobile object V accompanying the time change of the measurement data D42 and an object behavior detection feature D63 representing the movement of the object accompanying the time change of the object detection feature D61 from the measurement data D42 and the object detection feature D61. The process of extracting the mobile object behavior detection feature D62 and the object behavior detection feature D63 from the measurement data D42 and the object detection feature D61 by the behavior detection feature extractor 44c corresponds to the process related to behavior detection described above. The behavior detection feature extractor 44c stores the output mobile object behavior detection feature D62 and object behavior detection feature D63 in the memory circuit 30 and inputs them to the related behavior feature extractor 44d in the next stage.

When the learned behavior detection model M2 is divided into multiple sub-models (mobile behavior detection sub-model, object behavior detection sub-model), the behavior detection feature extractor 44c inputs the measurement data D42 and the object detection feature D61 to each of the multiple sub-models, and outputs the mobile behavior detection feature D62 and the object behavior detection feature D63, respectively.

The related behavior feature extractor 44d is a part having a function capable of executing a process of extracting related behavior feature D64 from the moving object behavior detection feature D62 and the object behavior detection feature D63 using the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42c. The related behavior feature extractor 44d inputs the "moving object behavior detection feature D62 and object behavior detection feature D63" input by the behavior detection feature extractor 44c to the learned related behavior detection model M3 as input data, and outputs the "related behavior feature D64" from the learned related behavior detection model M3 in response to this. As a result, the related behavior feature extractor 44d extracts the related behavior feature D64 representing the behavior that occurs in relation to the movement of the moving object V and the movement of the object in the still image that constitutes the moving image from the moving object behavior detection feature D62 and the object behavior detection feature D63. The process of extracting the related behavior feature D64 from the mobile object behavior detection feature D62 and the object behavior detection feature D63 by the related behavior feature extractor 44d corresponds to the process related to the related behavior detection described above. The related behavior feature extractor 44d stores the output related behavior feature D64 in the memory circuit 30 and also inputs it to the next stage classifier 44e.

The classifier 44e is a part having a function capable of executing a process of identifying the type of behavior that occurs in relation to the movement of the moving body V and the movement of an object in a still image that constitutes a moving image from the related behavior feature D64, using the learned type-specific model M4 generated by the type-specific model generation unit 42d. The classifier 44e inputs the "related behavior feature D64" input by the related behavior feature extractor 44d as input data to the learned type-specific model M4 generated by the type-specific model generation unit 42d, and outputs a "quantified value of the type of the above behavior" from the learned type-specific model M4 in response to this. In this way, the classifier 44e identifies the type of the above behavior represented by the related behavior feature.

For example, in the example of FIG. 5 described above, the classifier 44e identifies "a dangerous behavior in which the bicycle in the foreground, which is moving unsteadily, is heading in the direction of travel of the host vehicle, which is trying to turn left," as a type of behavior that occurs when the movement of the moving body V and the movement of an object are related, based on the relationship between the movements of the "bicycle in the foreground" and the "host vehicle (moving body V)." On the other hand, the classifier 44e identifies "a non-dangerous behavior in which the bicycle in the background, which is moving unsteadily, is heading in a direction different from the direction of travel of the host vehicle, which is trying to turn left," as a type of behavior that occurs when the movement of the moving body V and the movement of an object are related, based on the relationship between the movements of the "bicycle in the background" and the "host vehicle (moving body V)."

As another example, in the example of FIG. 8, the classifier 44e identifies, as a type of behavior that occurs as a result of the relationship between the movement of the moving body V and the movement of an object, for example, "safe behavior that is not a traffic violation, in which the own vehicle stops on the stop line at a red light," based on the relationship between the movement of the "red light," "stop line," and "own vehicle (moving body V)." On the other hand, in the example of FIG. 9, the classifier 44e identifies, as a type of behavior that occurs as a result of the relationship between the movement of the moving body V and the movement of an object, for example, "dangerous behavior that is a traffic violation, in which the own vehicle does not stop on the stop line at a red light (ignoring a signal)," based on the relationship between the movement of the "red light," "stop line," and "own vehicle (moving body V)."

The classifier 44e stores the quantified value of the output type of the above behavior in the memory circuit 30 as type identification result data (output data) D5.

The output unit 45 is a part having a function capable of executing a process of outputting based on the result of the identification of the type of behavior by the type identification unit 44. The output unit 45 of this embodiment is capable of executing a process of outputting the type identification result data D5 identified by the type identification unit 44 via the output device 20. The type identification result data D5 may be output as image information via a display constituting the output device 20, or may be output as sound information via a speaker constituting the output device 20. In addition, the type identification result data D5 may be output to other devices outside the situation detection system 1 via a data output device constituting the output device 20. The output unit 45 may output the type identification result data D5 in real time in accordance with the running of the vehicle, for example, or may output the type identification result data D5 at an appropriate timing in response to a user's operation via an operation input device constituting the input device 10.

As an example, the output unit 45 may output an alert via the output device 20 when the type identification unit 44 identifies the type of behavior as "dangerous behavior," i.e., when the type identification result data D5 indicates "dangerous behavior."

Next, the processing steps of the situation detection method in the situation detection system 1 will be described with reference to the flowchart in FIG. 10.

The situation detection method in the situation detection system 1 shown in FIG. 10 includes an acquisition step (step S1), a model generation step (steps S2 to S5), an input step (step S6), a type identification step (step S7), an output step (step S8), a determination step (step S9), and an alarm step (step S10). Here, the processing for each of the above steps is executed by the processing circuit 40 of the situation detection system 1.

First, the preprocessing unit 41 of the processing circuit 40 executes a step (step S1) of acquiring the first learning data set D31, the second learning data set D32, the third learning data set D33, and the fourth learning data set D34 as the learning data set D3. In this case, the preprocessing unit 41 may directly acquire the first learning data set D31, the second learning data set D32, the third learning data set D33, and the fourth learning data set D34 from another device outside the situation detection system 1 via the input device 10. The preprocessing unit 41 may also create and acquire the first learning data set D31, the second learning data set D32, the third learning data set D33, and the fourth learning data set D34 by performing various preprocessing on the raw data D0 input from another device outside the situation detection system 1 via the input device 10. The preprocessing unit 41 stores the acquired first learning data set D31, second learning data set D32, third learning data set D33, and fourth learning data set D34 in the memory circuitry 30.

Next, the model generation unit 42 of the processing circuit 40 executes a model generation step (steps S2 to S5) in which a trained model M is generated by machine learning using the multiple training data sets D3 acquired in the acquisition step (step S1).

More specifically, the object detection model generation unit 42a of the model generation unit 42 executes an object detection model generation step (step S2) of generating a trained object detection model M1 by machine learning using a plurality of first learning datasets D31. The behavior detection model generation unit 42b of the model generation unit 42 executes a behavior detection model generation step (step S3) of generating a trained behavior detection model M2 by machine learning using a plurality of second learning datasets D32. The related behavior detection model generation unit 42c of the model generation unit 42 executes a related behavior detection model generation step (step S4) of generating a trained related behavior detection model M3 by machine learning using a plurality of third learning datasets D33. The type-specific model generation unit 42d of the model generation unit 42 executes a type-specific model generation step (step S5) of generating a trained type-specific model M4 by machine learning using a plurality of fourth learning datasets D34.

The model generation unit 42 stores the generated trained models M (trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, trained type identification model M4) in the memory circuit 30. At this time, if a previously generated trained model M has already been stored in the memory circuit 30, the model generation unit 42 replaces the stored trained model M with the newly generated trained model M.

Next, the detection target input unit 43 of the processing circuit 40 executes an input step (step S6) of inputting the moving image data D41 and the measurement data D42 as the detection target data D4 to be detected to the type identification unit 44 of the processing circuit 40. In this case, the detection target input unit 43 may input the detection target data D4 received from another device outside the situation detection system 1 via the input device 10, or may input the detection target data D4 received by user operation via the input device 10. In addition, the detection target input unit 43 may temporarily store the input detection target data D4 in the memory circuit 30.

Next, the type identification unit 44 of the processing circuit 40 executes a type identification step (step S7) of identifying the type of behavior that occurs in relation between the movement of the moving body V and the movement of the object in the still image that constitutes the moving image from the moving image data D41 and the measurement data D42 (detection target data D4) input in the input step (step S6) using the learned object detection model M1, the learned behavior detection model M2, the learned relational behavior detection model M3, and the learned type identification model M4 generated in the model generation step (steps S2 to S5). In this case, the type identification unit 44 identifies the type of the above-mentioned behavior from the moving image data D41 and the measurement data D42 by the object detection feature extractor 44a, the integration unit 44b, the behavior detection feature extractor 44c, the relational behavior feature extractor 44d, and the classifier 44e working together. The type identification unit 44 stores the value that quantifies the type of the identified behavior in the memory circuit 30 as type identification result data D5.

Next, the output unit 45 of the processing circuit 40 executes an output step (step S8) of outputting type identification result data D5 for the type of behavior identified in the type identification step (step S7). In this case, the output unit 45 may output the type identification result data D5 as image information or sound information via the output device 20, or may output the type identification result data D5 via the output device 20 to another device outside the situation detection system 1.

Here, the output unit 45 further executes a determination step (step S9) of determining whether the type of behavior identified by the type identification unit 44 is a behavior that requires an alarm, such as "dangerous behavior." If the output unit 45 determines that the type of behavior identified by the type identification unit 44 is not a behavior that requires an alarm (step S9: No), the output unit 45 ends the processing according to this flowchart. If the output unit 45 determines that the type of behavior identified by the type identification unit 44 is a behavior that requires an alarm (step S9: Yes), the output unit 45 executes an alarm step (step S10) of outputting an alarm via the output device 20, and ends the processing according to this flowchart.

The above-mentioned situation detection method can be realized by executing a prepared situation detection program on a computer such as a personal computer or a workstation. This situation detection program causes the computer to execute each of the above-mentioned acquisition step (step S1), model generation step (steps S2 to S5), input step (step S6), type identification step (step S7), output step (step S8), judgment step (step S9), and warning step (step S10).

The situation detection system 1, situation detection method, and situation detection program described above can perform object detection by extracting "object detection features" representing objects detected in "video data" from the video data using the learned object detection model M1. The situation detection system 1, situation detection method, and situation detection program can perform behavior detection by extracting "mobile object behavior detection features" representing the movement of a moving object and "object behavior detection features" representing the movement of an object from "measurement data and object detection features" using the learned behavior detection model M2. The situation detection system 1, situation detection method, and situation detection program can perform related behavior detection by extracting "related behavior features" representing behaviors that occur in relation to the movement of a moving object and the movement of an object from the "mobile object behavior detection features and object behavior detection features" extracted as described above using the learned related behavior detection model M3. The situation detection system 1, situation detection method, and situation detection program can identify the "type of the behavior" from this "related behavior feature" using the learned type identification model M4.

As a result, the situation detection system 1, situation detection method, and situation detection program can use not only object detection and behavior detection, but also related behavior detection to detect relationships between moving bodies V and objects detected from video images, and can detect the meaning (type) of their mutual behavior by relating these movements. This allows the situation detection system 1, situation detection method, and situation detection program to perform appropriate situation detection, for example, more precise situation detection.

The situation detection system 1, situation detection method, and situation detection program can apply the more precise situation detection results obtained as described above to, for example, real-time safe driving diagnosis systems, education systems, hazard prediction systems, and automated driving systems. In this case, the situation detection system 1, situation detection method, and situation detection program can be used for various purposes, such as real-time driving warnings, evaluation of driving skills, detection and evaluation of dangerous driving and traffic violations, guidance and education on risk reduction behavior, creation of safe operation plans, and reflection in automated driving.

For example, the situation detection system 1, situation detection method, and situation detection program can use related behavior detection technology to detect related behaviors such as prying open vehicle doors or car break-ins using tools such as a bat or crowbar, and issue an alarm, which can be used for crime prevention, etc. In addition, the situation detection system 1, situation detection method, and situation detection program can be more effectively used for crime prevention, etc., by also using video data from security cameras installed on structures such as buildings and streetlights.

The situation detection system 1, the situation detection method, and the situation detection program can also be applied to a dispatch service system for taxis and the like, for example, by using related behavior detection technology. In this case, the situation detection system 1, the situation detection method, and the situation detection program can detect the intention to board a taxi based on the behavior of a person on the side of the road, such as staring in the direction of an approaching vehicle or raising a hand toward a vehicle, by performing related behavior detection on the movements of people around the vehicle and the movement of the vehicle, and can be used for dispatch services.

Furthermore, the situation detection system 1, the situation detection method, and the situation detection program can be applied to a service for providing information about available parking spaces and providing information about available parking spaces prediction, for example, by utilizing related behavior detection technology. The situation detection system 1, the situation detection method, and the situation detection program can also detect related behaviors, such as a person finishing loading their luggage or getting into their car in a full parking lot, by also using video image data from a camera or the like installed in the parking lot, and can identify, for example, a situation in which a free parking space is likely to become available within the next few minutes, and can be used for information services for registered users, etc.

In the embodiment described above, an example of a situation detection system 1 in which both the learning phase and the usage phase are integrated into one system has been described, but the embodiment is not limited to this.

For example, the situation detection system 1A according to the modified example illustrated in FIG. 11 differs from the situation detection system 1 described above in that it is configured separately from a trained model generation system 100 that executes each process in the learning phase, and a situation detection device 200 that executes each process in the usage phase. The situation detection device 200 may be mounted on a mobile body V together with, for example, a camera 50, a measuring device 51, etc. On the other hand, the trained model generation system 100 may be installed, for example, in a business establishment that operates the mobile body V.

The trained model generation system 100 includes an input device 110, an output device 120, a memory circuit 130, and a processing circuit 140, and performs processing to generate a trained model M by machine learning using a training dataset D3. In order to realize the various processing functions described above, the processing circuit 140 is functionally configured to include a preprocessing unit 141 and a model generation unit 142.

The preprocessing unit 141, like the preprocessing unit 41 described above, is capable of executing a process of acquiring a training data set D3 (a first training data set D31, a second training data set D32, a third training data set D33, and a fourth training data set D34). The preprocessing unit 141 stores the acquired multiple training data sets D3 in the memory circuitry 130.

The model generation unit 142, like the model generation unit 42 described above, can execute a process of generating a trained model M by machine learning using a plurality of training data sets D3 acquired by the preprocessing unit 141. Like the model generation unit 42 described above, the model generation unit 142 is functionally conceptually configured to include an object detection model generation unit 142a, a behavior detection model generation unit 142b, a related behavior detection model generation unit 142c, and a type-specific model generation unit 142d. The model generation unit 142 stores the generated trained models M (trained object detection model M1, trained behavior detection model M2, trained related behavior detection model M3, trained type-specific model M4) in the memory circuit 130.

The situation detection device 200 includes an input device 210, an output device 220, a memory circuit 230, and a processing circuit 240, and performs processing to identify the type of behavior that occurs in relation to the movement of a moving body V and the movement of an object in a still image that constitutes a moving image, using a trained model M. In order to realize the various processing functions described above, the processing circuit 240 is functionally configured to include a detection target input unit 243, a type identification unit 244, and an output unit 245.

The detection target input unit 243, like the detection target input unit 43 described above, is capable of executing a process of inputting moving image data D41 and measurement data D42 that become the detection target data D4 to be detected.

The type identification unit 244, like the type identification unit 44 described above, can execute a process of identifying the type of behavior that occurs in relation to the movements of multiple objects including the moving body V from the detection target data D4 input by the detection target input unit 243 using the trained model M. In this case, the type identification unit 244 can use, for example, the trained model M (trained object detection model M1, trained behavior detection model M2, trained relational behavior detection model M3, trained type identification model M4) that has been stored in advance in the memory circuit 230 via the output device 120 of the trained model generation system 100 and the input device 210 of the situation detection device 200. This trained model M is a model generated by the trained model generation system 100 as described above.

The output unit 245, like the output unit 45 described above, is capable of executing a process of outputting the type identification result data D5 identified by the type identification unit 44 via the output device 220.

The remaining configurations of the input devices 110, 210, output devices 120, 220, memory circuits 130, 230, and processing circuits 140, 240 are substantially the same as the input device 10, output device 20, memory circuit 30, and processing circuit 40 described above. In this modified example, a moving image of the surroundings of the moving body V is captured, for example, by a camera 50 mounted on the moving body V, and input from the camera 50 to the input device 210 as moving image data. Similarly, a measurement value according to the state of the moving body V is measured, for example, by a measuring device 51 mounted on the moving body V, and input from the measuring device 51 to the input device 210 as moving image data.

Even in this case, the situation detection system 1A, the trained model generation system 100, and the situation detection device 200 can appropriately detect the situation, similar to the situation detection system 1 described above, and can, for example, perform more precise situation detection.

In this modified example, the trained model M used in the situation detection device 200 is not limited to the model generated by the trained model generation system 100 as described above, but may be a trained model M generated by another system.

[Embodiment 2]
The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the second embodiment further differ from the first embodiment in that they use distance data. In the following, the same components as those in the above-mentioned embodiments are given the same reference numerals, and duplicated descriptions of the common configurations, actions, and effects are omitted as much as possible (the same applies below).

The situation detection system 1B of this embodiment shown in Figures 12 and 13 differs from the situation detection system 1 described above in that it performs related behavior detection using distance change features that represent changes in distance between a moving body V and objects surrounding the moving body V, in addition to moving body behavior detection features and object behavior detection features. Here, the situation detection system 1B differs from the situation detection system 1 described above in that the processing circuit 40 includes a model generation unit 42B and a type identification unit 44B instead of the model generation unit 42 and the type identification unit 44. The other configurations of the situation detection system 1B are substantially the same as those of the situation detection system 1 described above.

The input device 10 of this embodiment receives at least video data, measurement data, and distance data from various devices.

Distance data is data that indicates the distance between the moving body V and surrounding objects (e.g., vehicles, pedestrians, bicycles, obstacles, street lights, signs, utility poles, signs, stop lines, etc.) measured in conjunction with the capture of a moving image. The distance data is typically detected in the moving body V in synchronization with the capture of a moving image. The distance between the moving body V and the surrounding objects is detected, for example, by a distance sensor 52 mounted on the moving body V, and input from the distance sensor 52 to the input device 10 as distance data. The distance sensor 52 is, for example, mounted on the moving body V and is composed of various radars or sonars that detect distance using infrared rays, millimeter waves, ultrasonic waves, etc. In addition, the distance sensor 52 may be shared by, for example, the camera 50, and detect the distance by performing various processes on the moving image data captured by the camera 50.

The second learning dataset D32 of this embodiment is composed of the above-mentioned distance data in addition to the measurement data and object detection features, teacher data on the moving object behavior detection features, and teacher data on the object behavior detection features in addition to the teacher data on the distance change features. Here, the distance change feature is a feature that represents the change in the distance between the moving object V and the surrounding objects with the time change of the distance data, and represents the characteristics of the change in the distance according to the movement of the moving object V and the surrounding objects with the time change. Here, the second learning dataset D32 is composed of "measurement data, object detection features, and distance data" and "teacher data on the moving object behavior detection features corresponding to the measurement data, teacher data on the object behavior detection features corresponding to the object detection features, and teacher data on the distance change features corresponding to the distance data" linked as a set. In other words, the second learning dataset D32 in this embodiment is composed of the measurement data, the object detection feature, and the distance data quantified as input data D1 (explanatory variables), and teacher data regarding the mobile object behavior detection feature, the teacher data regarding the object behavior detection feature, and the teacher data regarding the distance change feature quantified as teacher data D2 (target variable).

The third learning dataset D33 of this embodiment is composed of a moving object behavior detection feature, an object behavior detection feature, a distance change feature, and teacher data related to the related behavior feature. The third learning dataset D33 is composed of a set of "moving object behavior detection feature, object behavior detection feature, and distance change feature" and "teacher data related to the related behavior feature corresponding to the moving object behavior detection feature, the object behavior detection feature, and the distance change feature." In other words, the third learning dataset D33 of this embodiment is composed of the moving object behavior detection feature, the object behavior detection feature, and the distance change feature quantified as input data D1 (explanatory variable), and teacher data related to the related behavior feature quantified as teacher data D2 (target variable).

The model generation unit 42B of this embodiment differs from the above-described model generation unit 42 in that it includes a behavior detection model generation unit 42Bb and a related behavior detection model generation unit 42Bc instead of the behavior detection model generation unit 42b and the related behavior detection model generation unit 42c. The other configurations of the model generation unit 42B are substantially the same as those of the above-described model generation unit 42.

The behavior detection model generation unit 42Bb generates a learned behavior detection model M2 by machine learning using a plurality of second learning datasets D32 acquired by the preprocessing unit 41. In this embodiment, the behavior detection model generation unit 42Bb performs machine learning of the learned behavior detection model M2 using the "measurement data, object detection feature, and distance data" of the second learning dataset D32 as input data D1 (explanatory variables) and "teacher data on moving object behavior detection feature, teacher data on object behavior detection feature, and teacher data on distance change feature" as teacher data D2 (target variable). As a result of the machine learning, the behavior detection model generation unit 42Bb generates a learned behavior detection model M2 that has been machine-learned to extract moving object behavior detection feature, object behavior detection feature, and distance change feature from the measurement data, object detection feature, and distance data. The learned behavior detection model M2 generated by the behavior detection model generation unit 42Bb is a model whose inputs are "measurement data, object detection features, and distance data" and whose output is "mobile object behavior detection features, object behavior detection features, and distance change features."

In addition, like the above-mentioned behavior detection model generation unit 42b, the behavior detection model generation unit 42Bb may use multiple second learning datasets D32 to individually generate multiple sub-models (mobile object behavior detection sub-model, object behavior detection sub-model, distance change detection sub-model) by machine learning as the learned behavior detection model M2.

The relational behavior detection model generation unit 42Bc can execute a process of generating a learned relational behavior detection model M3 by machine learning using a plurality of second learning datasets D32 acquired by the preprocessing unit 41. The relational behavior detection model generation unit 42Bc of this embodiment performs machine learning of the learned relational behavior detection model M3 using the "moving object behavior detection feature, object behavior detection feature, and distance change feature" of the third learning dataset D33 as input data D1 (explanatory variables) and the "teacher data regarding the relational behavior feature" as teacher data D2 (target variable). As a result of the machine learning, the relational behavior detection model generation unit 42Bc generates a learned relational behavior detection model M3 that has been subjected to machine learning to extract the relational behavior feature from the moving object behavior detection feature, object behavior detection feature, and distance change feature. The learned related behavior detection model M3 generated by the related behavior detection model generation unit 42Bc is a model whose inputs are "mobile object behavior detection features, object behavior detection features, and distance change features" and whose output is "related behavior features."

Then, in the use phase, the detection target input unit 43 of this embodiment inputs the moving image data D41, the measurement data D42, and the distance data D43 to the type identification unit 44B as the detection target data D4 to be detected.

In the use phase, the type identification unit 44B of this embodiment uses the learned object detection model M1 generated by the object detection model generation unit 42a, the learned behavior detection model M2 generated by the behavior detection model generation unit 42b, the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42c, and the learned type identification model M4 generated by the type identification model generation unit 42d to identify the type of the above behavior from the video image data D41, the measurement data D42, and the distance data D43 input by the detection target input unit 43.

Specifically, the type identification unit 44B of this embodiment is configured to include an integration unit 44Bb, a behavior detection feature extractor 44Bc, and a related behavior feature extractor 44Bd instead of the integration unit 44b, the behavior detection feature extractor 44c, and the related behavior feature extractor 44d. The other configuration of the type identification unit 44B is substantially the same as that of the type identification unit 44 described above.

The integration unit 44Bb integrates the "object detection feature D61" input by the object detection feature extractor 44a with the "measurement data D42" and "distance data D43" of the detection target data D4 input by the detection target input unit 43 so that they are combined at the same time in a chronological order, stores them in the memory circuit 30, and inputs them to the behavior detection feature extractor 44Bc in the next stage.

The behavior detection feature extractor 44Bc uses the learned behavior detection model M2 generated by the behavior detection model generation unit 42Bb to extract moving object behavior detection feature D62, object behavior detection feature D63, and distance change feature D65 from the measurement data D42, object detection feature D61, and distance data D43. The behavior detection feature extractor 44Bc inputs the "measurement data D42, object detection feature D61, and distance data D43" input by the integration unit 44Bb at the same time in time series as input data to the learned behavior detection model M2, and outputs the "moving object behavior detection feature D62, object behavior detection feature D63, and distance change feature D65" from the learned behavior detection model M2 in response to this. The behavior detection feature extractor 44Bc stores the output mobile object behavior detection feature D62, object behavior detection feature D63, and distance change feature D65 in the memory circuit 30, and also inputs them to the next-stage related behavior feature extractor 44Bd.

The related behavior feature extractor 44Bd uses the learned related behavior detection model M3 generated by the related behavior detection model generation unit 42Bc to extract the related behavior feature D64 from the moving object behavior detection feature D62, the object behavior detection feature D63, and the distance change feature D65. The related behavior feature extractor 44Bd inputs the "moving object behavior detection feature D62, the object behavior detection feature D63, and the distance change feature D65" input by the behavior detection feature extractor 44Bc as input data to the learned related behavior detection model M3, and outputs the "related behavior feature D64" from the learned related behavior detection model M3 in response to this. The related behavior feature extractor 44Bd stores the output related behavior feature D64 in the memory circuit 30 and inputs it to the next stage classifier 44e.

The classifier 44e inputs the "related behavior feature D64" input by the related behavior feature extractor 44Bd as input data to the learned type-specific model M4 generated by the type-specific model generation unit 42d, and outputs a "quantified value of the type of the above behavior" from the learned type-specific model M4 in response to this.

Even in this case, the situation detection system 1B, the situation detection method, and the situation detection program can appropriately detect the situation, similar to the situation detection system 1 described above, and can, for example, perform more precise situation detection.

Here, the situation detection system 1B, situation detection method, and situation detection program described above can extract "moving object behavior detection features, object behavior detection features, and distance change features" from "measurement data, object detection features, and distance data" using the learned behavior detection model M2. Then, the situation detection system 1, situation detection method, and situation detection program can extract "related behavior features" based on "moving object behavior detection features, object behavior detection features, and distance change features" using the learned related behavior detection model M3. Then, the situation detection system 1, situation detection method, and situation detection program can identify the type of the behavior from the related behavior features using the learned type identification model M4. As a result, the situation detection system 1, situation detection method, and situation detection program can perform higher-level situation detection based on the distance between the moving object V and the objects surrounding the moving object V.

The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to the above-described embodiments of the present invention are not limited to the above-described embodiments, and various modifications are possible within the scope of the claims.

For example, in the situation detection system 1 and situation detection system 1B described above, the output unit 45 may determine whether or not an alarm is required based on the distance data input by the detection target input unit 43.

In this case, as shown in FIG. 14, for example, after the output step (step S8), the output unit 45 executes a determination step (step S9-1) in which the type of behavior identified by the type identification unit 44, 44B is "dangerous behavior" or the like and determines whether or not there is a candidate for a warning in the "dangerous behavior". If the output unit 45 determines that the type of behavior identified by the type identification unit 44 is not "dangerous behavior" or the like or that there is no candidate for a warning in the "dangerous behavior" (step S9-1: No), it ends the processing according to this flowchart.

On the other hand, when the output unit 45 determines that the type of behavior identified by the type identification units 44, 44B is "dangerous behavior" or the like and that there is a candidate for the "dangerous behavior" that is the subject of an alarm (step S9-1: Yes), the output unit 45 determines whether or not the distance between the moving body V and the candidate for the alarm is close based on the distance data (step S9-2). The output unit 45 can determine whether or not the distance between the moving body V and the candidate for the alarm (surrounding object) is close by, for example, determining whether or not the distance between the moving body V and the candidate for the alarm (surrounding object) is equal to or less than a predetermined determination threshold.

If the output unit 45 determines that the distance between the moving body V and the candidate for the alarm is large (step S9-2: No), it determines that an alarm is not required and ends the processing according to this flowchart. On the other hand, if the output unit 45 determines that the distance between the moving body V and the candidate for the alarm is small (step S9-2: Yes), it determines that an alarm is required and executes an alarm step (step S10) to output an alarm via the output device 20 and ends the processing according to this flowchart.

In this case, the situation detection system 1, 1B can more reliably suppress unnecessary alarms and output an appropriate alarm in response to the detected situation.

The model generation units 42, 42B, and 142 described above have been described as generating the trained model M by machine learning with teacher data, but this is not limited thereto, and the trained model M may be generated in part or in whole by machine learning without teacher data to the extent possible. For example, the behavior detection model generation units 42b, 42Bb, and 142b may use a so-called "LSTM Autoencoder" to generate the trained behavior detection model M2 by machine learning without teacher data that does not use teacher data related to mobile object behavior detection features, etc.

The processing circuits 40, 140, and 240 described above have been described as each processing function being realized by a single processor, but this is not limited to the above. The processing circuits 40, 140, and 240 may be configured to realize each processing function by combining multiple independent processors and having each processor execute a program. Furthermore, the processing functions of the processing circuits 40, 140, and 240 may be realized by being appropriately distributed or integrated into a single or multiple processing circuits. Furthermore, the processing functions of the processing circuits 40, 140, and 240 may be realized in whole or in any part by a program, or may be realized as hardware using wired logic or the like.

The programs executed by the processors described above are provided by being pre-installed in the storage circuits 30, 130, 230, etc. Note that the programs may be provided by being recorded on a computer-readable storage medium in a format that can be installed on these devices or in a format that can be executed. The programs may also be provided or distributed by being stored on a computer connected to a network such as the Internet and downloaded via the network.

The situation detection system, situation detection method, situation detection program, trained model generation system, and situation detection device according to this embodiment may be configured by appropriately combining the components of the embodiments and modified examples described above.

1, 1A, 1B Situation detection system 10, 110, 210 Input device 20, 120, 220 Output device 30, 130, 230 Memory circuit 40, 140, 240 Processing circuit 41, 141 Preprocessing unit 42, 42B, 142 Model generation unit 42a, 142a Object detection model generation unit 42b, 42Bb, 142b Behavior detection model generation unit 42c, 42Bc, 142c Related behavior detection model generation unit 42d, 142d Type identification model generation unit 43, 243 Detection target input unit 44, 44B, 244 Type identification unit 44a Object detection feature extractor 44b, 44Bb Integration unit 44c, 44Bc Behavior detection feature extractor 44d, 44Bd Related behavior feature extractor 44e Classifier 45, 245 Output unit 50 Camera 51 Measuring device 52 Distance sensor 100 Trained model generation system 200 Situation detection device AL Machine learning algorithm D0 Raw data D1 Input data D2 Teacher data D3 Training data set D31 First training data set D32 Second training data set D33 Third training data set D34 Fourth training data set D4 Detection target data D41 Video image data D42 Measurement data D43 Distance data D5 Type identification result data D61 Object detection feature D62 Mobile object behavior detection feature D63 Object behavior detection feature D64 Related behavior feature D65 Distance change feature M Trained model M1 Trained object detection model M2 Trained behavior detection model M3 Trained related behavior detection model M4 Trained type identification model V Mobile object

Claims (6)

a pre-processing unit that acquires a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object accompanying a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object accompanying a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior.
an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection feature from the video data, using the plurality of first training data sets acquired by the preprocessing unit;
a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the second learning data sets acquired by the preprocessing unit;
a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the third learning data sets acquired by the preprocessing unit;
a type identification model generation unit that generates, by learning, a trained type identification model that identifies a type of the behavior from the related behavior feature using the plurality of fourth learning data sets acquired by the preprocessing unit;
a detection target input unit for inputting the moving image data and the measurement data to be detected;
a type identification unit that identifies a type of the behavior from the video data and the measurement data input by the detection target input unit, using the trained object detection model generated by the object detection model generation unit, the trained behavior detection model generated by the behavior detection model generation unit, the trained related behavior detection model generated by the related behavior detection model generation unit, and the trained type identification model generated by the type identification model generation unit.
Situation detection system. the second learning data set includes, in addition to the measurement data and the object detection feature, distance data representing a distance between the moving object and a surrounding object of the moving object measured in conjunction with capturing the video, teacher data regarding the moving object behavior detection feature, and teacher data regarding a distance change feature representing a change in the distance associated with a change in the distance data over time in addition to the teacher data regarding the object behavior detection feature,
the third learning data set includes the moving object behavior detection feature, the object behavior detection feature, the distance change feature, and training data regarding the related behavior feature,
the behavior detection model generation unit generates, by learning, the trained behavior detection model that extracts the moving object behavior detection feature, the object behavior detection feature, and the distance change feature from the measurement data, the object detection feature, and the distance data, using the plurality of second learning data sets acquired by the preprocessing unit;
the related behavior detection model generation unit generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the moving object behavior detection feature, the object behavior detection feature, and the distance change feature, using the third learning data sets acquired by the preprocessing unit;
the detection target input unit inputs the moving image data, the measurement data, and the distance data that are to be detected;
the type identification unit identifies the type of the behavior from the video data, the measurement data, and the distance data input by the detection target input unit, using the trained object detection model generated by the object detection model generation unit, the trained behavior detection model generated by the behavior detection model generation unit, the trained related behavior detection model generated by the related behavior detection model generation unit, and the trained type identification model generated by the type identification model generation unit;
The situation detection system according to claim 1 . acquiring a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior;
generating, by learning, a trained object detection model that extracts the object detection feature from the video data using the acquired plurality of first training data sets;
generating a trained behavior detection model by learning, the trained behavior detection model extracting the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the acquired second training data sets;
generating a learned related behavior detection model by learning, the learned related behavior detection model extracting the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the acquired third learning data sets;
generating a trained type identification model by learning, the trained type identification model identifying a type of the behavior from the related behavior feature by using the acquired plurality of fourth learning data sets;
inputting the moving image data and the measurement data to be detected;
and identifying a type of the behavior from the input video data and the measurement data using the generated trained object detection model, trained behavior detection model, trained relational behavior detection model, and trained type identification model.
How to detect the situation. obtain a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object associated with a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object associated with a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior;
generating, by learning, a trained object detection model that extracts the object detection feature from the video data using the acquired plurality of first training data sets;
generating a trained behavior detection model by learning, the trained behavior detection model extracting the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the acquired second training data sets;
generating a trained related behavior detection model by learning, the trained related behavior detection model extracting the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the acquired third training data sets;
generating a trained type identification model by learning, the trained type identification model being configured to identify a type of the behavior from the related behavior feature amount, using the acquired plurality of fourth learning data sets;
inputting the moving image data and the measurement data to be detected;
identifying a type of the behavior from the input video data and the measurement data using the generated trained object detection model, trained behavior detection model, trained relational behavior detection model, and trained type identification model;
Each process is executed by a computer.
Situation detection program. a pre-processing unit that acquires a first learning dataset including video data representing a video captured from a moving object and teacher data related to object detection features representing an object detected in the video data; a second learning dataset including measurement data representing a measurement value corresponding to a state of the moving object measured while the video was being captured and the object detection features, teacher data related to moving object behavior detection features representing the movement of the moving object accompanying a time change in the measurement data, and teacher data related to object behavior detection features representing the movement of the object accompanying a time change in the object detection features; a third learning dataset including the moving object behavior detection features and the object behavior detection features, and teacher data related to related behavior features representing behaviors occurring in relation to the movement of the moving object and the movement of the object; and a fourth learning dataset including the related behavior features and teacher data related to related behavior type data representing a type of the behavior.
an object detection model generation unit that generates, by learning, a trained object detection model that extracts the object detection feature from the video data, using the plurality of first training data sets acquired by the preprocessing unit;
a behavior detection model generation unit that generates, by learning, a trained behavior detection model that extracts the mobile object behavior detection feature and the object behavior detection feature from the measurement data and the object detection feature using the second learning data sets acquired by the preprocessing unit;
a related behavior detection model generation unit that generates, by learning, a trained related behavior detection model that extracts the related behavior feature from the mobile object behavior detection feature and the object behavior detection feature using the third learning data sets acquired by the preprocessing unit;
a type identification model generation unit that generates, by learning, a trained type identification model that identifies a type of the behavior from the related behavior feature using the plurality of fourth learning data sets acquired by the preprocessing unit.
A trained model generation system. a detection target input unit that inputs moving image data representing a moving image captured from a moving object and measurement data representing a measurement value corresponding to a state of the moving object measured in conjunction with capturing the moving image;
the detection target input unit includes a trained object detection model that has been trained to extract from the video data an object detection feature that represents an object detected in the video data; a trained behavior detection model that has been trained to extract from the measurement data and the object detection feature a mobile object behavior detection feature that represents the movement of the mobile object accompanying a time change in the measurement data and an object behavior detection feature that represents the movement of the object accompanying a time change in the object detection feature; a trained related behavior detection model that has been trained to extract from the mobile object behavior detection feature and the object behavior detection feature a related behavior feature that represents a behavior occurring in relation to the movement of the mobile object and the movement of the object; and a type identification unit that identifies the type of the behavior from the video data and the measurement data input by the detection target input unit using a trained type identification model that has been trained to identify the type of the behavior from the related behavior feature ,
the trained object detection model is generated by learning using a plurality of first learning data sets including the video data and teacher data related to the object detection features;
the trained behavior detection model is generated by learning using a plurality of second learning data sets including the measurement data and the object detection feature, teacher data regarding the mobile object behavior detection feature, and teacher data regarding the object behavior detection feature;
the trained related behavior detection model is generated by learning using a plurality of third learning data sets including the mobile object behavior detection feature and the object behavior detection feature, and teacher data related to the related behavior feature;
The learned type identification model is generated by learning using a plurality of fourth learning data sets including the related behavior feature amount and teacher data related to related behavior type data representing the type of the behavior .
Situation detection device.

Images