JP7749449B2 - Learning data generation device, automatic door system, learning data generation method, trained model generation method, control program, and recording medium - Google Patents

Description

The present invention relates to a training data generation device that generates training data used in machine learning, a control program for the training data generation device, a recording medium on which the control program is recorded, a trained model generation method using training data generated by the training data generation device, and an automatic door system that includes a detection unit that detects moving objects using the trained model.

In machine learning using deep learning and other methods, large amounts of training data are required for efficient learning. However, when creating training data for a trained model used in a sensor that detects moving objects, for example, the training data was previously created by manually labeling moving objects captured in captured images. This required a considerable amount of effort. As a result, various methods for efficiently creating training data have been proposed.

Patent document 1 describes a computer program that captures an image of a real object, derives the orientation of the real object using a 3D model corresponding to the real object, and associates the appearance information of the real object with a 2D model obtained from the 3D model.

Patent document 2 describes a training data generation device that models a 3D model with associated attributes and a background in a virtual space, projects the 3D space onto a 2D plane, generates labels for the 2D objects onto which the 3D model is projected, and associates the 2D images with the labels.

Patent document 3 describes a learning data generation device that generates scene data for generating learning data in which a specific object model has a specific color, sets an area for the specific object model, and associates an image for the scene data with the specific object area.

Patent document 4 describes an object recognition device that projects a created training image onto feature points in an eigenspace and outputs information about the feature points that are closest to the input image.

JP 2018-51410 A International Publication No. WO2020/183598 Japanese Patent Application Laid-Open No. 2019-23858 JP 2010-211732 A

However, there is room for improvement in the above-mentioned conventional technology, and it is possible to create more efficient and reliable training data. One aspect of the present invention was made in consideration of the above-mentioned problems, and its purpose is to realize a training data generation device that generates efficient and reliable training data.

To solve the above-mentioned problems, a training data generation device according to one aspect of the present invention comprises a video data generation unit that generates video data in which a three-dimensional model of a detection target moves in a virtual space; a variation data generation unit that generates brightness change data or difference data in a two-dimensional image projected from the video data; a labeling unit that assigns a label indicating that the detection target is an area in the brightness change data or difference data that corresponds to the detection target; and a training data generation unit that generates, as training data, a combination of the brightness change data or the difference data, the area corresponding to the detection target, and the label assigned by the labeling unit.

A training data generation device according to one aspect of the present invention uses brightness change data or difference data in a two-dimensional image that outputs video data to generate training data for training a trained model that estimates detection target regions in the two-dimensional image and labels assigned to the detection target regions.

With the above configuration, the created video data that includes the detection target is used, so the detection target area can be automatically identified and the detection target area can be automatically labeled. Furthermore, because brightness change data or difference data in the two-dimensional image from which the video data is output is used, the processing load can be reduced compared to when the video data is used as is. This allows for efficient creation of learning data. Furthermore, because labeling is performed using the generated video data, accurate labeling can be performed, resulting in the creation of highly reliable learning data.

In addition, a learning model using the learning data created by the learning data generation device can obtain estimation results using brightness data or difference data as input, thereby reducing the processing load in the estimation process of the learning model.

In a learning data generation device according to one aspect of the present invention, the display unit that displays the two-dimensional image may include an event generation unit that generates an event for each pixel on the display unit when the luminance value of that pixel exceeds a first threshold value or falls below a second threshold value, and a position output unit that outputs the position on the display unit of the pixel at which the event generation unit generated the event, and the variation data creation unit may create the luminance variation data using the position output by the position output unit.

The first threshold indicates a value greater than the current brightness value, and the second threshold indicates a value less than the current brightness value. With this configuration, pixels whose brightness values exceed the first threshold or fall below the second threshold are output along with their positions, making it easy to identify positions where brightness values have changed beyond a specified range, and creating brightness change data.

In a learning data generation device according to one aspect of the present invention, the video data created by the video data creation unit may not include background data.

With the above configuration, background data is not included, which reduces the volume of video data. Furthermore, when using learning data for automatic door detection sensors, it is sufficient to be able to detect passersby, the target of detection, and background data is not required, so there is no problem if background data is not included.

In the learning data generation device according to one aspect of the present invention, the detection target included in the video data created by the video data creation unit may be a moving object.

With the above configuration, moving objects can be detected.

In a learning data generation device according to one aspect of the present invention, the video data creation unit may generate multiple video data sets in which at least one of the traveling direction and traveling speed of the detection target differs.

The above configuration makes it possible to detect moving objects moving in various directions and at various speeds as detection targets.

In the learning data generation device according to one aspect of the present invention, the video data creation unit may create multiple video data sets in different environments in which the detection target exists.

The above configuration makes it possible to detect the target object regardless of the environment in which it is located. Here, the environment includes weather-related factors such as rain, wind, fog, snow, etc., vegetation swaying due to wind, etc., objects affected by weather, artificial lighting such as fluorescent lights, LEDs, and automobile headlights, cigarette smoke, steam, etc., etc.

In the learning data generation device according to one aspect of the present invention, the video data created by the video data creation unit may include a plurality of the detection targets.

The above configuration makes it possible to detect multiple detection targets.

To solve the above problem, an automatic door system according to one aspect of the present invention includes a sensor that detects the detection target using a learning model trained by machine learning using the learning data generated by the learning data generation device, and an automatic door that opens and closes based on the detection results of the sensor.

This configuration improves the detection accuracy of the detection sensor used to open automatic doors. Furthermore, detection processing can be performed using a learning model with a light processing load.

To solve the above problem, a training data generation method according to one aspect of the present invention includes a video data generation step of generating video data in which a three-dimensional model of a detection target moves in a virtual space; a variation data generation step of generating brightness change data or difference data in a two-dimensional image projected from the video data; a labeling step of assigning a label indicating that the detection target is located in the brightness change data or difference data to an area corresponding to the detection target; and a training data generation step of generating, as training data, a combination of the brightness change data or difference data, the area corresponding to the detection target, and the label assigned in the labeling step. This makes it possible to achieve the above-mentioned effects.

To solve the above problem, one aspect of the present invention provides a trained model generation method including a training data generation step of generating training data using the training data generation method, and a machine learning step of performing machine learning using the training data generated in the training data generation step.

The training data generation device according to each aspect of the present invention may be realized by a computer. In this case, the control program for the training data generation device, which causes the computer to operate as each unit (software element) of the training data generation device, thereby realizing the training data generation device on the computer, and the computer-readable recording medium on which the control program is recorded, also fall within the scope of the present invention.

One aspect of the present invention enables automatic labeling and reduces the processing load, resulting in the efficient creation of training data. Furthermore, since labeling is performed using generated video data, highly reliable training data can be created. Furthermore, a learning model with a light processing load can be created.

1 is a functional block diagram showing a configuration of a main part of a learning data generation device according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of a three-dimensional model used in a learning data generation device. FIG. 10 is a diagram showing an example in which a three-dimensional model is placed in a virtual space and converted into a two-dimensional image. FIG. 10 is a diagram illustrating an example of brightness change data of a two-dimensional image. FIG. 10 is a diagram showing an example of setting an area of a three-dimensional model in brightness change data. FIG. 10 is a diagram illustrating an example of learning data. FIG. 2 is a diagram illustrating a configuration of a main part of a learning device. 1 is a diagram showing the main configuration of an automatic door system. 10 is a flowchart showing a processing flow in the learning data generation device. FIG. 10 is a diagram showing a modified example of moving image data. FIG. 10 is a diagram showing a modified example of moving image data. FIG. 10 is a diagram showing a modified example of moving image data. FIG. 10 is a diagram showing a modified example of moving image data.

One embodiment of the present invention will be described in detail below. The learning data generation device 100 according to this embodiment generates learning data used to train the trained model 5. In this embodiment, an example will be given of creating learning data used to train the trained model 5 used to detect a detection target by the detection unit 2 provided in the automatic door system 1, but the learning data generated by the learning data generation device 100 is not limited to this. It may also be used as a learning model for sensors other than automatic doors, such as shutters.

Figure 1 is a functional block diagram showing an example configuration of a training data generation device 100 according to this embodiment. As shown in Figure 1, the training data generation device 100 includes a video data creation unit 10, a variation data creation unit 20, a labeling unit 30, a training data generation unit 40, and a display unit 50. Note that the display unit 50 may be provided external to the training data generation device 100.

The video data creation unit 10 creates video data in which a 3D model of the detection target moves in a virtual space. More specifically, the video data creation unit 10 first creates a 3D model of the detection target. The 3D model can be created using known technology and is, for example, a moving object such as a person or a vehicle. Information representing attributes is associated with the 3D model. Examples of attributes include gender and age for people, and type and color for vehicles.

Figure 2 shows an example of a 3D model to be created. Figure 2 shows an example of an adult male 201. As mentioned above, 3D models can be created using known techniques, and are created by setting various parameters (position, rotation, scaling, dimensions, etc.).

The video data creation unit 10 then places the created 3D model in a 3D virtual space and sets the movement of the 3D model. The movement of the 3D model is set by the user. This creates video data in which the 3D model moves within the 3D virtual space. Note that the video data creation unit 10 may create multiple video data sets in which the 3D model moves in different directions and at different speeds. It may also create video data sets in which the surrounding conditions of the 3D model are different. Different surrounding conditions refer to situations in which there is nothing around the 3D model, situations in which part of the 3D model is hidden by something other than the 3D model, etc.

The video data creation unit 10 may or may not create background data that serves as the background in the three-dimensional virtual space. The background refers to the parts of the virtual space other than the three-dimensional model.

The variation data creation unit 20 creates brightness variation data or difference data for a two-dimensional image onto which moving image data is projected. More specifically, the variation data creation unit 20 first projects a three-dimensional model in a three-dimensional virtual space onto a two-dimensional plane to render a two-dimensional object. Any known method can be used to render a two-dimensional object.

Figure 3 shows an example of a two-dimensional image in which a two-dimensional object is drawn on a two-dimensional plane. The example two-dimensional image shown in Figure 3 shows a two-dimensional image 211 of a created three-dimensional model of an adult male 201.

The variation data creation unit 20 then creates brightness variation data for a two-dimensional image 211 in which the three-dimensional model is projected onto a two-dimensional plane. Figure 4 shows an example of brightness variation data. Figure 4 shows a two-dimensional image created by accumulating brightness variation data for each pixel over a certain period of time. Details of the brightness variation data will be described later, but as shown in Figure 4, the brightness variation data makes it possible to identify the positions of pixels whose brightness values exceed a first threshold and pixels whose brightness values fall below a second threshold. Note that difference data between frames may be created instead of brightness variation data.

The labeling unit 30 assigns a label indicating that the region in the brightness change data or difference data corresponding to the detection target is the detection target. More specifically, the labeling unit 30 calculates the region in which the drawn two-dimensional object exists and labels the region using the attribute associated with the object. For example, if the two-dimensional object is a projection of a three-dimensional model with the attribute "male," the region in which the two-dimensional object exists is labeled "male." The attribute to be labeled may be all or some of the attributes associated with the corresponding three-dimensional model. The region in which the two-dimensional object exists may also be a rectangular region circumscribing the two-dimensional object. Figure 5 shows the region to be labeled by the labeling unit 30. In the example shown in Figure 5, a rectangular region 231 circumscribing an adult male 221 is set as the target region to be labeled. Here, rectangular region 231 is labeled "male."

The learning data generating unit 40 generates, as learning data, a set of brightness change data or difference data, an area indicating the detection target, and a label assigned by the labeling unit.
An example of training data is shown in Fig. 6. As shown in Fig. 6, training data 400 is a set of brightness change data or difference data, a detection target region, and a label, and serves as training data for a learning model in which the brightness change data or difference data is input data and the detection target region and the label are output data.

The display unit 50 is a display device that displays two-dimensional images. The display unit 50 includes an event generation unit 51 and a position output unit 52. Here, the display unit 50 is described as being provided in the training data generation device 100, but as mentioned above, the display unit 50 may be external to the training data generation device 100.

The event generation unit 51 generates an event when the brightness value of each pixel on the display unit 50 exceeds a first threshold value or falls below a second threshold value. The first and second threshold values are set as percentages of a reference value. For example, the first threshold value is set to +20% of the reference value, and the second threshold value is set to -15% of the reference value. The reference value is the current brightness value, and when an event occurs, the brightness value after the event becomes the reference value.

As described above, the display unit 50 that displays a two-dimensional image includes an event generation unit 51 that generates an event for each pixel on the display unit 50 when the luminance value of that pixel exceeds a first threshold value or falls below a second threshold value, and a position output unit 52 that outputs the position on the display unit 50 of the pixel at which the event generation unit 51 generated the event.

The position output unit 52 outputs the position on the display unit 50 of the pixel at which the event generation unit 51 generated an event. The position output unit 52 may also output the time the event occurred along with the position. The change data creation unit 20 creates brightness change data using the position output by the position output unit 52.

As described above, the learning data generation device 100 according to this embodiment comprises a video data generation unit 10 that generates video data in which a three-dimensional model of a detection target moves in a virtual space; a variation data generation unit 20 that generates brightness change data or difference data in a two-dimensional image projected from the video data; a labeling unit 30 that assigns a label indicating that the detection target is an area in the brightness change data or difference data that corresponds to the detection target; and a learning data generation unit 40 that generates, as learning data, a combination of the brightness change data or difference data, the area indicating the detection target, and the label assigned by the labeling unit.

[Configuration of the learning device]
Next, the learning device 200 will be described with reference to Fig. 7. As shown in Fig. 7, the learning device 200 includes a learning unit 210. The learning unit 210 performs machine learning using the learning data 400 generated by the learning data generation device 100 to generate a trained model 5. The trained model 5 is a trained model that receives brightness change data or difference data as input and outputs a detection target region and a label.

In other words, the trained model generation method by the training device 200 includes a machine learning step of performing machine learning using the training data 400 generated in the training data generation step by the training data generation device 100.

[Configuration of automatic door system]
Next, the automatic door system 1 will be described with reference to Figure 8. The automatic door system 1 shown in Figure 8 uses a trained model 5 trained using training data 400 created by a training data generation device 100 for the detection process of the detection target by the detection unit 2. Note that the operation of the automatic door system 1 can be achieved using known technology, so a detailed description will be omitted here.

As shown in Figure 8, the automatic door system 1 includes a detection unit (sensor) 2, a drive unit 3, an automatic door 4, and a trained model 5.

The detection unit 2 detects the presence of a person in a specified area and, if a person is present, transmits a drive instruction to the drive unit 3. As described above, in this embodiment, the detection unit 2 uses the trained model 5 to perform processing to detect whether or not a person is present in the specified area.

The drive unit 3 operates to open the automatic door 4 in response to instructions from the detection unit 2.

Automatic door 4 is an automatic door that is opened and closed by drive unit 3.

Note that, although the trained model 5 has been described here as being included in the automatic door system 1, the trained model 5 may not be included in the automatic door system 1, and the detection unit 2 may access an external trained model 5 to obtain estimation results using the trained model 5.

[Processing flow in the learning data generation device]
9 is a flowchart showing the flow of processing in the learning data generation device 100. As shown in FIG. 9, in the learning data generation device 100, first, the video data creation unit 10 creates a three-dimensional model (S101).

Next, the video data creation unit 10 uses the created 3D model to create video data in which the 3D model of the detection target moves in a virtual space (S102, video data creation step). The variation data creation unit 20 then projects the 3D model in the 3D virtual space onto a 2D plane, rendering the 2D object and creating a 2D image (S103). Then, it creates brightness variation data or difference data for the 2D image onto which the video data is projected (S104, variation data creation step).

Then, the labeling unit 30 assigns a label based on the attribute to the area in the brightness change data or difference data that corresponds to the detection target (S105, labeling step). The learning data generation unit 40 then generates learning data that is a combination of the brightness change data or difference data, the area indicating the detection target, and the label assigned by the labeling unit 30 (S106, learning data generation step).

The above is the process flow for generating training data in the training data generation device 100.

[Modification]
Next, modified examples of video data used to create the learning data 400 will be described with reference to Figures 10 to 13. In the above-described example, the video data was generated by placing the created three-dimensional model in a virtual space. There may be multiple three-dimensional models, as shown in Figure 10. In the example shown in Figure 10, three adult males, designated by 202 to 204, are placed in the virtual space as three-dimensional models. The three-dimensional models are not limited to adult males, and may also be females or children.

When the labeling unit 30 performs labeling, the rectangular areas 231-234 circumscribing each of the adult males 202-204 are designated as the target areas for labeling.

In this way, the video data created by the video data creation unit 10 may include multiple 3D models.

Also, as shown in Figure 11, the three-dimensional model may not be a person, but a vehicle or the like. In the example shown in Figure 11, the three-dimensional model is a forklift 205. In this case, the labeling unit 30 sets a rectangular area 235 circumscribing the forklift 205 as the target area to be labeled. A trained three-dimensional model trained using training data using a forklift or the like can be used as an opening/closing sensor for a shutter or the like through which a forklift passes.

Furthermore, as shown in Figures 12 and 13, various weather conditions may be set for the video data. Figure 12 shows an example in which a three-dimensional model 206 is placed in the rain. Figure 13 shows an example in which a three-dimensional model 207 is placed in the wind. In this way, by placing three-dimensional models in various weather conditions, it becomes possible to recognize the area in which the three-dimensional model exists under various weather conditions.

In this way, the video data creation unit 10 may create multiple video data sets in which the 3D model exists under different weather conditions. The video data creation unit 10 may also create video data sets in which the 3D model exists under various environments, not just weather conditions. Here, environment includes not only naturally occurring things such as the weather conditions described above, but also artificially created things. For example, it includes lights such as fluorescent lights, LEDs, and automobile headlights, cigarette smoke, steam, and other smoky elements. It also includes natural light and artificial light that transmits, reflects, refracts, and scatters.

In this way, the environment includes both naturally occurring and artificially occurring things. In other words, the environment can be defined as things that affect a three-dimensional model, such as (1) solids such as snow and vegetation, (2) liquids such as rain, and (3) gases such as wind, fog, and smoke.

[Software implementation example]
The functions of the learning data generation device 100 (hereinafter referred to as the "device") can be realized by a program that causes a computer to function as the device, and a program that causes a computer to function as each control block of the device (in particular, the video data creation unit 10, the variation data creation unit 20, the labeling unit 30, and the learning data generation unit 40).

In this case, the device includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the program. The functions described in each of the above embodiments are realized by executing the program using this control device and storage device.

The above program may be stored non-transitory on one or more computer-readable storage media. These storage media may or may not be included in the device. In the latter case, the program may be supplied to the device via any wired or wireless transmission medium.

Furthermore, some or all of the functions of each of the above control blocks can be realized by logic circuits. For example, integrated circuits incorporating logic circuits that function as each of the above control blocks are also included in the scope of the present invention. In addition, the functions of each of the above control blocks can also be realized by, for example, a quantum computer.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

1 Automatic door system 2 Detection unit (sensor)
3 Driving unit 4 Automatic door 5 Trained model 10 Video image data creation unit 20 Change data creation unit 30 Labeling unit 40 Training data generation unit 50 Display unit 51 Event generation unit 52 Position output unit 100 Training data generation device 200 Training device 210 Training unit 400 Training data

Claims (13)

a moving image data creating unit that creates moving image data in which a three-dimensional model of the detection target moves in a virtual space;
a variation data generating unit that generates brightness variation data or difference data between frames obtained by comparing brightness values with a threshold value in a two-dimensional image projected from the moving image data;
a labeling unit that assigns a label indicating that the detection target is a region in the brightness change data or the difference data that corresponds to the detection target;
a learning data generation unit that generates, as learning data, a set of the brightness change data or the difference data, an area corresponding to the detection object, and a label assigned by the labeling unit. the display unit that displays the two-dimensional image includes an event generation unit that generates an event in each pixel on the display unit when a luminance value of the pixel exceeds a first threshold value or falls below a second threshold value, and a position output unit that outputs a position on the display unit of the pixel at which the event generation unit has generated the event;
The learning data generating device according to claim 1 , wherein the variation data generating unit generates the luminance variation data using the positions output by the position output unit. The learning data generation device according to claim 1 or 2, wherein the video data created by the video data creation unit does not include background data. The learning data generation device according to any one of claims 1 to 3, wherein the detection target included in the video data created by the video data creation unit is a moving object. The learning data generation device of claim 4, wherein the video data creation unit generates multiple video data sets in which at least one of the traveling direction and traveling speed of the detection target differs. The learning data generation device according to claim 1 or 2, wherein the video data creation unit creates multiple video data sets in different environments in which the detection target exists. The learning data generation device according to any one of claims 1 to 6, wherein the video data created by the video data creation unit includes a plurality of the detection targets. a sensor that detects the detection target using a learning model that is machine-learned using the learning data generated by the learning data generation device according to any one of claims 1 to 7;
and an automatic door that opens and closes based on the detection results of the sensor. a moving image data creation step of creating moving image data in which the three-dimensional model of the detection target moves in a virtual space;
a variation data creation step of creating brightness variation data or difference data between frames obtained by comparing brightness values with a threshold value in a two-dimensional image projected from the moving image data;
a labeling step of assigning a label indicating that the detection target is a region in the brightness change data or the difference data corresponding to the detection target;
a learning data generation step of generating, as learning data, a set of the brightness change data or the difference data, an area corresponding to the detection target, and a label assigned in the labeling step. a learning data generating step of generating learning data by the learning data generating method according to claim 9;
a machine learning step of performing machine learning using the learning data generated in the learning data generation step. A control program for causing a computer to function as the learning data generation device described in claim 1, the control program causing the computer to function as the video data creation unit, the variation data creation unit, the labeling unit, and the learning data generation unit. A computer-readable recording medium having the control program described in claim 11 recorded thereon. a moving image data creating unit that creates moving image data in which a three-dimensional model of the detection target moves in a virtual space;
a variation data generating unit that generates brightness variation data or difference data in a two-dimensional image projected from the moving image data;
a labeling unit that assigns a label indicating that the detection target is a region in the brightness change data or the difference data that corresponds to the detection target;
a learning data generation unit that generates, as learning data, a set of the luminance change data or the difference data, an area corresponding to the detection target, and a label assigned by the labeling unit;
the display unit that displays the two-dimensional image includes an event generation unit that generates an event in each pixel on the display unit when a luminance value of the pixel exceeds a first threshold value or falls below a second threshold value, and a position output unit that outputs a position on the display unit of the pixel at which the event generation unit has generated the event;
The variation data creation unit creates the luminance variation data using the position output by the position output unit.