JP7437738B2 - Machine learning system, training dataset generation system, and machine learning program - Google Patents

Description

The present invention relates to a machine learning system, a learning dataset generation system, and a machine learning program.

Traditionally, cameras such as surveillance cameras and so-called AI (Artificial Intelligence) cameras have been placed indoors in stores, etc., and objects such as people reflected in frame images taken by these cameras are detected using learned object detection. There are known devices and systems in which a trained neural network model for object recognition is used to recognize the detected object using a neural network model for object recognition (for example, see Patent Document 1).

In machine learning of neural network models for inference, including neural network models for object detection and neural network models for object recognition, as described above, the probability distribution of the learning target (for example, in the case of a neural network model that recognizes a person's age) uses deep learning to reproduce the probability distribution of people's ages using a neural network model.

By the way, among the techniques that apply the neural network model for inference with respect to objects in images, there is a technique that performs Person Re-Identification, which assigns the same ID to the same person photographed with different video cameras. This is a neural network model for person identification that identifies people photographed by different video cameras based on the human features in each frame image that constitutes each video photographed by a plurality of video cameras. In general, the way people are captured changes greatly when different video cameras are used, parameters such as brightness and distortion vary greatly between video cameras, and because people move, the appearance of the same person can vary greatly, etc. Therefore, Person Re-Identification (identification of people photographed with different video cameras) is extremely difficult, and many researchers today are still competing to develop appropriate methods.

There is a dataset called Market1501 as a dataset for machine learning and testing of neural network models for Person Re-Identification (the site where you can download this dataset is https://github.com/vana77/Market -1501 Attrite). This dataset stores data for 751 people (image data (less than 10 frame image data taken from each angle for each person) and attribute data) taken with cameras installed in shopping centers. ing.

Furthermore, in order to create a data set for machine learning and testing of a neural network model for Person Re-Identification, there is a technology called Person X that generates a data set using three-dimensional computer graphics (3DCG). With this Person Dissecting-Person-Re-ID-from-the-Viewpoint-of-Viewpoint).

JP 2018-093283 Publication

However, in the first place, machine learning of (deep) neural network models for inference requires bias in the training data in the machine learning dataset (hereinafter referred to as the "learning dataset"), as shown in Figure 14. Or, due to the small amount of training data, the probability distribution of the learning target (for example, in the case of a neural network model that recognizes a person's age, the probability distribution of a person's age) cannot be used with a trained neural network model. The problem is that it cannot be accurately reproduced. For this reason, naturally, even in devices and systems that perform inference (detection and recognition) on objects such as people photographed with a conventional camera as described in Patent Document 1, the probability distribution of the learning target is calculated using a trained neural network. The problem was that it could not be accurately reproduced with a model. Note that "DL" and "DNN" in FIG. 14 are abbreviations for "Deep Learning" and "Deep Neural Network," respectively.

Taking the conventional neural network model for inference regarding objects in images as described in Patent Document 1 as an example, to explain in detail the bias of the training data in the above training data set, in general, objects in images The images used as learning data for the inference neural network model for are collected using cameras installed in existing (specific) environments such as stores. However, in such an actual (specific) environment, events that occur over a certain period of time will be biased, so images collected using cameras installed in existing environments will be biased. Put it away. For example, if a camera installed at a store entrance is used to create images of people in videos taken over a week, the clothing of the people in these images will depend largely on the season of the period, and Regarding gender and age, there is a high possibility that bias will occur if data is collected over a period of about one week. Further, whether or not to perform a specific action (for example, using a shopping cart) also depends on the customer at the time, so it is unclear whether the required number of image data of customers performing a specific action will be collected. Note that the lack of specific types of learning data, such as image data of customers performing the above-mentioned specific actions, can also be considered as a type of bias in learning data.

The bias in the training data in the above training data set is caused by the limited expressive power of the inference neural network model for objects in images, and the probability distribution of the above learning target is determined by the trained inference neural network. Exacerbating the problem that models cannot accurately reproduce.

The problem that the probability distribution of the learning target cannot be accurately reproduced by the trained neural network model for inference is remarkable in the trained neural network model for Person Re-Identification. Even if the Market 1501 data set mentioned above is used, the learning data included in the Market 1501 training data set is the data (image data and attribute data) of 751 people photographed by a camera installed in a certain shopping center. ) Therefore, in order to perform machine learning of a neural network model for Person Re-Identification, the amount of training data is small, and since it is based on image data taken at a certain time, The training data is highly biased. Further, regarding the learning data set generated using the above-mentioned Person Therefore, even if machine learning is performed using the Market 1501 training dataset or the training dataset generated by Person X, it is not possible to obtain a trained neural network model for Person Re-Identification that is suitable for practical use.

The present invention solves the above problems, and provides a machine learning system and learning data that can generate a highly accurate trained neural network model that can accurately reproduce the probability distribution of a learning target. The purpose is to provide a set generation system and a machine learning program.

In order to solve the above problems, a machine learning system according to a first aspect of the present invention uses a training data set including training images created using computer graphics to create a neural network for inference with respect to objects in the images. A machine learning system that performs machine learning of a network model, comprising: a data set generation unit that generates a learning data set corresponding to the probability distribution of entities of each component in a specific environment; and a data set generation unit that uses the learning data set. A neural network generation unit that generates a trained neural network model by performing deep learning; and a neural network generation unit that generates a trained neural network model by inputting a test image included in a test data set to the trained neural network model and generating an object in the test image. an inference unit that performs inference processing for the inference; a verification unit that verifies bias in the training data in the training data set of the trained neural network model based on the inference result by the inference unit; a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the learning data; and a learning that corresponds to the changed probability distribution of the entity of each component. post-updated training data set generation control that controls to generate a training data set by the data set generation unit and generate an updated training data set based on the training data in the training data set; and the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. and a learning repetition control unit that controls each of the learning images constituting the learning data set generated by the data set generating unit, each of the configurations selected from the probability distribution of the entity of each component in the specific environment. This is an image drawn based on the entities of each of the constituent elements of a plurality of patterns generated by adding random noise to the entities of the elements .

A machine learning system according to a second aspect of the present invention is a machine that performs machine learning of a neural network model for inference with respect to an object in an image using a learning data set that includes a learning image created using computer graphics. The learning system includes a dataset generation unit that generates a learning dataset corresponding to the probability distribution of entities of each component in a specific environment, and a learning system that performs deep learning using the training dataset. a neural network generation unit that generates a trained neural network model; and an inference unit that inputs a test image included in a test data set to the trained neural network model and performs inference processing on an object in the test image. , a verification unit that verifies the bias of the training data in the training data set of the trained neural network model based on the inference result by the inference unit; and a verification unit that verifies the bias of the training data found by the verification by the verification unit. a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to decrease the probability distribution of the entity of each component; an updated learning data set generation control unit that is generated by the generation unit and controls to generate an updated learning data set based on the learning data in the learning data set, and the neural network generation unit a learning repetition control unit that performs deep learning using the updated learning data set to generate a trained neural network model that is more accurate than the current trained neural network model; Identification of generating specific environment entity information, which is data of simulation results along a time series of entities in the specific environment corresponding to the entities in the specific environment, based on probability distributions of entities in each component in the specific environment. an environment generation section, the dataset generation section generates a test dataset for the neural network model corresponding to the specific environment entity information generated by the specific environment generation section, in addition to the learning dataset. The inference unit inputs the test image included in the test data set generated by the data set generation unit into the trained neural network model, and performs inference processing on the object in the test image. I do.

In this machine learning system, the learning repetition control unit causes the specific environment changing unit to: In order to reduce the bias of the learning data, the probability distribution of each component in the specific environment is changed, and the post-updated training data set generation control unit changes the actual state of each component after the change. a process of generating the learning data set corresponding to the probability distribution using the data set generation unit, and generating an updated learning data set based on the learning data in the learning data set; A process of generating a trained neural network model with higher accuracy than the current trained neural network model using the updated training data set by a neural network generation unit, and a process of generating the trained neural network model with higher accuracy than the current trained neural network model. Using a model, the inference unit performs inference on the object in the test image, and the verification unit determines the bias of the learning data in the current learning data set based on the inference result by the inference unit. It is desirable to repeat the verification process.

In this machine learning system, the specific environment changing unit changes the probability distribution of each component entity in the specific environment according to the value of the adjustment parameter, and according to the changed probability distribution of each component entity. , it is desirable to generate a different entity of each component each time with the same adjustment parameters.

In this machine learning system, it is preferable that the dataset generation unit generates annotation data of the learning dataset.

In this machine learning system, the neural network model for inference has the ability to identify people photographed with different video cameras based on the features of the person in each frame image constituting each video photographed with multiple video cameras. A neural network model for person identification may be included.

A learning dataset generation system according to a third aspect of the present invention is a learning dataset generation system that generates a learning dataset for performing machine learning of a neural network model for inference with respect to an object in an image. , the training data set includes training images created using computer graphics, and the test images included in the test data set are input to the current trained neural network model, and the test images in the test images are an inference unit that performs inference processing on an object; a verification unit that verifies bias of training data in a training data set of the currently trained neural network model based on the inference result by the inference unit; a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment reflected in the learning image so as to reduce the bias of the learning data found through verification; and each component after the change. An updated training dataset that controls the generation of a training dataset that corresponds to the probability distribution of the entity, and generates an updated training dataset based on the training data in this training dataset. Each of the learning images constituting the updated learning data set includes a generation control unit that applies random noise to the entity of each component selected from the probability distribution of the entity of each component in the specific environment. In addition, it is an image drawn based on the substance of each of the constituent elements of the generated plurality of patterns .

A learning dataset generation system according to a fourth aspect of the present invention is a learning dataset generation system that generates a learning dataset for performing machine learning of a neural network model for inference with respect to an object in an image. , the training data set includes training images created using computer graphics, and the test images included in the test data set are input to the current trained neural network model, and the test images in the test images are an inference unit that performs inference processing on an object; a verification unit that verifies bias of training data in a training data set of the currently trained neural network model based on the inference result by the inference unit; a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment reflected in the learning image so as to reduce the bias of the learning data found through verification; and each component after the change. An updated training dataset that controls the generation of a training dataset that corresponds to the probability distribution of the entity, and generates an updated training dataset based on the training data in this training dataset. a generation control unit, and a specific environment entity that is simulation result data along a time series of entities of the specific environment corresponding to the entities of each component based on the probability distribution of the entities of each component in the specific environment. a specific environment generation unit that generates information; and a data set generation unit that generates a test dataset for the neural network model corresponding to the specific environment entity information generated by the specific environment generation unit, The unit inputs the test image included in the test data set generated by the data set generation unit into the current trained neural network model, and performs inference processing on the object in the test image.

In this learning dataset generation system, the specific environment changing unit changes the probability distribution of each component entity in the specific environment according to the value of the adjustment parameter, and changes the probability distribution of each component entity after the change. It is desirable to generate a different entity of each component each time with the same adjustment parameters according to a probability distribution.

A machine learning program according to a fifth aspect of the present invention is for performing machine learning of a neural network model for inference with respect to an object in an image using a learning data set including a learning image created using computer graphics. This machine learning program includes a data set generation unit that generates a learning data set corresponding to the probability distribution of entities of each component in a certain specific environment, and a deep learning using the learning data set. A neural network generation unit generates a trained neural network model by inputting a test image included in a test dataset to the trained neural network model, and performs inference processing on objects in the test image. an inference unit that performs the following: a verification unit that verifies the bias of the training data in the training data set of the trained neural network model based on the inference result by the inference unit; a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the data; and a learning data set corresponding to the changed probability distribution of the entity of each component. an updated learning dataset generation control unit that controls the updated learning dataset to be generated by the dataset generation unit and to generate an updated learning dataset based on the learning data in the learning dataset; Learning controlled so that the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. Each learning image constituting the learning data set generated by the data set generation unit, which functions as a repetition control unit, is an entity of each component selected from the probability distribution of the entity of each component in the specific environment. This is an image drawn based on the substance of each component of a plurality of patterns generated by adding random noise to the above.

A machine learning program according to a sixth aspect of the present invention is for performing machine learning of a neural network model for inference with respect to an object in an image using a learning data set including a learning image created using computer graphics. This machine learning program includes a data set generation unit that generates a learning data set corresponding to the probability distribution of entities of each component in a certain specific environment, and a deep learning using the learning data set. A neural network generation unit generates a trained neural network model by inputting a test image included in a test dataset to the trained neural network model, and performs inference processing on objects in the test image. an inference unit that performs the following: a verification unit that verifies the bias of the training data in the training data set of the trained neural network model based on the inference result by the inference unit; a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the data; and a learning data set corresponding to the changed probability distribution of the entity of each component. an updated learning dataset generation control unit that controls the updated learning dataset to be generated by the dataset generation unit and to generate an updated learning dataset based on the learning data in the learning dataset; Learning controlled so that the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. a repetition control unit, and a specific environment entity that is simulation result data along a time series of entities of the specific environment corresponding to the entities of each component based on the probability distribution of the entities of each component in the specific environment. The data set generation unit functions as a specific environment generation unit that generates information, and in addition to the learning dataset, the neural network model corresponds to the specific environment entity information generated by the specific environment generation unit. The inference unit inputs the test images included in the test data set generated by the data set generation unit into the trained neural network model, and the inference unit inputs the test images included in the test data set generated by the dataset generation unit to Perform inference processing on objects.

In this machine learning program, the specific environment changing unit changes the probability distribution of each component entity in the specific environment according to the value of the adjustment parameter, and according to the changed probability distribution of each component entity. , it is desirable to generate a different entity of each component each time with the same adjustment parameters.

The machine learning system according to the first aspect of the present invention, the machine learning system according to the second aspect, the learning dataset generation system according to the third aspect, the learning dataset generation system according to the fourth aspect, and the fifth aspect According to the machine learning program and the machine learning program according to the sixth aspect, the probability distribution of each component entity in a specific environment is changed so as to reduce bias in learning data, and the probability distribution of each component entity after the change is It is possible to generate a learning data set corresponding to the probability distribution of an entity, and to generate an updated learning data set based on the learning data in this learning data set. In other words, this updated learning data set is a learning data set that reflects the learning data in the learning data set that has been regenerated so as to reduce the bias of the learning data in the learning data set. Therefore, the training data in this updated training dataset has less bias, so by performing deep learning using this updated training dataset, it is better than the current trained neural network model. It is possible to generate a highly accurate trained neural network model that can accurately reproduce the probability distribution of the learning target.

FIG. 1 is a block configuration diagram showing a schematic hardware configuration of a machine learning system according to an embodiment of the present invention. FIG. 2 is a functional block configuration diagram around a CPU in a learning server of the same machine learning system. A flowchart of machine learning processing in the same machine learning system. An explanatory diagram of parameters constituting each element of the application environment. An explanatory diagram of the generation (change from default value) of the entity of the application environment (store situation). (a) is a graph showing an example of the probability distribution of entities in the default store situation, (b) is an explanatory diagram showing an example of a slide bar for adjusting the probability distribution of entities in the store situation, and (c) is an explanatory diagram showing an example of the probability distribution of entities in the store situation. , a graph showing an example of the probability distribution of the entity of the store situation after setting the adjustment parameters using the slide bar. The figure which shows the example of the adjustment parameter setting screen which incorporated the function of the said slide bar. FIG. 3 is an explanatory diagram of a test data set generation process in the machine learning system. FIG. 2 is an explanatory diagram of a learning data set generation process in the machine learning system. An explanatory diagram of deep learning of a DNN model for each inference in the same machine learning system. An explanatory diagram of inference processing of each trained DNN model in the same machine learning system. An explanatory diagram of a process for verifying bias in learning data in the same machine learning system. FIG. 3 is an explanatory diagram of a process of generating adjustment parameter change data in the same machine learning system. An explanatory diagram of the problem that existed in conventional machine learning of neural network models for inference, in which the probability distribution of the learning target could not be accurately reproduced with a trained neural network model.

EMBODIMENT OF THE INVENTION Hereinafter, a machine learning system, a learning dataset generation system, and a machine learning program according to embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block configuration diagram showing a schematic hardware configuration of a machine learning system 10 (corresponding to a "machine learning system" and a "learning data set generation system" in the claims) according to the present embodiment. Although the present embodiment shows an example in which the machine learning system 10 is composed of only one computer (learning server 1), it may be a client-server type system. In the following description, the term "applied environment" refers to a specific environment (for example, a certain drug store, a certain convenience store, etc.) that appears in the learning images and test images. In this embodiment, an example in which the above-mentioned application environment is a store such as a drug store will be described. Further, in this embodiment, an example will be described in which the learning image and the test image are images created using three-dimensional computer graphics (3DCG).

As shown in FIG. 1, a learning server 1 that constitutes a machine learning system 10 includes a CPU 2 that controls the entire device and performs various calculations, and a communication unit 3 that distributes trained neural network models after machine learning. , a display 4, an input operation unit 5 for inputting adjustment parameters, a RAM (Random Access Memory) 6, and a hard disk 7 for storing various data and programs.

The program stored in the hard disk 7 includes a neural network model for inference, such as ReID (Person Re-Identification), for objects in images, using a learning data set that includes training images created with 3DCG. A machine learning program 15 for performing machine learning is included. This machine learning program 15 is a program for causing the learning server 1 to realize the functions of the machine learning system 10.

Further, the data stored in the hard disk 7 includes an application environment generation material storage section 11, a data set storage section 12, a DNN model storage section 13, and an inference result storage section 14.

The applied environment generation material storage unit 11 contains a 3D model of a person in this applied environment (a model of an image of a person created with 3DCG), various textures for reproducing the actual store in this applied environment, etc. Contains image materials created with 3DCG. Further, the applied environment generation material storage unit 11 includes statistical data regarding (detailed information on) people and stores in this applied environment (for example, in an area where a certain drugstore is located). This statistical data includes, for example, data on the probability distribution of human entities for each gender and age that can be generated in this application environment.

The above dataset storage unit 12 stores test datasets and learning data for inference DNN (Deep Neural Network) models (hereinafter referred to as "DNN models") for human detection, feature detection (vectorization), tracking, ReID, etc. It stores (saves) the data set for use. The DNN model storage unit 13 stores trained DNN models for inference such as person detection, feature detection, tracking, and ReID (“neural network model for person identification” in the claims). The inference result storage unit 14 stores the results of inference processing for objects in each test image included in the test data set stored in the data set storage unit 12 and data for changing adjustment parameters of the application environment (hereinafter referred to as (referred to as "adjustment parameter change data") is stored (saved).

Next, with reference to FIG. 2, the functional blocks of the CPU 2 in the learning server 1 will be described. The CPU 2 includes, as functional blocks, a control unit 20 (“updated learning data set generation control unit” and “learning repetition control unit” in the claims) and an application environment generation unit 21 (“specific environment generation unit” in the claims). ''), a data set generation section 22, a learning section 23 ("neural network generation section" in the claims), and an inference section 24. The above application environment generation section 21 includes a learning generation section 27, and the inference section 24 includes a verification section 25. Note that in this embodiment, the inference section 24 includes the verification section 25, but the inference section 24 and the verification section 25 may be separate functional blocks.

The control unit 20 described above controls other functional blocks within the CPU 2. The application environment generation unit 21 generates application environment entities (in this embodiment In this case, simulation result data (hereinafter referred to as "applied environment entity information") is generated in chronological order (hereinafter referred to as "applied environment entity information"). "Application environment entity information" corresponds to "specific environment entity information" in the claims. When generating the training data set for each inference DNN model such as person detection, feature detection, tracking, ReID, etc., the learning generation unit 27 performs adjustment parameter change processing based on the previous inference result and after this change. Based on the adjustment parameters, processing is performed to change the probability distribution of each component entity in the application environment.

The data set generation unit 22 generates a test data set for each of the inference DNN models corresponding to the application environment entity information generated by the application environment generation unit 21, and the probability of the entity of each component in the application environment. A training data set corresponding to the distribution is generated.

The learning unit 23 performs deep learning of the DNN model for each inference such as person detection, feature detection, tracking, and ReID using the learning dataset generated by the dataset generation unit 22. A trained DNN model for these inferences is generated. The inference unit 24 inputs each test image included in the test data set generated by the data set generation unit 22 to the trained DNN model for each inference, and generates each test image. Perform inference processing on the objects inside. The verification unit 25 included in the inference unit 24 performs learning on the training data set of the trained DNN model for each of the above inferences based on the results of inference processing (inference results) for the objects in each of the test images. Verify the bias in the data. Further, although the details will be described later, the verification unit 25 and the application environment generation unit 21 are configured to check the bias of the training data in the training data set of each trained DNN model for inference, which is found through the verification by the verification unit 25. The probability distribution of each component (person and store) entity in the application environment is changed so as to reduce the probability distribution. That is, the verification unit 25 and the application environment generation unit 21 correspond to the specific environment change unit in the claims.

The control unit 20 causes the data set generation unit 22 to generate a learning data set corresponding to the probability distribution of the entity of each component (person and store) after the modification by the verification unit 25 and application environment generation unit 21 described above. Then, control is performed to generate an updated learning data set based on the learning data in this learning data set. That is, the control unit 20 functions as an updated learning data set generation control unit in the claims.

In addition, the control unit 20 performs deep learning on each of the above-mentioned inference DNN models using the above-mentioned updated training data set, so that the control unit 20 performs deep learning on each of the above-mentioned inference DNN models using the above-mentioned updated learning data set. Control is performed to generate highly accurate trained DNN models for each inference. That is, the control section 20 also functions as a learning repetition control section in the claims.

Next, with reference to FIG. 3, the machine learning process performed by the machine learning system 10 described above will be described. First, prior to this machine learning process, the user needs to construct the base of the application environment using 3DCG by a system development company or the system development department of his/her own company (S1). When the applied environment is a store, as shown in FIG. 4, the applied environment is composed of the elements of "store", "people", "people's behavior", and "store situation". Each element is made up of various parameters. For example, as shown in Figure 4, a "store" is composed of parameters (a kind of evaluation axis) such as size, ceiling height, wall/floor texture, shelf position/orientation, etc. ” is composed of parameters such as gender, age, race, height, weight, clothing, and behavioral characteristics. "Human behavior" represents the type of behavioral characteristics included in the above-mentioned "human" parameter. The "store situation" is composed of parameters such as crowding level, gender distribution, age distribution, average customer staying time distribution, and behavior characteristic distribution. The initial values (default values) of each of the above parameters may be set automatically by the CPU 2 of the machine learning system 10 based on statistical data, or by the system development department of the user's company or the system development company. A person in charge of development may input from the input operation section 5. By arbitrarily adjusting some of the above parameters (default values) that make up each element of "store", "person", and "store situation", various patterns can be created. It is possible to express the probability distribution of the constituent elements of "store", "person", and "store situation". Here, "entity" means data representing these entities (for example, stores and people).

In this machine learning system 10, for example, for the component of "person", each time a different person's entity (data representing) is calculated from the probability distribution of the entity of "person" (according to the probability distribution of the entity of "person"). can be generated. Furthermore, in this machine learning system 10, it is possible to generate (data representing) different store entities each time from the probability distribution of the "store" entities. The probability distribution of the entity of "store status" is a probability distribution for generating and controlling stores and people (mainly people) in the time axis direction, and can be manually (or automatically) created by the user using the slide bar described later. ) by changing the adjustment parameters, and as a result, the substance of the "store situation" from now on will change. The 3DCG frame image 31 of the application environment shown in FIG. 4 is generated based on the entities of the above-mentioned "store", "person", and "store situation" components.

To build the base of the application environment in S1 above, we used various image materials created with 3DCG, such as a 3D model of a person, various textures to reproduce the actual store, and various statistical data (for example, a drug This also includes processing for storing data such as statistical data for the area where the store is located in the applied environment generation material storage section 11 described above.

When the application environment base construction process in S1 is completed, in the first application environment entity generation process, the user U uses the input operation unit 5 to change the application environment. (adjust) the adjustment parameters based on predetermined values (for example, the results of analysis of sales, customer demographics, customer trends, etc. of the store (application environment) and the statistical data about stores in the area). value) and instructs generation of an application environment, the application environment generation unit 21 changes the application environment from the default value and creates the entity of the changed application environment (to be more precise, the application environment after change). Data of computer simulation results along the time series of the entity (the entity of the store situation) is generated (S2).

More specifically, first, the user U uses the input operation unit 5 (mouse, etc.) to set adjustment parameters to predetermined values using a slide bar, which will be described later. Set some of the parameters (change from the initial value (default value)). Then, the application environment generation unit 21 sets other unset parameters based on the statistical data stored in the application environment generation material storage unit 11 and takes into account the values of the parameters that have already been set. is changed from the default value) to generate the probability distribution of the entities "store", "person", and "store status" according to the values of the above-mentioned already set parameters.

Specifically, the user U uses the input operation unit 5 (mouse, etc.) to adjust (set) parameters such as the size of the store by setting adjustment parameters to predetermined values using a slide bar described later. Then, the applied environment generation unit 21 saves unset parameters such as the height of the ceiling and the number of cameras to the applied environment generation material storage unit 11 according to the level of parameters such as the size of the store that have already been set. The probability distribution 32 of the "store" entity shown in FIG. 5 is generated based on the statistical data (mainly data expressed as a probability distribution) stored in the store.

In addition, the user U uses the input operation unit 5 to set adjustment parameters to predetermined values using a slide bar, thereby adjusting parameters such as the degree of crowding, gender distribution, age distribution, behavioral characteristic distribution, etc. shown in FIG. When (setting) is performed, the applied environment generation unit 21 generates the (unset) average customer stay time distribution according to the levels of the parameters such as the congestion level, gender distribution, age distribution, behavioral characteristic distribution, etc. that have already been set. The parameters are set based on the statistical data stored in the applied environment generation material storage unit 11, and the probability distribution 33 of the "store situation" entity shown in FIG. 5 is generated. According to the probability distribution 33 of this "store situation" entity, the applied environment generation unit 21 determines the timing (time) of generating (entering) and deleting (exiting) a person, and determines the gender and gender of the person when generating the person.・Determine age. More specifically, the applied environment generation unit 21 generates people as shown in FIG. (entering the store), deletion (leaving the store), etc., determine the gender and age of the person to be generated, as shown in Figure 5, based on the gender distribution and age distribution, and assign appropriate actions to the generated person.

Further, the applied environment generation unit 21 selects the unused parameters of the "person" shown in FIG. Determination parameters (parameters other than gender, age, and behavioral characteristics) are set based on data such as statistical data stored in the applied environment generation material storage unit 11, and a probability distribution of the "person" entity is generated. Note that the probability distribution 34 of the "person" entity shown in FIG. is the probability distribution of

Next, the applied environment generation unit 21 generates the store situation entity (applied environment entity) based on the probability distributions 32 to 34 of the entities of the components of “store”, “person”, and “store situation”. Time-series computer simulation result data (time-series information on the actions of customers (or clerks and customers) in the store) (the above-mentioned applied environment entity information) is generated. This computer simulation result data includes a 3D model of a person (a model of a human image created with 3DCG) read from the applied environment generation material storage unit 11, and various models for reproducing the actual store in this applied environment. Contains image data created with 3DCG, such as textures. The above store situation entities (data of computer simulation results along the time series) are generated from the probability distributions 32 to 34 of the entities "store", "person", and "store situation". Therefore, random elements are introduced. In short, even if the values of the adjustment parameters are exactly the same and the probability distributions 32 to 34 of the entities ``store'', ``person'', and ``store situation'' are the same, the applied environment generation unit 21 calculates these probabilities. From the distributions 32 to 34, different stores and people are generated each time.

Next, with reference to FIGS. 6(a), 6(b), and 6(c), a slide bar for adjusting (changing) the probability distribution of the above-mentioned "store status" entity will be described. Here, the actual probability distribution of the default (initial state) "store status" is based on the analysis results of the store's sales, customer base, customer trends, etc., and statistical data about stores in the area. Suppose. FIG. 6(a) shows an example of the probability distribution of the default "store status" entity. However, in Figures 6(a) and (c), the default "store status" is assumed to be the only parameters (evaluation axes) that constitute the probability distribution of the "store status" entity: the gender distribution and the age distribution. It shows an example of the probability distribution of the entity ``Situation''. Graphs 40a, 40b, 40c, and 40d in FIGS. 6A and 6C show appearance probability distributions of people of each age and gender.

If the probability distribution of the default "store situation" entity (in this example, the appearance of a person) is the probability distribution shown in FIG. , by sliding the slide bar 41 for adjusting the gender distribution and the slide bar 42 for adjusting the age distribution shown in FIG. 6(b) in the direction of the respective arrows in FIG. 6(b), the vector Set the tunable parameters expressed in the format. Thereby, the adjustment parameter becomes a vector such as [0.9, 0.0, . . . ], for example. The first element (0.9) in this vector represents the femininity level adjusted by the slide bar 41, and the second element (0.0) represents the senility level adjusted by the slide bar 42. represent.

By setting (adjusting) the adjustment parameters using the slide bar 41 and the slide bar 42 described above, the probability distribution of the entity of the (default) "store situation" shown in FIG. 6(a) is changed to the one shown in FIG. 6(c). Change as shown. That is, graphs 40a, 40b, 40c, and 40d indicated by broken lines in FIG. 6(c) change to graphs 43a, 43b, 43c, and 43d indicated by solid lines, respectively, as indicated by black arrows in FIG. 6(c). As the proportion of women increases, the proportion of young people also increases, as shown by the white arrow in FIG. 6(c). Although it is not shown in Figures 6(a) and (c) above because it is not possible to display axes with four or more dimensions, the axis of the probability distribution of the entity "Store status" includes "Average customer stay time distribution". There are also axes such as "", "behavioral characteristic distribution" (see Figure 4), etc., and the frequency distribution based on these axes (evaluation axes) represents the probability distribution of the "store situation" entity, and according to this probability distribution, the store The people inside are controlled (the timing (time) of creating (entering) and deleting (exiting) people, etc., determining the gender and age of people, and assigning actions). As mentioned above, the value of each parameter (evaluation axis) that makes up the probability distribution of the entity of the default "store status" is changed to the value of each element that makes up the adjustment parameter set with the slide bar (for example, [0 .9, 0.0, ...]), the probability distribution of the entity of the application environment (the entity of "store situation") is changed (variation ) can be made.

FIG. 7 shows an example of an adjustment parameter setting screen 50 incorporating the slide bar function described in FIG. 6(b) above. This adjustment parameter setting screen 50 is displayed on the display 4. The adjustment parameter setting screen 50 has an adjustment parameter setting column 51 on the right side, and images 64 taken by each camera are displayed on the left side. The captured image 64 of each camera changes when the user U adjusts each slide bar provided in the adjustment parameter setting field 51 using the input operation unit 5 (mouse or the like). The slide bars provided in the adjustment parameter setting column 51 in FIG. 7 are just examples; a slide bar column 52 for adjusting parameters of "store status" related to people (customers), and a slide bar column 52 for adjusting parameters of "store status" related to people (customers); It consists of a slide bar field 53 for parameter adjustment and a store type change field 54.

The slide bar column 52 for adjusting the parameters of "store status" related to the above people (customers) has a slide bar 55 for adjusting the number of customers, a slide bar 56 for adjusting the proportion of young people (for adjusting the proportion of people aged 0 to 29), (slide bar), middle-age group ratio adjustment slide bar 57 (slide bar for ratio adjustment of people aged 30 to 59), female ratio adjustment slide bar 58, and customer behavior variety adjustment slide bar 59. The customer number adjustment slide bar 55 described above is a slide bar for adjusting the "degree of congestion" in FIG. 4. The young age group ratio adjustment slide bar 56 and middle-aged age group ratio adjustment slide bar 57 described above are slide bars for adjusting the "age distribution" in FIG. 4.

The slide bar column 53 for adjusting the parameters of the above-mentioned "store" is composed of a camera number adjustment slide bar 60, a width adjustment slide bar 61, a lighting intensity adjustment slide bar 62, and a product inventory amount adjustment slide bar 63. ing. The camera number adjustment slide bar 60, width adjustment slide bar 61, and illumination intensity adjustment slide bar 62 are used to adjust the “number of cameras,” “width,” and “intensity” of “illumination” in FIG. 4, respectively. This is a slide bar for adjustment.

The above store type change field 54 is a window for selecting a store type. Further, at the lower right of the adjustment parameter setting screen 50, an image 65 of the entire store interior viewed from above is displayed.

Next, returning to the flowchart of FIG. 3, the processing after the application environment entity information generation processing described in S2 above will be described. The data set generation unit 22 of the CPU 2 generates a test data set corresponding to the current application environment entity information generated in S2 (S3). Specifically, the data set generation unit 22 generates data based on the application environment entity information (data of computer simulation results along the time series of the application environment entity (store situation entity)) generated by the application environment generation unit 21. Then, as shown in FIG. 8, processing is performed to generate frame images in chronological order from each camera (disposed in a store, which is an application environment), and annotation processing is performed on these frame images. Then, the dataset generation unit 22 associates each generated frame image (for example, the frame image 71 photographed with the first camera) with the annotation information (an XML file related to it) 73 of this frame image, and generates these data. is stored as test data in a test data set in the data set storage unit 12.

In the case of this embodiment, the annotation information of each frame image described above is annotation information about a person in each frame image. Note that 100, 110 and 200, 300 in the coordinates of the annotation information regarding the frame image 71 in FIG. Represents the x and y coordinates of the lower end point. Further, the ID in the annotation information is an ID assigned to the same person throughout the cameras. The person in each frame image described above is generated by the dataset generation unit 22 itself from the applied environment entity information, so the dataset generation unit 22 generates the person in each frame image based on the applied environment entity information. It is possible to generate accurate annotation information about

After the test data set generation process in S3 is completed, the control unit 20 of the CPU 2 uses the application environment generation unit 21 and the data set generation unit 22 to generate a probability distribution of the entity of each component in the current application environment. A process of generating a learning data set corresponding to (S4) is performed. Specifically, in the first training data set generation process, the learning generation unit 27 of the applied environment generation unit 21 uses the probability distribution 32a (see FIG. 9) of the store entity generated in S2 above and the human A process of selecting one store entity and one or more person entities (one person in FIG. 9) from the entity probability distribution 76a (see FIG. 9), and randomly selecting the selected store entity and person entity. The process of adding noise (normally distributed noise) and generating multiple patterns of store entities and human entities is repeated. Then, the dataset generation unit 22 generates a learning dataset including various patterns of learning images and their annotation data from the plurality of patterns of stores and human entities generated by adding the above random noise. In this way, random noise is added to each of the store and person entities to generate multiple patterns of store and person entities, and various patterns of learning images (frame images) are drawn based on these entities. The reason is to ensure sufficient distribution of training data in the generated training data set.

More specifically, the learning generation unit 27 generates the lens parameters, the position and orientation of the camera, the color of the store texture (texture of walls, floors, shelves, etc.), and the items displayed on the shelves in the selected store entity. In addition to generating multiple patterns (the actual store) by adding random noise to the material, lighting intensity, color, light direction, shadow, etc. Generate multiple patterns of people (substances) by adding random noise to the state, texture of clothes and skin, various materials, etc. The dataset generation unit 22 arranges each of these multiple patterns of people (entities) in each of the multiple patterns of store entities described above, and draws them as a frame image input from a camera placed inside the store entity. (for example, 78 and 79 in FIG. 9) and annotation data generation processing for these frame images are performed to generate a learning data set including learning data of various patterns. As a result, each learning image constituting the learning data set generated by the data set generation unit 22 is generated based on each of the stores and people generated by the applied environment generation unit 21, with random noise added. It becomes a drawn image. As shown in FIG. 9, the dataset generation unit 22 stores the generated learning dataset in the dataset storage unit 12.

Therefore, each of the learning images constituting the learning data set is a continuous frame image taken by the same camera in the test data set shown in FIG. 68b, 68c, and the consecutive frame images 69a, 69b, 69c taken by the second camera), the order of storage in the training data set is slightly different from that of the store in the previous training image (one This is an image of a person in a store (an arrangement of the store in the previous learning image) that is slightly different from the person in the previous learning image (an arrangement of the person in the previous learning image). That is, the image is a combination of images 78a and 79a of mutually different stores and images 78b and 79b of mutually different people, like the consecutive learning images 78 and 79 (in the order of storage) in FIG.

In addition, in the second and subsequent training data set generation processing, the learning generation unit 27 of the application environment generation unit 21 reduces the bias of the learning data found as a result of verifying the previous inference results. , the probability distribution 32a of the store entity and the probability distribution 76a of the human entity in the applied environment are changed, and the store entity and the person are determined from the changed probability distribution 32b of the store entity and the probability distribution 76b of the human entity. The difference is that the entities of stores and people are generated by selecting entities. However, details of the second and subsequent training data set generation processing will be described later for convenience of explanation.

When the training data set generation process in S4 is completed, the control unit 20 of the CPU 2 instructs the learning unit 23 to perform deep learning of each inference DNN model. In response to this, the learning unit 23 performs deep learning on the DNN models for human detection, feature detection (vectorization), tracking, and ReID inference using the learning data set generated in S4 above. , a trained DNN model for these inferences is generated (S5). More specifically, as shown in FIG. 10, the learning unit 23 reads each inference DNN model to be learned from the DNN model storage unit 13, loads it onto the memory, and uses the learning data generated in S4 above. After performing deep learning of the DNN model for these inferences using the set (the learning image 81 and annotation information 82 ("annotation data" in the claims) in each training data of The learned DNN model is saved in the DNN model storage unit 13 (S11 in FIG. 10). The human detection, feature detection (vectorization), tracking, and ReID DNN models in Figure 10 are the minimum required model group for ReID, and the inference DNN models that are subject to the above deep learning are , a DNN model for gender determination and age determination may be included.

When the human detection, feature detection (vectorization), tracking, and ReID trained DNN model generation processing described in S5 above are completed, the control unit 20 of the CPU 2 moves to the processing shown in FIG. The inference unit 24 is instructed to perform inference processing using the established DNN model. In response to this, the inference unit 24 first reads out the trained DNN models for person detection, feature detection (vectorization), tracking, and ReID generated in S5 above from the DNN model storage unit 13 and stores them on the memory. Load (S21 in FIG. 11). Then, the inference unit 24 uses the test image (for example, the test image 80 in FIG. 11) included in the test data set generated in S3 above to perform the person detection and feature detection (vector detection) generated in S5 above. ), tracking, and ReID to the learned DNN model group, execute inference processing using each learned DNN model for the person in the test image, and save this inference result in the inference result storage unit 14. (S6 in Figure 3).

Next, each trained DNN model in FIG. 11 will be explained. Among these trained DNN models, the person detection model detects people in the input frame image (for example, the test image 80 described above). The feature detection model (vectorization model) performs vectorization processing on the person detection result 84 (a cutout image of the person detected by the person detection model) by the above-mentioned person detection model, and uses the vector as the feature detection result 85. , save (memory). The tracking model is based on the person detection results 84 for people reflected in each frame image taken with the same camera and the vectors (feature detection results 85) for these people's images. The degree of similarity between images of people reflected in frame images is determined. Then, based on this degree of similarity, it is possible to identify whether or not the people in these images are the same person, thereby tracking the people in the images captured by the same camera (the same person can be traced across frames). (by assigning the same ID to the person, the movement of the person in question is tracked).

In addition, the ReID model uses vectors (feature detection results 85) for people captured in each frame image taken by each camera, and tracking results 86 by the tracking model (IDs for people captured in images taken by the same camera). The same ID is assigned to the same person even if the person appears in frame images taken with different cameras. fix). Then, the ReID model outputs, as a ReID result 87, the result of assigning an (re-assigned) ID to a person who appears in each frame image spanning between cameras. The inference unit 24 stores the inference results (person detection result 84, feature detection result 85, tracking result 86, and ReID result 87) by each of the trained DNN models described above in the inference result storage unit 14.

When the inference process using each trained DNN model in S6 is completed, the control unit 20 of the CPU 2 moves to the process in FIG. 12 to verify the bias of the learning data in the learning data set based on the above inference result. The verification unit 25 is instructed to perform the following. In response to this, the verification unit 25 uses the inference results by each trained DNN model stored in the inference result storage unit 14 and the annotation information 73 included in the test dataset stored in the data set storage unit 12. (Compare the inference results of each trained DNN model for the person in the test image with the annotation information 73 of each test data) to verify the bias of the training data in the current training data set (Figure 3 S7). To be more precise, the verification unit 25 calculates the bias of the training data in the current training data set and Estimate the type of data for which the current inference accuracy of each trained DNN model is insufficient. If there are types of data for which the inference accuracy of the current trained DNN models is insufficient, as shown in Figure 12, there may be many false positives for people in their 30s, or people whose orientation is 180° (rear view This results in a high number of false positives for people (people). Here, regarding the types of data for which the inference accuracy of each of the current trained DNN models is insufficient, it is considered that this type of learning data is insufficient. This lack of specific types of learning data can also be considered as a type of bias in learning data. Therefore, the verification process by the verification unit 25 described above can be considered as a process for verifying the bias of the learning data in the current learning data set as a whole.

The control unit 20 of the CPU 2 determines that, as a result of the verification process in S7, if the bias of the learning data is below a predetermined level (each trained DNN model has achieved sufficient accuracy, and the adjustment parameters are changed (adjusted)). is not necessary) (YES in S8), the process of the flowchart in FIG. 3 ends.

On the other hand, as a result of the verification process in S7 above, if the bias in the learning data exceeds a predetermined level (the bias has not been sufficiently eliminated) (NO in S8), the control unit 20 of the CPU 2 , instructs the verification unit 25 to generate adjustment parameter change data. Thereby, as shown in FIG. 13, the verification unit 25 generates data (adjustment parameter change data) for changing the value of the adjustment parameter so as to reduce the bias of the learning data found in the verification of S7 above. is generated and stored in the inference result storage unit 14. For example, as described above, if there are many false detections for people in their 30s, the adjustment parameter change data described above is set so as to increase the learning data for people in their 30s. In addition, if there are many false detections of people whose orientation is 180° (people from behind), set the above adjustment parameter change data to increase the training data (including training images) of people from behind. I do.

After the adjustment parameter change data generation process described above, the control unit 20 of the CPU 2 moves to preparation for the second learning data set generation process. In the second and subsequent training dataset generation processes, as a preparatory process, the learning generation unit 27 of the application environment generation unit 21 verifies the previous inference results using the above adjustment parameter change data. In order to reduce the bias in the learning data found as a result of S9). More specifically, as shown in FIG. 9, the learning generation unit 27 of the applied environment generation unit 21 generates the probability distribution 32a of the store entity and the probability distribution 76a of the human entity generated during the previous application environment generation. is multiplied by the adjustment parameter changed using the adjustment parameter change data described above, thereby changing the probability distribution of stores and people in the application environment.

Thereafter, the control unit 20 of the CPU 2 returns to the process of S4, and causes the data set generation unit 22 to generate a learning dataset corresponding to the changed probability distribution of the stores and people. Control is performed to generate an updated learning data set based on the learning data in this learning data set (S4). That is, the second learning data set generation process is performed.

Specifically, the control unit 20 first uses the learning generation unit 27 of the applied environment generation unit 21 to generate the probability distribution 32b of the store entity and the probability distribution 76b of the person entity after the above change. The process of selecting one or more store entities and one or more person entities, and the process of adding random noise to the selected store and person entities to generate multiple patterns of store and person entities are repeated. Then, the control unit 20 uses the data set generation unit 22 to generate multiple patterns of store entities (corresponding to store images 78a and 79a in FIG. 9) and human entities (corresponding to store images 78a and 79a in FIG. 9), which are generated by adding these random noises. By performing frame image generation processing based on the human images 78b and 79b in FIG. 9) and annotation processing on these frame images, learning corresponding to the probability distribution of the changed store and human entities is performed. Generate a dataset for Furthermore, the control unit 20 adds the learning data in the newly generated learning dataset to the previously generated learning dataset (or adds the learning data in the newly generated learning dataset to the problematic part of the previously generated learning dataset). (replace the training data with the training data in the newly generated training dataset), and then replace the training data set that reflects the training data in the newly generated training dataset with the updated training data. This is a data set for use.

When the above-mentioned generation process of the updated learning data set is completed, the control unit 20 of the CPU 2 causes the learning unit 23 to perform the above-mentioned person detection and feature detection (vectorization) using the updated learning data set. By performing deep learning on the DNN model for each inference of , tracking, and ReID, control is performed to generate a trained DNN model for each inference that is more accurate than the current trained DNN model for each inference. . Then, the control unit 20 uses the inference unit 24 to input the test image included in the test data set generated in S3 above to the trained DNN model for each inference regenerated above. , performs inference processing using each inference trained DNN model for the person in the test image, and stores the inference results in the inference result storage unit 14 (S6). Thereafter, the control unit 20 uses the verification unit 25 to perform the verification process in S7 above. Then, as a result of the verification process in S7, the control unit 20 performs adjustment until the bias of the learning data becomes equal to or less than a predetermined level (YES in S8) (each trained DNN model has achieved sufficient accuracy, (until changing (adjusting) the parameters is no longer necessary), repeat the above steps S9 and S4 to S7.

As described above, according to the machine learning system 10 of the present embodiment, the probability distribution of each component entity in the application environment is changed so as to reduce bias in learning data, and the probability distribution of each component entity after the change is A learning dataset corresponding to the probability distribution of the entity (probability distribution 32b of the store entity after change and probability distribution 76b of the human entity) is generated and updated based on the learning data in this learning dataset. A subsequent training dataset can be generated. In other words, this updated learning data set is a learning data set that reflects the learning data in the learning data set that has been regenerated so as to reduce the bias of the learning data in the learning data set. Therefore, the training data in this updated training dataset has little bias, so by performing deep learning on the DNN models for each of the above inferences using this updated training dataset, the current It is possible to generate a highly accurate trained DNN model for each inference that can more accurately reproduce the probability distribution of the learning target than the trained DNN model for each inference.

Further, according to the machine learning system 10 of the present embodiment, the application environment generation unit 21 (the learning generation unit 27 of the application environment generation unit 21) generates application Change the probability distribution of the entities of each component (stores and people) in the environment, and according to the probability distribution of the entities of each component after this change, the entities of each component (stores and people) that are different each time with the same adjustment parameters is now generated. Thereby, by generating a learning dataset from (images of) the entities of each of these components (stores and people), it is possible to generate a learning dataset including various patterns of learning data.

Further, according to the machine learning system 10 of the present embodiment, each learning image constituting the learning data set is applied with random noise to store and human entities selected from the probability distribution of stores and human entities in the application environment. The image was drawn based on multiple patterns of stores and people generated by adding . As a result, it is possible to obtain various patterns of training images from a set of store entities (images) and human entities (images), so it is possible to obtain a sufficient distribution of learning data in the training dataset. It can be realized.

In addition, according to the machine learning system 10 of this embodiment, the DNN model for inference that is the target of deep learning includes human feature values (vector ), a DNN model for person identification (ReID DNN model) that identifies people photographed with different video cameras is included. As stated in the "Problem to be Solved by the Invention" column, due to bias in the training data in the training dataset, the probability distribution of the learning target cannot be accurately reproduced by the trained DNN model. This problem is particularly noticeable in the trained DNN model for ReID. According to the machine learning system 10 of this embodiment, it is possible to reduce the bias of the learning data in the learning dataset, so this learning dataset can be used to perform deep learning of the DNN model for ReID. As a result, a trained DNN model for ReID with high accuracy can be generated.

Variant:
Note that the present invention is not limited to the configurations of the above-described embodiments, and various modifications can be made without changing the spirit of the invention. Next, a modification of the present invention will be described.

Variation 1:
In the above embodiment, the CPU 2 automatically performs the process of verifying the bias of the learning data in the learning data set and the process of changing the value of the adjustment parameter so as to reduce the bias of the learning data. Although an example has been shown, the user may perform these processes manually. That is, the user displays the inference results of each trained DNN model for the person in the test image and the annotation information of each test data on a display, and visually views the above inference results and annotation information. By comparing the data, the bias of the training data in the current training data set is verified, and based on the verification results, the adjustment parameters can be adjusted using the mouse or the like on the input operation section 5 using the various slide bars mentioned above. It may be changed.

Variation 2:
In the above embodiment, the trained DNN model for each inference used in the first inference process is created using a training dataset corresponding to the probability distribution of the store entity and the probability distribution of the human entity in the current application environment. Although the model is not limited to this, the trained DNN model for each inference used in the first inference process is based on the existing probability distribution of the store entity in the current application environment. A trained DNN model for each inference generated by performing deep learning using a learning data set other than the learning data set corresponding to the probability distribution of human entities may be used.

Variation 3:
In the above embodiment, the test data set is a test data set corresponding to the current application environment entity information (a test data set consisting of a chronological frame image of each camera and annotation information of this frame image). A test dataset containing data was used. However, the test data set used in the machine learning system and the learning data set generation system of the present invention is not limited to this, and may be an existing test data set or a data set in another application environment (for example, another store). It may be a test data set generated based on the situation (statistical information, etc.).

Modification example 4:
In the above embodiment, the actual store is generated as a 3DCG (data) based on the probability distribution of the store. Good too. In this case, random noise means cropping a part of the live-action store background image, enlarging the cropped part at a random enlargement rate and rotating it at a random angle, and further adding distortion due to lens parameters, brightness, and saturation. This corresponds to generating a store image that is randomly changed and finally adjusted to a predetermined frame size. In addition, in order to place a 3DCG person in the actual store background image, information regarding the installation position and angle of the store camera that corresponds to the security camera, as well as information regarding obstructions, is required, so please prepare this in advance. It is necessary to store it in the applied environment generation material storage section. In addition, by using a depth camera (a camera that can give depth information in pixel units to the captured RGB image) as a security camera, the installation position and installation angle of the above store camera can be automatically calculated. good.

1 Learning server (computer)
10 Machine learning system (machine learning system, learning dataset generation system)
15 Machine learning program 20 Control unit (post-update learning data set generation control unit, learning repetition control unit)
21 Application environment generation unit (specific environment generation unit, specific environment change unit)
22 Data set generation unit 23 Learning unit (neural network generation unit)
24 Inference section 25 Verification section (verification section, specific environment change section)
78, 79, 81 Learning image 82 Annotation information (annotation data)

Claims (12)

A machine learning system that performs machine learning of a neural network model for inference with respect to objects in images using a training dataset including training images created using computer graphics,
a dataset generation unit that generates a learning dataset corresponding to the probability distribution of the entity of each component in a certain specific environment;
a neural network generation unit that generates a trained neural network model by performing deep learning using the training data set;
an inference unit that inputs a test image included in a test data set into the trained neural network model and performs inference processing on an object in the test image;
a verification unit that verifies bias in the training data in the training data set of the trained neural network model based on the inference result by the inference unit;
a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the learning data found by the verification unit;
A learning data set corresponding to the probability distribution of the substance of each component after the change is generated by the data set generation unit, and the updated learning data is generated based on the learning data in this learning data set. a post-update learning dataset generation control unit that controls to generate a set;
Learning controlled so that the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. It is equipped with a repetition control section ,
Each training image constituting the training dataset generated by the dataset generation unit is created by adding random noise to the entity of each component selected from the probability distribution of the entity of each component in the specific environment. A machine learning system that is an image drawn based on the substance of each of the constituent elements of a plurality of generated patterns . A machine learning system that performs machine learning of a neural network model for inference with respect to objects in images using a training dataset including training images created using computer graphics,
a dataset generation unit that generates a learning dataset corresponding to the probability distribution of the entity of each component in a certain specific environment;
a neural network generation unit that generates a trained neural network model by performing deep learning using the training data set;
an inference unit that inputs a test image included in a test data set into the trained neural network model and performs inference processing on an object in the test image;
a verification unit that verifies bias in the training data in the training data set of the trained neural network model based on the inference result by the inference unit;
a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the learning data found by the verification unit;
A learning data set corresponding to the probability distribution of the substance of each component after the change is generated by the data set generation unit, and the updated learning data is generated based on the learning data in this learning data set. a post-update learning dataset generation control unit that controls to generate a set;
Learning controlled so that the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. a repetition control section;
Identification of generating specific environment entity information, which is data of simulation results along a time series of entities in the specific environment corresponding to the entities in the specific environment, based on probability distributions of entities in each component in the specific environment. Equipped with an environment generation section,
In addition to the learning data set, the data set generation unit generates a test data set for the neural network model that corresponds to the specific environment entity information generated by the specific environment generation unit,
The inference unit inputs a test image included in the test data set generated by the data set generation unit to the trained neural network model, and performs inference processing on the object in the test image. Machine learning system. The learning repetition control unit is configured to perform the following steps until the bias of the learning data in the current learning data set becomes equal to or less than a predetermined level as a result of the verification by the verification unit.
A process of changing the probability distribution of each component in the specific environment by the specific environment changing unit so as to reduce bias in the learning data;
The updated learning dataset generation control section generates the learning dataset corresponding to the probability distribution of the entity of each component after the change using the dataset generation section, and generates the learning dataset using the dataset generation section. A process of generating an updated training dataset based on the training data in the
A process of generating, by the neural network generation unit, a trained neural network model with higher accuracy than the current trained neural network model using the updated learning data set;
A process of inferring an object in the test image by the inference unit using the generated highly accurate trained neural network model;
Machine learning according to claim 1 or 2, characterized in that the verification unit repeats the process of verifying the bias of the learning data in the current learning data set based on the inference result by the inference unit. system. The specific environment changing unit changes the probability distribution of the entity of each component in the specific environment according to the value of the adjustment parameter, and according to the changed probability distribution of the entity of each component, for the same adjustment parameter, The machine learning system according to any one of claims 1 to 3, wherein a different entity of each component is generated each time. The machine learning system according to any one of claims 1 to 4 , wherein the dataset generation unit generates annotation data of the learning dataset. The neural network model for inference includes a neural network for person identification that identifies people photographed with different video cameras based on human features in each frame image constituting each video photographed with multiple video cameras. The machine learning system according to any one of claims 1 to 5 , characterized in that a network model is included. A learning dataset generation system for generating a learning dataset for performing machine learning of a neural network model for inference with respect to objects in images, the system comprising:
The training data set includes training images created using computer graphics,
an inference unit that inputs a test image included in a test data set into a currently trained neural network model and performs inference processing on an object in the test image;
a verification unit that verifies the bias of the training data in the training data set of the current trained neural network model based on the inference result by the inference unit;
a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment reflected in the learning image so as to reduce the bias of the learning data found by the verification unit;
Control is performed to generate a learning data set corresponding to the probability distribution of the substance of each component after the change, and to generate an updated learning data set based on the learning data in this learning data set. and a post-update training data set generation control unit ,
Each training image constituting the updated training data set is a plurality of patterns generated by adding random noise to the entity of each component selected from the probability distribution of the entity of each component in the specific environment. A learning dataset generation system that is an image drawn based on the substance of each component . A learning dataset generation system for generating a learning dataset for performing machine learning of a neural network model for inference with respect to objects in images, the system comprising:
The training data set includes training images created using computer graphics,
an inference unit that inputs a test image included in a test data set into a currently trained neural network model and performs inference processing on an object in the test image;
a verification unit that verifies the bias of the training data in the training data set of the current trained neural network model based on the inference result by the inference unit;
a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment reflected in the learning image so as to reduce the bias of the learning data found by the verification unit;
Control is performed to generate a learning data set corresponding to the probability distribution of the substance of each component after the change, and to generate an updated learning data set based on the learning data in this learning data set. a post-update training data set generation control unit,
Identification of generating specific environment entity information, which is data of simulation results along a time series of entities in the specific environment corresponding to the entities in the specific environment, based on probability distributions of entities in each component in the specific environment. environment generation section;
a data set generation unit that generates a test data set for the neural network model corresponding to the specific environment entity information generated by the specific environment generation unit;
The inference unit inputs the test image included in the test data set generated by the data set generation unit into the currently trained neural network model, and performs inference processing on the object in the test image. A learning dataset generation system. The specific environment changing unit changes the probability distribution of the entity of each component in the specific environment according to the value of the adjustment parameter, and according to the changed probability distribution of the entity of each component, for the same adjustment parameter, 9. The learning data set generation system according to claim 7 or 8, wherein a different entity of each component is generated each time. A machine learning program for performing machine learning of a neural network model for inference with respect to objects in images using a learning dataset including training images created with computer graphics,
computer,
a dataset generation unit that generates a learning dataset corresponding to the probability distribution of the entity of each component in a certain specific environment;
a neural network generation unit that generates a trained neural network model by performing deep learning using the training data set;
an inference unit that inputs a test image included in a test data set into the trained neural network model and performs inference processing on an object in the test image;
a verification unit that verifies bias in the training data in the training data set of the trained neural network model based on the inference result by the inference unit;
a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the learning data found by the verification unit;
A learning data set corresponding to the probability distribution of the substance of each component after the change is generated by the data set generation unit, and the updated learning data is generated based on the learning data in this learning data set. a post-update learning dataset generation control unit that controls to generate a set;
Learning controlled so that the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. Function as a repeat control section ,
Each training image constituting the training dataset generated by the dataset generation unit is created by adding random noise to the entity of each component selected from the probability distribution of the entity of each component in the specific environment. A machine learning program that is an image drawn based on the entities of each of the constituent elements of the generated plurality of patterns . A machine learning program for performing machine learning of a neural network model for inference with respect to objects in images using a learning dataset including training images created with computer graphics,
computer,
a dataset generation unit that generates a learning dataset corresponding to the probability distribution of the entity of each component in a certain specific environment;
a neural network generation unit that generates a trained neural network model by performing deep learning using the training data set;
an inference unit that inputs a test image included in a test data set into the trained neural network model and performs inference processing on an object in the test image;
a verification unit that verifies bias in the training data in the training data set of the trained neural network model based on the inference result by the inference unit;
a specific environment changing unit that changes the probability distribution of the entity of each component in the specific environment so as to reduce the bias of the learning data found by the verification unit;
A learning data set corresponding to the probability distribution of the substance of each component after the change is generated by the data set generation unit, and the updated learning data is generated based on the learning data in this learning data set. a post-update learning dataset generation control unit that controls to generate a set;
Learning controlled so that the neural network generation unit generates a trained neural network model with higher accuracy than the current trained neural network model by performing deep learning using the updated training data set. a repetition control section;
Identification of generating specific environment entity information, which is data of simulation results along a time series of entities in the specific environment corresponding to the entities in the specific environment, based on probability distributions of entities in each component in the specific environment. Function as an environment generation section,
In addition to the learning data set, the data set generation unit generates a test data set for the neural network model that corresponds to the specific environment entity information generated by the specific environment generation unit,
The inference unit inputs a test image included in the test data set generated by the data set generation unit to the trained neural network model, and performs inference processing on the object in the test image. Machine learning program. The specific environment changing unit changes the probability distribution of the entity of each component in the specific environment according to the value of the adjustment parameter, and according to the changed probability distribution of the entity of each component, for the same adjustment parameter, The machine learning program according to claim 10 or 11, wherein a different entity of each component is generated each time.