JP7609726B2 - Computer system, method for controlling input of data to a model, and control device - Google Patents

Description

The present invention relates to controlling the input of data into a model that determines behavior.

Machine learning is being used to develop algorithms that realize automatic control of controlled objects such as vehicles, robots, and drones. For example, end-to-end deep reinforcement learning is being used to develop algorithms that output vehicle control signals from images.

End-to-end deep reinforcement learning has problems in that it requires a large amount of training data, takes a huge amount of time to train, and is often difficult to train. Therefore, the development of algorithms that use object-based reinforcement learning has attracted attention.

In object-based reinforcement learning, objects are extracted from images, and a strategy for determining the behavior of the target is learned based on the information about the extracted objects.

Generally, a policy is given as a model such as a neural network. When the model is a neural network, the neural network has multiple input slots. Usually, when the input slot for inputting object information changes, the output value also changes. In other words, this affects the accuracy of the policy. Therefore, it is important to control the input of object information to the input slots.

As a method for controlling the input of object information to an input slot, for example, the technology described in Non-Patent Document 1 is known.

Non-Patent Document 1 describes a method of determining an object's score based on the object's appearance information, selecting an object based on the score, and inputting the selected object information into a neural network.

D. Wang, C. Devin, Q. -Z. Cai, F. Yu and T. Darrell, "Deep Object-Centric Policies for Autonomous Driving," 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8853 -8859, doi: 10.1109/ICRA.2019.8794224.

Because there is a high degree of diversity in how objects appear, it is necessary for a person to set the object appearance and the method of calculating the score, or to separately learn the method of calculating the priority. Therefore, there is an issue that the processing cost is high. In addition, the method described in Non-Patent Document 1 has an issue that the effect of improving the accuracy of the policy is small.

The present invention provides a computer system and method that realizes control of the input of object information into a model in order to obtain a highly accurate output.

A representative example of the invention disclosed in the present application is as follows. That is, a computer system includes at least one computer having an arithmetic unit, a storage device connected to the arithmetic unit, and an interface connected to the arithmetic unit, and holds model information that manages a model that determines the behavior of the control target using object data that represents the characteristics of objects included in an image surrounding the control target, and input control information that manages input rules for the object data to the model based on labels assigned based on the movement of the object, receives time-series data of the images, detects the object from each of the multiple images, selects a target image from the multiple images, calculates motion data representing the motion of the object detected from the target image using two or more images including the target image, stores the target image and the motion data in association with each other, assigns the label to the object detected from the image based on the motion data for each of the multiple images, and controls input of the object data of the object detected from the image to the model based on the input control information and the label for each of the multiple images.

According to the present invention, it is possible to realize input control of object data (object information) that improves the accuracy of model output. Problems, configurations, and advantages other than those described above will become clear from the explanation of the following embodiment.

FIG. 2 is a diagram illustrating an example of a hardware configuration of a computer included in the computer system of the first embodiment. FIG. 2 illustrates an example of a functional configuration of a computer according to a first embodiment. FIG. 2 illustrates an example of a functional configuration of a computer according to a first embodiment. 1 is a diagram illustrating an example of an image handled by the computer system of the first embodiment. 1 is a diagram illustrating an example of an image handled by the computer system of the first embodiment. 1 is a diagram illustrating an example of an image handled by the computer system of the first embodiment. 1 is a flowchart illustrating an example of a process executed by a computer according to a first embodiment. 11 is a diagram illustrating an example of a data structure of bounding box information generated by a computer according to a first embodiment. FIG. FIG. 13 is a diagram illustrating an example of a data structure of motion information generated by a computer according to the first embodiment. FIG. 13 is a diagram illustrating an example of a data structure of object label information generated by a computer according to the first embodiment. 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. 4A to 4C are diagrams illustrating an example of input control of object data by an input control unit according to the first embodiment. 11 is a flowchart illustrating an example of a bounding box correction process executed by a bounding box prediction unit according to the first embodiment. 10 is a diagram illustrating an example of a method for calculating a predicted bounding box by a bounding box prediction unit according to the first embodiment. FIG. FIG. 11 is a diagram illustrating an example of a functional configuration of a computer according to a second embodiment. FIG. 11 is a diagram illustrating an example of a functional configuration of a computer according to a second embodiment. FIG. 11 is a diagram illustrating an example of a data structure of cluster information according to the second embodiment. FIG. 11 is a diagram illustrating an example of a data structure of history information according to the second embodiment. 13 is a diagram illustrating an example of input control of object data by an input control unit according to a second embodiment. FIG. 13 is a flowchart illustrating a reclustering process executed by a computer according to a second embodiment.

The following describes an embodiment of the present invention with reference to the drawings. However, the present invention should not be interpreted as being limited to the description of the embodiment shown below. It will be easily understood by those skilled in the art that the specific configuration can be changed without departing from the concept or spirit of the present invention.

In the configuration of the invention described below, the same or similar configurations or functions are given the same reference symbols, and duplicate explanations are omitted.

The terms "first," "second," "third," and the like used in this specification are used to identify components and do not necessarily limit the number or order.

The position, size, shape, range, etc. of each component shown in the drawings, etc. may not represent the actual position, size, shape, range, etc., in order to facilitate understanding of the invention. Therefore, the present invention is not limited to the position, size, shape, range, etc. disclosed in the drawings, etc.

The computer system is composed of at least one computer 100. Figure 1 is a diagram illustrating an example of the hardware configuration of the computer 100 included in the computer system of the first embodiment.

The computer 100 has a processor 101, a memory 102, an auxiliary storage device 103, a network interface 104, and an input/output interface 105. Each hardware element is connected to each other via a bus.

The processor 101 is an arithmetic device that controls the entire computer 100, and executes programs stored in the memory 102. The processor 101 executes processing according to the programs, thereby operating as a functional unit (module) that realizes a specific function. In the following explanation, when processing is explained using a functional unit as the subject, this indicates that the processor 101 is executing a program that realizes the functional unit.

Memory 102 is a storage device that stores the programs executed by processor 101 and the information executed by the programs. Memory 102 is also used as a work area.

The auxiliary storage device 103 is a storage device that permanently stores data, such as a hard disk drive (HDD) and a solid state drive (SSD). The programs and information stored in the memory 102 may be stored in the auxiliary storage device 103. In this case, the processor 101 reads the programs and information from the auxiliary storage device 103 and loads them into the memory 102.

The network interface 104 is an interface for connecting to external devices via a network.

The input/output interface 105 is an interface for connecting to an input device 111 and an output device 112. Here, the input device 111 is a keyboard, a mouse, a touch panel, etc., and the output device 112 is a display, etc.

Note that the computer 100 does not have to have an input/output interface 105.

The computer system includes a computer 100 that learns a strategy for determining an action, and a computer 100 that determines an action using the learned strategy. Note that one computer 100 may perform both learning and action determination.

Figures 2A and 2B are diagrams showing an example of the functional configuration of the computer 100 of Example 1.

Figure 2A shows the functional configuration of computer 100-1 that learns a policy. Computer 100-1 has an object detection unit 200, a bounding box prediction unit 201, an object classification unit 202, a label assignment unit 203, an input control unit 204, and a learning unit 210. Computer 100-1 also holds classification definition information 230, occlusion detection definition information 231, label definition information 232, object data definition information 233, input control information 234, model information 235, and learning information 236. Self information 237 and reward information 238 are also input to computer 100-1 from the outside. Self information 237 and reward information 238 are input together with the input of image 250.

The classification definition information 230 is information for classifying objects. For example, the classification definition information 230 is definition information of a classifier. The occlusion detection definition information 231 is information on a method for detecting a state in which an object is occluded by another object. The label definition information 232 is information on a method for assigning a label based on the movement of an object. For example, the label definition information 232 is information for storing an entry that associates an object type with a label assignment rule based on the movement of the object to be evaluated. The object data definition information 233 is information on the data structure of object data (feature amount) to be input when determining a policy. The input control information 234 is information on the input rule of object data for a model that realizes the policy. The model information 235 is definition information of the model. The learning information 236 is information for controlling the learning process of the policy. For example, the learning information 236 includes a learning rate, etc. The self information 237 is information on the internal state of the control target and the surrounding environment. The reward information 238 is information on the reward for the action.

The object detection unit 200 detects an object from an image 250 of the surroundings of the control target. The object detection unit 200 generates a rectangular area (bounding box) that contains the object. The object detection unit 200 outputs the image 250 and the object's bounding box to the bounding box prediction unit 201 and the object classification unit 202, and outputs the image 250 to the label assignment unit 203.

The boundary box prediction unit 201 predicts a boundary box of an object using multiple images 250, compares the boundary box generated by the object detection unit 200 with the predicted boundary box, and corrects the boundary box based on the comparison result. The boundary box prediction unit 201 outputs the corrected boundary box to the label assignment unit 203.

The object classification unit 202 classifies the type of object contained in the image 250. In this embodiment, the object classification unit 202 classifies the type of object for each bounding box. The object classification unit 202 outputs the classification result to the label assignment unit 203.

The labeling unit 203 uses multiple images 250 to analyze the movement of objects included in the images 250, and assigns labels to the objects included in the images 250 based on the movement of the objects. The labeling unit 203 outputs the object labels and the images 250 to the input control unit 204.

The input control unit 204 generates object data 800 (see FIG. 8) for the object included in the image 250, and controls the input of the object data 800 to the model based on the label of the object.

The learning unit 210 learns a policy using the object data 800 input by the input control unit 204. The learning unit 210 includes a behavior decision unit 220 that decides an action based on the policy. The behavior decision unit 220 decides an action based on the policy defined by the model information 235, the self information 237, and the object data 800, and outputs it as behavior data 260. The behavior data 260 is, for example, a control signal for controlling the torque and steering of the vehicle.

The learning unit 210 learns a policy based on the learning information 236, the reward information 238, and the behavioral data 260, and reflects the learning results in the model information 235.

In Example 1, reinforcement learning is used as an example, but learning methods other than reinforcement learning may also be used.

Figure 2B shows the functional configuration of computer 100-2, which determines an action using a strategy. Computer 100-2 has an object detection unit 200, a bounding box prediction unit 201, an object classification unit 202, a label assignment unit 203, an input control unit 204, and an action determination unit 220. Computer 100-2 also holds classification definition information 230, occlusion detection definition information 231, label definition information 232, object data definition information 233, input control information 234, and model information 235, and also receives input of self information 237.

The functions and information possessed by computer 100-2 are the same as those possessed by computer 100-1, and therefore will not be described.

Note that with regard to each functional unit of the computers 100-1 and 100-2, multiple functional units may be combined into one functional unit, or one functional unit may be divided into multiple functional units for each function. For example, the object detection unit 200 may include a bounding box prediction unit 201.

The computer system may learn a model and set the learned model to a control object such as a vehicle, a robot, or a drone. In this case, the control object includes a camera that acquires images, a drive unit, and a control device such as a microcomputer having the functions shown in FIG. 2B. The control object controls the drive unit based on the behavior data 260 output from the control device.

Next, an example of an image 250 handled by the computer system of the first embodiment will be described.

Figures 3A, 3B, and 3C are diagrams illustrating an example of an image 250 handled by the computer system of the first embodiment.

As shown in FIG. 3A, the computer system of the first embodiment handles images captured by a camera attached to a vehicle 300. The camera captures images of a range 310. Range 310 includes objects (vehicles) 320-1, 320-2, 320-3, and 320-4. For the sake of explanation, a three-dimensional coordinate system is shown.

The camera outputs an image 250, for example, as shown in Figures 3B and 3C. Figure 3B shows image 250 in a coordinate system whose origin is the upper left corner of the image. Figure 3C shows image 250 in a coordinate system whose origin is the top center of the image.

The object detection unit 200 generates a bounding box 330 that contains the detected object. From the image 250 in Figures 3B and 3C, bounding boxes 330-1, 330-2, 330-3, and 330-4 that contain the objects 320-1, 320-2, 320-3, and 320-4 are generated.

In the image 250 of Figures 3B and 3C, only vehicles are present as objects 320, but objects of different object types than vehicles, such as people, road signs, and traffic lights, may also be included.

FIG. 4 is a flowchart illustrating an example of processing executed by the computer 100-1 of the first embodiment. FIG. 5 is a diagram illustrating an example of the data structure of bounding box information generated by the computer 100-1 of the first embodiment. FIG. 6 is a diagram illustrating an example of the data structure of motion information generated by the computer 100-1 of the first embodiment. FIG. 7 is a diagram illustrating an example of the data structure of object label information generated by the computer 100-1 of the first embodiment.

The computer 100-1 acquires a number of images 250 (time-series data of the images 250) arranged in chronological order (step S101). The acquired images 250 are stored in a work area.

The object detection unit 200 of the computer 100-1 performs object detection processing on each image 250 (step S102).

In the object detection process, the object detection unit 200 detects one or more objects from an image 250 and generates a bounding box 330 for the detected objects. Note that the method of detecting objects from an image is a known technique, so a detailed description is omitted.

The object detection unit 200 generates bounding box information 500 as a result of the object detection process for one image 250. The bounding box information 500 stores entries including an ID 501, a position 502, and a size 503. One entry corresponds to the bounding box 330 of one object. Note that the fields included in the entry are not limited to those described above. It is possible for the entry to not include any of the above-mentioned fields, and it may also include other fields.

ID 501 is a field that stores identification information of the bounding box 330 (object). Position 502 is a field that stores the coordinate values of the center of the bounding box 330. Note that instead of the coordinate values of the center, the coordinate values of the upper left corner of the bounding box 330 may be stored. Size 503 is a field that stores the width and height of the bounding box 330.

The object detection unit 200 also refers to the bounding box information 500 of each image 250 and tracks the bounding boxes 330 to associate the bounding boxes 330 between the images 250. Based on this association, the object detection unit 200 assigns the same ID to the bounding boxes 330 of the same object.

The bounding box prediction unit 201 of the computer 100-1 executes a bounding box correction process for each image 250 (step S103). The bounding box correction process will be explained using Figures 10 and 11.

The object classification unit 202 of the computer 100-1 performs object classification processing on each image 250 (step S104).

Classification of objects contained in an image can be done using known techniques, so a detailed description will be omitted. For example, a classification method using a CNN (Convolutional Neural Network) can be considered.

The labeling unit 203 of the computer 100-1 starts loop processing of the images 250 (step S105). Specifically, the labeling unit 203 selects one image 250 to be a target (target image 250). Here, it is assumed that the images 250 are selected in the direction from the most recent image 250 to the past. Note that images 250 may also be selected in the direction of time lapse.

The labeling unit 203 of the computer 100-1 starts a loop process of object types (step S106). Specifically, the labeling unit 203 selects one object type to be the target (target object type).

The labeling unit 203 of the computer 100-1 uses the target image 250 to calculate the motion data of the object of the target object type (step S107). Specifically, the following process is executed.

(S107-1) The label assignment unit 203 reads out the image 250 that is chronologically preceding the target image 250.

(S107-2) The labeling unit 203 reads out data on the bounding box 330 of the object of the target object type from the bounding box information 500 of each of the target image 250 and the read image 250.

(S107-3) The labeling unit 203 uses the data of the bounding box 330 to calculate an index representing the movement of the bounding box 330 of the object of the target object type. For example, the labeling unit 203 calculates the amount of change in the center of the bounding box 330 and the size (height and width) of the bounding box 330.

At this time, the labeling unit 203 may calculate features such as a collision grace period to be included in the object data 800. The collision grace period is calculated using any one of formulas (1), (2), and (3).

Here, w(t) represents the width of the bounding box 330 in the target image 250. Delta w represents the change in the width of the bounding box 330. h(t) represents the height of the bounding box 330 in the target image 250. Delta t represents the change in the height of the bounding box 330. s(t) represents the area of the bounding box 330 in the target image 250. The area is given as the product of the width and height. Delta s represents the change in the area of the bounding box 330.

(S107-4) The labeling unit 203 generates motion data composed of indicators and registers it in the motion information 600.

As shown in FIG. 6, the motion information 600 stores entries including an ID 601, a position change amount 602, a size change amount 603, and a collision grace period 604. One entry corresponds to the motion data of one object. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.

ID601 is the same field as ID501. Position change amount 602 is a field that stores the amount of change in the center of bounding box 330. Size change amount 603 is a field that stores the amount of change in size of bounding box 330. Collision grace period 604 is a field that stores the collision grace period. As shown in FIG. 6, the motion data can be treated as a vector whose components are indices.

The motion information 600 is generated for each object type. If the target image 250 is the oldest, there is no image 250 before that image 250, so the labeling unit 203 sets the index to 0. Motion data may be calculated using three or more images 250 that are consecutive in time series.

This concludes the explanation of the processing in step S107.

The label assignment unit 203 of the computer 100-1 assigns a label to the object of the target object type based on the movement information 600 and the label definition information 232, and generates object label information 700 (step S108).

In the label definition information 232, for example, the following rules for adding labels are defined.
(Assignment rule 1) Labels are assigned from "L1" to "Ln" in descending order of the amount of change in the y-axis from the center.
(Assignment rule 2) The motion data is sorted in order of the amount of change on the x-axis from the center, and labels from "L1" to "Ln" are assigned in order of the largest difference in the amount of change on the x-axis of the motion data at both ends.
n represents the number of objects.

Assignment rule 1 is a method of assigning a label using the physical quantity contained in the motion data itself as an index. Assignment rule 2 is a method of assigning a label based on an index that indicates the difference from the motion of surrounding objects, calculated from the physical quantity contained in the motion data.

When the target image 250 is the oldest image, the label assignment unit 203 simply reuses the label assignment results for the image 250 that is chronologically ahead.

As shown in FIG. 7, object label information 700 stores entries including an ID 701 and a label 702. One entry corresponds to one object. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.

ID701 is the same field as ID501. Label 702 is a field that stores the determined label. Figure 7 shows the result of assigning labels from "L1" to "Ln" in descending order of the amount of change on the y-axis of the center. n represents the number of objects.

The label assignment unit 203 of computer 100-1 determines whether processing has been completed for all object types (step S109).

If it is determined that processing has not been completed for all object types, the label assignment unit 203 of computer 100-1 returns to step S106 and executes the same processing.

If it is determined that processing has been completed for all object types, the label assignment unit 203 of the computer 100-1 determines whether processing has been completed for all images 250 (step S110).

If it is determined that processing has not been completed for all images 250, the label assignment unit 203 of the computer 100-1 returns to step S105 and executes the same processing.

If it is determined that processing has been completed for all images 250, the input control unit 204 of the computer 100-1 generates object data 800 for the objects contained in each image 250 based on the object data definition information 233 (step S111).

The input control unit 204 of the computer 100-1 controls the input of object data 800 to the action decision unit 220 for each image 250 based on the object label information 700 and the input control information 234 (step S112).

Here, a method for controlling the input of object data to the action decision unit 220 will be described. Figures 8A, 8B, 9A, and 9B are diagrams illustrating an example of input control of object data 800 by the input control unit 204 in Example 1.

When the behavior decision unit 220 is a neural network composed of an input layer 810, a hidden layer 811, and an output layer 812 as shown in Figures 8A and 8B, the input control information 234 stores information indicating the correspondence between labels and input slots. In this case, the input control unit 204 inputs object data 800 to an input slot corresponding to the label of the object.

Note that FIG. 8A shows input control of object data 800 when there is one object type, and FIG. 8B shows input control of object data 800 when there are two object types.

When the behavior decision unit 220 is composed of a recurrent neural network (RNN) 820 and a neural network (NN) 830 as shown in Figures 9A and 9B, the input control information 234 stores information associating labels with the input order of the object data 800 to the RNN 820. In this case, the input control unit 204 inputs the object data 800 to the RNN 820 in the order corresponding to the object labels.

Note that FIG. 9A shows input control of object data 800 when there is one object type, and FIG. 9B shows input control of object data 800 when there are two object types.

The learning unit 210 of the computer 100-1 executes the learning process using the object data 800 whose input has been controlled (step S113). After that, the computer 100-1 ends the series of processes.

Since the learning process is a known technique, a detailed description will be omitted. Note that the learning unit 210 may execute the learning process at a predetermined cycle rather than every time an image 250 is input. If the cycle has not elapsed, the processes of steps S112 and S113 can be omitted. If the learning process is not executed, the object data 800 is accumulated.

The process executed by computer 100-2 is the same as that executed by computer 100-1 in step S113, but with the output process of behavioral data 260.

The labeling unit 203 assigns a label based on the movement of the object (bounding box 330). Since the movement of an object has less diversity than the appearance of the object, it is possible to clearly assign meaning to the input. For example, input rules for the object data 800 can be given, such as large horizontal movement, close to the host vehicle, and fast moving speed. By controlling the input of the object data 800 based on the input rules, it is expected that the learning efficiency and the output accuracy can be improved.

FIG. 10 is a flowchart illustrating an example of a boundary box correction process executed by the boundary box prediction unit 201 of the first embodiment. FIG. 11 is a diagram illustrating an example of a method for calculating a predicted boundary box by the boundary box prediction unit 201 of the first embodiment.

The boundary box prediction unit 201 selects a target image 250 and performs the boundary box correction process described below on the target image 250. Note that it is assumed that the image 250 is selected in the past direction from the latest image 250.

The bounding box prediction unit 201 starts loop processing of objects included in the target image 250 (step S201). Specifically, the bounding box prediction unit 201 selects one bounding box 330 from the bounding box information 500 of the target image 250.

The bounding box prediction unit 201 calculates a predicted bounding box 330 of the selected object using the target image 250 and the image 250 that is chronologically preceding the target image 250 (step S202).

For example, the boundary box prediction unit 201 calculates the predicted boundary box 330 of the object 320-2 in the selected image 250 based on the boundary box 330 of the object 320-2 in the image 250 (t1) that is chronologically immediately preceding the target image 250 (t2), and the size and movement speed of the object 320-1 that overlaps the object 320-2.

The method for calculating the predicted bounding box 330 can be done using known technology, so a detailed explanation is omitted.

The bounding box prediction unit 201 calculates the error between the bounding box 330 of the selected object and the predicted bounding box 330 (step S203).

For example, the bounding box prediction unit 201 calculates the error in the x-axis of the center or the error in the height of the bounding box 330.

The bounding box prediction unit 201 determines whether the selected object is occluded by another object based on the calculated error and the occlusion detection definition information 231 (step S204).

The occlusion detection definition information 231 stores an error threshold, and the bounding box prediction unit 201 determines whether the selected object is occluded by another object based on the result of comparing the error and the threshold. For example, if the error is greater than the threshold, the bounding box prediction unit 201 determines that the selected object is occluded by another object.

If it is determined that the selected object is not occluded by another object, the bounding box prediction unit 201 proceeds to step S206.

If it is determined that the selected object is occluded by another object, the bounding box prediction unit 201 replaces the bounding box 330 of the selected object with the predicted bounding box 330 (step S205), and then proceeds to step S206.

Specifically, the bounding box prediction unit 201 sets the values of the predicted bounding box 330 to the position 502 and size 503 of the entry selected from the bounding box information 500.

By using a bounding box 330 that has been corrected to take into account the occlusion state, the object's movement can be accurately evaluated.

In step S206, it is determined whether processing has been completed for all objects contained in the target image 250 (step S206).

If it is determined that processing has not been completed for all objects contained in the target image 250, the bounding box prediction unit 201 returns to step S201 and executes the same processing.

When it is determined that processing has been completed for all objects contained in the target image 250, the boundary box prediction unit 201 ends the boundary box correction process.

The difference in the second embodiment is that labels are assigned using clusters constructed from past object movement data. Below, the second embodiment will be explained, focusing on the differences from the first embodiment.

The hardware configuration of the computer 100 constituting the computer system of the second embodiment is the same as that of the first embodiment.

Figures 12A and 12B are diagrams showing an example of the functional configuration of a computer 100 in Example 2.

FIG. 12A shows the functional configuration of a computer 100-1 that learns a policy. The computer 100-1 of the second embodiment differs from the computer of the first embodiment in that it holds cluster information 239 and history information 240.

FIG. 12B shows the functional configuration of a computer 100-2 that uses a policy to determine an action. The computer 100-1 of the second embodiment differs from the computer of the first embodiment in that it stores cluster information 239 and history information 240.

Figure 13 is a diagram showing an example of the data structure of cluster information 239 in Example 2.

Entries including a cluster ID 1301, a center of gravity 1302, and a range 1303 are stored. One entry corresponds to one cluster. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the fields described above, or may include other fields.

Cluster ID 1301 is a field that stores identification information of a cluster. Center of gravity 1302 is a field that stores the center of gravity of a cluster. Range 1303 is a field that stores the size of the cluster.

In the second embodiment, a cluster in a feature space is assumed, with the axes representing the amount of change in each of the x-coordinate, y-coordinate, width, and height contained in the motion data. Note that the feature space is not limited to this.

In addition, a cluster may be configured for each object type. In this case, cluster information 239 exists for each object type.

Figure 14 is a diagram showing an example of the data structure of history information 240 in Example 2.

The history information 240 stores entries including a position change amount 1401 and a size change amount 1402. One entry corresponds to motion data. Note that the fields included in an entry are not limited to those described above. An entry may not include any of the above-mentioned fields, or may include other fields.

Position change amount 1401 and size change amount 1402 are the same fields as position change amount 602 and size change amount 603.

When configuring clusters for each object type, the entry includes a field that stores the object type.

In the label definition information 232 of the second embodiment, for example, the following assignment rules are defined.
(Assignment rule 3) Assign a label "Ln" to cluster n.
n represents the number of clusters.

The process flow executed by the computer 100-1 in the second embodiment is the same as that in the first embodiment, but some processing is different.

In step S108, the label assignment unit 203 assigns a label to the object of the target object type based on the motion information 600, the label definition information 232, and the cluster information 239, and generates object label information 700.

Specifically, the label assignment unit 203 determines the cluster to which the object belongs based on the object's motion data and the cluster information 239. The label assignment unit 203 stores the motion data in the history information 240. The label assignment unit 203 assigns a label to the object of the target object type based on the cluster and label definition information 232.

The control method for inputting the object data 800 in the second embodiment can be the same as that in the first embodiment. In addition, a control method such as that shown in FIG. 15 can also be used.

Figure 15 is a diagram illustrating an example of input control of object data 800 by the input control unit 204 in Example 2.

The behavior decision unit 220 is composed of an RNN 820 and an NN 830. In this case, the label definition information 232 defines the label assignment rules for each cluster. Furthermore, the input control information 234 stores information that associates the label of each cluster with the input order of the object data 800 to the RNN 820. In this case, the input control unit 204 inputs the object data 800 of each cluster to the RNN 820 in the order corresponding to the object label.

Figure 16 is a flowchart explaining the re-clustering process executed by computer 100-1 in Example 2.

The computer 100-1 executes the reclustering process when a predetermined number of pieces of motion data have been accumulated, when a predetermined time has elapsed, or when an execution request has been received. Here, it is assumed that the label assignment unit 203 executes the process. However, it may also be executed by another functional unit.

The labeling unit 203 performs clustering using the history information 240 (step S301). A known method may be used for the clustering, so a detailed description is omitted.

The label assignment unit 203 refers to the cluster information 239 and determines whether the previous cluster can be associated with the new cluster (step S302).

For example, if the error in the center of gravity between the previous cluster and the new cluster is smaller than a threshold value for all clusters, and the error in the range is smaller than a threshold value, the labeling unit 203 determines that the previous cluster can be associated with the new cluster.

If it is determined that the previous cluster can be associated with the new cluster, the label assignment unit 203 updates the cluster information 239 (step S303) and terminates the reclustering process.

Specifically, the label assignment unit 203 updates the center of gravity 1302 and range 1303 of each cluster. Since the clusters are substantially unchanged before and after reclustering, the label definition information 232 and input control information 234 do not need to be changed.

If it is determined that the previous cluster cannot be associated with the new cluster, the label assignment unit 203 updates the cluster information 239 (step S304).

Specifically, the label assignment unit 203 initializes the cluster information 239 and sets an entry for a new cluster in the cluster information 239 based on the clustering result.

The label assignment unit 203 requests the user to reset the label definition information 232 and the input control information 234 (step S305).

In the second embodiment, since a label is set according to the cluster to which the label belongs, if the cluster changes, the label definition information 232 and the input control information 234 need to be reset. Therefore, if the cluster changes, the label assignment unit 203 prompts the user to reset the label definition information 232 and the input control information 234.

The label assignment unit 203 receives input of the label definition information 232 and the input control information 234 from the user (step S306). After that, the label assignment unit 203 ends the reclustering process.

The present invention is not limited to the above-described embodiments, but includes various modified examples. For example, the above-described embodiments are provided to explain the present invention in detail, and are not necessarily limited to those including all of the described configurations. In addition, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

The above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium on which it is stored constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs (Solid State Drives), optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.

In addition, the program code that realizes the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C/C++, perl, Shell, PHP, Python, Java (registered trademark), etc.

Furthermore, the program code of the software that realizes the functions of the embodiment may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read and execute the program code stored in the storage means or storage medium.

In the above examples, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. All components may be interconnected.

100 Computer 101 Processor 102 Memory 103 Auxiliary storage device 104 Network interface 105 Input/output interface 111 Input device 112 Output device 200 Object detection unit 201 Bounding box prediction unit 202 Object classification unit 203 Label assignment unit 204 Input control unit 210 Learning unit 220 Action determination unit 230 Classification definition information 231 Occlusion detection definition information 232 Label definition information 233 Object data definition information 234 Input control information 235 Model information 236 Learning information 237 Self information 238 Reward information 239 Cluster information 240 History information 250 Image 260 Action data 300 Vehicle 310 Range 320 Object 330 Bounding box 500 Bounding box information 600 Movement information 700 Object label information 800 Object data 810 Input layer 811 Hidden layer 812 Output layer 820 Recurrent Neural Network 830 Neural Network

Claims (14)

1. A computer system comprising:
At least one computer having a computing device, a storage device connected to the computing device, and an interface connected to the computing device;
The system holds model information for managing a model that determines a behavior of a control target using object data that represents features of objects included in an image of the control target, and input control information for managing input rules for the object data to the model based on labels that are assigned based on the movement of the object,
receiving time series data of the image;
Detecting the object in each of a plurality of the images;
selecting a target image from the plurality of images, calculating motion data representing a motion of the object detected from the target image using two or more of the images including the target image, and storing the target image and the motion data in association with each other;
for each of a plurality of said images, applying said label to said object detected in said image based on said motion data;
A computer system comprising: a computer that controls, for each of a plurality of said images, input of said object data of said object detected from said image to said model based on said input control information and said label. 2. The computer system of claim 1,
The motion data is a vector having physical quantities representing the motion of the object as components,
The computer system comprises:
holding label definition information for managing a rule for assigning the label based on an index calculated using the motion data;
Calculating the indicator using the motion data of the object detected from the image;
A computer system comprising: a computer that assigns the label to the object detected from the image based on the index and the label definition information. 2. The computer system of claim 1,
The motion data is a vector having physical quantities representing the motion of the object as components,
The computer system comprises:
holding cluster information for managing clusters configured from a plurality of pieces of motion data in a feature space having an axis of the physical quantity included in the motion data, and label definition information for managing a rule for assigning the label based on the cluster to which the cluster belongs;
Identifying the cluster to which the object detected from the image belongs based on the motion data and the cluster information;
A computer system comprising: a computer that assigns the label to the object detected from the image based on the identified cluster and the label definition information. 2. The computer system of claim 1,
A bounding box is calculated as an area including the object included in the image;
using two or more of the images including the target image, selecting a target object from among the objects included in the target image, and determining whether the target object is occluded by another of the objects;
correcting the bounding box of the target object if the target object is occluded by another of the objects;
A computer system comprising: a computer that calculates the motion data of the object detected from the target image based on an amount of change in the bounding box of the object in two or more images including the target image. 2. The computer system of claim 1,
Accepting time series data of learning images for learning the model;
Detecting the object from each of the plurality of learning images;
selecting a target learning image from the plurality of learning images, calculating the motion data of the object detected from the target learning image using two or more learning images including the target learning image, and storing the target learning image and the motion data in association with each other;
for each of the plurality of training images, assigning the label to the object detected from the training image based on the motion data;
for each of the plurality of learning images, controlling input of the object data of the object detected from the learning image to the model based on the input control information and the label;
A computer system comprising: a learning process that uses the object data whose input is controlled, thereby updating the model; and a result of the update that is reflected in the model information. A method for controlling input of data to a model, the method being executed by a computer system, comprising:
The computer system comprises:
At least one computer having a computing device, a storage device connected to the computing device, and an interface connected to the computing device;
The system holds model information for managing a model that determines a behavior of a control target using object data that represents features of objects included in an image of the control target, and input control information for managing input rules for the object data to the model based on labels that are assigned based on the movement of the object,
The data input control method includes:
a first step of receiving, by the at least one computer, time series data of the image;
a second step of the at least one computer detecting the object in each of a plurality of the images;
a third step in which the at least one computer selects a target image from the plurality of images, calculates motion data representing a motion of the object detected from the target image using two or more of the images including the target image, and stores the target image and the motion data in association with each other;
a fourth step of the at least one computer assigning, for each of a plurality of the images, the labels to the objects detected in the images based on the motion data;
a fifth step of controlling, for each of a plurality of the images, input of the object data of the object detected from the image to the model based on the input control information and the label by the at least one computer;
13. A method for controlling input of data to a model, comprising: A method for controlling input of data to a model according to claim 6, comprising the steps of:
The motion data is a vector having physical quantities representing the motion of the object as components,
the computer system holds label definition information that manages a rule for assigning the label based on an index calculated using the motion data;
The fourth step includes:
said at least one computer calculating said indicia using said motion data of said object detected from said image;
and a step in which the at least one computer assigns the label to the object detected from the image based on the index and the label definition information. A method for controlling input of data to a model according to claim 6, comprising the steps of:
The motion data is a vector having physical quantities representing the motion of the object as components,
the computer system holds cluster information for managing clusters configured of a plurality of pieces of motion data in a feature space having an axis of the physical quantity included in the motion data, and label definition information for managing a rule for assigning the label based on the cluster to which the cluster belongs;
The fourth step includes:
identifying, by the at least one computer, the cluster to which the object detected from the image belongs based on the motion data and the cluster information;
assigning the labels to the objects detected in the image based on the identified clusters and the label definition information, by the at least one computer;
13. A method for controlling input of data to a model, comprising: A method for controlling input of data to a model according to claim 6, comprising the steps of:
The second step includes:
said at least one computer calculating a bounding box that is an area that contains said object in said image;
the at least one computer using two or more of the images including the target image to select a target object from among the objects included in the target image and determine whether the target object is occluded by another of the objects;
and correcting the bounding box of the target object if the target object is occluded by another of the objects,
A method for controlling input of data to a model, characterized in that the third step includes a step in which the at least one computer calculates the motion data of the object detected from the target image based on the amount of change in the bounding box of the object in two or more images including the target image. A method for controlling input of data to a model according to claim 6, comprising the steps of:
The at least one computer receives time-series data of training images for training the model;
detecting the object from each of a plurality of training images by the at least one computer;
the at least one computer selecting a target learning image from the plurality of learning images, calculating the motion data of the object detected from the target learning image using two or more learning images including the target learning image, and storing the target learning image and the motion data in association with each other;
the at least one computer assigning the label to the object detected from the training image based on the motion data for each of the training images;
a step of controlling, for each of the plurality of learning images, an input of the object data of the object detected from the learning image to the model based on the input control information and the label, by the at least one computer;
updating the model by executing a learning process using the object data whose input is controlled by the at least one computer, and reflecting a result of the update in the model information;
13. A method for controlling input of data to a model, comprising: A control device that determines an action of a control target,
A computing device, a storage device connected to the computing device, and an interface connected to the computing device,
The method includes: storing model information for managing a model that determines a behavior of the control target using object data that represents features of objects included in an image of the control target; and storing input control information for managing input rules for the object data to the model based on labels that are assigned based on the movement of the object;
receiving time series data of the image;
Detecting the object in each of a plurality of the images;
selecting a target image from the plurality of images, calculating motion data representing a motion of the object detected from the target image using two or more of the images including the target image, and storing the target image and the motion data in association with each other;
for each of a plurality of said images, applying said label to said object detected in said image based on said motion data;
A control device characterized by controlling, for each of a plurality of images, input of the object data of the object detected from the image to the model based on the input control information and the label. The control device according to claim 11,
The motion data is a vector having physical quantities representing the motion of the object as components,
The control device includes:
holding label definition information for managing a rule for assigning the label based on an index calculated using the motion data;
Calculating the indicator using the motion data of the object detected from the image;
A control device which assigns the label to the object detected from the image based on the index and the label definition information. The control device according to claim 11,
The motion data is a vector having physical quantities representing the motion of the object as components,
The control device includes:
holding cluster information for managing clusters configured from a plurality of pieces of motion data in a feature space having an axis of the physical quantity included in the motion data, and label definition information for managing a rule for assigning the label based on the cluster to which the cluster belongs;
Identifying the cluster to which the object detected from the image belongs based on the motion data and the cluster information;
A control device which assigns the label to the object detected from the image based on the identified cluster and the label definition information. The control device according to claim 11,
A bounding box is calculated as an area including the object included in the image;
using two or more of the images including the target image, selecting a target object from among the objects included in the target image, and determining whether the target object is occluded by another of the objects;
correcting the bounding box of the target object if the target object is occluded by another of the objects;
A control device comprising: a control unit that calculates the motion data of the object detected from the target image based on an amount of change in the bounding box of the object in two or more images including the target image.

Images