JP7766451B2 - Object detection device - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act. Announced at the 46th Civil Engineering Informatics Symposium (website held) on September 28, 2021.

Article 30, Paragraph 2 of the Patent Act applies. Published on September 22, 2021 in the proceedings of the 46th Civil Engineering Informatics Symposium (website).

This invention relates to technology for extracting objects from images.

By training neural networks such as CNNs (convolutional neural networks), it is possible to extract objects such as people from images.

For example, an object detection model such as YOLO (You Only Look Once) can recognize objects in captured images and output a boundary box surrounding the object.

By using such an object detection model, it is possible to generate data for analyzing the movements of players and cars, for example, by extracting players from images of players moving around the field, or cars from images of cars passing through a public road.

Patent Publication No. 2020-160804

However, in order for such models to operate accurately, they need to be trained sufficiently. This means that a large amount of training data must be prepared, which poses a problem.

Object detection models like the one above estimate whether what is shown is background or an object. A model trained using images with a specific background can extract objects with high accuracy in images where the object exists in that background. However, high accuracy cannot be expected for images where the object exists in a different background. To achieve high-accuracy extraction, it is necessary to generate and train training data from images where the object exists in that different background, which is time-consuming.

SUMMARY OF THE INVENTION An object of the present invention is to solve the above problems and to provide an object detection device that is easy to learn.

Independently applicable features of this invention are listed below:

(1)(2)(3)(4) The new model generation device of the present invention is a device that generates a new object extraction model that can extract an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background, and includes an extracted object acquisition means that provides the different background object image to the existing object extraction model and acquires information about the extracted object, a training data generation means that generates training data based on the different background object image and the information about the object extracted by the extracted object acquisition means, and an additional training means that additionally trains the existing object extraction model based on the generated training data to generate a new object extraction model.

This makes it easy to create a model that can accurately extract an object captured against a desired background.

(5) The new model generation device according to the present invention is characterized in that the new model generation device or the training data generation device is configured as a server device accessible from a terminal device via a network.

Therefore, new models can be obtained by accessing them from a terminal device.

(6) In the novel model generation device according to the present invention, the training data generation means includes an acquisition means for acquiring information about the different background object image and the object extracted by the extracted object acquisition means, a non-background extraction means for extracting non-background parts that are not background from the different background object image using a background subtraction method, and an erroneous detection response means for excluding from the training data information about objects extracted by the extracted object acquisition means that do not have corresponding non-background parts.

This makes it possible to generate more appropriate training data in response to misclassifications made by existing models.

(7) The novel model generation device according to the present invention includes a training data generation means that includes an acquisition means for acquiring information about the different background object image and the object extracted by the extracted object acquisition means, a non-background extraction means for extracting non-background portions that are not background from the different background object image using a background subtraction method, and a detection omission response means that, if there is a region in the different background object image where the object is not extracted by the extracted object acquisition means, converts that region in the different background object image into a featureless image and uses it as training data.

This makes it possible to generate more appropriate training data in response to misclassifications made by existing models.

(8) The novel model generation device of this invention is characterized in that the object is a moving object including one of an athlete, a pedestrian, a car, an animal, and a ball.

Therefore, new models can be generated using these as objects.

(9)-(12) A new model generation system according to the present invention comprises a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and is capable of extracting an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model trained to extract an object from an object image having at least one object in the background,
the new model generation server device comprises: a server-side receiving means for receiving the different background object image and at least weight parameters of the object-extraction existing model transmitted from the terminal device; an extracted object obtaining means for providing the different background object image to the object-extraction existing model to obtain information on the extracted object; a learning data generating means for generating learning data based on the different background object image and the information on the object extracted by the extracted object obtaining means; an additional learning means for additionally learning at least the weight parameters of the object-extraction existing model using the learning data to generate at least weight parameters of an object-extraction new model; and a server-side transmitting means for transmitting at least the weight parameters of the generated object-extraction new model to the terminal device;
The terminal device includes an acquisition means for acquiring at least the weight parameters of the recorded existing object-extraction model, a terminal-side transmission means for transmitting the different background object image and at least the weight parameters of the existing object-extraction model to the new model generation server device, a terminal-side reception means for receiving at least the weight parameters of the new object-extraction model transmitted from the server device, and a new model generation means for replacing the existing object-extraction model with the received at least weight parameters to generate a new object-extraction model.

This makes it easy to create a model that can accurately extract an object captured against a desired background.

(13)(14) The object extraction device of the present invention is an object extraction device that extracts an object from an object video captured of the object, and includes: extracted object acquisition means that provides a portion of the object video to an existing object extraction model that has been trained to extract objects from object images and acquires information about the extracted object; training data generation means that generates training data based on the portion of the object video and the information about the object extracted by the extracted object acquisition means; additional training means that additionally trains the existing object extraction model based on the generated training data and generates a new object extraction model; and object extraction means that extracts an object from a portion other than at least a portion of the object video based on the new object extraction model.

Therefore, training data can be automatically generated to create a new object extraction model suited to the video from which the object is to be extracted, enabling accurate object extraction.

(15) (16) A new model generation system according to the present invention comprises a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and is capable of extracting an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model trained to extract an object from an object image having at least one object in the background,
the new model generation server device comprises: a server-side receiving means for receiving the different background object image transmitted from the terminal device; an additional learning data generating means for generating additional learning data based on the received different background object image; an additional learning means for additionally learning the object extraction existing model using the additional learning data to generate an object extraction new model; a server-side transmitting means for transmitting the generated object extraction new model or its parameters to the terminal device; and a model updating means for recording the object extraction new model as an object extraction existing model,
The terminal device includes a terminal-side transmitting means for transmitting the different background object image to the new model generation server device, and a terminal-side receiving means for receiving the object extraction new model or its parameters transmitted from the server device.

Therefore, existing models stored on the server device are updated and improved in accuracy each time they are used, which increases the accuracy of new models generated based on them.

In this embodiment, step S3 corresponds to the "extraction object acquisition means."

In this embodiment, step S10 corresponds to the "learning data generation means."

In this embodiment, step S21 corresponds to the "server-side receiving means."

In this embodiment, step S11 corresponds to the "additional learning means."

In this embodiment, step S23 corresponds to the "server-side transmission means."

In this embodiment, step S54 corresponds to the "terminal-side receiving means."

In this embodiment, step S55 corresponds to the "new model generation means."

In this embodiment, step S25 corresponds to the "model update means."

The term "device" is a concept that includes not only a single computer, but also multiple computers connected via a network, etc. Therefore, if the means of the present invention (or even part of the means) are distributed across multiple computers, these multiple computers constitute the device.

The term "program" refers not only to programs that can be executed directly by a CPU, but also to source-format programs, compressed programs, encrypted programs, etc.

1 is a functional configuration diagram of a novel model generating device according to an embodiment of the present invention. 1 shows the hardware configuration of a new model generation device. This is the architecture of the YOLO model. 10 is a flowchart of a new model generation program. 10 is a flowchart of a new model generation program. 10 is an example of a frame image used for training an existing model. 10 is an example of a different background frame image (different background object image). This is an image in which the player is extracted by applying the different background frame image of FIG. 7 to an existing model. This is an image obtained by subtracting the background from the image in FIG. 7. This image shows the player's areas that could not be extracted using the existing model de-featurized (blacked out). FIG. 10 is a functional configuration diagram of a new model generation system according to a second embodiment. This is the system configuration of the new model generation system. 1 shows the hardware configuration of a new parameter generation server device. 10 is a flowchart of a terminal program and a server program. 10 is a flowchart of a terminal program and a server program. FIG. 10 is a functional configuration diagram of an object extraction device according to a third embodiment. 10 is a flowchart of an object extraction program. 10 is a flowchart of an object extraction program. FIG. 10 is a functional configuration diagram of a new model generation system according to a fourth embodiment. This is the system configuration of the new model generation system. 10 is a flowchart of a terminal program and a server program. 10 is a flowchart of a terminal program and a server program.

1. First embodiment
1 shows the functional configuration of a new model generation device according to an embodiment of the present invention. The object extraction existing model 4 is a neural network model trained to extract objects from an object image having one or more objects in a background.

The purpose of the new model generation device in this embodiment is to obtain a new object extraction model based on this existing object extraction model that can accurately extract objects even in object images with different backgrounds.

The extracted object acquisition means 2 acquires an object image with a different background (different background object image). This is provided to the object extraction existing model 4 to extract the object and obtain extracted object information.

The training data generation means 6 generates training data based on the different background object images and object information. The additional training means 6 additionally trains the existing object extraction model 4 based on the generated training data.

In this way, a new object extraction model can be generated that can extract objects with high accuracy even in images of objects with different backgrounds, without manually creating learning data.

1.2 Hardware Configuration The hardware configuration of the new model generation device is shown in Figure 2. A CPU 10 is connected to a memory 12, a display 14, an SSD 16, a DVD-ROM drive 18, a keyboard/mouse 20, and a communication circuit 22.

The communications circuit 22 is a circuit for connecting to the Internet. The SSD 16 stores an operating system 30 and a new model generation program 32. The new model generation program 32 functions in cooperation with the operating system 30. These programs were originally stored on a DVD-ROM 36 and were installed on the SSD 16 via the DVD-ROM drive 18. The SSD 16 also stores an object extraction existing model 34.

1.3 New Model Generation Process In this embodiment, YOLO is used as the object detection model. The YOLO architecture is shown in Figure 3. After passing through multiple convolutional and max pooling layers, a fully connected layer outputs rectangular information indicating the area of the detected object (for example, the coordinates of two points in the upper left and lower right), the object label (object type), and the accuracy of the label.

In this embodiment, players are the objects to be detected. YOLO has a model that has been trained in advance to distinguish and detect various objects, including people. Using this trained model as a base, additional learning is performed based on training data such as images of players against various field backgrounds and players photographed from various angles.

Figure 6 shows an example of training data used to train an existing model. As shown in Figure 6A, frame images are extracted at intervals of a predetermined number from video images of players on the field (for example, one frame image is extracted every 10 frames), and the player's area is manually enclosed in a boundary box. This records the coordinate positions (top left and bottom right coordinates) of the frame images and the boundary boxes indicating the positions of players, coaches, etc. This becomes the training data. The above process is performed on a large number of frames to obtain training data.

In addition, as shown in Figure 6B, frame images from different fields are also prepared as training data.

The YOLO model is trained based on the training data shown in Figures 6A and 6B. This allows a basic trained model for player detection (an existing model for player detection) to be obtained. Note that it is also possible to train an untrained YOLO model using the above training data without using a trained YOLO model.

Figure 4 shows a flowchart of the new model generation program. The CPU 10 acquires a series of images (frame images) of a player captured against a background different from the background used for the learning described above (step S1). Here, "captured against a different background" refers to images of a player captured on a different field, or images of a player captured on the same field but from a different angle, etc.

For example, frame images such as those shown in Figure 7 are used. In this embodiment, frame images are extracted from captured video data (for example, one frame image is extracted for every 10 frames), and the CPU 10 acquires these. In this embodiment, frame images are extracted from a video of approximately several minutes in length.

Next, the CPU 10 selects the first of the frame images as the target frame image and performs the following process. Using the existing model 34 that has been trained for player detection, it detects players in the target frame image. This makes it possible to obtain the coordinates (coordinates of the upper left and lower right points) of the boundary boxes surrounding players, coaches, etc. on the frame image.

The coordinates of the midpoint of the bottom edge of the obtained boundary box are found, and it is determined whether this is inside the line 45 that surrounds the field in the image. If it is inside, it can be determined that the person is a player on the field. If it is outside, it can be determined that the person is a substitute player or coach, and the boundary box for that person is deleted.

When this is displayed as an image, it becomes a frame image in which the detected players are surrounded by a boundary box, as shown in Figure 8. Note that the line 45 surrounding the field is specified in advance by the operator using the mouse while the frame image is displayed on the display.

Next, the CPU 10 extracts object images other than the background from the target frame image using background subtraction (step S4). Note that if background subtraction is applied to an image from a time period with no shadows using a background from a time period with shadows, the shadows will be extracted as object images. Therefore, in this embodiment, a background subtraction image is obtained using dynamic background subtraction (MOG2) using a mixed normal distribution.

Dynamic background subtraction calculates a representative value for each pixel over a specified number of frames before and after the target frame (for example, 100 frames before and after), and generates a background image using these representative values. Therefore, if there is shadow in the image, a background image that includes the shadow is obtained, and the shadow itself is not extracted as an object.

The object images obtained in this way contain small noises that are not players. Therefore, in this embodiment, the standard height H (length in the Y-axis direction) of a player is determined according to the Y direction shown in Figure 9 (because the further back the player is, the smaller the player appears), and objects that are less than one-third of this standard height H are deemed not to be players and are deleted.

A rectangle is then generated surrounding the object obtained as an image, and only objects whose coordinates of the midpoint of the bottom edge are within the line 54 surrounding the field are retained. This allows for the acquisition of images of only the players on the field. Figure 9 shows a background difference image for the target frame image in Figure 7.

Next, the CPU 10 selects a location where there is a boundary box of a player detected using an existing player detection model, but no object image detected as a difference exists at the corresponding position (step S5). For example, in the player detection image of Figure 8, a player is detected as boundary box 40, but in the background difference image of Figure 9, no object appears at the corresponding position. Therefore, the CPU 10 selects this boundary box 40.

The CPU 10 then deletes the selected boundary box 40 (step S6). In other words, it is considered that the object was not detected as a player. If an object is detected as a player even though it does not appear in the background subtraction image, it is highly likely that the detection was erroneous. Therefore, it is not desirable to use this as training data.

In the example shown in Figure 8, only one boundary box 40 is found to be relevant, but if multiple boundary boxes are relevant, all of these boundary boxes will be deleted.

Next, the CPU 10 selects a location where an object is present in the background difference image of Figure 9 but no boundary box appears in the player detection image of Figure 8 (step S7). For example, object (player) 50 appears in the background difference image of Figure 9, but no boundary box appears in the corresponding position in the player detection image of Figure 8. Therefore, the CPU 10 selects player 50.

As shown in Figure 10, the CPU 10 sets a player area 65 (here a rectangular area, but any other shape that surrounds a player can be used) surrounding the player 50 in the target frame image and blacks out this area (step S8). In other words, it is assumed that there was no player in that area. Because an object (player) appeared in the background difference image but could not be detected as a player, it is highly likely that the image of the object (player) is not suitable for detection. Therefore, in order to remove the image of the object (player) from the target frame image, it is blacked out as described above. By blacking out the area, the image characteristics of that area are eliminated.

In addition, instead of painting it black, it may be painted solid with a specified color. Also, a background image may be inserted into the area of the object (player).

In the example of Figure 10, only one player area 65 is applicable, but if multiple areas are applicable, all of these areas will be painted black.

In this way, the coordinate information of the boundary boxes detected as players (excluding those deleted in step S6) is added to the target frame image that has been processed in steps S7 and S8, and this is used as learning data.

Once learning data is generated based on the target frame image as described above, the CPU 10 repeats steps S3 to S8 with the next frame as the target frame (steps S2 and S9). When processing has been completed for all frame images, learning data for additional learning will be obtained (step S10).

The generated training data contains new backgrounds that were not included in the backgrounds of the training data for existing models. Therefore, by additionally training existing models using this training data, it is possible to obtain models that can accurately determine new backgrounds as well.

The CPU 10 uses the training data generated in this way to additionally train the existing model 34 (step S11), thereby obtaining a detection model that can accurately detect players even against new backgrounds.

1.4 Other
(1) In the above embodiment, only players are detected. However, instead of or in addition to players, balls, coaches, substitute players, etc. may also be detected.

It can also be applied to moving objects in general, including not only athletes but also people such as pedestrians and cars.

(2) In the above embodiment, people (players) within the competition area are the detection targets. However, people (players) outside the competition area may also be the detection targets.

(3) In the above embodiment, the generated training data is used to additionally train an existing model trained against a specific background. However, the generated training data may also be used to train an untrained model (or a trained model provided by YOLO).

(4) In the above embodiment, training data is generated using frame images. However, training data may also be generated using still images other than frame images.

(5) In the above embodiment, in step S8, the player in the target frame image is painted black. However, it is also possible to leave the target frame image as is and add a boundary box to surround the player.

(6) In this embodiment, a YOLO (You Only Look Once) object detection model is used to recognize the position and area of the object (player) from the captured image, and the object is surrounded by a boundary box and output. Note that object detection algorithms other than YOLO, such as SSD (Single Shot MultiBox Detector) and R-CNN (Regions with CNN features), may also be used. Instance segmentation algorithms such as Mask R-CNN and YOLACT may also be used.

(7) The above modifications can be applied to other embodiments as long as they do not contradict the essence of the embodiments.

2. Second embodiment
11 shows the functional configuration of the new model generation system according to the second embodiment. This system includes a terminal device 80 and a parameter generation server device 60.

The terminal device 80 stores an existing object extraction model 84 that has been trained based on an object image having at least one object in a specified background. In this embodiment, YOLO is used as the model, as in the first embodiment. Therefore, the existing object extraction model 84 is constructed with the YOLO program, hyperparameters (such as the number of convolutions to prepare, the number of neural network layers, and normalization coefficients), and weight parameters for connecting each node.

This system incrementally trains an existing object extraction model 84 stored in a terminal device 80 so that it can detect objects in response to new background images (different background object images).

The acquisition means 82 of the terminal device 80 acquires the weight parameters of the existing object extraction model 84. The terminal-side transmission means 86 transmits these weight parameters and the different background object image 88 to the server device 60. In this embodiment, the hyperparameters are fixed in advance, so they are not transmitted to the server device 60.

The server-side receiving means 62 receives this. The extracted object acquisition means 64 constructs an existing object model 70 based on the received weighting parameters, and provides this with the received different background object image 88 to detect the object. The extracted object acquisition means 64 acquires information about the detected object (boundary box coordinates).

The training data generation means 66 generates training data based on the different background object image 88 from the terminal device and information about the detected object. The additional training means 88 additionally trains the existing object model 70 using the generated training data to obtain new weight parameters 72.

The server-side transmitting means 74 transmits the new weight parameters 72 to the terminal device. The terminal-side receiving means 94 receives the new weight parameters 72. The new model generating means 90 replaces the weight parameters of the existing object extraction model 84 with the received new weight parameters to generate a new object extraction model 92.

In this manner, by transmitting the weighting parameters and the different background object image from the terminal device 80 to the server device 60, new weighting parameters can be obtained and a new object extraction model 92 can be generated.

2.2 System Configuration and Hardware Configuration Fig. 12 shows the system configuration of the new model generation system. The parameter generation server device 60 is constructed as a server device on the Internet. Terminal devices 80a, 80b, ..., 80x can communicate with the parameter generation server device 60 via the Internet.

The hardware configuration of the terminal device 80 is the same as that shown in Figure 2 in the first embodiment. However, a terminal program is stored on the SSD 16 instead of the new model generation program 32.

Figure 13 shows the hardware configuration of the parameter generation server device 60. The CPU 110 is connected to a memory 112, an SSD 114, a DVD-ROM drive 116, and a communication circuit 118. The communication circuit 118 is a circuit for connecting to the Internet.

The SSD 114 stores an operating system 120, a server program 122, and an object extraction unlearned model 124. These programs were originally recorded on a DVD-ROM 126 and have been installed on the SSD 114 via the DVD-ROM drive 116. The programs may also be installed from another server device via the Internet.

2.3 New Model Generation Process FIGS. 14 and 15 show flowcharts of the server program 122 and the terminal program.

The SSD 16 of the terminal device 80 stores an existing object extraction model 34 constructed by YOLO or similar. As with the first embodiment, this existing object extraction model 34 has been trained using images of players captured against various backgrounds (for example, images such as those shown in Figures 6A and 6B).

The CPU 10 of the terminal device 80 (hereinafter sometimes referred to as the terminal device 80) acquires a series of images (frame images) of a player captured against a background different from the background used in the learning (step S51). Here, "captured against a different background" refers to images of a player captured on a different field, or images of a player captured on the same field but from a different angle, etc.

For example, frame images such as those shown in Figure 7 are used. In this embodiment, frame images are extracted from the captured video data (for example, one frame image is extracted every 10 frames), and the terminal device 80 acquires these. In this embodiment, frame images are extracted from a video of approximately several minutes in length.

Next, the terminal device 80 acquires the weighting parameters of the trained object extraction existing model 34 recorded on the SSD 16 (step S52). Subsequently, the terminal device 80 transmits the group of frame images with different backgrounds acquired in step S51 and the weighting parameters acquired in step S52 to the parameter generation server device 60 (step S53).

The CPU 110 of the parameter generation server device 60 (hereinafter sometimes abbreviated as the parameter generation server device 60) receives a group of frame images with different backgrounds and weight parameters of the existing model (step S21). Next, the parameter generation server device 60 reads the unlearned object extraction model 124 recorded on the SSD 114, and replaces the unlearned parameters with the weight parameters of the received existing model to generate an object extraction existing model (step S22).

Next, the parameter generation server device 60 selects the first of the received different background frame images as the target frame image and performs the following process. Using the existing object extraction model generated in step S22, players are detected for the target frame image (step S3). This allows the coordinates (coordinates of the upper left and lower right points) of the boundary boxes surrounding players, coaches, etc. on the frame image to be obtained. In the same way as in the first embodiment, only the boundary boxes of the players on the field are left. When this is displayed as an image, the frame image will have the detected players surrounded by boundary boxes, as shown in Figure 8.

Next, the parameter generation server device 60 extracts an object image other than the background from the target frame image using background subtraction (step S4). As in the first embodiment, a background subtraction image is obtained using a dynamic background subtraction method (MOG2) based on a mixed normal distribution. The method of removing small noise based on the length in the Y-axis direction is also the same as in the first embodiment. Furthermore, as in the first embodiment, only images of the players on the field are obtained, as shown in Figure 9.

Next, the parameter generation server device 60 selects a location where there is a boundary box of a player detected using an existing player detection model, but where there is no object image detected as a difference at the corresponding position (step S5). For example, in the player detection image of Figure 8, a player is detected as boundary box 40, but in the background difference image of Figure 9, no object appears at the corresponding position. Therefore, the parameter generation server device 60 selects this boundary box 40.

The parameter generation server device 60 then deletes the selected boundary box 40 (step S6). In other words, it is considered that it was not detected as a player. If an object is detected as a player even though it does not appear in the background subtraction image, it is highly likely that it was detected incorrectly. Therefore, it is not desirable to use this as training data.

In the example shown in Figure 8, only one boundary box 40 is found to be relevant, but if multiple boundary boxes are relevant, all of these boundary boxes will be deleted.

Next, the parameter generation server device 60 selects a location where an object is present in the background difference image of Figure 9 but no boundary box appears in the player detection image of Figure 8 (step S7). For example, object (player) 50 appears in the background difference image of Figure 9, but no boundary box appears in the corresponding position in the player detection image of Figure 8. Therefore, the parameter generation server device 60 selects player 50.

As shown in Figure 10, the parameter generation server device 60 sets a player area 65 surrounding the player 50 in the target frame image and paints this area black (step S8). In other words, it is assumed that there was no player in that area. Because an object (player) appeared in the background difference image but could not be detected as a player, it is highly likely that the image of the object (player) is not suitable for detection. Therefore, in order to remove the image of the object (player) from the target frame image, it is painted black as described above. Painting it black eliminates the image characteristics of that area.

In addition, instead of painting it black, it may be painted solid with a specified color. Also, a background image may be inserted into the area of the object (player).

In the example of Figure 10, only one player area 65 is applicable, but if multiple areas are applicable, all of these areas will be painted black.

In this way, the coordinate information of the boundary boxes detected as players (excluding those deleted in step S6) is added to the target frame image that has been processed in steps S7 and S8, and this is used as learning data.

Once learning data is generated based on the target frame image as described above, the parameter generation server device 60 repeats steps S3 to S8 with the next frame as the target frame (steps S2 and S9). Once processing has been completed for all frame images, learning data for additional learning will be obtained (step S10).

The generated training data contains new backgrounds that were not included in the backgrounds of the training data for existing models. Therefore, by additionally training existing models using this training data, it is possible to obtain models that can accurately determine new backgrounds as well.

The parameter generation server device 60 uses the training data generated in this way to additionally train the existing model generated in step S22 (step S11). This makes it possible to obtain a detection model that can accurately detect players even in new backgrounds.

The parameter generation server device 60 obtains the weight parameters of the model that has been additionally trained in this way and transmits them to the terminal device 80 (step S23).

The terminal device 80 receives the weight parameters transmitted from the parameter generation server device 60 (step S54). The terminal device 80 replaces the parameters of the existing object extraction model 34 recorded in the SSD 16 with the received weight parameters to generate a new object extraction model (step S55). By transmitting the weight parameters of the existing model and the different background object image from the terminal device 80 to the parameter generation server device 60 in the above manner, additional learned parameters can be obtained and a new object extraction model can be generated.

2.4 Other
(1) In the above embodiment, the weight parameters of the existing model are transmitted from the terminal device 80 to the server device 60. However, not only the weight parameters but also the hyperparameters may be transmitted. In this case, the parameter generation server device 60 additionally learns not only the weight parameters but also the hyperparameters and transmits them to the terminal device 80. In response to this, the terminal device 80 generates a new object extraction model based on the weight parameters and hyperparameters.

(2) In the above embodiment, weight parameters are transmitted from the terminal device 80 to the server device 60. However, the entire existing object extraction model may be transmitted to the server device 60, which may then perform additional learning and return a new object extraction model.

(3) The above modifications can be applied to other embodiments as long as they do not contradict the essence of the embodiments.

3. Third embodiment
3.1 Functional Configuration FIG. 16 shows the functional configuration of the object extraction device according to the third embodiment.

The existing object extraction model 4 is a neural network model trained to extract objects from each object image (still image) that makes up an object video.

The extracted object acquisition means 2 acquires a portion (e.g., the first few hundred frames) of the object video (different from the object video used to train the object extraction existing model 4). This is provided to the object extraction existing model 4 to extract the object and obtain extracted object information.

The training data generation means 6 generates training data based on the object video and object information. The additional training means 6 additionally trains the object extraction existing model 4 based on the generated training data. The object extraction means 7 provides the object video to the object extraction new model 9 to extract the object from the object video.

In this way, a new object extraction model 9 can be obtained that can accurately extract objects from object videos from which the objects are to be extracted, without having to manually create learning data, and the objects can be extracted.

3.2 Hardware Configuration The hardware configuration is the same as that of the first embodiment, except that the SSD 16 stores an object extraction program instead of the new model generation program 32.

17 and 18 show a flowchart of the object extraction program. The CPU 10 acquires a predetermined number of frames (for example, 1000 frames) from the start of the video from which players are to be extracted (step S1). The video from which players are to be extracted is, of course, different from the video used to train the existing model 34 (this includes cases where the background is different, or where the background is the same but the players are different).

Next, the CPU 10 selects the first of the frame images as the target frame image and performs the following process. Using the existing model 34 that has been trained for player detection, it detects players in the target frame image. This makes it possible to obtain the coordinates (coordinates of the upper left and lower right points) of the boundary boxes surrounding players, coaches, etc. on the frame image.

The coordinates of the midpoint of the bottom edge of the obtained boundary box are found, and it is determined whether this is inside the line 45 that surrounds the field in the image. If it is inside, it can be determined that the person is a player on the field. If it is outside, it can be determined that the person is a substitute player or coach, and the boundary box for that person is deleted.

When this is displayed as an image, it becomes a frame image in which the detected players are surrounded by a boundary box, as shown in Figure 8. Note that the line 45 surrounding the field is specified in advance by the operator using the mouse while the frame image is displayed on the display.

Next, the CPU 10 extracts object images other than the background from the target frame image using background subtraction (step S4). This process is the same as in the first embodiment.

Next, the CPU 10 selects a location where there is a boundary box of a player detected using an existing player detection model, but no object image detected as a difference exists at the corresponding position (step S5). For example, in the player detection image of Figure 8, a player is detected as boundary box 40, but in the background difference image of Figure 9, no object appears at the corresponding position. Therefore, the CPU 10 selects this boundary box 40.

The CPU 10 then deletes the selected boundary box 40 (step S6). In other words, it is considered that the object was not detected as a player. If an object is detected as a player even though it does not appear in the background subtraction image, it is highly likely that the detection was erroneous. Therefore, it is not desirable to use this as training data.

In the example shown in Figure 8, only one boundary box 40 is found to be relevant, but if multiple boundary boxes are relevant, all of these boundary boxes will be deleted.

Next, the CPU 10 selects a location where an object is present in the background difference image of Figure 9 but no boundary box appears in the player detection image of Figure 8 (step S7). For example, object (player) 50 appears in the background difference image of Figure 9, but no boundary box appears in the corresponding position in the player detection image of Figure 8. Therefore, the CPU 10 selects player 50.

As shown in Figure 10, the CPU 10 sets a player area 65 (here a rectangular area, but any other shape that surrounds a player can be used) surrounding the player 50 in the target frame image and blacks out this area (step S8). In other words, it is assumed that there was no player in that area. Because an object (player) appeared in the background difference image but could not be detected as a player, it is highly likely that the image of the object (player) is not suitable for detection. Therefore, in order to remove the image of the object (player) from the target frame image, it is blacked out as described above. By blacking out the area, the image characteristics of that area are eliminated.

In addition, instead of painting it black, it may be painted solid with a specified color. Also, a background image may be inserted into the area of the object (player).

In the example of Figure 10, only one player area 65 is applicable, but if multiple areas are applicable, all of these areas will be painted black.

In this way, the coordinate information of the boundary boxes detected as players (excluding those deleted in step S6) is added to the target frame image that has been processed in steps S7 and S8, and this is used as learning data.

Once learning data is generated based on the target frame image as described above, the CPU 10 repeats steps S3 to S8 with the next frame as the target frame (steps S2 and S9). Once processing has been completed for all acquired frame images, learning data for additional learning will be obtained (step S10).

The generated training data differs from the training data of existing models in that it is generated using a portion of the video from which players are to be extracted. Therefore, by additionally training the existing model using this training data, it is possible to obtain a new model that can make accurate judgments on new videos as well.

The CPU 10 uses the training data generated in this way to additionally train the existing model 34 (step S11). This makes it possible to obtain a new model that can accurately detect players even when using new videos.

Next, the CPU 10 acquires all frames of the video (step S12). Next, using the new model generated above, player detection is performed for all frames of the video (step S14). This makes it possible to track players, etc.

3.4 Other
(1) In the above embodiment, the training data is generated using frames from the beginning of the video. However, the training data may be generated using other parts of the video.

(2) In the above embodiment, player detection is performed using a new model for all frames of the video. However, it is also possible to use player detection results using an existing model (detection results obtained when creating training data) for part of the video, and player detection results using a new model for other parts.

(3) The above modifications can be applied to other embodiments as long as they do not contradict the essence of the embodiments.

4. Third embodiment
4.1 Functional Configuration Fig. 19 shows the functional configuration of a new model generation system according to the fourth embodiment. Terminal-side transmitting means 86 transmits an object image (corresponding to the different-background object image in the previous embodiment) to a new model generation server device 90. Server-side receiving means 62 receives this. Training data generating means 66 generates training data for additional training based on this object image. Additional training means 68 additionally trains the existing object extraction model 70 using this training data, thereby obtaining a new object extraction model 76.

The server-side transmitting means 74 transmits the generated object extraction new model 76 to the terminal device 80. The terminal-side receiving means 94 receives this object extraction new model 76.

In this way, a model 76 can be obtained that can accurately extract an object from an object image with any background.

The update means 78 updates the generated object extraction new model 76 as the object extraction existing model 70. As a result, the next time an object image is sent, further additional learning can be performed based on the additionally learned object extraction model. Therefore, as the use of the new model generation server device 90 increases, the accuracy of the base object extraction existing model 70 increases, and the accuracy of the model additionally learned from this increases as well.

4.2 System Configuration and Hardware Configuration Figure 20 shows the system configuration of the new model generation system. The new model generation server device 90 is constructed as a server device on the Internet. Terminal devices 80a, 80b, ..., 80x can communicate with the new model generation server device 90 via the Internet.

The hardware configuration of the terminal device 80 is the same as that shown in Figure 2 in the first embodiment. However, a terminal program is stored on the SSD 16 instead of the new model generation program 32.

The hardware configuration of the new model generation server device 90 is the same as that shown in Fig. 13 in the second embodiment, except that the object extraction existing model 76 is recorded in the SSD 114 instead of the object extraction unlearned model 124.

3.3 New Model Generation Process FIGS. 21 and 22 show a flowchart of the terminal program and a flowchart of the server program.

The SSD 114 of the new model generation server device 90 stores an existing object extraction model 34 constructed by YOLO or similar. As with the first embodiment, this existing object extraction model 34 has been trained using images of players captured against various backgrounds (for example, images like those in Figures 6A and 6B).

The terminal device 80 acquires a series of images (frame images) of the player captured against a background different from the background used for the learning (step S51).

In response to an operation by the operator, the terminal device 80 transmits this different background object image (frame image group) to the new model generation server device 60 (step S53). The new model generation server device 90 receives this frame image group (step S21).

Next, the new model generation server device 90 selects the first of the received different background frame images as the target frame image and performs the following process. Using the recorded existing object extraction model, players are detected for the target frame image (step S3). This allows the coordinates (coordinates of the upper left and lower right points) of the boundary boxes surrounding players, coaches, etc. on the frame image to be obtained. In the same way as in the first embodiment, only the boundary boxes of the players on the field are left. When this is displayed as an image, the frame image will have the detected players surrounded by boundary boxes, as shown in Figure 8.

Next, the new model generation server device 90 extracts object images other than the background from the target frame image using background subtraction (step S4). As in the first embodiment, a background subtraction image is obtained using dynamic background subtraction (MOG2) using a mixed normal distribution. The method of removing small noise based on the length in the Y-axis direction is also the same as in the first embodiment. Furthermore, as in the first embodiment, only images of the players on the field are obtained, as shown in Figure 9.

Next, the new model generation server device 90 selects a location where there is a boundary box of a player detected by an existing player detection model, but where there is no object image detected as a difference at the corresponding position (step S5). For example, in the player detection image of Figure 8, a player is detected as boundary box 40, but in the background difference image of Figure 9, no object appears at the corresponding position. Therefore, the new model generation server device 90 selects this boundary box 40.

Next, the new model generation server device 90 deletes the selected boundary box 40 (step S6).

In the example shown in Figure 8, only one boundary box 40 is found to be relevant, but if multiple boundary boxes are relevant, all of these boundary boxes will be deleted.

Next, the new model generation server device 90 selects a location where an object is present in the background difference image of Figure 9 but no boundary box appears in the player detection image of Figure 8 (step S7). For example, object (player) 50 appears in the background difference image of Figure 9, but no boundary box appears in the corresponding position in the player detection image of Figure 8. Therefore, the new model generation server device 90 selects player 50.

The new model generation server device 90 sets a player area 65 surrounding the player 50 in the target frame image, as shown in Figure 10, and paints this area black (step S8). In other words, it is assumed that no player was present in that area.

In the example of Figure 10, only one player area 65 is applicable, but if multiple areas are applicable, all of these areas will be painted black.

In this way, the coordinate information of the boundary boxes detected as players (excluding those deleted in step S6) is added to the target frame image that has been processed in steps S7 and S8, and this is used as learning data.

Once training data is generated based on the target frame image as described above, the new model generation server device 90 repeats steps S3 to S8 with the next frame as the target frame (steps S2 and S9). Once processing has been completed for all frame images, training data for additional training will be obtained (step S10).

The generated training data contains new backgrounds that were not included in the backgrounds of the training data for existing models. Therefore, by additionally training existing models using this training data, it is possible to obtain models that can accurately determine new backgrounds as well.

The new model generation server device 90 additionally learns the existing recorded model (step S11). This makes it possible to obtain a detection model that can accurately detect players even in new backgrounds.

The new model generation server device 90 transmits the new model obtained by additionally learning the existing model in this manner to the terminal device 80 (step S23).

The terminal device 80 receives the new model transmitted from the new model generation server device 90 (step S54). The terminal device 80 records this new model on the SSD 16.

In this way, a new model can be obtained by additional learning based on an existing model for an object image having a desired background.

4.4 Other
(1) In the above embodiment, the generated new model is transmitted to the terminal device 80. However, if a base model (an unlearned or learned model) exists on the terminal device 80 side, only the weight parameters of the new model may be transmitted. This is because the new model can be constructed on the terminal device 80 side.

(2) In the above embodiment, weight parameters are additionally learned. However, hyperparameters may also be additionally learned.

(3) In the above embodiment, the methods for generating additional learning data and performing additional learning are the same as those in the first and second embodiments. However, other methods for additional learning may be used. For example, learning data generated in the terminal device 80 may be transmitted to the server device 90 for additional learning.

(4) The above modifications can be applied to other embodiments as long as they do not contradict the essence of the embodiment.

Claims (18)

1. An apparatus for generating a new object extraction model that can extract an object from a different background object image having at least one object in a background different from an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background, the new object extraction model comprising:
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
additional learning means for additionally learning the existing object extraction model based on the generated learning data to generate a new object extraction model;
A novel model generating device comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A novel model generating device comprising: A new model generation program for realizing, by a computer, an apparatus for generating a new object extraction model capable of extracting an object from a different background object image having at least one object in a background different from an existing object extraction model trained to extract an object from an object image having at least one object in the background, the new model generation program comprising:
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
a new model generation program for functioning as additional learning means for additionally learning the existing object extraction model based on the generated learning data and generating a new object extraction model,
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A new model generation program comprising: 1. A training data generation device that generates training data for additional training based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in a background,
an extracted object acquisition means for providing a different background object image having at least one object in a background different from the background of the object image used to train the object extraction existing model to the object extraction existing model, and acquiring information about the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
A training data generation device comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A training data generation device comprising: A training data generation program for realizing a training data generation device that generates training data for additional learning based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in a background, the program comprising:
an extracted object acquisition means for providing a different background object image having at least one object in a background different from the background of the object image used to train the object extraction existing model to the object extraction existing model, and acquiring information about the extracted object;
a learning data generation program for causing the computer to function as learning data generation means for generating learning data based on the different background object image and information about the object extracted by the extracted object acquisition means,
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A learning data generation program comprising: 1. An apparatus for generating a new object extraction model that can extract an object from a different background object image having at least one object in a background different from an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background, the new object extraction model comprising:
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
additional learning means for additionally learning the existing object extraction model based on the generated learning data to generate a new object extraction model;
A novel model generating device comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A novel model generating device comprising: A new model generation program for realizing, by a computer, an apparatus for generating a new object extraction model capable of extracting an object from a different background object image having at least one object in a background different from an existing object extraction model trained to extract an object from an object image having at least one object in the background, the new model generation program comprising:
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
a new model generation program for functioning as additional learning means for additionally learning the existing object extraction model based on the generated learning data and generating a new object extraction model,
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A new model generation program comprising: 1. A training data generation device that generates training data for additional training based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in a background,
an extracted object acquisition means for providing a different background object image having at least one object in a background different from the background of the object image used to train the object extraction existing model to the object extraction existing model, and acquiring information about the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
A training data generation device comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A training data generation device comprising: A training data generation program for realizing a training data generation device that generates training data for additional learning based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in a background, the program comprising:
an extracted object acquisition means for providing a different background object image having at least one object in a background different from the background of the object image used to train the object extraction existing model to the object extraction existing model, and acquiring information about the extracted object;
a learning data generation program for causing the computer to function as learning data generation means for generating learning data based on the different background object image and information about the object extracted by the extracted object acquisition means,
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A learning data generation program comprising: In the new model generation device or the training data generation device according to any one of claims 1, 3, 5 and 7,
The new model generation device or the training data generation device is configured as a server device that can be accessed from a terminal device via a network. In the device or program according to any one of claims 1 to 9,
The object is a moving object including one of an athlete, a pedestrian, a car, an animal, and a ball.


A new model generation system comprising a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and capable of extracting an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model trained to extract an object from an object image having at least one object in the background,
The new model generation server device
a server-side receiving means for receiving the different background object image and at least weight parameters of the object extraction existing model transmitted from the terminal device;
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
additional learning means for additionally learning at least weight parameters of the existing object extraction model using the learning data to generate at least weight parameters of a new object extraction model;
a server-side transmitting means for transmitting at least weight parameters of the generated new object extraction model to the terminal device;
The terminal device
an acquisition means for acquiring at least weight parameters of the recorded object extraction existing model;
a terminal-side transmitting means for transmitting the different background object image and at least weight parameters of the object-extraction existing model to the new model generation server device;
a terminal-side receiving means for receiving at least the weight parameters of the new object extraction model transmitted from the server device;
a new model generating means for generating a new object extraction model by replacing the existing object extraction model with at least the received weight parameters;
In a novel model generation system comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A novel model generation system comprising: A server device that generates at least weight parameters of a new object extraction model that can extract an object from a different background object image having at least one object in a background different from an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background,
a server-side receiving means for receiving the different background object image and at least weight parameters of the object extraction existing model transmitted from a terminal device ;
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
additional learning means for additionally learning at least weight parameters of the existing object extraction model using the learning data to generate at least weight parameters of a new object extraction model;
a server-side transmitting means for transmitting at least weight parameters of the generated new object extraction model to the terminal device;
In a parameter generation server device comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A parameter generation server device comprising: A terminal device constituting a new model generation system includes a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and is capable of extracting an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model trained to extract an object from an object image having at least one object in the background,
an acquisition means for acquiring at least weight parameters of the recorded object extraction existing model;
a terminal-side transmitting means for transmitting the different background object image and at least weight parameters of the object-extraction existing model to the new model generation server device;
a terminal-side receiving means for receiving at least weight parameters of a new model for object extraction transmitted from a new model generation server device that has additionally learned at least weight parameters of the existing model for object extraction using learning data generated based on information of the object extracted from the different background object image by the existing model for object extraction;
a new model generating means for generating a new object extraction model by replacing the existing object extraction model with at least the received weight parameters;
In a terminal device comprising:
The new model generation server device
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A terminal device comprising: A terminal program for realizing, by a computer, the terminal device constituting a new model generation system that includes a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and that is capable of extracting an object from a different background object image having at least one object in a background different from the background based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background, the terminal program comprising:
an acquisition means for acquiring at least weight parameters of the recorded object extraction existing model;
a terminal-side transmitting means for transmitting the different background object image and at least weight parameters of the object-extraction existing model to the new model generation server device;
a terminal-side receiving means for receiving at least weight parameters of a new model for object extraction transmitted from a new model generation server device that has additionally learned at least weight parameters of the existing model for object extraction using learning data generated based on information of the object extracted from the different background object image by the existing model for object extraction;
a terminal program for causing the terminal to function as new model generation means for generating a new object extraction model by replacing the existing object extraction model with at least the received weight parameters,
The new model generation server device
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a false detection handling means for excluding information on objects that do not have corresponding non-background portions from the learning data, among the objects extracted by the extracted object obtaining means;
A terminal program comprising:

A new model generation system comprising a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and capable of extracting an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model trained to extract an object from an object image having at least one object in the background,
The new model generation server device
a server-side receiving means for receiving the different background object image and at least weight parameters of the object extraction existing model transmitted from the terminal device;
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
additional learning means for additionally learning at least weight parameters of the existing object extraction model using the learning data to generate at least weight parameters of a new object extraction model;
a server-side transmitting means for transmitting at least weight parameters of the generated new object extraction model to the terminal device;
The terminal device
an acquisition means for acquiring at least weight parameters of the recorded object extraction existing model;
a terminal-side transmitting means for transmitting the different background object image and at least weight parameters of the object-extraction existing model to the new model generation server device;
a terminal-side receiving means for receiving at least the weight parameters of the new object extraction model transmitted from the server device;
a new model generating means for generating a new object extraction model by replacing the existing object extraction model with at least the received weight parameters;
In a novel model generation system comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A novel model generation system comprising: A server device that generates at least weight parameters of a new object extraction model that can extract an object from a different background object image having at least one object in a background different from an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background,
a server-side receiving means for receiving the different background object image and at least weight parameters of the object extraction existing model transmitted from a terminal device;
an extracted object acquisition means for providing the different background object image to the object extraction existing model to acquire information on the extracted object;
a learning data generating means for generating learning data based on the different background object image and information on the object extracted by the extracted object obtaining means;
additional learning means for additionally learning at least weight parameters of the existing object extraction model using the learning data to generate at least weight parameters of a new object extraction model;
a server-side transmitting means for transmitting at least weight parameters of the generated new object extraction model to the terminal device;
In a parameter generation server device comprising:
The learning data generation means
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A parameter generation server device comprising: A terminal device constituting a new model generation system includes a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and is capable of extracting an object from a different background object image having at least one object in a background different from the background, based on an existing object extraction model trained to extract an object from an object image having at least one object in the background,
an acquisition means for acquiring at least weight parameters of the recorded object extraction existing model;
a terminal-side transmitting means for transmitting the different background object image and at least weight parameters of the object-extraction existing model to the new model generation server device;
a terminal-side receiving means for receiving at least weight parameters of a new model for object extraction transmitted from a new model generation server device that has additionally learned at least weight parameters of the existing model for object extraction using learning data generated based on information of the object extracted from the different background object image by the existing model for object extraction;
a new model generating means for generating a new object extraction model by replacing the existing object extraction model with at least the received weight parameters;
In a terminal device comprising:
The new model generation server device
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A new model generation server device comprising: A terminal program for realizing, by a computer, the terminal device constituting a new model generation system that includes a new model generation server device and a terminal device configured to be able to communicate with the new model generation server device, and that is capable of extracting an object from a different background object image having at least one object in a background different from the background based on an existing object extraction model that has been trained to extract an object from an object image having at least one object in the background, the terminal program comprising:
an acquisition means for acquiring at least weight parameters of the recorded object extraction existing model;
a terminal-side transmitting means for transmitting the different background object image and at least weight parameters of the object-extraction existing model to the new model generation server device;
a terminal-side receiving means for receiving at least weight parameters of a new model for object extraction transmitted from a new model generation server device that has additionally learned at least weight parameters of the existing model for object extraction using learning data generated based on information of the object extracted from the different background object image by the existing model for object extraction;
a terminal program for causing the terminal to function as new model generation means for generating a new object extraction model by replacing the existing object extraction model with at least the received weight parameters,
The new model generation server device
an acquisition means for acquiring information on the different background object image and the object extracted by the extracted object acquisition means;
a non-background extraction means for extracting a non-background portion that is not a background from the different background object image by a background subtraction method;
a detection omission response means for processing the area in the different background object image as a featureless image and using the image as learning data if there is an area in the non-background portion where the object is not extracted by the extraction object acquisition means;
A terminal program comprising: