JP7775316B2 - Learning data collection device, learning data collection method, and program - Google Patents

Description

The present invention relates to a learning data collection device, a learning data collection method, and a program.

Image recognition technology is needed to identify targets in ports or at sea.

Japanese Patent Application Laid-Open No. 2021-077202

Realizing such image recognition technology requires a large amount of training data, but manually annotating each image requires a huge amount of effort.

The present invention has been made in consideration of the above-mentioned problems, and its main purpose is to provide a learning data collection device, a learning data collection method, and a program that make it easy to collect learning data.

To solve the above problem, one aspect of the present invention provides a learning data collection device that includes an image acquisition unit that acquires a first image captured by a camera installed on a ship and includes a target in a harbor or on the sea, and a second image that includes the target and has higher resolution than the first image; an image recognition unit that estimates the type of the target from the second image; and an association unit that associates the type of the target estimated from the second image with the first image. This facilitates the collection of learning data.

In the above aspect, the second image may be an image captured by the camera at a location closer to the target than the location at which the first image was captured. This makes it possible to associate the first image with the type of target estimated from the second image captured at a location closer to the target.

In the above aspect, the second image may be an image captured by enlarging an area including the target in the first image using the optical zoom function of the camera. This makes it possible to associate the type of target estimated from the second image captured by enlarging it using the optical zoom function with the first image.

In the above aspect, the second image may be an image of the area including the target in the first image captured by a different camera with a higher resolution than the first camera. This makes it possible to associate the type of target estimated from the second image captured by the different camera with the higher resolution.

In the above aspect, the image recognition unit may estimate the type of the target from the second image and calculate the accuracy of the estimation, and the associating unit may associate the type of the target for which the accuracy is equal to or greater than a threshold with the first image. This makes it possible to associate the type of the target estimated from the second image for which the accuracy is equal to or greater than a threshold with the first image.

In the above aspect, the image recognition unit may estimate the type of the target from the first image and calculate the accuracy of the estimation, and the associating unit may associate the type of the target estimated from the second image with the first image for which the accuracy is less than a threshold. This makes it possible to associate the type of the target estimated from the second image with the first image for which the accuracy of the estimation is less than the threshold.

In the above aspect, the image recognition unit may estimate the type of the target from the image captured by the camera and calculate the accuracy of the estimation, and designate an image with the accuracy equal to or greater than a threshold as the second image, and an image with the accuracy less than the threshold as the first image. This makes it possible to designate an image with an estimated accuracy less than the threshold as the first image, and an image with an estimated accuracy equal to or greater than the threshold as the second image.

The above aspect may further include a position and orientation acquisition unit that acquires the position and orientation of the ship, and an identification unit that identifies the target detected in the first image and the target detected in the second image based on the position and orientation of the ship at the time the first image was captured, the in-image position of the target detected in the first image, the position and orientation of the ship at the time the second image was captured, and the in-image position of the target detected in the second image. This makes it possible to identify the target detected in the first image and the target detected in the second image.

In the above aspect, the associating unit may further associate the first image with additional data that represents the state of the ship or its surroundings at the time the first image was captured. This makes it possible to further include additional data in the learning data.

Another aspect of the present invention is a learning data collection method that acquires a first image captured by a camera installed on a ship, the first image including a target in a harbor or on the sea, and a second image including the target and having a higher resolution than the first image, estimates the type of the target from the second image, and associates the type of the target estimated from the second image with the first image. This facilitates the collection of learning data.

In the above aspect, a trained model for estimating the type of target may be retrained using a dataset including the type of target estimated from the first image and the second image. This makes it possible to further improve the recognition accuracy of the trained model.

In another aspect of the present invention, the program causes a computer to acquire a first image captured by a camera installed on a ship, the first image including a target in a harbor or on the sea, and a second image including the target and having a higher resolution than the first image; estimate the type of the target from the second image; and associate the type of the target estimated from the second image with the first image. This facilitates the collection of learning data.

FIG. 1 illustrates an example of a data collection system. FIG. 1 illustrates an example of a data collection device. FIG. 10 is a diagram showing an example of a first image. FIG. 10 is a diagram showing an example of a second image. FIG. 10 is a diagram illustrating an example of an image capturing point. FIG. 10 is a diagram showing an example of a recognition result of a first image. FIG. 10 is a diagram showing an example of a recognition result of a second image. FIG. 10 is a diagram illustrating an example of a temporary storage database. FIG. 10 is a diagram illustrating an example of a training dataset. FIG. 10 is a diagram illustrating an example of a data collection method.

Below, an embodiment of the present invention is described with reference to the drawings.

Figure 1 is a block diagram showing an example configuration of a data collection system 100. The data collection system 100 is a system that is installed on a ship and collects learning data for machine learning. In the following description, the ship on which the data collection system 100 is installed is referred to as the "own ship."

The data collection system 100 comprises a data collection device 1, a display unit 2, a radar 3, an AIS 4, a camera 5, a GNSS receiver 6, a gyrocompass 7, an ECDIS 8, and a wireless communication unit 9. These devices are connected to a network N, such as a LAN, and are capable of network communication with each other.

The data collection device 1 is a computer that includes a CPU, RAM, ROM, non-volatile memory, an input/output interface, etc. The CPU of the data collection device 1 performs information processing according to a program loaded from the ROM or non-volatile memory to the RAM.

The program may be supplied via an information storage medium such as an optical disc or memory card, or via a communications network such as the Internet or a LAN.

The display unit 2 displays radar images, camera images, electronic nautical charts, etc. The display unit 2 also displays display images generated by the data collection device 1.

The display unit 2 is, for example, a display device with a touch sensor, a so-called touch panel. The touch sensor detects the position on the screen pointed to by the user's finger or the like. Alternatively, the pointed position may be input using a trackball or the like.

Radar 3 emits radio waves around the ship and receives reflected waves, generating echo data based on the received signals. Radar 3 also identifies targets from the echo data and generates target tracking data (TT data) that represents the target's position and speed.

The AIS (Automatic Identification System) 4 receives AIS data from other ships around the ship or from land-based control. Instead of AIS, VDES (VHF Data Exchange System) may also be used. AIS data includes the identification code, name, position, course, speed, type, length, and destination of other ships.

Camera 5 is a digital camera that captures images of the outside world from the ship and generates image data. Camera 5 is installed, for example, on the bridge of the ship, facing the bow. Camera 5 may also be a so-called PTZ camera, which has pan/tilt and optical zoom functions.

It is preferable that the camera 5 is, for example, a wide-angle camera. Furthermore, the camera 5 is not limited to a visible light camera, and may also be an infrared camera.

The GNSS receiver 6 detects the ship's position based on radio waves received from the GNSS (Global Navigation Satellite System). The gyrocompass 7 detects the ship's heading. A GPS compass may also be used, not limited to a gyrocompass.

The ECDIS (Electronic Chart Display and Information System) 8 acquires the ship's position from the GNSS receiver 6 and displays the ship's position on an electronic chart. The ECDIS 8 also displays the ship's planned route on the electronic chart. A GNSS plotter may also be used instead of an ECDIS.

The wireless communication unit 9 includes various types of radio equipment, such as ultra-short wave, medium wave, and short wave band radio equipment, for enabling communication with other ships or land-based control.

In this embodiment, the data collection device 1 and the display unit 2 are independent devices, but this is not limited to this, and the data collection device 1 and the display unit 2 may also be an integrated device.

In this embodiment, the data collection device 1 is an independent device, but this is not limited to this and it may be integrated with another device such as ECDIS 8. In other words, the functional parts of the data collection device 1 may be realized by another device.

The display unit 2 is also an independent device, but this is not limited to this, and the display unit of another device such as ECDIS 8 may be used as the display unit 2 that displays the display image generated by the data collection device 1.

Figure 2 is a block diagram showing an example configuration of the data collection device 1. The data collection device 1 includes an image acquisition unit 11, a position and orientation acquisition unit 12, an image recognition unit 13, an identification unit 14, an association unit 15, a model holding unit 16, a temporary storage unit 17, and a data storage unit 18.

The functional units 11 to 15 included in the data collection device 1 are realized by the control unit 10 (processing circuitry) of the data collection device 1 executing information processing in accordance with a program. The memory units 16 to 18 included in the data collection device 1 are secured in the memory of the data collection device 1.

The image acquisition unit 11 acquires images captured by the camera 5. The image acquisition unit 11 sequentially acquires multiple images in time series from the camera 5 and sequentially provides them to the image recognition unit 13. The multiple images in time series are, for example, multiple still images (frames) included in video image data.

The images acquired by the image acquisition unit 11 are images captured by the camera 5 when the ship is navigating a port or other waters, and include landmarks in the port or at sea. Port landmarks include, for example, installations for handling cargo, such as cranes, or installations for mooring, such as quays. At-sea landmarks include, for example, ships and buoys.

Specifically, the image acquisition unit 11 acquires a first image P1 that includes a target object in a harbor or on the sea, and a second image P2 that includes the installation and has a higher resolution than the first image P1. In this embodiment, the second image P2 is an image captured by the camera 5 at a point closer to the target object than the point at which the first image P1 was captured.

Figures 3 and 4 are diagrams showing examples of the first image P1 and the second image P2. Figure 5 is a diagram showing examples of image capture locations. K1 and D1 represent the position and orientation of the ship SS at the time the first image P1 was captured, and K2 and D2 represent the position and orientation of the ship SS at the time the second image P2 was captured.

In the example shown in Figure 3, three cranes C1 to C3 are included in the center of the first image P1, appearing relatively small. In contrast, in the example shown in Figure 4, two cranes C1 and C2 are included relatively large across the entire second image P2. Note that in the example shown in Figure 4, crane C3 is outside the field of view of camera 5 and is not included in the second image P2.

As such, the second image P2 has a higher resolution than the first image P1, making it easier to identify the cranes C1 and C2. This is because, as shown in the example of Figure 5, the first image P1 was captured at position K1, which is relatively far from the quay PQ where the cranes C1 to C3 are installed, while the second image P2 was captured at position K2, which is relatively close to the quay PQ.

In other words, the first image P1 is an image in which the subsequent image recognition unit 13 can detect targets, but cannot classify them. On the other hand, the second image P2 is an image in which the image recognition unit 13 can detect targets and also classify them. Being able to classify means, for example, that the accuracy of estimating the type is at a sufficient level.

Therefore, whether an image is the first image P1 or the second image P2 may be determined by the subsequent image recognition unit 13. That is, as a result of classification by the image recognition unit 13, if the accuracy of the estimation is less than a threshold, the image may be determined to be the first image P1, and if the accuracy of the estimation is equal to or greater than the threshold, the image may be determined to be the second image P2.

Without being limited to this, the second image P2 may be, for example, an image captured by enlarging the range of the cranes C1 and C2 detected in the first image P1 using the optical zoom function of the camera 5, or an image captured by another camera with higher resolution than the camera 5.

The second image P2 may be, for example, a high-resolution image of the area of the cranes C1 and C2 detected in the first image P1. Furthermore, the first image P1 may be an infrared image captured by an infrared camera, and the second image P2 may be a visible light image captured by a visible light camera.

The position and orientation acquisition unit 12 shown in Figure 2 acquires the position and orientation of the ship at the time the image was captured by the camera 5 and associates it with the image acquired by the image acquisition unit 11. The position of the ship is the position of the ship detected by the GNSS receiver 6.

The heading of the ship is the ship's bow direction detected by the gyrocompass 7. In addition, the heading of the ship may include not only the ship's bow direction but also the ship's roll or pitch detected by an attitude sensor (not shown).

The image recognition unit 13 uses the trained model stored in the model storage unit 16 to estimate the type of target included in the image provided by the image acquisition unit 11. Specifically, the image recognition unit 13 detects the position within the image of the target included in the image, estimates the type of the target, and calculates the accuracy of the estimation.

The trained model is generated in advance through machine learning using training images as input data and the in-image positions and types of targets contained in the training images as training data. The trained model generated in this way outputs the in-image positions of targets contained in the images, the types of targets, and the accuracy of the estimation.

The trained model may be an object detection model such as SSD (Single Shot MultiBox Detector) or YOLO (You Only Look Once). Alternatively, the trained model may be a segmentation model such as Semantic Segmentation or Instance Segmentation.

Figures 6 and 7 are figures showing examples of recognition results of the first image P1 and the second image P2 by the image recognition unit 13.

The cranes C1 to C3 included in the first image P1 are surrounded by bounding boxes B1 to B3, and labels L1 to L3 describing the estimated type and its accuracy are attached to the bounding boxes B1 to B3.

Similarly, the cranes C1 and C2 included in the second image P2 are surrounded by bounding boxes B1 and B2, and labels L1 and L2 describing the estimated type and its accuracy are added to the bounding boxes B1 and B2.

The first image P1 has a relatively low resolution, so the accuracy of the estimation is relatively low. In the example of Figure 6, the estimated types of cranes C1 and C3 are incorrect and the accuracy of the estimation is low. For crane C2, the estimated type is correct, but the accuracy of the estimation is low.

On the other hand, the second image P2 has a relatively high resolution, so the accuracy of the estimation is relatively high. In the example of Figure 7, the estimated types of cranes C1 and C2 are correct and the accuracy of the estimation is also high.

The temporary storage unit 17 shown in Figure 2 temporarily stores the first image P1. In this embodiment, among the images acquired by the image acquisition unit 11, an image including a target whose estimation accuracy is less than a threshold as a result of recognition by the image recognition unit 13 is stored in the temporary storage unit 17 as the first image P1.

Figure 8 is a diagram showing an example of a temporary storage database for managing the first image P1 stored in the temporary storage unit 17. The temporary storage database includes fields such as "target ID," "image," "position in image," "type," "accuracy," "image capture position," "image capture direction," "estimated position," and "accompanying data."

"Target ID" is an identifier for identifying the target. "Image" represents the file name of the first image P1. In this example, target IDs 001 to 003 correspond to cranes C1 to C3 shown in the example of Figure 6.

"Image position" represents the image position of the target detected in the first image P1. The image position is expressed, for example, by the coordinates of the upper left and lower right corner points of the bounding box surrounding the target.

"Type" indicates the type of target estimated by the image recognition unit 13. In this example, the type of target ID: 001, 003 is mistakenly estimated as a radio tower or bridge rather than a crane. "Accuracy" indicates the accuracy of the estimation. Accuracy is expressed, for example, as a value between 0 and 1, with the closer to 1 the accuracy is.

"Imaging position" represents the position of the ship at the time the first image P1 was captured. The ship's position is expressed, for example, by latitude and longitude. "Imaging direction" represents the imaging direction of camera 5 at the time the first image P1 was captured. The imaging direction of camera 5 corresponds to the ship's bow direction.

"Estimated position" refers to the estimated position of the target calculated based on the position in the image, the imaging position, the imaging direction, etc. Other parameters used to calculate the estimated position include the angle of view and resolution of the camera 5.

Specifically, the estimated position is first calculated as a relative position to the ship from the position in the image and the imaging direction, and then converted to an absolute position using the imaging position. The estimated position is expressed, for example, by latitude and longitude, just like the imaging position.

"Additional data" is data that represents the state of the ship or its surroundings at the time the first image P1 was captured. The additional data includes data such as weather conditions, such as fog or rain, or area attributes, such as a port or perspective.

The identification unit 14 shown in Figure 2 identifies targets detected in the first image P1 and targets detected in the second image P2. In this embodiment, of the images acquired by the image acquisition unit 11, an image in which the estimation accuracy of all detected targets is equal to or greater than a threshold as a result of recognition by the image recognition unit 13 is provided to the identification unit 14 as the second image P2.

When the identification unit 14 receives the second image P2, it extracts the first image P1 that contains the same target as the target detected in the second image P2 from the first images P1 stored in the temporary memory unit 17.

The identification unit 14 determines that the two targets are the same when the estimated position of the target detected in the first image P1 is the same as or similar to the estimated position of the target detected in the second image P2. As described above, the estimated position of the target is calculated based on the position in the image, the image capture position, the image capture direction, etc.

The association unit 15 associates the first image P1 with the target type estimated from the second image P2, for which the accuracy of the estimation is equal to or greater than a threshold. The association unit 15 then stores a dataset including the first image P1 and the target type associated therewith in the data storage unit 18 as a learning dataset for machine learning.

Specifically, the associating unit 15 associates the type of target estimated from the second image P2 by the image recognition unit 13 with the first image P1, which includes the same target as in the second image P2 and which has been extracted from the temporary storage unit 17 by the identification unit 14. The associating unit 15 may also associate additional data with the first image P1.

Figure 9 is a diagram showing an example of a training dataset stored in the data storage unit 18. The training dataset includes a first image P1, the in-image position and type of a target contained in the first image P1, and ancillary data.

The target type is the type of target estimated from the second image P2. That is, the target type estimated from the first image P1 with an estimation accuracy below the threshold is replaced with the target type estimated from the second image P2 with an estimation accuracy above the threshold.

This makes it possible to associate the first image P1, in which the target cannot be classified, with a highly reliable classification estimated from the second image P2, thereby obtaining an appropriate learning dataset.

In the example of Figure 8 above, the type of some of the objects in image IMG01 was mistakenly estimated to be a radio tower or a bridge rather than a crane, but in the example of Figure 9, the type of objects estimated from the second image P2 was applied, and as a result, all of the types in image IMG01 were determined to be cranes.

Returning to the explanation of Figure 2, the learning dataset stored in the data storage unit 19 is used to retrain the trained model by the learning device 200. The learning device 200 is, for example, one or more server computers installed on land.

When communication with the learning device 200 is established, for example, when the data collection device 1 calls at port, it transfers the learning dataset stored in the data storage unit 18 to the learning device 200. Furthermore, the data collection device 1 deletes the learning dataset that has been transferred from the data storage unit 18 to free up space.

The learning device 200 re-learns the trained model using the training dataset acquired from the data collection device 1. Specifically, the learning device 200 re-learns using the images included in the acquired training dataset as input data and the target's position in the image, type, and accompanying data as training data.

When a new version of the trained model is prepared in the learning device 200, the data collection device 1 obtains the new version of the trained model from the learning device 200 and replaces the old version of the trained model stored in the model storage unit 16.

In this way, by repeating the cycle of retraining the trained model using the training dataset collected by the data collection device 1 and then collecting training datasets using the retrained trained model by the data collection device 1, continuous improvement of the trained model is possible.

Figure 10 is a diagram showing an example of the procedure of a data collection method implemented in the data collection device 1. The CPU of the data collection device 1 executes the information processing shown in the figure in accordance with a program.

First, the data collection device 1 acquires an image captured by the camera 5 (S11, processing as the image acquisition unit 11). Next, the data collection device 1 acquires the position and attitude of the ship itself and associates them with the image (S12-S13, processing as the position and attitude acquisition unit 12).

Next, the data collection device 1 performs image recognition (S14, processing by the image recognition unit 13). Specifically, the data collection device 1 uses the trained model to detect the position within the image of targets contained in the image, estimate the type of target, and calculate the accuracy of the estimation.

Next, the data collection device 1 designates the image containing the target whose estimation accuracy is less than the threshold as the first image P1 (S15: YES), registers the first image P1 and information related to the target in a temporary memory database (see Figure 8), and terminates the processing.

On the other hand, the data collection device 1 designates the image in which the estimation accuracy of all targets is above the threshold as the second image (S15: NO), and by referring to the temporary memory database, determines whether there is a first image P1 that includes the same targets as the second image P2 (S17, S18, processing as the identification unit 14).

If a first image P1 exists that includes the same target as the second image P2 (S18: YES), the data collection device 1 associates the type of target estimated from the second image P2 with the type of target whose accuracy is equal to or greater than a threshold value with the first image P1 and saves it (S19, processing as the association unit 15), and terminates the processing.

The above describes an embodiment of the present invention, but the present invention is not limited to the embodiment described above, and various modifications are possible for those skilled in the art.

For example, a camera other than camera 5 may be installed on the ship, and the image captured by the other camera may be designated as first image P1, the image captured by camera 5 may be designated as second image P2, and the identification result of second image P2 may be associated with first image P1. This makes it possible to collect a training dataset for the other camera. In this case, it is preferable to correct the position within the image according to the parallax based on the differences in position, resolution, and angle of view between camera 5 and the other camera.

1 Data collection device, 2 Display unit, 3 Radar, 4 AIS, 5 Camera, 6 GNSS receiver, 7 Gyrocompass, 8 ECDIS, 9 Wireless communication unit, 11 Image acquisition unit, 12 Position and attitude acquisition unit, 13 Image recognition unit, 14 Identification unit, 15 Association unit, 16 Model holding unit, 17 Temporary storage unit, 18 Data storage unit, 100 Data collection system, 200 Learning device

Claims (12)

an image acquisition unit that acquires a first image including a target in a harbor or on the sea, both of which are captured by a camera installed on the ship, and a second image including the target and having a higher resolution than the first image;
an image recognition unit that estimates the type of the target from the second image;
an associating unit that associates the type of the target estimated from the second image with the first image;
A learning data collection device comprising: The second image is an image captured by the camera at a point closer to the target than the point at which the first image was captured.
The learning data collection device according to claim 1 . The second image is an image captured by enlarging an area including the target in the first image using an optical zoom function of the camera.
The learning data collection device according to claim 1 . The second image is an image of a range including the target in the first image captured by another camera having a higher resolution than the camera.
The learning data collection device according to claim 1 . the image recognition unit estimates the type of the target from the second image and calculates the accuracy of the estimation;
The associating unit associates the type of the target whose certainty is equal to or greater than a threshold with the first image.
5. The learning data collection device according to claim 1. the image recognition unit estimates the type of the target from the first image and calculates an accuracy of the estimation;
the associating unit associates the type of the target estimated from the second image with the first image for which the accuracy is less than a threshold value;
6. The learning data collection device according to claim 1. the image recognition unit estimates the type of the target from the image captured by the camera and calculates an accuracy of the estimation, and determines an image whose accuracy is equal to or greater than a threshold as the second image, and determines an image whose accuracy is less than the threshold as the first image;
7. The learning data collection device according to claim 1. a position and attitude acquisition unit that acquires the position and orientation of the ship;
an identification unit that identifies the target detected in the first image and the target detected in the second image based on the position and orientation of the ship at the time the first image was captured, the position within the image of the target detected in the first image, the position and orientation of the ship at the time the second image was captured, and the position within the image of the target detected in the second image;
Further provided with
8. The learning data collection device according to claim 1. The associating unit further associates, with the first image, incidental data that represents a state of the ship or its surroundings at the time the first image was captured.
9. The learning data collection device according to claim 1. acquiring a first image including a target in a port or on the sea, both of which are captured by a camera installed on a ship, and a second image including the target and having a higher resolution than the first image;
Estimating the type of the target from the second image;
Associating the type of the target estimated from the second image with the first image;
Methods for collecting data for training. further retraining a trained model for estimating the type of the target using a dataset including the type of the target estimated from the first image and the second image;
The learning data collection method according to claim 10. acquiring a first image including a target object at a port or on the sea, both of which are captured by a camera installed on a ship, and a second image including the target object and having a higher resolution than the first image;
Estimating a type of the target from the second image; and
Associating the type of the target estimated from the second image with the first image;
A program that causes a computer to execute the following.