JP7832141B2 - Information processing device, information processing method, and program - Google Patents

Description

This invention relates to an information processing device, an information processing method, and a program.

A technique for annotating images using machine learning to detect desired objects is known (see, for example, Patent Document 1).

Japanese Patent Publication No. 2019-46094

However, conventional techniques lacked sufficient accuracy in segmenting the object's region, resulting in ineffective methods of annotating the detected object onto the image.

This invention takes these circumstances into consideration and aims to provide an information processing device, an information processing method, and a program that can effectively perform annotation by accurately segmenting the region of an object on an image.

One aspect of the present invention is an information processing apparatus comprising: a detection unit for detecting the region of an object on an image of the ground surface captured from above; an estimation unit for estimating the orientation of the region; and an image processing unit for shaping the region based on the orientation.

According to one aspect of the present invention, annotation can be effectively performed by accurately segmenting the region of an object on an image.

This figure shows an example of the configuration of the information processing device 100 according to the embodiment. This flowchart shows the sequence of processing steps of the processing unit 110 according to the embodiment. This figure illustrates a method for detecting the area of a pedestrian crossing using the first machine learning model MDL1. This figure illustrates a method for estimating various orientations using the second machine learning model, MDL2. This is a diagram illustrating how to shape the area of a pedestrian crossing. This figure shows an example of annotated satellite imagery. This figure shows an example of the hardware configuration of the information processing device 100 in this embodiment.

The following describes embodiments of the information processing apparatus, information processing method, and program of the present invention with reference to the drawings.

[overview]
One aspect of this embodiment of the information processing device detects the region of an object on an image of the ground surface taken from above, and estimates the orientation of the region of the object.

Images of the Earth's surface taken from above include, for example, satellite images (satellite photographs), aerial images (aerial photographs), and aerial photographs (aerial photographs).

The objects detected on an image can be arbitrarily determined depending on the type of service or application provided to the user. For example, when providing users with satellite or aerial imagery as map data, anything visible in those images can be considered an object. More specifically, when navigating a user to a destination using map data, traffic control structures such as pedestrian crossings, traffic lights, school zones, and railway crossings along the route to the destination may be selected as objects. The following explanation will use "pedestrian crossings" as an example object.

Furthermore, when map data such as satellite imagery and aerial imagery is used for disaster prevention against natural disasters such as landslides and river flooding, objects that could potentially cause these disasters (for example, solar panels installed after forests have been cleared) may be selected as targets.

The information processing device detects the region of an object (e.g., a pedestrian crossing) in an image and estimates its orientation. Based on the estimated orientation, it then reshapes the region of the object. This reshaping of the object's region improves the accuracy of segmentation. As a result, when adding annotations to satellite imagery, such as pedestrian crossings, the annotation process becomes more effective.

[Information Processing Device]
Figure 1 is a diagram showing an example of the configuration of an information processing device 100 according to the embodiment. For example, the information processing device 100 is a web server or application server that sends and receives information with a user's terminal device.

The user's terminal device is a computer device with communication and display functions, such as a smartphone, personal computer, or tablet terminal. Specifically, the terminal device includes a communication interface for communicating with external devices such as the content provision device 50 via a network NW, a display, and a GNSS (Global Navigation Satellite System) receiver. The network NW includes the Internet, LAN (Local Area Network), WAN (Wide Area Network), and cellular networks.

The communication interface includes, for example, a network card such as a NIC (Network Interface Card) and a wireless communication module. The display includes, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display. These displays show a GUI (Graphical User Interface) for accepting various user input operations. The GNSS receiver determines the location of the terminal device.

Furthermore, the terminal device is equipped with a processor such as a CPU (Central Processing Unit), and the CPU executes a User Agent (UA), which then displays various content on the screen.

The information processing device 100, when a web browser or application is launched as a user agent (UA) on the user's terminal device and a request is sent from that UA, provides various content to the terminal device as a response to that request. This content includes, for example, satellite images with annotations indicating objects such as pedestrian crossings.

As shown in the figure, the information processing device 100 comprises a communication unit 102, a processing unit 110, and a storage unit 130.

The communication unit 102 is, for example, a communication interface such as a network card for connecting to a network NW.

The processing unit 110 comprises an acquisition unit 112, a region detection unit 114, an orientation estimation unit 116, an image processing unit 118, a communication control unit 120, and a learning unit 122. These components are realized, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of the components of the processing unit 110 may be realized by hardware (including circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), and GPU (Graphics Processing Unit), or by the collaboration of software and hardware.

The storage unit 130 is implemented using, for example, a storage device such as an HDD (Hard Disc Drive), flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), ROM (Read Only Memory), or RAM (Random Access Memory). The storage unit 130 stores firmware and application programs executed by the processor. The programs may be pre-stored in the storage unit 130, or they may be stored on a removable storage medium (non-transient storage medium) such as a DVD or CD-ROM, and installed in the storage unit 130 when the storage medium is inserted into the drive device.

The memory unit 130 also stores map data 132, which includes satellite and aerial images, and a training dataset 134. The map data 132 includes various images of the Earth's surface taken from above, such as satellite and aerial images. The training dataset 134 is a dataset (a combination of input data and correct output data) prepared for training the machine learning model described later.

[Processing flow of information processing equipment]
The processing details of the processing unit 110 will be explained below using a flowchart. Figure 2 is a flowchart showing the sequence of processing steps of the processing unit 110 according to this embodiment. The processing steps in this flowchart may be executed repeatedly, for example, at a predetermined cycle.

First, the acquisition unit 112 acquires the satellite image IMG1 (step S100). For example, if the satellite image IMG1 is stored as map data 132 in the storage unit 130, the acquisition unit 112 may read and acquire the satellite image IMG1 from the storage unit 130. Alternatively, the acquisition unit 112 may acquire the satellite image IMG1 from an external server (e.g., a data source server) via the communication unit 102. Furthermore, if a non-transient storage medium (e.g., fresh memory) containing the satellite image IMG1 is connected to the drive device of the information processing device 100, the acquisition unit 112 may read the satellite image IMG1 from the storage medium. As described above, the acquisition unit 112 may acquire aerial images or other images instead of satellite images. The following explanation assumes that the image to be acquired is the satellite image IMG1.

Next, the region detection unit 114 detects the region of a pedestrian crossing, which is an example of an object, on the satellite image IMG1 (which may also be an aerial or aerial image) acquired by the acquisition unit 112 (step S102).

For example, the region detection unit 114 uses the first machine learning model MDL1, which is one of the segmentation models, to detect the area of a pedestrian crossing on the satellite image IMG1 (performing segmentation).

The first machine learning model, MDL1, may be a segmentation model implemented using a neural network such as a CNN (Convolutional Neural Network(s)).

The first machine learning model, MDL1, is trained based on a first training dataset in which a given satellite image is associated with the regions of the correct objects (e.g., crosswalks) that should be detected on that satellite image. If the object is a crosswalk, its region will be a quadrilateral with four vertices.

Figure 3 illustrates a method for detecting a pedestrian crossing area using the first machine learning model MDL1. The area detection unit 114 inputs the satellite image IMG1 acquired by the acquisition unit 112 into the first machine learning model MDL1. As described above, the first machine learning model MDL1 is trained based on the first training dataset. Therefore, upon receiving the satellite image IMG1, the first machine learning model MDL1 outputs the location and size of the pedestrian crossing area. Furthermore, the first machine learning model MDL1 may output a score (hereinafter referred to as the confidence score) representing the likelihood, or reliability, of the location and size of the area.

Returning to the flowchart explanation, the image processing unit 118 then crops (extracts) the area of the traverse from the satellite image IMG1, and generates a cropped image IMG2 from the cropped portion of the satellite image IMG1 (step S104).

When detecting the area of a pedestrian crossing using the first machine learning model MDL1, the area of the pedestrian crossing is often detected as a quadrilateral, but it may also be detected as a pentagon with five vertices or a polygon with more than five vertices.

If the area of a pedestrian crossing is detected as a polygonal area with five or more vertices, the image processing unit 118 may approximate the polygonal area of the pedestrian crossing to a quadrilateral with four vertices (or a pentagon with five vertices) based on the following conditions.
(1) First, approximate the area of the crosswalk with a quadrilateral polygon having four vertices.
(2) If the difference between the area of the approximated quadrilateral polygon and the area of the original crosswalk is less than 25% of the area of the original crosswalk, or if the area of the original crosswalk is less than 128 when the area of one pixel is set to 1, the original crosswalk area will be used as is without approximation.
(3) Next, the area of the pedestrian crossing, which is approximated by the quadrilateral polygon, is further approximated by a pentagonal polygon with 5 vertices.

Next, the image processing unit 118 determines whether the aspect ratio of the cropped image IMG2 is within an acceptable range (step S106). For example, if the cropped image IMG2 input to the second machine learning model MDL2 (described later) is assumed to be square (i.e., if a second training dataset containing square images is used during training), the acceptable range for the aspect ratio may be, for example, the same as the aspect ratio of a square image (1:1), or one that allows for an error of a few percent in the aspect ratio of a square image.

If the aspect ratio of the cropped image IMG2 is outside the acceptable range (i.e., the cropped image IMG2 is not square), the image processing unit 118 pads the cropped image IMG2 so that the aspect ratio is within the acceptable range (step S108).

For example, the image processing unit 118, as padding, adds an arbitrary pixel value (for example, a pixel value of the same color as the road surface) to one or all of the top, bottom, left, or right sides of the matrix representing the cropped image IMG2, thereby making the shape of the cropped image IMG2 closer to a square.

On the other hand, if the aspect ratio of the cropped image IMG2 is within an acceptable range (i.e., if the cropped image IMG2 is square), the image processing unit 118 omits the padding process in S118.

Generally, the shape of pedestrian crossings is determined according to the road and surrounding traffic conditions, and pedestrian crossings can sometimes be extremely elongated vertically or horizontally. When cropped images of pedestrian crossings of various shapes (IMG2) are input into the second machine learning model (MDL2), the orientations estimated by MDL2 (details below) tend to deviate from the orientations assumed during training. For example, if the orientations assumed during training were 0 degrees, 90 degrees, and 180 degrees, angles other than these are more likely to be estimated by MDL2.

In contrast, by adjusting the aspect ratio of the cropped image IMG2 input to the second machine learning model MDL2 (described later) using padding, the accuracy of the output results of the second machine learning model MDL2 can be improved. For example, by inputting the cropped image IMG2 into the second machine learning model MDL2 after unifying the aspect ratio, the second machine learning model MDL2 becomes more likely to output the orientation (0 degrees, 90 degrees, 180 degrees) that was assumed during training.

Furthermore, if the cropped image IMG2 input to the second machine learning model MDL2 is assumed to be rectangular rather than square (i.e., if a second training dataset containing rectangular images is used during training), the acceptable range of aspect ratios compared in the judgment process of S106 may be determined according to the aspect ratio of a rectangle, rather than a square.

Next, the orientation estimation unit 116 estimates the orientation V1 of the pedestrian crossing included in the cropped image IMG2 and the orientation V2 of the white line drawn on the road as the pedestrian crossing (step S110).

The direction V1 of the crosswalk is the direction in which pedestrians cross the crosswalk, and it intersects with the longitudinal direction of the white line (the direction in which the white line extends). The direction V2 of the white line is the longitudinal direction of the white line (the direction in which the white line extends). The direction V1 of the crosswalk is an example of the "first direction," and the direction V2 of the white line is an example of the "second direction."

For example, the orientation estimation unit 116 uses a second machine learning model MDL2, implemented using a neural network such as a CNN (Convolutional Neural Network(s)), to estimate the orientation V1 of the crosswalk and the orientation V2 of the white line on the cropped image IMG2.

The second machine learning model, MDL2, is trained on a second training dataset in which the orientation of the crosswalk (V1) and the orientation of the white line (V2) are associated with the orientation of the ground truth object region to be estimated on a cropped image used for training. The cropped image used for training is an image extracted from a satellite image containing the crosswalk region, and is, for example, a square image as described above.

Figure 4 illustrates the method for estimating various orientations using the second machine learning model MDL2. The orientation estimation unit 116 inputs the cropped image IMG2, generated by the image processing unit 118, into the second machine learning model MDL2. As described above, the second machine learning model MDL2 is trained based on the second training dataset. Therefore, upon receiving the cropped image IMG2, the second machine learning model MDL2 outputs the orientation of the crosswalk V1 and the orientation of the white line V2 as the orientation of the object's region.

Next, the image processing unit 118 reshapes the area of the pedestrian crossing detected by the area detection unit 114 based on the direction V1 of the pedestrian crossing and the direction V2 of the white line estimated by the direction estimation unit 116 (step S112).

Figure 5 illustrates a method for shaping the area of a pedestrian crossing. (a) in the figure represents the cropped image IMG2, and R on the cropped image IMG2 represents the area of the pedestrian crossing detected by the area detection unit 114. Although the cropped image IMG2 is shown as a rectangle for convenience, it may actually be a square. Furthermore, even if the cropped image IMG2 is rectangular, it may be converted to a square by padding as described above.

Figure (b) indicates that the orientation of the pedestrian crossing V1 and the orientation of the white line V2 are estimated on the cropped image IMG2 of (a). As shown in the figure, the detected pedestrian crossing region R may not completely mask the actual pedestrian crossing region. This is because the segmentation model is model-based using the first machine learning model MDL1. Therefore, the image processing unit 118 reshapes the detected pedestrian crossing region R using the first machine learning model MDL1 to completely mask the actual pedestrian crossing region, based on the orientation of the pedestrian crossing V1 and the orientation of the white line V2.

As shown in (c), the image processing unit 118 first divides the four sides of the rectangle detected as the crosswalk area R into two pairs, with opposite sides forming a pair. Specifically, the image processing unit 118 classifies the four sides of the rectangle detected as the crosswalk area R into two sets: two sides that are close to the angle of the crosswalk direction V1 (smaller angle difference with V1 and more parallel) and two sides that are close to the angle of the white line direction V2 (smaller angle difference with V2 and more parallel). The two sides that are close to the angle of the crosswalk direction V1 (smaller angle difference with V1 and more parallel) are an example of the "first pair of opposite sides," and the two sides that are close to the angle of the white line direction V2 (smaller angle difference with V2 and more parallel) are an example of the "second pair of opposite sides."

As shown in (d), the image processing unit 118 then rotates two sides that are close in angle to the direction V1 of the crosswalk (i.e., the angle difference with V1 is small and they are more parallel) so that they approach the direction V1 of the crosswalk. Similarly, the image processing unit 118 rotates two sides that are close in angle to the direction V2 of the white line (i.e., the angle difference with V2 is small and they are more parallel) so that they approach the direction V2 of the white line. In this case, the image processing unit 118 may rotate the sides around their midpoints.

As shown in (e), the image processing unit 118 defines the region enclosed by the four sides with corrected angles (i.e., the reshaped region) as the new crosswalk region R#. In this case, if the area of the new crosswalk region R# is 50% or less of the area of the crosswalk region R before reshaping, the image processing unit 118 will not adopt the new crosswalk region R# and will instead adopt the crosswalk region R before reshaping. Furthermore, even if the overlapping portion between the new crosswalk region R# and the crosswalk region R before reshaping is 50% or less of the area of the crosswalk region R before reshaping, the image processing unit 118 may still adopt the crosswalk region R before reshaping instead of the new crosswalk region R#.

Through this series of processes, the area R of the pedestrian crossing is shaped, and the processing in this flowchart is completed. When the communication control unit 120 receives a request for map data transmitted from the user's terminal device via the communication unit 102, it may, as a response to that request, transmit a satellite image to the user's terminal device via the communication unit 102, with the shaped pedestrian crossing area R# (or the unshaped pedestrian crossing area R if the area conditions are not met) annotated.

Figure 6 shows an example of annotated satellite imagery. In the figure, AN represents the annotation, which overlays the reshaped crosswalk area R# onto the crosswalk in the satellite image. This display of annotation AN allows users to more realistically perceive the location and shape of objects such as crosswalks, enabling them to use navigation and other services without feeling any discomfort or unnaturalness.

[Training machine learning models]
The training methods for the first machine learning model MDL1 and the second machine learning model MDL2 described above are explained below. The learning unit 122 trains the first machine learning model MDL1 using the first training dataset described above, and trains the second machine learning model MDL2 using the second training dataset described above. When the learning unit 122 trains the second machine learning model MDL2, it sets the loss function as follows. The term "direction" mentioned above can be treated as synonymous with "angle". The numerical values output by the second machine learning model MDL2 as angles V1 and V2 are in the range of 0 to 1, and this numerical range of 0 to 1 corresponds to the angle range of 0 degrees to 180 degrees.

(i) The angle output by the second machine learning model MDL2 (hereinafter referred to as the estimated angle) differs from the correct angle included in the second training dataset by 90 degrees (0.5) or more.
(ii) If both the estimated angle and the correct angle are less than 18 degrees (0.1) or 162 degrees (0.9) or greater,
(iii) The correct angle is corrected to be closer to the estimated angle (closer to 0 degrees or 180 degrees), and then the loss function is optimized using BCELoss (Binary Cross Entropy Loss).

When the angles V1 and V2 output (estimated) by the second machine learning model MDL2 in response to the input cropped image IMG2 are approximately 0 degrees or approximately 180 degrees, the estimated angle and the correct angle may be in opposite directions. For example, if the correct angle is approximately 179.8 degrees (equivalent to approximately 0.999 in the output values of the second machine learning model MDL2), the estimated angle may be approximately 0 degrees (equivalent to approximately 0.0001 in the output values of the second machine learning model MDL2).

Although nearly 180 degrees and nearly 0 degrees are practically the same angle, it is best to consider the estimated angle, which is the output value of the second machine learning model MDL2, as correct even if it is a different value from nearly 0 (e.g., around 0.0001) or nearly 1 (e.g., around 0.999).

Therefore, the second machine learning model MDL2 is trained based on the loss function that includes the exception conditions (i) to (iii) above. This improves the training efficiency and accuracy of the second machine learning model MDL2.

According to the embodiments described above, the information processing device 100 detects the area of an object (e.g., a pedestrian crossing) on an image of the Earth's surface taken from above, such as satellite imagery, aerial imagery, or aerial photograph, and estimates the orientation of the object's area. The information processing device 100 then reshapes the object's area based on the estimated orientation. By reshaping the object's area in this way, the accuracy of segmentation is improved. As a result, when adding annotations to satellite imagery or other data containing objects such as pedestrian crossings, the annotation can be performed more effectively.

<Hardware Configuration>
The information processing device 100 of the above-described embodiment can be realized by a hardware configuration such as that shown in Figure 7. Figure 7 is a diagram showing an example of the hardware configuration of the information processing device 100 of the embodiment.

The information processing device 100 consists of a NIC 100-1, a CPU 100-2, a RAM 100-3, a ROM 100-4, a secondary storage device 100-5 such as flash memory or an HDD, and a drive device 100-6, all interconnected via an internal bus or dedicated communication lines. A portable storage medium, such as an optical disc, is mounted in the drive device 100-6. A program stored in the secondary storage device 100-5 or the portable storage medium mounted in the drive device 100-6 is loaded into the RAM 100-3 by a DMA controller (not shown), and executed by the CPU 100-2 to realize the processing unit 110. The program referenced by the CPU 100-2 may be downloaded from another device via a network NW.

Although embodiments for carrying out the present invention have been described above using examples, the present invention is not limited in any way to these embodiments, and various modifications and substitutions can be made without departing from the spirit of the invention.

100...Information processing unit, 102...Communication unit, 110...Processing unit, 112...Acquisition unit, 114...Region detection unit, 116...Orientation estimation unit, 118...Image processing unit, 120...Communication control unit, 122...Learning unit

Claims (7)

A detection unit that detects the area of an object on an image of the ground surface taken from above,
An estimation unit for estimating the orientation of the said region,
An image processing unit that shapes the region based on the orientation,
A communication unit that communicates with a terminal device that displays the aforementioned image upon receiving a request from a user,
The system includes a communication control unit that transmits an annotation image, which is the image to which the shaped region of the object has been annotated, to the terminal device via the communication unit,
The annotation image is displayed on the terminal device as a response to the request.
Information processing device. The aforementioned object is a pedestrian crossing,
The detection unit detects the area of the pedestrian crossing on the image,
The estimation unit estimates a first direction, which is the direction in which a pedestrian crosses the crosswalk, and a second direction, which is the longitudinal direction of the white line drawn on the road as the crosswalk.
The image processing unit shapes the area of the pedestrian crossing based on the first direction and the second direction.
The information processing apparatus according to claim 1. The area of the aforementioned pedestrian crossing is a quadrilateral area consisting of four sides.
The image processing unit reshapes the quadrilateral region such that the first pair of opposite sides of the four sides is parallel to the first direction, and the second pair of opposite sides of the four sides is parallel to the second direction.
The information processing apparatus according to claim 2. The detection unit uses a first machine learning model to detect the region of the object on the image.
The first machine learning model is a machine learning model that has been trained on a first training dataset in which a training target image is associated with the region of the correct object to be detected on the training target image.
The information processing apparatus according to claim 1 or 2. The estimation unit uses a second machine learning model to estimate the orientation of the detected region on a cropped image, which is an image extracted from the image.
The second machine learning model is a machine learning model trained on a second training dataset in which the orientation of the object region to be estimated on the cropped image of the training target is associated with the cropped image of the training target, which is an image obtained by cutting out the region of the object from a training target image.
The information processing apparatus according to claim 1 or 2. A computer-based information processing method,
Detecting the area of an object on an image of the ground surface taken from above,
To estimate the orientation of the said region,
To shape the region based on the orientation,
To communicate with a terminal device that displays the aforementioned image upon receiving a request from a user,
This includes transmitting an annotated image, which is the image to which the shaped region of the object has been annotated, to the terminal device via a communication unit .
The annotation image is displayed on the terminal device as a response to the request.
Information processing methods. A program to be executed by a computer,
Detecting the area of an object on an image of the ground surface taken from above,
To estimate the orientation of the said region,
To shape the region based on the orientation,
To communicate with a terminal device that displays the aforementioned image upon receiving a request from a user,
This includes transmitting an annotated image, which is the image to which the shaped region of the object has been annotated, to the terminal device via a communication unit .
The annotation image is displayed on the terminal device as a response to the request.
program.