JP7570210B2 - Information processing device, control method, program, and storage medium - Google Patents

Description

The present invention relates to the detection of change points on roads, such as dividing lines.

Technologies for detecting road change points, such as lane markings, have been known for some time. For example, Patent Document 1 discloses a map change point detection device that detects map change points, such as lane markings, related to roads, by referring to a high-precision map with absolute accuracy that serves as a reference before the change.

JP 2019-3606 A

In general, it is very costly to prepare a high-precision map with absolute accuracy to use as a reference representing the state before the change, so it has been difficult to develop maps of wide areas using such references.

The present invention has been made to solve the above problems, and its main objective is to provide an information processing device that can suitably compare data for change point detection without using a pre-change reference with absolute accuracy.

The claimed invention is
a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
having
the first image and the second image are images associated with absolute coordinates,
The reference position setting means is an information processing device that sets the reference position represented by absolute coordinates .
The claimed invention also includes:
a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
having
the slit image generating means sequentially generates the slit images by cutting out the first image and the second image at predetermined widths along the road starting from the reference position;
The slit image comparison means is an information processing device that sequentially compares the slit images cut out from the first image and the second image by the slit image generation means.
The claimed invention also includes:
a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
a change point candidate determining means for determining the presence or absence of a change point candidate related to the road based on a comparison result for each of the slit images,
The change point candidate determination means is an information processing device that determines the presence or absence of a change point candidate related to a lane marking of the road.
The claimed invention also includes:
a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a reference image generating means for generating a reference image by cutting out an image including the reference position from the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference image;
a slit image matching means for matching the first image and the second image for each slit image,
The slit image generating means is an information processing device that determines a position and an angle for cutting out the slit image based on a comparison result of a reference image cut out from the first image and the second image.
The claimed invention also includes:
a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
having
The information processing device is configured such that the first image and the second image are ortho-images including at least the road.

The claimed invention also includes:
A computer-implemented control method, comprising:
acquiring a first image, which is an image of a road generated based on data measured in the first measurement;
acquiring a second image of the road generated based on data measured in a second measurement;
setting a reference position for the first image and the second image;
generating a slit image cut out from the first image and the second image based on the reference position;
comparing the first image with the second image for each of the slit images;
The first image and the second image are orthoimages including at least the road.
It is a control method.

The claimed invention also includes:
acquiring a first image, which is an image of a road generated based on data measured in the first measurement;
acquiring a second image of the road generated based on data measured in a second measurement;
setting a reference position for the first image and the second image;
generating a slit image cut out from the first image and the second image based on the reference position;
causing a computer to execute a process of comparing the first image and the second image for each of the slit images ;
The program is such that the first image and the second image are ortho-images including at least the road .

1 is a schematic configuration of a change-point detection system. FIG. 2 is a block diagram showing a functional configuration of a change-point detection device. 10 is a flowchart illustrating an example of a procedure for a process related to change point detection. FIG. 2 shows a functional block diagram of a controller of the change-point detection device. 1 shows an example of a first orthoimage clearly showing a first reference image and a first slit image. 13 shows an example of a second orthoimage clearly showing a second reference image and a second slit image. 13 is an example of a flowchart executed by a change-point detection device in a first modified example.

According to a preferred embodiment of the present invention, the information processing device includes a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement, a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement, a reference position setting means for setting a reference position for the first image and the second image, a slit image generation means for generating a slit image cut from the first image and the second image based on the reference position, and a slit image matching means for matching the first image and the second image for each slit image. According to this aspect, the information processing device can match the first image and the second image for each slit image and preferably detect the presence or absence of a change that occurred between the first measurement and the second measurement. Note that the "first measurement" and the "second measurement" are not limited to one measurement, and may correspond to multiple measurements.

In one aspect of the information processing device, the first image and the second image are images associated with absolute coordinates, and the reference position setting means sets the reference position represented by absolute coordinates. With this aspect, the information processing device can preferably align the first image and the second image with reference to the reference position.

In another aspect of the information processing device, the slit image generating means sequentially generates the slit images cut out at predetermined widths from the first image and the second image along the road starting from the reference position, and the slit image matching means sequentially matches the slit images cut out from the first image and the second image by the slit image generating means. With this aspect, the information processing device can preferably detect the presence or absence of changes in the road and its surroundings for each slit image.

In another aspect of the information processing device, the slit image generating means changes at least one of the position and angle of the cutout of the slit image to be cut out next from the second image based on the result of the comparison of the slit images by the slit image comparison means. With this aspect, the information processing device can suitably generate slit images that show the same location in the first image and the second image, and perform comparison with high accuracy.

In another aspect of the information processing device, when the slit image generating means sequentially performs matching of the slit images cut out from the first image and the second image, the slit image matching means performs matching of the next slit image based on the matching result of the immediately preceding slit image. Even with this aspect, the information processing device can preferably improve matching accuracy by using the matching result of the slit image for matching the next slit image.

In another aspect of the information processing device, the information processing device further includes a change point candidate determination means for determining whether or not there are any change point candidates related to the road based on the comparison result for each slit image. With this aspect, the information processing device can preferably determine whether or not there are any change point candidates for each compared slit image.

In another aspect of the information processing device, the change point candidate determination means determines whether there is a change point candidate for the lane markings of the road. In a preferred example, the change point candidate determination means determines whether there is a change point candidate for the solid or dashed lane markings of the road, and the slit image is set to a range for each lane of the road or a range including the left and right shoulders. When the range including the left and right shoulders is set, the information processing device can preferably perform matching between slit images by generating a slit image including the left and right shoulders of the road, even when a solid lane marking with few features is set as the target for detecting change points. In another preferred example, the first image and the second image are images whose pixel values are determined based on the measured reflection intensity. As a result, the information processing device can preferably perform matching to detect the presence or absence of a change in the lane markings using the first image and the second image that accurately reflect the positions of the lane markings with high reflection intensity.

In another aspect of the information processing device, the information processing device further includes a reference image generating means for generating a reference image by cutting out an image including the reference position from the first image and the second image, and the slit image generating means determines a position and angle for cutting out the slit image based on a comparison result of the reference image cut out from the first image and the second image. With this aspect, the information processing device can suitably determine a position and angle for cutting out the slit image so that the image represents a location where the slit images being compared match.

In another aspect of the information processing device, at least one of the first image and the second image is an image that combines data measured by driving the road multiple times. Such an information processing device can detect with high accuracy the presence or absence of changes that have occurred between the first measurement and the second measurement by using the first image and the second image that are not affected by occlusion due to the road congestion at the time of measurement, etc.

In another aspect of the information processing device, the first image and the second image are orthoimages that include at least the road. In this case, the first image and the second image are images of a bird's-eye view of the target road. This allows the information processing device to suitably detect the presence or absence of changes in the road that have occurred between the first measurement and the second measurement.

According to another preferred embodiment of the present invention, a control method executed by a computer includes acquiring a first image, which is an image of a road generated based on data measured in a first measurement, acquiring a second image, which is an image of the road generated based on data measured in a second measurement, setting a reference position for the first image and the second image, generating a slit image cut out from the first image and the second image based on the reference position, and comparing the first image and the second image for each slit image. By executing this control method, the computer can preferably detect the presence or absence of a change that has occurred between the first measurement and the second measurement.

According to another preferred embodiment of the present invention, the program causes a computer to execute a process of acquiring a first image, which is an image of a road generated based on data measured in a first measurement, acquiring a second image, which is an image of the road generated based on data measured in a second measurement, setting a reference position for the first image and the second image, generating a slit image cut out from the first image and the second image based on the reference position, and comparing the first image and the second image for each slit image. By executing this program, the computer can preferably detect the presence or absence of a change that has occurred between the first measurement and the second measurement. Preferably, the program is stored in a storage medium.

Below, a preferred embodiment of the present invention will be described with reference to the drawings.

First Example
(1) System Overview
Fig. 1 is a schematic diagram of a change-point detection system according to Example 1. The change-point detection system shown in Fig. 1 is a system that detects points (change points) where time-series changes occur in white lines (division lines) on a road based on measurement data generated by a measurement vehicle traveling on the road, and mainly includes a measurement vehicle and a change-point detection device 4.

The measurement vehicle generates measurement data by measuring the road and the surrounding environment as it travels along the road. There may be multiple measurement vehicles. The measurement vehicle mainly has an onboard device 1, an external sensor 2, and a positioning sensor unit 3.

The external sensor 2 is one or more sensors that measure (sense) the environment around the measurement vehicle. The external sensor 2 is, for example, a lidar (Light Detection and Ranging, or Laser Illuminated Detection and Ranging), a camera, or both. When the external sensor 2 includes a lidar, the sensor emits a pulsed laser in a predetermined angular range in the horizontal and vertical directions to discretely measure the distance to an object in the external world, and generates point cloud data consisting of measurement points that represent the three-dimensional position and irradiation intensity of the object. When the external sensor 2 includes a camera, the sensor generates an image of the scenery captured from the vehicle. Date and time data indicating the measurement date and time (generation date and time) is added to the output data of the external sensor 2. The external sensor 2 may be provided at any position on the measurement vehicle, or multiple external sensors 2 may be provided on the measurement vehicle.

The positioning sensor unit 3 is a group of sensors that generate movement information that indicates the time-series position and orientation (direction of travel) of the external sensor 2. The positioning sensor unit 3 includes sensors such as a GPS receiver, an acceleration sensor, a gyro sensor, and an IMU (Inertial Measurement Unit). The GPS receiver may generate highly accurate position information that indicates the absolute position (e.g., three-dimensional position of latitude, longitude, and altitude) of the measurement vehicle based on an RTK positioning method (i.e., an interferometric positioning method). The positioning sensor unit 3 may be a sensor provided in the external sensor 2 so as to directly detect the position and orientation of the external sensor 2.

The vehicle-mounted device 1 is electrically connected to the external sensor 2 and the positioning sensor unit 3 by wire or wirelessly, and stores data generated by the external sensor 2 and the positioning sensor unit 3. The vehicle-mounted device 1 is also capable of data communication with the change-point detection device 4, and transmits measurement data "D1" based on the output data of the external sensor 2 and the positioning sensor unit 3 to the change-point detection device 4. Specifically, the vehicle-mounted device 1 transmits measurement data D1 including data measured by the external sensor 2 at multiple positions and movement information of the external sensor 2 based on the output of the positioning sensor unit 3 to the change-point detection device 4. The vehicle-mounted device 1 may be configured as a drive recorder together with the positioning sensor unit 3.

The vehicle-mounted device 1 may also perform processing to appropriately correct the data output by the external sensor 2 and the positioning sensor unit 3. For example, the vehicle-mounted device 1 may detect the difference in the reference time within each of the external sensor 2 and the positioning sensor unit 3, and correct at least one of these date and time data based on the above-mentioned time difference so that the output data of the external sensor 2 and the date and time data included in the movement information are synchronized.

The change point detection device 4 receives measurement data D1 including output data of the external sensor 2 and movement information of the external sensor 2 from the vehicle-mounted device 1, and performs processing to detect a change point based on the received measurement data D1. The change point detection device 4 is an example of an "information processing device."

Note that the configuration of the change point detection system shown in FIG. 1 is an example, and various modifications may be made to the configuration shown in FIG. 1. For example, instead of acquiring the measurement data D1 through data communication with the vehicle-mounted device 1, the change point detection device 4 may acquire the measurement data D1 by reading the measurement data D1 stored in the storage medium by the vehicle-mounted device 1 from the storage medium. In this case, the above-mentioned storage medium is electrically connected to the vehicle-mounted device 1 during measurement of the measurement vehicle, so that the vehicle-mounted device 1 writes the measurement data D1. After measurement on the measurement vehicle, the above-mentioned storage medium is electrically connected to the change point detection device 4, so that the change point detection device 4 reads the measurement data D1. In addition, the external sensor 2 and the positioning sensor unit 3 may each hold log data generated in a storage medium independently, and the storage medium may be read into the change point detection device 4 to supply the measurement data D1 to the change point detection device 4. In this case, the change point detection system may not have the vehicle-mounted device 1. In addition, the change point detection device 4 may be composed of multiple devices. In this case, the change point detection device 4 executes pre-assigned processes among multiple devices, and transmits and receives necessary data between the devices.

(2) Configuration of the change-point detection device
2 is a block diagram showing the functional configuration of the change-point detection device 4. The change-point detection device 4 has an interface 41, a memory 42, and a controller 43. These elements are connected to each other via a bus line.

The interface 41 performs interface operations related to the exchange of data between the change point detection device 4 and an external device. In this embodiment, the interface 41 receives the measurement data D1 generated by the vehicle-mounted device 1. The interface 41 may be a wireless interface such as a network adapter for wireless communication with the vehicle-mounted device 1, or may be a hardware interface for reading the measurement data D1 from a storage medium or the like that stores the measurement data D1. The interface 41 may also perform interface operations with various peripheral devices such as an input device, a display device, and a sound output device.

The memory 42 is composed of various types of volatile and non-volatile memory, such as a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk drive, and a flash memory. The memory 42 stores a program for the controller 43 to execute a predetermined process. The program executed by the controller 43 may be stored in a storage medium other than the memory 42.

The memory 42 also stores the measurement data D1, the first ortho-image D21, the second ortho-image D22, and change point candidate information D3.

The measurement data D1 is the measurement data D1 received by the change point detection device 4 from the vehicle-mounted device 1. The first ortho-image D21 is an ortho-image of a target road (also called the "target road") for which change points are to be detected, based on the measurement data D1 generated by a first measurement of the target road. The "first measurement" is a measurement for generating data for the target road as a reference that serves as a standard before the change, and corresponds, for example, to a measurement made during the first travel of the measurement vehicle on the target road. The first ortho-image D21 is linked to information regarding the absolute position (also called "absolute position information"). This absolute position information is generated, for example, based on movement information of the external sensor 2 that corresponds to the output data of the external sensor 2 used to generate the ortho-image.

The second orthoimage D22 is an orthoimage of the target road and its surroundings generated in a second measurement of the target road. The "second measurement" is a measurement for determining whether or not there has been a change in the target road since the first measurement, and corresponds, for example, to a measurement made by a measurement vehicle traveling the target road for the second time. Absolute position information is associated with the second orthoimage D22.

The measurement data D1 used to generate the first ortho-image D21 and the measurement data D1 used to generate the second ortho-image D22 may be data generated by different measurement vehicles. Furthermore, the first ortho-image D21 and the second ortho-image D22 may be images generated for each road, or may be multiple images generated for one road, or may be images representing multiple roads. The first ortho-image D21 and the second ortho-image D22 are examples of a "first image" and a "second image".

The change point candidate information D3 is information that represents candidate change points detected based on the first orthoimage D21 and the second orthoimage D22.

Here, a supplementary explanation is given regarding the generation of ortho-images. The change point detection device 4 generates a 3D model of the surroundings of the target road section based on the output data of the external sensor 2 included in the measurement data D1 and the position and orientation (measurement direction) of the external sensor 2 at the time of measurement indicated by the movement information of the external sensor 2. The change point detection device 4 then generates an ortho-image by performing ortho-transformation on the generated 3D model. The change point detection device 4 then associates the generated ortho-image with absolute position information based on the movement information of the external sensor 2. In this case, a first ortho-image D21 is generated based on the measurement data D1 obtained in the first measurement, and a second ortho-image D22 is generated based on the measurement data D1 obtained in the second measurement.

Here, if the external sensor 2 is a LIDAR, an orthoimage is generated by converting the 3D measurement point cloud output by the LIDAR into a 2D image. In this case, the pixels of the generated orthoimage may have a reflection intensity value, a Z coordinate value (i.e., altitude information indicated by the measurement point corresponding to the pixel), or both of these values. The orthoimage may also be generated by combining the output data of the camera and the LIDAR. For example, in this case, the LIDAR and the camera may be synchronized to generate an orthoimage in which the parts outside the LIDAR's illumination range are interpolated using the camera's output data, or an orthoimage in which the parts outside the camera's shooting range are interpolated using the LIDAR's output data.

At least one of the measurement data D1, the first ortho-image D21, the second ortho-image D22, and the change-point candidate information D3 may be stored in a storage device external to the change-point detection device 4, such as a hard disk connected to the change-point detection device 4 via the interface 41. The storage device may be a server device that communicates with the change-point detection device 4. The storage device may be composed of multiple devices. The first ortho-image D21 and the second ortho-image D22 may be generated by a device other than the change-point detection device 4 and then stored in the memory 42.

The controller 43 includes one or more processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and a TPU (Tensor Processing Unit), and controls the entire change point detection device 4. In this case, the controller 43 performs processing related to change point detection of the lane markings by executing a program stored in the memory 42, etc. For example, the controller 43 generates a first ortho-image D21 and a second ortho-image D22 based on the measurement data D1 received from the vehicle-mounted device 1 via the interface 41, and stores them in the memory 42. The controller 43 also generates change point candidate information D3 based on the first ortho-image D21 and the second ortho-image D22 and stores them in the memory 42. The controller 43 functions as a "first image acquisition means," a "second image acquisition means," a "reference position setting means," a "reference image generation means," a "slit image generation means," a "slit image matching means," a computer that executes a program, etc.

The processing executed by the controller 43 is not limited to being realized by software using a program, but may be realized by any combination of hardware, firmware, and software. The processing executed by the controller 43 may also be realized by using a user-programmable integrated circuit, such as an FPGA (Field-Programmable Gate Array) or a microcomputer. In this case, the program executed by the controller 43 in this embodiment may be realized using this integrated circuit. In this way, the controller 43 may be realized by various hardware. Furthermore, the functions of the controller 43 may be realized by the cooperation of multiple computers, for example, using cloud computing technology.

(3) Change point detection
Next, the change point detection of the division line performed by the change point detection device 4 will be described. In summary, the change point detection device 4 detects candidate change points by aligning the first ortho-image D21 and the second ortho-image D22 with reference to a predetermined feature or the like, and then comparing the slit images of these images. In this way, the change point detection device 4 detects candidate change points at low cost without requiring a reference image with absolute accuracy, and detects candidate change points with high accuracy by narrowing the comparison range.

(3-1) Processing Flow FIG. 3 is a flowchart showing an example of a procedure for processing related to change-point detection executed by the change-point detection device 4.

First, the change point detection device 4 acquires the first ortho-image D21 and the second ortho-image D22 to be compared (step S10). In this case, the change point detection device 4 acquires the first ortho-image D21 and the second ortho-image D22 representing the target road for which the change points of the lane markings are to be detected, based on the absolute position information linked to the first ortho-image D21 and the second ortho-image D22, respectively.

Next, the change point detection device 4 searches for a position (also called the "reference position Ps") that serves as a reference for aligning the first ortho-image D21 acquired in step S10 with the second ortho-image D22 acquired in step S10 (step S11). The reference position Ps is, for example, the position of a feature such as a sign, billboard, or mailbox that exists around the target road, and is expressed in absolute coordinates (world coordinates).

Next, the change point detection device 4 cuts out an image including the reference position Ps (also called the "reference image") from the first ortho-image D21 and the second ortho-image D22 acquired in step S10. The range from which the reference image is cut out is determined, for example, to be a rectangular area of a predetermined width that includes the reference position Ps and both ends of the target road in the width direction and along the target road. Hereinafter, the reference image cut out from the first ortho-image D21 will be referred to as the "first reference image", and the reference image cut out from the second ortho-image D22 will be referred to as the "second reference image".

Next, the change point detection device 4 performs matching (collation) between the first and second reference images (step S13). In this case, using any matching method, the first and second reference images are matched so that at least the reference position Ps coincides, and the difference position (required translation movement) and difference angle (required rotation angle) between the first and second reference images are calculated. In this case, for example, the change point detection device 4 moves at least one of the first and second reference images to calculate a difference position such that the reference position Ps coincides. In addition, when an index value (match degree) indicating the degree of matching such as the degree of correlation is calculated while rotating the first and second reference images relatively, the change point detection device 4 calculates the rotation angle at which the match degree is highest as the difference angle. In this case, the change point detection device 4 may calculate the above-mentioned rotation angle (difference angle) using any root-finding algorithm such as Newton's method.

Next, the change point detection device 4 uses the reference image as a reference and generates images (also called "slit images") cut into slit shapes (rectangle shapes) from the first ortho-image D21 and the second ortho-image D22 (step S14). In this case, the change point detection device 4 uses the most recently cut reference image or slit image as a reference and generates slit images by cutting out rectangular areas of a predetermined width from the first ortho-image D21 and the second ortho-image D22 along the target road. In this way, the change point detection device 4 uses the reference image as a reference and sequentially cuts slits of a predetermined width into the first ortho-image D21 and the second ortho-image D22 along the direction in which the road extends to generate slit images.

Hereinafter, the slit image cut out from the first ortho-image D21 will be referred to as the "first slit image," and the slit image cut out from the second ortho-image D22 will be referred to as the "second slit image."

Next, the change point detection device 4 matches (collates) the first slit image and the second slit image generated in step S14 (step S15). In this case, an arbitrary matching method is used to match the first slit image and the second slit image, and a match degree indicating the degree of matching is calculated. In this case, for example, when the change point detection device 4 calculates the match degree while moving and rotating the position of at least one of the first slit image and the second slit image, it calculates the combination of the difference position and difference angle that gives the highest match degree using an arbitrary bulb algorithm.

Then, the change point detection device 4 determines whether the first slit image and the second slit image match (step S16). For example, if the highest degree of match obtained in the matching process of step S15 is greater than a predetermined threshold, the change point detection device 4 determines that the first slit image and the second slit image match. On the other hand, if the degree of match is equal to or less than the threshold, the change point detection device 4 determines that the first slit image and the second slit image do not match. The threshold is stored in advance, for example, in the memory 42.

Then, when the change point detection device 4 determines that the first slit image and the second slit image match (step S16; Yes), it stores the difference position and difference angle that give the highest degree of match (step S17). The difference position and difference angle stored in step S17 may be used to determine the cut-out position of the next slit image in step S14, as described below, or may be used in the matching process of step S15 for the next cut-out slit image.

After executing step S17, the change point detection device 4 returns the process to step S14, and generates the next slit image in step S14. Note that, in generating the slit image in step S14, if the generated slit image includes the next reference position, or if the range from which the slit image is to be cut out does not exist in the first ortho-image D21 and the second ortho-image D22 (is cut off in the first ortho-image D21 and the second ortho-image D22), the change point detection device 4 ends the process of the flowchart. In this case, the change point detection device 4 executes the flowchart for the new reference position or the new first ortho-image D21 and the second ortho-image D22.

On the other hand, if the change point detection device 4 determines that the first slit image and the second slit image do not match (step S16; No), it generates change point candidate information D3 (step S18). In this case, the change point detection device 4 determines that a change point may have occurred in the road portion represented by the second slit image that did not match the first slit image, and stores the target first slit image and second slit image in memory 42 as change point candidate information D3.

(3-2) Functional Blocks FIG. 4 shows an example of a functional block of the controller 43 of the change point detection device 4. The controller 43 functionally includes a reference position setting unit 51, a first reference image generating unit 52A and a second reference image generating unit 52B, a reference image matching unit 53, a first slit image generating unit 54A and a second slit image generating unit 54B, a slit image matching unit 55, and a change point candidate determining unit 56. Hereinafter, when the first reference image generating unit 52A and the second reference image generating unit 52B are not distinguished from each other, they are also simply referred to as the "reference image generating unit 52". Similarly, when the first slit image generating unit 54A and the second slit image generating unit 54B are not distinguished from each other, they are simply referred to as the "slit image generating unit 54".

The reference position setting unit 51 sets a reference position Ps for the first ortho-image D21. This process corresponds to the process of step S11 in FIG. 3. As a result, the reference position setting unit 51 sets the location of a point, a line, a feature, etc. as the reference position Ps on the absolute coordinates (world coordinates). For example, when the first ortho-image D21 is an image generated by a camera image, the reference position setting unit 51 determines a characteristic position detected by a feature point detection technique as the reference position Ps. In another example, when the first ortho-image D21 is an image generated by point cloud data of a lidar, the reference position setting unit 51 determines a characteristic position (e.g., a relatively high position) detected based on the Z coordinate value included in each pixel as the reference position Ps. In yet another example, when the first ortho-image D21 is an image generated by point cloud data of a lidar, the reference position setting unit 51 determines a position where pixels with high reflection intensity are densely gathered (e.g., a cluster of a predetermined number of pixels or more) as the reference position Ps. Alternatively, the reference position setting unit 51 may determine the reference position Ps based on map information, or may determine a position selected based on manual input (user input) as the reference position Ps. Then, the reference position setting unit 51 notifies the first reference image generating unit 52A and the second reference image generating unit 52B of the set reference position Ps.

The reference image generating unit 52 generates a reference image. This process corresponds to the process of step S12 in FIG. 3. Specifically, the first reference image generating unit 52A generates a first reference image based on the first ortho-image D21 and the reference position Ps set by the reference position setting unit 51. For example, the first reference image generating unit 52A cuts out the first reference image from the first ortho-image D21 so as to include the reference position Ps and both ends in the width direction of the target road, and to be a rectangular area of a predetermined width along the target road. Similarly, the second reference image generating unit 52B generates a second reference image based on the second ortho-image D22 and the reference position Ps searched by the reference position setting unit 51. In this case, for example, the second reference image generating unit 52B cuts out the second reference image from the second ortho-image D22 so as to include the reference position Ps and both ends in the width direction of the target road, and to be a rectangular area of a predetermined width along the target road. The width of the reference image in the width direction of the road may be set to the same width as the orthoimage. Here, since the reference position Ps is a position in the absolute coordinate system, the first reference image and the second reference image are images that represent a common road portion that includes at least the reference position Ps.

The reference image matching unit 53 matches the first reference image with the second reference image. This process corresponds to step S13 in FIG. 3. As a result, the reference image matching unit 53 matches the first reference image with the second reference image so that the reference position Ps coincides, and supplies the matching results, that is, the difference position and difference angle, to the first slit image generation unit 54A and the second slit image generation unit 54B.

The slit image generating unit 54 generates slit images, which is a process equivalent to step S14 in FIG. 3. Specifically, the first slit image generating unit 54A sequentially generates first slit images, which are images obtained by cutting the target road into strips along the direction in which the road extends, starting from the first reference image including the reference position Ps from the first ortho-image D21. In this case, the first slit image generating unit 54A recognizes the traveling direction of the external sensor 2 at the time of measurement as the direction in which the road extends, based on the movement information included in the measurement data D1, for example, and extracts an image cut out by making slits at intervals of a predetermined width in a direction perpendicular to the direction, as the first slit image. Similarly, the second slit image generating unit 54B sequentially generates second slit images, which are images obtained by cutting the target road into strips along the direction in which the road extends, starting from the second reference image including the reference position Ps from the second ortho-image D22.

The width of the slit image in the road width direction may be a width that includes both ends of the target road in the width direction (including the edges of the target road such as curbs), or may be a width that includes at least both ends of the lanes of the target road. In addition, for roads with multiple lanes, a slit image may be created for each lane, and the widths of both ends of each lane may be set to the width of the slit image, or a slit image may be created for each lane, and the widths of both ends of each lane may be set to the width of the slit image. In addition, if there is a median strip, the width of the slit image may be set to a range that includes the median strip in addition to the target road or target lanes. In yet another example, the width of the slit image may be set to the same width as the orthoimage. Specifically, the width of the slit image is determined taking into consideration the number of feature points in the matching (whether the dividing line that is the subject of change point detection is a solid line or a dashed line), etc.

Here, the slit image generating unit 54 may change the position and angle at which at least one of the first slit image or the second slit image is cut out based on the difference position and difference angle supplied from the slit image matching unit 55. This allows the slit image to be cut out appropriately, taking into account the matching result between the immediately preceding slit images, so that there is no misalignment between the first slit image and the second slit image to be cut out next.

The slit image matching unit 55 calculates the combination of the difference position and the difference angle that will give the highest degree of match by matching (matching) the first slit image and the second slit image. This process corresponds to step S15 in FIG. 3. For example, the slit image matching unit 55 supplies the difference position and the difference angle that are the matching results to the first slit image generating unit 54A and the second slit image generating unit 54B. In this case, as described above, the difference position and the difference angle are used to cut out the slit image. In another example, the slit image matching unit 55 may store the difference position and the difference angle generated by matching in the memory 42 as the initial values for searching for the optimal solution of the difference position and the difference angle in matching the next slit image. Even in this case, the accuracy of matching the next slit image can be preferably improved.

The change point candidate determination unit 56 determines whether there is a change point candidate based on the first slit image, the second slit image, and their degree of match (when they match best) supplied from the slit image matching unit 55. This process corresponds to steps S16 and S17 in FIG. 3. For example, if the degree of match is equal to or less than a predetermined threshold, the change point candidate determination unit 56 determines that there is a possibility that a change has occurred in the road portion indicated by the first slit image and the second slit image. Therefore, in this case, the change point candidate determination unit 56 regards the first slit image and the second slit image as images representing change point candidates, and stores these images in the memory 42 as change point candidate information D3.

The change point candidate determination unit 56 may further perform a process of determining a change point based on the change point candidate information D3. For example, the change point candidate determination unit 56 generates multiple second ortho-images D22 of the target road based on the measurement data D1 measured over multiple time periods, and generates change point candidate information D3 by comparing with the first ortho-image D21. Then, when the change point candidate determination unit 56 determines that the same point has been extracted as a change point candidate in the second ortho-images D22 corresponding to multiple time periods based on the change point candidate information D3, it determines the point as a change point. In this way, the change point candidate determination unit 56 may determine as a change point a point where a change has been detected over multiple time periods. In another example, the change point candidate determination unit 56 may determine the change point candidate indicated by the change point candidate information D3 as a change point as it is.

(3-3) Specific Examples of Reference Images and Slit Images Next, specific examples of methods for generating reference images and slit images will be described with reference to FIGS.

Figure 5 shows an example of a one-lane road (one lane road on one side) of a first ortho-image D21 clearly showing the first reference image and the first slit image. In the case of a road with multiple lanes, a slit image may be created for each lane, or a slit image may be created for multiple lanes together. In addition, in the case of a road with a median strip, in addition to the slit image creation method for a road with multiple lanes, a slit image with a width set so that the median strip is included in addition to the target road or target lane may be created. Also, Figure 6 shows an example of a second ortho-image D22 corresponding to the first ortho-image D21 shown in Figure 5 and clearly showing the second reference image and the second slit image. The dashed frame in Figures 5 and 6 represents an example of the cut-out range of each reference image or slit image. Note that the width of the reference image or slit image may be the same as the width of the ortho-image.

The first ortho-image D21 shown in FIG. 5 and the second ortho-image D22 shown in FIG. 6 are generated based on the output data of the lidar, and each pixel has a pixel value corresponding to the reflection intensity. In the first ortho-image D21, the pixel regions 60A-62A corresponding to the demarcation lines and the pixel regions 63A, 64A corresponding to the signs are formed by pixels with high pixel values. Similarly, in the second ortho-image D22, the pixel regions 60B-62B corresponding to the demarcation lines and the pixel regions 63B, 64B corresponding to the signs are formed by pixels with high pixel values. In addition, the first ortho-image D21 and the second ortho-image D22 are linked to an absolute coordinate system, and the upper left corner position of the first ortho-image D21 shown in FIG. 5 corresponds to absolute coordinates (X1, Y1), and the upper left corner position of the second ortho-image D22 shown in FIG. 6 corresponds to absolute coordinates (X2, Y2).

In this case, the reference position setting unit 51 first searches the pixel regions 63A and 64B of the sign as the reference position Ps, and among them, the pixel region 63A is set as the first reference position Ps. Then, the first reference image generating unit 52A determines the cut-out range of the first reference image "A1" including the pixel region 63A, and the second reference image generating unit 52B determines the cut-out range of the second reference image "B1" including the pixel region 63B. In this case, for example, these reference image generating units 52 determine as the reference image a rectangular image cut out by a predetermined width in a direction perpendicular to the traveling direction of the external sensor 2 indicated by the movement information of the external sensor 2 at the time of measurement. In another example, the reference image generating unit 52 may recognize the extension direction of the road by image recognition processing and set the reference image based on the recognized extension direction of the road. In addition, the width of the reference image is set to a length that includes the left and right edges of the road, or at least the center divider or the target lane when the road includes a center divider or multiple lanes, like the width of the slit image. The cut-out range of the first reference image A1 and the cut-out range of the second reference image A2 may be determined based on manual input. After that, the reference image matching unit 53 matches the first reference image A1 with the second reference image A2 and calculates the difference position and difference angle.

Then, the first slit image generating unit 54A determines a rectangular area of a predetermined width adjacent to the first reference image A1 along the road as the cutout range of the first slit image "a1". Also, the second slit image generating unit 54B determines a range adjacent to the second reference image A2 along the road, which is moved by the difference position and difference angle calculated by matching the first reference image A1 and the second reference image A2, as the cutout range of the second slit image "a2". Then, the first slit image generating unit 54A determines a rectangular area of a predetermined width adjacent to the first slit image a1 along the road as the cutout range of the first slit image "b1". Also, the second slit image generating unit 54B determines a range adjacent to the second slit image a2 along the road, which is moved by the difference position and difference angle calculated by matching the first slit image a1 and the second slit image a2, as the cutout range of the second slit image "b2". Using a similar procedure, the cut-out ranges of the first slit image "c1" to "e1" and the second slit image "c2" to "e2" are determined and matched.

Furthermore, after matching the first slit image e1 and the second slit image e2, the slit image determined from the first ortho-image D21 next to the first slit image e1 includes the pixel area 64A of the sign that is the reference position, so the first reference image generating unit 52A sets a first reference image B1 that includes this pixel area 64A. Similarly, the second reference image generating unit 52B sets a second reference image B2 that includes the pixel area 64A that corresponds to the newly set reference position Ps. Thereafter, matching between the first reference image B1 and the second reference image B2, generation of the slit image, and matching of the generated slit image are performed in sequence.

In this way, the change point detection device 4 sequentially compares the first ortho-image D21 and the second ortho-image D22 for each partial image (slit image and reference image), and uses the comparison result of the partial image in comparing the next partial image. This allows the change point detection device 4 to compare the first ortho-image D21 and the second ortho-image D22 with high accuracy, and detect areas that are candidates for change points. Note that the difference position and difference angle, which are the matching results of the slit image matching unit 55, may be used as initial values for the difference position and difference angle in matching the next slit image, instead of being used to determine the cut-out range of the slit image. Note that in Figures 5 and 6, the reference position is on the road edge, but it does not necessarily have to be on the road edge.

Here, we will provide a supplementary explanation of the effects of this embodiment. When matching large images, the accuracy of the matching tends to be low due to image distortion and shaking. Therefore, in this embodiment, matching is performed using slit images that are cut out from a small range. This narrows the comparison range, which has the effect of preventing unnecessary matching. In addition, in this embodiment, the matching process is performed continuously without gaps, so that the matching results of the previous image (position deviation and rotation angle) can be suitably used to correct the creation of the next image. As a result, the same location can be compared even between different driving images. In addition, instead of preparing a reference image with absolute accuracy as in the past, change points can be detected by relative comparison of orthoimages obtained each time the measurement vehicle travels. Therefore, the cost of creating a reference image with absolute accuracy can be suitably reduced.

As described above, the controller 43 of the change point detection device 4 in this embodiment acquires the first ortho-image D21, which is an image of the road generated based on the data measured in the first measurement. The controller 43 also acquires the second ortho-image D22, which is an image of the road generated based on the data measured in the second measurement. The controller 43 also sets a reference position Ps for the first ortho-image and the second ortho-image D22. Then, the controller 43 generates a slit image cut out from the first ortho-image D21 and the second ortho-image D22 based on the reference position Ps. Then, the controller 43 compares the first ortho-image D21 and the second ortho-image D22 for each slit image. This allows the controller 43 to suitably perform comparison between images for change point detection without preparing a pre-change reference with absolute accuracy.

(4) Modifications
Next, modified examples that can be applied to the above-mentioned embodiment will be described. The following modified examples may be applied to the above-mentioned embodiment in any combination.

(First Modification)
The change point detection device 4 may use a plurality of ortho-images based on the measurement data D1 obtained by a plurality of drives (e.g., a week's worth of drives, or drives during the day and night) as the first ortho-image D21. Similarly, the change point detection device 4 may use a plurality of ortho-images based on the measurement data D1 obtained by a plurality of drives as the second ortho-image D22.

Figure 7 is an example of a flowchart executed by the change point detection device 4 in the first modified example. The flowchart in Figure 7 includes a process for generating a reference image, etc., based on an image obtained by accumulating multiple orthoimages.

First, the change point detection device 4 acquires the orthoimage to be processed (step S20). In this case, at least one of the first orthoimage D21 and the second orthoimage D22 may be multiple orthoimages obtained by traveling multiple measurement vehicles. The change point detection device 4 extracts these orthoimages from the memory 42.

Next, the change point detection device 4 searches for the reference position Ps for the first ortho-image D21 acquired in step S20 (step S21). Then, the change point detection device 4 accumulates the ortho-images based on the searched reference position Ps (step S22). Here, a case where multiple ortho-images corresponding to the second ortho-image D22 are prepared will be described. In this case, the change point detection device 4 aligns the multiple ortho-images acquired in step S20 as the second ortho-image D22 based on the reference position Ps, and generates an image (also called an "accumulated image") by accumulating these ortho-images after alignment. In this case, the change point detection device 4 may generate an image by accumulating multiple ortho-images only in a partial area including the reference position Ps as the accumulated image of the second ortho-image D22. Similarly, when multiple ortho-images corresponding to the first ortho-image D21 are prepared, the change point detection device 4 generates an integrated image by accumulating the ortho-images that have been aligned based on the reference position Ps.

Then, the change point detection device 4 cuts out and generates a reference image from each of the first ortho-image D21 and the second ortho-image D22 (step S23). Here, if the change point detection device 4 generates an integrated image of the first ortho-image D21 or the second ortho-image D22 in step S22, it cuts out a reference image from the integrated image. Then, the change point detection device 4 matches the first reference image generated in step S23 with the second reference image (step S24). After that, the change point detection device 4 executes the same processes as steps S14 to S18 in FIG. 3 as steps S25 to S29.

According to this modified example, an integrated image that summarizes driving over a specified period of time is generated, and change points during the specified period can be detected. In addition, for roads that have congested times, an integrated image that includes orthoimages of non-congested times can be generated, making it possible to preferably generate orthoimages of roads that are not affected by occlusion due to congestion.

(Second Modification)
The change point detection device 4 may change the method of creating the slit image depending on the type of lane marking that is the subject of change point detection.

For example, when detecting change points on solid dividing lines, the change point detection device 4 takes into consideration the fact that the solid line itself is featureless and difficult to compare, and generates a first ortho-image D21, a second ortho-image D22, a reference image, and a slit image so that the left and right shoulders of the road (passable area) are included in the images. This enables the change point detection device 4 to favorably increase the number of feature points that serve as references when matching slit images, and favorably improve the matching accuracy.

On the other hand, when the change point detection device 4 detects a dashed dividing line as a change point, the length of the slit image in the direction of travel is made longer than the length of the dashed line and shorter than the length of two dashed lines. This allows the change point detection device 4 to suitably improve the matching accuracy between slit images.

(Third Modification)
The detection target for the change point is not limited to the lane markings. Instead of or in addition to this, any feature (guard rail, etc.) extending along the road may be used as the detection target for the change point. Even in this case, the change point detection device 4 can detect the change point with high accuracy by comparing each slit image.

In the above-described embodiment, the program can be stored using various types of non-transitory computer readable media and supplied to a computer, such as a controller. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R/Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)).

Although the present invention has been described above with reference to the embodiments, the present invention is not limited to the above embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In other words, the present invention naturally includes various modifications and amendments that a person skilled in the art could make in accordance with the entire disclosure, including the scope of the claims, and the technical ideas. In addition, the disclosures of the above cited patent documents and the like are incorporated into this document by reference.

REFERENCE SIGNS LIST 1 Vehicle-mounted device 2 External sensor 3 Positioning sensor unit 4 Change point detection device 41 Interface 42 Memory 43 Controller

Claims (13)

a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
having
the first image and the second image are images associated with absolute coordinates,
The reference position setting means sets the reference position represented by absolute coordinates. a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
having
the slit image generating means sequentially generates the slit images by cutting out the first image and the second image at predetermined widths along the road starting from the reference position;
The slit image comparison means is an information processing device that sequentially compares the slit images cut out from the first image and the second image by the slit image generation means. 3. The information processing apparatus according to claim 2 , wherein the slit image generating means changes at least one of a position and an angle for cutting out the slit image to be next cut out from the second image based on a result of the comparison of the slit images by the slit image comparing means. The information processing device according to claim 2, wherein when the slit image generation means sequentially performs matching of the slit images cut out from the first image and the second image, the slit image matching means performs matching of the next slit image based on the matching result of the immediately preceding slit image. a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
a change point candidate determining means for determining the presence or absence of a change point candidate related to the road based on a comparison result for each of the slit images,
The change point candidate determination means determines whether or not there is a change point candidate related to a dividing line of the road. The change point candidate determination means determines whether or not there is a change point candidate regarding a solid or dashed dividing line of the road,
The information processing device according to claim 5 , wherein the slit image is set in a range including each lane of the road or a range including left and right shoulders of the road. The information processing apparatus according to claim 5 , wherein the first image and the second image are images whose pixel values are determined based on measured reflection intensities. a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a reference image generating means for generating a reference image by cutting out an image including the reference position from the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference image;
a slit image matching means for matching the first image and the second image for each slit image,
The slit image generating means determines a position and an angle for cutting out the slit image based on a comparison result of a reference image cut out from the first image and the second image. a first image acquisition means for acquiring a first image, which is an image of a road generated based on data measured in a first measurement;
a second image acquisition means for acquiring a second image, which is an image of the road generated based on data measured in a second measurement;
a reference position setting means for setting a reference position for the first image and the second image;
a slit image generating means for generating a slit image cut out from the first image and the second image based on the reference position;
a slit image matching means for matching the first image and the second image for each slit image;
having
The information processing device, wherein the first image and the second image are orthoimages that include at least the road. The information processing device according to claim 1 , wherein at least one of the first image and the second image is an image obtained by combining data measured by traveling a plurality of times on the road. A computer-implemented control method, comprising:
acquiring a first image, which is an image of a road generated based on data measured in the first measurement;
acquiring a second image of the road generated based on data measured in a second measurement;
setting a reference position for the first image and the second image;
generating a slit image cut out from the first image and the second image based on the reference position;
comparing the first image with the second image for each of the slit images;
The first image and the second image are orthoimages including at least the road.
Control methods. acquiring a first image, which is an image of a road generated based on data measured in the first measurement;
acquiring a second image of the road generated based on data measured in a second measurement;
setting a reference position for the first image and the second image;
generating a slit image cut out from the first image and the second image based on the reference position;
causing a computer to execute a process of comparing the first image and the second image for each of the slit images ;
The first image and the second image are orthoimages including at least the road.
program. A storage medium storing the program according to claim 12 .

Images