JP7742710B2 - Processing device, processing method, and program - Google Patents

Description

The present invention relates to a processing device, a processing method, and a program.

Orthoimages of road surfaces are used to create maps used in autonomous driving and navigation systems, as well as for road management.

Patent Document 1 discloses a device that generates road surface orthoimages based on data collected by a mobile measurement vehicle.

JP 2015-90591 A

For example, when trying to generate an orthoimage using images acquired from various vehicle-mounted cameras, the conversion parameters for orthogonalization will differ depending on the mounting direction and angle of the camera. Furthermore, even if the mounting direction and angle of the camera are adjusted, they may still change due to vibration or impact.

One example of the problem that this invention aims to solve is how to perform appropriate ortho transformation regardless of the camera's mounting direction or angle.

The invention described in claim 1 is
an acquisition unit that acquires a reference image;
and a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image.

The invention described in claim 14 is
an acquisition step of acquiring a reference image;
and a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image.

The invention described in claim 15 is
A program for causing a computer to execute each step of the processing method according to claim 14.

FIG. 1 is a block diagram illustrating a configuration of a processing apparatus according to a first embodiment. FIG. 10 is a diagram illustrating an example of a reference image. 10A and 10B are diagrams illustrating an example of an edge-detected image obtained by performing edge detection on a reference image. 10A is a diagram showing the vertex coordinates of the outer edge of a partial region in a reference image, and FIG. 10B is a diagram showing the vertex coordinates of the outer edge of a partial region after ortho-transformation. 1 is a flowchart illustrating the flow of a processing method according to the first embodiment. FIG. 1 is a diagram illustrating an example of a usage environment of a processing apparatus according to a first embodiment. FIG. 10 is a diagram illustrating a computer for realizing the processing device. FIG. 10 is a block diagram illustrating the configuration of a processing apparatus according to a second embodiment. 10 is a flowchart illustrating a processing method executed by a processing device according to a second embodiment. FIG. 10 is a diagram illustrating an example of edges detected for an object. FIG. 10 is a diagram for explaining a method for deriving a shape score. 11 is a flowchart illustrating a method by which a calculation unit according to a third embodiment derives the size of an object. 13A and 13B are diagrams illustrating a method in which a calculation unit according to a fourth embodiment derives the size of an object. FIG. 10 is a diagram illustrating the configuration of a processing apparatus according to a fifth embodiment. 10A and 10B are diagrams for explaining a method in which a transformation unit determines a target image and performs projective transformation on the target image. 13 is a flowchart illustrating the flow of processing performed by a processing apparatus according to a fifth embodiment.

Embodiments of the present invention will be described below with reference to the drawings. Note that in all drawings, similar components will be designated by similar reference numerals, and descriptions will be omitted where appropriate.

In the following description, unless otherwise specified, each component of the processing device 10 is represented as a functional block rather than a hardware configuration. Each component of the processing device 10 is realized by any combination of hardware and software, centered around the CPU of any computer, memory, a program loaded into memory, a storage medium such as a hard disk that stores the program, and a network connection interface. There are many variations in the implementation methods and devices.

(First embodiment)
Fig. 1 is a block diagram illustrating the configuration of a processing device 10 according to the first embodiment. Fig. 2 is a diagram illustrating an example of a reference image 20. The processing device 10 includes an acquisition unit 120 and a calculation unit 140. The acquisition unit 120 acquires the reference image 20. The calculation unit 140 calculates parameters for projective transformation based on the shape and size of an object 200 in the reference image 20. This will be described in detail below.

For example, while a moving object such as a car or motorcycle is traveling on a road, a camera captures images of the surrounding area. The road is captured in these captured images. If it were possible to perform projective transformation (ortho-transformation) on images captured by the camera of any moving object, not just a dedicated vehicle, it would be possible to easily collect a large amount of image data needed to generate road surface orthoimages.

However, even if a camera is installed on a moving object, the camera's orientation may change due to vibrations, human contact, or other factors. For example, adjusting the camera's position and measuring its orientation every time the moving object moves would be extremely time-consuming. In contrast, the processing device 10 according to this embodiment can calculate the parameters required for projective transformation using a reference image 20 captured of a reference object 200. The calculated parameters can then be used to perform appropriate projective transformation, thereby reducing labor.

As will be described in detail below, the calculation unit 140 extracts a partial region 22 containing a mark from the reference image 20, and calculates parameters based on the mark contained in the partial region 22. The object 200 contained in the reference image 20 is, for example, a mark placed on the ground. Note that the ground may be, for example, underground, on an elevated road, or the surface of a structure, as long as it is a surface on which a mobile object can move. Examples of the object 200 include dividing lines painted on a road and crosswalks. The object 200 may also be a feature on the floor of a garage. The object 200 may also be a reference object installed for a dedicated purpose or for other purposes. In this case, the reference object can be managed to prevent unexpected changes. The object 200 is, for example, rectangular in plan view in real space.

The method by which the calculation unit 140 calculates parameters will be described with reference to Figures 2 to 4(b). Figure 2 is a diagram showing an example of a reference image 20. The reference image 20 shows a road and an object 200 on the road. In this example, the object 200 is a white line on the road. The white line is a dashed line separated by a predetermined length. The length and width of each white line are predetermined. The calculation unit 140 performs edge detection on the reference image 20. Figure 3 is a diagram showing an example of an edge-detected image obtained by performing edge detection on the reference image 20. Next, the calculation unit 140 extracts a partial region 22 that includes the object 200 in the edge-detected image. The calculation unit 140 can detect the object 200 from the reference image 20 using an existing image processing method. In the example shown in this figure, the partial region 22 includes two objects 200 and the area sandwiched between the two objects 200. Furthermore, at least a portion of the outer edge of the partial region 22 overlaps the outer edge of the object 200. At least a portion of the outer edge of the partial region 22 connects the edges of the two objects 200. However, the partial region 22 may include only one object 200, and the outer edge of the object 200 and the outer edge of the partial region 22 may entirely coincide. However, by including the partial region 22 in two objects 200 and the area sandwiched between the two objects 200, highly accurate parameter calculation using a wide partial region 22 is possible.

Next, the calculation unit 140 acquires the vertex coordinates of the outer edge of the partial region 22 in the reference image 20. The coordinate system here is one in which the reference image 20 is placed on the xy plane. Fig. 4(a) is a diagram showing the vertex coordinates of the outer edge of the partial region 22 in the reference image 20. In this diagram, the coordinates of the four vertices of the partial region 22 are shown: ( _x1 , _y1 ), ( _x2 , _y2 ), ( _x3 , _y3 ), and ( _x4 , _y4 ).

The calculation unit 140 also derives the vertex coordinates of the outer edge of the partial region 22 after orthorectification based on the thickness (width) and length of the object 200 in real space. The coordinate system here is assumed to be an XY plane in which the orthorectified image is placed. FIG. 4B is a diagram showing the vertex coordinates of the outer edge of the partial region 22 after orthorectification. This diagram shows the four vertex coordinates ( _X1 , _Y1 ), ( _X2 , _Y2 ), ( _X3 , _Y3 ), and ( _X4 , _Y4 ) of the partial region 22. For example, the distance _d1 between ( _X1 , _Y1 ) and ( _X4 , _Y4 ) is determined based on the length _Lt of the actual object 200. Furthermore, the distance _d2 between ( _X1 , _Y1 ) and ( _X2 , _Y2 ) is determined based on the width _Wt of the actual object 200 and the actual distance _dt between the two objects 200. That is, _d1 / _d2 = _Lt /(2× _Wt + _dt ) holds. Then, by calculating the vertex coordinates of the partial region 22 before and after the projective transformation in this way, the calculation unit 140 can obtain the transformation parameters used for the projective transformation.

If the distance _dt is unknown, the ratio between the distance between two objects 200 in the reference image 20 and the width of the object 200 may be calculated, and the distance _d2 may be determined based on that ratio. That is, if the distance between two objects 200 in the reference image 20 is α times the width of the object 200, then the coordinates should be set so that _d1 / _d2 = _Lt /(2× _Wt +α× _Wt ) holds.

The calculated parameters can be used when performing projective transformation on images captured by the same camera of the same moving object. For example, parameter calculation is performed for each series of videos. Specifically, one of the images constituting the series of videos is used as the reference image 20, and parameters are calculated. The calculated parameters can be used to perform projective transformation on the other images constituting the video, or the reference image 20. Note that parameter calculation does not necessarily have to be performed for each series of videos. However, because the camera's posture may change after a certain amount of time has passed, it is preferable to perform parameter calculation for the same camera at least every predetermined period of time.

Figure 5 is a flowchart illustrating the flow of a processing method according to this embodiment. The processing method according to this embodiment comprises an acquisition step S10 and a calculation step S30. In the acquisition step S10, a reference image 20 is acquired. In the calculation step S30, parameters for projective transformation are calculated based on the shape and size of the object 200 in the reference image 20. The processing method according to this embodiment is executed by the processing device 10. The processing performed by the processing device 10 according to this embodiment is described in detail below.

In acquisition step S10, the acquisition unit 120 acquires the reference image 20. The acquisition unit 120 can acquire the reference image 20, for example, from a camera (imaging device) mounted on a mobile object. The camera is, for example, an in-vehicle camera. The reference image 20 is, for example, an image including a road. The reference image 20 may also be one of multiple images that make up a video. The reference image 20 may be stored in a storage device in advance, and the acquisition unit 120 may read and acquire the reference image 20 from a storage device accessible by the acquisition unit 120. The reference image 20 may also be an image captured by a camera other than a camera mounted on a mobile object. However, the reference image 20 includes the object 200.

Figure 6 is a diagram illustrating an example of the usage environment of the processing device 10 according to this embodiment. For example, a server machine 50 may collect images from cameras 42 mounted on multiple mobile objects 40, and the acquisition unit 120 of the processing device 10 may acquire the reference image 20 from the server machine 50.

The calculation unit 140 acquires size information indicating the size of the object 200 in real space and calculates parameters using the size information. Specifically, the calculation unit 140 reads and acquires the size information stored in a storage unit accessible to the calculation unit 140. In this embodiment, the shape of the object 200, the size of the object 200, and the distance between two objects 200 are known. For example, if the object 200 is a white line painted on a road, the size information is determined based on road marking standards. Also, if the object 200 is an installed reference object, the size information is determined based on the size of the reference object. The size information may also include information indicating the distance between multiple objects 200. Note that size information may differ depending on the location, such as between a highway and other roads. In this case, the storage unit stores size information for each location, for example. The calculation unit 140 can then select and read size information from the storage unit that corresponds to the shooting position of the reference image 20.

In calculation step S30, the calculation unit 140 calculates the parameters required for projective transformation using the reference image 20 and size information, as described above with reference to Figures 2 to 4(b). That is, the calculation unit 140 extracts the object 200 from the reference image 20, and calculates the parameters using at least the size of the object 200 in the reference image 20 and the size of the object 200 in real space based on the size information. For example, if the object 200 is rectangular in plan view in real space, the calculation unit 140 calculates the parameters so that the ratio of the long side to the short side of the object 200 is the same for the object 200 in real space and the object 200 in an orthoimage obtained using the parameters. Note that if the partial region 22 includes multiple objects 200, the calculation unit 140 may also calculate the parameters using the distance between the multiple objects 200.

In more detail, for example, the calculation unit 140 performs a process to correct lens distortion on the reference image 20, and then performs edge detection. Then, the object 200 is detected based on the detected edges. A partial region 22 including the object 200 is extracted. The partial region 22 is, for example, a region that becomes rectangular when projectively transformed. The calculation unit 140 calculates x and y coordinates (for example, the coordinates of the four vertices of a rectangle) that indicate the outer edge of the partial region 22 in the reference image 20. The calculation unit 140 also calculates x and y coordinates (for example, the coordinates of the four vertices of a rectangle) that indicate the outer edge of the partial region 22 in the orthoimage after projective transformation based on the acquired size information. Based on the calculated x and y coordinates and x and y coordinates, the calculation unit 140 derives transformation parameters. An existing method can be used as a method for deriving the transformation parameters based on the calculated x and y coordinates and x and y coordinates.

As described above, the processing device 10 according to this embodiment calculates appropriate parameters using the acquired reference image 20. Therefore, regardless of the orientation or angle of the camera or other device that captured the image, it is possible to perform projective transformation and obtain a good orthoimage.

The hardware configuration of the processing device 10 is described below. Each functional component of the processing device 10 may be realized by hardware that realizes that functional component (e.g., a hardwired electronic circuit, etc.), or by a combination of hardware and software (e.g., a combination of an electronic circuit and a program that controls it). Below, further explanation is given of the case where each functional component of the processing device 10 is realized by a combination of hardware and software.

Figure 7 is a diagram illustrating a computer 1000 for implementing the processing device 10. The computer 1000 may be any computer. For example, the computer 1000 may be an SoC (System On Chip), a Personal Computer (PC), a server machine, a tablet terminal, or a smartphone. The computer 1000 may be a dedicated computer designed to implement the processing device 10, or may be a general-purpose computer.

The computer 1000 has a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input/output interface 1100, and a network interface 1120. The bus 1020 is a data transmission path through which the processor 1040, the memory 1060, the storage device 1080, the input/output interface 1100, and the network interface 1120 exchange data with one another. However, the method of interconnecting the processor 1040 and other components is not limited to bus connection. The processor 1040 may be any of a variety of processors, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main storage device implemented using RAM (Random Access Memory) or the like. The storage device 1080 is an auxiliary storage device implemented using a hard disk, SSD (Solid State Drive), memory card, ROM (Read Only Memory), or the like.

The input/output interface 1100 is an interface for connecting the computer 1000 to input/output devices. For example, the input/output interface 1100 is connected to an input device such as a keyboard and an output device such as a display device.

The network interface 1120 is an interface for connecting the computer 1000 to a network. This communication network may be, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The network interface 1120 may connect to the network via a wireless connection or a wired connection.

The storage device 1080 stores program modules that realize each functional component of the processing device 10. The processor 1040 reads each of these program modules into the memory 1060 and executes them to realize the function corresponding to each program module.

As described above, according to this embodiment, the calculation unit 140 calculates parameters for projective transformation based on the shape and size of the object 200 in the reference image 20. Therefore, appropriate ortho transformation is possible regardless of the mounting direction or angle of the camera.

Second Embodiment
8 is a block diagram illustrating the configuration of a processing device 10 according to a second embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to the first embodiment, except for the points described below. In the processing device 10 according to this embodiment, an acquisition unit 120 acquires a plurality of images including a reference image 20. The processing device 10 according to this embodiment further includes a selection unit 130 that selects the reference image 20 from the plurality of images acquired by the acquisition unit 120. This will be described in detail below.

The processing device 10 calculates parameters using the reference image 20. Here, the accuracy of the parameters depends on the quality of the reference image 20. Therefore, it is preferable to select and use a reference image 20 that is suitable for calculating the parameters. In the processing device 10 according to this embodiment, the acquisition unit 120 selects multiple images, and the selection unit 130 selects one of these images that is suitable as the reference image 20. The calculation unit 140 then calculates the parameters using the selected reference image 20.

Figure 9 is a flowchart illustrating a processing method executed by the processing device 10 according to this embodiment. In the processing method according to this embodiment, a selection step S20 is performed between the acquisition step S10 and the calculation step S30. In the processing method according to this embodiment, a plurality of images including a reference image 20 are acquired in the acquisition step S10. Then, in the selection step S20, the reference image 20 is selected from the plurality of images.

The acquisition unit 120 acquires, for example, a video captured by a camera mounted on a moving object. A video consists of multiple images. However, the acquisition unit 120 may also acquire multiple images that do not constitute a video. The multiple images may be stored in a storage device in advance, and the acquisition unit 120 may acquire the reference image 20 by reading it from a storage device accessible by the acquisition unit 120. For example, as shown in FIG. 6, a server machine 50 may collect multiple images from cameras 42 mounted on multiple moving objects 40, and the acquisition unit 120 of the processing device 10 may acquire the multiple images from the server machine 50. The multiple images acquired by the acquisition unit 120 may include images that do not include the target object 200.

The method by which the selection unit 130 selects the reference image 20 is described below. As described below, there are multiple perspectives for selecting a preferred reference image 20. The selection unit 130 may select the reference image 20 based on any one of the perspectives, or may select the reference image 20 based on multiple perspectives. When selecting the reference image 20 based on multiple perspectives, for example, the selection unit 130 may calculate a score for each perspective, combine these scores to calculate an overall score, and select the reference image 20 based on the overall score.

The selection unit 130 first extracts images that include the entire object 200 from the multiple images. Specifically, the selection unit 130 performs processing to correct lens distortion on the multiple images, and then performs edge detection. Next, the selection unit 130 detects the object 200 in the images based on the detected edges. Then, from the multiple images, it extracts images that include the entire object 200, and selects the reference image 20 from the extracted images.

<Method based on the position of the object 200>
The selection unit 130 can select the reference image 20 based on the position of the object 200 in the image. Specifically, it is preferable to select an image in which the object 200 is located in a region closer to the camera, i.e., lower in the image, as the reference image 20. This increases the accuracy of edge detection and improves the image quality of the orthoimage generated when that region is projectively transformed. Note that the lower side of the image is the side closer to the moving object in real space, and the upper side of the image is the side farther from the moving object in real space.

For example, the selection unit 130 selects the image in which the object 200 is positioned lowest among the images as the reference image 20. When calculating the score, the selection unit 130 assigns a higher score to an image in which the object 200 is positioned lowest among the images.

<Method based on the state of the moving object>
If the multiple images are images obtained by a camera (image capture device) attached to the moving object, the selection unit 130 can select the reference image 20 based on the condition of the moving object when the image was obtained. The condition of the moving object is, for example, the speed and attitude of the moving object at the time of image capture, and more specifically, at least one of the orientation, speed, roll, and pitch of the moving object at the time of image capture. It is preferable to evaluate whether the condition of the moving object at the time of image capture was stable, and select an image captured under stable conditions as the reference image 20. The processing device 10 can acquire sensor information indicating the condition of the moving object from a sensor provided on the moving object.

For example, the selection unit 130 acquires sensor information of the moving object for a predetermined period of time, including the timing at which each image was captured, based on the capture time of the image. The sensor information includes, for example, sensor values from one or more sensors selected from a speed sensor, a direction sensor, a roll sensor, and a pitch sensor. The selection unit 130 calculates the standard deviation for each sensor value included in the acquired sensor information within the predetermined period of time. Then, it adds up the standard deviations of one or more sensors. For example, the selection unit 130 selects the image with the smallest total standard deviation as the reference image 20. Furthermore, when calculating the score, the selection unit 130 assigns a higher score to images with smaller total standard deviations.

<Method based on edge detection results>
The selection unit 130 can select the reference image 20 based on the results of edge detection performed on the image. If the object 200 is a mark such as a white line painted on a road or a crosswalk, the paint may have faded or the like. Therefore, it is preferable to select as the reference image 20 an image in which edges can be clearly detected based on the strength at the time of edge detection, whether the positional relationship of the detected edges is close to the outline of the object 200, etc.

Specifically, the selection unit 130 calculates the straightness score, shape score, and strength score of the edge as follows. The sum of the calculated straightness score, shape score, and strength score is then set as the score based on edge detection. The selection unit 130 can select the image with the highest score based on edge detection as the reference image 20.

Linearity score
The selection unit 130 derives the length of the edge detected for the object 200. Then, the longer the derived edge length is within the range of an upper limit, the higher the linearity score is assigned to the image. On the other hand, the lowest score is assigned to the image whose derived edge length exceeds the upper limit.

Figures 10(a) to 10(c) are diagrams illustrating edges detected for the object 200. Figure 10(a) shows the detection result for a good object 200. A long edge 210 is detected according to the length of the object 200. Figure 10(b) shows the detection result when the object 200 is blurred. Part of the edge 210 is broken. Figure 10(c) is an example in which noise or something other than the object 200 has been detected. The edge 210 is excessively long.

<<Shape score>>
FIG. 11 is a diagram illustrating a method for deriving a shape score. The selection unit 130 derives the inclination of the edge 210 detected for the object 200 for all images and calculates the average value of the derived inclinations. The average value of the edge inclinations for all images (the inclination indicated by line 220 in the figure) is then compared with the inclination of the edge 210 detected for the object 200 in each image, and a higher shape score is assigned to an image in which an edge with an inclination closer to the average value is detected. Note that if each image contains multiple objects 200 with different inclinations, the selection unit 130 groups the objects 200 in the multiple images that have similar inclinations as collinear objects 200. The edges of the grouped objects 200 are then used to calculate the average value for each group. The inclination of the edge 210 of the object 200 in each image is then compared with the average value for the group to which that object 200 belongs.

<<Intensity score>>
The selection unit 130 derives the intensity at the time of detection for the edge detected for the object 200. Then, the higher the intensity of an image, the higher the intensity score is assigned to the image.

<Method based on shooting position>
The selection unit 130 can select an image whose shooting position satisfies a predetermined condition as the reference image 20. In this case, information indicating the shooting position of each image is associated with each of the multiple images. The acquisition unit 120 then acquires the information indicating the shooting position of each image along with the multiple images.

For example, if the object 200 is a white line painted on a road, the spacing between the white lines is not set to a specific value on ordinary roads. Furthermore, depending on the slope of the road surface and the curvature of the road, there may be cases where the white lines are not rectangular. Therefore, it is preferable to check the shape of the object 200 at each location in advance using high-precision map data or aerial photographs, etc., determine the object 200 to be used as a reference, and select an image of that object 200 as the reference image 20.

Recommended area information indicating areas on a map where a reference object 200 can be photographed is stored in advance in a storage unit accessible by the selection unit 130. The selection unit 130 acquires this recommended area information from the storage unit. The selection unit 130 can then select, as the reference image 20, an image whose photographing position is within the area indicated in the recommended area information. Furthermore, when calculating the score, the selection unit 130 assigns a higher score to an image whose photographing position is within the area indicated in the recommended area information than to an image that is not within the area.

<Method based on overall score>
The selection unit 130 can select the reference image 20 by combining the evaluation results based on the above-mentioned aspects, i.e., the position of the object 200, the state of the moving object, the edge detection result, and the shooting position. Specifically, the selection unit 130 calculates a weighted sum of the scores derived from each aspect as a total score. The weight of each score in the weighted sum is determined in advance. The selection unit 130 selects the image with the highest total score as the reference image 20.

Note that, although examples of methods by which the selection unit 130 selects the reference image 20 have been described above, the selection unit 130 may select the reference image 20 using methods other than these examples.

The calculation unit 140 calculates parameters using the reference image 20 selected by the selection unit 130 in the same manner as in the first embodiment. Note that the calculation unit 140 only needs to calculate parameters using the results of processing such as correction, edge detection, and detection of the object 200 performed by the selection unit 130, and does not need to perform these processes again.

The hardware configuration of the computer that realizes the processing device 10 according to this embodiment is shown, for example, in FIG. 7, similar to the first embodiment. However, the storage device 1080 of the computer 1000 that realizes the processing device 10 according to this embodiment further stores a program module that realizes the functions of the selection unit 130.

As described above, according to this embodiment, the same actions and effects as those of the first embodiment can be obtained. In addition, according to this embodiment, the processing device 10 further includes a selection unit 130 that selects a reference image 20 from the multiple images acquired by the acquisition unit 120. Therefore, highly accurate parameters can be calculated using a reference image 20 that is suitable for parameter calculation.

(Third embodiment)
FIG. 12 is a flowchart illustrating a method by which the calculation unit 140 according to the third embodiment derives the size of the object 200. The processing device 10 according to this embodiment is the same as the processing device 10 according to the first or second embodiment, except for the points described below. In this embodiment, the acquisition unit 120 acquires a plurality of images, including a reference image 20, constituting a moving image. This moving image is acquired by a camera (image capture device) attached to a moving object. The calculation unit 140 then calculates the size of the object 200 in real space using the moving image and the speed of the moving object when the moving image was acquired. Furthermore, the calculation unit 140 calculates parameters using the size of the object 200 in real space. This will be described in detail below.

In the first and second embodiments, the size of the object 200 is known, and the calculation unit 140 calculates parameters using size information prepared in advance. In contrast, in the processing device 10 according to this embodiment, the calculation unit 140 identifies the size of the object 200 (size in real space) using multiple images. Then, the parameters are calculated using the identified size and the reference image 20. Therefore, parameters can be calculated even if the size of the object 200 is unknown.

In the processing device 10 according to this embodiment, the acquisition unit 120 acquires multiple images in the same manner as in the second embodiment. In this embodiment, the partial region 22 is, for example, rectangular in plan view in real space. The calculation unit 140 then assumes multiple lengths L and widths W of the partial region 22 and identifies the appropriate length L and width W from among these. Specifically, the calculation unit 140 calculates the estimated movement distance of the moving object between multiple consecutive images using the assumed values of the length L and the assumed values of the width W, and compares this with the movement distance calculated from the detection value of the sensor of the moving object. In this way, the most likely length L and width W are identified.

The multiple images acquired by the acquisition unit 120 constitute a series of moving images. The calculation unit 140 then performs the following processing on the multiple images in the chronological order in which they were captured. That is, in S303, which will be described later, the multiple images are read in order in the chronological order in which they were captured.

With reference to Figure 12, the method by which the calculation unit 140 determines the size of the partial region 22 will be described. When the acquisition unit 120 acquires multiple images, the calculation unit 140 performs the following processing to determine the size of the partial region 22. S301 to S312 constitute loop processing A. In loop processing A, processing is repeated while changing the assumed value of L for each loop. Multiple assumed values of L are determined in advance, and loop processing A ends when processing has been performed for all of these assumed values. S302 to S311 constitute loop processing B. In loop processing B, processing is repeated while changing the assumed value of W for each loop. Multiple assumed values of W are determined in advance, and loop processing B ends when processing has been performed for all of these assumed values. The assumed values of L and W are set within the range of possible sizes for the actual partial region 22.

S303 to S310 constitute loop processing C. In loop processing C, multiple images acquired by the acquisition unit 120 are repeatedly read in order for each loop and processed. When processing has been completed for all of the multiple images, loop processing C ends.

In S304, an attempt is made to detect the object 200 and extract a partial region 22 from the loaded image. Here, image correction and edge detection may be performed as appropriate prior to detection. When a partial region 22 is extracted, the calculation unit 140 reads the position of the object 200 on the image. Note that even if the partial region 22 includes multiple objects 200, the calculation unit 140 only needs to read the position of at least one object 200.

Next, in S305, the calculation unit 140 determines whether the reading of the position of the object 200 in S304 was successful. If the reading was not successful, the process proceeds to S310. On the other hand, if the reading was successful, in S306 the calculation unit 140 calculates the distance between the camera that captured the image and the object 200 based on the assumed values of L and W and the position of the object 200 on the image. Note that an existing method can be used to calculate the distance between the camera and the object 200 based on the coordinates of the four vertices of the actual rectangular object 200, the coordinates of the four vertices of the object 200 on the image, and the camera characteristics. Here, information indicating the camera characteristics can be obtained by the processing device 10 from the camera that captured the image.

Next, in S307, the calculation unit 140 compares the distance between the camera and the object 200 obtained in this S306 with the distance between the camera and the object 200 obtained in the image from the previous loop processing, and calculates the movement distance Dc of the moving object.

In addition, in S308, the calculation unit 140 acquires information indicating the speed of the moving object when the image was captured from the moving object's sensor. Then, based on the speed when the image was captured, the calculation unit 140 calculates the distance Da traveled by the moving object between when the previous image was captured and when the current image was captured.

Then, in S309, the calculation unit 140 calculates the score so that the smaller the difference between the movement distance Dc and the movement distance Da, the higher the score.

When loop process A ends in S312, that is, when processing has been completed for all assumed values of L, all assumed values of W, and all images, the calculation unit 140 calculates the average score for each combination of assumed values of L and W in S313. The assumed values of L and W for the combination with the highest calculated average value are identified as the size of the object 200 to be used in parameter calculation. Note that instead of calculating the average score for each combination of assumed values of L and W, the calculation unit 140 may calculate the average score for each assumed value of L. In that case, the assumed value of L with the highest average value is identified as the L of the object 200. The calculation unit 140 may also calculate the average score for each assumed value of W, and identify the assumed value of W with the highest average value as the W of the object 200.

In this way, the calculation unit 140 determines the size of the partial region 22, and uses the determined size instead of size information to calculate parameters in the same manner as described in the first embodiment.

In the above-described method, instead of calculating the distance between the camera that captured the image and the object 200, the calculation unit 140 may calculate the distance between the camera and the area on the actual road surface that corresponds to the partial area 22. Even in this case, the size of the partial area 22 can be determined in a similar manner.

2 to 4(b), the length _Lt of the object 200 and the length L of the partial region 22 are the same. Therefore, the length Lt of the object 200 is determined by determining the length _L of the partial region 22. Furthermore, in the examples shown in FIGS. 2 to 4(b), W=2× _Wt + _dt holds. The calculation unit 140 only needs to determine the size of the object 200 in at least one direction. For example, the calculation unit 140 may determine both the length _Lt of the object 200 and the width _Wt of the object 200 by determining the size of the partial region 22, or may determine only one of them.

As described above, this embodiment provides the same functions and effects as the first embodiment. In addition, according to this embodiment, the calculation unit 140 identifies the size of the object 200 using multiple images. Therefore, parameters can be calculated even if the size of the object 200 is unknown.

(Fourth embodiment)
The processing device 10 according to the fourth embodiment is the same as the processing device 10 according to the third embodiment except for the method by which the calculation unit 140 derives the length L _t of the object 200 .

In the processing device 10 according to this embodiment, the acquisition unit 120 acquires multiple images constituting a video. The calculation unit 140 performs correction and edge detection processing on the multiple images as necessary. The calculation unit 140 extracts a first image and a second image as follows from the multiple images. The first image and the second image include the same object 200. The position of one end of the object 200 in the length _Lt direction in the first image is the same as the position of the other end of the object 200 in the length _Lt direction in the second image. The calculation unit 140 can extract the first image and the second image from the multiple images using an existing method. The object 200 is, for example, a white line painted on a road.

The calculation unit 140 calculates the difference between the capture timing of the extracted first image and the capture timing of the second image. The calculation unit 140 also acquires the speed of the moving object between these capture timings. The calculation unit 140 then calculates the amount of movement of the moving object between the capture timing of the first image and the capture timing of the second image using the calculated timing difference (time) and the speed of the moving object. The calculation unit 140 specifies the calculated amount of movement as the length _Lt of the object 200. Note that the length _Lt of the object 200 is the length in a direction approximately parallel to the traveling direction of the moving object.

FIG. 13 is a diagram illustrating a method by which the calculation unit 140 according to this embodiment derives the size of the partial region 22. In the example of this diagram, steps S302 to S311 are as described in the third embodiment. In the example of this diagram, in S320, the calculation unit 140 identifies the length _Lt of the object 200 using the method described above. Then, the processing of steps S302 to S311 is performed using the identified length _Lt. It is assumed that the length _Lt of the object 200 and the length L of the partial region 22 are the same. In this example, it is not necessary to perform loop processing that changes L. Then, when loop processing B ends in S311, the calculation unit 140 calculates the average score for each assumed value of W in S321. The assumed value of W with the highest calculated average is then identified as the width W of the object 200 to be used in parameter calculation. The calculation unit 140 performs parameter calculation using the length L and width W thus identified.

Note that the width W may be known. That is, the calculation unit 140 may read and obtain information indicating the width W that is previously stored in the storage unit, and use this information together with the value of the length L derived as described above to calculate the parameters.

As described above, this embodiment provides the same functions and effects as the first embodiment. In addition, according to this embodiment, the calculation unit 140 identifies the size of the object 200 using multiple images. Therefore, parameters can be calculated even if the size of the object 200 is unknown.

Fifth Embodiment
14 is a diagram illustrating an example of the configuration of a processing device 10 according to a fifth embodiment. The processing device 10 according to this embodiment is the same as the processing device 10 according to at least one of the first to fourth embodiments, except that it further includes a transformation unit 160 that performs projective transformation on a target image using calculated parameters. This will be described in detail below.

In this embodiment, the transformation unit 160 may determine the target image to be projectively transformed. For example, the transformation unit 160 may perform projective transformation on the partial region 22, expand the projectively transformed partial region 22, perform the inverse projective transformation on the expanded partial region 22, and determine the target image using the result of the inverse transformation.

For example, when generating an orthoimage of a road surface from an image taken with an in-vehicle camera, it is necessary to determine the range within the captured image over which projective transformation will be performed. However, the appropriate range for projective transformation will vary depending on the camera's mounting position, angle, and road surface inclination, among other factors. By appropriately specifying the transformation range depending on the image, a highly accurate orthoimage can be obtained.

Figure 15 is a diagram illustrating how the transformation unit 160 determines the target image 30 and performs projective transformation on the target image 30. The transformation unit 160 performs projective transformation on the subregion 22 using the parameters calculated by the calculation unit 140. The projectively transformed subregion 22 is then expanded by a predetermined magnification. In the example shown in this figure, the transformation unit 160 expands the projectively transformed subregion 22 in the upward and left-right directions of the image. The transformation unit 160 identifies the outer boundary line 220 of the expanded subregion 22. If the subregion 22 is a polygon, the outer boundary line 220 of the expanded subregion 22 can be identified by the coordinates indicating the vertices of the line 220. Next, the transformation unit 160 inversely transforms the expanded subregion 22. That is, the inversely transformed line 220 is identified by calculating the coordinates of its vertices. Next, the transformation unit 160 determines the portion of the reference image 20 inside the inversely transformed line 220 as the target image 30. The transformation unit 160 then performs projective transformation on the determined target image 30 to obtain an orthoimage. If the line 220 after inverse transformation extends beyond the reference image 20, a predetermined value is set for the pixels in the extending portion. In this way, the transformation unit 160 can generate an orthoimage.

The expansion magnification when expanding the projectively transformed partial region 22 can be determined in advance based on, for example, the number of lanes on the photographed road.

Note that the target image 30 is not limited to the above example. The transformation unit 160 can perform projective transformation on any image as the target image 30. However, the target image 30 may include, for example, a road. It is preferable that the target image 30 and the reference image 20 are images obtained using the same camera of the same moving object. It is also preferable that the reference image 20 and the target image 30 constitute the same video.

In the example of Figure 15, the target image 30 is a part of the reference image 20. In other words, the reference image 20 and the target image 30 are different images. However, the reference image 20 and the target image 30 may be the same image. Also, the conversion unit 160 may use the partial region 22 as the target image 30 without expanding it.

The transformation unit 160 can perform projective transformation of the target image 30 using existing methods, using the parameters calculated by the calculation unit 140. The transformation unit 160 may also generate multiple orthoimages from multiple target images 30, align the multiple generated orthoimages based on sensor information acquired from a moving object, and combine them into a single image.

The hardware configuration of the computer that realizes the processing device 10 according to this embodiment is the same as that of the first embodiment, and is shown, for example, in FIG. 7. However, the storage device 1080 of the computer 1000 that realizes the processing device 10 according to this embodiment further stores a program module that realizes the functions of the conversion unit 160.

Figure 16 is a flowchart illustrating the flow of processing performed by the processing device 10 according to this embodiment. The processing method according to this embodiment includes an acquisition step S10, a calculation step S30, and a transformation step S40. In the transformation step S40, the target image is projectively transformed using the calculated parameters.

As described above, this embodiment provides the same functions and effects as the first embodiment. In addition, according to this embodiment, the processing device 10 further includes a transformation unit 160 that performs projective transformation on the target image using the calculated parameters. Therefore, highly accurate orthoimages can be obtained regardless of the mounting direction or angle of the camera.

Although the embodiments and examples have been described above with reference to the drawings, these are merely examples of the present invention, and various configurations other than those described above can also be adopted.
Below, examples of reference forms are given.
1. An acquisition unit that acquires a reference image;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image.
Processing equipment.
2. In the processing device according to 1.,
the object is a mark on the ground,
The calculation unit
extracting a partial area including the mark from the reference image;
The parameter is calculated based on the mark included in the partial area.
Processing equipment.
3. In the processing device according to 1. or 2.,
The apparatus further includes a transformation unit that performs projective transformation on a target image using the calculated parameters.
Processing equipment.
4. In the processing device according to 2.,
a transformation unit that performs projective transformation on a target image using the calculated parameters;
The conversion unit
projectively transforming the subregion;
Expanding the projectively transformed subregion;
The expanded subregion is subjected to an inverse projective transformation;
The result of the inverse transformation is used to determine the target image.
Processing equipment.
5. In the processing apparatus according to 3. or 4.,
The reference image and the target image are different images.
Processing equipment.
6. In the processing apparatus according to any one of 1. to 5.,
The object is a dividing line painted on the road.
Processing equipment.
7. In the processing apparatus according to any one of 1. to 6.,
the acquisition unit acquires a plurality of images including the reference image,
a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit
Processing equipment.
8. In the processing device according to 7.,
The selection unit selects the reference image based on a position of the object in the image.
Processing equipment.
9. In the processing device according to 7. or 8.,
the plurality of images are images obtained by an imaging device attached to a moving object,
The selection unit selects the reference image based on the situation of the moving object when the image was obtained.
Processing equipment.
10. In the processing apparatus according to any one of 7. to 9.,
The selection unit selects the reference image based on a result of edge detection performed on the image.
Processing equipment.
11. In the processing apparatus according to any one of 7. to 10.,
The plurality of images are associated with information indicating the photographing positions of the images,
The selection unit selects an image whose photographing position satisfies a predetermined condition as the reference image.
Processing equipment.
12. In the processing apparatus according to any one of 1. to 11.,
The calculation unit acquires size information indicating the size of the object in real space, and calculates the parameter using the size information.
Processing equipment.
13. In the processing apparatus according to any one of 1. to 11.,
the acquisition unit acquires a plurality of images that constitute a moving image, including the reference image;
the video is a video obtained by an imaging device attached to a moving object,
The calculation unit
calculating a size of the object in real space using the moving image and the speed of the moving object when the moving image was obtained;
The parameters are calculated using the size of the object in real space.
Processing equipment.
14. An acquisition step of acquiring a reference image;
and calculating parameters for projective transformation based on the shape and size of the object in the reference image.
Processing method.
15. A program that causes a computer to execute each step of the processing method described in 14.

10 Processing device 20 Reference image 22 Partial region 30 Target image 40 Moving object 42 Camera 50 Server machine 120 Acquisition unit 130 Selection unit 140 Calculation unit 160 Conversion unit 200 Target object 210 Edge 1000 Computer

Claims (19)

an acquisition unit that acquires a reference image;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image;
a transformation unit that performs projective transformation on a target image using the calculated parameters ,
The calculation unit
extracting a partial region including the object from the reference image;
Calculating vertex coordinates of the partial region before and after projective transformation based on the size of the object in real space;
The parameters are calculated using the vertex coordinates before and after the projective transformation.
Processing equipment. an acquisition unit that acquires a reference image;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image;
a transformation unit that performs projective transformation on a target image using the calculated parameters,
the calculation unit extracts a partial region including the object from the reference image;
The conversion unit
projectively transforming the subregion;
Expanding the projectively transformed subregion;
The expanded subregion is subjected to an inverse projective transformation;
A processing device that uses the results of the inverse transform to determine the target image. 3. The processing apparatus according to claim 1 ,
A processing device in which the reference image and the target image are different images. an acquisition unit that acquires a plurality of images that constitute a moving image, including a reference image;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image,
the video is a video obtained by an imaging device attached to a moving object,
The calculation unit
calculating a size of the object in real space using the moving image and the speed of the moving object when the moving image was obtained;
A processing device that calculates the parameters using the size of the object in real space. an acquisition unit that acquires a plurality of images including a reference image;
a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit ;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image,
The calculation unit
extracting a partial region including the object from the reference image;
Calculating vertex coordinates of the partial region before and after projective transformation based on the size of the object in real space;
The parameters are calculated using the vertex coordinates before and after the projective transformation.
Processing equipment. 6. The processing apparatus according to claim 5 ,
The plurality of images are associated with information indicating the photographing positions of the images,
The selection unit is a processing device that selects an image whose photographing position satisfies a predetermined condition as the reference image. an acquisition unit that acquires a plurality of images including a reference image;
a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image,
The selection unit is a processing device that selects the reference image based on the position of the object in the image. an acquisition unit that acquires a plurality of images including a reference image;
a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image,
the plurality of images are images obtained by an imaging device attached to a moving object,
The selection unit is a processing device that selects the reference image based on the state of the moving object when the image was obtained. an acquisition unit that acquires a plurality of images including a reference image;
a selection unit that selects the reference image from the plurality of images acquired by the acquisition unit;
a calculation unit that calculates parameters for projective transformation based on the shape and size of the object in the reference image,
The selection unit is a processing device that selects the reference image based on a result of edge detection performed on an image. The processing apparatus according to any one of claims 1 to 3 and 7 to 9 ,
The calculation unit is a processing device that acquires size information indicating the size of the object in real space and calculates the parameters using the size information. The processing apparatus according to any one of claims 1 to 10 ,
A processing device in which the object is a mark on the ground. The processing apparatus according to any one of claims 1 to 11 ,
The processing device in which the object is a dividing line drawn on a road. an acquisition step of acquiring a reference image;
a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image ;
a transformation step of projectively transforming a target image using the calculated parameters ,
In the calculation step,
extracting a partial region including the object from the reference image;
Calculating vertex coordinates of the partial region before and after projective transformation based on the size of the object in real space;
A processing method for calculating the parameters using the vertex coordinates before and after the projective transformation. an acquisition step of acquiring a reference image;
a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image;
a transformation step of projectively transforming a target image using the calculated parameters,
In the calculation step, a partial region including the object is extracted from the reference image;
In the converting step,
projectively transforming the subregion;
Expanding the projectively transformed subregion;
The expanded subregion is subjected to an inverse projective transformation;
A processing method using the result of the inverse transformation to determine the target image. an acquisition step of acquiring a plurality of images including a reference image and constituting a moving image;
a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image,
the video is a video obtained by an imaging device attached to a moving object,
In the calculation step,
calculating a size of the object in real space using the moving image and the speed of the moving object when the moving image was obtained;
A processing method for calculating the parameters using the size of the object in real space. acquiring a plurality of images including a reference image;
a selection step of selecting the reference image from the plurality of images acquired in the acquisition step;
a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image,
In the selecting step, the reference image is selected based on the position of the object in the image. acquiring a plurality of images including a reference image;
a selection step of selecting the reference image from the plurality of images acquired in the acquisition step;
a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image,
the plurality of images are images obtained by an imaging device attached to a moving object,
In the selecting step, the reference image is selected based on the state of the moving object when the image was acquired. acquiring a plurality of images including a reference image;
a selection step of selecting the reference image from the plurality of images acquired in the acquisition step;
a calculation step of calculating parameters for projective transformation based on the shape and size of the object in the reference image,
In the selection step, the reference image is selected based on the result of edge detection performed on the image. A program that causes a computer to execute each step of the processing method according to any one of claims 13 to 18 .