JP7320459B2 - Image processing device, imaging device, roadside unit, and image processing method - Google Patents

Description

The present invention relates to an image processing device, an imaging device, a roadside device, and an image processing method.

Installation of roadside units on roads that communicate various information with vehicles is under study. For example, the roadside unit captures images of objects to be measured, such as vehicles and people approaching an intersection with poor visibility, with a camera, and notifies vehicles and people heading to the intersection from a road other than the object to be measured as information about their positions. It is being considered to be used to

In order to calculate the position of the measurement target based on the image captured by the camera, it is necessary to calibrate the camera and calculate calibration parameters corresponding to the position and orientation of the camera in the world coordinate system. In order to calibrate the camera of the roadside unit, the measurement vehicle equipped with the mark is photographed by the camera, the position of the mark in the image is acquired, and the position of the measurement vehicle is calculated in the world coordinate system from the information received by the GPS receiver. Acquiring the position of a mark has been proposed (see Patent Document 1).

JP 2012-010036 A

However, camera calibration using a measurement vehicle is a complicated task.

Accordingly, an object of the present disclosure, which has been made in view of the problems of the conventional technology as described above, is to provide an image processing device, an imaging device, a roadside device, and an image processing method that simply perform camera calibration. .

In order to solve the above-mentioned problems, the image processing device according to the first aspect includes:
an image interface that acquires a captured image from the imaging unit;
Positions in a world coordinate system of each of a plurality of specific feature points on the road surface or the floor surface of the captured image captured by the imaging unit, and the plurality of specific feature points captured by the imaging unit at an arbitrary time a memory that stores the position in each captured image as a reference position;
A controller capable of detecting the plurality of specific feature points in the captured image and calculating calibration parameters for calibrating the position and orientation of the imaging unit based on the plurality of specific feature points and the position of the world coordinate system. and
The controller detects the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time, and detects a predetermined ratio or more of the plurality of specific feature points. If the position in the captured image has changed from the reference position, the calibration parameters are recalculated.

In addition, the imaging device according to the second aspect is
an imaging unit that generates a captured image by imaging;
Positions in a world coordinate system of each of a plurality of specific feature points on a road surface or a floor surface of an image captured by the imaging unit, and the plurality of specific feature points captured by the imaging unit at an arbitrary time a memory that stores the position in each captured image as a reference position;
A controller capable of detecting the plurality of specific feature points in the captured image and calculating calibration parameters for calibrating the position and orientation of the imaging unit based on the plurality of specific feature points and the position of the world coordinate system. and
The controller detects the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time, and detects a predetermined ratio or more of the plurality of specific feature points. If the position in the captured image has changed from the reference position, the calibration parameters are recalculated.

In addition, the roadside unit according to the third aspect is
an imaging unit that generates a captured image by imaging;
Positions in a world coordinate system of each of a plurality of specific feature points on a road surface or a floor surface of an image captured by the imaging unit, and the plurality of specific feature points captured by the imaging unit at an arbitrary time a memory that stores the position in each captured image as a reference position;
A controller capable of detecting the plurality of specific feature points in the captured image and calculating calibration parameters for calibrating the position and orientation of the imaging unit based on the plurality of specific feature points and the position of the world coordinate system. and
The controller detects the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time, and detects a predetermined ratio or more of the plurality of specific feature points. If the position in the captured image has changed from the reference position, the calibration parameters are recalculated.

Further, the image processing method according to the fourth aspect is
An image processing method by an image processing device,
The image processing device is
Positions in the world coordinate system of each of a plurality of specific feature points on the road surface or the floor surface of the captured image captured by the imaging unit, and each of the plurality of specific feature points captured by the imaging unit at an arbitrary time. The position in the captured image of is stored as a reference position,
Detecting the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time,
when positions in the captured image of a predetermined ratio or more among the plurality of specific feature points have changed from the reference position, based on the plurality of specific feature points and the position of the world coordinate system A calibration parameter for calibrating the position and orientation of the imaging unit is calculated.

According to the image processing device, imaging device, roadside device, and image processing method according to the present disclosure configured as described above, camera calibration can be performed simply.

1 is a block diagram showing a schematic configuration of a roadside machine including an image processing device according to this embodiment; FIG. 2 is a block diagram showing a schematic configuration of the image processing apparatus of FIG. 1; FIG. FIG. 3 is a flowchart for explaining manual calibration processing executed by the controller in FIG. 2; FIG. FIG. 3 is a flowchart for explaining automatic calibration processing executed by the controller in FIG. 2; FIG.

An embodiment of an image processing apparatus to which the present disclosure is applied will be described below with reference to the drawings.

As shown in FIG. 1 , a roadside device 11 including an image processing device 10 according to an embodiment of the present disclosure includes an imaging unit 12 and an imaging device 13 including the image processing device 10, a memory 14, a communication unit 15, and a controller 16. Consists of

The image capturing unit 12 captures a scene around the roadside unit 11 and generates an image. The image processing device 10 determines whether there is an object to be measured around the roadside unit 11 based on the image captured by the imaging unit 12 . In this embodiment, the objects to be measured are, for example, humans, box-shaped vehicles, motorcycles, and bicycles. When there is an object to be measured, the image processing device 10 calculates the existing position of the object to be measured in the real space, in other words, the position in the world coordinate system. The memory 14 stores the position of the roadside unit 11 . The communication unit 15 communicates with an external device by wire or wirelessly. The external device is, for example, a terminal device such as a navigation device used in a vehicle adjacent to the roadside unit 11, or a server device that recognizes the current position of a running vehicle and transmits traffic information to the vehicle. The external device may be a terminal device for manual calibration of the imaging unit 12 . When the captured image includes the object to be measured, the controller 16 causes the communication unit 15 to transmit the position of the object to be measured in the real space together with the position of the roadside unit 11 to an external device that notifies the user of the position of the object to be measured. to control.

Details of the imaging unit 12 and the image processing device 10 will be described below.

The imaging unit 12 is fixed to a structure having a height capable of capturing an outdoor scene including a road surface, such as a signaling device, a utility pole, and a streetlight. The position and posture at which the imaging unit 12 is fixed may be determined in advance, or may be determined at the time of fixing. The position and orientation of the imaging unit 12 respectively mean the position of the imaging unit 12 with respect to the reference position and the inclination of the imaging unit 12 with respect to the reference direction in the world coordinate system. In the present disclosure, the “world coordinate system” means a coordinate system set based on the three-dimensional space outside the imaging unit 12 . The imaging unit 12 images a subject within an angle of view and generates an image signal corresponding to the subject.

As shown in FIG. 2, the image processing apparatus 10 includes an image interface 17, an information interface 18, a memory 19, and a controller 20. FIG.

The image interface 17 acquires the captured image from the imaging unit 12 . The information interface 18 communicates with the controller 16 of the roadside unit 11 .

The memory 19 includes arbitrary storage devices such as RAM (Random Access Memory) and ROM (Read Only Memory). Memory 19 stores various programs that cause controller 20 to function and various information that controller 20 uses.

The memory 19 stores the position of each of the plurality of specific feature points in the world coordinate system. A specific feature point is a point that can be detected by the controller 20 in a captured image captured by the imaging unit 12 fixed to each arbitrary structure. Further, the specific feature point is a point in the captured image that corresponds to the corner of the road marking, the lower end of the structure, etc., which are continuously located at the same place on the road surface or floor surface in the real space. The corners of road markings are, for example, the four corners of crosswalks and the corners of white lines. The lower end of the structure is, for example, the corner of a vending machine. The position of the specific feature point in the world coordinate system is stored in the memory 19 by using a terminal device for manual calibration, which will be described later. Also, the memory 19 stores a reference position. The reference position is the position in the captured image of each of the plurality of specific feature points captured by the imaging unit 12 at any time.

The memory 19 may store a conversion formula or conversion table for coordinates from the two-dimensional image coordinate system to the world coordinate system. The image coordinate system may have vertical and horizontal axes in the image captured by the imaging unit 12 . The world coordinate system may have axes in the vertical direction of the real space and two directions parallel to the horizontal plane and perpendicular to each other. The conversion formula or conversion table may be calibrated with calibration parameters that calibrate the position and orientation of the imaging unit 12 . A conversion formula or a conversion table is used for coordinate conversion of the position coordinates of an object on the road surface or the floor surface in the captured image from the image coordinate system to the world coordinate system.

Controller 20 includes one or more processors and memory. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor that specializes in specific processing. A dedicated processor may include an Application Specific Integrated Circuit (ASIC). The processor may include a programmable logic device (PLD). A PLD may include an FPGA (Field-Programmable Gate Array). The controller 20 may be either a System-on-a-Chip (SoC) with one or more processors working together, or a System In a Package (SiP).

When the controller 20 receives a request for manual calibration from the manual calibration terminal device via the information interface 18 and the communication unit 15, the controller 20 performs manual calibration as described below.

In executing manual calibration, the controller 20 obtains the positions of each of the plurality of specific feature points in the world coordinate system, obtains the positions of the plurality of specific feature points in the captured image, and calculates the calibration parameters. . The number of specified feature points may be equal to or greater than the number necessary for performing calibration, preferably 5 or more, more preferably 10 or more, and even more preferably 20 or more.

The controller 20 acquires the world coordinate system positions of the feature points via the information interface 18, for example, from a GPS receiver or a terminal device for manual calibration. The controller 20 stores the acquired positions of the feature points in the world coordinate system in the memory 19 .

In order to acquire the position from the GPS receiver, for example, the GPS receiver is positioned within the angle of view of the imaging unit 12, such as the corner of a road marking on the road surface, or the bottom corner of a vending machine on the floor. Then, the controller 20 is placed at a position that can be easily identified from the captured image. A GPS receiver calculates its position in the horizontal plane based on the signals it receives from several satellites. When the controller 20 receives the mounting position of the GPS receiver directly from the GPS receiver or via a terminal device for manual calibration, the controller 20 adds the height of the road surface or the floor surface as a coordinate in the height direction, Get as the position in the world coordinate system of a specific feature point. Since the user is required to place the GPS receiver at a plurality of positions, the controller 20 obtains the world coordinate system positions of a plurality of specific feature points.

Alternatively, in order to acquire the position from the terminal device for manual calibration, for example, by the user's operation on the terminal device, an aerial photograph around the structure to which the imaging unit 12 is fixed is superimposed on the display of the terminal device. A map or survey plan is displayed. By the user's operation, the controller 20 picks up an image from the map or the survey map, such as the corner of the road marking on the road surface within the angle of view of the imaging unit 12, or the corner of the vending machine on the floor. , a plurality of positions on the road surface or floor surface that are considered to be easy to identify are specified. When the controller 20 receives a plurality of specified positions in the aerial photograph from the terminal device for manual calibration, the controller 20 adds the height of the road surface or floor surface as a coordinate in the height direction to obtain a plurality of specific feature points. as a position in the world coordinate system of .

The controller 20 controls the information interface 18 so as to transmit the captured image acquired from the imaging unit 12 to the terminal device for manual calibration in order to acquire the position of each of the plurality of specific feature points in the captured image. The controller 20 acquires, via the information interface 18, the position in the captured image that is associated with each feature point in the position of the world coordinate system for the transmitted captured image. Note that the terminal device for calibration displays the captured image and a map showing the positions of the feature points on the horizontal plane based on the positions of the feature points in the world coordinate system. The user is required to input the position of each feature point in the captured image while looking at the position of each feature point on the map. The controller 20 stores the positions of the acquired feature points in the captured image in the memory 19 .

In calculating the calibration parameters, the controller 20 first calculates temporary calibration parameters. The controller 20 converts the position of each of the plurality of specific feature points in the captured image and the acquired position of each of the feature points in the world coordinate system by executing, for example, a 5-point algorithm. The parameters that calibrate the table are calculated as temporary calibration parameters.

The controller 20 converts the position of each feature point in the world coordinate system to the position in the captured image using a conversion formula or conversion table calibrated using the temporary calibration parameters. The controller 20 calculates the distance between the converted position in the captured image and the position in the captured image of the corresponding feature point acquired from the terminal device for manual calibration.

If the calculated interval is less than the interval threshold, the controller 20 temporarily determines the feature point corresponding to the interval as the reference feature point used for automatic calibration.

When the calculated interval is equal to or greater than the interval threshold, the controller 20 distinguishably indicates the position converted from the world coordinate system and the position obtained from the terminal device for manual calibration on the captured image. Superimpose two point images colored with different specific colors. The controller 20 controls the information interface 18 so as to transmit the captured image on which the point image is superimposed to the terminal device for manual calibration.

The controller 20 determines the position of the feature point corresponding to the point image in the captured image to be re-specified, or the position in the world coordinate system to which the feature point is re-specified, for the transmitted captured image on which the point images are superimposed. to get one of It should be noted that the user of the terminal device for manual calibration may re-specify the position of the feature point in the captured image or You are prompted to re-specify the location of the world coordinate system.

The controller 20 selects a combination of the redesignated position in the captured image and the position in the world coordinate system stored in the memory 19 or For the combination of the captured image position and the re-specified world coordinate system position, perform the 5-point algorithm along with the captured image position and the world coordinate system position of the reference feature point, again provisional calibration Calculate parameters.

The controller 20 again converts the position of the feature point in the world coordinate system to the position in the captured image using the conversion formula or conversion table calibrated using the temporary calibration parameters. The controller 20 calculates the distance between the converted position in the captured image and the position in the captured image of the corresponding feature point acquired from the terminal device for manual calibration.

The controller 20 repeats the calculation of temporary calibration parameters, the conversion from the position in the world coordinate system to the position in the captured image, and the calculation of intervals until the number of reference feature points reaches or exceeds the number threshold. The number threshold value may be any number that is equal to or greater than the number required for calibration, and is preferably 5 or more, more preferably 10 or more, and even more preferably 20 or more.

The controller 20 determines the temporary calibration parameters when the number of reference feature points is greater than or equal to the reference threshold as the initial values of the calibration parameters. The controller 20 calibrates the conversion formula or conversion table in the memory 19 using the determined calibration parameters, and stores it again in the memory 19 .

The controller 20 converts the position of the reference feature point in the world coordinate system to the position in the captured image using a calibrated conversion formula or conversion table. The controller 20 stores the converted position in the captured image in the memory 19 as the reference position of the feature point. The controller 20 extracts the feature amount of the reference feature points from the captured image. The controller 20 stores the extracted feature quantity in the memory 19 .

In other words, when the controller 20 acquires a new captured image from the imaging unit 12 at a time different from the arbitrary time at which the captured image used for calculating the reference position is captured after the manual calibration ends. , automatic calibration can be performed. Automatic calibration may be performed each time a captured image of one frame is acquired, may be performed at predetermined frame intervals, or may be performed periodically. In this embodiment, automatic calibration is performed at a predetermined frame period.

For automatic calibration, the controller 20 detects a plurality of reference feature points from the newly acquired captured image based on the feature amount stored in the memory 19 . After performing manual calibration, points in real space corresponding to reference feature points may be covered by obstacles such as vehicles. In such a case, the reference feature point may not be detected from the captured image. If the number of detected reference feature points is outside the execution reference range, the controller 20 stops calculating subsequent calibration parameters for the captured image. The execution reference range may be the minimum number of feature points required to calculate the calibration parameters, for example, a range of 5 or more, or the number of reference feature points stored in the memory 19, such as 60%. may be in the range of a predetermined ratio or more.

If the number of detected reference feature points is within the execution reference range, the controller 20 detects the positions of a plurality of reference feature points in the newly acquired captured image. The controller 20 determines whether the position of the detected reference feature point in the captured image has changed from the reference position stored in the memory 19 by an interval threshold or more.

If the number of reference feature points whose position in the captured image has changed from the reference position by the interval threshold or more is equal to or greater than a predetermined ratio of the number of detected reference feature points, the controller 20 determines whether each of the plurality of reference feature points , the detected positions in the captured image and the positions in the world coordinate system stored in the memory 19 are used to recalculate the calibration parameters. The predetermined ratio is an appropriate ratio at which it can be determined that the position or orientation of the imaging unit 12 has changed, and may be, for example, 50%, preferably 30%, and more preferably 20%. %. If part of all the reference feature points stored in the memory 19 cannot be detected from the new captured image, the controller 20 uses the detected feature points that are part of all the reference feature points. to calculate the calibration parameters. If the number of reference feature points whose position in the captured image has changed from the reference position by the interval threshold or more is less than a predetermined ratio of the number of detected reference feature points, the controller 20 calculates subsequent calibration parameters. Not performed.

Controller 20 calibrates the conversion formula or conversion table stored in memory 19 using the recalculated calibration parameters. The controller 20 stores the converted conversion formula or conversion table in the memory 19 after calibration. The controller 20 converts the position of the reference feature point stored in the memory 19 in the world coordinate system to the position in the captured image using a calibrated conversion formula or conversion table. The controller 20 stores the converted position in the captured image in the memory 19 as a new reference position. The controller 20 uses the new reference position to perform subsequent automatic calibration.

After executing manual calibration or automatic calibration, the controller 20 determines whether there is an object to be measured around the roadside unit 11 and calculates the position of the object to be measured.

When the controller 20 acquires one frame of captured image from the imaging unit 12, the controller 20 determines whether or not the object to be measured is included. The controller 20 determines whether or not the measurement target is included based on whether the measurement target can be detected in the captured image. The controller 20 detects, for example, by a conventionally known detection method such as pattern matching or deep learning.

Upon detecting the object to be measured, the controller 20 detects the position of the lower end of the object to be measured on the side of the road surface or floor. The controller 20 converts the detected position into a position in the world coordinate system using a conversion formula or conversion table stored in the memory 19 . The controller 20 controls the information interface 18 so as to transmit the measurement object and the position of the world coordinate system to the external device.

Next, manual calibration processing executed by the controller 20 in this embodiment will be described using the flowchart of FIG. The manual calibration process starts when a request for manual calibration is obtained from the terminal for manual calibration.

In step S100, the controller 20 controls the information interface 18 to transmit the captured image acquired from the imaging unit 12 to the terminal device for manual calibration. After transmission, the process proceeds to step S101.

In step S101, the controller 20 acquires the position in the world coordinate system and the position in the captured image of each of the plurality of specific feature points from the terminal device for manual calibration that transmitted the captured image in step S100. determine. If not, the process returns to step S101. If so, the process proceeds to step S102.

In step S102, the controller 20 determines the position of each of the plurality of specific feature points confirmed to be acquired in step S101 in the world coordinate system and the captured image, and the position of the feature point confirmed to be acquired in step S109. Based on this, a provisional calibration parameter is calculated. After the calculation, the process proceeds to step S103.

In step S103, the controller 20 converts the positions in the world coordinate system of each of the plurality of specific feature points confirmed to be acquired in step S101 into positions in the captured image based on the temporary calibration parameters calculated in step S103. do. After the return, the process proceeds to step S104.

In step S104, the controller 20 calculates, for all feature points, the distance between the positions of the corresponding feature points in the captured image whose acquisition was confirmed in step S101 and the positions in the captured image converted in step S103. do. After the calculation, the process proceeds to step S105.

In step S105, the controller 20 temporarily determines a feature point whose interval calculated in step S104 is less than the interval threshold as a reference specific point. After determination, the process proceeds to step S106

In step S106, the controller 20 determines whether or not the number of reference specific points temporarily determined in step S105 is greater than or equal to the number threshold. If not, the process proceeds to step S107. If so, the process proceeds to step S110.

In step S107, the controller 20 confirms the acquisition in step S101 of the feature points for which the interval calculated in step S104 is equal to or greater than the interval threshold value, that is, the feature points other than the reference feature points temporarily determined in step S105. Two distinguishable point images are superimposed on the captured image at each of the positions in the captured image converted in step S103 and the positions in the captured image converted in step S103. After superimposition, the process proceeds to step S108.

In step S108, the controller 20 controls the information interface 18 so as to transmit the captured image on which the point images are superimposed in step S107 to the terminal device for manual calibration. After transmission, the process proceeds to step S109.

In step S109, the controller 20 acquires the re-specified positions in the world coordinate system and the positions in the captured image of each of the plurality of specific feature points from the terminal device for manual calibration that transmitted the captured image in step S109. determine whether or not If not, the process returns to step S109. If so, the process returns to step S102.

In step S110, the controller 20 determines the temporary calibration parameter calculated in step S102 as the initial value of the calibration parameter. The controller 20 calibrates the conversion formula or conversion table in the memory 19 using the determined calibration parameters, and stores it again in the memory 19 . The controller 20 determines the reference specific point temporarily determined in step S105 as the reference specific point to be used thereafter. After confirmation, the process proceeds to step S111.

In step S111, the controller 20 uses the conversion formula or conversion table calibrated in step S110 to convert the positions of each of the plurality of specific feature points confirmed to be acquired in step S101 in the world coordinate system to the positions in the captured image. A reference position is calculated by conversion. The controller 20 stores the calculated reference position in the memory 19 . After the calculation, the process proceeds to step S112.

In step S112, the controller 20 extracts the feature amount of the reference specific point determined in step S110 from the captured image. The controller 20 associates the extracted feature amounts with each reference feature point and stores them in the memory 19 . After storage, the manual calibration process ends.

Next, automatic calibration processing executed by the controller 20 in this embodiment will be described using the flowchart of FIG. The automatic calibration process starts at a predetermined frame period after the manual calibration process ends.

In step S<b>200 , the controller 20 detects reference feature points from the newly acquired captured image based on the feature amounts of each of the plurality of reference feature points stored in the memory 19 . After detection, the process proceeds to step S201.

In step S201, the controller 20 determines whether or not the number of reference feature points detected in step S200 is within the execution reference range. If not within the execution reference range, the auto-calibration process ends. If so, the process proceeds to step S202.

In step S202, the controller 20 detects the position of the reference feature point in the captured image. After detection, the process proceeds to step S203.

In step S203, the controller 20 detects the number of reference feature points whose positions in the captured image detected in step S202 differ from the reference positions stored in the memory 19 by an interval threshold or more, detected in step S200. It is determined whether or not the number is equal to or more than half of the number of reference feature points. If not, the auto-calibration process ends. If so, the process proceeds to step S204.

In step S204, the controller 20 recalculates the calibration parameters using the positions of the plurality of reference feature points in the captured image detected in step S202 and the positions in the world coordinate system stored in the memory 19. . The controller 20 calibrates the conversion formula or conversion table in the memory 19 using the recalculated calibration parameters and stores them again in the memory 19 . After recalculation, the process proceeds to step S205.

In step S205, the controller 20 calculates the reference position by converting the position in the world coordinate system stored in the memory 19 into the position in the captured image using the conversion formula or conversion table calibrated in step S204. . The controller 20 stores the calculated reference position in the memory 19 . After the calculation, the automatic calibration process ends.

The image processing apparatus 10 of the present embodiment configured as described above includes a memory 19 for storing, as a reference position, the position of each of a plurality of specific feature points captured at any time in the captured image in the imaging unit 12, A plurality of reference feature points are detected in the captured image captured by the imaging unit 12, and if the positions in the captured image of more than half of the plurality of reference feature points have changed from the reference positions, the calibration parameters are re-set. calculate. With such a configuration, the image processing apparatus 10 can determine a point whose position in the world coordinate system does not change continuously, such as the corner of a white line on the road surface, as a reference feature point, and store the reference position. Thus, calibration parameters can be calculated without arranging a mark within the angle of view of the imaging unit 12 for calibration. Further, the image processing apparatus 10 estimates the deviation of the position or orientation of the imaging unit 12 from the change in the reference position, and calculates the calibration parameters when the deviation becomes large. Therefore, the calibration is executed when necessary. obtain. Therefore, the image processing apparatus 10 can simply calibrate the imaging unit 12 .

Further, the image processing apparatus 10 of the present embodiment calculates the initial values of the calibration parameters based on the positions of the plurality of specific feature points in the captured image and the positions of the world coordinate system obtained from the information interface 18. . With such a configuration, the image processing apparatus 10 can acquire the position in the captured image and the position in the world coordinate system of the specific feature point specified by the user of the terminal device for manual calibration from the terminal device. be. Therefore, the image processing apparatus 10 can calculate the calibration parameters using the feature point positions that reduce the possibility of misrecognition by the computer. Therefore, the image processing device 10 can perform highly accurate calibration.

Further, in the image processing apparatus 10 of the present embodiment, the position in the world coordinate system includes the position calculated based on the signal received by the GPS receiver at the point corresponding to the feature point. With such a configuration, the image processing apparatus 10 eliminates the need for the user to specify the position in the world coordinate system. Therefore, the image processing apparatus 10 can make the initial calibration even easier.

Further, in the image processing apparatus 10 of the present embodiment, the position of the world coordinate system includes the position designated on the map or the survey map. With such a configuration, the image processing apparatus 10 eliminates the need to place the GPS receiver in real space. Therefore, the image processing device 10 can reduce the preparation for initial calibration compared to the method of mounting the GPS receiver.

Further, the image processing apparatus 10 of the present embodiment stores in the memory 19 the positions in the captured image of the plurality of reference feature points transformed from the positions in the world coordinate system using the calibration parameters as reference positions. With such a configuration, the image processing apparatus 10 can perform the recalculation of the calibration parameters, and the position of the feature point specified in the captured image at any time as a target for comparison with the position of the feature point in the new captured image. Rather than position, the suitable position may be used.

Further, when the image processing apparatus 10 of the present embodiment cannot detect some of the plurality of reference feature points in the captured image, the image processing apparatus 10 performs calibration using reference feature points other than the one of the plurality of reference feature points. Calculate parameters. With such a configuration, the image processing apparatus 10 can calculate the calibration parameters even if all the reference feature points cannot be detected due to frequent travel of the vehicle or the like.

Further, the image processing apparatus 10 of the present embodiment stops calculating the calibration parameters when the number of reference feature points that can be detected in the captured image is outside the execution reference range. In calculating the calibration parameters, there is a lower limit to the number of feature points whose positions in the captured image and the positions in the world coordinate system are known, and the greater the number, the higher the calibration accuracy. Therefore, the image processing apparatus 10 having the configuration described above prevents the calculation of the calibration parameters when the calibration parameters cannot be calculated or when the calibration accuracy is low. As a result, the image processing apparatus 10 can calculate calibration parameters with high calibration accuracy.

Although the present invention has been described with reference to the drawings and examples, it should be noted that various variations and modifications will be readily apparent to those skilled in the art based on this disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention.

For example, in the present embodiment, the image processing apparatus 10 is configured to calculate initial values of calibration parameters, calculate reference positions, and extract feature amounts of reference feature points in manual calibration. Part may be performed by the terminal for manual calibration. Even if the image processing apparatus 10 acquires the initial values of the calibration parameters, the reference position, and the feature amount from the terminal device for manual calibration, the same effects as in the present embodiment can be obtained.

REFERENCE SIGNS LIST 10 image processing device 11 roadside device 12 imaging unit 13 imaging device 14 memory 15 communication unit 16 controller 17 image interface 18 information interface 19 memory 20 controller

Claims (10)

an image interface that acquires a captured image from the imaging unit;
Positions in a world coordinate system of each of a plurality of specific feature points on the road surface or the floor surface of the captured image captured by the imaging unit, and the plurality of specific feature points captured by the imaging unit at an arbitrary time a memory that stores the position in each captured image as a reference position;
A controller capable of detecting the plurality of specific feature points in the captured image and calculating calibration parameters for calibrating the position and orientation of the imaging unit based on the plurality of specific feature points and the position of the world coordinate system. and
The controller detects the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time, and detects a predetermined ratio or more of the plurality of specific feature points. An image processing device that recalculates the calibration parameter when a position in the captured image has changed from the reference position. The image processing device according to claim 1,
an information interface that acquires the positions of the plurality of specific feature points in the captured image captured by the imaging unit and the positions of each of the plurality of specific feature points in a world coordinate system,
The controller calculates an initial value of the calibration parameter based on the positions of the plurality of specific feature points in the captured image and the position of the world coordinate system obtained from the information interface. In the image processing device according to claim 2,
The position in the world coordinate system acquired by the information interface includes a position calculated based on signals received by a GPS receiver at points corresponding to feature points. In the image processing device according to claim 2,
The position of the world coordinate system acquired by the information interface includes a position designated on a map or a survey map. In the image processing device according to any one of claims 1 to 4,
The controller stores in the memory, as the reference position, the positions of the plurality of specific feature points in the captured image that are transformed from the positions in the world coordinate system using the calibration parameters. In the image processing device according to any one of claims 1 to 5,
When a part of the plurality of specific feature points cannot be detected in the captured image, the controller calculates the calibration parameter using feature points other than the part of the plurality of specific feature points. Image processing device. In the image processing device according to claim 6,
The controller stops calculating the calibration parameter when the number of the plurality of specific feature points that can be detected in the captured image is outside an execution reference range. an imaging unit that generates a captured image by imaging;
Positions in a world coordinate system of each of a plurality of specific feature points on a road surface or a floor surface of an image captured by the imaging unit, and the plurality of specific feature points captured by the imaging unit at an arbitrary time a memory that stores the position in each captured image as a reference position;
A controller capable of detecting the plurality of specific feature points in the captured image and calculating calibration parameters for calibrating the position and orientation of the imaging unit based on the plurality of specific feature points and the position of the world coordinate system. and
The controller detects the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time, and detects a predetermined ratio or more of the plurality of specific feature points. An imaging device that recalculates the calibration parameters when a position in the captured image has changed from the reference position. an imaging unit that generates a captured image by imaging;
Positions in a world coordinate system of each of a plurality of specific feature points on a road surface or a floor surface of an image captured by the imaging unit, and the plurality of specific feature points captured by the imaging unit at an arbitrary time a memory that stores the position in each captured image as a reference position;
A controller capable of detecting the plurality of specific feature points in the captured image and calculating calibration parameters for calibrating the position and orientation of the imaging unit based on the plurality of specific feature points and the position of the world coordinate system. and
The controller detects the plurality of specific feature points in the captured image captured by the imaging unit at a time different from the arbitrary time, and detects a predetermined ratio or more of the plurality of specific feature points. A roadside unit that recalculates the calibration parameters if the position in the captured image has changed from the reference position. An image processing method by an image processing device,
The image processing device is
Positions in a world coordinate system of each of a plurality of specific feature points on a road surface or a floor surface of an image captured by an imaging unit, and each of the plurality of specific feature points captured by the imaging unit at an arbitrary time. The position in the captured image of is stored as a reference position,
Detecting the plurality of specific feature points in an image captured by the imaging unit at a time different from the arbitrary time,
when positions in the captured image of a predetermined ratio or more among the plurality of specific feature points have changed from the reference position, based on the plurality of specific feature points and the position of the world coordinate system An image processing method for calculating calibration parameters for calibrating the position and orientation of an imaging unit.

Images