JP7633143B2 - Stereo camera, stereo camera system, and arrangement method - Google Patents

Description

The present invention generally relates to a stereo camera for detecting, ranging and/or identifying objects.

In recent years, a technology has been disclosed for a stereo camera having a mechanism capable of controlling camera settings such as zoom, pan, and tilt, which has at least two cameras and estimates the camera parameters when the camera lens position is changed by an instruction to operate the camera (see Patent Document 1).

Also disclosed is an image capture device equipped with a rotation mechanism that rotates an imaging camera in at least one direction, in which image data output from the image capture device has fθ characteristics, and the center of rotation of the rotation mechanism is approximately at the center of the entrance pupil of the optical system (see Patent Document 2).

JP 2017-135495 A JP 2004-289367 A

Object detection systems using cameras need to be able to detect objects at longer distances and with wider angles in response to demands for wider detection ranges. Cameras play an important role as visual recognition sensors, and so longer distances can be achieved by lengthening the focal length of the lens, but the image sensor size cannot be increased proportionally, resulting in a smaller and narrower angle of view. Conversely, to widen the angle of view, the focal length must be shortened, resulting in a loss of resolution and making it difficult to detect objects at longer distances.

One way to deal with this is to use a lens with distortion, such as a fisheye lens, to achieve high resolution near the optical axis and a wide angle by compressing the periphery. However, the projection methods used in fisheye lenses, equidistant projection (f θ), equisolid angle projection (2f sin(θ/2)), and orthogonal projection (f sinθ), do not provide sufficient results unless there is a certain angle of view.

On the other hand, the pan and tilt operations that mechanically change the orientation of the camera, as described in Patent Documents 1 and 2, are a rational method of ensuring long-distance performance with the optical performance of the camera, and realizing a wide angle with the pan and tilt operations. However, Patent Documents 1 and 2 make no mention of cases where the entire angle of view needs to be detected simultaneously, and the technologies described in Patent Documents 1 and 2 make it difficult to realize long-distance and wide-angle views.

The present invention was made in consideration of the above points, and aims to propose a stereo camera etc. that can achieve long distance and wide angle.

In order to solve this problem, the present invention provides a stereo camera including a first camera, a first rotation mechanism for rotating the first camera, a second camera, and a second rotation mechanism for rotating the second camera, in which a first rotation axis of the first rotation mechanism and a second rotation axis of the second rotation mechanism are each perpendicular to a plane including both a first optical axis of the first camera and a second optical axis of the second camera, the first rotation axis intersects with the first optical axis, and the second rotation axis intersects with the second optical axis, such that the distance from the intersection of the first rotation axis and the first optical axis to the first entrance pupil position of the first camera is equal to the distance from the intersection of the second rotation axis and the second optical axis to the second entrance pupil position of the second camera.

In the above configuration, the distance from the rotation axis of the rotation mechanism to the entrance pupil position of each camera is the same, so for example, by performing coordinate transformation of the image using rotation angle information, it is possible to achieve a longer distance and wider angle detection range while maintaining the distance measurement accuracy of the detected object.

The present invention makes it possible to realize a stereo camera that can handle longer distances and wider angles. Other issues, configurations, and advantages will become clear from the description of the embodiments below.

FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. FIG. 1 is a diagram illustrating an example of a stereo camera for explaining a first embodiment. FIG. 11 is a diagram showing distance measurement errors for explaining the first embodiment; 1 is a diagram illustrating an example of a configuration of an object detection system according to a first embodiment. 3A and 3B are diagrams illustrating the positional relationship between a camera and a rotation mechanism according to the first embodiment. FIG. 2 is a diagram for explaining coordinate transformation according to the first embodiment. FIG. 4 is a diagram for explaining distance calculation according to the first embodiment; 10A and 10B are diagrams for explaining inconveniences caused when the rotation axis is not at the entrance pupil in the first embodiment; FIG. 4 is a diagram for explaining calibration according to the first embodiment. FIG. 4 is a diagram for explaining calibration according to the first embodiment. 13 is a diagram showing the positional relationship between a camera and a rotation mechanism according to the second embodiment. FIG. 13A and 13B are diagrams for explaining a method for solving the problem when the rotation axis is not at the entrance pupil according to the second embodiment. FIG. 11 is a diagram for explaining coordinate transformation according to the second embodiment.

(I) First Embodiment Hereinafter, one embodiment of the present invention will be described in detail. However, the present invention is not limited to the embodiment.

In this embodiment, we will particularly describe a stereo camera that has a long-distance, wide FOV (Field of view) detection range and high-precision distance measurement performance, and a stereo camera system that uses the stereo camera.

The stereo camera system described in this embodiment includes multiple means for solving the above problem. For example, the stereo camera is characterized in that "camera 1 and camera 2 constituting the stereo camera each have a rotation mechanism for changing the shooting direction, and when the entrance pupil positions of each camera are P1 and P2 and a line including P1 and P2 is L, the rotation axes R1 and R2 of the rotation mechanisms are perpendicular to the line L passing through P1 and P2 and the optical axes O1 and O2 of each camera, and are parallel to each other." In addition, the stereo camera system is characterized in that "based on the rotation angle θ of the rotation mechanism, correction processing is performed on at least one of the captured images of each camera or processed images obtained by performing image processing on the captured images."

Each stereo camera has a rotation mechanism that changes the shooting direction, and by aligning the rotation axis of the rotation mechanism with the entrance pupil position of each camera and using the rotation angle information to perform coordinate conversion of the image, it is possible to achieve a longer distance and wider angle detection range and space saving while maintaining the distance measurement accuracy of the detected object.

The designations "first," "second," "third," and the like in this specification are used to identify components and do not necessarily limit the number or order. Furthermore, numbers for identifying components are used in different contexts, and a number used in one context does not necessarily indicate the same configuration in another context. Furthermore, this does not prevent a component identified by a certain number from also serving the function of a component identified by another number.

Next, an embodiment of the present invention will be described with reference to the drawings. The following description and drawings are examples for explaining the present invention, and some parts have been omitted or simplified as appropriate for clarity of explanation. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

In the following description, identical elements in the drawings are given the same numbers and descriptions are omitted as appropriate. When describing similar elements without distinguishing between them, the common portion (excluding the branch number) of the reference sign including the branch number is used, and when describing similar elements with distinction, the reference sign including the branch number may be used. For example, when describing a camera without distinction between them, it may be written as "camera 401", and when describing individual cameras with distinction between them, it may be written as "camera 401-1", "camera 401-2", etc.

[Each camera rotates to save space. The diagonal model has distance measurement errors.]
Generally, the method of changing the camera orientation causes some inconveniences. In central projection (f tan θ), which is a normal camera projection method, the image appears larger the further away from the optical axis. For example, the width of a person standing at the corner of the camera screen appears wider on the image than the width of a person standing in front of the camera. This can be solved by using equidistant projection (f θ). When equidistant projection is used, the image of an object is equal regardless of the angle of view, so the image size of the same object can be kept constant even if the camera orientation is changed by panning, tilting, etc. In addition, by aligning the rotation axis of the camera with the entrance pupil of the camera, the distance to the object can be accurately kept constant even if the camera orientation is changed, and the variation in image size due to the angle of view can be further reduced.

Here, when changing the shooting direction in a stereo camera, the relative positions and orientations of the two cameras must be accurately maintained. For this reason, as shown in Figures 1, 2, and 3, changing the orientation of the stereo camera 100 as a whole and maintaining the relative positions of the cameras 101 can more easily achieve a wide angle. However, unlike a monocular camera, when changing the orientation of the stereo camera 100, it is necessary to ensure a large space where the cameras 101 and the fixing mechanism for the cameras 101 do not interfere with other components. This increases depending on the required rotation angle and baseline length of the cameras 101, so it is difficult to install the stereo camera 100, which is particularly suitable for long distances and has a long baseline, in a narrow driver's seat or cab in an automobile, train, etc.

Therefore, as shown in Figures 4, 5, 6, etc., by configuring each camera 401 of the stereo camera 400 to have an independent rotation axis, it is possible to achieve a wide angle without providing a space larger than the horizontal projection area of the device. However, this method has the disadvantage that an error occurs in the distance measurement result when the orientation of the camera 401 is tilted. For example, if the case of Figure 5 is replaced with that of Figure 7, the image sizes I1 and I2 of the camera 401-1 (camera 1) and the camera 401-2 (camera 2) are respectively:
I1=Ob/L1・fc...Formula (1)
I2=Ob/L2・fc=Ob/(L1−B・sin(Δθ))・fc...Formula (2)
It becomes.
Here, Ob is the size of the target object, L1 and L2 are the distances from camera 1 and camera 2 to the target object, respectively, fc is the focal length of camera 1 and camera 2, B is the base line length, and Δθ is the orientation (rotation angle) of camera 1 and camera 2. It can be seen from both equations that the image sizes of the target object obtained by camera 1 and camera 2 are different. In addition, since the ratio of the image sizes also differs depending on the distance to the target object, simply enlarging or reducing one of the images will result in a distance measurement error.

In this embodiment, as an example of a stereo camera system, a stereo camera mounted on a train and an object detection system 800 using the stereo camera will be described. FIG. 8 shows a schematic configuration of the object detection system 800. First, the role of each block will be described. Reference numerals 801 and 802 indicate cameras constituting the stereo camera. Reference numerals 811 and 812 indicate a rotation mechanism (ROTATION MECH.) that changes the shooting direction of the cameras 801 and 802. Reference numeral 820 indicates a controller unit (CAMERA & ANGLE CONTROLLER) that controls the shooting timing of the cameras 801 and 802 and the operation of the rotation mechanisms 811 and 812. Reference numeral 830 indicates a geometric correction unit (GEOMETRIC CORRECTION) that corrects image distortion, image orientation, image size, etc. for the images acquired by the cameras 801 and 802. Reference numeral 840 indicates a coordinate conversion unit (COORDINATE CORRECTION) that mainly performs coordinate conversion of pixel values based on the rotation angle of the rotation mechanisms 811 and 812. Reference numeral 850 denotes a disparity image generating unit (DISPARITY) that generates a disparity image using an image that has been subjected to geometric correction and coordinate transformation. Reference numeral 860 denotes an object detection unit (IMAGE ANALYSIS OBJECT DETECTION) that performs distance measurement, image analysis, object detection, etc. using the disparity image and the original image.

The functions of the object detection system 800 (controller unit 820, geometric correction unit 830, coordinate conversion unit 840, parallax image generation unit 850, object detection unit 860, etc.) may be realized, for example, by a processor reading a program stored in an auxiliary storage device into a main storage device and executing it (software), or may be realized by hardware such as a dedicated circuit, or may be realized by a combination of software and hardware. Note that one function of the object detection system 800 may be divided into multiple functions, or multiple functions may be combined into one function. For example, the geometric correction unit 830, the coordinate conversion unit 840, and the parallax image generation unit 850 may be a correction unit. In addition, some of the functions of the object detection system 800 may be provided as separate functions or may be included in other functions. For example, geometric correction and coordinate conversion may be performed simultaneously. In addition, the stereo camera including the cameras 801 and 802 may include, for example, a controller unit 820, a geometric correction unit 830, a coordinate conversion unit 840, and a parallax image generation unit 850, and a computer capable of communicating with the stereo camera may include an object detection unit 860. Also, for example, the stereo camera including the cameras 801 and 802 may include, for example, a controller unit 820, and a computer capable of communicating with the stereo camera may include, for example, a geometric correction unit 830, a coordinate conversion unit 840, a parallax image generation unit 850, and an object detection unit 860. Also, some of the functions of the object detection system 800 may be realized by another computer capable of communicating with the object detection system 800.

Next, the relationship between each block will be described. The rotation mechanisms 811 and 812 rotate at a rotation angle instructed by the controller unit 820, and change the shooting direction of the cameras 801 and 802 fixed to each of the rotation mechanisms 811 and 812. The cameras 801 and 802 shoot at the timing instructed by the controller unit 820, and send the acquired images (image data) to the geometric correction unit 830. For each image sent to the geometric correction unit 830, the geometric correction unit 830 performs geometric correction processing including lens distortion correction and parallelization processing, and sends the processed geometrically corrected image to the coordinate conversion unit 840. The coordinate conversion unit 840 performs coordinate conversion of the geometrically corrected image sent from the geometric correction unit 830 based on the rotation angle information sent from the controller unit 820, and sends the coordinate conversion image to the parallax image generation unit 850 and the object detection unit 860. The parallax image generating unit 850 generates a parallax image using a coordinate transformation image based on the images sent from each of the cameras 801 and 802, and sends the generated parallax image to the object detection unit 860. The object detection unit 860 performs distance measurement, object detection, object identification, etc. using the sent parallax image and coordinate transformation image, and sends information necessary for vehicle control to a downstream system (not shown). For example, if the downstream system is a brake control system, the brake control system controls the vehicle brakes based on the received information (for example, information indicating that an obstacle has been detected on the track).

Figure 9 is a diagram showing the positional relationship between the cameras 801, 802 and the rotation mechanisms 811, 812. Reference numerals 901, 902 indicate the lenses of each camera 801, 802. Note that the cameras 801, 802 are provided with one or more lenses 901, 902, typically with multiple lenses, but FIG. 9 shows only one lens for ease of explanation. Reference numerals 911, 912 indicate the image pickup elements of each camera 801, 802. Reference numerals 921, 922 indicate the entrance pupil positions (entrance pupils) of each camera 801, 802. Reference numerals 931, 932 indicate the optical axes of each lens 901, 902 (cameras 801, 802). Reference numerals 941, 942 indicate the rotation axes of each rotation mechanism 811, 812 with dashed lines. These rotation axes 941, 942 pass through the entrance pupils 921, 922, respectively, and are arranged so as to be perpendicular to the optical axes 931, 932 and the line L. Here, the line L is a line that passes through both the entrance pupils 921, 922. In general, the optical axes of both cameras in a stereo camera are arranged to be parallel, so in other words, the rotation axes 941, 942 are arranged so as to be perpendicular to a plane that includes both optical axes 931, 932, and to pass through each entrance pupil 921, 922.

The rotation angle θ is based on the state in which the optical axes 931, 932 are perpendicular to the line L (θ = 0 deg), and the following explanation will be based on this premise.

[Conversion formula]
Next, the detailed operation of the coordinate conversion unit 840 will be described with reference to FIG. 10. FIG. 10 is a schematic diagram of the camera 801 facing in the direction of a rotation angle θ=Δθ, and also shows the arrangement of the camera 801 when θ=0 using a dotted line. The camera 801 in the figure is a central projection model after geometric correction, and is equivalent to a pinhole model. Reference numeral 1001 denotes a lens modeled after the lens 901. As shown in FIG. 10, basically, in the pinhole model, the pinhole position and the entrance pupil position of the camera 801 coincide with each other. Note that the camera 802 is similar to the camera 801, and a description thereof will be omitted.

The coordinate conversion by the coordinate conversion unit 840 is an operation of converting the value of an actual pixel B into the value of a virtual pixel A, where the pixel that a light ray incident from a certain direction reaches when the rotation angle of the camera 801 is Δθ is pixel B, and the pixel that the light ray reaches when the rotation angle θ=0 is pixel A. Specifically, when the coordinates of each pixel in the pre-conversion image are B(Ub, Vb), the coordinates of a reference pixel (e.g., center pixel) in the pre-conversion image are B(Ub0, Vb0), the coordinates of each pixel in the post-conversion image (post-correction image) are A(Ua, Va), the coordinates of a reference pixel (e.g., center pixel) in the post-conversion image are A(Ua0, Va0), the focal length of the camera 801 is fc, and the pixel pitches in the x and y directions are px and py, the coordinates after conversion are expressed as follows, using the coordinates before conversion and the rotation angle Δθ:
Ua=fc・tan(Δθ+α)/px+Ua0...Formula (3)
Va=Vb-Vb0+Va0...Formula (4)
α=atan((Ub-Ub0)・px/fc) ...Formula (5)
This can be expressed as:

When the rotation angle θ = 0, the light ray incident from the angle of view (Δθ + α) reaches outside the range of the image sensor 911, so the image sensor 911 cannot capture the image. Therefore, by changing the camera orientation by the rotation angle Δθ, the light ray reaches within the range of the image sensor 911, and α can be obtained from the coordinates of the imaged pixel. Therefore, the known rotation angle Δθ can be used to obtain the angle of incidence (Δθ + α) of the light ray, and the coordinates of the imaged pixel of the light ray when the rotation angle θ = 0 can be calculated.

[Disparity and distance obtained by transformation]
Next, an example of distance calculation of a target object will be shown using Fig. 11. Here, the target object is a point light source, and the cameras 801 and 802 are not shown, but the entrance pupils 921 and 922 of the cameras 801 and 802 are indicated by black circles. The U coordinate of the image-forming pixel of the light ray when the rotation angle θ = 0 is expressed as (Ua - Ua0) from the above formula, so if the same coordinates of the cameras 801 and 802 are expressed as (Ua1 - Ua10) and (Ua2 - Ua20), then
(Ua1-Ua10)=tan(Δθ+α)・fc/px...Formula (6)
(Ua2-Ua20)=tan(Δθ)・fc/px...Equation (7)
Here,
Y/L=tan(Δθ)...Formula (8)
(B+Y)/L=tan(Δθ+α)...Formula (9)
Since the parallax D is (Ua1-Ua10)-(Ua2-Ua20), the above formula gives:
D=B/L・fc/px...Formula (10)
Using the measured value of the parallax D, the distance L of the light spot can be obtained.
L=B/D・fc/px...Formula (11)

[What happens if the entrance pupil is not the rotation axis?]
Here, the inconvenience that occurs when the rotation axis 941 is not the entrance pupil 921 will be described using the camera 801. FIG. 12 is a diagram in which the rotation axis 941 is provided on the image sensor 911 and the camera orientation is set to a rotation angle Δθ. The dashed line indicates the optical axis 931 of the rotation angle Δθ of the camera 801, and the solid thick line indicates the image plane 1201 of the image sensor 911. The light ray direction β can be calculated from the coordinates of the image of the light point imaged on the image plane 1201 and the focal length. However, unlike the case of FIG. 11, the light ray direction (Δθ+α) in the camera orientation θ=0 state cannot be calculated even if the light ray direction β is used. This is because, even if it is known that the target object exists on the light ray direction β, the light ray direction in the camera orientation θ=0 state can take different values such as α0 and α1 depending on the distance of the target object. This occurs because the pinhole positions in the camera orientations θ=0 and Δθ do not match.

[Proofreading]
The above parallax and distance can be obtained accurately when the rotation mechanisms 811 and 812 operate ideally, but in reality, there are mechanical rotation errors and mechanical errors caused by the inclination and backlash of the rotation axes 941 and 942, which can cause non-negligible parallax errors and distance measurement errors, or problems such as the parallax not being obtained at all. In this case, calibration of the conversion formula according to the rotation angle Δθ is required. Calibration requires initial calibration corresponding to the initial mechanical error at the time of system manufacture, and time-dependent calibration corresponding to the mechanical error caused by changes over time. The purpose of time-dependent calibration can be achieved using the same equipment and method as the initial calibration, but it may also be operated so that the calibration operation is performed in the background during business hours. Below, the methods of initial calibration and time-dependent calibration will be described in order.

[Proofreading (initial, with chart)]
An example of initial calibration of the conversion equation according to the rotation angle Δθ will be described with reference to Figs. 13 and 14. In both figures, a stereo camera 1300 is shown at the bottom of the paper, and charts 1301 and 1302 are shown at the top of the paper. It is desirable that the charts 1301 and 1302 have a high-contrast pattern arranged all over so that parallax can be obtained. For example, checker marks, evenly spaced dots or doughnut-shaped marks, random patterns, etc. are available. The distance between the charts 1301 and 1302 and the cameras 801 and 802 is ideally set to a distance sufficient to allow focus and sufficient distance to allow both cameras 801 and 802 to capture a common area on the charts 1301 and 1302.

First, the chart image is acquired with the cameras 801 and 802 facing in such a way that the straight line L described in FIG. 9 and the optical axes 931 and 932 of the cameras 801 and 802 are perpendicular to each other. At this time, for example, the area a1 shown in FIG. 13 is captured on the left side of the camera 801 (left camera), and the area a2 is captured on the right side of the same camera 801.

Next, the orientation of cameras 801 and 802 is changed by half the angle of view to the state shown in FIG. 14. If the orientation of cameras 801 and 802 moves ideally, the image obtained by geometrically and coordinate-transforming area a2, which is the left half of the image acquired in the state of FIG. 14, should match the image obtained by geometrically transforming area a2 acquired in the state of FIG. 13. However, if there is a mechanical error during rotation, the two images will not match, so the amount of misalignment between corresponding pixels in both images is found and calibration is performed to eliminate the amount of misalignment. The amount of misalignment for each pixel can be obtained by block matching or feature point matching of both images, but any means may be used as long as the amount of misalignment can be obtained.

Area a3 can be calibrated by extrapolating the calibration of area a2. Thereafter, the amount of misalignment between the right half area a(n) of the image rotated n times the half angle of view and the left half area a(n+1) of the image rotated (n+1) times is calibrated. The area to the left of area a1 is also calibrated by changing the rotation angle to the negative side. The camera 802 on the right side of the paper in Figures 13 and 14 can be calibrated in the same way. The initial calibration is completed with the above procedure.

[Calibration (over time, no chart)]
The method of time-dependent calibration can be basically the same as that of initial calibration. In a maintenance yard where maintenance and repairs are performed, a dedicated chart can be provided or the shaking of the cameras 801 and 802 can be eliminated, so that the accuracy can be almost the same as that of the initial calibration. On the other hand, when performing calibration on a vehicle in operation, some ingenuity is required. In particular, when the cameras 801 and 802 are shaken, it is not possible to determine whether the mismatch of the images used for calibration is due to the shaking or mechanical error. Therefore, it is desirable to perform the calibration operation when there are no passengers or luggage getting on or off the vehicle, when there is no movement of passengers or luggage inside the vehicle, or when the same situation is detected. In addition, it is desirable to avoid environments that include low-contrast images that are unsuitable for block matching, feature point matching, etc., set conditions to obtain images with a certain level of contrast, accumulate calibration information, and build a mechanism to perform calibration after obtaining information of a certain level or more.

[Discrete value]
Considering the above calibration method, it is desirable that the camera orientation (rotation angle Δθ) has discrete values and that a calibration value is provided for each angle. It is also desirable that the actual shooting direction is changed discretely and that a calibration value is applied for each angle.

[Calibration does not have to be in units of half angle of view]
However, as described above, the discrete values do not need to match the half angle of view of the cameras 801 and 802. The finer the discrete values, the longer the calibration takes, but the higher the accuracy of extrapolation is expected, and high-precision distance measurement can be performed in practical use. Conversely, if the discrete interval is made larger, the calibration time can be shortened, but there is a concern that errors during extrapolation will increase.

[The accuracy of the extrapolated parts other than the left and right half pairs is questionable. Adjust the linearity and equal spacing as a whole.]
If errors due to extrapolation cause actual harm, calibration may be performed by arranging straight lines, evenly spaced marks, or orthogonal lines, etc., over the entire area of the charts 1301 and 1302, to ensure linearity, even spacing, and orthogonality in the calibrated image. This can eliminate practical problems.

[Landing from the same direction]
Furthermore, mechanical errors during rotation can be reduced by devising control over the rotation mechanisms 811 and 812. In other words, the controller unit 820 controls the rotation mechanisms 811 and 812 so that they always move in one direction and then stop, regardless of the rotation angle at which they are stopped. This reduces the hysteresis of the rotation mechanisms 811 and 812 and the backlash of the rotation mechanisms 811 and 812, and can reduce mechanical errors.

[Calibration formula]
When the calibration value in the x direction is Ka and the calibration value in the y direction is Kb,
Ua'=Ua+Ka...Formula (12)
Va'=Va+Kb...Formula (13)
Here, Ua=fc·tan(Δθ+α)/px+Ua0, and Va=Vb−Vb0+Va0.

As described above, by arranging the rotation axis that changes the camera orientation so that it passes through the entrance pupil and is perpendicular to the line connecting each optical axis and the entrance pupil, and performing a geometric transformation to central projection on the images captured by both cameras and a coordinate transformation based on the rotation angle information, it is possible to provide a stereo camera that can achieve long distances, a wide angle of view, and space savings while providing high-precision distance measurement performance, and an object detection system using this stereo camera.

(II) Second embodiment [Example in which the entrance pupil is not equal to the rotation axis. However, the rotation angles of the left and right cameras are made the same]
In the first embodiment, the rotation axes 941, 942 of the rotation mechanisms 811, 812 of the cameras 801, 802 pass through the entrance pupils 921, 922 of the cameras 801, 802, but similar effects can be obtained without being limited to this. That is, in Fig. 9, the rotation axes 941, 942 pass through the entrance pupils 921, 922, but in this embodiment, accurate distance measurement can be performed even if the rotation axes 941, 942 do not pass through the entrance pupils 921, 922, but instead pass through points on the optical axes 931, 932 that are equidistant and in the same direction from the entrance pupils 921, 922, as shown in Fig. 15.

The components of the object detection system 800 are the same as those in FIG. 8, which describes the first embodiment, and therefore are omitted. The relationship between the components is also the same as in the first embodiment, and therefore is omitted. The difference from the first embodiment is in the way of thinking about the entrance pupil position that moves when the cameras 801 and 802 face the rotation angle Δθ. In other words, in the first embodiment, it was shown using FIG. 12 that the entrance angle α cannot be calculated from the entrance angle β, but in this embodiment, it is assumed that the stereo camera itself has moved the same amount as the entrance pupils 921 and 922. Specifically, as shown in FIG. 16, when θ=0, the cameras 801 and 802 are arranged at positions where their optical axes 931 and 932 overlap with the dashed line, but when they face the direction of the rotation angle Δθ, it is assumed that the stereo camera has shifted by fc tan(Δθ) to the right of the paper and by (1-fc cos(Δθ)) below the paper. At this time, the orientation of the camera 801 and the orientation of the camera 802 are in the same state.

Thinking about it this way, the light ray directions β and γ are calculated from the coordinates of the light point imaged by camera 801 as shown in Figure 17, and using these values and the rotation angle Δθ, it is possible to obtain the coordinates of the imaged pixels of the light point in the stereo camera after shifting so that the black circles in the figure coincide with the entrance pupils 921, 922 and the optical axes 931, 932 coincide with the thick dotted lines. The shift amount of the stereo camera is minute compared to the calculated object position, and can therefore be ignored in many cases.

In this embodiment, the rotation axis of the rotation mechanism does not pass through the entrance pupil, but passes through a position equally shifted from the entrance pupil, and the orientation of both cameras is controlled to be equal, making it possible to perform accurate distance measurement using a coordinate conversion unit similar to that of the first embodiment. Therefore, it is possible to provide a stereo camera that achieves a wide angle of view, long distances, and has highly accurate distance measurement performance, and an object detection system using the stereo camera.

As described above, the rotation axis is perpendicular to the plane containing the optical axis of each camera and passes through each optical axis. Furthermore, the point where each rotation axis intersects with the optical axis can be equally expressed as the distance ΔX from the entrance pupil. The value of ΔX can be reduced by appropriate correction whether it is zero or a non-zero value, but the former allows coordinate transformation with simpler calculations. However, in general, it may be advantageous from the standpoint of mechanical error, stability, etc. to have the rotation axis closer to the center of gravity of the rotated body. From this point of view, it can also be said that it is reasonable to set the range of ΔX from the imaging surface of the image sensor to the frontmost surface of the lens.

(III) Supplementary Notes The above-described embodiment includes, for example, the following contents.

In the above embodiment, the present invention has been described as being applied to an object detection system for trains, but the present invention is not limited to this, and can be widely applied to a variety of systems, devices, and programs, including a method for an operator, manufacturing machine, or other task force to position the camera and rotation mechanism in a stereo camera.

In the above-described embodiment, some or all of the programs may be installed in the device from a program source. The program source may be, for example, a program distribution server connected via a network or a computer-readable recording medium (e.g., a non-transitory recording medium). In the above description, two or more programs may be realized as one program, and one program may be realized as two or more programs.

The above-described embodiment has the following characteristic configurations:

(1)
A stereo camera (e.g., stereo camera 1300) including a first camera (e.g., camera 801), a first rotation mechanism (e.g., rotation mechanism 811) that rotates the first camera, a second camera (e.g., camera 802), and a second rotation mechanism (e.g., rotation mechanism 812) that rotates the second camera, wherein each of a first rotation axis (e.g., rotation axis 941) of the first rotation mechanism and a second rotation axis (e.g., rotation axis 942) of the second rotation mechanism is a rotation axis of the first camera. The first rotation axis is perpendicular to a plane containing both a first optical axis (e.g., optical axis 931) and a second optical axis (e.g., optical axis 932) of the second camera, the first rotation axis intersects the first optical axis, and the second rotation axis intersects the second optical axis, and the distance from the intersection of the first rotation axis and the first optical axis to a first entrance pupil position (e.g., entrance pupil 921) of the first camera is equal to the distance from the intersection of the second rotation axis and the second optical axis to a second entrance pupil position (e.g., entrance pupil 922) of the second camera.

(2)
The first rotation axis passes through the first entrance pupil position, and the second rotation axis passes through the second entrance pupil position (see, for example, FIG. 9 ). In the above configuration, since the rotation axes pass through the entrance pupil position, for example, it is possible to perform coordinate transformation with high accuracy.

(3)
A stereo camera system (e.g., object detection system 800) using the stereo camera includes a correction unit (e.g., coordinate conversion unit 840) that performs a correction process (e.g., coordinate conversion) on at least one of a first image of the first camera or a processed image (e.g., geometric correction image) obtained by applying image processing to the first image (e.g., image captured by camera 801) based on a rotation angle of rotation angle information sent from a controller unit (e.g., controller unit 820) to the first rotation mechanism and the second rotation mechanism, and performs a correction process (e.g., coordinate conversion) on at least one of a second image of the second camera (e.g., image captured by camera 801) or a processed image (e.g., geometric correction image) obtained by applying image processing to the second image based on the rotation angle.

(4)
The correction unit includes a geometric correction unit (e.g., geometric correction unit 830) that performs geometric correction on each of the first image and the second image, a coordinate conversion unit (e.g., coordinate conversion unit 840) that performs coordinate conversion of pixel values based on the rotation angle for each of the first geometrically corrected image and the second geometrically corrected image that have been geometrically corrected by the geometric correction unit, and a parallax image generation unit (e.g., parallax image generation unit 850) that compares the first coordinate converted image and the second coordinate converted image that have been coordinate converted by the coordinate conversion unit to generate a parallax image including a parallax value for each pixel value.

(5)
The correction unit performs geometric correction and coordinate conversion of pixel values based on the rotation angle on the first image, and performs geometric correction and coordinate conversion of pixel values based on the rotation angle on the second image.

(6)
The coordinate conversion unit calculates, according to a rotation angle Δθ from a reference angle θ=0, the following: where each pixel value of the image before correction is B (Ub, Vb), a reference pixel of the image before correction is B (Ub0, Vb0), each pixel value of the image after correction is A (Ua, Va), a reference pixel of the image after correction is A (Ua0, Va0), a focal length is fc, and pixel pitches in the x and y directions are px, py.
Ua=fc・tan(Δθ+α)/px+Ua0...(Formula 1)
Va=Vb-Vb0+Va0...(Formula 2)
α=atan((Ub-Ub0)・px/fc)...(Formula 3)
(For example, see FIG. 10).

(7)
A rotation angle at which the first optical axis and the second optical axis are perpendicular to a line containing the first entrance pupil position and the second entrance pupil position is set as a reference angle, and the stereo camera system includes a controller unit (e.g., controller unit 820) that sends angle information indicating a rotation angle of a discrete value to the first rotation mechanism and the second rotation mechanism.

(8)
When at least one of the first camera and the second camera is used to acquire a reference image at a reference angle, a first image is acquired at a first rotation angle of the camera, and a second image is acquired at a second rotation angle of the camera, the correction unit calibrates the corrected image of the first image based on the reference image, and calibrates the corrected image of the second image based on the calibrated image of the first image (see, for example, Figure 13).

(9)
The angle difference between the n-th rotation angle and the (n+1)-th rotation angle for acquiring the images used for the calibration is equal to or smaller than a half angle of view (see FIG. 13, for example).

(10)
The correction unit uses the overlapping area between the first image and the second image to calibrate the correction formula for each pixel so that the corrected image of the second image matches the corrected image of the first image, and constructs the area of the second image other than the overlapping area by extrapolating from the calibration result of the overlapping area (e.g., see Figure 13).

(11)
The stereo camera system includes a controller unit (e.g., controller unit 820) that rotates the first camera and the second camera in one direction when positioning the first rotation mechanism and the second rotation mechanism at any rotation angle.

Furthermore, the above-mentioned configurations may be modified, rearranged, combined, or omitted as appropriate without departing from the spirit and scope of the present invention.

It should be understood that an item in a list in the format "at least one of A, B, and C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C). Similarly, an item in a list in the format "at least one of A, B, or C" can mean (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

800...object detection system, 801...camera, 802...camera, 811...rotation mechanism, 812...rotation mechanism, 921...entrance pupil (entrance pupil position), 922...entrance pupil (entrance pupil position), 931...optical axis, 932...optical axis, 941...rotation axis, 942...rotation axis.

Claims (10)

A stereo camera system using a stereo camera including a first camera, a first rotation mechanism that rotates the first camera, a second camera, and a second rotation mechanism that rotates the second camera ,
a first rotation axis of the first rotation mechanism and a second rotation axis of the second rotation mechanism are each perpendicular to a plane including both a first optical axis of the first camera and a second optical axis of the second camera;
the first rotation axis intersects the first optical axis, and the second rotation axis intersects the second optical axis;
a distance from an intersection of the first rotation axis and the first optical axis to a first entrance pupil position of the first camera is equal to a distance from an intersection of the second rotation axis and the second optical axis to a second entrance pupil position of the second camera;
a correction unit that performs a correction process on at least one of a first image of the first camera or a processed image obtained by performing image processing on the first image, based on a rotation angle of rotation angle information sent from a controller unit to the first rotation mechanism and the second rotation mechanism, and performs a correction process on at least one of a second image of the second camera or a processed image obtained by performing image processing on the second image, based on the rotation angle;
The correction unit is
a geometric correction unit that performs geometric correction on each of the first image and the second image;
a coordinate conversion unit that performs coordinate conversion of pixel values on each of the first geometrically corrected image and the second geometrically corrected image that have been geometrically corrected by the geometric correction unit, based on the rotation angle;
Equipped with
The coordinate conversion unit calculates, according to a rotation angle Δθ from a reference angle θ=0, the following: where each pixel value of the image before correction is B (Ub, Vb), a reference pixel of the image before correction is B (Ub0, Vb0), each pixel value of the image after correction is A (Ua, Va), a reference pixel of the image after correction is A (Ua0, Va0), a focal length is fc, and pixel pitches in the x and y directions are px, py:
Ua=fc・tan(Δθ+α)/px+Ua0...(Formula 1)
Va=Vb-Vb0+Va0...(Formula 2)
α=atan((Ub-Ub0)・px/fc)...(Formula 3)
Perform coordinate transformation according to
Stereo camera system . 2. The stereo camera system according to claim 1,
the first axis of rotation passes through the first entrance pupil position and the second axis of rotation passes through the second entrance pupil position.
Stereo camera system . 2. The stereo camera system according to claim 1 ,
The correction unit is
a parallax image generating unit that compares a first coordinate transformed image, which has been subjected to coordinate transformation by the coordinate transformation unit, with a second coordinate transformed image, to generate a parallax image including a parallax value for each pixel value;
Stereo camera system. 2. The stereo camera system according to claim 1 ,
the correction unit performs geometric correction and coordinate conversion of pixel values based on the rotation angle on the first image, and performs geometric correction and coordinate conversion of pixel values based on the rotation angle on the second image.
Stereo camera system. 2. The stereo camera system according to claim 1,
a rotation angle at which the first optical axis and the second optical axis are perpendicular to a line including the first entrance pupil position and the second entrance pupil position is set as a reference angle;
a controller unit that transmits angle information indicating a rotation angle of a discrete value to the first rotation mechanism and the second rotation mechanism;
Stereo camera system. 2. The stereo camera system according to claim 1 ,
when at least one of the first camera and the second camera is used to acquire a reference image at a reference angle, a first image is acquired at a first rotation angle of the camera, and a second image is acquired at a second rotation angle of the camera, the correction unit calibrates the corrected image of the first image based on the reference image, and calibrates the corrected image of the second image based on an image after calibration of the first image;
Stereo camera system. 7. The stereo camera system according to claim 6 ,
an angle difference between the n-th rotation angle and the (n+1)-th rotation angle for acquiring the image used for the calibration is equal to or smaller than a half angle of view;
Stereo camera system. 7. The stereo camera system according to claim 6 ,
the correction unit calibrates a correction equation for each pixel using an overlapping area between the first image and the second image so that a corrected image of the second image coincides with a corrected image of the first image, and configures an area of the second image other than the overlapping area by extrapolating from the calibration result of the overlapping area.
Stereo camera system. 2. The stereo camera system according to claim 1,
a controller unit that rotates the first camera and the second camera in one direction when the first rotation mechanism and the second rotation mechanism are positioned at any rotation angle;
Stereo camera system. A method for arranging a camera and a rotation mechanism in a stereo camera used in a stereo camera system , comprising:
the stereo camera includes a first camera, a first rotation mechanism that rotates the first camera, a second camera, and a second rotation mechanism that rotates the second camera;
a first rotation axis of the first rotation mechanism and a second rotation axis of the second rotation mechanism are each perpendicular to a plane including both a first optical axis of the first camera and a second optical axis of the second camera;
the first rotation axis intersects the first optical axis, and the second rotation axis intersects the second optical axis;
a distance from an intersection of the first rotation axis and the first optical axis to a first entrance pupil position of the first camera is equal to a distance from an intersection of the second rotation axis and the second optical axis to a second entrance pupil position of the second camera;
the stereo camera system includes a correction unit that performs a correction process on at least one of a first image of the first camera or a processed image obtained by performing image processing on the first image based on a rotation angle of rotation angle information sent from a controller unit to the first rotation mechanism and the second rotation mechanism, and performs a correction process on at least one of a second image of the second camera or a processed image obtained by performing image processing on the second image based on the rotation angle;
The correction unit is
a geometric correction unit that performs geometric correction on each of the first image and the second image;
a coordinate conversion unit that performs coordinate conversion of pixel values on each of the first geometrically corrected image and the second geometrically corrected image that have been geometrically corrected by the geometric correction unit, based on the rotation angle;
Equipped with
The coordinate conversion unit calculates, according to a rotation angle Δθ from a reference angle θ=0, the following: where each pixel value of the image before correction is B (Ub, Vb), a reference pixel of the image before correction is B (Ub0, Vb0), each pixel value of the image after correction is A (Ua, Va), a reference pixel of the image after correction is A (Ua0, Va0), a focal length is fc, and pixel pitches in the x and y directions are px, py:
Ua=fc・tan(Δθ+α)/px+Ua0...(Formula 1)
Va=Vb-Vb0+Va0...(Formula 2)
α=atan((Ub-Ub0)・px/fc)...(Formula 3)
Perform coordinate transformation according to
Placement method.

Images