JP7660964B2 - Imaging device and program - Google Patents

Description

The present invention relates to an imaging device and a program.

Conventionally, stereo cameras, which have two cameras arranged on the left and right, are mounted on vehicles such as automobiles and are used to measure distance information from the vehicle to an object. In addition to distance information, stereo cameras are also used to recognize various objects around the vehicle. Such stereo cameras contribute to improving safety during vehicle driving assistance.

Note that object recognition can be performed using only images captured by a single camera, even without distance information. For this reason, it has been considered to configure a stereo camera with cameras that have different angles of view, namely a wide-angle camera and a narrow-angle camera. With such a stereo camera, object recognition can be performed over a wide range using the wide-angle camera, and the distance to the object can be measured in the area where the imaging ranges of the narrow-angle camera and wide-angle camera overlap.

Patent Document 1 discloses a technique for calculating a vertical timing delay time to correct the misalignment from the amount of misalignment between the trimming positions of the left and right images, and instructing a vertical timing adjuster to make the adjustment so that the vertical misalignment approaches zero spatially. By making such adjustments, it is possible to obtain a high-quality stereoscopic image with no misalignment between the left and right cameras.

However, when the rolling shutter method is used in conventional in-vehicle stereo cameras equipped with cameras with different angles of view, there is a problem that the scanning timing of the left and right cameras of the in-vehicle stereo camera is misaligned.

More specifically, the rolling shutter method scans pixels in the sensor in sequence. For this reason, if the left and right cameras have different angles of view (amount of pixels scanned), there will be a timing discrepancy during scanning even if they start scanning at the same time. If a timing discrepancy occurs during scanning like this, the same object will be captured at different times by the left and right cameras, resulting in errors in the distance information.

The present invention has been made in consideration of the above, and aims to reduce the misalignment in scanning timing that occurs between left and right imaging units that have different angles of view.

In order to solve the above-mentioned problems and achieve the object, the present invention provides a method and apparatus for detecting a scanning start timing of one of the first imaging unit and the second imaging unit by a delay time which is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary area in a first captured image captured by the first imaging unit and an arbitrary area in a second captured image captured by the second imaging unit, wherein the arbitrary area in the first captured image and the arbitrary area in the second captured image are areas in which the same subject exists, and the delay instruction unit delays the scanning start timing so that a difference in scanning time to the subject approaches zero .

The present invention has the advantage of being able to reduce the misalignment in scanning timing that occurs between left and right imaging units with different angles of view.

FIG. 1 is a schematic plan view showing a configuration of an imaging device according to a first embodiment. FIG. 2 is a schematic plan view showing the configuration of the first imaging section and the second imaging section. FIG. 3 is a block diagram showing the hardware configuration of a control system of the imaging apparatus. FIG. 4 is a diagram showing an example of magnification adjustment and trimming in the correction unit. FIG. 5 is a diagram showing an example of distance measurement by the imaging apparatus. FIG. 6 is a diagram showing an example of the horizontal angle of view of the first and second imaging units when they are arranged so that the Z axis is the optical axis direction. FIG. 7 is a diagram showing an example of the vertical angle of view of the first and second imaging units when the Z axis is set as the optical axis direction. FIG. 8 is a diagram illustrating an example of the difference in scanning timing between the image pickup element of the first imaging unit and the image pickup element of the second imaging unit. FIG. 9 is a diagram showing an example of the difference in scanning timing between front and rear lines of a subject. FIG. 10 is a diagram showing a case where the scanning timing for the subject is different between the first imaging unit and the second imaging unit. FIG. 11 is a diagram showing a case where the difference in scanning timing for a subject between the first imaging unit and the second imaging unit is brought close to zero. FIG. 12 is a diagram showing an example in which the scanning timing is delayed. FIG. 13 is a block diagram illustrating a hardware configuration of a control system of the image pickup apparatus according to the second embodiment. FIG. 14 is a diagram showing an example of the vanishing point. FIG. 15 is a flowchart showing the flow of processing for reflecting the delay time from the vanishing point. FIG. 16 is a diagram showing an example in which an imaging device according to the embodiment is provided in an object detection device for an automobile. FIG. 17 is a diagram showing a schematic configuration of an object detection device for an automobile to which the imaging device according to the embodiment is applied. FIG. 18 is a diagram showing an example of the configuration of an object detection device for an automobile to which the imaging device according to the embodiment is applied. FIG. 19 is a top view of an aircraft equipped with an imaging device according to an embodiment of the present invention. FIG. 20 is a front view of an aircraft equipped with an imaging device according to an embodiment of the present invention.

Embodiments of the imaging device and program are described in detail below with reference to the attached drawings.

(First embodiment)
Fig. 1 is a schematic plan view showing the configuration of an imaging device 10 according to a first embodiment, and Fig. 2 is a schematic plan view showing the configuration of a first imaging section 1 and a second imaging section 2. The stereo imaging device 10 has the first imaging section 1 and the second imaging section 2 as a stereo camera that captures an image of a subject.

The first imaging unit 1 and the second imaging unit 2 are arranged with a predetermined distance between them, the baseline length B (e.g., 60 mm). The imaging device 10 can synchronize the scanning timing to capture images of the same subject using the first imaging unit 1 and the second imaging unit 2.

As shown in FIG. 2, the first imaging unit 1 and the second imaging unit 2 each include an optical lens 8 (8a, 8b) and an imaging element 9 (9a, 9b). The first imaging unit 1 and the second imaging unit 2 are fixed together by a stay member 12 with a baseline length B between them.

The optical lens 8a is, for example, a lens with a focal length f: 2.5 mm. The optical lens 8b is, for example, a lens with a focal length f: 5 mm.

The first imaging unit 1 is a wide-angle camera. The second imaging unit 2 is a narrow-angle camera. In other words, the first imaging unit 1 and the second imaging unit 2 have different angles of view due to different resolutions, and also have different focal lengths.

The imaging element 9a of the first imaging unit 1 is a high-resolution CMOS image sensor that uses a rolling shutter method. The imaging element 9a has a pixel pitch pp of 3 μm. The rolling shutter method is a method in which the pixels in the sensor are scanned in sequence.

The imaging element 9b of the second imaging unit 2 is a low-resolution CMOS image sensor that uses a rolling shutter method. The imaging element 9b has a pixel pitch pp of 3 μm.

Next, we will explain the control system.

The imaging device 10 has a hardware configuration that utilizes a normal computer, and includes a control device such as a CPU (Central Processing Unit), a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and an external storage device such as a HDD that stores a control program.

The CPU of the imaging device 10 executes the control program to function as a memory unit 3, a correction unit 6, a parallax calculation unit 7, and a delay instruction unit 5 as shown in FIG. 3.

The control program executed by the imaging device 10 of this embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. The control program executed by the imaging device 10 of this embodiment may also be provided or distributed via a network such as the Internet.

The control program executed by the imaging device 10 of this embodiment may also be configured to be provided by being pre-installed in a ROM or the like.

Figure 3 is a block diagram showing the hardware configuration of the control system of the imaging device 10. As shown in Figure 3, the imaging device 10 includes a storage unit 3, a correction unit 6, a parallax calculation unit 7, and a delay instruction unit 5.

The memory unit 3 stores the captured image 11, the corrected image 14, the parallax image 15, and the delay time 17.

Here, the captured image captured by the first imaging unit 1 is referred to as the "first captured image." Also, the captured image captured by the second imaging unit 2 is referred to as the "second captured image." Also, when there is no need to distinguish between the first captured image and the second captured image, they are simply referred to as "captured images." Captured image 11 is either the first captured image or the second captured image.

The correction unit 6 adjusts the magnification and performs trimming so that the first captured image and the second captured image appear to be images captured at the same focal length, and corrects the coordinate shift of the first captured image (second captured image) caused by the position shift of the first imaging unit 1 (second imaging unit 2).

As described above, the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, have different angles of view due to the difference in resolution, and also have different focal lengths. Therefore, it is necessary to adjust the focal length in order to obtain the distance to the subject, which will be described later. The correction unit 6 adjusts the magnification and performs trimming in order to adjust the focal length.

Figure 4 is a diagram showing an example of magnification adjustment and trimming in the correction unit 6. In Figure 4, the image captured by the first imaging unit 1, which is a wide-angle camera, is the first captured image 11-1, and the image captured by the second imaging unit 2, which is a narrow-angle camera, is the second captured image 11-2.

4, the focal length of the first imaging unit 1 is f ₁ , and the focal length of the second imaging unit 2 is f _2. It is known that the magnification can be calculated from the ratio of the focal lengths, and the magnification m of the second captured image 11-2 with respect to the first captured image 11-1 is (f ₂ /f ₁ ). Therefore, the correction unit 6 enlarges the first captured image 11-1 by m times, and trims it to the same sizes X ₁ ', Y ₁ ' as the sizes X ₂ , Y ₂ of the second captured image 11-2, to create a first corrected image 14-1 with the same magnification and image size as the second captured image 11-2.

In this embodiment, as described above, the focal length _f1 of the optical lens 8a of the first imaging unit 1 is 2.5 mm, and the focal length _f2 of the optical lens 8b of the second imaging unit 2 is 5 mm. In such a case, the focal length of the captured image is adjusted, for example, by adjusting the magnification and trimming, as follows:
Apparent focal length of image captured by first imaging unit 1: 5 mm
Apparent focal length of image captured by second imaging unit 2: 5 mm

The disparity calculation unit 7 calculates the disparity from the first corrected image obtained by correcting the first captured image by the correction unit 6 and the second captured image. The disparity calculation unit 7 then uses the calculated disparity to generate a disparity image 15 including information on the disparity of the coordinates of the captured image 11. The specific method of calculating the disparity is the same as that of a well-known parallel isometric stereo camera.

The disparity calculation unit 7 calculates the correlation between the image area around the feature point extracted from the reference captured image and the corresponding candidate area on the captured image to be compared, shifting them by one pixel at a time, and calculates the disparity with an accuracy of less than a pixel unit by interpolating the correlation value near the maximum correlation value using a parabola or the like. The disparity calculation unit 7 of this embodiment generates a disparity image 15 with an accuracy of less than 1/10 pixel.

Here, we will explain the principle of distance measurement in the imaging device 10 using the first imaging unit 1 and the second imaging unit 2.

Here, Fig. 5 is a diagram showing an example of distance measurement by the imaging device 10. As shown in Fig. 5, the first imaging unit 1 has a focal length f of the optical lens 8a, an optical center _O0 of the optical lens 8a, and an imaging surface _S0 of the imaging element 9a, and the Z axis is the optical axis direction. The second imaging unit 2 has a focal length f of the optical lens 8b, an optical center _O1 of the optical lens 8b, and an imaging surface _S1 of the imaging element 9b, and the Z axis is the optical axis direction.

The first imaging unit 1 and the second imaging unit 2 are positioned parallel to the X-axis and separated by a baseline length B (e.g., 60 mm).

It is assumed that the subject A is located at a distance d from the optical center O ₀ of the first imaging unit 1 in the optical axis direction.

The first imaging unit 1 forms an image of the subject A at _P0 , which is the intersection point of the straight line A- _O0 and the imaging plane _S0 . Meanwhile, the second imaging unit 2 forms an image of the subject A at _P1 , which is the intersection point of the straight line A- _O1 and the imaging plane _S1 .

Here, _P0 ' is the intersection point between a line that passes through the optical center _O1 of the second imaging unit 2 and is parallel to the line A- _O0 , and the imaging surface _S1 . Also, p is the distance between _P0 ' and _P1 . The parallax calculation unit 7 calculates the distance p, which is the amount of positional shift on the images of the same subject A captured by the first imaging unit 1 and the second imaging unit 2, as the parallax. Then, the parallax calculation unit 7 uses this parallax to generate a parallax image 15 including information on the parallax of the coordinates of the captured image 11.

5, the triangle AO ₀ -O ₁ and the triangle O ₁ -P ₀ '-P ₁ are similar. Therefore, the distance d to the subject A can be calculated from the base line length B, the focal length f, and the parallax p as shown in the following formula.
d = B × f / p

The imaging device 10 periodically and repeatedly performs imaging, correction, and parallax calculation of the captured image using the above-mentioned components (first imaging unit 1, second imaging unit 2, memory unit 3, correction unit 6, and parallax calculation unit 7) to continuously generate a parallax image 15.

However, in the imaging device 10 equipped with the first imaging unit 1 and the second imaging unit 2 having different angles of view (amount of pixels scanned), as in this embodiment, when the rolling shutter method is adopted as the shutter method, there is a problem that even if scanning is started simultaneously, a timing difference occurs during scanning in the first imaging unit 1 and the second imaging unit 2. If a timing difference occurs during scanning in this way, the same subject will be imaged at different times on the left and right, resulting in an error in the distance information.

Here, we will explain an example in which an error occurs in distance information when a rolling shutter method is adopted as the shutter method in an imaging device 10 equipped with a first imaging unit 1 and a second imaging unit 2 with different angles of view.

Here, FIG. 6 is a diagram showing an example of the horizontal angle of view of the first imaging unit 1 and the second imaging unit 2 when the Z axis is arranged as the optical axis direction. As shown in FIG. 6, when the Z axis is arranged as the optical axis direction, the horizontal angle of view of the first imaging unit 1, which is a wide-angle camera, is represented by the symbol 1a, and the horizontal angle of view of the second imaging unit 2, which is a narrow-angle camera, is represented by the symbol 2a.

Figure 7 is a diagram showing an example of the vertical angle of view of the first imaging unit 1 and the second imaging unit 2 when the Z axis is arranged as the optical axis direction. As shown in Figure 7, when the Z axis is arranged as the optical axis direction, the vertical angle of view of the first imaging unit 1, which is a wide-angle camera, is represented by the symbol 1b, and the vertical angle of view of the second imaging unit 2, which is a narrow-angle camera, is represented by the symbol 2b.

Figure 8 is an exemplary diagram showing the difference in scanning timing between the image sensor 9a of the first imaging unit 1 and the image sensor 9b of the second imaging unit 2. Figure 8 explains the difference Δtv in scanning timing for subject A when the scanning timing of the image sensor 9a of the first imaging unit 1, which is a wide-angle camera, and the image sensor 9b of the second imaging unit 2, which is a narrow-angle camera, are aligned.

The rolling shutter type image sensor 9 (9a, 9b) scans the image line by line from the top to the bottom of the screen. As shown in FIG. 8, the vertical line scanned to subject A in the image sensor 9a of the first image sensor 1 is v1, and the vertical line scanned to subject A in the image sensor 9b of the second image sensor 2 is v2. As shown in FIG. 8, the number of vertical lines scanned to subject A is v1 > v2.

Also, as shown in FIG. 8, the scanning time per line in the image sensor 9a of the first imaging unit 1 is th1, and the scanning time per line in the image sensor 9b of the second imaging unit 2 is th2. Therefore, the scanning time tv1 to subject A in the image sensor 9a of the first imaging unit 1 is expressed as (th1×v1), and the scanning time tv2 to subject A in the image sensor 9b of the second imaging unit 2 is expressed as (th2×v2).

That is, as shown in Figure 8, when the scanning time th1 per line at the image sensor 9a of the first imaging unit 1 is different from the scanning time th2 per line at the image sensor 9b of the second imaging unit 2, a time difference Δtv as shown in the following equation (1) occurs in the scanning timing of subject A.
Δtv=tv1-tv2=(th1×v1)-(th2×v2)...(1)

The following describes an example where th1<th2.

Figure 9 is an exemplary diagram showing the difference in scanning timing between the front and rear lines of subject A. Figure 9 explains the difference in scanning timing between the front and rear lines of subject A when the scanning timing of the image sensor 9a of the first imaging unit 1, which is a wide-angle camera, and the image sensor 9b of the second imaging unit 2, which is a narrow-angle camera, are aligned. As shown in Figure 9, the scanning time th1 per line of the image sensor 9a of the first imaging unit 1 and the scanning time th2 per line of the image sensor 9b of the second imaging unit 2 have a relationship of th1 < th2, so the time difference Δtv in the scanning timing of subject A becomes smaller with each scan.

Here, we will explain the problem that occurs when the scanning timing for subject A is out of sync between the first imaging unit 1 and the second imaging unit 2.

Fig. 10 is a diagram showing a case where the scanning timing for subject A is different between the first imaging unit 1 and the second imaging unit 2. Fig. 10 shows that, with the imaging surface _S'0 of the first imaging unit 1 and the imaging surface _S'1 of the second imaging unit 2, if the first imaging unit 1 images the subject A and the subject A moves a distance g to the left during a time Δth, the second imaging unit 2 images the subject _A2 after the movement. As a result, as shown in Fig. 10, the parallax becomes large, and a distance d' of the position of subject A' that is closer than the actual distance of subject A is obtained.

As described above, the apparent focal length f of the optical lens 8 (8a, 8b) after trimming to adjust the focal length is 5 mm, the baseline length B between the first imaging unit 1 and the second imaging unit 2 is 60 mm, and the pixel pitch pp is 3 μm.

Furthermore, the scanning time th1 per line of the first imaging unit 1 is 20 μs, the number of lines tv1 scanned to subject A is 480, the scanning time th2 per line of the second imaging unit 2 is 50 μs, and the number of lines tv2 scanned to subject A is 240.

The difference in scanning timing Δtv for subject A is -2.4 ms according to the above formula (1).

Here, let's assume that the distance d to subject A is 30 m, and subject A is moving to the left at a speed of 60 km/h.

The distance g that the subject A moves during the 2.4 ms from when the first imaging unit 1 scans the subject A until the second imaging unit 2 captures the image of the subject A is given by:
g=60km/h×2.4ms=0.04m
It becomes.

Here, the increment in parallax caused by moving the distance g is defined as G. Furthermore, triangle AA ₂ -O ₁ and triangle O ₁ -P ₁ -P ₁ ' are similar. Therefore, the increment in parallax G is approximately 2.2 pixels according to the following formula.
G=(((f×g)/d))/pp

Meanwhile, the parallax value p for a distance of 30 m to the original subject A is about 3.3 pixels, since the triangles A-O ₀ -O ₁ and O ₁ -P ₀ '-P ₁ are similar, so P = ((B × f)/d)/pp. Therefore, the parallax value p' is about 5.5 pixels.

10, the triangle A'-O ₀ -O ₁ and the triangle O ₁ -P ₀ '-P ₁ are similar, so d'=B×f/(p'×pp), and the distance d' to the subject A' is approximately 18 m.

In other words, the imaging device 10 measures subject A as being 18 m away when in reality it is 30 m away. If the vehicle is traveling and the braking distance is assumed to be 20 m, the object detection device of the automobile equipped with the imaging device 10 will predict, based on this result alone, that a collision will occur because an object is present within the braking distance. In other words, the control system may determine that emergency braking or avoidance operations are necessary.

However, automotive object detection devices usually have a margin in place, such as requiring multiple collision detections before control can begin. Therefore, erroneous control will not necessarily occur in the above situation, but it is natural that it is desirable for the stereo camera module itself to have high reliability. In this way, the combination of the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, can cause problems such as the subject appearing to be moving.

Therefore, in this embodiment, the imaging device 10 adjusts the scanning timing. Specifically, as shown in FIG. 3, the imaging device 10 includes a delay instruction unit 5. Also, as shown in FIG. 3, the imaging device 10 stores a delay time 17 in the storage unit 3.

The delay instruction unit 5, details of which will be described later, reads the delay time 17 from the memory unit 3 and instructs the imaging device 10 to delay the scanning start timing of the first imaging unit 1 and the second imaging unit 2.

Here, FIG. 11 is a diagram showing a case where the difference in scanning timing for subject A between the first imaging unit 1 and the second imaging unit 2 is brought closer to zero. FIG. 11 explains the delay time Δtv of the scanning start timing when the difference in scanning timing for subject A between the imaging element 9a of the first imaging unit 1, which is a wide-angle camera, and the imaging element 9b of the second imaging unit 2, which is a narrow-angle camera, is brought closer to zero.

As shown in FIG. 8, when scanning is started at the same scanning timing, the time until the subject A is scanned is as follows for the image sensor 9a of the first image capturing unit 1:
tv1 = th1 × v1
In the image sensor 9b of the second image capturing unit 2,
tv2 = th2 × v2
The difference Δt v in the scanning timing of the first imaging unit 1 and the second imaging unit 2 with respect to the subject A is expressed as follows:
Δtv=tv1-tv2=(th1×v1)-(th2×v2)
It can be seen that.

Therefore, as shown in FIG. 11, by shifting the scanning start timing of the image sensor 9a of the first imaging unit 1 and the image sensor 9b of the second imaging unit 2 by Δtv, the scanning timing shift of the subject A can be reduced.

That is, the imaging device 10 stores the difference Δtv in the scanning timing of the first imaging unit 1 and the second imaging unit 2 for the subject A in the storage unit 3 as the delay time 17.

Then, the delay instruction unit 5 of the imaging device 10 reads the delay time 17 from the memory unit 3 and instructs the first imaging unit 1 and the second imaging unit 2 to delay the scanning timing.

Figure 12 is a diagram showing an example of delayed scanning timing. As shown in Figure 12, the delay instruction unit 5 of the imaging device 10 delays the difference between the scanning timing of the image sensor 9a of the first imaging unit 1, which is a wide-angle camera, and the scanning timing of the image sensor 9b of the second imaging unit 2, which is a narrow-angle camera, based on the scanning timing deviation Δtv, which is the delay time 17 read out from the storage unit 3, so that the difference approaches zero.

In the example shown in FIG. 12, the scanning time th1 per line in the image sensor 9a of the first imaging unit 1 and the scanning time th2 per line in the image sensor 9b of the second imaging unit 2 have a relationship of th1<th2, so the difference in scanning timing increases with each scan. However, in the example shown in FIG. 12, the deviation in scanning timing between the lines before and after subject A is reduced compared to the example shown in FIG. 9.

Thus, according to this embodiment, when starting scanning with the wide-angle camera (first imaging unit 1) and narrow-angle camera (second imaging unit 2) that have imaging elements with different resolutions, one of the imaging elements is delayed in advance by the difference in the scanning completion timing between the left and right imaging elements by the time difference in the scanning completion timing, and the scanning start trigger of the imaging element is changed so that the scanning timing of the left and right imaging elements is the same in any area (area where the same subject is present). This makes it possible to reduce the difference in scanning timing that occurs between the left and right imaging units that have different angles of view.

In the above explanation, the case of subject A is used as an example, but it goes without saying that the target for reducing the scanning timing deviation may be set to any region.

In addition, in the above explanation, an example was given in which th1<th2, but even when th1>th2, a time difference of Δtv occurs in the scanning timing of subject A, and by shifting the scanning start timing of the image sensor 9a of the first imaging unit 1 and the image sensor 9b of the second imaging unit 2 by Δtv, the difference in the scanning timing of subject A can be reduced.

Second Embodiment
Next, the second embodiment will be described. In the first embodiment, an area in which the same subject exists is applied as an arbitrary area in the captured image, but in the second embodiment, an area in which a vanishing point exists where the direction of a stationary object moving on the captured image converges is applied as an arbitrary area in the captured image, which is different from the first embodiment. In the following description of the second embodiment, the description of the same parts as in the first embodiment will be omitted, and only the parts different from the first embodiment will be described.

Fig. 13 is a block diagram showing the hardware configuration of a control system of the imaging device 10 according to the second embodiment. As shown in Fig. 13, the imaging device 10 further includes a vanishing point detection unit 4 that detects vanishing points and performs other operations in addition to the imaging device 10 shown in Fig. 3.

Now let's explain vanishing points.

Figure 14 is an exemplary diagram showing vanishing points. The example shown in Figure 14 shows a case where the first imaging unit 1, which is a wide-angle camera, and the second imaging unit 2, which is a narrow-angle camera, are arranged with the optical axis direction at a depression angle with respect to the Z axis. In this case, the vertical angle of view of the first imaging unit 1, which is a wide-angle camera, is represented by the symbol 1c, and the vertical angle of view of the second imaging unit 2, which is a narrow-angle camera, is represented by the symbol 2c.

For vehicles with a high vehicle height, such as trucks, the camera is mounted higher, which reduces the reflection of the road surface in the vicinity. As shown in Figure 14, it is possible to add a depression angle so that the road surface is reflected.

Normally, collision detection in a vehicle control system is performed for objects in the traveling direction. For this reason, it is important to align the scanning timing with the subject A that appears in the traveling direction D of the imaging device 10, as shown in FIG. 14.

As shown in FIG. 14, subject A, which appears in the traveling direction D of the imaging device 10, appears at the same position as the vanishing point where the directions of motion of stationary objects converge on the captured image. Therefore, the imaging device 10 can align its scanning timing with the vanishing point on the captured image.

The vanishing point detection unit 4 detects a vanishing point by reading out the first captured image 11-1 captured by the first imaging unit 1 and the second captured image 11-2 captured by the second imaging unit 2. Methods for detecting a vanishing point from captured images include, for example, the methods described in the following documents.
Reference: Yamamoto Muneta and Kaneko Kunihiko (2011). Parallelization of vanishing point trajectory estimation processing from a moving camera image database using MapReduce, 3-4. https://db-event.jpn.org/deim2011/proceedings/pdf/e9-2.pdf

Then, the vanishing point detection unit 4 writes the delay time at which the scanning timing at the vanishing point on the captured image of the first imaging unit 1 and the second imaging unit 2 coincides, as the delay time 17, into the memory unit 3.

In the above explanation, the vehicle travel direction is detected using the vanishing point on the captured image, but this is not limited to this, and the vehicle travel direction may be detected using something other than the vanishing point on the captured image.

Next, we will explain in detail how the vanishing point detection unit 4 reflects the delay time from the vanishing point.

Figure 15 is a flowchart showing the flow of the process of reflecting the delay time from the vanishing point. As shown in Figure 15, the vanishing point detection unit 4 reads out the first captured image 11-1 captured by the first imaging unit 1, and detects the vanishing point F1 (step S1).

Next, the vanishing point detection unit 4 reads out the second captured image 11-2 captured by the second imaging unit 2 and detects the vanishing point F2 (step S2).

The detection of the vanishing point F1 of the first imaging unit 1 and the detection of the vanishing point F2 of the second imaging unit 2 may be processed in parallel.

Next, the vanishing point detection unit 4 calculates a delay time Δtv of the scanning timing from the number v1 of vertical lines of the vanishing point F1 and the number v2 of vertical lines of the vanishing point F2 (step S3).
Δtv=tv1-tv2=(th1×v1)-(th2×v2)

Then, the vanishing point detection unit 4 writes the scan timing delay time Δtv to the memory unit 3 as the delay time 17 (step S4).

Thus, according to this embodiment, when starting scanning of the wide-angle camera (first imaging unit 1) and narrow-angle camera (second imaging unit 2) having imaging elements with different resolutions, one of the imaging elements is delayed in advance by the difference in the scanning completion timing of the left and right imaging elements by the time difference in the scanning completion timing, and the scanning start trigger of the imaging element is changed so that the scanning timing of the left and right imaging elements is the same in any area (an area where there is a vanishing point where the directions of moving stationary objects converge). This makes it possible to reduce the difference in scanning timing that occurs between the left and right imaging units with different angles of view.

(Example)
(Example of application to automobiles)
Next, the imaging device 10 of each of the above-described embodiments can be provided in a moving object such as a robot device, an automobile, an aircraft, a ship, etc. An application example in which the imaging device 10 is applied to a moving object will be described below.

First, FIG. 16 is a diagram showing an example in which an imaging device 10 according to an embodiment is provided in an object detection device for an automobile. As shown in FIG. 16, an object detection device equipped with imaging device 10 is provided on the windshield of an automobile, etc., and captures images of pedestrians, road signs, the road surface, a preceding vehicle, guard rails, etc., within a predetermined imaging range ahead in the traveling direction. As described above, imaging device 10 has a stereo camera configuration, and captures two images, one for the left eye and one for the right eye.

Figure 17 is a diagram showing the general configuration of an object detection device. The object detection device outputs two images captured by the imaging device 10 to a vehicle ECU (Engine Control Unit) 50. The vehicle ECU 50 analyzes the parallax image based on each image from the imaging device 10, and performs control of the automobile's engine control, braking control, lane keeping assist, steering assist, etc., based on the analysis results.

Figure 18 is a diagram showing the configuration of an object detection device. As shown in this Figure 18, an imaging device 10 equipped with a first imaging unit 1 and a second imaging unit 2 is connected to an image processing device 30. The first imaging unit 1 and the second imaging unit 2 each include lenses 8a, 8b, imaging elements 9a, 9b, and sensor controllers 23a, 23b. The sensor controller 23 performs, for example, exposure control, image read control, communication with external circuits, and image data transmission control for each of the imaging elements 9a, 9b.

The image processing device 30 is provided, for example, in the vehicle ECU 50 shown in FIG. 17. The image processing device 30 has, for example, a data bus line 300, a serial bus line 302, a CPU (Central Processing Unit) 304, an FPGA (Field-Programmable Gate Array) 306, a ROM (Read Only Memory) 308, a RAM (Random Access Memory) 310, a serial IF (Interface) 312, and a data IF (Interface) 314.

The imaging device 10 is connected to the image processing device 30 via a data bus line 300 and a serial bus line 302. The CPU 304 controls the operation of the entire image processing device 30 and executes image processing and image recognition processing. Images captured by the image sensors 9a, 9b of the first imaging unit 1 and the second imaging unit 2 are written to the RAM 310 of the image processing device 30 via the data bus line 300. Change control data for the sensor exposure value, change control data for the image readout parameters, and various setting data from the CPU 304 or FPGA 306 are transmitted and received via the serial bus line 302.

The FPGA 306 performs real-time processing on the image data stored in the RAM 310, such as gamma correction, distortion correction (parallelization of left and right images), and parallax calculation using block matching, to generate the above-mentioned parallax image, which it then writes back to the RAM 310. Note that the parallax image is information that corresponds to the vertical position, horizontal position, and depth position of an object.

The CPU 304 controls the sensor controllers 23a and 23b of the imaging device 10, and performs overall control of the image processing device 30. The CPU 304 acquires, for example, the vehicle's CAN (Controller Area Network) information as parameters (vehicle speed, acceleration, steering angle, yaw rate, etc.) via the data IF 314.

Vehicle control data, which is recognition data for objects such as preceding vehicles, people, guardrails, and the road surface, is supplied to the vehicle ECU 50 via the serial IF 312 and is used in, for example, an automatic braking system or a driving assistance system provided as a control function of the vehicle ECU 50. The automatic braking system controls the brakes of the vehicle. The driving assistance system performs lane keeping assistance and steering assistance for the vehicle.

(Example of application to aircraft)
Next, Fig. 19 and Fig. 20 are diagrams showing an example of installation of the imaging device 10 of the embodiment on an aircraft. Fig. 19 shows a top view of the aircraft provided with the imaging device of the embodiment, and Fig. 20 shows a front view of the aircraft provided with the imaging device of the embodiment. As shown in Fig. 19 and Fig. 20, the aircraft is a remotely controlled aircraft having a frame 101, a drive unit 102, rotors 103, and legs 104. The aircraft also has an imaging device 10 that captures still images or moving images of a subject.

As shown in FIG. 19, the frame 101 is arranged such that the web portion 101a and the flange portions 101b arranged at both ends of the web portion 101a form an H-shape in a plan view. However, the shape of the frame 101 is not limited to this. For example, the frame 101 may have four arms extending radially from the center of the aircraft and arranged in an X-shape at right angles to each other. The frame 101 may also have a structure in which arms extend radially from the center in the same number as the driving devices 102 and rotors 103.

The driving device 102 is a motor that rotates the rotor 103. As an example, a brushless motor can be used as the driving device 102. The driving devices 102 are attached to the tips of the two flanges 101b, respectively, and are connected to the rotating shafts of the rotors 103. Each driving device 102 is controlled independently of the other by the control device 100.

The rotors 103 are attached to the respective drive devices 102, and rotate when the drive devices 102 are driven. The rotation speed of the rotors 103 is adjusted by independently controlling the drive of each drive device 102. Although FIG. 19 illustrates the flying object having four rotors 103, three or more rotors 103 are sufficient for stable flight as an flying object. Furthermore, it is preferable to provide an even number of rotors 103, such as four, six, or eight. In addition, the rotors 103 are not limited to a form in which multiple rotors 103 are arranged on approximately the same plane as shown in FIG. 19 and FIG. 20, but may be a form in which multiple rotors 103 arranged on approximately the same plane are stacked in multiple stages.

The legs 104 are connected to the underside of the frame 101 at approximately the center. The lower part of the legs 104 is formed in an inverted V shape to protect the aircraft from impacts when it lands on the ground.

The control device 100 is provided, for example, in the web portion 101a of the frame 101. The control device 100 controls each drive device 102 etc. in response to signals from various sensors provided on the aircraft according to the purpose and operation signals sent from the outside.

The control device 100 also independently controls the drive of each drive device 102, thereby independently controlling the rotational speed of each rotor 103, and controls the flight vehicle's movements such as ascent, descent, forward movement, and backward movement, as well as rotation around the yaw axis (Z axis) and movement into the air for hovering (stationary flight). The control device 100 also tilts the attitude of the flight vehicle left and right to control the flight vehicle's movement in the left and right directions, as well as forward and backward movement.

The imaging device 10 is mounted on the web portion 101a via a spacer that absorbs vibrations transmitted through the web portion 101a of the frame 101. This reduces the effect that vibrations of the frame 101 and the web portion 101a have on the measurement of the distance to the structure. As will be described later, the support member (housing) of the imaging device 10 is made of a carbon fiber reinforced plastic member (CFRP: Carbon Fiber Reinforced Plastic) with adjusted fiber orientation.

This allows the imaging device 10 to reduce the effects of external loads, such as bending, twisting, and stretching, that are applied by direct external factors such as wind and contact. Therefore, the imaging device 10 is able to measure the measurement target (object) with high accuracy by using the above-mentioned spacer and CFRP with adjusted fiber direction.

Such an imaging device 10 captures images of the surroundings of the flying object. As an example, the imaging device 10 is a so-called stereo camera device equipped with an imaging element 9. The imaging device 10 is fixed to the web portion 101a of the frame 101. The imaging device 10 is provided on the web portion 101a so that the optical axis is horizontal when the flying object is flying horizontally parallel to the ground. The imaging device 10 is also provided on the web portion 101a so that the imaging direction coincides with the forward direction of the flying object.

In addition, by providing multiple imaging devices 10, it may be possible to capture a full celestial sphere (360°) panoramic image around the flying object (full celestial sphere camera device).

In this way, the imaging device 10 of the embodiment can be installed in a moving object such as an automobile or an aircraft. It can also be applied to industrial robots, etc. In either case, the same effects as described above can be obtained.

Such novel embodiments may be implemented in various other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. The embodiments and modifications of the embodiments are included within the scope and spirit of the invention, and are included in the scope of the invention and its equivalents as set forth in the claims.

Reference Signs List 1 First imaging unit 2 Second imaging unit 4 Vanishing point detection unit 5 Delay instruction unit 10 Imaging device

Patent No. 5683025

Claims (6)

A first imaging unit having a rolling shutter type imaging element;
a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit;
a delay instruction unit that delays a scanning start timing for one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit;
Equipped with
the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which the same subject exists,
the delay instruction unit delays the scanning start timing so that a difference in scanning time to the subject approaches zero.
1. An imaging device comprising: A first imaging unit having a rolling shutter type imaging element;
a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit;
a delay instruction unit that delays a scanning start timing for one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit;
Equipped with
the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions where a vanishing point exists where directions of movement of a stationary object converge on the first captured image and the second captured image,
the delay instruction unit delays the scanning start timing so that the scanning timings at the vanishing points coincide with each other.
1. An imaging device comprising: a vanishing point detection unit that detects a vanishing point to which directions of movement of a stationary object converge on the first captured image and the second captured image;
3. The imaging device according to claim 2 . A first imaging unit having a rolling shutter type imaging element;
a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit;
A computer that controls an imaging device comprising:
a delay instruction unit that delays a scanning start timing of one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit ,
the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions in which the same subject exists,
the delay instruction means delays the scanning start timing so that a difference in scanning time to the subject approaches zero.
Program for. A first imaging unit having a rolling shutter type imaging element;
a second imaging unit having a rolling shutter type imaging element having a different resolution from that of the first imaging unit;
A computer that controls an imaging apparatus comprising:
a delay instruction unit that delays a scanning start timing of one of the first imaging unit and the second imaging unit by a delay time that is a difference in scanning completion timing between the first imaging unit and the second imaging unit in an arbitrary region in a first captured image captured by the first imaging unit and an arbitrary region in a second captured image captured by the second imaging unit,
the arbitrary region in the first captured image and the arbitrary region in the second captured image are regions where a vanishing point exists where directions of movement of a stationary object converge on the first captured image and the second captured image,
the delay instruction means delays the scanning start timing so that the scanning timings at the vanishing points coincide with each other;
Program for. causing the computer to function as a vanishing point detection means for detecting a vanishing point to which directions of movement of a stationary object converge on the first captured image and the second captured image;
6. The program according to claim 5 .

Images