JP7653369B2 - Stereo Camera System - Google Patents

Description

The present invention relates to a stereo camera system for measuring three-dimensional surface shapes. This may be used to obtain depth information from an object moving along a conveyor belt in front of the stereo camera system. The stereo camera system may form part of a production monitoring system.

Production monitoring systems that include conveyor systems and cameras for optically scanning products transported by the conveyor system are known and are used, for example, in package sorting facilities and production lines.

Usually the height of the product or object to be scanned is determined by a measuring device, e.g. a light curtain. The camera has an adjustable lens, which is set to a pre-determined height of the object to be scanned in order to correctly scan the object's surface. The camera may have a zoom lens with a variable focal length so that it can be set appropriately to scan objects of different heights. It is also possible to use a lens with a fixed focal length, the focus of which can be tracked to ensure that objects of different heights are sharply depicted on the sensor.

In the case of a zoom lens, multiple lenses are moved relative to one another. Zoom lenses have a movable mechanism inside the lens that presses the multiple lenses against each other. In the case of a zoom lens, the scale can be kept constant along the adjustment path.

When using a lens with a fixed focal length, the entire lens is moved along the optical axis so that the area of sharpness is located at the desired height. The shorter the optical path length, the larger the imaging scale.

Zoom lenses and fixed focal length lenses both require moving mechanical parts. This movement leads to wear. Industrial applications require millions of cycles of use, which is usually well beyond the lifespan of these moving parts.

Where a product monitoring system requires the evaluation of both optical and height information, various methods have been established.

One possibility is the so-called light section method, in which a thin, straight strip of light is shone on the object to be measured and recorded by an area scan camera. The distortion caused by the object in the strip of light may be used to estimate the object's surface.

Stereoscopic vision offers a further option: a camera system can record an object from two different viewing directions and determine the topological surface data of the object from the corresponding pixel offsets. However, the disadvantage of this method is that it is limited to one focal plane.

Such a stereo camera system may comprise two camera modules, with which the same object area may be scanned from two different viewing directions. The camera modules are equipped with lenses for imaging the object area onto the sensors of the respective camera modules. The optical axis of the lens is the axis of symmetry of the lens, which runs approximately transverse to the surface of the lens or lenses contained therein. The object-facing principal surface of the lens is referred to as the lens surface. The sensor surface is the plane on which the face of the light-sensitive sensor lies. The object surface is the surface that is sharply imaged by the lens onto the face of the sensor.

In conventional stereo camera systems, the lens plane, object plane, and sensor plane are arranged parallel to one another. In what follows, the parallel alignment of the object plane, lens plane, and sensor plane is referred to as a "parallel arrangement."

The key to resolving three-dimensional surface shapes with a stereo camera system is the base angle (vertex angle, Basis-Winkel) α, which is the angle that the viewing direction of the stereo camera system's two camera modules occupies on the target area. The base angle α is basically set by the size of the base (base) B, which is the distance between the faces of the two sensors of the two camera modules S1 and S2. In other words, the larger the base B, the larger the base angle α, and the higher the vertical resolution of the stereo camera system (Figure 8).

In the following, unless otherwise specified, it is assumed that the mutual distances between the sensor surface SP, lens surface LP, and object surface OP are constant, and only the base B determines the base angle α.

A lens has an aperture angle γ that gives the maximum angle of incidence of the incoming light with respect to the optical axis, within which the object may be imaged substantially without distortion (Figure 8). Depending on the lens design, the maximum aperture angle of the lens may vary. In the following, the beam passing through the lens is described as the "beam path". The beam path includes all beams within the aperture angle and spans between the face of the sensor and the object.

However, in the parallel stereo camera system described above, if the base is too large, the portions enclosed by the incidence angles will not overlap or will otherwise only slightly overlap in the object region, so the sharply imaged portion of the object region will be small or will no longer exist, so the base may not be as large as desired.

Only the overlap region, i.e. the region of the object that can be captured simultaneously by both sensors, can be used to evaluate the three-dimensional surface shape. The edge region of this overlap region is imaged with a large lens aperture angle γ. Therefore, distortions are greatest at the edge region of the overlap region. In addition, the imaging performance, which may be characterized inter alia by color defects and the modulation transfer function (MTF), is worse at the edge region of the overlap region than in the center of the overlap region. The color defects represent a loss of color imaging. The MTF represents a decrease in the contrast of details.

"Scheimpflug stereocamera for particle image velocity in liquid flow", Ajay K. Prasad and Kirk Jensen, APPLIED OPTICS 34, 30 (1995), describes a stereo video camera with Scheimpflug conditions and an area sensor. This system is used to determine the velocity distribution of small particles in a liquid.

In such a Scheimpflug camera, the parallel arrangement is eliminated by tilting the lens plane LP relative to the sensor plane, which allows a large base without the object area being limited by the angle of incidence. In a Scheimpflug camera, the so-called Scheimpflug condition is fulfilled, in which the lens plane, the sensor plane and the object plane intersect at a common intersection axis. There is a Scheimpflug angle β between the object plane OP and the sensor plane SP (FIG. 9). If the Scheimpflug condition is fulfilled, the object area is sharply imaged by the lens onto the sensor. In the above-mentioned literature, intersection axes at an infinite distance from the stereo camera system are sometimes described. In this special case, the Scheimpflug condition corresponds to a parallel arrangement. However, in the following discussion, it applies that the Scheimpflug condition is not fulfilled if these planes or axes intersect at infinity. Therefore, the above planes or axes must intersect at a finite point so that the Scheimpflug condition is fulfilled. One or more mirrors may be provided in the beam path such that the beam path is folded so that the actual lens, sensor and object planes do not intersect at a common intersection axis. In such an arrangement, the Scheimpflug condition is met if the respective lenses, sensors and objects are equally effectively positioned in an unfolded beam path without an associated mirror, and the lens, sensor and object planes intersect at a common intersection axis.

On the other hand, for a stereo camera system with the Scheimpflug condition, a wider base angle may be selected for a constant lens and image quality than for a stereo camera system without the Scheimpflug condition. A larger base may lead to a smaller angle of incidence and therefore be optimized due to the tilted position of the lens. As a result, lenses with a smaller aperture angle γ are also suitable. Due to the smaller aperture angle, the image quality, especially the MTF, is more constant in the overlap region in a stereo camera system with the Scheimpflug condition compared to a stereo camera system without the Scheimpflug condition. Linear distortions along the overlap region are taken into account, since they can be corrected by parallelization processes. The larger the base chosen, the more the sensor and lens planes need to be tilted with respect to the object plane in order to reduce the angle of incidence.

A further problem with stereo camera systems without the Scheimpflug condition is back reflections. Due to the narrow base angle in the parallel arrangement, the lens not only images the object on the sensor, but also the sensor on the object. This image of the lens can be imaged back to the sensor again by reflection on a reflective surface of the object. In this way, the sensor can capture a mirror image of itself. A wider base angle can be selected, for example by tilting the lens with a Scheimpflug, thereby avoiding such back reflections. Even if the reflection remains, the image of the reflection may also show, for example, a neutral background. The effect of the reflection is therefore reduced.

Other measurement systems equipped with stereo cameras in which the Scheimpflug condition is met are disclosed in WO 2014/017977 A1, U.S. Pat. No. 6,671,397 B1 and WO 02/040970 A1.

WO 2014/017977 A1 discloses a method and apparatus for determining coplanarity in an integrated circuit housing. To determine the planarity, two sensors, each with two beam paths, are arranged to follow the Scheimpflug principle.

US Patent No. 6,671,397 B1 discloses a metrology system with a camera with a lens and a separate sensor. The metrology system has a first camera with a lens and a separate sensor that are positioned such that their planes intersect at the object plane according to the Scheimpflug principle. A second camera is positioned perpendicular to the object plane. The camera system is an area scan camera.

WO 02/040970 A1 discloses an optical method and apparatus for inspecting large planar objects, which in one embodiment uses two wafer-holding area-scan cameras arranged in a Scheimpflug condition.

The advantages of a stereo camera system with the Scheimpflug condition have been explained above. However, in the case of a stereo camera system with the Scheimpflug condition, a negative effect on the quality of the stereo vision occurs when using area sensors. This is due to the so-called keystone effect, which describes the distortion of the projected sensor face.

A two-dimensional image (picture) of the object area is acquired using an area sensor. In conventional stereo camera systems, one side of the area sensor is arranged parallel to the base. Therefore, the main axis of the acquired two-dimensional image extends either in a direction parallel to the base or laterally to the base. Hereinafter, the direction parallel to the base will be referred to as the base direction, and the direction laterally to the base will be referred to as the lateral direction.

When imaging an object area with a stereo camera system, image compression occurs not only in the lateral direction but also in the base direction. This compression occurs because the individual points of a planar object are located at different distances from the sensor face. The further two points are from the sensor face, the closer they are imaged. The larger the base, the stronger the compression. In the lateral direction, lateral compression occurs, and in the base direction, base compression occurs.

For example, with a square object area, if the base and center placement of the camera system is made as small as possible, the square object area will be imaged with little distortion. However, if the base is made larger, the image of the square will become trapezoidal or keystone shaped. Corners that are farther away from each other will appear smaller than the distance between the corners that are closer to the lens.

This effect occurs in stereo camera systems with and without the Scheimpflug condition. In those without the Scheimpflug condition, the effect is similarly limited by the narrowness of the base. In stereo camera systems with the Scheimpflug condition, the effect is especially pronounced since the base may be particularly large.

The keystone effect results in a loss of resolution. Resolution is the minimum distance two point-like objects need to be apart before they are recognized as separate objects. Compression causes two points far from the face of the sensor to be imaged closer together. At a certain distance from the lens, the points may no longer be recognized as separate. The resolution parallel to the base is hereafter referred to as the "base resolution" and the resolution lateral to the base as the "lateral resolution".

In addition, the keystone effect increases the calculation time. In order to use the images for the evaluation of the 3D surface shape, both the base compression and the lateral compression have to be calculated from the image data by a rectification process. Rectification is a method to remove geometric distortions in the image data that can occur, for example, due to central projection photography. Rectification shifts the individual pixels so that the same position of the object is represented when the pixels of the two sensors are superimposed. In addition, the distortion is corrected so that the calculated disparity depends linearly on the height profile of the object.

In stereoscopic vision, each point in the image of one camera module is assigned to each point in the image of the second camera module. The rectangular sensor records a trapezoid T1 of the object surface if the sensor position is tilted relative to the object due to the keystone effect (Figure 10). The other area sensor records a second trapezoid T2 of the object surface. In the point assignment, these trapezoids overlap, with each long edge coinciding with the short edge of the other trapezoid. This results in a hexagonal overlap area (hatched in Figure 10).

The disadvantage here is that some of the information recorded in the non-overlapping areas is lost. Only the overlapping areas can be used to evaluate the 3D surface shape.

Furthermore, the aperture angle of the lens must be large enough to completely cover the overlap area. This applies when the diagonal D of the hexagonal overlap area is detected by the lens.

DE 10 2015 11 11 20 A1 discloses a method for scanning a surface with a stereo camera comprising at least two camera modules, which are arranged to acquire one image each from a common area of the surface to be scanned. The camera modules may comprise a line camera with several different sensor lines. The individual line images are superimposed by geometric imaging.

Line sensors and parallel stereo camera systems have the same problems as area sensors and parallel stereo camera systems. As mentioned above, the base cannot be made any desired size because if the base is too large, the portions of the object area enclosed by the angles of incidence will overlap slightly or not at all, resulting in a small or non-existent portion of the object area that can be sharply imaged.

In principle, stereo camera systems that do not satisfy either the Scheimpflug condition or the parallel arrangement are conceivable. For example, only the object plane OP and the sensor plane SP could be parallel to each other, but the lens plane LP could be tilted with respect to the other two planes, so that the object region is not limited by a limited angle of incidence. The imaging plane, which corresponds to the plane in which the lens sharply images the object surface, is arranged at an angle (obliquely) with respect to the object surface or the object axis. Only in the area where the imaging plane and the sensor plane intersect is the object region sharply imaged on the camera sensor (Figure 11). The greater the inclination of the lens axis with respect to the object axis, the narrower the focus area in which the object is sharply imaged. Also, the greater the depth of field TS, the wider the focus area. With a narrow focus area, only a small part of the object provides the information necessary for the evaluation of the three-dimensional surface shape.

However, to image 3D surface shapes, sharp imaging is very important. Likewise, it is desirable to use the entire surface of the sensor.

In the following, a "stereo camera system that does not meet the Scheimpflug condition" means a stereo camera system with a parallel arrangement.

Known from "Zeilenkameras zur Inspection der Dichtflaechen einer O-Ring-Nut", Photonik 5/2012 is a system in which three line cameras are operated in parallel, each imaging one side of the groove. The side walls of the groove are imaged by the line cameras with the Scheimpflug condition.

U.S. Patent No. 7,724,362 B1 discloses a system for inspecting macro wafers at angled incidence, where the wafer is detected by a line camera whose beam path is directed at the wafer at an angle that satisfies the Scheimpflug condition.

DE 10 2013 103 897 A1 discloses a camera module for scanning an object line by line. The camera module comprises a linear sensor and a lens for imaging the object onto the sensor. The camera module has a number of line sensors, which are arranged at varying distances from the lens, so that image lines at different distances from the lens are imaged on each line sensor.

International Publication No. 2014/017977A1 U.S. Pat. No. 6,671,397 B1 International Publication No. 02/040970A1 DE 10 2015 11 11 20A1 U.S. Pat. No. 7,724,362 B1 DE 10 2013 103 897 A1

"Scheimpflug stereocamera for particle image velocimetry in liquid flow", Ajay K. Prasad and Kirk Jensen, APPLIED OPTICS 34, 30 (1995) "O-Ring-Nut", Photonik 5/2012

The present invention is based on the problem of creating a stereo camera system that can provide good spatial resolution over a large scanning area in a simple manner.

A further object of the present invention is to create a stereo camera system that allows reliable scanning of spatial objects.

A further object of the present invention is to create a stereo camera system that can cover a large imaging area in a simple manner.

A further object of the present invention is to create a stereo camera system with a compact design.

A further object of the present invention is to create a stereo camera system suitable for imaging different object planes.

One or more of the problems are solved by the subject matter of the independent claims. Advantageous developments and preferred variants form the subject matter of the dependent claims.

The stereo camera system according to the present invention for measuring the three-dimensional surface shape of an object has at least two linear sensor areas and at least one lens for imaging the object on the sensor areas. The lens is arranged such that the object area is imaged by two independent beam paths, each on one of the sensor areas. The stereo camera system is characterized in that the Scheimpflug condition is fulfilled along at least one of the beam paths.

As explained above, a stereo camera system that takes the Scheimpflug condition into account has a larger base than a parallel stereo camera system without the image being distorted by the lenses. The larger base results in higher resolution of the 3D surface shape.

Such a stereo camera system may be used for line-by-line scanning of an object area intersecting the line direction, with the stereo camera system being moved relative to the object. The direction parallel to the line sensor is called the line direction and the direction perpendicular to the line sensor is called the scan direction. The distance of two points located towards each other in the scan direction are recorded with the same resolution, regardless of the keystone effect and their position in the line direction. This means that the image is not compressed in the scan direction. This improves the resolution in the scan direction.

Similarly, trapezoid formation is prevented. Both line sensors can cover the same area with their entire sensor line. Therefore, no information is lost. The overlapping areas of the line sensors are hereafter referred to as overlapping lines.

In the case of a line sensor, the aperture angle of the lens needs to cover only the overlap line, which is smaller than the diagonal of the overlap area of two area sensors that form a hexagon as described above if the sensors have the same length. Therefore, the required aperture angle of a stereo camera system with Scheimpflug conditions and area sensors is larger than that of a stereo camera system with Scheimpflug conditions and line sensors. Therefore, a stereo camera system with Scheimpflug conditions and line sensors has less distortion than a stereo camera system with Scheimpflug conditions and area sensors .

Reduced compression also reduces the need for parallelization, which reduces computation time and therefore makes image data available sooner.

In summary, it can be said that by using a stereo camera system with the Scheimpflug condition, a large base may be chosen without the occurrence of distortions as in a stereo camera system without the Scheimpflug condition. A large base results in a precise (high) resolution in the height direction. In the Scheimpflug condition, the aperture angle of the lens can be small, resulting in good image quality. However, due to the keystone effect, in the case of area sensors, the Scheimpflug condition leads to undesirable distortions in both directions. If a line sensor is used, the distortions only occur in one direction, resulting in a higher image quality. Furthermore, since the projection width of the line sensor is smaller than the projection diagonal of the area sensor, the required aperture angle of the lens is smaller for the line sensor than for the area sensor. In addition, in contrast to the area sensor, all pixels of the sensor are used.

Therefore, a stereo camera system that satisfies the Scheimpflug condition and has a line sensor may be able to obtain images with high vertical resolution using relatively simple means.

It is preferred that the Scheimpflug condition is satisfied for all beam paths. In principle, the beam paths may have different sensor areas and lens arrangements. Since all beam paths satisfy the Scheimpflug condition, a larger base may be selected compared to a stereo camera system where only one beam path satisfies the Scheimpflug condition.

Preferably, the beam paths are arranged with mirror symmetry. This arrangement makes it easier to adjust the camera system. Asymmetries can sometimes occur and be corrected. It also makes it easier to evaluate the image data, as each sensor is equidistant from the object region. Therefore, the acquired pixels are also offset by the same distance.

Preferably, one or more of the beam paths are deflected by at least one mirror. The beam path may thereby be folded by the mirror. The image of the measured object is reflected one or more times by the mirror, thereby maintaining the length of the beam path. However, the volume captured by the arrangement of optical lenses is partly significantly reduced. In particular, a large base can be selected for the stereo camera system, but the two sensor halves are still close to each other. This means that a more compact design is possible compared to a structure without a folded beam path.

In connection with the folding of the beam path, a distinction may be made between object-side folding, in which the mirror folding the beam path is arranged between the lens and the object, sensor-side folding, in which the mirror folding the beam path is arranged between the lens and the sensor, and double-sided folding, in which the mirror folding the beam path is arranged not only between the lens and the object but also between the lens and the sensor. In the case of object-side folding, the working distance, which is the distance between the lens and the object, is reduced.

The beam paths are preferably each projected onto a separate portion of one sensor. These portions each form a sensor area of one of the two or more camera modules. The sensor may be divided into multiple sensor areas, each of which is assigned a beam path. For example, in the case of a stereo camera system with two beam paths, the sensor is divided into two areas. If the stereo camera system has three beam paths, the sensor is divided into three. This also increases the stability of the system. In addition, it allows for an easy-to-adjust and compact design.

Alternatively, the camera can be designed so that each beam path is projected onto a separate sensor, with one mirror for each beam path, and the beam paths arranged in a Scheimpflug configuration. This design allows for a larger sensor surface than a single sensor in a compact form factor, leading to improved resolution.

Yet another method is characterized in that the beam paths are each projected onto a separate sensor, each beam path is provided with two mirrors, and the beam paths are arranged in a Scheimpflug configuration. This design allows for a compact geometry.

Preferably, the linear sensor areas are arranged adjacent to one another on a line, each represented by a line sensor. This line arrangement may facilitate adjustment, as the linear sensor areas may be fixed to a common support. For example, if the support is tilted, there is no need to realign (readjust) the sensor areas.

For stereoscopic methods, it is advantageous if the offset is done in one direction only. This means that the sensors are arranged parallel or perpendicular to the sensor axis. This is supported by the line arrangement.

According to a further modification, the linear sensor areas may be arranged parallel and adjacent to each other. Again, the offset of the two beam paths is only in one direction.

The linear sensor area can be a pixel line of an area sensor. Existing stereo line camera systems can be easily modified to do so.

A common lens may be provided for all beam paths. In this way, a particularly compact stereo line camera system may be created. Reducing the number of lenses may also reduce costs.

The two beam paths may have different distances between the object plane and the sensor plane. In this connection, it may be advantageous to smooth the optical path due to the difference in distance by one or more glass elements provided in each of the short beam paths.

The glass element slows down the speed of light within the glass element, and therefore the light takes longer to reach the sensor. This has the same effect as lengthening the light path. In this way, distance variations may be evened out.

In a further aspect of the invention, a stereo line camera system for measuring three-dimensional surface shapes includes at least one color sensor having a plurality of pixels sensitive to different colors and at least one hyperchromatic lens is provided for imaging the object onto the sensor. The color sensor is positioned toward the object such that different focal planes are each imaged into pixels of the same color.

In hyperchromatic lenses, the focus depends on the wavelength. Therefore, each color line of the sensor is assigned to a different focal plane.

This may be used for measuring three-dimensional surface shapes. Different heights in the object area each have a different focal plane. This ensures that each height in a particular focus area is sharply imaged on the sensor. The height resolution here corresponds to the color resolution. This method is described below as hyperchromatic depth determination. It is particularly suitable for detecting small differences in height.

Preferably, the hyperchromatic depth determination is applied together with a stereoscopic method that implements the Scheimpflug condition. In the hyperchromatic depth determination, fine height differences are detected, whereas in the stereoscopic method, height differences are detected over a larger range.

Preferably, the color lines are tilted relative to the lens. The tilt is about an axis parallel to the individual color lines, maintaining the Scheimpflug condition along the lines. This tilt changes the distance of the color lines from the lens and also moves the focal plane, which may effectively change the maximum height difference measurable in hyperchromatic depth determination.

Preferably, the color channels of the color sensor are evaluated separately, so that different focal planes can be calculated separately.

In a further aspect of the invention, a stereo line camera for measuring three-dimensional surface shapes comprises at least two area sensors designed to have a plurality of pixel lines arranged parallel and adjacent to each other, and at least two lenses for imaging an object onto one of the area sensors, respectively. The stereo line camera is characterized in that an object area having two independent beam paths is imaged simultaneously onto one of the sensors, respectively, such that a linear object area is imaged onto the pixel lines of the area sensors, but in the line plane of the beam paths intersecting the linear object area, the Scheimpflug condition is satisfied, and the area sensors are tilted about an axis such that the surface is not parallel to the object area to be scanned, so that the individual lines of the area sensors image different planes of the linear object area.

Due to the inclination, each pixel line has a different distance from the lens. Along the line, the Scheimpflug condition is maintained. Each line therefore has its own focal plane. In this way, different heights of the object area are sharply imaged on the different pixel lines. The height resolution here corresponds to the line resolution. This method is referred to below as "inclination depth determination" (Latin: inclination = inclination).

The maximum difference in height measurable in slope depth determination may be altered by changing the slope.

Height resolution can be improved by combining the dip depth determination with a stereoscopic method with the Scheimpflug condition, which reduces measurement errors since the height is detected by two different methods.

Preferably, the sensor is a color sensor and the lens is a hyperchromatic lens. This may further split the individual focal planes created by the tilt of the area sensor, improving focal plane resolution.

Preferably, the image data is rectified online, which harmonizes the image data of the stereo image pair line by line and corrects for offsets and distortion differences due to the two lenses. This rectification, which can also be done offline, adjusts the position data of each distortion so that the image data is displayed undistorted and may be further calculated. This position data does not require an additional transformation step, which increases the speed of calculations.

Line sensors using charge-coupled device (CCD) sensors have shift registers that store the detected brightness values. These shift registers are read out in succession. Typically, when reading out the shift registers for each line, each pixel is assigned a uniform positional value.

Preferably, the pixel lines of the sensors are recorded line-synchronously and synchronized to start imaging at the same time. This significantly reduces the time required for the parallelization process. Especially in combination with the parallelization process, this allows an acquired pixel of one camera module to be assigned to the same point on the object at the same time as a second acquired pixel of the second camera module. This has the advantage that in the stereo calculation, each line of the sensor area is automatically assigned to the appropriate line of the second sensor area. This means that the stereo calculation is only required in one dimension.

A further aspect of the present invention relates to a surface shape detection device. The surface shape detection device has at least one stereo camera system as described above for acquiring a three-dimensional surface shape of an object to be measured. The surface shape detection device includes a conveying device for conveying the object to be measured or the camera system, a synchronizing device for synchronizing the conveying speed with a linear scan of the three-dimensional surface shape, and an evaluation unit for evaluating the measurement result of the three-dimensional surface shape.

The conveying device may be, for example, a conveyor belt.

The synchronization device provides a signal to the stereo camera system and/or the evaluation unit to synchronize the transport speed with the measurement of the 3D surface shape. If this signal is sent to the stereo camera system, it can trigger the measurement. If the signal is sent to the evaluation unit, the speed of the transport device is determined. Also, if the image generation frequency is known, it can be determined how far one image line is from the next image line.

The synchronizing device may for example be an incremental encoder which is triggered by the transport device. If the transport device is a conveyor belt, for example by means of a wheel coupled to the conveyor belt, the position of the wheel can be determined by the synchronizing device. At a certain wheel position, a signal is emitted. Alternatively, the synchronizing device could be provided with a camera unit which detects markings on the conveyor belt and with its help emits a signal.

The synchronization device may be in the form of a module of the evaluation unit. If the image generation frequency and the speed of the transport device are known, the distance between two recorded image lines may be calculated.

The evaluation unit is a module on the computational unit that calculates the 3D surface shape from the recorded image data.

The present invention will now be described in detail by way of example with reference to the accompanying drawings, which in each case are schematic side views.

1 is a first embodiment of a camera with two lenses and two sensors. 2 is a second embodiment of a camera with two lenses, four mirrors, and one sensor. 1 is a variation of the second embodiment of a camera with two lenses, four mirrors, and two sensors. 1 is a further modification of the second embodiment of the camera with two color filters, two lenses, four mirrors, and two sensors. 3 is a third embodiment of a camera with two lenses, two mirrors, and two sensors. It is a hyperchromatic lens in which the various focal planes lie on different color lines. 4 is a fourth embodiment of a camera with three lenses, four mirrors, and a color line sensor. 11 is a variation of the fourth embodiment of the camera with one lens, four mirrors, a glass element, and a color line sensor. 1 is a block diagram showing a surface shape detection device using a stereo camera according to the present invention; 1 is a schematic layout diagram of sensors and lenses of two camera modules of a stereo camera not according to the present invention. FIG. 2 is a schematic layout diagram of sensors and lenses of two camera modules of a stereo camera. This is the sensor plane projected onto the object plane. 1 is a block diagram showing a surface shape detection device using a stereo camera according to the present invention;

In a first embodiment, the stereo camera system 25 according to the present invention includes two camera modules 1a and 1b (FIG. 1).

Each camera module 1 is formed by a lens 2 and a line sensor 3. The line sensors 3 use their respective sensor areas 13 to acquire (capture) the image formed thereon by the lens 2. The two camera modules 1 are arranged to scan a common object area 6 from two different viewing directions.

For each camera module 1, the camera axis 5 passes through the center of the sensor area 13 and the center of the lens 2, respectively. The two camera axes 5 therefore run in the field of view of the sensor area 13 to the object area 6.

The two camera axes 5 are located on a single optical plane. The two line sensors 3 and their sensor areas 13 are also located on this optical plane. This optical plane coincides with the drawing plane in Figure 1.

The base B is the distance between the centers of the two sensor areas 13. The camera axis 5 intersects with the object axis 10, forming (sandwiching) a base angle α.

Since this camera system uses a line sensor 3, only the sensor axis 8, lens axis 9 and object axis 10 are relevant for further consideration, which are straight lines that lie in the optical plane and at the same time lie in the sensor, lens and object planes defined above.

The line sensor 3 acquires an image of the object area 6 reproduced on the line sensor 3 by the lens 2. The line sensor 3 is a CCD sensor and includes a shift register and a one-dimensional array of photodetectors, each of which serves to record one pixel. In each line sensor 3, a sensor axis 8 runs coincident with the sensor line.

The lens 2 of the camera module 1 is positioned relative to each line sensor 3 such that the object area 6 is sharply depicted in the plane of the line sensor 3. The lens 2 may be formed of a single lens or multiple lenses. The optical axis of the lens 2 is the axis of symmetry of the lens 2 and runs perpendicular to the lens axis 9 in the optical plane.

The object region 6 is located on the object axis 10 and is sharply imaged onto the sensor region 13.

The sensor axis 8, lens axis 9, and object axis 10 intersect at a common point, but this common point does not exist at infinity. This satisfies the Scheimpflug condition.

The second embodiment (Figure 2a) is described below, with the same reference numbers used for elements that are the same as those in the first embodiment. The above description applies to the same elements unless otherwise specified below.

In the second embodiment, the stereo camera system 25 again includes two camera modules 1a, 1b. Each camera module 1 includes a lens 2, two mirrors 11, 12, and a line sensor area 13. The camera axis 5 is deflected by the two mirrors 11, 12. The line sensor areas 13 of each camera module 1a, 1b are aligned, one behind the other, and together form a line sensor 3.

Mirrors 11 and 12 are positioned between lens 2 and line sensor area 13 so that object area 6 falls onto (enters) line sensor area 13.

Sensor axis 8 is reflected by mirror 11. Sensor axis 8' (not shown) is the reflected sensor axis 8. Sensor axis 8' is now reflected by mirror 12. Sensor axis 8" (not shown) is the reflected sensor axis 8', i.e. the doubly reflected sensor axis 8.

The base B of the stereo camera system 25 is given by the distance between the centers of the projections of the two line sensors 3, which is the point where the line sensor 3 would be if the image of the object area 6 was not reflected.

The sensor axis 8", the lens axis 9, and the object axis 10 intersect at a common point, but this common point does not exist at infinity. This satisfies the Scheimpflug condition.

The second embodiment differs from the first embodiment in that two mirrors 11, 12, each for camera module 1a, 1b, deflect the focal points of the object axes 10 such that they lie on a common axis, the associated sensor axis 8. Therefore, there is no need to have two separate sensors, instead both beam paths may be projected onto a line sensor 3 that is split into two line sensor areas 13.

In this embodiment, the sensor axis 8 is parallel to the object axis 10. However, it is also conceivable that these two axes are not parallel. In this case, both camera modules are not of mirror symmetric design.

Alternatively, the stereo camera system 25 has a gap between the two line sensor areas 13 (FIG. 2b). Each line sensor 3 includes only one line sensor area 13. This embodiment has some similarities to the HyperSkop, but does not fall under the Scheimpflug condition.

A further option of this variant is for the two optical axes to intersect and for a first image of the object area 6 to be projected onto the opposite line sensor 3b and line sensor area 13b, respectively. Without the Scheimpflug condition, this arrangement is known as a pseudoscope.

In a further embodiment, the stereo camera system 25 has a design according to the second embodiment (FIG. 2c), but the sensor areas 13 are not arranged adjacent to each other in a line along the sensor axis 8, but instead are arranged parallel to each other perpendicular to the sensor axis 8. These sensor areas 13 may each be formed by a different color line of the line sensor 3, as will be described in detail below. Both beam paths are arranged perpendicular to the line color sensor 3 with an offset from each other, so that the two different beam paths meet slightly offset on the line color sensor 3. The two mirrors 11 are mounted adjacent to each other. And a color filter 26 is arranged along each beam path. As a result, the image of one camera module 1a is displayed on the line color sensor 3 as, for example, a red image (image), and the image of the other camera module 1b is displayed on the line color sensor 3 as, for example, a green image (image). Thereby, the line color sensor 3 may simultaneously acquire two or even more images of the object area 6. Here, the surface of the object must not be one of the colors of the color filters 26. Because otherwise the image information of the surface would be filtered out. Therefore, the beam path with the color filter of the first color hits the sensor line that detects the first color. The beam path with the color filter of the second color hits the sensor line that detects the second color.

An object surface that is white or grayscale is preferred, where each color in the color spectrum has the same intensity value. If light passes through each color filter 26, the integrated intensity of the light is reduced equally for all colors, regardless of the color of the color filter. If the color channels of the line sensor 3 are evaluated as grayscale, the resulting images of both beam paths will differ in grayscale intensity only due to different viewing angles to the object, and not due to different colors in the beam paths.

In this embodiment, the color filter 26 is mounted in front of the lenses 2a, 2b (Fig. 2c). However, the color filter 26 may also be located between the mirrors 11 and 12, or between the line color sensor 3 and the mirror 11.

Alternatively, instead of color filters, different polarizing filters may be used along the beam paths, as long as the line sensor 3 can individually acquire suitably different polarized images of the object region 6. Each beam path of the stereo camera system 25 therefore has a different polarization when it hits the line sensor 3. This method may also be used for holography, provided that the phase and intensity of the images are acquired. In this case, the object surface may not change the polarization, since the polarizing filters would otherwise filter out image data required for calculating the height data.

The third embodiment (Figure 3) is described below, with the same reference numbers used for elements that are the same as those in the first and second embodiments. The above description applies to the same elements unless otherwise specified below.

In the third embodiment, the stereo camera system 25 again includes two camera modules 1a, 1b. Each camera module 1 includes a lens 2, a mirror 11, and a line sensor 3. The camera axis 5 is deflected by the mirror 11.

The mirror 11 is positioned between the lens 2 and the line sensor 3 so that the object area 6 falls on the line sensor 3.

The sensor axis 8 is reflected by the mirror 11. The sensor axis 8' (not shown) is the reflected sensor axis 8. The two sensor axes 8 of the two camera modules 1 are not necessarily parallel to each other in this embodiment.

The base B of the stereo camera system 25 is given by the distance between the centers of the projections of the two line sensors 3, which is the point where the line sensor 3 would be if the image of the object area 6 was not reflected.

The sensor axis 8', lens axis 9, and object axis 10 intersect at a common point, but this common point is not at infinity. This satisfies the Scheimpflug condition.

In contrast to the second embodiment, the two line sensor areas 13 are no longer aligned, one behind the other. In addition, due to the mirror 11, the sensor 3 may be arranged perpendicular or nearly perpendicular to the object area 6, which makes it possible to choose a very compact design.

The fourth embodiment (Figure 5) is described below, with the same reference numbers used for elements that are the same as those in the first embodiment. The above description applies to the same elements unless otherwise specified below.

In the fourth embodiment, the stereo camera system 25 includes three camera modules 1a, 1b, 1c. Instead of a simple line sensor, the stereo camera system 25 includes a three-line sensor 15. The three-line sensor 15 includes three parallel line sensor areas 13 arranged next to each other. The sensor axis 8 runs perpendicular to the optical plane.

Each camera module 1a, 1c includes a lens 2, two mirrors 11, 12, and one of three line sensor areas 13. Camera module 1b includes a lens 2 and one of three line sensor areas 13. Camera modules 1 scan a common object area 6 from three different viewing directions.

For the outer camera modules 1a, 1c, the sensor axis 8 is reflected by mirror 11. Sensor axis 8' (not shown) is the reflected sensor axis 8. Sensor axis 8' is now reflected by mirror 12. Sensor axis 8" (not shown) is the reflected sensor axis 8', i.e. the doubly reflected sensor axis 8.

If three camera modules 1 are used in the stereo camera system 25, three base lengths B are possible for each combination of two camera modules 1. The base B is given by the distance between the centers of the projections of the two line sensors 3, the projections of the line sensors being the points where the line sensors 3 would be if the image of the object area 6 was not reflected.

The sensor axis 8 or 8", the lens axis 9, and the object axis 10 intersect at a common point, but this common point does not exist at infinity. This satisfies the Scheimpflug condition.

Alternatively, instead of the three lenses 2a-2c, the stereo camera system 25 includes only a single lens 2 that surrounds all three camera modules 1a, 1b, 1c. To compensate for this, the middle camera module 1b has a glass element 19 (Figure 6).

The fourth embodiment differs from the preceding embodiments in that the line sensor areas 13 are arranged parallel to each other, rather than aligned one behind the other.

In an alternative variant of the above-described embodiment, the lens of each camera module 1 is a hyperchromatic lens 14, the focus of which depends, inter alia, on the respective wavelength (see FIG. 4). In addition, the three-line sensor 15 is a three-line color sensor 15 having red 16a, green 16b and blue 16c lines. Due to the hyperchromatic lens 14, each of the red, green and blue lines (16a, 16b, 16c) of the three-line color sensor 15 has a different focal plane 17a, 17b, 17c, respectively. Thus, each color line 16a, 16b, 16c measures a different height measurement area 18a, 18b, 18c. In this case, the height resolution corresponds to the color resolution.

Preferably, the color lines 16 are tilted relative to the lens 14. The tilt is about an axis parallel to the individual color lines 16. The Scheimpflug condition is maintained along the color lines 16. This tilt changes the distance of the color lines 16 from the lens 14, which also moves the focal plane 17.

A further option of the present invention is that the camera module 1 has an area sensor instead of a line sensor 3. An area sensor is a two-dimensional array of photodetectors, each photodetector corresponding to one pixel. In this embodiment, the area sensor can be thought of as a series of parallel line sensors 3.

The area sensors are tilted about a tilt axis parallel to the individual line sensors 3. Each line, together with its assigned sensor axis 8 or 8' or 8", intersects the lens axis 9 and the object axis 10 at a single point and is therefore positioned to satisfy the Scheimpflug condition.

Here, the amount of angle between the normal and the optical axis of the area sensor is less than 90°. Each line of the area sensor has a different height measurement area 18. Adjacent height measurement areas lead to an enlarged height area. As a result, the same camera setup may be used for objects of different heights. Each height of the object is assigned at least one corresponding line of the area sensor with an associated height measurement area.

In an alternative variation of the embodiment described above, instead of a straight mirror 11, a curved mirror is used so that the image of the object 4 to be measured reaches the line sensor 3 without distortion.

The above-mentioned embodiments each include two or three camera modules 1. In principle, any desired number of camera modules may be provided for the stereoscopic determination of the surface shape, but at least two camera modules are necessary. The more camera modules there are, the smaller the statistical error. However, the costs also increase. The number of base lengths follows a triangular number. For example, with two camera modules 1 only one base is possible, with three camera modules 1 three, with four six options and with five ten different base lengths are possible.

The surface shape detection device 20 (Figure 7) is described below.

The surface shape detection device includes a stereo camera system 25 described in one of the above embodiments relating to any one of Figures 1 to 6 for measuring the three-dimensional surface shape of the object 4 to be measured, a transport device 21 for transporting the object 4 to be measured, a synchronization device 22 for synchronizing the transport speed with the measurement of the three-dimensional surface shape, and an evaluation device 23 for evaluating the measurement results of the three-dimensional surface shape.

In this embodiment, the conveying device 21 is a conveyor belt.

The synchronization device 22 in this embodiment is an incremental encoder 22. The incremental encoder is connected to the conveyor belt 21 so as to rotate during the movement of the conveyor belt. In the course of the rotation around a given rotation angle, a clock signal is generated, whereby the conveyor belt covers a given distance between two successive clock signals. These clock signals are passed to the evaluation device 23. The evaluation device 23 is also connected to the camera module 1 so as to obtain the image signals acquired by the sensors 3 arranged in the camera modules. The evaluation device 23 is designed to control the time at which each camera module scans the object 4 to be measured. In this way, the evaluation device 23 is able to synchronize the scan of the object 4 to be measured with the respective movements of the conveyor belt and the object 4 to be measured according to the clock signals received from the incremental encoder 22. This synchronization is preferably arranged so that the object to be measured is scanned in the direction of movement 7 at the same local distance. Several successive one-dimensional scans may be combined to give a two-dimensional image in the evaluation device 23. Three-dimensional data on the surface shape are thus obtained.

The acquired image may be corrected. On the one hand, by successively and sensitively switching the individual pixels of the line camera, shift errors resulting from the movement of the measured object 4 relative to the camera can be corrected. Thus, at high speeds, the image has a constant tilt position relative to the direction of movement 7.

In addition, the image may be rectified, which removes geometric distortions in the image data that may arise, among other things, due to the Scheimpflug geometry.

Subsequent correction, called parallelization, may be performed online directly during image registration. After taking a picture of the image line, the data is directly corrected. This involves digitizing the sensor signal followed by pre-processing of the image signal in the camera's FPGA (Field Programmable Gate Array). If the distortion parameters of the two camera modules are known, e.g. from a system calibration, the same correction model may be applied on the PC as before. Therefore, no further correction is required on the computing device, which increases the processing speed.

The images thus corrected are used for depth reconstruction of the imaging plane of the object 4 to be measured. A typical method of depth reconstruction is stereotriangulation. This involves assigning to each point of a first image a second point of a second image. The distance between the two points depends on the actual spatial depth of the point, more precisely on its distance from the object region 6 of the sensor 3 and the base B of the stereo camera system 25. In this way, each point of the first image is assigned depth information.

Alternatively, blocks, i.e. groups of pixels, such as 3x3 matrices, can be assigned to each other. This method is called block matching.

In this way, for each image line acquired, an image line containing depth information is calculated. If multiple image lines of the object 4 to be measured are acquired, the lines with depth information may be combined to generate a three-dimensional surface shape of the object to be measured. The distances of the individual lines may be calculated with the help of information from the synchronization device 22.

If a line sensor area 13 is used instead of multiple line sensors 3, the image data of all camera modules 1 is generated on the same line sensor 3. The resulting digital double image is split in the evaluation device 23. The two resulting images are corrected as described above and used for depth construction.

If a three-line sensor 15 is used to acquire the object to be measured, a separate image is generated for each color. For each color, the separate image is corrected as described above and used for depth construction. If a hyperchromatic lens is used, each color has a different height measurement area 18a, 18b, 18c, so that an additional estimate of the depth profile of the surface of the object to be measured may be calculated via image sharpness detection. Each point of the image is a point of the image of the other two color areas, but assigned to the same camera module 1. The position of the point does not differ in the three images. Thus, the image sharpness of this point differs depending on the depth and color of the actual point on the object to be measured 4. If the point is deep, it is displayed sharp in the blue image, at a medium depth in the green image and at a low depth in the red image. The sharpness detection may be performed, for example, by a contrast comparison of the images in a pixel area. The sharper a point is, the greater the contrast.

In this way, by combining it with stereoscopic calculations, it is possible to determine the surface shape in detail.

Rather than calculating each color image separately, the RGB images may be combined to calculate a grayscale image that is then analyzed.

If an area sensor is used to capture the object to be measured, a separate image is generated for each line. For each line, the separate image is corrected and used for depth construction as described above. Then, in each separate image, the sharpness of an actual point on the object to be measured 4 is determined. Thus, the line that captures the point the sharpest may be determined. Since the position and focal length of the line are known, the distance of the point is determined from these factors. If all the points are connected together, the height profile of the object to be measured 4 is obtained.

In addition to the three-line sensor 15 described above, other combinations are also possible, such as a sensor with three RGB sensor points lined up, a sensor with three sensor points lined up that also recognizes grayscale, and a sensor with three sensor points lined up that only recognizes grayscale.

The sensor may be a CCD sensor, but other sensor types are also contemplated, such as a complementary metal-oxide semiconductor (CMOS).

Reference numerals 1 camera module 2 lens 3 sensor 4 object to be measured 5 camera axis 6 object area 7 direction of movement 8 sensor axis 9 lens axis 10 object axis 11 mirror 1
12 Mirror 2
13 Line sensor area 14 Hyperchromatic lens 15 Three-line sensor 16 Color line 17 Focal plane 18 Height measurement area 19 Glass element 20 Surface shape detection device 21 Transport device 22 Synchronization device 23 Evaluation device 25 Stereo camera system 26 Color filter α Base angle β Scheimpflug angle γ Lens aperture angle B Base

Claims (18)

A stereo camera system for measuring a three-dimensional surface shape of an object, comprising:
At least two linear sensor areas;
at least one lens for imaging an object onto the sensor area;
the sensor areas and the at least one lens are arranged such that an object area is imaged simultaneously by two independent beam paths, each onto one of the sensor areas;
A stereo camera system, characterized in that the Scheimpflug condition is satisfied along at least one of the beam paths. The stereo camera system of claim 1, characterized in that the Scheimpflug condition is satisfied along all beam paths. The stereo camera system of claim 1 or 2, characterized in that one or more of the beam paths are deflected by at least one mirror. The stereo camera system according to any one of claims 1 to 3, characterized in that each of the beam paths is projected onto a different part of one sensor, and the sensor is divided into the number of beam paths. The stereo camera system according to any one of claims 1 to 3, characterized in that the beam paths are each projected onto a separate sensor, at least one mirror is provided in each beam path, and the beam paths are configured in a Scheimpflug arrangement for each beam path. The stereo camera system according to any one of claims 1 to 3, characterized in that the beam paths are each projected onto a separate sensor, at least two mirrors are provided in each beam path, and the beam paths are configured in a Scheimpflug arrangement for each beam path. The stereo camera system according to any one of claims 1 to 6, characterized in that the linear sensor areas are arranged adjacent to each other on a line, and each is represented by a single line sensor. The stereo camera system according to any one of claims 1 to 5, characterized in that the linear sensor areas are arranged parallel to and adjacent to each other. The stereo camera system of claim 8, characterized in that the linear sensor area is a pixel line of an area sensor. The stereo camera system according to any one of claims 1 to 9, characterized in that a common lens is provided for all beam paths. The stereo camera system according to any one of claims 1 to 10, characterized in that a glass element is provided in at least one of the beam paths. A stereo camera system for measuring a three-dimensional surface shape of an object according to any one of claims 1 to 11,
at least one color sensor having a plurality of pixels sensitive to different colors;
at least one hyperchromatic lens for imaging the object onto the color sensor;
11. A stereo camera system, comprising: a color sensor arranged with respect to the lens such that different focal planes are each imaged into pixels of the same color. The stereo camera system of claim 12, characterized in that the color channels of the color sensor are evaluated separately. A stereo camera system for measuring a three-dimensional surface shape of an object, comprising:
At least two area sensors, the area sensors being designed to have a plurality of pixel lines arranged parallel and adjacent to each other;
at least two lenses for respectively imaging an object onto one of the area sensors;
an object region having two independent beam paths is imaged simultaneously onto one of the area sensors,
A linear object area is imaged onto the pixel line of one area sensor;
12. A stereo camera system, characterized in that the Scheimpflug condition according to any one of claims 1 to 11 is satisfied in a line plane of the beam path that intersects with the linear object area, and the area sensor is tilted about an axis such that its surface is not parallel to the scanned object area, so that individual lines of the area sensor image different planes of the linear object area. 15. The stereo camera system according to claim 14, wherein the lens is a hyperchromatic lens according to claim 12. 16. The stereo camera system according to claim 1 , wherein the acquired data is recorded online for subsequent correction of the parallelization process. The stereo camera system of any one of claims 1 to 16, characterized in that the pixel lines of the sensors are synchronized to record line-synchronously and start imaging at the same time. A surface shape detection device, comprising at least
A stereo camera system according to any one of claims 1 to 17 for measuring a three-dimensional surface shape of an object to be measured,
A conveying device for conveying the object to be measured;
a synchronization device for synchronizing a conveying speed with the measurement of the three-dimensional surface shape;
and an evaluation device for evaluating a measurement result of the three-dimensional surface shape.

Images