JP3937103B2 - Image processing apparatus and image processing method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method capable of detecting the moving direction of the apparatus itself.
[0002]
[Prior art]
For example, in a self-propelled robot or the like, in order to accurately move to the destination, it is necessary to accurately detect the direction in which the robot is currently moving.
[0003]
Conventionally, in such a self-propelled robot or the like, for example, an image of the moving direction of the robot is captured by a CCD camera or the like, and feature points of the captured image are extracted, and then an optical flow of the feature points (Optical Flow) : Optical movement), etc., to determine the moving direction.
[0004]
[Problems to be solved by the invention]
However, in such a method, for example, the feature point of the image to be captured is the left when the robot is translated in the left direction or when the robot is rotated in the left direction (direction change). Since it moves from right to left (the feature point moves from left to right), it is difficult to distinguish them. In other words, since the amount of information to be acquired is limited, there is a problem that it is difficult to accurately determine the moving direction.
[0005]
In order to solve the above-described problems, it is conceivable to equip a plurality of CCD cameras, pick up images in different directions, and detect the moving direction of the robot itself from the moving direction of each image. However, such a method requires a plurality of CCD cameras, so that there is a problem that the cost is high.
[0006]
The present invention has been made in view of the above situation, and an object of the present invention is to realize an image processing apparatus capable of accurately and easily detecting a moving direction at low cost.
[0007]
[Means for Solving the Problems]
The image processing apparatus according to claim 1 includes an imaging unit having a predetermined field of view, and a reflection unit that reflects light in at least one direction different from the optical axis direction of the field of view of the imaging unit and enters the imaging unit. The first extraction means for extracting an image signal corresponding to the optical axis direction from among the image signals output from the imaging means, and the incident light from the reflection means among the image signals output from the imaging means Second extraction means for extracting an image signal, feature point extraction means for extracting respective feature points of the image signal extracted by the first and second extraction means, and feature points extracted by the feature point extraction means And calculating means for calculating the moving direction of the image pickup means from the optical flow.
[0008]
The image processing method according to claim 3, an imaging step having a predetermined field of view, and a reflecting step that reflects light in at least one direction different from the optical axis direction of the field of view of the imaging step and enters the imaging step. The first extraction step for extracting an image signal corresponding to the optical axis direction among the image signals output from the imaging step, and the incident light from the reflection step among the image signals output from the imaging step A second extraction step for extracting the image signal, a feature point extraction step for extracting each feature point of the image signal extracted by the first and second extraction steps, and a feature point extracted by the feature point extraction step And a calculating step for calculating a moving direction of the imaging step from the optical flow.
[0009]
In the image processing apparatus according to claim 1, the imaging unit has a predetermined field of view, and the reflection unit reflects light in at least one direction different from the optical axis direction of the field of view of the imaging unit, so that the imaging unit The image signal corresponding to the optical axis direction is extracted from the image signal incident and output from the imaging unit, and the first extraction unit extracts the image signal output from the imaging unit. The corresponding image signal is extracted by the second extraction means, the feature points of the image signals extracted by the first and second extraction means are extracted by the feature point extraction means, and extracted by the feature point extraction means The calculating means calculates the moving direction of the imaging means from the optical flow of the feature points. For example, with respect to an imaging means having a predetermined field of view such as a CCD camera, the reflection means composed of four reflecting mirrors reflects light from a total of four directions up, down, left and right with respect to the optical axis of the imaging means. The first extraction means extracts an image corresponding to the light transmitted through the transmission part, and the light in the optical axis direction transmitted through the transmission part provided in the central part of the four reflecting mirrors is incident. The second extracting means extracts images corresponding to the light in the vertical and horizontal directions reflected by the four reflecting mirrors, respectively, and the feature point extracting means is extracted by the first and second extracting means, respectively. The feature points of the images are extracted, and the calculation means calculates the moving direction of the imaging means from the optical flows of the feature points of the total of five images extracted by the feature point extraction means.
[0010]
In the image processing method according to claim 3, the imaging step has a predetermined field of view, and the reflection step reflects light in at least one direction different from the optical axis direction of the field of view of the imaging step. The image signal corresponding to the optical axis direction is extracted from the image signal incident and output from the imaging step, and the first extraction step extracts the image signal output from the imaging step. The corresponding image signal is extracted by the second extraction step, the feature points of the image signal extracted by the first and second extraction steps are extracted by the feature point extraction step, and extracted by the feature point extraction step. The calculating step calculates the moving direction of the imaging step from the optical flow of the feature points. For example, for an imaging step having a predetermined field of view as in a CCD camera, light from a total of four directions, up, down, left and right with respect to the optical axis of the imaging step, is reflected by reflection steps each consisting of four reflecting mirrors. In addition to the incident light, the light in the optical axis direction that has passed through the transmission part provided in the central part of the four reflecting mirrors is incident, and the first extraction step extracts an image corresponding to the light that has passed through the transmission part. The second extraction step extracts images corresponding to the light in the vertical and horizontal directions reflected by the four reflecting mirrors, and the feature point extraction step extracts each of the feature points extracted by the first and second extraction steps. In the calculation step, the moving direction of the imaging step is calculated from the optical flows of the feature points of the total five images extracted in the feature point extraction step.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram showing a configuration example of an embodiment of the present invention. In this figure, the reflecting mirror 1 (reflecting means) is composed of four substantially trapezoidal plane mirrors (upper image mirror 1-1, lower image mirror 1-2, left direction mirror, as shown in the front view of FIG. The upper base of the mirror 1-3 and the right-facing mirror 1-4) is opposed to the CCD camera 2 (imaging means), and the oblique sides are connected to each other so that the reflecting surface faces outward. It is formed and fixed at a position away from the CCD camera 2 by a predetermined distance. That is, these four mirrors are arranged so that the lower base of the trapezoid is closer to the back of the page than the upper base, and when viewed from the upper side of the page, has a convex shape. These four mirrors reflect light in the vertical and horizontal directions with respect to the optical axis of the CCD camera 2 and enter the CCD camera 2. Further, a transmissive portion (a transmissive portion having four sides of the upper base of the reflecting mirrors 1-1 to 1-4) provided in the center portion transmits light in the optical axis direction and enters the CCD camera 2. It is made like that.
[0012]
FIG. 3 is a diagram for explaining the positional relationship between the reflecting mirrors 1-1 to 1-4 and the CCD camera 2. As shown in FIG. As shown in this figure, the reflecting mirrors 1-1 to 1-4 are arranged in front of the CCD camera 2, and as shown in FIG. Is reflected and incident on the CCD camera 2. Further, light in the direction of the optical axis of the CCD camera 2 is incident from a transmission part having four sides on the upper bottom part of the reflecting mirrors 1-1 to 1-4.
[0013]
The light incident on the CCD camera 2 in this way is converted into a corresponding image signal and output. FIG. 5 is a diagram illustrating an example of an image output from the CCD camera 2. As shown in this figure, the image output from the CCD camera 2 includes a front image transmitted through the central transmission part of the reflecting mirrors 1-1 to 1-4, an upper image reflected by the reflecting mirror 1-1, and a reflection. It is a composite image of the lower image reflected by the mirror 1-2, the left image reflected by the reflecting mirror 1-3, and the right image reflected by the reflecting mirror 1-4.
[0014]
The front image extraction unit 3-1 (first extraction unit) is configured to extract a front image (an image corresponding to light transmitted through the center of the reflecting mirror) illustrated in FIG. The upper image extraction unit 3-2 (second extraction unit) extracts an image corresponding to the light reflected by the reflecting mirror 1-1. Similarly, the lower image extracting unit 3-3 (second extracting unit), the left image extracting unit 3-4 (second extracting unit), or the right image extracting unit 3-5 (second extracting unit). Part) extract images corresponding to the light reflected by the reflecting mirrors 1-2 to 1-4, respectively.
[0015]
The feature point extraction units 4-1 to 4-5 (feature point extraction means) are predetermined feature points (for example, from the image signals extracted by the image extraction units 3-1 to 3-5 in the front, top, bottom, left, and right directions. , A portion where the pixel value changes greatly compared to the adjacent portion).
[0016]
The optical flow calculation units 5-1 to 5-5 (calculation means) calculate the optical flow (feature point moving direction) of the feature points extracted by the feature point extraction units 4-1 to 4-5. ing. These optical flows are calculated as the presence / absence of movement in the up / down / left / right directions, the presence / absence of enlargement / reduction, or the presence / absence of rotation in the right or left direction.
[0017]
The movement direction determination unit 6 (calculation unit) is configured to determine the movement direction of the CCD camera 2 by combining the optical flows calculated by the optical flow calculation units 5-1 to 5-5.
[0018]
Next, the operation of the above embodiment will be described.
[0019]
Now, it is assumed that the embodiment shown in FIG. 1 is fixed to a predetermined portion of a self-running robot (hereinafter abbreviated as a robot). In such a case, for example, it is assumed that the robot moves forward (in the optical axis direction of the camera).
[0020]
At that time, the forward and vertical / horizontal image extraction units 3-1 to 3-5 extract the forward image and vertical / horizontal image shown in FIG. 5 from the image signal output from the CCD camera 2. And supplied to the feature point extraction units 4-1 to 4-5.
[0021]
For example, the feature point extraction units 4-1 to 4-5 extract points from which the pixel values greatly change compared to the adjacent portions from the respective images, and use the extracted points as feature points. Then, the extracted feature points are supplied to the optical flow calculation units 5-1 to 5-5.
[0022]
The optical flow calculation units 5-1 to 5-5 indicate the moving direction of the feature points extracted by the feature point extraction units 4-1 to 4-5, the current feature point position, and the previous feature. It is calculated by comparing the position of the point. For example, when the self-running robot is moving forward (in the optical axis direction of the CCD camera 2), the feature points of the front image move (diffuse) radially around the optical axis.
[0023]
At that time, the feature point of the upper image moves from top to bottom. Further, in the lower image, the feature points move from the bottom to the top, from the left to the right in the left image, and from the right to the left in the right image.
[0024]
FIG. 6 shows the moving direction of the feature points when the CCD camera 2 is moved or rotated in a predetermined direction.
[0025]
As shown in this figure, when the CCD camera 2 is moved forward (in the optical axis direction), as described above, the front image is enlarged, and the images in the up, down, left, and right directions are down, up, and right, respectively. Or move left.
[0026]
Further, when the CCD camera 2 is moved backward, the front image is reduced, and the images in the vertical and horizontal directions are moved upward, downward, left, or right, respectively.
[0027]
When the CCD camera 2 is translated upward, the upper image is enlarged, the lower image is reduced, and the front, left, and right images are all moved downward.
[0028]
When the CCD camera 2 is translated downward, the upper image is reduced, the lower image is enlarged, and the front, left, and right images are all moved upward.
[0029]
When the CCD camera 2 is translated to the left, the front, upper, and lower images all move to the right, the left image is enlarged, and the right image is reduced.
[0030]
When the CCD camera 2 is translated to the right, the front, upper, and lower images all move to the left, the left image is reduced, and the right image is enlarged.
[0031]
When the CCD camera 2 is rotated upward (when the lens of the CCD camera 2 is rotated upward and the main body is moved downward), the front image, the left image, and the right image All move down, and the upper and lower images both move up.
[0032]
When the CCD camera 2 is rotated downward (when the lens of the CCD camera 2 is rotated downward and the main body is rotated upward), the front image, the left image, and the right image All move up, and both the upper and lower images move down.
[0033]
When the CCD camera 2 is rotated to the left (when the CCD camera 2 is viewed from above, the lens of the CCD camera 2 is rotated to the left and the main body is moved to the right), the front The image, the upper image, and the lower image all move to the right, and both the left image and the right image move to the left.
[0034]
When the CCD camera 2 is rotated to the right (when the CCD camera 2 is rotated so that the lens of the CCD camera 2 moves to the right and the main body moves to the left when the CCD camera 2 is viewed from above), The image, the upper image, and the lower image all move to the left, and both the left image and the right image move to the right.
[0035]
When the CCD camera 2 is rotated to the left about its optical axis (when the main body is rotated counterclockwise when the CCD camera 2 is viewed from behind), the front image is directed to the right (clockwise). ), And the upper, lower, left, and right images move to the right, left, up, or down, respectively.
[0036]
Further, when the CCD camera 2 is rotated to the right about the optical axis (when the CCD camera 2 is viewed from the back, the main body is rotated clockwise), the front image is displayed in the left direction ( The image is rotated in the counterclockwise direction, and the upper, lower, left, and right images respectively move to the left, right, down, or up.
[0037]
The moving direction determination unit 6 shows a correspondence relationship between the moving direction of the feature point of each image (front, top, bottom, left and right images) and the moving (or rotating) direction of the CCD camera 2 as shown in FIG. A table is stored in a memory (not shown), and the movement direction of the CCD camera 2 (or by comparing the movement direction of the feature points output from the optical flow calculation units 5-1 to 5-5 with this table) , Rotation) direction is determined. For example, as described above, when the feature points of the front image are enlarged and the feature points in the up, down, left, and right directions are respectively moved down, up, right, or left (corresponding to the first row in FIG. 6). In the case), it is determined that the CCD camera 2 is moving forward.
[0038]
In the above embodiment, the four reflecting mirrors 1-1 to 1-4 are provided in the vertical and horizontal directions. For example, only the upward reflecting mirror 1-1 may be provided. Good. In that case, as shown in FIG. 6, a distinction between parallel translation to the left and rotation to the left (both the front image and the upper image move to the right) and parallel translation to the right and rotation to the right (front image) , Both of the upper images move to the left) cannot be distinguished, but other discrimination is possible.
[0039]
Therefore, for example, if the upper image reflecting mirror 1-1 and the left image reflecting mirror 1-3 are provided, the left image is enlarged in the distinction between the parallel translation to the left and the rotation to the left. Or whether it moves to the left or not. Further, the distinction between parallel translation to the right and rotation to the right can be made by determining whether the left image is reduced or moved to the right.
[0040]
In addition, what is necessary is just to select suitably the number of these reflective mirrors 1-1 thru | or 1-4, and an installation place according to the apparatus to apply.
[0041]
In the embodiment shown in FIG. 1, an embodiment using hardware is adopted. However, for example, it can be realized by using a personal computer as shown in FIG.
[0042]
FIG. 7 is a block diagram showing a configuration example of another embodiment of the present invention. In this figure, a personal computer 20 includes a CPU (Central Processing Unit) 20-1 (first extraction means, second extraction means, feature point extraction means, calculation means), ROM (Read Only Memory) 20-2, A RAM (Random Access Memory) 20-3, an IF (Interface) 20-4, and a bus 20-5 are included. A CCD camera 2 is connected to the IF 20-4, and a reflector 1 similar to that shown in FIG. 2 is disposed in front of the CCD camera 2.
[0043]
The CPU 20-1 performs various calculations and controls each unit of the apparatus. The ROM 20-2 stores various programs and data. The RAM 20-3 is configured to temporarily store data in the middle of calculation when the CPU 20-1 executes various calculations. The IF 20-4 converts the image signal output from the CCD camera 2 into a digital signal and reads it, and outputs the determination result of the moving direction of the CCD camera 2 determined by the personal computer 20. The bus 20-5 is configured to connect the CPU 20-1, the ROM 20-2, the RAM 20-3, and the IF 20-4 to each other to exchange data.
[0044]
FIG. 8 is a flowchart for explaining an example of processing executed in the embodiment shown in FIG. When this process is executed, the CPU 20-1 of the personal computer 20 extracts forward, vertical and horizontal images (see FIG. 5) from the image output from the CCD camera 2 in step S1.
[0045]
In step S <b> 2, the CPU 20-1 extracts each feature point (for example, a point where the pixel value changes greatly compared to the adjacent portion) from the image extracted in step S <b> 1.
[0046]
In step S3, the CPU 20-1 compares the feature points detected in the previous process and stored in the RAM 20-3 with the feature points extracted in the current process, and calculates the optical flow in each image. To do.
[0047]
In step S4, the CPU 20-1 collates the optical flow of each image calculated in step S3 with the table shown in FIG. 6, and determines the direction in which the CCD camera 2 is moved (or rotated). Then, the determined result is output via the IF 20-4.
[0048]
In step S5, the CPU 20-1 stores the feature points extracted in the above process (the process of step S2) in a predetermined area of the RAM 20-3. The feature points stored in this way are referred to as feature points obtained in the previous process in the next process.
[0049]
In subsequent step S6, CPU 20-1 determines whether or not to end the process. As a result, when it is determined that the process is to be continued (NO), the process returns to step S1 and the same process as described above is repeated. If it is determined that the process is to be ended (YES), the process is ended (END).
[0050]
According to the embodiment described above, the moving direction of the CCD camera 2 can be easily and accurately detected by the combination of the personal computer 20 and software.
[0051]
In the above embodiment, the four reflecting mirrors 1-1 to 1-4 are arranged in front of the CCD camera 2, but for example, only the reflecting mirror 1-1 is arranged. May be rotated about the optical axis of the CCD camera 2 to extract an image when the reflecting mirror moves to a specific position (a position corresponding to the reflecting mirrors 1-1 to 1-4).
[0052]
【The invention's effect】
According to the image processing device according to claim 1 and the image processing method according to claim 3, an image signal is generated from the optical axis direction and light in at least one other direction different from the optical axis direction, An image signal corresponding to the axial direction is extracted, an image signal corresponding to light in a direction different from the optical axis direction is extracted, and feature points of each of the two extracted image signals are extracted, and the extracted features are extracted. Since the moving direction is calculated from the optical flow of the points, the moving direction of the apparatus can be detected accurately and easily.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration example of an embodiment of the present invention.
FIG. 2 is a front view showing an arrangement state of the reflecting mirrors 1-1 to 1-4 shown in FIG.
3 is a diagram showing a positional relationship between the reflecting mirrors 1-1 to 1-4 and the CCD camera 2 shown in FIG.
4 is a diagram showing a positional relationship when the reflecting mirrors 1-1 to 1-4 and the CCD camera 2 shown in FIG. 1 are viewed from the lateral direction.
FIG. 5 is a diagram showing an example of an image captured by the embodiment shown in FIG. 1;
6 is a diagram showing a correspondence relationship between the moving direction (or rotation direction) of the CCD camera 2 shown in FIG. 1 and the moving direction (or enlargement / reduction or rotation direction) of each image. FIG.
FIG. 7 is a block diagram showing a configuration example of another embodiment of the present invention.
FIG. 8 is a flowchart illustrating processing executed in the embodiment shown in FIG.
[Explanation of symbols]
1-1 to 1-4 Reflector (reflecting means), 2 CCD camera (imaging means), 3-1 Front image extracting means (first extracting means), 3-2 Upper image extracting section (second extracting means) ), 3-3 Lower image extraction unit (second extraction unit), 3-4 Left image extraction unit (second extraction unit), 3-5 Right image extraction unit (second extraction unit), 4 -1 to 4-5 Feature point extraction unit (feature point extraction unit), 5-1 to 5-5 Optical flow calculation unit (calculation unit), 6 Movement direction determination unit (calculation unit), 20-1 CPU (first Extraction means, second extraction means, feature point extraction means, calculation means)

Claims (3)

Imaging means having a predetermined field of view;
Reflecting means that reflects light in at least one direction different from the optical axis direction of the visual field of the imaging means and enters the imaging means;
A first extraction unit for extracting an image signal corresponding to the optical axis direction from among the image signals output from the imaging unit;
A second extraction unit that extracts an image signal corresponding to incident light from the reflection unit out of the image signal output from the imaging unit;
Feature point extracting means for extracting each feature point of the image signal extracted by the first and second extracting means;
An image processing apparatus comprising: a calculation unit that calculates a moving direction of the imaging unit from an optical flow of the feature point extracted by the feature point extraction unit. The reflecting means is formed by connecting the upper base portion of four substantially trapezoidal plane mirrors to the imaging means and connecting the oblique sides to each other so that the reflecting surface faces outward,
The imaging means includes light in the optical axis direction that has passed through a transmission part having four sides of the upper bottom part of the four plane mirrors, and vertical and horizontal directions reflected by the four plane mirrors relative to the optical axis. The image processing apparatus according to claim 1, wherein the reflected light is incident. An imaging step having a predetermined field of view;
A reflecting step that reflects light in at least one direction different from the optical axis direction of the visual field of the imaging step and enters the imaging step;
A first extraction step of extracting an image signal corresponding to the optical axis direction among the image signals output from the imaging step;
A second extraction step of extracting an image signal corresponding to incident light from the reflection step out of the image signal output from the imaging step;
A feature point extraction step of extracting each feature point of the image signal extracted by the first and second extraction steps;
An image processing method comprising: a calculation step of calculating a moving direction of the imaging step from an optical flow of the feature points extracted by the feature point extraction step.

Images