JP4380376B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to Augmented Reality technology that makes it appear as if there is a virtual object in real space, and more particularly to a system related to layout simulation in which a virtual object is arranged at a predetermined location.

  Augmented Reality is a technology that reinforces real-world information with virtual world data (CG: Computer Graphics) built inside the computer, and is expected to open up an application field that is completely different from Virtual Reality, which handles only the world of CG. A lot of research has been done.

For example, in Non-Patent Document 1, a black frame marker with a picture inside is used, a space in front of the eye is photographed with a camera, and when a marker is detected in the camera image, a 3D CG is synthesized at that position. A technique for generating an image is disclosed. In this technique, the three-dimensional relative position between the camera and the black frame is calculated based on the size and shape of the black frame shown in the camera image, and the image written inside the black frame is stored in advance in association with the CG to be synthesized. The CG corresponding to the pattern written inside the black frame is synthesized. This makes it possible to synthesize a plurality of different CGs by changing the pattern of the marker.
Billinghust & Kato, “Collaborative Mixed Reality.”, IN Proceedings of the First International Symposium on Mixed Reality (ISMR'99). Mixed Reality -Merging Real and Virtual Worlds, pp. 261-284. Kato, Amagasaki, Kei Tachibana, “Alignment Method for Augmented Reality Based on Matching Templates Generated Online from Texture Images”, Transactions of the Virtual Reality Society of Japan, Vol. 7, no. 2, pp119-128 (2002)

  However, when the number of CGs to be synthesized is increased in the above-described conventional technique, it is necessary to increase the number of patterns used for the marker correspondingly. However, when the number of patterns is increased, the number of patterns in the system is increased. There was a problem that a CG that was mistakenly recognized and different from the intended CG would be synthesized. In other words, the use of a large number of markers tends to make it difficult to distinguish between markers, and when a large number of markers are placed, the area of the marker becomes smaller depending on the distance and orientation of the markers. It becomes difficult to distinguish. Therefore, there is a problem that erroneous recognition is likely to occur. In addition, when a large number of markers are used, there is a problem that information to be searched increases and high-speed processing becomes difficult.

  The present invention has been made in view of the above points, and in a technique for synthesizing CG using a marker, it is possible to reliably synthesize CG even if the number of candidate CGs to be synthesized is increased. The purpose is to provide.

The above-described problem is an image processing apparatus for combining an image corresponding to a marker with an image obtained by photographing a subject including the marker with a camera, and a marker information temporary storage unit for temporarily storing marker information And marker information extraction storage means for extracting marker information of some of the plurality of markers from marker information, and storing the extracted marker information in the marker information temporary storage means, 3D position calculation means for detecting a marker from an image captured by the camera using the marker information stored in the marker information temporary storage means and calculating the 3D position of the marker; and the 3D position calculation means Based on the calculation result obtained by the above, the image corresponding to the marker is synthesized with the position of the marker in the image photographed by the camera and output to the image display means. With the door, said marker information extracting storage means by said camera or said camera obtains an image of the captured marker with a different camera, comparing the image of the marker and the image of the plurality of markers, A marker image having the highest similarity to the acquired marker image is detected from the plurality of marker images, and marker information of the detected marker is stored in the marker information temporary storage unit. It can be solved by an image processing apparatus.

According to the present invention, since marker detection and image synthesis are performed using marker information of some markers extracted from a plurality of markers, marker image detection is possible even when the number of the plurality of markers is very large. The number of collation target markers used sometimes can be suppressed to a predetermined number or less. Therefore, erroneous recognition of the marker can be reduced. Further, with such a configuration, desired marker information can be easily stored in the marker information temporary storage means.

  The marker information can be configured as information including marker image information, position information corresponding to at least three feature points in the marker, and three-dimensional CG data.

  According to the present invention, in the technique of combining a CG with an image using a marker, even if the number of candidate CGs to be combined is increased, the CG can be reliably combined with the corresponding marker position. .

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  First, the schematic configuration of the embodiment of the present invention will be described with reference to FIG. In FIG. 1, information of a plurality of markers and a three-dimensional CG associated with the markers are stored in advance in the image processing apparatus 3 (four types of markers are shown in the example of FIG. 1).

  A camera 1 and a camera 2 are connected to the image processing apparatus 3. First, the camera 1 captures one of the stored markers. The image processing apparatus 3 recognizes which of the plurality of markers stored in advance is the marker photographed by the camera 1, and stores only the three-dimensional CG corresponding to the marker information. Store in a separate area.

  Then, using the camera 2, an actual landscape where the marker is arranged is photographed. The image processing apparatus 3 detects a marker stored in the other area from the landscape image. Then, the three-dimensional position of the marker with respect to the camera 2 is calculated from the position and shape of the marker in the image and the marker information stored corresponding to the marker. Then, the image processing device 3 synthesizes the three-dimensional CG associated with the marker at the three-dimensional position in the image and outputs it to the display 4.

  In this way, by storing only information on markers that will be used in advance in another area, and by using only markers stored in another area as targets for verification, the number of targets to be verified can be reduced. Reduction and faster processing can be achieved.

  FIG. 2 is a diagram showing in more detail the configuration of the image processing apparatus 3 according to the embodiment of the present invention. In the image processing apparatus 3 according to the present embodiment, n markers are used as targets for placing a virtual object in real space, and the marker information I1 to n is stored in advance in the image processing apparatus 3. To do. One marker information Ix (1 ≦ x ≦ n) includes marker image information Mx, three-dimensional position specifying information Dx for specifying the three-dimensional position of the marker, and three-dimensional CG data Gx synthesized on the marker. The three-dimensional position specifying information Dx for specifying the three-dimensional position of the marker is, for example, a coordinate value in the marker of at least three feature points of the marker. Each feature point is a point that can be identified from an image including the marker.

  As shown in FIG. 2, the image processing apparatus 3 uses the marker 1 camera 1, the marker information storage unit 5 that stores the marker information I 1 to n, and the marker information stored in the marker information storage unit 5. A comparison unit 6 that identifies a marker in the image, m registers Ry (1 ≦ y ≦ m, m ≦ n) for temporarily storing marker information identified based on the comparison result, and a camera 2 for taking a panoramic view The three-dimensional position information obtained by the three-dimensional position calculator 8 that calculates the three-dimensional position of the marker reflected in the image P2 of the panoramic camera 2 based on the information in the register R is obtained. An image composition unit 9 that renders three-dimensional CG data corresponding to the marker stored in the register R on the camera image and an image display unit 4 that displays the composite image are provided.

  Next, an outline of the operation in the configuration shown in FIG. 2 will be described.

  When the user holds the marker over the camera 1, the image processing device 3 compares the camera image with the n pieces of marker image information by the comparison unit 6 and selects marker information corresponding to the closest marker image information. The selected marker information (marker image information, 3D position specifying information, 3D CG data) is stored in the register R. For example, by performing this operation on each of m different markers, marker information is stored in each of the m registers R. Further, instead of holding each of the m markers over the camera 1, m cameras may be simultaneously photographed with the camera 1.

  Next, the user captures a scene including the marker on the camera 2. The three-dimensional position calculation unit 8 detects a marker from the camera image P2 of the camera 2 based on the marker image information stored in the register R. Then, the relative three-dimensional position of the marker with respect to the camera 2 is obtained based on the three-dimensional position specifying information stored in the register R corresponding to the marker. The obtained result is given to the image synthesizing unit 9, and the image synthesizing unit 9 synthesizes the camera image P2 and the three-dimensional CG data corresponding to the marker from the value, and displays them on the image display unit 4.

  The processing by the comparison unit 6 and the processing for detecting a marker from the camera image P2 can be realized by an image comparison algorithm such as DP matching. In the processing by the comparison unit 6, when the marker is projected with a certain degree of resolution (about VGA), it can be compared with about several thousand registered marker images. A method for calculating the three-dimensional position based on the three-dimensional position specifying information and the marker image in the camera image P2 has been proposed by Kato et al. (See Non-Patent Document 2). When processing such as 30 to 15 frames per second is performed, the number of markers that can be handled in the three-dimensional position calculation is at most several tens, and the number m of registers may be set to a value smaller than this number.

  Next, the processing in the image processing apparatus 3 will be described in more detail with reference to FIG.

  First, the storing process of the marker information in the register R by the comparison unit 6 will be described. First, j = 1 is set (step 1). Here, j is a variable corresponding to the marker photographed by the camera 1. The comparison unit 6 acquires an image of the camera 1 (step 2) and sets i = 1 (step 3). Here, i is a variable corresponding to the marker information stored in the marker information storage unit 5.

  The comparison unit 6 compares the i-th marker image information stored in the marker information storage unit 5 with the camera image acquired in step 2, and calculates a similarity (step 4). If i = i + 1 (step 5) and i is not larger than n (no in step 6), it means that the comparison target marker image information still remains, so the processing from step 4 is performed again. If i is larger than n (Yes in Step 6), the marker information having the highest similarity among the n comparison results is stored in the j-th register R (Step 7). If j = j + 1 and j is not larger than m (the number of registers R) (no in step 8), the process returns to step 2. If j is larger than m (Yes in Step 8), the process returns to Step 1.

  After the storage process in the register R as described above, the image of the camera 2 and the three-dimensional CG are combined. Even when the number of pieces of marker information stored in the register R is less than m, the combining process can be performed as appropriate.

  First, the three-dimensional position calculation unit 8 acquires an image photographed by the camera 2 (step 11) and sets j = 1 (step 12). The three-dimensional position calculation unit 8 detects the marker corresponding to the marker image from the camera image by the camera 2 using the marker image information stored in the j-th register R. (Step 13). When the marker is detected (Yes in Step 14), three feature points of the marker in the camera image are detected (Step 15), and the position of the feature point reflected in the camera image and the like are stored in the register R. The three-dimensional position (the camera coordinates of the marker) of the marker is calculated from the j-th three-dimensional position specifying information (step 16). Then, the j-th three-dimensional CG stored in the register R is drawn at the position of the marker in the camera image (step 17). Subsequently, j = j + 1 is set. If j is not larger than m, the process returns to step 13 (no in step 18), and the same processing as described above is performed for the next marker stored in the register R. If j is larger than m (Yes in Step 18), the processing is completed for m markers in the register R, and a camera image in which a 3D CG is synthesized at the marker position is output to the image display unit 4. (Step 19).

  By using the method according to the present embodiment, even if there are about several thousand kinds of markers that can be used for the three-dimensional CG synthesis, by using the camera 1 in advance, the marker that is to be used for photographing can be selected. Since it is recorded in the register R in the image processing apparatus 3 and then the image of the camera 2 and the three-dimensional CG are synthesized, 3 types of 3 more than the conventional system in which it is necessary to perform search processing for all markers. Dimensional CG can be handled. Further, when the number of markers to be photographed on the camera 1 exceeds m, the register R overflows. In this case, this problem can be avoided by deleting the oldest registration. In the method according to the present embodiment, the number of CGs that can be simultaneously displayed in the image display unit 4 is limited to m. However, in many applications, it is sufficient to display at most m.

  In the above embodiment, recognition of a marker for registering the marker in the register R is performed by image comparison using the camera 1. Instead of such a method, by attaching a barcode to the marker and using a barcode reader instead of the camera 1, the marker can be recognized more reliably than the image comparison method.

  In addition, by sharing the cameras 1 and 2 and giving an image of a specific area of the camera to the comparison unit 6, it is possible to realize the above method with a single camera. Further, if the comparison unit 6 having sufficiently high calculation processing capability is used, sufficient processing can be performed even if an image of a full screen is given to the comparison unit 6 instead of a specific area. That is, in this case, a landscape including the marker is photographed, the marker is detected from the image, the marker information of the marker is registered in the register R, and then photographing for CG synthesis is performed.

  The image processing apparatus 3 according to the present embodiment can be realized by causing a computer including a CPU, a memory, a hard disk, and the like to execute a program that performs the processing described in the present embodiment. That is, the comparison unit 6, the three-dimensional position calculation unit 8, the image composition unit 9, and the like are functional units that are realized by causing a computer to execute a program. J and i in FIG. 3 are variables in the program. The marker information storage 5 and the register R are realized by a storage device in a computer such as a memory or a hard disk. Further, the above program can be stored in a recording medium such as a CD-ROM or an SD card and installed in the computer from the recording medium.

  1 and FIG. 2, for example, the comparison unit 6 and the marker information storage unit 5 are provided in one computer, and the three-dimensional position calculation unit 8, the register R, the image composition unit 9, and the image display unit. 4 may be provided in another computer, and these computers may be connected via a network. Even in such a case, both computers are referred to as an image processing apparatus.

  As described above, according to the embodiment of the present invention, marker information used for image composition is extracted from a large amount of marker information stored in advance and held in another area, and the marker held in the other area Marker detection and CG synthesis are performed using only information, so even if the number of types of markers increases, the number of markers to be detected is limited to a fixed number and the number of verification targets decreases. It can be reduced and the processing speed can be increased.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

It is a figure for demonstrating the schematic structure of embodiment of this invention. It is a figure which shows the structure of the image processing apparatus 3 in embodiment of this invention. 3 is a flowchart of processing in the image processing apparatus 3.

Explanation of symbols

1, 2 Camera 3 Image processing device 4 Image display unit (display)
5 Marker information storage unit 6 Comparison unit 8 3D position calculation unit 9 Image composition unit R Register

Claims (7)

An image processing apparatus for combining an image corresponding to a marker with an image obtained by photographing a subject including the marker with a camera,
Marker information temporary storage means for temporarily storing marker information;
Marker information extraction and storage means for extracting marker information of some of the plurality of markers from marker information and storing the extracted marker information in the marker information temporary storage means;
Using the marker information stored in the marker information temporary storage means, detecting a marker from an image taken by the camera, and calculating a three-dimensional position of the marker;
Based on the calculation result by the three-dimensional position calculation means, comprising an image synthesis means for synthesizing an image corresponding to the marker in the position of the marker in the image photographed by the camera, and outputting to the image display means ,
The marker information extraction and storage means obtains an image of a marker photographed by the camera or a camera different from the camera, and compares the marker image with the images of the plurality of markers, thereby obtaining the plurality of markers. An image processing apparatus , wherein an image of a marker having the highest similarity to the acquired marker image is detected from the acquired images, and the marker information of the detected marker is stored in the marker information temporary storage means .   The image processing apparatus according to claim 1, wherein the marker information includes image information of the marker, position information corresponding to at least three feature points in the marker, and three-dimensional CG data. An image processing method executed by an image processing device for combining an image corresponding to a marker with an image obtained by photographing a subject including the marker with a camera,
The marker information of a part of the plurality of markers is extracted from the marker information of the plurality of markers, and the extracted marker information is stored in the marker information temporary storage means for temporarily storing the marker information. Marker information extraction storage step;
Using the marker information stored in the marker information temporary storage means, detecting a marker from an image captured by the camera, and calculating a three-dimensional position of the marker;
An image synthesis step of synthesizing an image corresponding to the marker in the position of the marker in the image captured by the camera based on the calculation result of the three-dimensional position calculation step, and outputting the image to the image display means ;
In the marker information extracting and storing step, the image processing apparatus acquires a marker image captured by the camera or a camera different from the camera, and compares the marker images with the plurality of marker images. The marker image having the highest similarity to the acquired marker image is detected from the plurality of marker images, and the marker information of the detected marker is stored in the marker information temporary storage means. An image processing method. The image processing method according to claim 3 , wherein the marker information includes image information of the marker, position information corresponding to at least three feature points in the marker, and three-dimensional CG data. A program for causing a computer to function as an image processing device for synthesizing an image corresponding to a marker on an image obtained by photographing a subject including the marker with a camera,
Marker information temporary storage means for temporarily storing marker information,
Marker information extraction storage means for extracting marker information of some of the plurality of markers from the marker information of the plurality of markers and storing the extracted marker information in the marker information temporary storage means;
3D position calculation means for detecting a marker from an image captured by the camera using the marker information stored in the marker information temporary storage means, and calculating the 3D position of the marker;
A program for synthesizing an image corresponding to a marker at a marker position in an image photographed by the camera based on a calculation result by the three-dimensional position calculation unit, and for causing the image display unit to function as an image synthesis unit. ,
The marker information extraction and storage means obtains an image of a marker photographed by the camera or a camera different from the camera, and compares the marker image with the images of the plurality of markers, thereby obtaining the plurality of markers. A program for detecting a marker image having the highest similarity to the acquired marker image and storing the marker information of the detected marker in the marker information temporary storage means . The program according to claim 5 , wherein the marker information includes image information of the marker, position information corresponding to at least three feature points in the marker, and three-dimensional CG data. A computer-readable recording medium on which the program according to claim 5 or 6 is recorded.

Images