JP7693191B2 - Aerial image generating device, distortion correction function generating method, and aerial image generating method - Google Patents

Description

The present invention relates to an aerial image generating device that displays an aerial image when a transparent or reflective object is present on the optical path, a distortion correction function generating method, and an aerial image generating method.

There is a known technology that uses optical elements or the like to reflect and refract light emitted from a light source, forming an aerial image (real image) at a specified position in a space where no screen or display is located. Aerial images give an observer who sees them a mysterious sensation, and are therefore used in a variety of applications, including virtual reality (VR) and augmented reality (AR).

In virtual reality and augmented reality, it is important that the image display does not simply convey information to the observer, but rather attracts the observer's attention and allows natural interaction with real objects. In that sense, image projection onto real objects has attracted attention because it can add dynamic and interactive expression to static buildings and natural landscapes.

Patent Document 1 describes a technology for forming an aerial image using retrotransmitting optical elements made of micro-mirror array plates (MMAPs) to retrotransmit incident light and form an aerial image, with a high degree of freedom in terms of installation location and with the contents displayed on the display unit not being directly visible to the observer.

Patent document 2 describes a technology for an image processing device that generates high-quality images for projection, which calculates correction parameters for correcting the projected image and corrects the pixel positions that make up the projected image based on the calculated correction parameters and data related to the installation state of the projection unit.

The technology described in Patent Document 2 prepares an image for calculating correction parameters, such as a white image or a grid image, and calculates the correction parameters necessary for geometric transformation of the image to be projected onto the output device. It also generates virtual space data including data on the shape and reflection characteristics of a virtual subject placed in the virtual space, and position data of a virtual camera. The technology described in Patent Document 2 corrects the pixel positions that make up the projected image using these correction parameters and virtual space data.

There is also a technique described in Non-Patent Document 1 as a method for displaying an image inside a transparent object. The optical system described in Non-Patent Document 1 also uses retrotransmissive optical elements (MMAPs), and these MMAPs are used to form an image of light from a light source object inside the transparent object. The technique described in Non-Patent Document 1 furthermore places a mirror image of the transparent object to be imaged on the light source side, thereby compensating for distortion caused by refraction of the transparent object.

The optical system described in Non-Patent Document 1 is an optical system for presenting aerial images (real images) that give the impression of floating within a transparent object without compromising the appearance of the transparent object.

JP 2020-60752 A JP 2016-72691 A

"Tetsuya Yokose, Tatsuya Okawa, Takeshi Naemura, InFloasion: Optical system for forming an image inside a transparent object," IEICE Technical Report, Vol. 118, pp. 89-94, 2018

However, in conventional image display methods such as those described in Patent Documents 1 and 2, the image display surface is limited to being hollow or on the surface of an object, making it difficult to express an image embedded inside a transparent object. In other words, when forming an image inside a solid transparent object, the change in the optical path due to refraction that occurs at the object's boundary surface cannot be ignored, so even if an attempt is made to display an image inside a transparent object, the image cannot be displayed correctly at the desired position.

Furthermore, in the optical system described in Non-Patent Document 1, in order to achieve a sense of realism in the image, only the refractive distortion occurring on the light source side is subject to compensation, and as a result, there is a problem in that the refractive distortion occurring in the viewing direction cannot be compensated for.
Furthermore, in the optical system described in Non-Patent Document 1, a mirror image of the target transparent object or a transparent object cut at the image display position is required on the light source side, but it is difficult to prepare a compensating transparent object for each shape and arrangement method of the transparent object.

The object of the present invention is to provide an aerial image generating device, a distortion correction function generating method, and an aerial image generating method that can generate an aerial image without distortion at a preset spatial position even if there is a transparent object or a reflective object in the optical path from the light source.

In order to solve the above problem, the aerial image generating device of the present invention includes an image processing unit that generates a distortion-corrected image that applies distortion to an input image signal, a distortion-corrected image storage unit that stores the distortion-corrected image generated by the image processing unit, a distortion-corrected image display light source that uses the distortion- corrected image stored in the distortion-corrected image storage unit as a light source, an aerial image optical system that has at least one of a transparent object that transmits light and a reflective object that reflects light and a retrotransmissive optical element in an optical path between the distortion-corrected image display light source and the aerial image, and that receives light emitted from the distortion-corrected image display light source and generates an aerial image as a real image at a predetermined spatial position, and a light source distortion correction function generating unit that generates a distortion correction function based on a calculation image obtained by recording pixel values of the transparent object and/or the reflective object installed in the aerial image optical system and the light beam that has passed through the retrotransmissive optical element at the light source installation position of the actual machine . The image processing unit generates the distortion-corrected image so that the distortion of the aerial image provided by the aerial image optical system is corrected.

In addition, the distortion correction function generating method of the present invention includes steps (1) to (3) in CG space, and the aerial image generating method of the present invention includes, in addition to steps (1) to (3) in CG space, steps (4) and (5) in the actual machine.
(1) Designing an aerial image optical system including a positional relationship between a transparent object or a reflective object and a retrotransmissive optical element;
(2) outputting an ideal camera image by rendering a portion of the ideal image arranged at the aerial image display position in the CG space within the angle of view of the CG camera corresponding to the viewpoint;
(3) A step of generating a distortion correction function from an image for calculation generated by emitting a light ray having a pixel value of each pixel of an ideal camera image from a CG camera, and recording the pixel values of all pixels of the ideal camera image when the light ray reaches a light source installation position of the actual device via an aerial image optical system;
(4) generating a distortion-corrected image by applying distortion to an input image signal using a distortion correction function, and generating a distortion-corrected image display light source based on the distortion-corrected image;
(5) A step of placing the aerial image optical system and the distortion-corrected image display light source designed in the designing step (1) in a real space, receiving light emitted from the distortion-corrected image display light source, and generating an aerial image as a real image at a predetermined spatial position.

According to the present invention, it is possible to generate an aerial image in a predetermined space in which image distortion caused by a transparent object or a reflective object is compensated for.
Problems, configurations and effects other than those described above will become apparent from the following description of the embodiments.

1 is a block diagram showing an overall configuration of an aerial image generating device according to an embodiment of the present invention. 1A to 1C are diagrams illustrating four steps of an aerial image generating method according to an embodiment of the present invention. 3 is a diagram for explaining the design process of the optical system in process A shown in FIG. 2 . FIG. FIG. 3 is a diagram for explaining the rendering of an ideal camera image in process B shown in FIG. 2 . 3 is a flowchart showing the procedure of an ideal camera image generation process in relation to step B shown in FIG. 2 . 3 is a flowchart showing a procedure for generating a distortion correction function in step C shown in FIG. 2 . 3 is a diagram illustrating a method for generating a distortion correction function in step C shown in FIG. 2. FIG. 3 is a diagram showing an example of generating an aerial image inside a transparent object from a light source image generated based on the distortion correction function generated in step D shown in FIG. 2 . FIG. 1A is a diagram for explaining the structure of an MMAPs (retrotransmissive optical element) and FIG. 1B is a diagram for explaining the principle of MMAPsBRDF. 13A is a diagram for explaining the difference between a calculation image generated by MMAPs and a calculation image generated by MMAPsBRDF. FIG. 11A and 11B are diagrams showing a comparison between an image inside a transparent object obtained by a conventional method and an image inside a transparent object obtained by an embodiment of the present invention in a mounting test. 13 is a diagram showing that the effects of the embodiment of the present invention were confirmed by a mounting test. 1A to 1C are diagrams showing modified examples of an embodiment of the present invention.

The basic configuration and embodiments of the present invention will be described in detail below with reference to the attached drawings. In this specification and drawings, elements having substantially the same configuration or function are given the same reference numerals, and duplicated descriptions will be omitted.

<Overall configuration of the aerial image generating device>
1 is a diagram showing the overall configuration of an aerial image generating device 100 according to an embodiment of the present invention. In general, when an aerial image is generated at a position inside a transparent object or at a position passing through a transparent object, the light rays passing through the transparent object are refracted, causing distortion in the aerial image.
It is therefore necessary to prevent distortion of the shape and color of the image produced in the air.

As shown in FIG. 1, the aerial image generating device 100 of this embodiment includes a light source distortion correction function generating unit 101. As described below, this light source distortion correction function generating unit 101 is a functional block that calculates the distortion of the image when it passes through each optical system and derives a correction function to prevent distortion from occurring in the aerial image, and is prepared in advance outside the actual machine when forming the aerial image.

The aerial image generating device 100 of this embodiment also includes an image processing unit 102 that performs distortion processing, a distortion-corrected image storage unit 103, a distortion-corrected image display light source 104, and an aerial image optical system 105. The distortion-corrected image storage unit 103, the distortion-corrected image display light source 104, and the aerial image optical system 105 together constitute an aerial image display unit.

The image processing unit 102 is supplied with an image signal on which the aerial image is based and a distortion correction function generated by the light source distortion correction function generation unit 101, and the distortion correction function adds a distortion to the supplied image signal according to a specific optical system (described later). The corrected image (distortion corrected image) to which distortion has been added by the image processing unit 102 is recorded in the distortion corrected image storage unit 103. Note that the image signal on which the aerial image is based is an image signal that displays an undistorted image when input to a general display device.

The distortion-corrected image stored in the distortion-corrected image storage unit 103 then becomes the distortion-corrected image display light source 104, and an aerial image is displayed as a real image at a predetermined spatial position via the aerial image optical system 105. The details of the aerial image optical system 105 will be described later, but it includes at least MMAPs, which are retrotransmissive optical elements, and a transparent object (or a reflective object).

<Steps of the aerial image generation method>
FIG. 2 is a process diagram of an aerial image generating method for generating an aerial image using the aerial image generating apparatus of this embodiment.
As shown in Fig. 2, the aerial image generating method of this embodiment includes process A of designing the aerial image optical system 105 shown in Fig. 1 and the arrangement of the distortion corrected image display light source 104 and the aerial image optical system 105, process B of rendering an ideal camera image by simulation using CG, process C of generating a distortion correction function from an image for calculation created using the ideal camera image, and process D of applying distortion to an image signal using the generated distortion correction function to display the image at a predetermined spatial position. Note that process A is design within CG space, and process D is a process in which a corresponding actual device is assembled in real space for the aerial image optical system designed in process A and implemented within the actual device.

Here, steps A to C are processes in CG space. Taking pictures in CG space with a camera is called rendering in CG, and one method for this rendering is ray tracing.
Hereinafter, a procedure for generating an aerial image in the aerial image generating apparatus and aerial image generating method of this embodiment will be described in the order of steps A to D.

<Design of the Aerial Image Optical System 105>
The aerial image optical system 105 shown in Fig. 1 is an optical system for generating a real image in a specified space by using a special optical element such as a retrotransmissive optical element MMAPs with light from a light source. This makes it possible to express an image in which the image appears to float in mid-air.

As shown in Figures 3(a) and (b), the aerial image optical system 105 in this embodiment is composed of MMAPs 10 for generating an aerial image 40 that is installed at an angle, and a transparent object 30 of any shape that is placed at the position where the aerial image 40 is generated or between the position where the aerial image 40 is generated and the MMAPs 10.

Figure 3(a) shows an example of an optical system in which an aerial image 40 is generated inside a transparent object 30, and Figure 3(b) shows an example of an optical system in which an aerial image 40 is generated outside the transparent object 30. Here, a material with a smooth surface and high transparency is used as the transparent object 30. The distance between the transparent object 30 and the MMAPs 10 is designed to be the same as the distance L between the display serving as the light source 20 and the MMAPs 10.

In addition, in FIG. 3(a), in order to display the aerial image 40 inside the transparent object 30, the size of the aerial image 40 needs to be smaller than the cross section of the transparent object 30 at the imaging position. Although not shown in FIG. 3, it is also possible that the transparent object 30 is placed between the light source 20 and the MMAPs 10.

First, the structure and function of the retrotransmissive optical element MMAPs10 used in this aerial image optical system will be described.
The MMAPs 10 is an optical element that reflects light incident from the light source 20 twice and transmits it to form a real image (aerial image 40) at a plane-symmetrical position. Specifically, the MMAPs 10 is configured with a plurality of micromirrors arranged in a plane, and deflects light incident at 45° to the plane by 90° to form a real image at a plane-symmetrical position. By configuring the image formation positions of the light source 20 and the aerial image 40 as described above, the brightness of the aerial image formed by the MMAPs 10 is maximized.

MMAPs10 are used in digital signage, amusement facilities, and entertainment in general because they are easy to install and create an aerial image 40 in a position that is plane-symmetrical to the light source 20. However, with MMAPs10, light attenuation occurs due to two reflections, and there is also a considerable amount of stray light, so not all incident light can be used as the aerial image 40. It is said that less than 50% of the incident light is used as the aerial image 40.

<Rendering ideal camera images>
3A and 3B, when an aerial image 40 is generated, a transparent object 30 that is present between the MMAPs 10 and the aerial image generation position or between the light source 20 and the MMAPs 10 causes distortion in the aerial image 40. For this reason, in this embodiment, an ideal camera image is rendered by CG as shown in step B of FIG.
Here, the ideal camera image refers to an image rendered from a portion of a space defined in a computer (hereinafter referred to as "CG space") that includes an ideal image and is within the angle of view of the CG camera. The CG camera is a camera that defines the origin and emission range of rays in the CG space in order to perform ray tracing in rendering.

Figures 4 and 5 are diagrams for explaining the rendering of the ideal camera image shown in step B of Figure 2. As shown in Figure 4, first, in CG space, an ideal image is placed at the position where the aerial image 40 shown in Figure 3 is to be displayed. This ideal image is the image that is to be displayed inside and outside the transparent object, and here, an image with a color gradation in which all pixels have different colors is generated in CG space as the ideal image.

The ideal camera image shown in FIG. 4 is an image after rendering of the CG space including the ideal image has been completed. In other words, the initial state of the ideal camera image is a state in which none of the pixels are colored. In other words, the entirety of the angle of view of the CG camera 50 is colorless. From this state, an image is created by rendering the portion of the CG space including the ideal image that is within the angle of view of the CG camera 50 using the method described later in FIG. 5, resulting in the ideal camera image shown in FIG. 4.

Therefore, in addition to the ideal image shown in FIG. 4, the ideal camera image also includes an outer frame portion outside the ideal image. In the rest of this specification, images captured using a real camera or rendered portions of the CG space within the angle of view of the CG camera 50 are referred to as "images," and those displayed as light sources and those generated by processes other than rendering are referred to as "images."

<Procedure for generating ideal camera images>
Fig. 5 is a flow chart showing the procedure for generating the ideal camera image explained in Fig. 4. First, as shown in Fig. 4, an ideal image is arranged at the aerial image display position in the CG space (step S10). This ideal image in the CG space is an image light source displayed on a plane.
Next, pixels to be processed are selected for rendering the portion within the angle of view of the CG camera 50 in the CG space including the ideal image arranged in step S10 (step S11).

As already explained, the part of the CG space that includes the ideal image that is within the angle of view of the CG camera 50 is rendered, but the angle of view of the CG camera 50 includes both the ideal image and the area outside of it. The selection of pixels to be processed in step S11 means selecting pixels one by one within the angle of view of the CG camera 50. Naturally, this selection also includes pixels that are in the area outside of the ideal image.

Next, a light beam is emitted from the CG camera 50 toward the pixel selected in step S11 (hereinafter referred to as the "selected pixel") (step S12). This process of step S12 is performed on all pixels within the angle of view of the CG camera 50 in the CG space that includes the ideal image.

Then, it is determined whether the light ray emitted from the CG camera 50 intersects with the ideal image (step S13). If the light ray emitted from the CG camera 50 intersects with the ideal image in step S13 (YES in step S13), the value of the pixel in the ideal image at the position where the light ray intersects, i.e., the pixel value, is calculated (step S14).

Furthermore, if the light ray emitted from the CG camera 50 does not intersect with the ideal image (NO in step S13), that is, if the point of intersection is a part of the outer frame other than the ideal image within the angle of view of the CG camera 50, the ideal image is not on the extension line of the light ray from the CG camera 50, so the pixel value is not calculated (step S15).
After the processes in steps S14 and S15 are completed, it is then determined whether or not the processes for all pixels of the CG camera 50 have been completed (step S16). If the processes for all pixels of the CG camera 50 have not been completed in step S16 (NO in step S16), the processes in steps S11 to S16 are repeated until the processes for all pixels have been completed.

If processing of all pixels of the CG camera 50 has been completed in step S16 (YES in step S16), all pixels of the CG camera 50 including the pixel values of the ideal image calculated in step S14 are output as an ideal camera image (step S17). This ends the generation process of the ideal camera image.

<How to generate distortion correction function>
Fig. 6 is a flow chart showing the process of generating a distortion correction function, which is the process of step C in Fig. 2. First, the transparent object 30, the MMAPs 10, and the light source installation positions of the actual machine are set in the CG space based on step A in Fig. 2 (step S20). Next, the CG camera 50 is placed at the camera position of the aerial image optical system, and a processing pixel in the ideal camera image is selected (step S21). Then, a light ray having the pixel value of the selected pixel in the ideal camera image is emitted from the camera position (corresponding to the viewpoint position) (step S22).

Next, the next ray path when the light ray emitted from the CG camera 50 in step S22 intersects with the transparent object 30 is calculated (step S23). Note that although the transparent object 30 is used here, the light ray path may change when reflected by a reflective object (reflective surface) instead of the transparent object 30.

When the light ray that has passed through the transparent object 30 intersects with the MMAPs 10, the next light ray path is calculated (step S24). Note that the transparent object 30 may be between the MMAPs 10 and the light source position, in which case the processes of steps S23 and S24 are interchanged.

Next, when a light ray having the pixel value of a processed pixel in the ideal camera image intersects with the position where the light source is installed in the actual device, the pixel value at that time is linked to the position of intersection and recorded in a memory unit (not shown) (step S25). Then, it is determined whether the processing from step S21 to step S25 has been completed for all processed pixels in the ideal camera image (step S26).

In step S26, if the pixel values for all processed pixels in the ideal camera image have not been recorded (NO in step S26), steps S21 to S25 are repeated until the pixel values for all processed pixels in the ideal camera image have been recorded.

Then, in step S26, if the pixel values of all the processing pixels in the ideal camera image are recorded (YES in step S26), a calculation image having the stored pixel values is output (step S27). Finally, a distortion correction function is generated from the calculation image generated in step S27 (step S28).
The distortion correction function generated in step S28 is a function that indicates to which position each pixel in the ideal image captured in the ideal camera image has moved in the calculation image calculated in step S27, and represents a function that distorts the input image signal obtained from the calculation image. The process of generating the distortion correction function is completed by the processes of steps S20 to S28 described above.

As described above, in step C of Fig. 2 and Fig. 6, a distortion correction function that determines how the image signal is distorted from the calculation image is obtained. That is, the distortion correction function is a function that determines how the input image signal is distorted, and by applying this distortion correction function to the image signal, a distortion-corrected image display light source 104 (see Fig. 1) for displaying the aerial image 40 is obtained.
Therefore, the distortion correction function can be said to be a function that indicates the correspondence relationship of the positions to be moved in order to convert the pixel values of the input image signal into the distortion-corrected image display light source 104 .

As mentioned above, the distortion correction function is found from the correspondence between each pixel of the ideal camera image and each pixel of the calculation image. By recording not only the positional relationship of each pixel as the correspondence but also the change in pixel value of each corresponding pixel, the distortion correction function can correct the color change of the image caused by a transparent object or a reflective object. Therefore, in this specification, "distortion" includes not only a change in shape (deformation) but also a change in color, and by using the distortion correction function of the present invention, it is possible to not only correct the deformation of the aerial image, but also to correct the color of the aerial image.

Next, with reference to FIG. 7, the procedure for generating an image for calculation from the ideal camera image described in the flowchart of FIG. 6 will be described starting from the change in the optical path along the optical system.
7, first consider an optical system in which MMAPs 10 and a transparent object 30 are interposed between a light source position _P4 and a viewpoint position _P0 , and place a CG camera 50 at the viewpoint position _P0 . Therefore, the viewpoint position _P0 becomes the CG camera position _P0 .

A light ray having a pixel value c of a specific point in the ideal camera image is emitted from the CG camera 50 from the CG camera position _P0 toward the emission direction _v0 . This light ray is incident on the origin _P1 of the transparent object 30, where it is refracted and its optical path is changed, becoming a light ray emitted toward the emission direction _v1 . In addition, when this light ray is emitted from point _P2 of the transparent object 30, its optical path is changed, becoming a light ray emitted toward the emission direction _v2 , and entering point _P3 of the MMAPs 10.

Then, at point _P3 of the MMAPs 10, the light is emitted in the changed emission direction _v3 , and the pixel values of the light are recorded when the light reaches the light source position _P4 . The set of pixel values recorded at the position where the _light reaches the light source position P4 becomes the image for calculation. Note that the pixel value of the light that reaches the light source position _P4 corresponds to the pixel value c of a specific point of the ideal camera image emitted from the CG camera 50.
By repeating the above process for all pixels of the ideal camera image, a calculation image with distortion according to the optical system of transparent objects 30, MMAPs 10, etc. placed along the route is formed at light source position _P4 .

A distortion correction function is generated from the calculation image obtained in Fig. 7, and this distortion correction function is used to generate a light source image for displaying an ideal image without distortion in the space inside the transparent object 30 or through the transparent object 30. Then, by placing this generated light source image having distortion at light source position _P4 , an aerial image without distortion is generated at the position of aerial image 40 in Figs. 3(a) and (b).

Figure 8 is a diagram explaining process D, and shows a specific example in which a distorted light source image generated by the procedure shown in the flowchart of Figure 6 and the method described in the schematic diagram of Figure 7 is placed at the actual light source position in real space, and an undistorted aerial image 40 is displayed inside a spherical transparent object 30.
As shown in Fig. 8, an image of a goldfish was prepared as a light source image including distortion. This light source image corresponds to the distortion-corrected image display light source 104 shown in Fig. 1.

The distortion-corrected image display light source 104 having an image of a goldfish is bent 90 degrees by MMAPs 10, which is a distance L from its center, and an image is formed at the center of the transparent object 30, which is also a distance L from MMAPs 10. Note that the goldfish image in the transparent object 30 shown in Figure 8 is close to an actual goldfish image, but this is because the image used as the light source is distorted using the distortion correction function obtained in Figure 6 of this embodiment. It has been verified that if there is no distortion correction for the light source image using this distortion correction function, the image will differ from the actual goldfish image.

<Noise reduction by MMAPsBRDF>
Next, in order to improve the efficiency of light source image generation and reduce noise in the generated light source image, we will explain MMAPs BRDF (Bidirectional Reflection Distribution Function), which has only the functions of MMAPs that are necessary for forming an aerial image.

As shown in FIG. 9(a), MMAPs10 is composed of two layers of mirror arrays covered with glass, and the incident light is reflected once at each layer, forming an aerial image of the light from the light source at a position symmetrical to the plane of MMAPs10. However, depending on the angle of incidence of the light, some light is reflected only at one layer, resulting in stray light that creates an image that is disturbing to the observer, and some light is transmitted without being reflected even once.

In addition to the two-layer mirror array, reflection also occurs due to the glass, so not all light forms an aerial image. In fact, when light from a light source forms an aerial image, the brightness is attenuated to less than half. Therefore, if MMAPs10 is used directly to generate an image for calculation, there is a 50% or higher probability that the light path it traces will generate stray light, transmitted light, or reflected light, resulting in poor efficiency in generating the image for calculation.

Therefore, in the aerial image generating device of this embodiment, as shown in Figure 9 (b), a BRDF (hereinafter referred to as "MMAPsBRDF") that has only the function of emitting incident light in a plane-symmetrical direction is used in CG space. MMAPsBRDF is a function that mathematically describes the retrotransmission of light rays in order to simulate it in CG space, and is realized by calculation processing in CG space, but does not actually exist physically. Note that while BRDF itself is a general term used in CG, the term MMAPsBRDF is a name coined by the inventors.

The MMAPsBRDF is, so to speak, a function that expresses only the property that incident light is emitted in a plane-symmetric direction. In other words, even without using a large number of mirror arrays like the MMAPs10, if there is a single plane to which the MMAPsBRDF is applied, that plane can emit light in a plane-symmetric direction and form an aerial image 40, just like the MMAPs10.

The principle and effects of MMAPsBRDF will be explained below.
9B, the MMAPsBRDF has only the function of emitting light incident in a direction i in a plane-symmetric direction o. Therefore, when using the MMAPsBRDF, it is necessary to calculate the emission vector o from the incidence vector i and the surface normal n.

If we define vector h shown in Figure 9(b) as the vector obtained by dropping a perpendicular line from the origin of incident vector i to the surface, then equation 1 can be established with simple vector calculations. Here, vector h is the dot product of vector i and unit vector n multiplied by unit vector n, and can be expressed by equation 2. Therefore, the desired exit vector o can be obtained by substituting equation 2 into equation 1, as in equation 3.

(Equation 1)
o = -i + 2h
(Equation 2)
h = (i n) n
(Equation 3)
o=-i+2(i・n)n

Figure 10 shows the difference between an image for calculation (a) generated using a conventional MMAPs10 and an image for calculation (b) generated by applying MMAPsBRDF to a planar object. Note that the light source image was generated using a CPU-calculated rendering program written by the inventors in C++.

As can be seen from Figures 10(a) and (b), the calculation image (b) generated using MMAPsBRDF has clearly less noise than the calculation image (a) generated by MMAPs10.

It was also found that the calculation time to generate the calculation image (b) using MMAPsBRDF was significantly reduced compared to the time to generate the calculation image (a) using MMAPs10. That is, when the number of samples per pixel was set to 100 and the ideal image resolution was set to 512 x 512 pixels, the calculation took 2552.8 seconds when MMAPs10 was used, whereas it took 5.1 seconds when MMAPsBRDF was used.

This is thought to be because MMAPs10 has a two-layer mirror array with 1,380 mirrors on each layer, resulting in a large number of objects in the CG space, whereas MMAPsBRDF can form an aerial image on a single plane.

<Implementation test>
In order to confirm the effects of the aerial image generating device 100 of this embodiment, the following experiments (1) to (4) were carried out.
(1) A case where an aerial image is directly displayed at the image display position and no transparent object is placed between the aerial image and the MMAPsBRDF (hereinafter referred to as the “desired image”)
(2) When the image of the light source of the aerial image is set to the same image as that used for the above-mentioned "desired image" and the image is displayed inside a transparent object (hereinafter referred to as "uncompensated image")
(3) When an image is displayed inside a transparent object using the InFloatation optical system described in Non-Patent Document 1 (hereinafter referred to as "compensated image (InFloatation)")
(4) A case where the image of the light source is set to the image of the light source generated by correcting the distortion, and the image is displayed inside a transparent object (hereinafter, referred to as "image with compensation (this embodiment)").

FIG. 11 is a diagram showing the results of capturing a desired image and an image displayed within a transparent object and measuring the image formation positions of the images.
For the images to be displayed, namely, "desired image,""uncompensatedimage," and "compensated image (InFloatation)," the light source image shown in FIG. 11(a) was used, and for the "compensated image (present embodiment)," the light source image shown in FIG. 11(b) was used.

In the light source image shown in FIG. 11(a), one side of the grid point to be displayed is 30 mm, but in the light source image for the "image with compensation (present embodiment)" in FIG. 11(b), the display surface of the light source image is made larger, a 50 mm square, so that the pixel values propagated by the light rays can be reliably recorded in the generation of the calculation image shown in step S27 of the flowchart in FIG. 6.

Figure 11(c) shows the "desired image", Figure 11(d) shows the "uncompensated image", Figure 11(e) shows the "compensated image (InFloatation)", and Figure 11(f) shows the "compensated image (present embodiment)". Thus, while the image presentation area at the light source in Figures 11(c) to (e) is 30 mm square, Figure 11(f) is 50 mm square.

As shown in Figures 11(a) to (f), the image displayed within the transparent object is tilted, but since the experiment was performed on a horizontal optical table and the light source, MMAPs, and transparent object were not tilted, it is believed that the tilt of the image is due to the thickness and bending of the MMAPs.

The positions of the images displayed in the images captured under each of the above conditions (1) to (4) were compared. In this comparison, the differences in the positions of the lattice points in the formed image were examined. That is, the positions of 49 lattice points in the formed image were recorded, with the rightward direction being the positive direction in the x direction and the downward (vertical) direction being the positive direction in the y direction. At this time, the captured image was a digital image with 4000 pixels vertically and 6000 pixels horizontally, and as shown in Figure 11 (c), the width of the lattice points in the desired image in the x direction was 1165 pixels.

After recording the positions of the 49 grid points, the differences in the coordinates of the corresponding grid points in each image were calculated for the following three combinations:
Case A: (1) desired image and (2) uncompensated image. Case B: (1) desired image and (3) compensated image (InFloatation).
Case C: (1) Desired image and (4) Compensated image (present embodiment)

Figure 12 shows the results. As can be seen from Figure 12, the difference in the number of pixels from the desired image was 276.1 pixels on average in case A, 155.4 pixels on average in case B, and 62.8 pixels on average in case C. This confirms that case C, which is the method of this embodiment, significantly improves compensation of image distortion within transparent objects compared to the other methods shown in cases A and B. However, considering that the width in the x direction of the grid points of the desired image in (1) is 1156 pixels, the width of one grid is about 191 pixels, so it must be said that even in this embodiment, the image deviates from the desired image by about one-third of one grid.

<Modification>
The aerial image generating device and the aerial image generating method of this embodiment have been described above by taking an example in which a transparent object 30 is placed between the viewpoint and the MMAPs 10, but the present invention is not limited to this embodiment. Fig. 13 shows an example in which the layout relationship of the optical system is different from that of the optical system shown in Fig. 3.
Fig. 13(a) shows an optical system in which light emitted from a light source LS is reflected by a reflecting object R and then enters the MMAPs, and Fig. 13(b) shows an optical system in which light emitted from a light source LS is incident on the MMAPs after passing through a transparent object T. Fig. 13(c) shows an optical system in which light from a light source LS is incident on the MMAPs, leaves the MMAPs, and is then reflected by a reflecting object R.

The optical systems shown in Figures 13(a) to (c) are merely examples of possible applications, and the present embodiment is not limited to these optical systems, and of course includes various applications and modifications within the scope of the claims. Optical systems that include both transparent objects and reflective objects in the optical path are also contemplated, as are optical systems that include multiple transparent objects and multiple reflective objects in the optical path.

100...aerial image generating device, 101...light source distortion correction function generating unit, 102...image processing unit (distortion processing unit), 103...distortion corrected image storage unit, 104...distortion corrected image display light source, 105...aerial image optical system, 10...retrotransmissive optical element (MMAPs), 20, LS...light source, 30, T...transparent object, 40, MI...aerial image, 50...CG camera, R...reflective object

Claims (6)

an image processing unit that generates a distortion-corrected image by applying distortion to an input image signal;
a distortion-corrected image storage unit that stores the distortion-corrected image generated by the image processing unit;
a distortion-corrected image display light source using the distortion-corrected image stored in the distortion-corrected image storage unit as a light source;
an aerial image optical system having at least one of a transparent object that transmits light and a reflective object that reflects light, and a retrotransmissive optical element, in an optical path between the distortion-corrected image display light source and an aerial image, the aerial image optical system receiving light emitted from the distortion-corrected image display light source and generating an aerial image as a real image at a predetermined spatial position;
a light source distortion correction function generating unit that generates a distortion correction function based on an image for calculation obtained by recording pixel values of the transparent object and/or the reflective object installed in the aerial image optical system and the light beam passing through the retrotransmissive optical element at a light source installation position of an actual device;
The image processing unit generates the distortion corrected image such that distortion of the aerial image provided by the aerial image optical system is corrected. In a CG space corresponding to the aerial image optical system,
An ideal camera image obtained by rendering the ideal image arranged at the aerial image display position within the angle of view of a CG camera corresponding to the viewpoint;
2. The aerial image generating device of claim 1, wherein each pixel within the angle of view of the CG camera is selected, and a light ray having a pixel value of the ideal camera image corresponding to the position of the pixel is emitted from the CG camera in the direction of the pixel, and then the image processing unit generates the distortion corrected image by applying distortion to the input image signal using the distortion correction function generated by the light source distortion correction function generation unit. The spatial location where the aerial image is displayed is a location inside or outside the transparent object.
3. An aerial image generating device according to claim 1 or 2. The retrotransmissive optical element used in generating the distortion correction function is a planar object having an MMAPsBRDF that mathematically describes the retrotransmission of light rays in order to simulate it in CG space.
4. The aerial image generating device according to claim 1. In CG space,
Designing an aerial image optical system including a positional relationship between a transparent object or a reflective object and a retrotransmitting optical element;
A step of outputting an ideal camera image by rendering the ideal image arranged at the aerial image display position within an angle of view of a CG camera corresponding to a viewpoint;
a step of emitting a light ray having a pixel value of each pixel of the ideal camera image from the CG camera, and recording the pixel values of all pixels of the ideal camera image when the light ray reaches a light source installation position of the actual device via the aerial image optical system, thereby generating a distortion correction function from an image for calculation. In CG space,
Designing an aerial image optical system including a positional relationship between a transparent object or a reflective object and a retrotransmitting optical element;
A step of outputting an ideal camera image by rendering the ideal image arranged at the aerial image display position within an angle of view of a CG camera corresponding to a viewpoint;
A step of generating a distortion correction function from an image for calculation generated by emitting a light ray having a pixel value of each pixel of the ideal camera image from the CG camera, and recording the pixel values of all pixels of the ideal camera image when the light ray reaches a light source installation position of the actual device via the aerial image optical system;
In the actual machine,
generating a distortion-corrected image by applying distortion to an input image signal using the distortion correction function, and generating a distortion-corrected image display light source based on the distortion-corrected image;
and placing the aerial image optical system designed in the designing step and the distortion-corrected image display light source in real space, receiving light emitted from the distortion-corrected image display light source, and generating an aerial image as a real image at a predetermined spatial position.

Images