JP7789622B2 - Image Processing System - Google Patents

Description

The present invention relates to VR (Virtual Reality)/AR (Augmented Reality) technology, and in particular to technology that is effective when applied to image processing systems that realize image/video-based VR/AR.

Systems are being considered and developed that use VR/AR technology to allow users to have simulated experiences that make them feel as if they are in another space or time. For example, technologies are being considered and developed that allow users to experience traveling to a distant place (another space) from the comfort of their own home, or to experience what a scene they are currently seeing while traveling looked like in the past (another time) in a realistic way.

As an example of technology related to the latter, U.S. Patent No. 10,127,730 (Patent Document 1) describes a mechanism that uses VR/AR technology to display content on a user's device that simulates an attraction that existed in the past at that location, based on information about the user's current location. This technology makes it possible to superimpose past footage of that location onto the scene currently being viewed.

As an example of a technology for overlaying other content on the current scene, Japanese Patent No. 6420605 (Patent Document 2) describes an image processing device that uses image recognition-based AR technology to track objects of any shape, enabling robust tracking with a small database size and processing load, even when there are large changes in viewpoint, such as shooting angle or distance.

U.S. Pat. No. 10,127,730 Patent No. 6420605

Using conventional technology such as that described in Patent Document 1, it is possible to realize a VR/AR system that, for example, displays a past scene from a location superimposed on an image of the current scene.

However, with a system like the technology described in Patent Document 1 that acquires content related to past scenes at a location based solely on location information, there are issues with the means of acquiring location information. For example, accurate location information may not be acquired indoors or underground, errors of several tens of centimeters to several meters may occur, or the scene may be completely different depending on the direction the user is facing even at the same location. This can make it difficult to select or determine an appropriate past scene that corresponds to the current scene.

Furthermore, when superimposing other images onto a video of the current scene in a VR/AR system, the objects in the superimposed image are often treated as three-dimensional models in order to follow the user's movements and changes in the direction they are facing, as in the conventional technology described in Patent Document 2. However, when using three-dimensional models, the objects are deformed using CG, resulting in a less realistic scene and a heavy data processing load.

The object of the present invention is to provide an image processing system that effectively and efficiently selects appropriate past scenes to overlay on images of current scenes using VR/AR, while reducing the load imposed by the overlay process.

The above and other objects and novel features of the present invention will become apparent from the description of this specification and the accompanying drawings.

A brief summary of the representative inventions disclosed in this application is as follows:

A representative embodiment of the present invention is an image processing system that uses an image processing device to overlay a past video of a corresponding past scene onto a current video of a current scene captured by a camera device and displays the overlay on a display device. The image processing device has a pre-processing unit that extracts one or more key frames from each of a plurality of past videos and records them as pre-processed past videos in association with the past videos from which they were extracted; an image search unit that identifies a first key frame similar to the current video from the pre-processed past videos by image search and identifies the first past video corresponding to the first key frame; and an image synthesis unit that converts the first past video onto the current video so as to minimize deviations between feature points in each video, and overlays the converted first past video onto the display device.

The effects achieved by representative inventions disclosed in this application can be briefly explained as follows:

In other words, according to a representative embodiment of the present invention, when a past scene is superimposed on an image of a current scene using VR/AR, it is possible to effectively and efficiently select an appropriate past scene to superimpose. It is also possible to reduce the load involved in the superimposition.

1 is a diagram illustrating an overview of an example of the configuration of an image processing system according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of image processing according to an embodiment of the present invention. FIG. 1 is a diagram showing an overview of an example of a data structure according to an embodiment of the present invention. 10 is a flowchart outlining an example of a preprocessing flow according to an embodiment of the present invention. 1 is a flowchart outlining an example of the flow of image search and synthesis processing according to an embodiment of the present invention.

Embodiments of the present invention will be described in detail below with reference to the drawings. In all drawings used to explain the embodiments, identical parts will generally be given the same reference numerals, and repeated explanations will be omitted. However, parts that have been described using reference numerals in one drawing may be referred to using the same reference numerals when explaining other drawings, although they will not be shown again.

<Overview>
Systems that use VR/AR technology to allow people to experience a virtual trip from the comfort of their own home have been explored and developed in response to the self-restraint on going out due to the spread of COVID-19. However, from the perspective of supporting local areas in the with-COVID/post-COVID era and revitalizing local areas through digitalization, it is necessary to aim for VR/AR systems that focus on promoting travel to local areas.

For example, as an experience before going on a trip, users could be able to view local scenery in real time and participate in events from the comfort of their own home, which could foster a desire to actually visit the location next time. Also, as an experience before a trip or while the user is actually traveling, for example, past scenes (videos) of the location that cannot be viewed directly in real time at the time could be superimposed on video of the current location, creating a sense of realism, which could foster a desire to return on another occasion.

The past scenes (videos) that can be overlaid here could be anything that existed or happened at some point in the past but is not shown in the current scene, such as scenes that no longer exist, such as abandoned railways or demolished buildings, events that were held at a specific time in the past, such as legendary live events or famous sports matches, or commentary by local reporters, guides or famous commentators. It could also be anything that still exists today but in a different state than it is now, such as the best scenery for a particular season, weather or time of day for sightseeing, such as rows of cherry blossoms in full bloom, a clear day, or a unique view that can only be seen at dusk or during high or low tide.

The image processing system, which is one embodiment of the present invention and will be described below, uses VR/AR to overlay (combine) past scenes from the same location onto a captured image of the current scene, allowing the user to have the experience described above.

<System Configuration>
1 is a diagram showing an overview of an example of the configuration of an image processing system according to one embodiment of the present invention. The image processing system 1 includes various devices, such as an image processing device 10 that performs image processing related to VR/AR, a photographing device 20 that photographs a scene in response to a user's instruction and acquires a moving image thereof, and a display device 30 that displays the moving image, etc., photographed by the photographing device 20 and presents it to the user. The photographing device 20 and the display device 30 are connected to the image processing device 10 by wired or wireless communication means or connection means (not shown).

The image capture device 20 can be configured, for example, as a digital video camera, webcam, action camera, dashcam, or other device with video capture capabilities. If the display device 30 (described below) also has a camera function, it may be configured as an integrated unit with the display device 30. The display device 30 can be configured, for example, as a device with a video image display function, such as a monitor or PC (Personal Computer) display. It may also be a device with a camera function that can function as the image capture device 20 in addition to a display function, such as a smartphone, tablet device, or VR/AR goggles.

The image processing device 10 is configured, for example, from a server device, a virtual server built on a cloud computing service, a PC, etc., and realizes various functions related to VR/AR image processing by using a CPU (Central Processing Unit) (not shown) to execute middleware such as an OS (Operating System), DBMS (Database Management System), and web server program, which are deployed to memory from a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), as well as software running on top of them.

This image processing device 10 has various units implemented as software, such as a pre-processing unit 11, an image search unit 12, and an image synthesis unit 13. It also has image data such as a past video set 14 and a pre-processed past video set 15, which are recorded and stored as a database, file, etc.

The pre-processing unit 11 has the function of performing pre-processing such as foreground/background separation and scene division, as described below, on past videos of past scenes captured in various locations and recorded in the past video set 14 in advance in order to provide them for VR/AR processing, and recording the generated image data as the pre-processed past video set 15. The image search unit 12 has the function of performing an image search to obtain similar past videos from the pre-processed past video set 15 based on video images of the current scene captured by the imaging device 20. The image synthesis unit 13 has the function of overlaying (synthesizing) the past video obtained by the image search unit 12 onto the video images of the current scene captured by the imaging device 20 and displaying them on the display device 30.

All or part of the image search unit 12 and image synthesis unit 13 may be implemented on a display device 30 configured as a smartphone, VR/AR goggles, or the like, rather than on the image processing device 10. The details of preprocessing, the composition of the preprocessed past video set 15, and the details of searching for past videos and overlaying (synthesizing) them with the current video image will be described later.

<Image processing example>
2 is a diagram outlining an example of image processing in one embodiment of the present invention. The diagram in the upper left of Fig. 2 shows an example of a video (hereinafter sometimes referred to as "current video") of a current scene at a certain location captured by the camera device 20, and the diagram in the upper right of Fig. 2 shows an example of a video (hereinafter sometimes referred to as "past video") of a past scene at the same location (for convenience, both are shown as one of consecutive still images (frames) in the video). The past video shows the same "castle" as the current video, but does not show the "cherry blossom branch" that appears in the current video, and instead shows "two horses" that existed in the past but are not in the current video.

In this embodiment, as shown in the lower left diagram of Figure 2, the previous video is transformed onto the current video, overlaid, and composited before playback. By using two-dimensional moving images captured live rather than CG as the previous video to be overlaid, the processing load for selecting and overlaying the previous video to be overlaid is reduced, processing is sped up, and a more realistic image can be produced. Note that when overlaying, for example, corresponding feature points (e.g., objects in the image (such as the vertices of the "castle" in the example of Figure 2)) are extracted for the current video and the previous video to be overlaid, and a two-dimensional transformation matrix (affine transformation matrix) is calculated to match corresponding feature points by rotating and moving the image. This transformation matrix is applied to the previous video to transform (deform) the previous video, and the corresponding feature points are overlaid and composited onto the current video. Note that the calculation and operation of the affine transformation matrix can be performed using known methods (for example, libraries such as OpenCV exist) in which the source and destination are represented as matrices and solved using the least squares method.

Rather than overlaying the entire past video onto the current video, it is also possible to cut out and composite only the foreground objects from the past video, as shown in the diagram at the bottom right of Figure 2. This diagram shows an example of a situation where only the "two horses" portion from the past video is cut out and composited. It is also possible to switch between a method of overlaying the entire past video and a method of overlaying only the foreground objects from the past video. For example, the method may be switched at the user's command, or may be switched depending on the type of display device 30 used by the user (e.g., a smartphone or VR/AR goggles).

<Data structure>
3 is a diagram outlining an example of a data structure in one embodiment of the present invention. In this embodiment, when superimposing a video of a past scene (past video 14a) onto a video of a current scene (current video 21), it is necessary to identify the past video 14a that captured the same subject as the current video 21. In this case, the past video 14a is identified by performing a similar image search for the current video 21. This increases the number of cases where the past video 14a to be superimposed can be identified even when sufficient location information about the location where the current video 21 was captured or the location of the subject cannot be obtained.

In this embodiment, when performing a similar image search, the search target is not the past video 14a itself, but rather key frames 15c, which are one or more characteristic still images (frames) that represent the past video 14a. This enables efficient similar image searches. The key frames 15c are extracted from the past video 14a by the pre-processing unit 11, and are recorded and stored in association with the past video 14a as one piece of data in the pre-processed past video group 15.

To extract keyframe 15c, the preprocessing unit 11 first separates foreground objects from the background in the past video 14a, obtaining a foreground video 15b from which only the foreground objects have been extracted, and a background video 15a from which only the background has been extracted. These are also recorded and stored in association with the past video 14a as one piece of data in the preprocessed past video set 15. Since the foreground object in the past video 14a is typically an object that existed in the past but does not appear in the current video 21, separating it and then performing a similar image search with the current video 21 can more efficiently and effectively identify the past video 14a.

Then, for the background video 15a from which the foreground object has been separated, characteristic frames (frames to be overlaid) are extracted as key frames 15c. In this embodiment, the background video 15a is divided at the point where there is a major scene change, and the first frame of each scene is extracted as key frame 15c, but this is not limited to this. A frame at the end or in the middle of a scene may also be used as key frame 15c, as long as it can be treated as a characteristic frame representing each scene.

Scene division is determined, for example, when the change in the six degrees of freedom (6DoF, consisting of three translational degrees of freedom for movement along the axes of a three-dimensional Cartesian coordinate system and three rotational degrees of freedom around the axes) between frame images exceeds a predetermined threshold, but is not limited to this.

In a similar image search using the current video 21, a search is performed for key frames 15c extracted from multiple past videos 14a recorded in the past video set 14, and the past video 14a corresponding to the matching key frame 15c is identified. The identified past video 14a or the foreground video 15b separated and extracted from the past video 14a is then transformed using the above-mentioned two-dimensional affine transformation matrix and superimposed on the current video 21 for synthesis.

<Processing flow>
4 is a flowchart outlining an example of the flow of preprocessing according to an embodiment of the present invention. In this preprocessing, the preprocessing unit 11 extracts key frames 15c for similar image search from each past video 14a stored in advance in the past video set 14, and records the extracted key frames 15c as one data item in the preprocessed past video set 15.

First, the pre-processing unit 11 separates the foreground and background of the past video 14a to be processed (S01). That is, a process (Image Matting) is performed in which the foreground object (object) is cut out from the past video 14a and extracted as the foreground video 15b, and this is then separated from the past video 14a to obtain the background video 15a. This process can be implemented using a number of existing technologies, methods, libraries, etc. that are in practical use, such as AI image processing technologies such as CNN (Convolutional Neural Network). The separated background video 15a and foreground video 15b are associated with the past video 14a and recorded as the pre-processed past video group 15. Note that subsequent processing is performed on the separated background video 15a.

Once the background video 15a is acquired, the type of video is then determined (S02). Six-degree-of-freedom information for each frame is acquired according to the type of video, and the amount of change between frames is obtained. If the background video 15a is a video with six-degree-of-freedom information, the amount of change between frames is simply acquired (S05). If the background video 15a is a stereo video captured with a stereo camera, the six degrees of freedom are estimated using, for example, a known triangulation technique (S03), and the amount of change between frames is acquired (S05). Furthermore, if the background video 15a is a monocular video captured with a monocular camera, the six degrees of freedom are estimated by tracking feature points within the video using, for example, a known V-SLAM (Visual Simultaneous Localization and Mapping) technique (S04), and the amount of change between frames is acquired (S05).

Then, when the amount of change in the six degrees of freedom between frames acquired in step S05 exceeds a predetermined threshold, the scene is divided between those frames (S06). The amount of change in the six degrees of freedom may be a simple sum of the amounts of change in each of the six degrees of freedom, or a weighted sum of the amounts of change in one or more specific degrees of freedom. For example, of the six degrees of freedom (moving forward/backward, left/right, up/down, tilting forward/backward (pitch), rotating the head left/right (yaw), tilting the head left/right (roll)) that typically characterize the behavior of a user viewing VR/AR images, yaw is often the most central movement, so a large weight may be assigned to the amount of change in yaw. The degrees of freedom to be weighted and the weighting value may be changed depending on the content of the video. Instead of the sum of the amounts of change in the six degrees of freedom, a scene may be divided when the amount of change in one or more specific degrees of freedom exceeds a predetermined threshold.

As a simpler method, instead of using the amount of change in six degrees of freedom between frames, it is possible to obtain, for example, a simple amount of change (difference) between the image data between frames, and when this amount of change exceeds a predetermined threshold, divide the scene between those frames.

Once the background video 15a has been divided into scenes, the first frame of each scene is extracted as a key frame 15c (S07). Data for a key frame group consisting of one or more extracted key frames 15c is then associated with information on the previous video 14a to be processed from which it was extracted, and recorded as a preprocessed previous video group 15 (S08). The above series of processes are performed on all previous videos 14a to be processed, completing the preprocessing.

Figure 5 is a flowchart outlining an example of the flow of image search and synthesis processing in one embodiment of the present invention. In this image search and synthesis processing, the image search unit 12 and image synthesis unit 13 superimpose and synthesize the corresponding past video 14 onto a video image of the current scene captured by the imaging device 20, and display the resulting composite image on the display device 30.

First, the image search unit 12 acquires local location information (S11). Here, the local location refers to the location of the camera device 20 currently capturing the video 21, or the location of the subject currently being captured in the video 21, or any location nearby. If the camera device 20 is an information processing terminal such as a smartphone equipped with a camera function and a GPS (Global Positioning System) function, the camera device 20 acquires location information as latitude and longitude information using the GPS function and transmits this to the image processing device 10 via communication means (not shown), which is then acquired by the image search unit 12. The location information may be acquired from an information processing terminal equipped with a GPS function separate from the camera device 20 located at the local location, or the photographer or user may input the name of a local facility or landmark and acquire the corresponding location information via the Internet or the like.

Next, the image search unit 12 acquires one or more candidate past videos 14a from the preprocessed past video set 15 based on the acquired local location information (S12). For example, past videos 14a shot within a predetermined distance from the local location information are acquired as candidates. Then, key frames 15c associated with each acquired past video 14a are acquired from the preprocessed past video set 15 (S13). This narrows down the key frames 15c to be used in the similar image search described below. To enable this narrowing down, each past video 14a is assumed to be recorded in association with location information about the shooting location or subject. Note that if there are no candidate past videos 14a corresponding to the current location information, the process is terminated and the image of the current video 21 shot by the shooting device 20 is displayed as is on the display device 30.

Once the narrowed-down key frames 15c are obtained, key frames 15c similar to the current video 21 being captured by the imaging device 20 are identified from among them by a similar image search (S14), and the past video 14a corresponding to the identified key frame 15c is selected from the candidates acquired in step S12 (S15). Note that the method of similar image search is not particularly limited, and known libraries using AI technology, etc., can be used as appropriate.

Once the past video 14a is selected, the image synthesis unit 13 then extracts corresponding feature points between the key frame 15c identified in step S14 and the image of the current video 21, and calculates a two-dimensional transformation matrix (affine transformation matrix) for image synthesis that matches each corresponding feature point by rotating and translating them (S16). The past video 14a selected in step S15 is then transformed (deformed) by applying the affine transformation matrix calculated in step S16, and this is superimposed on the video of the current video 21, synthesized, and played (S17). The image played with the past video 14a superimposed on the moving image of the current video 21 is displayed on the display device 30.

When playing back the past video 14a, the entire past video 14a may be played back from the beginning, or the past video 14a may be played back from a point corresponding to the key frame 15c identified in step S14. The point corresponding to the key frame 15c may be, for example, the beginning of a scene that includes the key frame 15c, or the key frame 15c itself.

Then, in step S17, it is determined whether the amount of deviation (the deviation of each feature point after applying the affine transformation matrix) when the current video 21 and the past video 14a are superimposed and combined exceeds a predetermined threshold (S18). The amount of deviation compared to the threshold may be the sum of the values at each feature point, or it may be the deviation for any one feature point. It is also possible to make a determination using both of these. If the amount of deviation is below the threshold (No in step S18), it means that the contents of the current video 21 and the past video 14a still match, so the process returns to step S17 and continues playback by combining the past video 14a with the current video 21. On the other hand, if the amount of deviation exceeds the threshold (Yes in step S18), it means that the current video 21 and the past video 14a no longer match, so the process of combining the past video 14a is terminated.

Then, the current video 21 is displayed as is on the display device 30, and the series of processes from step S11 is repeated to reselect the past video 14a that corresponds to the current video 21. Through the series of processes described above, it is possible to continue displaying the current video 21 while appropriately selecting and superimposing an appropriate past video 14a for display.

As described above, image processing system 1, which is one embodiment of the present invention, uses VR/AR to overlay (combine) past scenes from a location onto a captured image of the current scene, and displays the images. This system effectively and efficiently selects appropriate past scenes to overlay, while also reducing the load imposed by the overlay process.

That is, in one embodiment of the present invention, the image processing system 1 uses two-dimensional past videos 14a, not CG images but live-action images, as past scenes to be overlaid on the current video 21. Key frames 15c are extracted and associated with each past video 14a in advance based on the amount of change in three-dimensional position and direction (six degrees of freedom). Then, candidate past videos 14a are narrowed down from the past video set 14 using position information related to the current video 21. Key frames 15c related to each narrowed-down past video 14a are then extracted that are similar to the current video 21 using a similar image search between two-dimensional images, and the corresponding past video 14a is identified. A two-dimensional affine transformation matrix is then generated so that the extracted key frames 15c and the feature points of the current video 21 match (i.e., minimize the deviation), and the identified past video 14a is transformed (deformed) using this transformation matrix and overlaid on the current video 21.

These techniques reduce the processing load of selecting and overlaying the past video 14a to be overlaid on the current video 21, speeding up processing and producing realistic VR/AR images.

The invention made by the inventor has been specifically described above based on the embodiments, but it goes without saying that the present invention is not limited to the above embodiments and can be modified in various ways without departing from the spirit of the invention. Furthermore, the above embodiments have been described in detail to clearly explain the invention, and the invention is not necessarily limited to those that include all of the described configurations. Furthermore, it is possible to add, delete, or replace some of the configurations of the above embodiments with other configurations.

Furthermore, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole in hardware, for example by designing them as integrated circuits. Furthermore, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing programs that realize each function. Information such as programs, tables, and files that realize each function can be stored in a storage device such as memory, hard disk, or SSD, or on a storage medium such as an IC card, SD card, or DVD.

In addition, in the above diagrams, the control lines and information lines shown are those considered necessary for explanation, and do not necessarily represent all control lines and information lines in the actual implementation. In reality, it is safe to assume that almost all components are interconnected.

This invention can be used in image processing systems that realize image/video-based VR/AR.

1...Image processing system,
10... image processing device, 11... pre-processing unit, 12... image search unit, 13... image synthesis unit, 14... past video group, 14a... past video, 15... pre-processed past video group, 15a... background video, 15b... foreground video, 15c... key frame,
20...camera, 21...current video,
30...Display device

Claims (4)

An image processing system in which a past video in which a corresponding past scene is shot is superimposed on a current video relating to a current scene shot by a shooting device and displayed on a display device by an image processing device,
The image processing device includes:
a preprocessing unit that extracts one or more keyframes from each of a plurality of past videos, associates the extracted keyframes with the past videos, and records the preprocessed past videos;
an image search unit that identifies a first key frame similar to the current video from the preprocessed past video by image search, and identifies a first past video corresponding to the first key frame;
an image synthesis unit that converts and superimposes the first past video onto the current video so as to minimize deviations between feature points in the respective video images , plays back the first past video from a position corresponding to the first key frame, and displays the first past video on the display device;
An image processing system comprising: 2. The image processing system according to claim 1 ,
The pre-processing unit acquires values of six degrees of freedom, consisting of three degrees of freedom of translation and three degrees of freedom of rotation, for each frame image of each of the past videos, and when the amount of change in the values of the six degrees of freedom between frame images exceeds a predetermined threshold, divides the scenes in each of the past videos between the frame images and extracts frame images representing each scene as the key frames . 3. The image processing system according to claim 2 ,
The pre- processing unit calculates the amount of change in the six degrees of freedom between each frame image of each of the past videos by weighting the amount of change in yaw in the three rotational degrees of freedom among the six degrees of freedom more heavily than the other degrees of freedom. 2. The image processing system according to claim 1,
The image search unit identifies the first key frame by image search from among the multiple key frames recorded as the pre-processed past video, narrowing down the key frames based on location information related to the current video.