JP4179251B2 - Intermediate image creation device - Google Patents

Description

  The present invention relates to an apparatus for photographing a subject simultaneously from various directions using an imaging apparatus. In particular, the present invention relates to a method and apparatus for increasing time resolution by shifting an imaging timing of adjacent imaging apparatuses to create an intermediate image.

  In moving body imaging, the time resolution, that is, the number of frames that can be captured per second, depends greatly on the performance of the imaging apparatus used. As a method of increasing temporal resolution in a pseudo manner, when two images before and after are used as material images, a plurality of feature points are selected for objects existing in the material image, and feature points between the material images are selected. A method of creating an intermediate image by performing association and interpolation is common. When all feature points existing in both images can be associated, no problem occurs in the creation of the intermediate image. However, the part that is displayed at one moment is hidden when the angle of the object changes or another object comes forward at the next moment, or the part that is hidden when occlusion occurs However, there is a problem that feature points cannot be associated with each other, and an error occurs in the created intermediate image.

  As a method for creating an intermediate image, Japanese Patent Application Laid-Open No. 2004-23271 (Patent Document 1) discloses a method for creating an intermediate image from two images by dividing the subject in a key frame image into layers according to the depth. There is a method of creating a high-precision intermediate image by creating an intermediate image by assigning corresponding points for each layer and synthesizing the intermediate images for each layer.

JP 2004-23271 A

  In Patent Document 1, since an intermediate image can be created according to the depth, an intermediate image can be created without distorting the image when there is a movement such as the movements of subjects in different layers. It is possible to avoid difficulty in matching feature points caused by occlusion. However, since it is impossible to restore the portion hidden by occlusion, it is possible to create an intermediate image that does not collapse the subject by being able to associate feature points. It cannot be supplemented, and an intermediate image in which an incomplete part remains is created.

  Therefore, in increasing the time resolution by creating the intermediate image, methods for dealing with errors due to occlusion are limited, and incomplete portions often remain. For this reason, in order to generate a more accurate intermediate image, it is necessary to effectively compensate for a portion hidden by occlusion.

  The present invention has been devised in view of the above problems, and an object thereof is to improve time resolution by creating an accurate intermediate image.

  In order to solve the above problems, the following apparatus is proposed.

  A plurality of cameras installed so as to surround one subject, a synchronization signal generator connected to odd-numbered cameras, a synchronization signal generator connected to even-numbered cameras, and a server PC connected to the synchronization signal generator In accordance with the imaging timing command means for transmitting the synchronization timing to each synchronization signal generator at the server PC, and the imaging timing signal transmitted by each synchronization signal generator at the same time as receiving the synchronization timing A means for capturing a photographed image taken by a camera connected to the synchronization signal generator, a means for creating the same camera intermediate image based on the photographed image, and a single frame intermediate image based on the photographed image Means for extracting overlapping intermediate images from the generated same camera intermediate image and the generated same frame intermediate image And an error determining means for designating an error area of the extracted overlapping intermediate image by an input device, and an image synthesizing means for creating an intermediate image by synthesizing the low error area extracted by the error area designation. Providing the device.

  As an effect of the present invention, an intermediate image created from an image from the same camera and an intermediate image created from an image in the same frame are combined to create an intermediate image by combining portions with less error, thereby achieving a more accurate intermediate Images can be created.

  Embodiments of the present invention will be described in detail below with reference to the drawings.

  FIG. 1 is a diagram showing the configuration of the apparatus of this embodiment. A configuration with 10 cameras will be described as an example.

  In this apparatus, ten cameras (10, 20) are installed so as to surround the subject (1). Each camera uses a camera capable of external synchronization control having a function of releasing a shutter in accordance with the timing of the synchronization signal generated by the synchronization signal generator (30).

  A plurality of synchronization signal generators (30) are prepared in order to intentionally shift the synchronization timing of adjacent cameras and perform shooting. In the case of a configuration in which shooting is performed at two shooting timings, two synchronization signal generators are required. Numbering in order from any camera, odd-numbered cameras (10-a, 10-c, 10-e, 10-g, 10-i), even-numbered cameras (20-b, 20- d, 20-f, 20-h, 20-j), the odd-numbered camera is connected to the synchronization signal generator (30-a), and the even-numbered camera is connected to the synchronization signal generator (30-b). ).

  Each synchronization signal generator (30-a) (30-b) is connected to the server PC (40), and the server PC (40) is synchronized with each synchronization signal generator (30-a) (30-b). Control.

  The synchronization signal generator (30-a) and the synchronization signal generator (30-b) double the frame rate at the time of shooting by alternately outputting a synchronization signal.

FIG. 2 illustrates a material image and a created image position in the present embodiment.
The frame numbers of the images taken by the cameras (10-a, 10-c,...) Are assigned odd numbers 1, 3, 5, 7,. The frame numbers of the captured images of the cameras (20-b, 20-d...) Captured at different timings are numbered in the order of 2, 4, 6, 8,.

  At this time, the position where the material image (80) exists is as shown by a solid rectangle in FIG. The intermediate image (90) to be created is as shown by the grid rectangle between the material images (80).

  FIG. 3A is a diagram showing an internal configuration in the present embodiment.

  In order to generate a synchronization signal at an imaging timing at which images can be alternately acquired by adjacent cameras, first, from the imaging timing command unit (110), two synchronization signal generators (30-a) (30-b) ), A signal for starting shooting is transmitted at an intentionally shifted timing. The difference in timing is half the shooting timing of consecutive frames. In the odd-numbered camera (10) connected to the synchronization signal generator (30-a), shooting is performed in accordance with the synchronization signal transmitted from the synchronization signal generator (30-a), and the synchronization signal generator ( It is assumed that the even-numbered camera (20) connected to 30-b) performs shooting in accordance with the synchronization signal transmitted from the synchronization signal generator (30-b).

  The video shot by each camera is first stored as a shot image (210).

  In the same camera intermediate image creation unit (120) in order to increase the time resolution, the feature point of the subject in the image is specified in the frame before and after the time, and the feature point is interpolated in association with the same camera intermediate image (220 ). In the intermediate image creation, an existing intermediate image creation technique is used.

  In addition, in the same frame intermediate image creation unit (130) in order to increase the viewpoint resolution, in the image taken by the adjacent camera, by specifying the feature point of the subject in the image and interpolating the feature point in association with each other, The same frame intermediate image (230) is created.

  From the created same camera intermediate image (220) and the same frame intermediate image (230), the overlapping intermediate image extraction unit (140) extracts an image at the same position. For example, in an image taken with the camera (10-c), an intermediate image created from the images of the frame 1 and the frame 3 and an image taken with the camera (20-b) and the camera (20-d) The intermediate image created from the frame number 2 image is at the same position.

  The error determination unit (150) visually determines whether or not a defect has occurred in the contour portion of the subject in the two extracted overlapping intermediate images. In particular, due to occlusion in which a part of the subject is visible or hidden, the contour line of the subject in the created intermediate image becomes ambiguous or blurred.

  If two images are compared and there is no defect in either image, or if only one of the images is defective, the low error image selection unit (160) will not cause a defect. Select.

  When two images are compared and both of them have a defect, visually check which position of the intermediate image has the defect, and the error position extraction unit (170) uses a mouse or a keyboard. Specify by enclosing the defective part using the input device (70). The low error region selection unit (180) compares the two images and selects a portion with less defects, and the image composition unit (190) combines and combines the low error regions to generate an intermediate image with less defects. create.

  FIG. 3B is a diagram showing an internal configuration when there are three types of shooting timing patterns in the present embodiment.

  When there are three types of shooting timing patterns, three synchronization signal generators (30-a), (30-b), and (30-c) are required. The first camera connects to (30-a), the second camera connects to (30-b), the third camera connects to (30-c), and the fourth camera connects back to (30-a) To do. It is assumed that all cameras are connected in order. When the shooting timing of one synchronization signal generator is 1, the shooting timing of adjacent synchronization signal generators is shifted by 1/3 timing.

FIG. 4 is a diagram showing a system configuration of the server PC (40) of the present embodiment.
This system includes a CPU (50) that performs calculation and control based on a program, a main storage device (100) such as a memory that stores data and programs, and a large-capacity storage such as a hard disk. Device (200).

  In addition, a synchronization signal generator (30-a, 30-b) that controls the shooting timing of the camera, a plurality of cameras (10) connected to the synchronization signal generator (30-a), a synchronization signal generator (30- It consists of a plurality of cameras (20) connected to b), an input device (70) for operating a program and selecting an error region, and a BUS (60) for connecting them together.

  In the auxiliary storage device (200), a captured image (210) for storing an image captured by the camera (10, 20), an identical camera intermediate image (220) for storing an intermediate image created from the same camera video, the same Includes the same frame intermediate image (230) that stores the intermediate image created from the frame video, and the intermediate image (250) created by extracting the low error area from the same frame intermediate image (240) as the same camera intermediate image (220) It is.

  The main memory device (100) has a command to deliberately deviate the timing from the synchronization signal transmitter (30-a, 30-b) for controlling the shooting timing of the camera (10) and camera (20). The same camera intermediate image creation unit (120) that creates an intermediate image of the same camera based on the photographed image (210) taken by the camera (10) and the camera (20) The same frame intermediate image creating unit (130) for creating an intermediate image of the same frame, the overlapping intermediate image extracting unit (140) for extracting an overlapping intermediate image from the same frame intermediate image as the same camera intermediate image, intermediate image generation An error determination unit (150) that determines whether there is an error due to occlusion at the time, a low error image selection unit (160) that selects an image with little error as an intermediate image, and extracts an error position by specifying a range with an input device Error position extraction unit (170), comparing two images A low error region selection unit (180) that selects a portion with a small error and an image composition unit (190) that combines a region with a small error to create a low error image are included.

  FIG. 5A is a flowchart showing the flow in the present embodiment.

  First, in step 1000, a captured image is acquired.

  In step 1110, an intermediate image is created by using two images before and after the image taken by the same camera, and stored in the same camera intermediate image (220). The same camera intermediate image creation method will be described in detail with reference to FIG. This intermediate image creation method is also applied to the same frame intermediate image creation method in step 1120.

  In step 1120, an intermediate image is created using two images taken by adjacent cameras in the same frame and stored in the same frame intermediate image (230).

  In step 1200, whether or not an error due to occlusion has occurred in the same camera intermediate image (220) created in step 1110 is visually determined.

  In step 1300, it is determined whether there is an error in the same camera intermediate image. If there is no error, in step 1400, the corresponding same camera intermediate image is stored as an intermediate image (240).

  If an error occurs in the determination in step 1300, in step 1500, an image overlapping the same camera intermediate image in which an error has occurred is extracted from the same frame intermediate image (230) created in step 1120.

  In step 1600, it is determined whether or not there is an error in the same frame intermediate image. If there is no error, in step 1700, the corresponding same frame intermediate image is stored as an intermediate image (240).

  If there is an error in the determination in step 1600, that is, if the same camera intermediate image and the same frame intermediate image both contain errors due to occlusion, etc., an error region is specified in step 1800. . Using an input device (70) such as a mouse or a keyboard, an area is specified in two images including an error by a method such as a rectangle or a freehand.

  In step 1900, a portion having no error and a portion having a small error are extracted from the two images, and in step 2000, an intermediate image having a small error is created by synthesizing the extracted low error region. 240).

  FIG. 5B is a flowchart showing the flow of the intermediate image creation method in the present embodiment.

  In the same camera intermediate image in step 1110, when occlusion occurs, a method of creating a layer for each depth, which is an existing method, and creating an intermediate image for each layer may be used. Specifically, first, in step 1111, the depth of the subject in the image is determined, and in step 1112, the subject is separated into layers corresponding to the depth. Next, in step 1113, feature points are extracted for each layer. In step 1114, feature points are associated for each layer. In step 1115, an intermediate image is created for each layer. Finally, in step 1116, intermediate images are created by superimposing and integrating the created intermediate images for each layer in order from the deepest.

  FIG. 6 is a diagram showing a specific example in the present embodiment.

  Here, description will be given by taking as an example the creation of an intermediate image of frame number 1 and frame number 3 of the camera (10-c). The subject in the image of camera (10-c) frame number 1 is (10-c_1), and the subject in the image of camera (10-c) frame number 3 is (10-c_3).

  Also, the subject in the image of the camera (20-b) frame number 2 is (20-b_2), and the subject in the image of the camera (20-d) frame number 2 is (20-d_2).

  Given these material data, consider the creation of an image with frame number 2 in the camera (10-c).

  FIG. 7 shows an intermediate image created from images taken with the same camera. An intermediate image created from the material (10-c_1) (10-c_3) is defined as (10-c_2 [Camera]).

  FIG. 8 shows an intermediate image created from images taken by adjacent cameras in the same frame. An intermediate image created from the material (20-b_2) (20-d_2) is defined as (10-c_2 [Frame]).

  FIG. 9 is a diagram in which a low error intermediate image is created from the created intermediate image.

  Error part due to the effect of occlusion on the subject (10-c_2 [Frame]) in the same frame intermediate image (80), Error part due to the effect of occlusion on the subject (10-c_2 [Camera]) in the same image intermediate image (90) is designated using the input device, and a portion with a small error region is selected and synthesized to create a low error intermediate image, and an intermediate image (10-c_2) is created.

  As an application example of the present invention, the time resolution can be improved, for example, in checking a sports form or viewing a sporting event, and it is created by a shooting method for shooting a single subject from all directions using a large number of cameras. It can be applied to all images.

FIG. 1 is an apparatus configuration diagram of the photographing apparatus according to the first embodiment. FIG. 2 is a diagram showing a material image and a created image position in the present embodiment. FIG. 3A is a diagram showing a system configuration of the server PC 40 in the present embodiment. FIG. 3B is a diagram showing a system configuration of the server PC 40 when there are three types of shooting timing patterns in the present embodiment. FIG. 4 is a diagram showing an internal configuration in the present embodiment. FIG. 5A is a flowchart showing the processing flow in the present embodiment. FIG. 5B is a flowchart showing the flow of creating an intermediate image in the present embodiment. It is the figure which showed the specific example in this embodiment. FIG. 7 is a diagram showing creation of the same camera intermediate image in the present embodiment. FIG. 8 is a diagram showing the same frame intermediate image creation in the present embodiment. FIG. 9 is a diagram showing intermediate image creation in the present embodiment.

Explanation of symbols

1 ... subject, 10 ... camera, 20 ... camera, 30 ... sync signal generator, 40 ... server PC, 50 ... CPU, 60 ... bus, 70 ... Input device

Claims (2)

A plurality of cameras installed so as to surround one subject, a synchronization signal generator connected to odd-numbered cameras, a synchronization signal generator connected to even-numbered cameras, and a server PC connected to the synchronization signal generator In the intermediate image creation device configured by
Photographing timing command means for transmitting a synchronization timing to each synchronization signal generator in the server PC;
Means for capturing a photographed image photographed by a camera connected to the synchronization signal generator in response to a photographing timing signal transmitted by each synchronization signal generator simultaneously with receiving the synchronization timing;
Means for creating the same camera intermediate image based on the captured image;
Means for creating the same frame intermediate image based on the captured image;
Means for extracting an intermediate image overlapping from the created same camera intermediate image and the created same frame intermediate image;
Error determining means for designating an error region of the extracted overlapping intermediate image by an input device;
An intermediate image creating apparatus comprising image composition means for creating an intermediate image by synthesizing low error areas extracted by the error area designation. The intermediate image creation device according to claim 1,
The intermediate image creating apparatus characterized in that the difference in shooting timing between adjacent cameras is half the shooting timing of consecutive frames.

Images