JP5225313B2 - Image generating apparatus, image generating method, and program - Google Patents

Description

  The present invention relates to a technique for synthesizing multi-viewpoint images shot from a plurality of viewpoints, and generates an image with a wide field of view (a mosaic image such as a panoramic image) by connecting images shot by a plurality of cameras. The present invention relates to an image generation method and a program.

  Multi-viewpoint images taken from a plurality of viewpoints are combined to generate a panoramic image that captures a wide range of scenes, a high-resolution image, or the like, which is called mosaicing. A panoramic image obtained by moving images taken from a plurality of cameras is hereinafter referred to as a mosaic image.

  When there is a moving subject (moving object) in a shooting scene, it is difficult to naturally represent a moving object existing in an area that is being shot by a plurality of cameras on a composite image. As a method of dealing with the reflection of moving objects, a method for realizing the synthesis of moving images by arranging the cameras so that the projection centers of the images on each camera are substantially coincident, that is, the parallax between the cameras can be ignored. (For example, refer nonpatent literature 1) Or there exists the method of synthesize | combining an image without being influenced by a moving object (for example, refer nonpatent literature 2, 3).

  However, the former has a restriction that the camera must be arranged so that the parallax is completely eliminated. Moreover, the latter method is intended for still images, and when this method is applied to a moving image in which a moving object is reflected, it is difficult to synthesize a moving image that does not give a sense of incongruity. For example, if the result of region segmentation for composite image (still image) composition is applied to each frame of a moving image as it is and a plurality of still images are combined, and the combined result is viewed as a moving image, the moving object suddenly Unnatural movement that disappears from the screen or appears suddenly.

  Hereinafter, the details of a conventional method for synthesizing a mosaic image from images that cannot be ignored from a plurality of viewpoints, which is an elemental technology assumed in this case, will be described.

  In conventional mosaicing, the alignment of images taken from multiple viewpoints is projected onto a predetermined mosaic surface using the relationship between corresponding points between images and a projective transformation matrix obtained from known camera parameters. To do. This is performed under the assumption that subjects (landscapes, cars, people, etc.) shown in the image are on the same two-dimensional plane in the three-dimensional space.

  For example, when considering a subject sufficiently distant from the camera such as a landscape, the subject's own depth Δd is often small with respect to the distance d from the camera, and it can be approximated that the subject in the scene is at the same distance from the camera. . As a result, the subject is considered to exist on the same two-dimensional plane, and can be aligned by projecting onto the mosaic surface using the projective transformation matrix.

  However, when the distance between the camera and the subject is not sufficiently large and the depth of the subject cannot be ignored, blurring due to parallax (double image) occurs in the synthesized image. In many camera scenes, there are many subjects whose depth cannot be ignored, and in mosaicing, it is important to prevent such blurring. Even if the majority of subjects in a scene are on a two-dimensional plane, the composite image may be blurred due to an estimation error of the projective transformation matrix due to the influence of noise or the like.

  Therefore, in order to prevent double images, it is common to divide the overlapped area after alignment and use only one of the multi-viewpoint images for each divided area. . The flow of the region dividing method according to the prior art will be described below.

  1) One of a plurality of images is set as a reference image, and the remaining images are projected onto a mosaic surface where the reference image exists by a projective transformation matrix to perform alignment.

2) Perform region division on overlapping portions of images after alignment. As an example, FIG. 6 shows an area division of an overlapping portion between the reference image A on the mosaic plane and the image B _i (i = 1, 2,..., M) projected on the remaining mosaic plane. Here, M represents the number of viewpoints of the image to be synthesized. The overlapping portion, delimits so as not to split the moiety as blurring is conspicuous (part a person of interest), making the region A and the region B _i. The region where the blurring is conspicuous is a place where there is a subject (a building with many edges) having a fine texture and a subject (a person or a face) that is easily noticed by a person. On the contrary, the region where the blur is not conspicuous is an image portion in which a subject including a lot of low frequency components such as a sky with less texture, a grassland, a wall, or the like is shown.

  Specifically, after the image is converted into an edge image (luminance gradient information) by a sobel filter or a laplacian filter, the difference value of the edge image is evaluated as a portion of interest, or a person or face is It is detected as a process, and the detected area is evaluated as a portion of interest. In addition, when evaluating a difference in hue in consideration of a difference in characteristics between cameras, a difference value of pixel values between overlapping images is also evaluated. Based on this evaluation, in the conventional method, the boundary is determined so as to avoid a region that is noticed in the overlapping portion. A description of the graph and a conventional method of region division in the overlapping region using the graph (Non-Patent Documents 2 and 3) will be described later.

  3) After dividing the overlapping portion, the image after projective transformation is trimmed to the boundary, and the trimmed image is pasted on the mosaic surface to generate an image as shown in FIG.

(Division of overlapping areas)
An example of area division using a graph is shown below. First, an area that is noticed by a person in the overlapping area is examined. In Figure 8, and displays an image by superposing the images A and B _i in FIG. 6 in the overlapping region. The solid line, the image A, the dashed line represents the image _{B i.} As shown in this figure, when all the images in the overlapping area are displayed, a double image can be formed on a subject having a depth such as a person. Therefore, if a boundary is set on a subject that is noticed by a person, the cut subject is displayed in the combined image. The image is divided by obtaining a route that minimizes the cumulative total. The specific evaluation is performed for each pixel by the method shown in 2) above. This result is hereinafter referred to as a cost image.

  Next, a graph is created from the pixels of the cost image. The graph is composed of nodes (nodes) and edges (sides). In the example shown in FIG. 9, the nodes of the graph are associated with the pixels of the cost image, and the four connected directions that are adjacent directions of the pixels are defined as edges. The number on the edge is the cost of the edge. This edge cost is determined by the sum of evaluation values of adjacent pixels.

  Finally, the division of the graph such that the edge cost (edge cost) of the graph is minimized is obtained. FIG. 10 is a conceptual diagram illustrating an example of dividing a two-dimensional graph generated from a cost image with respect to an overlapping region. The division result of this graph corresponds to the division of the overlapping area. As a specific method, a Dijkstra method (for example, Non-Patent Document 4) for obtaining the shortest path, a graph cut method (for example, Non-Patent Document 5), or the like is used. Thereby, no division is performed on an edge near a pixel (node) having a large pixel value of the cost image, and a division is performed on an edge around a pixel having a small pixel value.

Kiyoshi Ikeda, Tomokazu Sato, Naokazu Yokoya, “Generation of high-resolution panoramic moving images by calibrating an omnidirectional multi-camera system,” Image Lab., 2005.2, pp.5-9, 2005 J. Davis, “Mosaics of scenes with moving objects,” Proc. Computer Vision and Pattern Recognition Conference, pp.354-360, 1998. Takeaki Iiyoshi, Wataru Mitsuhashi, “Automatic Removal of Partial Foreground for Image Mosaic Generation,” Vol.J92-D No.8 pp.1403-1413,2009 Dijkstra, E. W, “A note on two problems in connexion with graphs,” Numerische Mathematik 1, 269-271, 1959 Hiroshi Ishikawa, “Graph Cut,” Information Processing Society of Japan, 2007-CVIM-158- (26), No.31, pp.193-204,2007

  When the method of combining a plurality of still images taken from a plurality of viewpoints according to the prior art is applied to a moving image as it is, there is a problem that “flickering” and “shake” described below occur. Hereinafter, “flickering” and “swing” will be described.

  FIG. 11 is a conceptual diagram showing an example in which shaking and blurring occur in the area dividing method according to the conventional technique. When the conventional technology is applied to a moving image, when the synthesized image is viewed as a moving image, a phenomenon that a wide area of the image is shaken or flickering that a stationary object moves occurs. In the example shown in FIG. 11, since the position of the overlapping boundary is different between the frame f and the frame f + 1, the subjects of the frame f and the frame f + 1 are displayed by images from different cameras.

  In general, the images of the projected images on the mosaic surfaces of different cameras are deviated, and flickering or shaking occurs at this time when viewed as a moving image. That is, in the prior art, the generation of the boundary line in the time axis direction causes a problem.

  In other words, in the prior art, images are synthesized independently for each frame, so when the generated mosaic image is viewed as a moving image, the “still flickering” that appears to be moving the subject that is originally stationary, Since a phenomenon such as “shaking” occurs in a wide area as if the camera was shaken, the image quality deteriorates.

  In addition, in the conventional technique, since a background with many textures is not used as a boundary of division, an area may be divided so as to pass through a moving person or the like, so that an uncomfortable image is generated. A moving subject such as a person can be detected from tracking or a difference image using the preceding and following frames, and the edge cost near the subject can be increased to prevent the moving subject from being divided.

  However, since the related art does not consider the relevance of area division between frames, it is difficult to suppress the “flickering” and “shake” of the image as described above.

  The present invention has been made in consideration of such circumstances, and the object thereof is not to cut a moving object reflected in a moving image, and “flickering” or “swaying” accompanying synthesis. An object of the present invention is to provide an image generation apparatus, an image generation method, and a program capable of synthesizing moving images captured by a plurality of cameras having parallax while reducing the phenomenon.

  In order to solve the above-described problem, the present invention generates a mosaic image by synthesizing a plurality of images in which a part of an area overlaps a multi-viewpoint image captured by a plurality of imaging units as a moving image. An image generating apparatus, using a shooting parameter of each shooting means estimated in advance from the correspondence of feature points between a plurality of images, a moving image shot at the same time zone from two or more viewpoints, A projection multi-viewpoint image projected onto a predetermined reference plane is created, and a plurality of overlapping areas for the projection multi-viewpoint image are obtained. A cost image generating means for calculating a cost value for each pixel using a function for evaluating the degree of attention of the image, and for each overlapping area obtained by the converting means, each pixel is a node (node), a node while Create a graph in the spatial direction with the connected edges as edges. At that time, the cost of the node of interest and the adjacent node in the time direction and the adjacent node that is the same in the time direction and adjacent to the surrounding node. Using the value, the edge cost in the spatial direction is calculated so that the edge cost increases if the cost value between the node of interest and the adjacent node is high, and among the node of interest and a predetermined number of frames If the cost value between the target node and the adjacent node is high using the cost value of the adjacent node in the time direction with the pixel located at the same position as the target node in the spatial direction, the edge cost increases. The time / space direction graph generation means for calculating the edge cost in the time direction and the edge cost in the space direction calculated by the time / space direction graph generation means If the same imaging means as the preset imaging means is not allocated to the node of interest for each node in the overlap area obtained by the conversion means based on the edge cost in the time direction Based on an evaluation function that calculates the total cost of a cost function to be added and a cost function that causes an edge cost when assigning images of different imaging means to adjacent nodes in the spatio-temporal direction, based on the evaluation function By solving the optimization problem that minimizes the total cost, the image to be assigned to each node in each overlap area is determined, and the overlap area in the space / time direction is divided. An image identified by the assigned imaging means for each pixel in each overlapping area divided by the area dividing means and the time / space direction overlapping area dividing means. An image generation apparatus comprising: an image synthesis unit that assigns pixel values of an image and synthesizes the image.

  According to the present invention, in the above-described invention, the cost image generation unit is configured to determine a threshold value in which a change in image characteristics with respect to each overlapping region obtained by the conversion unit is determined in advance as a function for evaluating a degree of attention of a subject image. An evaluation function is used such that the cost of the above pixels becomes a high value.

  According to the present invention, in the above invention, the cost image generating unit is configured to determine a cost of a pixel whose luminance change between a predetermined number of consecutive frames is equal to or more than a predetermined threshold and a luminance change of pixels in a peripheral region. An evaluation function is used such that the cost of pixels equal to or higher than a predetermined threshold is high.

  According to the present invention, in the above-described invention, the cost image generating unit is configured such that the cost function increases as the difference between the brightness values of the projected multi-viewpoint images in the overlapping regions is larger as the evaluation function. It is characterized by using.

  According to the present invention, in the above invention, the image synthesizing unit further performs a blurring process using a color or luminance of a pixel in the vicinity of the boundary in the overlapping region, using a color or luminance of a pixel of an adjacent projected multi-viewpoint image By performing the above, a blending process for making the boundary between images inconspicuous is performed.

  In addition, in order to solve the above-described problem, the present invention combines a plurality of images overlapping a part of a region with respect to a multi-viewpoint image captured by a plurality of imaging units as a target for a mosaic image. An image generation method for generating a moving image shot at the same time zone from two or more viewpoints using shooting parameters of the respective shooting means estimated in advance from correspondences between feature points between a plurality of images. A projection step of creating a projected multi-viewpoint image projected onto a predetermined reference plane, obtaining a plurality of overlapping areas for the projected multi-viewpoint image, and the subject image is focused on each overlapping area A cost image generation step for calculating a cost value for each pixel by using a function for evaluating the degree of the image, and a node (node) for each pixel and an edge (side) obtained by connecting the nodes to each overlapping region To A graph of the inter-direction is created, and at that time, the target node and the surrounding nodes, that is, using the cost values of the adjacent nodes that are the same in the time direction and adjacent in the spatial direction, The edge cost in the spatial direction is calculated so that the edge cost increases if the cost value with the adjacent node is high, and the same position as the target node in the spatial direction among the target node and a predetermined number of frames. The edge cost in the time direction is calculated so that the edge cost is higher if the cost value between the node of interest and the adjacent node is higher, using the cost value of the adjacent node in the time direction and the adjacent node in Based on the spatio-temporal direction graph generation step, the edge cost in the spatial direction, and the edge cost in the temporal direction, A cost function that adds a cost when the same imaging means as the preset imaging means is not assigned to the node of interest, and an edge cost when assigning images of different imaging means to adjacent nodes in the space-time direction By solving an optimization problem that minimizes the total cost of the evaluation function based on the evaluation function that calculates the total cost with the cost function that causes And determining the image to be assigned, the time and space direction overlapping area dividing step for dividing the overlapping area in the time and space direction, and the assigned imaging means for each pixel of each of the divided overlapping areas And an image composition step of assigning pixel values of the image to be synthesized and synthesizing the images.

  According to the present invention, in the above invention, the cost image generation step includes a threshold value in which a change in image characteristics for each overlapping area obtained by the conversion unit is determined in advance as a function for evaluating a degree of attention of the subject image. An evaluation function is used such that the cost of the above pixels becomes a high value.

  According to the present invention, in the above invention, the cost image generation step includes a cost of a pixel whose luminance change between a predetermined number of consecutive frames is equal to or more than a predetermined threshold, and a luminance change of pixels in a peripheral region. An evaluation function is used such that the cost of pixels equal to or higher than a predetermined threshold is high.

  According to the present invention, in the above invention, the cost image generation step includes an evaluation function in which the cost value increases as the difference between the luminance values of the projected multi-viewpoint images in the overlapping regions is larger as the evaluation function. It is characterized by using.

  According to the present invention, in the above invention, the image composition step further includes blurring processing using the color or luminance of a pixel in an adjacent projection multi-viewpoint image with respect to the color or luminance of a pixel near the boundary in the overlapping region. By performing the above, a blending process for making the boundary between images inconspicuous is performed.

  Further, the present invention is a program for causing a computer to execute the above-described image generation method.

  According to the present invention, it is possible to generate a mosaic image in which flickering of an image and a wide range of image fluctuations are suppressed.

It is a block diagram which shows the structure of the mosaic image generation apparatus by embodiment of this invention. It is a flowchart for demonstrating the mosaicing process by this embodiment. It is a conceptual diagram for demonstrating the cause which "sway" by the prior art and "flicker" occur. It is a conceptual diagram which shows an example of the graph structure of the time and the space direction in this embodiment. It is a conceptual diagram which shows the reason which can suppress "sway" and "flicker" by this embodiment. It is a conceptual diagram which shows mosaicing which made the example of two images with the area | region division method by a prior art. It is a conceptual diagram which shows mosaicing which made the example of two images with the area | region division method by a prior art. It is a conceptual diagram which shows the example which expanded and displayed the blurring in the duplication area | region by the area | region division method by a prior art. It is a conceptual diagram which shows an example of the production | generation method of the cost image in the duplication area | region by the area | region division method by a prior art. It is a conceptual diagram which shows an example which divides | segments the two-dimensional graph produced | generated from the cost image about the duplication area | region by the area | region division method by a prior art. It is a conceptual diagram which shows an example which the shake and blurring by the area | region division method by a prior art produce.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
An object of the present invention is to generate a mosaic image of a moving image obtained by photographing a wide range of scenes, and to solve the problems of the prior art. For example, an object of the present invention is to generate a mosaic image from a moving image of a wide range of scenes acquired by a plurality of cameras installed in a concert hall or a stadium such as soccer.

  In the mosaic image generation method according to the above-described prior art, the overlapping area is divided and only one image is displayed in each area in order to prevent occurrence of blurring due to parallax (double image) in the overlapping area in the composite image. However, if this method is used for a moving image (video) as it is, the boundary position of the divided area changes for each frame, so that the subject that is originally stationary appears to be moving, “Wide range of shaking” occurs when the camera shakes.

  The present invention aims to reduce this problem, and does not divide the overlapping area for each frame, but incorporates a continuous dividing method in the time direction into the mosaic generating method. Perform region segmentation considering continuity in the time axis direction. As a specific method, the degree (cost) that a person is paying attention to the area is calculated for a plurality of frames, and the area that the person is paying attention is not divided in time and space. It is characterized by performing area division.

  FIG. 1 is a block diagram showing a configuration of a mosaic image generating apparatus according to an embodiment of the present invention. In FIG. 1, a subject photographing means 101 is a multi-viewpoint image acquisition system composed of a plurality of cameras. For example, a system that photographs a game on a soccer field or a stage performance in a concert hall with a plurality of cameras. It is. The subject photographing means 101 supplies the photographed video signal S1 to the camera image acquisition means 102.

  The mosaic image generation apparatus 100 includes a camera image acquisition unit 102, a mosaic image generation unit 103, an image storage unit 106, and a generated image output unit 104. The camera image acquisition unit 102 acquires the video signal S1 from the subject photographing unit 101 and supplies it to the image storage unit 106 as image data D1.

  The image storage unit 106 includes a camera image storage unit 106a and a composite image storage unit 106b. The camera image storage unit 106a stores the image data D1 from the camera image acquisition unit 102. The composite image storage unit 106b stores composite image data D6 output from the mosaic image generation unit 103 described later.

  The image storage unit 106 includes a camera image storage unit 106a and a composite image storage unit 106b, but may be the same storage medium. For example, it is a storage device such as a hard disk drive (HDD) or a solid state drive (SSD). By storing in the image storage unit 106 an image of a scene captured in advance by subject shooting by the camera of the subject shooting unit 101, the processing by the image composition unit 103e of the mosaic image generation unit 103 can be executed offline. It becomes possible.

  The mosaic image generation unit 103 takes out the image data D1 stored in the camera image storage unit 106a, and finally generates composite image (mosaic image) data D6. The mosaic image generation unit 103 includes a conversion unit 103a for converting to a reference projection plane (mosaic plane), a cost image generation unit 103b, a graph generation unit 103c in time / space direction, a graph division unit 103d in time / space direction, and an image composition. It comprises means 103e.

  The conversion means 103a for converting to the reference projection plane performs projective conversion on the image captured by each camera. More specifically, using the camera parameters of each camera preliminarily estimated from the correspondence between the feature points between a plurality of images, a moving image that has been shot at the same time zone that was shot from two or more viewpoints is stored in advance. Create an image (projection multi-viewpoint) projected on a plane that serves as a reference. Overlapping areas (N) are obtained for this projected multi-viewpoint image.

  The cost image generation means 100b generates a cost image for the image of F frames for the overlapping area of the projective transformed image. More specifically, for each overlapping region, an image of a subject in the image, such as a region with many edges such as a contour, a region with a lot of high frequency components such as a fine texture region, a moving subject, a human face, etc. The cost value is calculated for each pixel by using a function for evaluating the degree of attention of the pixel. A cost image is obtained by assigning this cost value to each pixel in each overlapping region. More specifically, the cost image generation unit 100b determines the cost of pixels having a change in image characteristics between a predetermined number of consecutive frames, for example, a pixel whose luminance change is equal to or greater than a predetermined threshold, and pixels in the surrounding area. The evaluation function is used so that the cost of a pixel having a change in image characteristics, for example, a luminance change equal to or higher than a predetermined threshold value is high. The evaluation function may be an evaluation function in which the cost value increases as the difference in data representing the image characteristics of the projected multi-viewpoint image in each overlapping region, for example, the difference in luminance value increases.

  The spatio-temporal graph generation means 103c creates a graph from images of F frames for the overlapping region. More specifically, for each overlapping region, a graph in the spatial direction is created in which each pixel is a node (node), and the connection between the nodes is an edge (side). At that time, the edge cost and the edge cost in the time direction are calculated. The edge cost in the spatial direction is obtained by calculating the cost value between the target node and the adjacent node by using the cost value of the target node and the adjacent node, that is, the adjacent node in the time direction and adjacent in the spatial direction. If it is higher, the edge cost is higher. In addition, the edge cost in the time direction uses the cost value of the target node and the adjacent node in the time direction that is a pixel at the same position as the target node in the spatial direction in a predetermined number of frames. Thus, if the cost value between the node of interest and the adjacent node is high, the edge cost is calculated to be high.

  The time / space direction graph dividing means 103d divides the overlapping region into regions for each label. More specifically, the overlapping area in the spatio-temporal direction is divided by solving a mathematical problem that determines which camera number (label) image is allocated to each node of the overlapping area. That is, when the same label as the preset label is not assigned to the node of interest, the cost function for adding the cost and the edge cost when assigning different camera images to nodes adjacent in the space-time direction Solve the optimization problem that minimizes the value of the evaluation function (that is, the total cost) based on the evaluation function that calculates the total cost of the cost function that causes The camera image to be assigned to is determined.

  The image synthesizing unit 103e connects the image of the overlapping area and the image of the non-overlapping area for each frame. More specifically, the pixel value of the image specified by the assigned camera number is assigned to each pixel in each overlapping region, and the image is synthesized. At this time, blending processing that makes the boundary between images inconspicuous by performing blurring processing using the color or luminance of the pixel of the adjacent projected multi-viewpoint image for the color or luminance of the pixel near the boundary in the overlapping region May be implemented.

  The generated image output means 104 reads the composite image data D6 stored in the composite image storage means 106b as output image data and outputs it to the generated image display means 105 as a video signal S2 for display display.

  The generated image display unit 105 is, for example, a display device such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), or a PDP (Plasma Display Panel) connected to the generated image output unit 104 such as a display terminal. The generated image display unit 105 displays a composite image in accordance with the video signal S2 from the generated image output unit 104. Note that the generated image display unit 105 may be, for example, a two-dimensional planar device or a curved display device surrounding the device user.

  Next, a mosaic image generation method performed by the mosaic image generation apparatus 100 according to the present embodiment will be described. FIG. 2 is a flowchart for explaining a mosaic image generating method by the mosaic image generating apparatus 100 of the present embodiment. The subject photographing means (hereinafter also referred to as a camera) 101 can be arranged in any way, but in the present embodiment, in order to ensure a certain common field of view (overlapping region), in a grid pattern or on a straight line, It is assumed that a plurality of cameras are arranged in parallel or in a radial pattern with a specific subject as a gazing point.

First, as a preprocessing, a projection transformation matrix from each camera to a reference camera is obtained (step S1). At this time, the camera number is m (= 1, 2,..., M), and the frame number of the image is f (= 1, 2,..., F). Next, projective conversion is performed on the image captured by each camera by the conversion means 103a to the reference projection plane (step S2). The images to be converted are all except the image of the reference camera. In the subsequent steps, the obtained image _{Im, f} after projective transformation is used. Here, m indicates the camera number in step S1, and f indicates the frame number in step S1. Next, for the image _{Im, f} after projective transformation, the overlapping region is set to R _i (i = 1, 2,..., N) (step S3). As shown in FIG. 11, when the number of cameras is two, the number of overlapping areas is 1 (N = 1). In the following steps, the same processing is performed from i = 1 to N.

Next, the overlap region R _i, to generate a cost image for frame F sheets of images (step S4). The cost image is an evaluation result of the degree of human attention, for example, an evaluation value that increases the cost of a moving car or a building that is stationary but has strong edge strength and has a fine texture. It becomes. Examples of cost image evaluation methods are shown below.

The moving subject was detected by the inter-frame difference. For a stationary object, a pixel having a large edge strength is examined by applying a Laplacian filter, and it is assumed that there is a stationary object on a pixel whose edge strength continues for a threshold value that is between several frames. Assuming that the K images I _{k, f} overlap in the overlapping region R _i in the frame f, the cost image ImgC ₁ (R _i ) can be set as in the following equation (1).

  However, α and β are adjustment constants determined experimentally, and x and y are variables indicating coordinates when coordinate axes are set in the horizontal direction and the vertical direction on the image.

M (R _i ) and S (R _i ) are functions that are processed for each pixel. Move is a function that outputs 1 for each pixel, and 0 for each pixel if there is a moving object image in the image I _{k, f} , and M (R _i ) is the result of K images by Move. It is a function that outputs a value added for each pixel. Stay evaluates 1 if the image I _{k, f} has a stationary object, 0 otherwise, for each pixel, and S (R _i ) adds the results of K images by Stay for each pixel. It is a function that outputs the value.

In order to consider the difference in characteristics between cameras and the influence of parallax, for example, a cost image ImgC ₂ represented by the following equation (2) can be set.

However, u and v are combinations of K images, and Z is a set of combinations of K to 2 images. D (R _i ) is a function that outputs a 2-norm for a vector w obtained by taking the difference for each R, G, B component for each pixel for the images I _{u, f} and I _{v, f} .

Next, the overlap region _{R i,} to create the graph from the F frames of the image (step S5). Here, FIG. 3 is a conceptual diagram for explaining the cause of the occurrence of “sway” and “blur” according to the prior art. As shown in FIG. 3, in the conventional method, the area is divided into frames f−1, f, and f + 1. Therefore, the boundary is set to the frame f due to the influence of the moving subject (vehicle B in the illustrated example). Different for each of −1, f, and f + 1. Therefore, in the present invention, a division method that considers the relevance in the time direction is performed.

Here, FIG. 4 is a conceptual diagram showing an example of a graph structure of time and space directions in the present embodiment. First, using a pixel of the cost image as a node, an edge (side) is created in the time axis direction in the same manner as in the spatial direction to create a three-dimensional graph (see FIG. 4). The edge cost of the graph is determined based on the cost image. For example, the spatial direction may be the sum or product value of the cost image pixel values (see (1)). For the time direction, the sum of the pixel values in the cost image ImgC ₁ or the product value multiplied by the constant term γ is used as the edge cost in the time direction (see (2)). However, when the image is large, it is possible not to create an edge for each adjacent pixel, but to reduce the calculation cost by using a graph using a reduced cost image.

  FIGS. 5A to 5C are conceptual diagrams showing the reason why “shake” and “blur” according to the present embodiment can be suppressed. FIG. 5 shows the effect of generating an edge in the time direction. In FIG. 5, it is assumed that the subject to which the person pays attention is the vehicle B and the sign C. FIG. 5A shows a pixel in which a target pixel (node) of a cost image is connected between frames. As shown in FIG. 5B, when the optimum region division is performed for each frame as in the prior art, it is a stationary object due to the surface created by connecting the division boundaries of the frames at frames f and f + 1. It can be seen that the edges of the sign C (broken lines) may intersect.

  In FIG. 5C, in the cross section: A-A, an edge between nodes with a moving subject (vehicle B) is indicated by a broken line, and an edge of a node with a stationary subject (sign C) is indicated by a one-dot broken line. The edges between the other nodes are indicated by solid lines. As can be seen from this figure, even in the time direction, if the division is performed so as not to cut an edge having a large edge cost, a double image when the moving image is made can be prevented. In other words, if a three-dimensional graph created by a plurality of frames is divided so as to minimize the edge cost, it is possible to prevent the region division from crossing the subject of interest in the time direction.

  In terms of parameters, it is possible to prevent the boundary from crossing the subject in the spatial direction by increasing the values of α and β. In the time direction, by increasing the value of γ, it is possible to suppress the subject from crossing in the time direction, so that there is an effect of suppressing the shaking and flickering as described above.

  However, if γ is too large, there is a possibility that the boundary is set so as to divide the subject in the spatial direction. In the example shown in FIGS. 5A and 5B, for example, when the moving vehicle B stops for a while in the center of the overlapping area, There is also a possibility that the boundary may continue to be set on a part of the sign C when, for example, a boundary is set on the mark C or when a part of the sign C is not detected as a stationary object. Therefore, these parameters are determined by experimental adjustment.

  Next, by solving the optimization problem that minimizes the evaluation function E, the overlapping region is divided into regions for each label (step S6). When a pixel (node) of a cost image in a certain overlapping area is p, a label defined for each node is L = {1, 2,..., M}. However, the label number coincides with the camera number and indicates which camera image is displayed on the pixel. At this time, a cost function E expressed by the following equation (3) is defined for the three-dimensional graph.

Here, q is an adjacent pixel in the spatial direction and time direction with respect to p, and L (x) is a label of the position of the node x. C _d is a cost function for a specific node pair, and C _i is a cost function for adjacent relations of nodes for all nodes.

There are various methods for calculating C _d and C _i , and an example thereof is shown in the above equation (3). _Cd is a cost if the pixel of a certain node p is the same as a preset label, and the cost is 0 otherwise, and a cost Kb is output as a function, and Kb is set experimentally by the user. Shall.

Therefore, if it is known in advance that a certain node is a stationary object, the user costs a very large value compared to the edge cost for the label of the camera number specific to that node and the label. By setting as, the pixel label can be designated by the user. In the present embodiment, the node of the overlap region in contact with non-overlapping area region is set to the camera number of the image areas not overlapping region C _d. For other nodes, C _d = 0.

C _i is a function that outputs a cost of 0 if p and the adjacent pixel q have the same label, and an edge cost otherwise.

  If the cost function is designed as described above, it can be solved as a label combination optimization problem that minimizes the cost function. Examples of a specific algorithm to be calculated include a graph cut method based on a maximum flow minimum cut theorem (Non-Patent Document 5), a reliability propagation method, and the like.

  Here, as a practical problem, when the number of frames of the target moving image is tens of thousands to hundreds of thousands, there is a possibility that processing cannot be performed collectively due to the limit of the memory of the computer. If one hour of moving images is handled, the number of frames is 30 (fps) * 60 (seconds) * 60 (minutes) = 108000 frames. As an example of dealing with this, there is a method of processing by dividing a moving image into several frames.

For example, when a 1000-frame moving image is divided into two, 1-500 frames are batch-processed by the above method, and 501-1000 frames are batch-processed by the above method. However, if the processing is simply divided, the continuity of time is not maintained between the two moving images, so that the image shakes or flickers. Therefore, the region division result label of the 500th frame is given as an initial value to the node of the 501st frame. In other words, the label specifies the portion of the C _d, by inputting a result of processing in the last, it is possible to perform the time-sequential process.

Next, for each frame f, a camera image _{Im, f} corresponding to the label L (f, p) of the pixel p is pasted, and an image of the overlapping region _Ri is synthesized (step S7). Here, it is also possible to perform a blending process so that the boundary between the pasted images is not noticeable. As a simple example, for the pixels within several pixels near the boundary line, the pixel values of the camera image in contact with the boundary line are mixed. When the images of the camera m and the camera m + 1 are adjacent to each other, the pixels that are several pixels from the boundary line are 2 in the middle of the pixel values of the images of the camera m and the camera m + 1 (R, G, and B components are added). (Divided value) will be displayed.

Then, all the overlap regions _{R i (i = 1, ...} , N) determines whether or not it is completed for (step S8), and all the overlap regions _{R i (i = 1, ...} , N) is finished Steps S4 to S7 described above are repeated.

Then, when all the overlapping regions R _i (i = 1,..., N) are finished, the overlapping region image and the non-overlapping region image are connected for each frame (step S9), and processing is performed for all overlapping regions. When is over, it ends.

  According to this embodiment, the continuity of a plurality of frames is taken into consideration, and a global optimum solution can be obtained as a moving image. On the other hand, in the prior art, since a method for obtaining a composite region of an image for each frame is used, when considered as a moving image, only a local optimum solution has been obtained.

  Also, in this embodiment, a method of detecting by a method suitable for each such as a moving object or a stationary object is incorporated in the framework, thereby causing image flickering or shaking that causes image quality degradation. It is possible to efficiently suppress the phenomenon. In the prior art, since the area is obtained for each frame, the boundary is only a line for dividing the two-dimensional graph. In the present embodiment, the two-dimensional surface for dividing the three-dimensional graph is used. There is a difference in seeking.

  As mentioned above, although the invention made | formed by this invention was concretely demonstrated based on embodiment, this invention is not limited to embodiment, Of course, it can change in the range which does not deviate from the summary. It is.

DESCRIPTION OF SYMBOLS 100 Mosaic image generation apparatus 101 Subject imaging means 102 Camera image acquisition means 103 Mosaic image generation means 103a Conversion means to reference projection plane 103b Cost image generation means 103c Time / space direction graph generation means 103d Time / space direction graph division means 103e Image composition means 104 Generated image output means 105 Generated image display means 106 Image storage means 106a Camera image storage means 106b Composite image storage means

Claims (11)

An image generation device that generates a mosaic image by combining a plurality of images with overlapping regions in a multi-viewpoint image captured by a plurality of imaging means for a moving image.
A plane serving as a predetermined reference for moving images shot at the same time zone from two or more viewpoints using the shooting parameters of the respective shooting means preliminarily estimated from the correspondence of feature points between a plurality of images. A projecting multi-viewpoint image projected onto the image, and converting means for obtaining a plurality of overlapping areas for the projected multi-viewpoint image;
Cost image generation means for calculating a cost value for each pixel using a function that evaluates the degree to which an image of a subject is noted for each overlapping region obtained by the conversion means;
For each overlapping area obtained by the converting means, a graph in the spatial direction is created with each pixel as a node (node) and a connection between the nodes as an edge (side). Using the cost value of the node of interest and the surrounding nodes, that is, the adjacent nodes that are the same in the time direction and adjacent in the spatial direction, the edge cost increases if the cost value of the node of interest and the adjacent node is high In addition, the edge cost in the spatial direction is calculated, and the node of interest and the adjacent node in the temporal direction adjacent to the pixel at the same position as the node of interest in the predetermined number of frames. Spatio-temporal direction graph that uses the cost value to calculate the edge cost in the time direction so that the edge cost increases if the cost value between the node of interest and the adjacent node is high And forming means,
Based on the edge cost in the spatial direction and the edge cost in the time direction calculated by the spatio-temporal direction graph generation unit, for each node in the overlap area obtained by the conversion unit, A cost function that adds a cost when the same imaging means as the set imaging means is not allocated, and a cost function that generates an edge cost when assigning images of different imaging means to adjacent nodes in the space-time direction And determining an image to be assigned to each node of each overlapping region by solving an optimization problem that minimizes the total cost of the evaluation function, using the evaluation function for calculating the total cost as Then, the time / space direction overlapping area dividing means for dividing the space / time direction overlapping area,
Image synthesizing means for synthesize the image by assigning the pixel value of the image specified by the assigned imaging means to each pixel of each overlapping area divided by the space-time direction overlapping area dividing means. An image generation apparatus characterized by the above. The cost image generating means includes
As a function for evaluating the degree of attention of the subject image, an evaluation function is used so that the change in image characteristics for each overlapping area obtained by the conversion means has a high cost value for pixels having a predetermined threshold value or more. The image generating apparatus according to claim 1. The cost image generating means includes
The cost of a pixel whose luminance change between a predetermined number of consecutive frames is equal to or higher than a predetermined threshold and the cost of a pixel whose luminance change in a peripheral region is higher than a predetermined threshold are high. The image generation apparatus according to claim 2, wherein a simple evaluation function is used. The cost image generating means includes
The image generation apparatus according to claim 3, wherein an evaluation function is used such that a cost value increases as a difference in luminance values of projected multi-viewpoint images increases in each overlapping region. . The image composition means includes
Furthermore, the blending process that makes the boundary between the images inconspicuous by performing blurring processing using the color or brightness of the pixel of the adjacent projected multi-viewpoint image with respect to the color or brightness of the pixel near the boundary in the overlapping region 5. The image generation apparatus according to claim 1, wherein: An image generation method for generating a mosaic image by synthesizing a plurality of images in which a part of a region overlaps for a multi-viewpoint image captured by a plurality of imaging units as a target of a moving image,
A plane serving as a predetermined reference for moving images shot at the same time zone from two or more viewpoints using the shooting parameters of the respective shooting means preliminarily estimated from the correspondence of feature points between a plurality of images. Creating a projected multi-viewpoint image projected onto the projection multi-viewpoint image, and obtaining a plurality of overlapping areas for the projected multi-viewpoint image;
A cost image generation step of calculating a cost value for each pixel using a function for evaluating a degree of attention of the subject image for each overlapping region;
For each overlapping area, create a graph in the spatial direction with each pixel as a node (node) and a connection between the nodes as an edge (side). At that time, the target node and the target node and its surroundings Edge in the spatial direction so that if the cost value between the node of interest and the adjacent node is high, the edge cost is high, using the cost value of the adjacent node in the time direction and adjacent in the spatial direction. Calculate the cost, using the cost value of the target node and a neighboring node adjacent in the time direction with a pixel in the same position as the target node in the spatial direction in a predetermined number of frames, A time / space direction graph generation step of calculating an edge cost in a time direction so that an edge cost becomes higher if a cost value between the node of interest and the adjacent node is higher;
Based on the edge cost in the spatial direction and the edge cost in the time direction, the cost when the same imaging unit as the preset imaging unit is not allocated to the node of interest for each node in the overlapping region The evaluation function is based on an evaluation function that calculates a total cost of a cost function that adds an image and a cost function that generates an edge cost when images of different imaging means are assigned to adjacent nodes in the spatio-temporal direction. By solving the optimization problem that minimizes the total cost due to determining the image to be assigned to each node of each overlapping area, the overlapping area in the space-time direction is divided into the space-time direction Overlapping region dividing step;
An image generation method comprising: an image composition step of assigning a pixel value of an image specified by an assigned imaging means to each pixel in each of the divided overlapping areas and compositing the images. The cost image generation step includes:
As a function for evaluating the degree of attention of the subject image, an evaluation function is used so that the change in image characteristics for each overlapping area obtained by the conversion means has a high cost value for pixels having a predetermined threshold value or more. The image generation method according to claim 6. The cost image generation step includes:
The cost of a pixel whose luminance change between a predetermined number of consecutive frames is equal to or higher than a predetermined threshold and the cost of a pixel whose luminance change in a peripheral region is higher than a predetermined threshold are high. The image generation method according to claim 7, wherein a simple evaluation function is used. The cost image generation step includes:
The image generation method according to claim 8, wherein an evaluation function is used such that a cost value increases as a difference between luminance values of projected multi-viewpoint images increases in each overlapping region as the evaluation function. . The image composition step includes
Furthermore, the blending process that makes the boundary between the images inconspicuous by performing blurring processing using the color or brightness of the pixel of the adjacent projected multi-viewpoint image with respect to the color or brightness of the pixel near the boundary in the overlapping region The image generation method according to claim 6, wherein the image generation method is performed.   A program for causing a computer to execute the image generation method according to any one of claims 6 to 10.

Images