JP4556745B2 - Image composition apparatus and method, and recording medium - Google Patents

Description

  The present invention relates to an image synthesizing apparatus and method, and a recording medium, which synthesize a panoramic overall image in which a plurality of image areas are pasted by sequentially changing the imaging direction.

  Conventionally, an electronic still camera that has been widely used converts light that has passed through a lens by imaging a subject into an image signal by a solid-state imaging device such as a CCD, and records this on a recording medium. The signal can be reproduced. Many electronic still cameras include a monitor that can display a captured still image, and a specific one of the still images recorded so far can be selected and displayed. In this electronic still camera, the shooting range in one shooting is a narrow range limited by the angle of view of the lens, and a wide range of shooting beyond the angle of view of the lens cannot be performed.

  For this reason, camera systems that capture panoramic images have been proposed in recent years. This camera system is classified into a multi-lens system that captures a wide field of view at a time with a plurality of lenses, and a monocular system that continuously captures individual unit images while sequentially shifting the capturing direction using a monocular lens.

  The multi-lens camera system has an advantage that a panoramic overall image can be generated by almost the same operation as a normal camera, but has a disadvantage that the entire system becomes expensive.

  On the other hand, a monocular camera system (for example, refer to Patent Document 1) is relatively inexpensive, but generates a whole image by joining individual unit images taken continuously so that the joints are not conspicuous. The image processing technology to do is essential. In other words, since the individual fields of view are different from each other due to the fact that the shooting direction is sequentially shifted, the amount of positional deviation between adjacent unit images is estimated using a technique such as pattern matching. This is because it is necessary to correct. In addition, in order to obtain panoramic whole images from overlapping unit images, a method for selecting an optimal unit image based on the obtained boundary line by using the Voronoi division method related to the distance from the center of each unit image is proposed. Has been. Further, a method of smoothly connecting adjacent unit images by using an average of pixel values in mutually overlapping image regions, or a method of selecting an optimal unit image by using a median filter has been proposed.

Japanese Patent Laid-Open No. 11-46317

  By the way, in the above-described monocular camera system, the shooting times of individual unit images that are continuously shot are different. Therefore, when a moving object is continuously shot and an entire image is generated, The contour becomes unnatural.

  The present invention has been proposed to solve such a problem, and provides an image composition apparatus and method, and a recording medium, which can alleviate the unnaturalness of an image caused by the movement of an object. With the goal.

In order to solve the above-described problem, an image composition device according to the present invention is an image composition device in which a subject is photographed while sequentially changing a photographing direction, and a plurality of images obtained by photographing the subject are superimposed to generate a composite image Whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlaid A determination unit that determines whether there is a difference in pixel value by the determination unit, and a first image and a second image that are filtered by the processing unit , A superimposing unit that superimposes the contribution ratio of the first image so as to decrease toward the photographing direction, and an output unit that outputs a composite image superimposed by the superimposing unit are provided .

The image composition device according to the present invention is an image composition device that generates a composite image by superimposing a plurality of images in which a subject is photographed while sequentially changing the photographing direction, and photographing the subject . Determining means for comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlapped to determine whether there is a difference between the pixel values; It is determined that there is a difference in pixel values between the superimposing unit that superimposes the first image and the second image so that the contribution ratio of the first image decreases in the shooting direction, and the determination unit. processing means for filtering position, is characterized in that an output means for outputting the filtered composite image by said processing means.

An image composition method according to the present invention is an image composition method in which a subject is photographed while sequentially changing the photographing direction, and a composite image is generated by superimposing a plurality of images obtained by photographing the subject . A determination step of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the image is superimposed with a second image captured next to the first image ; The contribution ratio of the first image is the processing step of filtering the position determined to have a difference in pixel value by the determination step, and the first image and the second image filtered by the processing step. There the overlapping step of overlapping so as to reduce towards the shooting direction, is characterized by an output step of outputting a composite image superimposed by the process superimposed above.

An image composition method according to the present invention is an image composition method in which a subject is photographed while sequentially changing the photographing direction, and a composite image is generated by superimposing a plurality of images obtained by photographing the subject . A determination step of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are superimposed It is determined that there is a difference in pixel values between the superimposing step of superimposing the first image and the second image so that the contribution ratio of the first image decreases toward the shooting direction, and the determination step. a processing step of filtering the position, is characterized by an output step of outputting the filtered composite image by the above processing steps.

Further, the recording medium according to the present invention is a computer-readable recording in which a subject is photographed by sequentially changing the photographing direction, and a program for generating a composite image by superimposing a plurality of images obtained by photographing the subject is recorded. Whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are superimposed on the medium A determination step for determining whether or not, a processing step for performing a filtering process at a position where a pixel value is determined to be different by the determination step, and a first image and a second image that are filtered by the processing step a superimposition step of the contribution rate of the first image is superimposed so as to reduce toward the shooting direction, it is characterized by an output step of outputting a composite image superimposed by the process superimposed above .

The recording medium according to the present invention is a computer-readable recording in which a subject is photographed while sequentially changing the photographing direction, and a program for generating a composite image by superimposing a plurality of images obtained by photographing the subject is recorded. Whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are superimposed on the medium A determination step for determining whether or not the first image and the second image are overlapped so that the contribution ratio of the first image decreases toward the photographing direction, and pixels are determined by the determination step. a processing step of filtering the determined position and there is a difference in value, is characterized by an output step of outputting the filtered composite image by the above processing steps.

  By detecting a predetermined image area of the moving object and blurring the image area to generate a composite image, the unnaturalness of the outline of the moving object can be reduced.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  The present invention is applied to, for example, an imaging apparatus 1 as shown in FIG. The imaging apparatus 1 includes an imaging unit 10 that images a subject. The imaging unit 10 shields a lens 10a for forming image light from the subject and image light incident through the lens 10a. A diaphragm drive unit 10b that adjusts the diaphragm amount by a shutter blade (not shown) and the like, and a CMOS (Complementary Metal-Oxide Semiconductor) image sensor 11 that generates an electrical imaging signal C1 based on an input subject image. .

  The imaging apparatus 1 also uses a CDS (Correlated Double Sampling) circuit 12 for compensating for variations in the imaging signal C1 generated by the CMOS image sensor 11, and an imaging signal C2 supplied from the CDS circuit 12 in analog / digital. An A / D conversion unit 13 that performs conversion processing, and a digital signal processor that temporarily stores image data as a digitized imaging signal C2 supplied from the A / D conversion unit 13 and performs predetermined processing on the image data ( DSP 15, a codec processing unit 16 that encodes image data from the connected DSP 15, and a memory 17 that stores image data supplied from the codec processing unit 16.

  The imaging apparatus 1 includes a D / A conversion unit 18 that performs digital / analog conversion processing on image data supplied from the DSP 15, and a video encoder that converts image data from the D / A conversion unit 18 into a video signal. Unit 19 and a video encoder unit 19, and includes a monitor unit 20 for displaying an image based on the video signal to the user. The imaging apparatus 1 is connected via an internal bus 14 connected thereto. A central processing unit (CPU) 21 for controlling the entire imaging apparatus 1, an operation unit 22 connected to the internal bus 14 for a user to perform various operations, and a transmission from the CPU 21 via the internal bus 14. A timing generator 23 for controlling a signal processing system from the CMOS image sensor 11 to the DSP 15 based on a control signal generated; Which are connected respectively to the scan 14 and a motor 24 and light meter 26.

  The imaging unit 10 executes an automatic aperture control operation, an automatic focus control operation, and the like based on the operation signal supplied from the CPU 21. The imaging unit 10 adjusts the shooting direction in the horizontal and vertical directions based on the operation signal, and opens and closes shutter blades (not shown) according to the aperture value input via the operation unit 22. Adjust the iris amount with.

  The CMOS image sensor 11 generates an imaging signal C1 obtained by converting a subject image incident through the lens unit 10a and the aperture driving unit 10b into an electrical signal, and outputs this to the CDS circuit 12. Note that the CMOS image sensor 11 can select a part of a subject image formed on the imaging surface and efficiently read out only the pixel value of the region.

  The CDS circuit 12 removes the noise of the imaging signal C1 supplied from the CMOS image sensor 11 using a correlated double sampling circuit, or performs a process for amplifying the gain, and uses this as an imaging signal C2 for A / D. The data is output to the conversion unit 13. The A / D conversion unit 13 performs an analog / digital conversion process on the imaging signal C2 supplied from the CDS circuit 12, and outputs this to the DSP 15. Incidentally, the respective operation timings in the CDS circuit 12 and the A / D converter 13 are controlled by the timing generator 23 so as to continue image capture at a constant frame rate.

  The DSP 15 is a block having a signal processing processor (not shown) and an image RAM. The image represented by the imaging signal C2 from the A / D conversion unit 13 is supplied as stream data configured at a constant frame rate under the control of the timing generator 23, and is temporarily stored in the image RAM. The A signal processor (not shown) is set to perform pre-programmed image processing on the image stored in the image RAM. The image processed in the image RAM (not shown) is transmitted to either or both of the codec processing unit 16 and the D / A conversion unit 18.

  The codec processing unit 16 compresses the data amount of the image transmitted from the DSP 15 by a predetermined method. Here, compression coding may be performed based on a standard such as JPEG (Joint Photographic Experts Group).

  The memory 17 is composed of, for example, a semiconductor memory, a magnetic recording medium, a magneto-optical recording medium, or the like, and is a medium for recording the image data compressed by the codec processing unit 16 at a predetermined address. By configuring the memory 17 as a detachable recording medium, the captured image can be transferred to another PC or the like and viewed, or various searches and processing can be executed.

  The monitor unit 20 displays an image converted into an analog signal by the D / A conversion unit 18 and further converted into a video signal by the video encoder unit 19. The monitor unit 20 may be configured with a liquid crystal display element or the like provided on the side surface of the casing of the imaging apparatus 1 so that the user can check the content of the shooting in real time while executing the imaging process.

  The CPU 21 is connected to a ROM for storing a control program to be executed, a DRAM as a work area used for data accumulation and development, and the like as a central processing unit for controlling the entire image pickup apparatus 1 via the internal bus 14. Play the role of The CPU 21 generates an operation signal based on the operation signal D <b> 1 supplied from the operation unit 22 and the information related to the brightness transmitted from the exposure meter 26, and transmits the operation signal to the imaging unit 10 via the internal bus 14.

  The operation unit 22 includes a key for the user to freely adjust the shooting angle of view and the shooting direction, or to freely adjust the aperture value and the exposure time in the imaging unit 10. The operation unit 22 generates an operation signal D1 corresponding to information input by the user, and transmits the operation signal D1 to the CPU 21 via the internal bus 14. Further, the operation unit 22 includes a shutter button 221, and generates an operation signal D 1 for starting or stopping imaging based on a pressing input operation of the shutter button 221 by the user. To the CPU 21.

  The motor 24 is configured as a drive source for turning the imaging unit 10 such as a stepping motor. The motor 24 rotates in response to an operation signal from the CPU 21. Thereby, the imaging direction of the imaging unit 10 can be changed to the horizontal direction or the vertical direction.

  The exposure meter 26 is a sensor for identifying the brightness of the object imaged by the imaging unit 10, and transmits information regarding the identified brightness to the CPU 21.

  A subject image picked up by the image pickup apparatus 1 having the above-described configuration is converted into an electric signal by the CMOS image sensor 11 to become an image pickup signal C1, noise is removed by the CDS circuit 12 to become an image pickup signal C2, and further A Analog / digital conversion processing is performed in the / D conversion unit 13. Further, the image indicated by the imaging signal C2 is stored in an image RAM (not shown) in the DSP 15, subjected to predetermined image processing, and then subjected to digital / analog conversion processing by the D / A conversion unit 18 to be monitored 20. Or is encoded by the codec processing unit 16 and recorded in the memory 17.

  The present invention can be further applied to an imaging apparatus 3 described below. With respect to the same configuration and elements of the imaging apparatus 3 as those of the imaging apparatus 1 described above, the description of FIG.

  For example, as shown in FIG. 2, the imaging device 3 captures an image of a subject via an imaging unit 10 including a lens unit 10a, and can rotate in the direction of arrow A in FIG. A first casing 31 and a second casing 32 that is disposed below the first casing and can be held with one hand of the user are provided. Incidentally, the second casing 32 is provided with a shutter button 221 and a display unit 43 for the user to visually recognize the captured image.

  FIGS. 3A and 3B show the internal configuration of the imaging apparatus 3. The first housing 31 includes a lens unit 10a, a CMOS image sensor 11, and a first electronic circuit 41 having at least a CDS circuit 12 and an A / D conversion unit 13.

  The second casing 32 is connected to the first casing 31 via a rotation shaft 38, and controls the motor 51 for rotating the rotation shaft 38 and the rotation angle of the rotation shaft 38. In the photo interrupter 52 and the shielding plate 53 for carrying out, a stopper 54 for physically limiting the rotational operation of the rotary shaft 38, and a connecting hole provided in the upper surface of the second housing 32 A bearing 55 that is inserted and realizes a smooth rotation operation by supporting the rotating shaft 38, a second electronic circuit 42 for controlling each component of the imaging device 3, and each component And a battery 44 for operating the element.

  As shown in FIG. 3B, the second electronic circuit 42 includes a DSP 15, a memory 17, a CPU 21, an operation unit 22, a shutter button 221, a display unit 43, a motor, a photo interrupter, An interface 47 for transmitting / receiving image data to / from an external device is provided, and a wiring 56 for transmitting / receiving an electrical signal to / from the first electronic circuit 41 is connected.

  Incidentally, the second casing 32 is provided with buttons for switching between the general shooting mode and the panoramic shooting mode and various buttons of a general digital camera in addition to the above-described components. Yes.

  The motor 51 is configured by a stepping motor or the like that rotates the rotary shaft 38 at an angular velocity corresponding to the supplied drive pulse. The first housing 31 is connected to one end side of the rotating shaft 38 rotated by the motor 51, and a stopper 54 is fixed to the middle. A shielding plate 53 is fixed to the other end of the rotating shaft 38. That is, the first casing 31 rotates in accordance with the rotation operation of the rotating shaft 38 based on the motor 51, and the shielding plate 53 and the stopper 54 rotate in the same manner. Incidentally, the relative displacement in the rotation angle of the rotating shaft 38 by the motor 51 can be detected by the number of pulses applied to the motor 51.

  The display unit 43 includes all configurations similar to the D / A conversion unit 18, the video encoder unit 19, and the monitor unit 20, and displays an image generated via a liquid crystal display element provided on the side surface of the second housing 32. Since the display unit 43 is provided apart from the first casing 31 rotated by the rotation shaft 38, visibility can be improved without being controlled by the rotation of the rotation shaft 38.

  The photo interrupter 52 includes a light emitter 52a and a light receiver 52b which are provided vertically. The light receiving body 52b continues to receive the optical signal from the light emitting body 52a. However, when the shielding plate 53 is rotated to the vicinity of the photo interrupter 52 according to the rotation of the rotating shaft 38, the light is received. The signal is shielded by the shielding plate 53. That is, it is possible to identify the rotation state of the rotary shaft 38 from the shielding state of the optical signal received by the light receiving body 52b.

  In addition, when the rotating shaft 38 rotates beyond the movable range, the shielding plate 53 may shield the optical signal, and the motor 51 may be stopped according to the shielding. FIG. 4 shows the photo interrupter 52 and the shielding plate 53 taken from the B direction in FIG. As shown in FIG. 4, within the movable range, since the shielding plate 53 does not block the optical signal received by the light receiving body 52b, the rotating shaft 38 is freely rotated without stopping the motor 51. be able to. On the other hand, when the movable range is exceeded, the shielding plate 53 blocks the optical signal received by the light receiving body 52, and the motor 51 stops. If the movable range is exceeded, the rotational operation of the first casing 31 may be physically suppressed via the stopper 54.

  In this way, by introducing a mechanism that limits the rotation range of the first casing 31 relative to the second casing 32, data transmission from the first casing 31 to the second casing 32 is performed. Without using a special joint part using a photocoupler, it can be realized by using a flexible wiring material used in a normal movable part.

  A user who uses the imaging apparatus 3 having the above-described configuration performs shooting while holding the second casing 32. In such a case, by pressing the shutter button 221 provided outside the second housing 32, the above-described shooting start timing and shooting end timing can be designated. By providing the shutter button 221 at a position where the second casing 32 is touched by the fingertip of the user holding the second casing 32 with one hand, it is possible to support the start and end of photographing with a small movement of the fingertip.

  Accordingly, by holding the second casing 32 with one hand and pressing the shutter button 221, the first casing 31 gradually rotates from the state shown in FIG. As shown in (b), the shooting direction of the imaging unit 10 changes. Further, by further pressing the shutter button 221, as shown in FIG. 5C, the first casing 31 is further rotated, and as a result, the shooting direction of the imaging unit 10 is further changed. That is, the first casing 31 is configured to be rotatable so that the shooting direction of the imaging unit 10 is sequentially changed around the rotation axis 38 connected.

  Incidentally, when the shutter button 221 is continuously pressed in the state shown in FIG. 5C, the rotation of the motor 51 is suppressed by the photo interrupter 52 described above, and then the first casing 31 is moved. Further, it may be rotated in the opposite direction.

  Incidentally, the second casing 32 is mounted with components having large masses such as a battery 44 and a motor 51 and a second electronic circuit 42 composed of many circuits, whereas the first casing 32 is mounted on the second casing 32. On the body 31, components having a relatively small mass such as the lens unit 10a necessary for photographing, the CMOS image sensor 11, the first electronic circuit 41, and the like are mounted. That is, the first casing 31 is configured to have a lighter mass than the second casing 32. As a result, it is possible to reduce the mechanical load on the parts such as the rotary shaft 38 and the bearing 55 for rotating the first casing 31, and further on the motor 51 and the like, and to keep the cost of the entire product low. It is also possible to realize a general consumer digital camera having a panoramic shooting function at a low cost.

  FIG. 6 is a flowchart showing a photographing process by the imaging device 3.

  The imaging apparatus 3 proceeds to step S61 after performing hardware diagnosis and initialization.

  In step S61, as with a normal camera, the user adjusts the aperture amount, exposure, and the like while using assistance such as an automatic exposure function. As a result, various shooting parameters are determined.

  Next, the process proceeds to step S62, and the pressing of the shutter button 221 is checked to identify the photographing start timing. If the shutter button 221 is pressed, the process directly proceeds to step S63, and if not, the process returns to step S61.

  When the process proceeds to step S63, the imaging unit 10 performs imaging. Data acquired by imaging is transferred to the DSP 15 via the CDS circuit 12 and the A / D converter 13.

  In step S64, the DSP 15 performs image processing to synthesize a panoramic overall image. FIG. 7 shows how a single panoramic overall image is synthesized by joining continuously captured images. Here, it is assumed that the unit images photographed continuously are photographed so that overlapping regions are generated. With respect to this overlapping region, the whole image as shown in FIG. 7 can be synthesized by connecting them based on a conventionally known synthesis method. Incidentally, as this synthesis method, for example, a synthesis technique such as alpha blending may be used.

  Next, the process proceeds to step S65, and the shooting direction by the imaging unit 10 is slightly changed by rotating the first casing 31 with respect to the second casing 32. The amount of rotation of the first casing 31 is adjusted so that an overlapping area is generated between the unit images taken at least continuously. This rotation amount can also be obtained in advance by geometric calculation when the imaging device 3 is designed. Moreover, the amount of rotation of the first casing 31 may be finely adjusted dynamically by mounting a function for detecting the amount of shaking of the entire image pickup apparatus 3 during shooting.

  In step S65, the rotation amount of the motor 51 is set as small as possible, and the continuous shooting is performed by increasing the number of shots per unit time, thereby shooting the distortion due to parallax and moving subjects. The discontinuity of the image can be eliminated, and it becomes possible to synthesize a high-quality whole image. In order to perform such shooting, intermittent and highly accurate control of the motor 51 is required. As described above, the first casing 31 has a mass higher than that of the second casing 32. Therefore, these can be realized at low cost.

  In step S66, the pressing of the shutter button 221 is checked again to identify the timing of the end of shooting. If it is confirmed in step S66 that the shutter button 221 has been pressed, the process returns to step S63 to continue shooting. On the other hand, if it is not confirmed in step S66 that the shutter button 221 has been pressed, the process proceeds to step S67 in order to shift to the photographing end operation.

  In step S67, the combined panoramic overall image is stored in the memory 17 as a process at the end of shooting.

  As described above, by mounting the shutter button 221 on the second casing 32 that is separated from the first casing 31, it is possible to instruct the start and end of photographing only with a small operation of the fingertip. Moreover, the visibility of the obtained image can be improved by providing the display part 43 in the 2nd housing | casing 32. FIG.

  In addition, since at least the parts necessary for photographing such as the lens unit 10a and the CMOS image sensor 11 are mounted in the first casing 31 that is actually rotated, the mass of the rotating part is reduced. The cost of the components that support the rotating portion of the first housing 31 and the motor 51 that rotationally drives the components can be reduced. In addition, by mounting a component serving as a driving source such as the motor 51 in the second casing 32 held by the user, it is possible to suppress the influence of the vibration accompanying the rotation of the motor 51 as much as possible.

  Further, by limiting the rotation range of the first casing 31 using the photo interrupter 52, data transfer between the rotating portion and the holding portion can be realized using a normal and inexpensive flexible wiring material. .

  In addition, in this imaging device 3, you may make it perform all operation | movement of the imaging device 1 mentioned above. As a result, in addition to the effects of the imaging device 3 itself described above, the effects of the imaging device 1 such as an improvement in image quality by overlapping the cut-out slit-like image regions and a reduction in the cost of the entire system can be achieved synergistically. It is also possible to construct a simple shooting system.

  The imaging device 3 to which the present invention is applied is not limited to the above-described embodiment, and the DSP 15, the CPU 21, the display unit 43, the motor 51, the operation unit 22 (shutter button 221), and the battery 44 are included. Any one or more may be provided in the second housing 32.

  By the way, in the case of composing one panoramic overall image by joining images that have been continuously photographed in this way, for example, when photographing a scene where a bright object is moving in front of a dark landscape, the contour of the object Was unnatural. This unnatural outline will be explained using a case where two cameras are continuously shot with a single camera while changing the shooting direction and a panoramic image is created from these two images. Here, it is assumed that the positional relationship on the image corresponding to the angle difference in the shooting direction when two images are shot is (p, 0). That is, the positions of the two images are shifted by p in the horizontal direction.

  FIG. 8 is a diagram illustrating the first image. This image shows a scene where a white rectangular object is moving in front of a dark landscape.

  FIG. 9 is a diagram showing a second image. The second image is taken at a position shifted by (p, 0) with respect to the first image shown in FIG. Since there is a difference in shooting time between the first image and the second image, the white rectangular object has moved from the time when the first image was shot.

  Here, as shown in FIG. 10, a panoramic image is generated by superimposing the second image by shifting it by (p, 0) with respect to the first image. Here, the panoramic image is an image obtained by photographing a wider part than the first image (or the second image).

  FIG. 11 is a diagram illustrating a panoramic image obtained by superimposing the first image and the second image so as to have an appropriate positional relationship. In this panoramic image, since there is a difference between two image times, an unnatural outline appears in the left and right peripheral portions of a white rectangular object.

  Next, how the above-described unnatural contour is generated will be described with reference to FIG. FIG. 12 is a diagram for explaining the degree of black and white on the straight line a of the panoramic image shown in FIG. In FIG. 12, the degree of black and white is shown using 0 (black) to 255 (white).

  As shown in FIG. 12, the positions of the “first image” shown in FIG. 12A and the “second image” shown in FIG. 12B are shifted by p. Further, when the “first image” is photographed, the white rectangular object exists at positions B to D as shown in FIG. 12A, and the “second image” is photographed. In this case, the white rectangular object moves and exists at positions C to E as shown in FIG. Here, the overlapping portion of the “first image” and the “second image” is from A to F.

  By the way, in general, when a composite image is formed by overlapping images, a weighted average using a position-dependent weight function is used in order to smoothly connect the overlapping boundary portions. That is, the boundary portion can be smoothed by setting the weight function so that the contribution ratio of the peripheral portion of the superimposed image becomes zero. This technique is described, for example, in Fig. 2 of PJ Burt and EH Adelson, 鄭 multiresolution spline with application to image mosaics, ACM Trans. On Graphics, Vol. 2, no. 4, pp. 217-236, 1983. Yes.

  Based on this technique, the “weight (contribution rate) of the first image” is set to 1 at point A and 0 at point F as shown in FIG. The weight (contribution rate) of the second image is a value obtained by subtracting the weight (contribution rate) of the first image from 1.

  Next, the degree of black and white of the “composite image” shown in FIG. First, the section from A to B will be described. In this section, since the “first image” and the “second image” are both black (value is 0), the “composite” is the result image regardless of the contribution ratio of each image. "Image" is black (value is 0).

  In the section from B to C, the “first image” is white (value is 255), and the “second image” is black (value is 0). Then, from B to C, the “contribution ratio of the first image” shown in FIG. 12C decreases, and as a result, the black and white of the “composite image” shown in FIG. The degree decreases from B to C.

  Further, in the section from C to D, the “first image” and the “second image” are both white (value is 255), so whatever the contribution rate of each image is, The “composite image” that is the result image is white (value is 255).

  In the section from D to E, the “first image” is black (value is 0), and the “second image” is white (value is 255). Since the “contribution ratio of the first image” decreases from B to C, as a result, the degree of black and white of the “composite image” shown in FIG. It becomes a big value toward.

  Further, in the section from E to F, the “first image” and the “second image” are both black (value is 0), so whatever the contribution rate of each image is, The “composite image” that is the result image is black (value is 0).

  On the left side of A, the “first image” is used as it is as a “synthesized image”, and on the right side of F, the “second image” is used as it is as a “synthesized image”.

  As can be seen from the section from B to C and the section from D to E shown in the “composite image” in FIG. "Image" is the outline of a white rectangular object, from black (background) (left from A) to white (B), gradually becoming black (B to C), and further white (moving object) (C), It will be strange. That is, in an actual image, as shown in FIG. 11, an unnatural outline appears in the left and right peripheral portions of a white rectangular object.

  Therefore, in the present embodiment, an unnatural contour is reduced by detecting a peripheral portion of a moving object and blurring only that portion. Specifically, a low-pass filter (hereinafter referred to as LPF) that limits a high-frequency component band is applied to a peripheral portion of a moving object.

  FIG. 13 is a block diagram illustrating a configuration of the image processing apparatus 100 for realizing image processing using the LPF in the imaging apparatus 3 in hardware. Hereinafter, the image processing in the DSP 15 will be described as being realized by hardware, but the present invention is not limited to this.

  The image processing apparatus 100 includes an image input terminal 101 that inputs a plurality of images, an image position input terminal 102 that inputs position information of the plurality of images, and an LPF that determines an LPF application range of each image based on the position information. An application range determination circuit 103, an LPF circuit 104 that performs LPF processing, a selection circuit 105 that selects an image input from the image input terminal 101 or an image that has been subjected to LPF processing based on the LPF application range, and the selected A superimposing circuit 106 that superimposes images and an image output terminal 107 that outputs a superimposed composite image are provided.

  A plurality of images are input to the image input terminal 101. For example, in the example of the composite image shown in FIG. 11, the first image and the second image are input.

  The image position input terminal 102 receives position information of a plurality of images input to the image input terminal 101. For example, in the example illustrated in FIG. 10, position information (p, 0) indicating the positional relationship between the first image and the second image is input.

  The LPF application range determination circuit 103 arranges each image input from the image input terminal 101 based on the position information input from the image position input terminal 102, and whether or not the values of overlapping pixels match each other. To determine the LPF application range. In addition, by comparing the pixel values at corresponding positions of the images (two or more) that are to be superimposed, it is determined whether or not the values match. It is possible to determine whether the selected object is a stationary object or a moving object.

  The LPF circuit 104 performs low-pass filter processing on each image input from the image input terminal 101.

  The selection circuit 105 selects, for each pixel of each image determined by the LPF application range determination circuit 103, an image pixel processed by the LPF circuit 104 or an image pixel input from the image input terminal 101. When the LPF application range determination circuit 103 determines that the values of the pixels match, the pixel of the image input from the image input terminal 101 is selected, and when it is determined that the values of the pixels are different, the LPF circuit 104 The pixel of the processed image is selected. That is, a process of applying an LPF (low-pass filter) only to the pixel of the moving object portion in each pixel of each image.

  The superimposing circuit 106 generates a composite image using the pixels selected by the selection circuit 105.

  The image output terminal 107 outputs the synthesized image synthesized by the superposition circuit 106.

  Next, the operation of the image processing apparatus 100 will be described using the flowchart shown in FIG. First, in step S <b> 1, a plurality of images are input to the image input terminal 101 and the positional relationship between these images is input to the image position input terminal 102.

  In step S2, a moving object determination process is performed. That is, when the LPF application range determination circuit 103 arranges each image so as to have a positional relationship indicated by the position information, it determines whether or not the values of the overlapping pixels are different to determine the LPF application range.

  In addition, when determining whether or not the values of overlapping pixels differ, there is a possibility of erroneous determination due to noise or the like. Judgments can be reduced. In other words, even if it is determined that “the values of the pixels are different” at a certain pixel position, if the “pixel values are the same” at all the surrounding pixel positions, the attention is paid. Since it is considered that the determined pixel is erroneously determined due to noise, it is forcibly determined that “the values of the pixels are the same”. On the other hand, if it is determined that “the values of the pixels are the same” at a pixel position of interest, but it is determined that “the values of the pixels are different” at all the surrounding pixel positions, Since it is considered that the determined pixel is erroneously determined due to noise, it is forcibly determined that “the values of the pixels are different”. This is because the outline of the moving object is reflected in the image with a certain size, and it is determined that only one point is isolated and “the values of the pixels are different (or the same)”. It is because it is thought that it was misjudged for.

  In step S3, adaptive LPF processing is performed. That is, in the moving object determination processing in step S2, if it is determined that “the values of the pixels are different” among the pixels of each image, the LPF circuit 104 performs LPF processing on the pixel. I do. In the LPF process, for example, the calculation shown in Expression (1) is performed.

  In Expression (1), d (s, t) is a pixel value of the input image at the position (s, t). Further, (x, y) is a position determined as “the values of the pixels are different” in the determination in the moving object determination process. At this position (x, y), as shown in the equation (1), a weighted average is performed with the surrounding pixels of the front, rear, left and right, and high frequency components are removed.

  In step S4, an overlay process is performed. That is, the superposition circuit 106 superimposes the images after the adaptive LPF processing by arranging them so as to be in the positional relationship indicated by the positional information, and creates a result image.

  In step S5, the result image created by the overlay process is output.

  By such image processing, unnaturalness in the contour of the moving object can be reduced.

  Next, a synthesized image synthesized by the above-described image processing will be described with reference to FIG. In FIG. 15, the black and white degree of the image is shown using 0 (black) to 255 (white). In FIG. 15, the positions of the “first image” and the “second image” are shifted by p as in FIG. 12. Further, the overlapping portion between the “first image” and the “second image” is A to F as in FIG. Similarly to FIG. 12, when the “first image” was photographed, the white rectangular object was present at positions B to D shown in FIG. 15, and the “second image” was photographed. In this case, the white rectangular object moves and exists at a position from C to E.

  The LPF application range determination circuit 103 calculates pixel values at positions corresponding to the “first image” shown in FIG. 15A and the “second image” shown in FIG. Compare. That is, the pixel values of the two images at each position when the “second image” is placed at a position shifted by p with respect to the “first image” are compared. Here, as is apparent from FIG. 15, it is determined that “the pixel values are the same” in the section from A to B, the section from C to D, and the section from E to F. Further, it is determined that “the pixel values are different” in the section from B to C and in the section from D to E. In other words, the section from A to B, the section from C to D, and the section from E to F are projected images of a stationary object, the section from B to C, and the section from D to E. In the section, it is a projected image of a moving object.

  By the way, when the LPF process is applied to all the areas of the “first image” shown in FIG. 15A, the vicinity of the position B and the vicinity of the position D are shown in FIG. The image is blurred. Here, A to F shown in FIG. 16 are the same positions as A to F shown in FIG. FIG. 16A is the “first image” shown in FIG.

  Similarly, when the LPF process is performed on all the areas of the “second image” shown in FIG. 15B, the images in the vicinity of the C position and in the vicinity of the E position are blurred.

  Therefore, if a composite image is generated by superimposing images that have been subjected to LPF processing for all regions, an unnatural composite image that is totally blurred will result.

  Therefore, in the image processing of the present embodiment, the LPF process is performed only on the area determined as “the pixel value is different” with respect to the “first image” shown in FIG. That is, the selection circuit 105 sets the pixels in the section from B to C and the section from D to E in the first image as pixels subjected to LPF processing by the LPF circuit 104. This processing result is defined as a “first LPF image” as shown in FIG.

  Also for the “second image”, the LPF process is performed only on the area determined as “the pixel value is different”. That is, the selection circuit 105 sets the pixels in the section from B to C and the section from D to E in the second image as pixels subjected to the LPF processing by the LPF circuit 104. This processing result is defined as a “second LPF image” as shown in FIG.

  As described above, by performing the LPF process only on the area determined as “the pixel value is different”, the position from B to C and the position from D to E which are the outline portions of the white rectangular object of the “first image” The position and the positions from B to C and the positions from D to E, which are the outline portions of the white rectangular object of the “second image”, are blurred images.

  Next, the “first LPF image” and the “second LPF image” are superimposed. Also in this case, as in FIG. 12, the “weight (contribution rate) of the first LPF image” is set to be 1 at point A and 0 at point F as shown in FIG. To do. The weight (contribution rate) of the second LPF image is a value obtained by subtracting the weight (contribution rate) of the first LPF image from 1.

  Thus, when the contribution rate changes depending on the position, the unnaturalness of the contour portion of the moving object is alleviated. This will be described by dividing into sections A to B, sections B to C, sections C to D, sections D to E, and sections E to F shown in FIG.

  The “synthesized image from the LPF image” shown in FIG. 15F is generated by the superimposing circuit 106 by the “first LPF image” shown in FIG. 15C and the “second image shown in FIG. The image is created by superimposing the “LPF image”.

  First, the section from A to B will be described. In this section, the “first LPF image” shown in FIG. 15C and the “second LPF image” shown in FIG. 15D are both black (value is 0). Whatever the contribution ratio of each image, the “synthesized image from the LPF image” shown in FIG. 15F is black (value is 0).

  In the section from B to C, the “first LPF image” shown in FIG. 15C changes from black (value is 0) to white (value is 255), and is shown in FIG. The “second LPF image” also changes from black (value 0) to white (value 255). Since the “contribution rate of the first LPF image” shown in FIG. 15E decreases from B to C, the “composite image from the LPF image” shown in FIG. Compared with the “composite image” shown in FIG. 12D, the transition changes smoothly from B to C.

  In the section from C to D, the “first LPF image” shown in FIG. 15C and the “second LPF image” shown in FIG. 15D are both white (value is 255). Therefore, regardless of the contribution ratio of each image, the “synthesized image from the LPF image” shown in FIG. 15F is white (value is 255).

  In the section from D to E, the “first LPF image” shown in FIG. 15C changes from white (value is 255) to black (value is 0), and is shown in FIG. The “second LPF image” also changes from white (value is 255) to black (value is 0). Since the “contribution ratio of the first LPF image” shown in FIG. 15E decreases from D to E, the “composite image from the LPF image” shown in FIG. Compared with the “composite image” shown in FIG. 12D, the change is smoother from D to E.

  In the section from E to F, both the “first LPF image” shown in FIG. 15C and the “second LPF image” shown in FIG. 15D are black (value is 0). Therefore, regardless of the contribution ratio of each image, the “synthesized image from the LPF image” shown in FIG. 15F is black (value is 0).

  On the left side of A, the first image is used as it is as a “composite image from LPF images”, and on the right side of F, the second image is used as it is as a “composite image from LPF images”.

  In this way, a composite image having a monochrome level shown in FIG. 15 (f) is generated. As is apparent from a comparison of the “composite image from the LPF image” shown in FIG. 15F and the “composite image” shown in FIG. It is possible to reduce the unnaturalness of the contour of the moving object in the composite image when the white rectangular object is moving in front of the landscape.

  In the above-described image processing, the LPF process is performed on the area determined as “the pixel value is different” of each input image. After the input images are synthesized, “the pixel value is different”. The LPF process may be performed on the area determined as.

  Next, image processing for performing LPF processing on this composite image will be described with reference to FIG. Since (a) to (d) in FIG. 17 correspond to (a) to (d) in FIG. 12, respectively, description thereof is omitted here.

  In this image processing, LPF processing is performed on the area determined as “the pixel value is different” for the “composite image” shown in FIG. 17E, that is, the “composite image” shown in FIG. That is, the LPF process is performed only on the section from B to C and the section from D to E. This processing result is an “image obtained by applying LPF to a composite image” shown in FIG.

  As apparent from a comparison between the “image obtained by applying LPF to the composite image” shown in FIG. 17E and the “composite image” shown in FIG. 12D, the image processing described with reference to FIG. By applying the above, it is possible to reduce the unnaturalness of the contour of the moving object in the composite image when the white rectangular object is moving in front of the dark landscape.

  As described above, by comparing the pixel values at corresponding positions of two or more images to be superimposed, the object projected there is a stationary object or moves. It is possible to discriminate whether the object is

  Further, by applying LPF processing that has an effect of blurring an object only at a portion determined to be moving, unnaturalness of the composite image can be eliminated. In addition, the portion determined to be stationary is not subjected to the LPF process, so that a clear image is obtained.

  The target image to be subjected to the LPF process may be an input image or a composite image obtained as a result of superposition.

It is a figure for demonstrating about the structure of the imaging device of the panoramic image to which this invention is applied. It is a figure for demonstrating about the external appearance of the imaging device of the panoramic image which has a 1st housing | casing and a 2nd housing | casing. It is a figure for demonstrating about another structure of the imaging device to which this invention is applied. It is a figure for demonstrating about the method of controlling rotation of the 1st housing | casing by a photo interrupter. It is a figure for demonstrating about the example which rotates a 1st housing | casing gradually by pressing a shutter button. It is a flowchart which shows the imaging | photography process by an imaging device. It is a figure shown about a mode that a continuous panoramic image is connected and one panoramic whole image is synthesize | combined. It is a figure which shows the 1st image. It is a figure which shows the 2nd image. It is a figure for demonstrating a panoramic image. It is a figure which shows the panoramic image which overlap | superposed the 1st image and the 2nd image so that it might become appropriate positional relationship. It is a figure for demonstrating the degree of black-and-white on the straight line a of the panoramic image shown in FIG. It is a block diagram which shows the structure of an image processing apparatus. It is a flowchart which shows operation | movement of an image processing apparatus. It is a figure for demonstrating the synthesized image synthesize | combined by the image process. It is a figure for demonstrating the LPF process with respect to the whole image. It is a figure for demonstrating other image processing.

Explanation of symbols

1, 3 imaging device, 10 imaging unit, 11 CMOS image sensor, 12 CDS circuit, 13 A / D conversion unit, 14 internal bus, 15 DSP, 16 codec processing unit, 17 memory, 18 D / A conversion unit, 19 video Encoder unit, 20 monitor unit, 21 CPU, 22 operation unit, 23 timing generator, 24 motor, 26 exposure meter, 100 image processing device, 101 image input terminal, 102 image position input terminal, 103 LPF application range determination circuit, 104 LPF Circuit, 105 selection circuit, 106 overlay circuit, 107 image output terminal

Claims (10)

An image synthesizer that shoots a subject while sequentially changing the shooting direction, and generates a composite image by superimposing a plurality of images of the subject .
Judgment of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlaid Means,
Processing means for filtering the position determined by the determination means to have a difference in pixel value;
Superimposing means for superimposing the first image and the second image filtered by the processing means so that the contribution ratio of the first image decreases toward the photographing direction ;
Output means for outputting a composite image superimposed by the superimposing means;
An image synthesizing apparatus. The image synthesizing apparatus according to claim 1, wherein the determination unit determines whether there is a difference between the pixel values based on a determination result of pixel values of peripheral pixels. The image synthesizing apparatus according to claim 1, wherein the processing means performs blurring processing using a low-pass filter. An image synthesizer that shoots a subject while sequentially changing the shooting direction, and generates a composite image by superimposing a plurality of images of the subject .
Judgment of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlaid Means,
Superimposing means for superimposing the first image and the second image so that the contribution ratio of the first image decreases toward the shooting direction ;
Processing means for filtering the position determined by the determination means to have a difference in pixel value;
Image synthesis apparatus and an output means for outputting the filtered composite image by said processing means. An image composition method in which a subject is photographed by sequentially changing the photographing direction, and a composite image is generated by superimposing a plurality of images in which the subject is photographed.
A determination step of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are superimposed. When,
A processing step of performing a filtering process at a position where the pixel value is determined to be different by the determination step;
A superimposing step of superimposing the first image and the second image filtered by the above-described processing step so that the contribution ratio of the first image decreases toward the photographing direction ;
Image composition method and an output step of outputting a composite image superimposed by the process superimposed above. The image composition method according to claim 5 , wherein in the determination step, it is determined whether there is a difference between the pixel values based on a determination result of pixel values of neighboring pixels. The image synthesizing method according to claim 5 , wherein in the processing step, blurring is performed by a low-pass filter. An image composition method in which a subject is photographed by sequentially changing the photographing direction, and a composite image is generated by superimposing a plurality of images in which the subject is photographed.
Judgment of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlaid Process,
A superimposing step of superimposing the first image and the second image so that the contribution ratio of the first image decreases toward the photographing direction ;
A processing step of performing a filtering process at a position where the pixel value is determined to be different by the determination step;
Image composition method and an output step of outputting the filtered composite image by the above processing steps. Subject is photographed sequentially changing the photographing direction, a computer-readable recording medium on which a program is recorded for generating a composite image by superimposing a plurality of images which the subject is photographed,
Judgment of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlaid Process,
A processing step of performing a filtering process at a position where the pixel value is determined to be different by the determination step;
A superimposing step of superimposing the first image and the second image filtered by the processing step so that the contribution ratio of the first image decreases toward the photographing direction ;
A recording medium comprising: an output step of outputting the composite image superposed by the superposition step. Subject is photographed sequentially changing the photographing direction, a computer-readable recording medium on which a program is recorded for generating a composite image by superimposing a plurality of images which the subject is photographed,
Judgment of comparing whether or not there is a difference in pixel values by comparing pixel values at corresponding positions when the first image and the second image captured next to the first image are overlaid Process,
A superimposing step of superimposing the first image and the second image so that the contribution ratio of the first image decreases toward the shooting direction ;
A processing step of performing a filtering process at a position where the pixel value is determined to be different by the determination step;
Recording medium and an output step of outputting the filtered composite image by the above processing steps.

Images