JP6928772B2 - Image processing device - Google Patents

Description

The present invention relates to an image processing apparatus, program and method for restoring an original interlaced image from an enlarged interlaced image. Restoring the original interlaced image includes restoring the field image included in the original interlaced image.

As is well known, an interlaced image is an image in which two field images are alternately combined so that one constitutes an odd number of rows and the other constitutes an even number of rows. Therefore, when there is a time lag between the two field images and the subject is moving, the same subject appears in the interlaced image with the odd-numbered rows and even-numbered rows staggered in the moving direction, and horizontal stripes called combing noise appear. A pattern appears.

There are various types of image sharpening methods, such as brightness and contrast adjustment, vividness and hue adjustment, noise removal, sharpness, blur and blur correction, enlargement, distortion correction, and tilt correction. However, in some cases, these processes may emphasize the problems of the image or disperse it in the surroundings, and rather deteriorate the image quality. For example, if the interlaced image is not deinterlaced, that is, if processing such as sharpness or distortion correction is performed without separating the two field images, combing noise is emphasized or deformed, so that the image quality is clearly improved. Deteriorates. Therefore, it is preferable to deinterlace the interlaced image before performing the sharpening process. As described above, the interlaced image has a regularity that two field images are arranged alternately. Therefore, according to this regularity, the interlace can be easily released (for example, Patent Document 1). reference).

Japanese Unexamined Patent Publication No. 2014-033438

By the way, some players dedicated to security cameras arbitrarily scale the image when displaying a video or outputting a file regardless of the intention of the user. Also, when an interlaced image is acquired via a capture device, the image size may be changed automatically. At this time, if the magnification at the time of enlargement of the enlarged interlaced image (hereinafter, may be referred to as an enlarged interlaced image) is unknown, the interlaced image before enlargement (hereinafter, may be referred to as an original interlaced image) is restored. Cannot be separated into two field images. However, if the enlarged interlaced image is sharpened without deinterlacing, the image quality will deteriorate as described above.

Further, when the image is enlarged at least in the vertical direction for some reason as described above, new rows may be inserted into the image at predetermined intervals according to the enlargement magnification. Therefore, when the interlaced image is enlarged at least in the vertical direction, the regularity of the alternating arrangement of the two original field images is broken in the enlarged interlaced image. Therefore, if the position of the additional line at the time of enlargement is not known in the enlarged interlaced image, the original interlaced image cannot be restored and cannot be separated into two field images. However, if the enlarged interlaced image is sharpened without deinterlacing, the image quality will deteriorate as described above.

An object of the present invention is to restore an original interlaced image from an enlarged interlaced image.

The image processing device according to the first aspect of the present invention is an image processing device that restores the original interlaced image from an enlarged interlaced image in which the original interlaced image is enlarged, and includes a pattern detection unit and a restoration unit. The pattern detection unit detects a striped pattern due to combing noise included in the magnified interlaced image by performing image processing on the magnified interlaced image, and based on the striped pattern, the enlarged interlaced image is described as described above. Calculate the reduction ratio to the original interlaced image. The restoration unit creates at least one of a restoration image obtained by restoring the original interlaced image and a field image constituting the restored image based on the reduction magnification.

The image processing apparatus according to the second aspect of the present invention is the image processing apparatus according to the first aspect, and the pattern detection unit changes from the enlarged interlaced image to the original interlaced image based on the striped pattern. The temporary reduction ratio of the above is calculated, and a plurality of candidate pixels that are candidates for the number of vertical pixels of the original interlaced image are set based on the provisional reduction ratio and the number of vertical pixels of the enlarged interlaced image. A plurality of the reduction ratios corresponding to the number of candidate pixels are calculated, and the restoration unit creates at least one of the restoration image and the field image for each of the plurality of reduction ratios.

The image processing apparatus according to the third aspect of the present invention is the image processing apparatus according to the second aspect, and the number of the plurality of candidate pixels is an even number.

The image processing apparatus according to the fourth aspect of the present invention is an image processing apparatus according to any one of the first to third aspects, and the restoration unit uses a linear interpolation algorithm to obtain the enlarged interlaced image. The restored image is created by reducing the image at a reduction ratio.

The image processing apparatus according to the fifth aspect of the present invention has the original interlaced image from the enlarged interlaced image magnified at a certain magnification by adding additional lines at predetermined intervals to the original interlaced image. It is an image processing device that restores an image, and includes a pattern detection unit and a restoration unit. The pattern detection unit detects a striped pattern due to combing noise included in the magnified interlaced image by performing image processing on the magnified interlaced image, and includes the magnified interlaced image based on the striped pattern. Calculate the parameters that specify the position of the additional line. The restoration unit removes the additional row from the enlarged interlaced image based on a parameter for specifying the position of the additional row, thereby restoring the original interlaced image and a field image constituting the restored image. Create at least one of them.

The image processing apparatus according to the sixth aspect of the present invention is the image processing apparatus according to the fifth aspect, and the parameter for specifying the position of the additional row includes the enlargement magnification.

The image processing apparatus according to the seventh aspect of the present invention is an image processing apparatus according to the fifth or sixth aspect, and the pattern detection unit captures an image having a striped pattern every other line at the magnification. The enlarged pattern image, which is an image enlarged at the same magnification as the above, and the enlarged interlaced image are aligned so that the striped patterns of both images match most, and the addition is based on the result of the alignment. Calculate the parameters that specify the row position.

The image processing apparatus according to the eighth aspect of the present invention is an image processing apparatus according to any one of the fifth to seventh aspects, and further includes a bleeding correction unit for removing bleeding of the restored image.

The image processing apparatus according to the ninth aspect of the present invention is the image processing apparatus according to the eighth aspect, and the bleeding correction unit calculates a correction coefficient from a pixel value included in the bleeding pattern image and uses the correction coefficient as the correction coefficient. Based on this, the pixel value of the restored image is corrected. The blur pattern image is an image in which an image having a striped pattern every other line is enlarged at the same magnification as the enlargement magnification and then reduced at a reduction magnification that is the reciprocal of the enlargement magnification.

The image processing apparatus according to the tenth aspect of the present invention is an image processing apparatus according to any one of the first to ninth aspects, and the striped pattern includes the period of the striped pattern.

The image processing apparatus according to the eleventh aspect of the present invention is the image processing apparatus according to the tenth aspect, and the enlarged interlaced image is a moving image including a plurality of frames. The pattern detection unit creates a noise-enhanced image in which the combing noise is emphasized from the plurality of frames, creates a difference image showing a difference between each line of the noise-enhanced image, and includes the difference image in each line of the difference image. A vertical profile representing the characteristics of the pixel values is created, and the period of the striped pattern is calculated from the position of the peak of the power spectrum of the vertical profile.

The image processing program according to the twelfth aspect of the present invention is an image processing program that restores the original interlaced image from an enlarged interlaced image in which the original interlaced image is enlarged, and causes a computer to execute the following steps.
(1) A step of detecting a striped pattern due to combing noise included in the magnified interlaced image by image processing the magnified interlaced image (2) Based on the striped pattern, the magnified interlaced image is described as described above. Step of calculating the reduction magnification to the original interlaced image (3) Based on the reduction magnification, a step of creating at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image.

The image processing method according to the thirteenth aspect of the present invention is an image processing program that restores the original interlaced image from an enlarged interlaced image in which the original interlaced image is enlarged, and includes the following steps.
(1) A step of detecting a striped pattern due to combing noise included in the magnified interlaced image by image processing the magnified interlaced image (2) Based on the striped pattern, the magnified interlaced image is described as described above. Step of calculating the reduction magnification to the original interlaced image (3) Based on the reduction magnification, a step of creating at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image.

In the image processing program according to the 14th aspect of the present invention, the original interlaced image is enlarged from the enlarged interlaced image at a certain magnification by adding additional lines at predetermined intervals to the original interlaced image. An image processing program that restores an image and causes a computer to perform the following steps.
(1) A step of detecting a striped pattern due to combing noise included in the magnified interlaced image by image processing the magnified interlaced image (2) Included in the magnified interlaced image based on the striped pattern. Step of calculating a parameter for specifying the position of the additional row (3) Restoring the original interlaced image by removing the additional row from the enlarged interlaced image based on the parameter for specifying the position of the additional row. A step of creating at least one of the restored image and the field image constituting the restored image.

In the image processing method according to the fifteenth aspect of the present invention, the original interlaced image is enlarged from the enlarged interlaced image at a certain magnification by adding additional lines at predetermined intervals to the original interlaced image. An image processing method for restoring an image, which includes the following steps.
(1) A step of detecting a striped pattern due to combing noise included in the magnified interlaced image by image processing the magnified interlaced image (2) Included in the magnified interlaced image based on the striped pattern. Step of calculating a parameter for specifying the position of the additional row (3) Restoring the original interlaced image by removing the additional row from the enlarged interlaced image based on the parameter for specifying the position of the additional row. A step of creating at least one of the restored image and the field image constituting the restored image.

In the image processing apparatus according to the 16th aspect of the present invention, the original interlaced image is enlarged from the enlarged interlaced image at a certain magnification by adding additional lines at predetermined intervals to the original interlaced image. It is an image processing device that restores an image, and includes a detection unit and a restoration unit. The detection unit detects duplicate rows having the same or substantially the same pixel value included in the enlarged interlaced image by performing image processing on the enlarged interlaced image. The restoration unit creates at least one of a restoration image obtained by restoring the original interlaced image and a field image constituting the restored image by removing the overlapping lines from the enlarged interlaced image.

According to the first aspect of the present invention, by performing image processing on the enlarged interlaced image, a striped pattern due to combing noise included in the enlarged interlaced image is detected. Then, based on this striped pattern, the reduction magnification for restoring the original interlaced image from the enlarged interlaced image is calculated. Then, based on this reduction magnification, at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image is created. From the above, the original interlaced image can be restored from the enlarged interlaced image.

According to the fifth aspect of the present invention, by performing image processing on the enlarged interlaced image, a striped pattern due to combing noise included in the enlarged interlaced image is detected. Then, based on this striped pattern, the position of the additional line at the time of enlargement included in the enlarged interlaced image is specified. Then, this additional line is removed, and at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image is created. From the above, the original interlaced image can be restored from the enlarged interlaced image.

According to the 16th aspect of the present invention, by performing image processing on the enlarged interlaced image, overlapping rows having the same or substantially the same pixel value included in the enlarged interlaced image are detected. Then, by removing the duplicated lines, at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image is created. From the above, the original interlaced image can be restored from the enlarged interlaced image.

The block diagram of the image processing apparatus which concerns on 1st Embodiment of this invention. The figure of the basic screen before the image data is imported. The figure of the basic screen after the image data is imported. The figure which shows the still image group belonging to 1 timeline. The flowchart which shows the flow of the process of the special deinterlacing which concerns on 1st Embodiment of this invention. An example of a noise-enhanced image created by self-weighted averaging from the difference image of the enlarged interlaced image. An example of a noise-enhanced image created by mere averaging from the difference image of the same enlarged interlaced image as in FIG. 6A. An example of a difference image showing the difference between the lines of a noise-enhanced image of an enlarged interlaced image by the nearest neighbor method. An example of a difference image showing the difference between the lines of a noise-enhanced image of an enlarged interlaced image by the bilinear method. An enlarged view of the inside of the square frame shown in FIG. 7A. An enlarged view of the inside of the square frame shown in FIG. 7B. The graph of the vertical profile derived from the difference image of FIG. 7B. The graph of the power spectrum of the vertical profile of FIG. (A) A pattern image assuming combing noise. (B) An enlarged pattern image obtained by enlarging (a) by the bilinear method. (C) A difference image showing the difference between the lines of (b). (D) The image of (c) aligned with respect to (e). (E) A difference image showing the difference between lines of a noise-enhanced image created from an enlarged interlaced image. The graph of the pixel value of the difference image of FIG. 11C. The figure which shows the appearance that the enlarged pattern image is arranged by shifting it downward one pixel at a time. The figure of the reduced image which reduced each enlarged pattern image of FIG. 13 by the nearest neighbor method. (A) Enlarged pattern image of FIG. 11 (b). (B) A reduced image obtained by reducing (a) by the nearest neighbor method. (C) The pattern image of FIG. 11 (a). (D) An image obtained by extracting a part of the image shown in the frames of (b) and (c). The flowchart which shows the flow of the process of the special deinterlacing which concerns on a modification. An example of a magnified interlaced image. FIG. 6 is a diagram showing the result of performing normal deinterlacing on the enlarged interlaced image of FIG. 17A (comparative example). It is a figure which shows the result of having performed the reduction process (S3) of the special deinterlacing cancellation of FIG. 5 with respect to the enlarged interlace image of FIG. 17A. It is a figure which shows the result of having performed the blur correction processing (S4) of the special deinterlacing cancellation of FIG. 5 on the enlarged interlace image of FIG. 17A. FIG. 6 is a diagram showing the result of deinterlacing the special interlace of FIG. 5 with respect to the enlarged interlaced image of FIG. 17A (Example 1). FIG. 6 is a diagram showing the result of performing normal deinterlacing on the reduced image of FIG. 17C (Example 2). The flowchart which shows the flow of the process of the special deinterlacing which concerns on 2nd Embodiment of this invention. (A) A pattern image assuming combing noise. (B) An image obtained by enlarging the pattern image of (A) by a normal bilinear method and then reducing it to the original size by a normal bilinear method. (C) An image obtained by enlarging the pattern image of (A) by a normal bilinear method and then reducing it to the original size by the bilinear method according to the second embodiment. The PSNR graph of each reduced image and the original pattern image when the pattern image of FIG. 19A is enlarged by the bilinear method at various magnifications and then reduced to the original size. FIG. 17A is a diagram showing the result of performing special deinterlacing by the algorithm according to the second embodiment on the enlarged interlaced image of FIG. 17A (Example 3). The figure which shows the sample of 6 kinds of original interlaced images which concern on Examples 4-6. Restored images when sample 1 is magnified at 1.3 times and 1.6 times and restored to its original size by the bilinear method and the normal bilinear method according to the second embodiment (Example 5 and Comparative Example 5). ). Restored images when sample 2 is magnified at 1.3 times and 1.6 times and restored to its original size by the bilinear method and the normal bilinear method according to the second embodiment (Example 5 and Comparative Example 5). ). Restored images when sample 3 is magnified at 1.3 times and 1.6 times and restored to its original size by the bilinear method and the normal bilinear method according to the second embodiment (Example 5 and Comparative Example 5). ). Restored images when sample 4 is magnified at 1.3 times and 1.6 times and restored to the original size by the bilinear method and the normal bilinear method according to the second embodiment (Example 5 and Comparative Example 5). ). Restored images when sample 5 is magnified at 1.3 times and 1.6 times and restored to its original size by the bilinear method and the normal bilinear method according to the second embodiment (Example 5 and Comparative Example 5). ). Restored images when sample 6 is magnified at 1.3 times and 1.6 times and restored to its original size by the bilinear method and the normal bilinear method according to the second embodiment (Example 5 and Comparative Example 5). ). The graph which shows the relationship between the magnification and PSNR in the case of sample 3. The graph which shows the relationship between the magnification and PSNR in the case of sample 6. An image obtained by deinterlacing the image of FIG. 25 according to sample 3 (Example 6 and Comparative Example 6).

Hereinafter, the image processing apparatus, the program, and the method according to some embodiments of the present invention will be described with reference to the drawings.

<1. First Embodiment>
<1-1. Image processing device configuration>
The image processing device 1 shown in FIG. 1 is an image processing device according to the first embodiment of the present invention. The image processing device 1 is a general-purpose personal computer as hardware. The image processing program 2 is provided and installed in the image processing device 1 from a computer-readable recording medium 60 such as a CD-ROM or a USB memory, or via a network such as a LAN or the Internet. The image processing program 2 is application software for supporting image processing for moving images and still images. The image processing program 2 causes the image processing device 1 to execute the steps included in the operations described later.

The image processing device 1 includes a display 10, an input unit 20, a storage unit 30, and a control unit 40. These units 10 to 40 are connected to each other via a communication line 5 such as a bus line or a cable, and can communicate with each other as appropriate. The display 10 is composed of a liquid crystal display or the like, and displays a screen or the like described later to the user. The input unit 20 is composed of a mouse, a keyboard, a touch panel, and the like, and receives an operation from the user on the image processing device 1. The storage unit 30 is a non-volatile storage area composed of a hard disk, a flash memory, or the like. The control unit 40 is composed of a CPU, a ROM, a RAM, and the like.

The image processing program 2 is stored in the storage unit 30. A software management area 50 is secured in the storage unit 30. The software management area 50 is an area used by the image processing program 2. An original image area 51 and a processed file area 52 are secured in the software management area 50. The roles of the regions 51 and 52 will be described later.

The control unit 40 virtually operates as the display control unit 41 and the image processing unit 42 by reading and executing the image processing program 2 stored in the storage unit 30. The display control unit 41 controls the display of screens, windows, buttons and all other elements displayed on the display 10. The image processing unit 42 executes various types of image processing. The image processing unit 42 virtually operates as the detection unit 42a, the restoration unit 42b, and the correction unit 42c during the execution of the special deinterlacing process described later. Details of the operation of each part 41, 42, 42a to 42c will be described later.

<1-2. Operation of image processing device>
When the control unit 40 detects that the user has performed a predetermined operation via the input unit 20, the control unit 40 starts the image processing program 2. When the image processing program 2 is started, the basic screen W1 (see FIG. 2) is displayed on the display 10.

<1-2-1. Importing image data>
The basic screen W1 receives a command from the user to import image data into the original image area 51. The image data captured in the original image area 51 is subject to reproduction processing and image processing described later. The control unit 40 takes in image data from the still image file or the moving image file into the original image area 51. In the present specification, the still image file is a still image format data file, and the moving image file is a moving image format data file.

When importing image data from a still image file, the user specifies one still image file or one folder by operating the input unit 20. In the former case, the control unit 40 causes the user to input the address path and the file name in the storage unit 30 of the still image file. In the latter case, the control unit 40 causes the user to input the address path and the folder name in the storage unit 30 of the folder. After that, the control unit 40 saves the designated still image file or all the still image files in the designated folder as a still image file group in the original image area 51. In addition, in this specification, the term "group" is not limited to a plurality of elements, and may be one.

On the other hand, when the image data is taken from the moving image file, the user inputs the address path and the file name in the storage unit 30 of the moving image file 1 by operating the input unit 20. When the display control unit 41 detects that the user has specified a moving image file, the display control unit 41 superimposes and displays a moving image import window (not shown) on the basic screen W1. The video capture window accepts the user to select any section of the entire timeline of the specified video file. When the control unit 40 detects that the user has selected a specific section via the input unit 20, the control unit 40 generates a still image file group corresponding to the frame group included in the selected section on a one-to-one basis. After that, the control unit 40 saves the still image file group in the original image area 51. Therefore, in the present embodiment, the image data to be reproduced and image processed, which will be described later, is not a moving image file but a still image file.

Note that the control unit 40 selects the files included in the still image file group even if the still image file group captured in the original image area 51 is not derived from the moving image file but is derived from the still image file. Recognize that they are arranged along the timeline. The array is automatically determined from the file attributes (file name, creation date and time, modification date and time, etc.).

<1-2-2. Playback processing>
When the still image file group is taken into the original image area 51, the display control unit 41 superimposes the display window W2 (see FIG. 3) on the basic screen W1 and displays it. Display windows W2 are created for the number of timelines of the still image files captured in the original image area 51.

In the display window W2, first, one still image file (for example, a still image file corresponding to the first frame on the timeline) included in the still image file group captured in the original image area 51 is displayed. .. After that, as will be described later, the frame displayed in the display window W2 is switched in response to a user operation.

As shown in FIG. 3, a window selection pull-down menu T1, a play button T2, a frame advance button T3, a frame return button T4, and a timeline bar T5 are arranged on the basic screen W1.

Even if there are a plurality of display windows W2, there is only one active display window W2. The window selection pull-down menu T1 accepts from the user the selection of which display window W2 is to be activated. Hereinafter, the timeline corresponding to the active display window W2 is referred to as an active timeline, and the frame group belonging to the active timeline is referred to as an active frame group. Further, the frame currently displayed in the active display window W2 is called an active frame.

The display control unit 41 can reproduce the active frame group as a moving image in the active display window W2. The play button T2 receives a command from the user to play the active frame group as a moving image. When the display control unit 41 detects that the user has pressed the play button T2 via the input unit 20, the frames included in the active frame group are sequentially frame-advanced along the timeline in the active display window W2. Display in the format of. The reproduction starts from the active frame at the time when the reproduction button T2 is pressed. Further, the play button T2 receives an instruction to stop playback from the user. When the display control unit 41 detects that the user presses the play button T2 via the input unit 20 during playback, the display control unit 41 fixes the display in the active display window W2 to the active frame at that time.

Each of the frame advance button T3 and the frame return button T4 receives an instruction from the user to switch the active frame to the next frame after the active frame along the active timeline.

The timeline bar T5 is an object that graphically represents the active timeline. The timeline bar T5 is equally divided by the number of frames included in the active frame group in the direction in which the bar extends. The nth division area from the left on the timeline bar T5 corresponds to the nth frame on the active timeline (n is a natural number).

As shown in FIG. 3, the display control unit 41 displays the divided area A1 corresponding to the selected frame group and the divided area A2 corresponding to the non-selected frame group in different display formats on the timeline bar T5. .. The selected frame group is a frame group belonging to the currently selected section on the active timeline. The non-selected frame group is a frame group belonging to a section that is not currently selected on the active timeline.

The timeline bar T5 receives from the user the selection of an arbitrary section on the active timeline. The section selected at this time may be a continuous section or may be a discontinuous section as shown in FIG. Specifically, the user can select an arbitrary number of arbitrary frames from the active frame group by operating the divided area on the timeline bar T5 via the input unit 20. Multiple division areas can be selected at the same time. The display control unit 41 immediately switches the active frame to the frame corresponding to the latest selected division area each time the division area on the timeline bar T5 is selected by the user. The image processing unit 42 recognizes the selected frame group as an image processing target described later.

<1-2-3. Image processing>
The image processing unit 42 performs brightness / contrast / saturation adjustment, noise removal, sharpness, blur / blur correction, enlargement / reduction, distortion correction, tilt correction, normal deinterlacing, special deinterlacing, etc. for the selected frame group. It is possible to execute multiple image processing modules of. The image processing module is incorporated in the image processing program 2.

By operating the basic screen W1 via the input unit 20, the user can select any image processing module in any order and any number of times. Each time the image processing unit 42 detects that the user has selected the image processing module, the image processing unit 42 executes the image processing module for the selected frame group at that time. To execute the image processing module for the selected frame group means to execute the image processing module for each frame included in the selected frame group.

As the image processing module is sequentially executed once, twice, three times, ... For the frame, the frame is sequentially executed as the first, second, third, ... It will be processed. The 0th frame corresponds to a still image file stored in the original image area 51. The (m + 1) th-th frame corresponds to the still image file after the image processing module is executed once for the still image file of the m-th frame (m is an integer of 0 or more). The image processing unit 42 sequentially generates still image files corresponding to the first and subsequent frames, and stores these still image files separately in the processed file area 52.

FIG. 4 is a conceptual diagram showing how the image group belonging to the timeline of 1 is managed by the image processing program 2. In FIG. 4, the N axis on the horizontal axis indicates the order of frames on the timeline, and the M axis on the vertical axis indicates the order of processing. The quadrangle corresponding to the coordinates (n, m) in the NM space of FIG. 4 represents the image I (n, m). Image I (n, m) is the m-th order image of the nth frame on the timeline (n is a natural number and m is an integer of 0 or more).

The control unit 40 manages the value of the currently selected coordinate m as the parameter m _{s for each frame.} Immediately after the still image files are imported into the original image area 51, the coordinate _ms has an initial value of 0. Thereafter, every time the image processing module is executed once, the coordinate m _s of the frame is incremented by one. The user, by performing a predetermined operation via the input unit 20 can be changed freely coordinate m _s of any frame. To execute the image processing module on a frame means to execute the image processing module on the image of the _msth order of the frame. Therefore, to change the coordinate m _s is meaningful that changes the target of the execution of the image processing module. Further, the display frame is to display the image coordinates m _s of the frame. Therefore, to change the coordinate m _s is also means to change the object to be displayed in the active display window W2.

<1-3. Special deinterlacing>
Hereinafter, the special deinterlacing process, which is one of the image processes implemented in the image processing program 2, will be described. The interlaced image is an image in which two field images are alternately combined so that one constitutes an odd-numbered row and the other constitutes an even-numbered row. Normal deinterlacing refers to separating an interlaced image into an image consisting of odd-numbered rows and an image consisting of even-numbered rows, that is, into two original field images according to this regularity.

On the other hand, special deinterlacing means separating the enlarged interlaced image into two original field images. The magnified interlaced image is an image in which the original interlaced image is magnified at least at a magnification in the vertical direction, in other words, an image in which additional lines are added at predetermined intervals according to the magnified magnification to the original interlaced image. (Including when additional columns have been added). Therefore, in the magnified interlaced image, the presence of these additional rows breaks the regularity of the alternating arrangement of the two field images before compositing. Therefore, the enlarged interlaced image is a special interlaced image that cannot be deinterlaced by a normal deinterlacing process. The special deinterlacing is a process for deinterlacing such a special interlaced image. Specifically, by removing additional rows from the magnified interlaced image, a restored image is created in which the magnified interlaced image is restored to its original size at least vertically, and this restored image is used as an odd-numbered field image and an even-numbered row. Separated from the field image. At this time, the position of the additional line included in the enlarged interlaced image is specified by image processing based on the striped pattern due to the combing noise included in the enlarged interlaced image. The combing noise is noise peculiar to an interlaced image that appears as a striped pattern in the horizontal direction. That is, it is noise generated when subjects moving in odd-numbered rows and even-numbered rows are alternately shifted in the moving direction due to the time difference between the two field images before composition.

Hereinafter, the special deinterlacing algorithm will be described in detail with reference to FIG. FIG. 5 is a flowchart showing the flow of the special deinterlacing process according to the present embodiment. The special deinterlacing process is executed for each frame included in the selected frame group. However, since the special deinterlacing process according to the present embodiment requires at least two frames, it is executed only when the selected frame group includes at least two frames. Therefore, the process according to the flowchart of FIG. 5 starts when the user orders the execution of this process in a state where two or more frames are selected on the active timeline via the input unit 20.

In the first step S1, the detection unit 42a detects _{a striped pattern due to combing noise of frames F 1} , F ₂ , ..., F _N (N is an integer of 2 or more) included in the selected frame group. .. The striped pattern is detected by the following algorithm. First, the detection unit 42a creates a difference image FDif _{n between two frames F n} and F _{n + 1,} which are temporally adjacent to each other with respect to n = 1, 2, ..., N-1. Specifically, according to the following equation, for x = 1, 2, ..., W, y = 1, 2, ..., H, each pixel value FDif _{n, x included in the difference image FDif n , Y} is calculated. Note that the frame F _n is an image of H (vertical) × W (horizontal) pixels, FDif _{n, x, y} have pixel values in y rows and x columns of _{FDif n , and Dat n, x, y} are There is a pixel value of y row x column of frame F _n.

In the difference image FDif _n , the stationary object disappears and the moving body portion is relatively emphasized. Since the combing noise appears in the moving body portion reflected _{in the frames F 1} , F ₂ , ..., F _N , the combing noise is emphasized in _{the difference image FDif n.}

Next, the detection unit 42a creates an image WFDif in which _{the difference images FDif 1} , FDif ₂ , ..., FDif _{N-1 are superimposed. Specifically, each pixel value WFDif x, y} included in the image WFDif is calculated for x = 1, 2, ..., W, y = 1, 2, ..., H according to the following equation. do. Note that WFDif _{x, y} is a pixel value of the y row x column of the image WFDif. The image WFDef is an image in which the combing noise contained in the frames F ₁ , F ₂ , ..., F _{N is further emphasized by superimposing the difference images FDif 1} , FDif ₂ , ..., FDif _N-1. Hereinafter, it may be referred to as a noise-enhanced image).

In the equation of Equation 2, superposition is performed by self-weighted average. That is, the noise-enhanced image WFDif according to the equation of _{Equation 2 averages the pixel values included in the difference images FDif 1} , FDif 2, ..., FDif _N-1 for each pixel with their own pixel values as weights. It is an image that was made. The calculation method of the noise-enhanced image WFDif is not limited to this, and for example, it may be a simple average image of _{the difference images FDif 1} , FDif ₂ , ..., FDif _N-1. Further, without superimposing the difference images FDif ₁ , FDif ₂ , ..., FDif _N-1 , any one difference image FDif _n can be used as it is as the noise-enhanced image WFDif, and the subsequent processing can proceed.

FIG. 6A is a noise-enhanced image WFDif created from 96 enlarged interlaced images according to the equation of Equation 2. On the other hand, FIG. 6B is a noise-enhanced image WFDef created as a mere average image of _{the difference images FDif 1} , FDif ₂ , ..., FDif _{N-1 of the same 96 enlarged interlaced images.} As is clear from a comparison of these, it can be seen that the combing noise is further emphasized by performing the self-weighted averaging.

Next, the detection unit 42a creates a difference image LDif representing the difference between adjacent lines of the noise-enhanced image WFDif. Specifically, according to the following equation, for x = 1, 2, ..., W, y = 1, 2, ..., H-1, each pixel value LDif _{x, included in the difference image LDif x,} Calculate _y. In addition, LDif _{x, y} is a pixel value of y row x column of the difference image LDif.

FIG. 7A is a difference image LDif created from the image of FIG. 6A according to the equation of Equation 3. FIG. 7B is a difference image LDif created according to the equation of Equation 3 from 60 enlarged interlaced images different from those in FIG. 6A.

Next, the detection unit 42a creates a vertical profile Ypro that represents the characteristics of the pixel values included in each row of the difference image LDif. _{Specifically, Ypro y} is calculated for y = 1, 2, ..., H-1 according to the following formula. In addition, Ypro is a matrix of (H-1) rows and 1 column, and Ypro _y is a value of the yth row of Ypro.

Next, the detection unit 42a Fourier transforms a data string of (H-1) values included in the vertical profile Ypro to derive a power spectrum. However, at this time, the values from the first line to the H'line included in the vertical profile Ypro are used. H'is the maximum value of the power of 2 below (H-1).

Since the vertical profile Ypro shows the characteristics of the waveform resulting from the striped pattern due to combing noise, the Ypro power spectrum shows peaks corresponding to the period of change in the width of the striped pattern. Therefore, the period C _{p of the} striped pattern can be calculated from the peak position Y peak of the power spectrum of H'and Ypro according to the following equation.

8A and 8B are enlarged views in the square frame shown on the difference image LDif of FIGS. 7A and 7B, respectively. From FIG. 8A, it can be seen that the interval of the striped pattern has a periodicity of 8 lines (that is, C _p = 8). It can also be seen that there are three black or blackish lines in each of the eight lines corresponding to one cycle. The black or blackish lines appearing on the difference image LDif showing the difference between each line of the noise-enhanced image WFDif are the same or substantially the same as the pixel values of the adjacent lines in the noise-enhanced image WFDif. show. In other words, a black or blackish line represents a line in which the combing noise is widened when the original interlaced image is enlarged to the enlarged interlaced image, that is, an additional line. In a normal interlaced image, such black or blackish lines do not exist because the striped pattern changes every other line. Thus, the original number of lines C _org in one cycle, the following equation is calculated by subtracting the number Cnt black or dark row of per period C _p from the cycle C _p. In the example of FIGS. 7A and 8A, _Corg = 8-3 = 5 lines.

On the other hand, from FIGS. 7B and 8B, it is difficult to visually distinguish black or blackish lines. However, C _p and Cnt can be calculated from the difference image LDif. FIG. 9 is a graph of the vertical profile Ypro derived from the difference image LDif of FIG. 7B, and FIG. 10 shows the power spectrum thereof. In the example of FIG. 10, since H'= 512 and the peak position Ypeak = 51, C _p = 512/51 = 10.039 ... From _{the equation of Equation 5, and the period C p} is 10 rows. Is specified. Further, looking at FIG. 9, a downwardly convex portion appears once per cycle of 10 rows. The downwardly convex portion corresponds to a black or blackish line that may be difficult to detect visually. Therefore, in this example, Cnt = 1.

_{From the above, the detection unit 42a calculates the period C p} from the power spectrum of the vertical profile Ypro according to the equation of Equation 5 _{, and further lowers the period C p} from the array of values of the vertical profile Ypro. Calculate the number Cnt of the convex parts. Further, from these C _{p and Cnt, Corg} is calculated according to the formula of Equation 6.

Next, the detection unit 42a calculates _{the vertical enlargement magnification R E} from the original interlaced image to the enlarged interlaced image by the following formula based on _{C p} and Cnt which are parameters representing the striped pattern.

Thus, in the example of FIGS. 7A and 8A, a magnification ratio _{R E = 8/5 = 1.6} . On the other hand, in the example of FIG. 7B and 8B, an enlarged magnification _{R E = 10/9 = 1.111} ··· times.

By the way, FIGS. 7A and 8A are images derived from an enlarged interlaced image enlarged by the nearest neighbor (nearest neighbor interpolation) method, and FIGS. 7B and 8B show one pixel value from a plurality of peripheral pixel values. It is an image derived from an enlarged interlaced image enlarged by the bilinear method, which is one of the interpolation algorithms for determining. In the former Nearest Neighbor method, since the additional line at the time of enlargement is a copy of the adjacent line, C _p and Cnt can be visually specified. On the other hand, in the latter example in which the additional row at the time of enlargement is a mixed value of the pixel values of a plurality of rows as in the bilinear method, it is difficult to visually identify the row. In this regard, the algorithm for detecting the striped pattern by the equations of several 5 to 7 can correspond not only to the nearest neighbor method but also to the enlarged interlaced image enlarged by an algorithm such as the bilinear method.

In the following step S2, the detection unit 42a calculates a parameter for specifying the position of the additional line included _{in the frame F n} which is the enlarged interlaced image based on the result of step S1. This parameter is calculated by the following algorithm.

The detection unit 42a creates a pattern image CP0 assuming combing noise as shown in FIG. 11A. The pattern image CP0 is an image having a striped pattern in which rows having white pixel values and rows having black pixel values are alternately arranged every other row. The parameter for specifying the position of the additional row is calculated by a simulation of generating a special interlaced image (an enlarged image of the interlaced image) from the pattern image CP0. Specifically, the detection unit 42a generates an image obtained by enlarging the pattern image CP0 at magnification R _E in the longitudinal direction by a predetermined interpolation method (magnified pattern image) CP1 (see FIG. 11 (b)). In the example of FIG. 11, the bilinear method is used. In the bilinear method, a new pixel value of each pixel is generated by linear interpolation based on the distance between the pixel and the four peripheral pixels. Therefore, in the enlarged pattern image CP1 of FIG. 11, pixels having an intermediate density between white and black are used. Has appeared. However, as the interpolation method used here, _{the interpolation method when the frame F n} is created should be selected. Therefore, the interpolation method is not limited to the bilinear method, and any method such as the nearest neighbor method or the bicubic method can be selected. You can select the interpolation method. Since the interpolation method when the frame F _n is created is often unknown, the same processing should be performed by various interpolation methods, and the best result should be selected by the user himself or automatically. Can be done.

Further, the detection unit 42a creates a difference image CP2 (see FIG. 11C) representing the difference between adjacent lines of the enlarged pattern image CP1 according to the following equation similar to the equation of Equation 3. FIG. 12 is a graph of pixel values of the difference image CP2. Specifically, according to the following equation, for x = 1, 2, ..., W, y = 1, 2, ..., H-1, each pixel value MDif _{x, included in the difference image CP2} Calculate _y. In addition, MDif _{x, y} is the pixel value of y row x column of the difference image CP2, and N _{x, y} is the pixel value of y row x column of the enlarged pattern image CP1.

Subsequently, the detection unit 42a shifts the difference image CP2 in the vertical direction so that the difference image LDif and the difference image CP2 match the shades of both images most. Then, the shift amount of the difference image CP2 at the time of the most matching is specified as _{the offset value O 1.} In this shift, the enlarged pattern image CP1 which is an enlarged interlaced image created from the pattern image CP0 and the frames F ₁ , F ₂ , ..., F _N which are also enlarged interlaced images are striped in both images. It means that the patterns of are aligned so that they match best.

FIG. 13 shows a state in which the enlarged pattern image CP1 is arranged by shifting it downward by one pixel. Further, FIG. 14 shows a reduced image CP3 obtained by reducing the longitudinal direction at a magnification of the reciprocal of the magnification R _E by the nearest neighbor method each magnified pattern picture CP1 FIG. The detection unit 42a creates such a reduced image CP3, compares each reduced image CP3 with the pattern image CP0 of the same size, and identifies the reduced image CP3 in which the shades of both images match most. The reduced image CP3 is created by the nearest neighbor method in order to prevent the smoothing effect by linear interpolation from further progressing, but another interpolation method can be selected. In the example of FIG. 14, the fifth reduced image CP3 from the left, that is, the reduced image CP3 obtained by reducing the enlarged pattern image CP1 shifted downward by four lines is most similar to the pattern image CP0. The detection unit 42a specifies such a shift amount as an offset value O _2. That is, the offset value O ₂ is a parameter at the time of the reduction process that can restore the original pattern image CP0 most correctly when the pattern image CP0 is enlarged in the vertical direction and then reduced to the original size. be.

The detection unit 42a sets the offset value O = O ₁ + O _{2 obtained by adding the above offset values O 1} and O ₂ to a parameter representing the position of an additional line included in the frames F ₁ , F ₂ , ..., F _N. Identify as. That is, the frames F ₁ , F ₂ , ..., F _N are shifted in the vertical direction by the offset value O, and then deleted when the frames are reduced in the vertical direction by the reciprocal of the _{enlargement magnification R E.} The line that becomes corresponds to the additional line. In this sense, the magnifying power R _E can also be said to be a parameter that specifies the position of the additional row. That is, the enlargement magnification R _E represents one cycle of the insertion pattern of the additional row, and the offset value O represents the start position of this cycle.

In the following step S3, the restoration unit 42b shifts the frames F ₁ , F ₂ , ..., F _N in the vertical direction by the offset value O, respectively, based on the results of steps S1 and S2, and then the nearest neighbor. _{Restored images G 1} , G ₂ , ..., _GN are created by reducing in the vertical direction at the reduction magnification of the reciprocal of the enlargement magnification R _{E by the method.} That is, the restoration unit 42b _{removes from each frame F 1} , F ₂ , ..., F _N the additional line added at least at the time of expansion in the vertical direction to the frame. At this time, the nearest neighbor method is used to prevent the smoothing effect from further progressing, but another interpolation method such as the bilinear method or the bicubic method can be selected.

In step S4, the restored image G _1, G _2, ···, removes blurring of G _N. 15 (a) shows the enlarged pattern image CP1 of FIG. 11 (b), and FIG. 15 (c) shows the pattern image CP0 of FIG. 11 (a). FIG. 15B shows a reduced image CP3 of the fifth enlarged pattern image CP1 from the left, which is most similar to the pattern image CP0 in FIG. It can be said that the reduced image CP3 of FIG. 15B is an image (hereinafter, may be referred to as a bleeding pattern image) representing the pattern of bleeding caused by the enlargement / reduction of the pattern image CP0. FIG. 15 (c) is an ideal state as a restored image, but as is clear from a comparison of FIGS. 15 (b) and (c), the intermediate density generated at the time of enlargement can be eliminated even if the image is reduced. There is no. Therefore, this bleeding is eliminated according to the following algorithm. Here, too, the simulation based on the pattern image CP0 is executed in the same manner as when the offset value O is determined.

Restored image G _1, G _2, · · ·, for each pixel of G _N bleeding is corrected using the pixel values of the pixels of the pixel and its vertically. FIG. 15 (d) is an image obtained by extracting a part of the image shown in the frame of FIGS. 15 (b) and 15 (c). Then, in the figure, correction is performed so that the pixel values of A, B, and C included in the blur pattern image CP3 become the pixel values of D included in the pattern image CP0. Specifically, the correction unit 42c calculates the _{correction coefficients α y} and β _{y according to the following equation.} However, y is the restored image G _1, G _2, · · ·, a vertical coordinate of one cycle (C _org) content of striped pattern of G _N.

The correction unit 42c shifts the frame including the three lines shown in FIGS. 15B and 15C by one pixel, and has correction coefficients α _y , β with _{respect to y = 1, 2, ..., Corg.} Calculate _y. Then, the correction coefficient alpha _y, based on the beta _y, the restored image G _1, G _2, · · ·, calculates a correction value obtained by removing the blur from each pixel value of G _N. Specifically, the restored image G _1, G _2, · · ·, for each pixel of G _N, then the pixel value of the pixel is B, the number 9 pixel value of the upper and lower pixels as A and C Substituting into 1 equation, the obtained D is used as the correction value of the pixel value of the pixel. In this case, the correction coefficient alpha _y used for each pixel, as beta _y, correction coefficient C _org as alpha _y, from the beta _y, based on the offset value O described above, corresponds to the row number of the pixel Α _y and β _y are selected.

In step S5, the restoration unit 42b are restored image G _1, G ₂ corrected by the step S4, · · ·, to create a two field images from each G _N. That is, as in the case of normal deinterlacing, each restored image G _n after correction in step S4 is separated into a _{field image O n} composed of odd-numbered rows and a field image E _{n composed of even-numbered rows.} Since two frames are created from one frame F _n , in the present embodiment, one new timeline is created, and the field images O ₁ and O are assumed to belong to the timeline. _2, ···, O _N and _{E 1, E 2, ···,} E N is stored in the original image area 51. At this time, an odd-numbered line first or an even-numbered line first is determined according to a predetermined setting or by automatic determination, and a new moving image is created. On new timeline, in the case of the odd lines first _{are, O 1, E 1, O} 2, E 2, ···, O N, frames are arranged in the order of E _N, in the case of the even lines first is _{, E 1, O 1, E} 2, O 2, ···, frames are arranged in the order of E _N, O _N.

<1-4. Uses>
The image processing program 2 can handle image processing for a wide variety of still images and moving images, and can also be used, for example, by an organization such as the police to analyze surveillance images of a security camera for investigating an incident. .. In a security camera and a recording device and a playback device connected to the security camera, the interlaced image may be unintentionally enlarged. However, in a situation where such an interlaced image must be analyzed, the above-mentioned special deinterlacing is canceled. Processing is useful.

<1-5. Modification example>
Although the first embodiment has been described above, the present invention is not limited to the above embodiment, and various modifications can be made without departing from the spirit of the present invention. For example, the following changes can be made.

<1-5-1>
Instead of the special deinterlacing process according to the above embodiment, the special deinterlacing process shown in FIG. 16 can be executed. This process is suitable for restoring an enlarged interlaced image obtained by enlarging the original interlaced image by the nearest neighbor method.

First, in step S11, the detection unit 42a performs image processing on the _{frame F n} included in the selected frame group for n = 1, 2, ..., N to obtain duplicate rows included in _{the frame F n.} To detect. The overlapping line referred to here is a line having the same or substantially the same pixel value in adjacent lines. More specifically, the detecting unit 42a according to the following equation, y = 1, 2, · · ·, to the H-1, HDif _n representing the difference between Dat _{n, y + 1} and Dat _{n, y} Create _{, y.}

Note that Dat _{n and y} are matrices representing an array of pixel values in the y-th row of the frame F _n. Then, when the total value of the absolute values of all the elements included in _{HDif n and y is equal to or less than a predetermined value, the (y + 1) line of the frame F n} is determined to be an overlapping line with the y line.

In the following step S12, the restoration unit 42b creates the _{restoration image G n} by removing the overlapping rows detected in the step S11 from the _{frame F n for n = 1, 2, ..., N.} .. In the nearest neighbor method, the image is magnified by inserting an additional row in which the data of the adjacent row is copied as it is at a cycle corresponding to the enlargement magnification. Therefore, the duplicate line removed here is an additional line when the _{original interlaced image is enlarged to the frame F n.} Therefore, the restored image G _n is an image reduced by a reduction magnification that is the reciprocal of the enlargement magnification when the original interlaced image is _{enlarged to the frame F n.} Further, at this time, the pixel value of the latter row may be determined based on the overlapping row to be removed and the row determined to be duplicated (the row not to be removed). This is because even when enlarged by the nearest neighbor method, the additional line does not always exactly match the copy source line due to the influence of noise and the like. Therefore, the effect of noise can be canceled by synthesizing (for example, averaging) the two rows.

Subsequent step S13 is the same as step S5. That is, the restoration unit 42b are restored image G _1, G ₂ by step S12, · · ·, a G _N, to create a two field images constituting each.

<1-5-2>
In the special deinterlacing process according to the first embodiment, the position of the additional line is specified at the stage when step S2 is completed. Thus, while deleting these additional lines, restore omit steps S3 image G _1, G _2, · · ·, without creating a G _N, directly frames F _1, F _2, · · · , F _N can also be deinterlaced.

<2. Second Embodiment>
Hereinafter, the special deinterlacing process according to the second embodiment will be described. Since the hardware configuration of the image processing apparatus that executes this process is the same as that of the first embodiment, the same reference numerals are given to each element in the following description.

<2-1. Special deinterlacing>
Similar to the first embodiment, in the special deinterlacing according to the second embodiment, the enlarged interlaced image is separated into the original two field images. The enlarged interlaced image referred to here is an image in which the original interlaced image is enlarged at least at a magnification in the vertical direction, in other words, an image in which the number of vertical pixels of the original interlaced image is increased according to the magnification (). Including the case where the number of horizontal pixels increases). Therefore, if the magnification at the time of generating the enlarged interlaced image is not known, the original size cannot be restored and the original interlaced image cannot be restored, and the two field images cannot be separated. Further, in the enlarged interlaced image, there is a possibility that the regularity of the alternating arrangement of the two field images before compositing is broken due to the increase in the number of vertical pixels. Therefore, the enlarged interlaced image is a special interlaced image that cannot be deinterlaced by a normal deinterlacing process. The special deinterlacing according to the second embodiment is also a process for deinterlacing such a special interlaced image. Specifically, at least the vertical reduction magnification for restoring the original interlaced image from the enlarged interlaced image is calculated, and a restored image obtained by restoring the enlarged interlaced image to at least the original size in the vertical direction is created, and this restored image is created. Is separated into an odd-numbered row field image and an even-numbered row field image. At this time, the reduction magnification is specified by image processing based on the striped pattern due to combing noise included in the enlarged interlaced image.

Hereinafter, the special deinterlacing algorithm according to the second embodiment will be described in detail with reference to FIG. FIG. 18 is a flowchart showing a flow of processing for deinterlacing the special interlace according to the second embodiment. Similar to the first embodiment, this process is also executed for each frame included in the selected frame group. Further, since this process also requires at least two frames, it is executed only when the selected frame group includes at least two frames.

In the first step S21, as in step S1, the detection unit 42a performs _{image processing on the frames F 1} , F ₂ , ..., F _N (N is an integer of 2 or more) included in the selected frame group. , Detects a striped pattern due to combing noise contained in this. _{Specifically, the detection unit 42a has two frames F n} and F _{n + 1,} which are temporally adjacent to n = 1, 2, ..., N-1, as in step S1. Create a difference image FDif _n. Next, the detection unit 42a creates an image WFDif in which _{the difference images FDif 1} , FDif ₂ , ..., FDif _{N-1 are superimposed.} Further, the detection unit 42a creates a difference image LDif representing the difference between adjacent lines of the noise-enhanced image WFDif. Subsequently, the detection unit 42a creates a vertical profile Ypro that represents the characteristics of the pixel values included in each row of the difference image LDif.

ただし、Ｙｐｗｒは、以下のとおり定義されるパワースペクトルである。

Subsequently, as in step S1, the detection unit 42a Fourier transforms the data sequence of (H-1) values included in the vertical profile Ypro to derive the power spectrum. As mentioned in the description of the first embodiment, the vertical profile Ypro shows the characteristics of the waveform generated from the striped pattern due to the combing noise. Therefore, the Ypro power spectrum shows the period of change in the width of the striped pattern. The corresponding peak appears. In the present embodiment, the detection unit 42a automatically detects the peak position Ypeak from the power spectrum of Ypro according to the following algorithm.

However, Ypwr is a power spectrum defined as follows.

Next, the detection unit 42a calculates _{the period C p of the} striped pattern from the positions Y peak of H and the peak according to the following equation.

Next, the detection unit 42a, based on C _p as a parameter representing the striped pattern, according to the following equation to calculate the vertical reduction ratio R _R from enlarged interlaced image to the original interlaced image. Note that the reduction ratio R _R is a reduction ratio which is updated at the next step S22, in other words, is a temporary reduction ratio.

Considering the equation of Eq. 14, for example, if the original interlaced image is multiplied by 1.1 to generate an enlarged interlaced image, one pixel is added for every 10 pixels, so C _p is 10 + 1 = It is calculated as 11. In this case, according to the equation of Equation 14, 10/11 times is the reduction magnification R _R , but the reciprocal of the enlargement magnification R _E is 1.1 times, which shows that the calculation can be performed correctly. Further, when the original interlaced image is multiplied by 1.5 to generate an enlarged interlaced image, one pixel is added for every two pixels, so C _p is calculated as 2 + 1 = 3. In this case, according to the equation of Equation 14, 2/3 times is the reduction magnification R _R , but the reciprocal of the enlargement magnification R _E is 1.5 times, which shows that the calculation can be performed correctly.

The algorithm for detecting the striped pattern by the equations 13 and 14 is an enlarged interlaced image enlarged by various algorithms such as the nearest neighbor method, the bilinear interpolation method, and the bicubic method. On the other hand, it can be applied.

In step S22, the detection unit 42a is temporary reduction ratio R _R and the frame F _1, F _2, · · ·, based on the H is the number of vertical pixels of F _N, the restored image G _1, G _2, · _··· , Set a _{plurality of candidate pixel number orgs} that are candidates for the number of vertical pixels of GN. _Horg is set according to the following formula. In the following formula, h is a temporary value in the middle of calculation, and [] is a function that rounds down after the decimal point to obtain an integer.

The number of vertical pixels in an interlaced image is usually an even number. Therefore, when h becomes an odd number, h in the lower equation is added / subtracted in conjunction with the addition / subtraction of 1 in the upper equation, and becomes an even number. In addition, 1 is added or subtracted in the above equation because if H _org = [ _{HR R} ] is simply set, H _org may deviate from the true value. Therefore, the detection unit 42a _{calculates two h by adding / subtracting 1 to H / R R} , and calculates the number of two candidate pixels H _org. The value of 1 to be added or subtracted is an experimentally determined value. Then, when an even number exists between these two candidate pixel number H _org , the even number is also set as the candidate pixel number H _org . For example, when the number of vertical pixels H = 100 and _RR = 0.9 of the enlarged interlaced image, h = 89,91. Therefore, the calculated two H _org as 88 and 92, in this example, 90 also becomes H _org sandwiched therebetween.

In step S23, the detection unit 42a, in accordance with the following equation, calculates a plurality of reduction ratio R _R 'respectively corresponding to a plurality of candidate pixel number H _org.

In the following step S24, _{the restored images G 1} , G ₂ , ..., G in which the frames F ₁ , F ₂ , ..., F _N are reduced to the original size for _{each of the plurality of reduction magnifications R R'} Create _N. In the present embodiment, the restoration unit 42b reduces the frames F ₁ , F ₂ , ..., F _N by the bilinear method of linear interpolation.

_{Specifically, in the present embodiment, the frame F n} is reduced for n = 1, 2, ..., N according to the following equation, and the restored image G _n is created. Dst _{i, j} means the pixel value of the restored image G _n in column i, row j (i = 1, 2, ..., WR _R ', j = 1, 2, ..., H. R _R ').

The meaning of each symbol in the above formula is as follows. Note that offset is 0 or -0.5, _{depending on the difference in the algorithm when frames F 1} , F ₂ , ..., F _N are created from the original interlaced image, that is, the starting point of the enlargement calculation. Is a value determined according to whether is the pixel edge or the pixel center. _{In the present embodiment, Dst i and j} are calculated with both the patterns of offset = 0 and offset = −0.5, that is, the restored image G _n is generated.

The algorithm of the bilinear method according to the second embodiment is a modification of a part of the normal bilinear method. _{That is, in the normal bilinear method, the pixel value of the newly generated pixel is added with weights u 0} , u ₁ , v ₀ , v ₁ based on the distance between the pixel and its peripheral pixels, and the pixels of the peripheral pixels are added. It is calculated by adding the values P ₀ , 0, P ₁ , 0, P 0, 1, and P ₁ , _1. However, as it is reflected in the numerical formula 17, in the present embodiment, P _{1, 0,} has become coefficients P _{0, 1} is a negative value.

The effect of restoring the enlarged interlaced image to the original interlaced image by the above algorithm will be described. The image of FIG. 19A is a pattern image assuming combing noise similar to CP0 of FIG. 11A. The image of FIG. 19 (B) is an image obtained by enlarging the pattern image of FIG. 19 (A) at a magnification of 125% by a normal bilinear method and then reducing it to the original size by a normal bilinear method. On the other hand, the image of FIG. 19C was enlarged by the normal bilinear method at a magnification of 125%, and then reduced to the original size by the bilinear method according to the second embodiment. It is an image. As can be seen from the figure, according to the bilinear method according to the second embodiment, although it is not perfect, the original interlaced image can be restored more correctly.

Further, FIG. 20 shows a case where the pattern image of FIG. 19 (A) is enlarged from 1.00 times to 2.00 times by a normal bilinear method at a magnification of 0.01 and then reduced to the original size. , It is a graph of PSNR (peak signal-to-noise ratio) of each reduced image and the original pattern image. FIG. 20 shows two graphs, one is reduced by the normal bilinear method and the other is reduced by the bilinear method according to the second embodiment. According to the figure, it can be seen that the latter case has a larger PSNR, and the larger the magnification is, the more remarkable the tendency is. From this, it can be seen that the original interlaced image can be restored more correctly according to the bilinear method according to the second embodiment.

Tables 1 to 4 below show the pixel values after scaling when the original interlaced image with 1 horizontal pixel is enlarged at a certain magnification and then reduced at the reciprocal of the reduction magnification. The composition ratio of the pixel value is shown. In other words, Tables 1 to 4 below show the degree of agreement between the _{pixel values P 0, j} before scaling in the following equation and the number 17 Dst _{i, j after scaling. Note that Dst 0, j} in the following equation means the pixel value of row j of the image after scaling (j = 1, 2, ..., H). Further, in the following equation, P _{0, j} is the pixel value of row j of the original interlaced image, and v _j is the coefficient thereof.

Table 1 shows the results of enlargement by the ordinary bilinear method at a magnification of 125% and then reduced to the original size by the bilinear method according to the second embodiment. Table 2 shows the results of reduction to the original size by the ordinary bilinear method at the same magnification. After enlarging with, the result of reducing to the original size by the usual bilinear method is shown. On the other hand, Table 3 shows the results of enlargement by the normal bilinear method at a magnification of 175% and then reduced to the original size by the bilinear method according to the second embodiment, and Table 4 shows the results of reduction to the original size at the same magnification. After enlarging by the bilinear method, it is reduced to the original size by the normal bilinear method.

The original number of pixels used in the bilinear method for generating one new pixel value is usually two when the number of horizontal pixels is one. Then, the number of pixels used for adjusting the size to the original size by the bilinear method again is also two pixels. Therefore, the two pixels after scaling are generated from the original two pixels or three pixels, and the composition ratio thereof is shown in Tables 1 to 4. In order to completely restore the original image after scaling, the pixel values of the same pixels must correspond to 1: 1. The two pixel values of are averaged to generate the pixel value after scaling. On the other hand, according to Tables 1 and 3, when the reduction is performed by the bilinear method according to the second embodiment, the pixel values of the same pixels correspond to each other at a ratio of 1: 1 before and after the enlargement / reduction. The tendency is remarkable. Therefore, from this as well, it can be seen that the original interlaced image can be restored more correctly according to the bilinear method according to the second embodiment.

In subsequent step S25, restoring portion 42b, for each combination of a plurality of reduction ratio R _R 'and two Offset value, the restored image G _1, G ₂ step S24, · · ·, 2 sheets each of G _N Create a field image of. This step is executed in the same manner as in step S5. Further, the display control unit 41, recovery with respect to each combination of the plurality of reduction ratio R _R 'and two Offset value image _{G 1, G 2, ···,} G N is displayed on the display 10. The user can visually select the best one from these.

Examples of the present invention will be described below, but the present invention is not limited thereto.

(1) Examples 1 and 2
As Comparative Example 1, FIG. 17B shows the result of performing normal deinterlacing on the enlarged interlaced image (video) shown in FIG. 17A.

Further, the enlarged interlaced image also shown in FIG. 17A was deinterlaced as shown in the flowchart of FIG. FIG. 17C shows the result of the reduction process in step S3, FIG. 17D shows the result of the bleeding correction process in step S4, and FIG. 17E shows the result of deinterlacing in step S5 (Example 1). Shown. Further, FIG. 17F shows the result (Example 2) of performing normal deinterlacing on the image of FIG. 17C.

As is clear from comparison between Example 1 (FIG. 17E) and Example 2 (FIG. 17F) and Comparative Example 1 (FIG. 17B), the original images are correctly restored in each of Examples 1 and 2 with respect to Comparative Example 1. You can see that it is done. Further, as is clear from comparing Example 1 and Example 2, it can be seen that the original image can be restored more accurately by the bleeding correction process in step S4.

(2) Example 3
With respect to the enlarged interlaced image (video) of FIG. 17A, a plurality of reduction magnifications were estimated by the same method as in steps S21 to S24, and a plurality of restored images were restored. Then, the most suitable restored image was visually selected from the plurality of restored images, and deinterlacing was performed. As a result, the result shown in FIG. 21 was obtained. Although the influence of each other's patterns remains on the two field images of FIG. 21, the influence is less than that of Comparative Example 1 of FIG. 17B showing the result of normal deinterlacing, and the second embodiment is related to the second embodiment. It can be seen that the original image can be restored more correctly by deinterlacing the special image.

(3) Examples 4 to 6
In order to confirm the effect of the special deinterlacing according to the second embodiment, six types of images (videos) shown in FIG. 22 were prepared. These images are normal interlaced images that have not been magnified. In FIG. 22, the smallest size image is sample 5 and has 320 × 240 pixels, and the maximum size image is sample 6 and has 640 × 480 pixels. Other images also retain their relative size. Further, the sample 3 is an image in which people appearing small are moving, and the sample 6 is an image in which the camera shakes greatly in the vertical direction.

After enlarging the above six types of images from 1.000 times to 2.000 times at a magnification of 0.001 by the usual bilinear method, a plurality of candidate reductions are performed by the same method as in steps S21 to S23. The magnification was estimated. Since the size of the original image before enlargement is known for samples 1 to 6, it was examined whether or not the correct answer was included in the plurality of candidate reduction magnifications. The results are shown in Table 5 as the correct answer rate. Further, at the time of estimating the reduction magnification, 50 frames were used for each sample to generate the self-weighted average image.

Considering the correct answer rate, the ratio of the correct answers of the reduction ratio included in the candidates is 98.5% to 99.5%, and it can be said that the candidates include almost the correct answers. In addition, when the ratio of the middle value of the candidates being the correct answer was investigated, it was 79.8% to 97.55%, which was also a high value. Accordingly relates step S22, although H _org is described issues that may deviate from the true value, it was found that it is possible to make the intermediate value and the correct answer. If h in the first equation of Equation 15 is an even number, there are two candidates. In this case, the one of the two candidates that is closer to the value before the fractional part is truncated is selected, and the middle one is referred to here. It was set as a value.

Regardless of whether or not the correct answer is included in the reduction magnification candidates, a plurality of restored images were created for the enlarged interlaced image (video) derived from each sample by the same method as in steps S21 to S24 (implementation). Example 4). At the same time, a plurality of restored images were created for the enlarged interlaced image (video) derived from each sample by a normal bilinear method (Comparative Example 4). Then, one frame is randomly selected from these restored images, and the result of evaluating the degree of matching of the patterns before and after scaling by PSNR is shown in Table 5. According to this, an improvement effect (restoration effect) of 1.833 to 5.508 was recognized. When the moving object is small as in sample 3, the difference between Example 4 and Comparative Example 4 is small. On the other hand, it was confirmed that the improvement effect was particularly large in an image such as Sample 6 in which the movement was extremely vigorous. Similar comparisons were made for the images after deinterlacing, but the results were similar.

In FIGS. 23 to 28, the images of the samples 1 to 6 are magnified by the usual bilinear method at magnifications of 1.3 times and 1.6 times, respectively, and the original bilinear method and the normal bilinear method according to the second embodiment are used. The restored image when restored to the size of (Example 5, Comparative Example 5) is shown. Further, as an example, the relationship between the magnification and PSNR in the case of Samples 3 and 6 is shown in FIGS. 29 and 30. As shown in FIGS. 29 and 30, in Sample 3, the effect of the algorithm according to the second embodiment diminishes at a low magnification, but an increase in the effect is confirmed at a high magnification, as in the case of FIG. Further, in sample 6, the effect of the algorithm according to the second embodiment was always confirmed. Further, from FIGS. 23 to 28, it can be visually seen that the algorithm according to the second embodiment is always more effective than the ordinary bilinear method.

FIG. 31 shows an image obtained by deinterlacing the image of FIG. 25 according to the sample 3 (Example 6, Comparative Example 6). As shown in the figure, in Example 6, although the contrast is slightly stronger than that in Comparative Example 6, it was found that there is no particular problem even at a low magnification, and the superiority of the algorithm according to the second embodiment is shown. rice field.

1 Image processing device 2 Image processing program 42a Detection unit (Pattern detection unit)
42b Restoration unit 42c Correction unit (bleeding correction unit)

Claims (16)

An image processing device that restores the original interlaced image from an enlarged interlaced image in which the original interlaced image is enlarged.
By image processing the enlarged interlaced image, a striped pattern due to combing noise included in the magnified interlaced image is detected, and the enlarged interlaced image is reduced to the original interlaced image based on the striped pattern. A pattern detector that calculates the magnification and
An image processing device including a restored image obtained by restoring the original interlaced image based on the reduced magnification and a restoration unit for creating at least one of a field image constituting the restored image. The pattern detection unit
Based on the striped pattern, a temporary reduction magnification from the enlarged interlaced image to the original interlaced image is calculated.
Based on the provisional reduction magnification and the number of vertical pixels of the enlarged interlaced image, a plurality of candidate pixels that are candidates for the number of vertical pixels of the original interlaced image are set.
A plurality of the reduction magnifications corresponding to the plurality of candidate pixels are calculated, and the reduction ratios are calculated.
The restoration unit creates at least one of the restoration image and the field image for each of the plurality of reduction magnifications.
The image processing apparatus according to claim 1. The plurality of candidate pixels are all even numbers.
The image processing apparatus according to claim 2. The restoration unit creates the restoration image by reducing the enlarged interlaced image at the reduction magnification using an algorithm of linear interpolation.
The image processing apparatus according to any one of claims 1 to 3. An image processing device that restores the original interlaced image from an enlarged interlaced image magnified at a certain magnification by adding additional lines to the original interlaced image at predetermined intervals.
By image processing the enlarged interlaced image, a striped pattern due to combing noise included in the magnified interlaced image is detected, and the position of the additional line included in the magnified interlaced image is determined based on the striped pattern. A pattern detector that calculates the parameters that specify
By removing the additional row from the enlarged interlaced image based on the parameter for specifying the position of the additional row, at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image is created. Restoration part and
An image processing device. The parameter that identifies the position of the additional row includes the magnification factor.
The image processing apparatus according to claim 5. The pattern detection unit
The magnified pattern image, which is an image obtained by enlarging an image having a striped pattern every other line at the same magnification as the magnified magnification, and the magnified interlaced image so that the striped patterns of both images match most. Align and
Based on the result of the alignment, a parameter for specifying the position of the additional row is calculated.
The image processing apparatus according to claim 5 or 6. A bleeding correction unit for removing bleeding of the restored image is further provided.
The image processing apparatus according to any one of claims 5 to 7. The bleeding correction unit calculates a correction coefficient from the pixel value included in the bleeding pattern image, and corrects the pixel value of the restored image based on the correction coefficient.
The blur pattern image is an image in which an image having a striped pattern every other line is enlarged at the same magnification as the enlargement magnification and then reduced at a reduction magnification that is the reciprocal of the enlargement magnification.
The image processing apparatus according to claim 8. The striped pattern includes the striped cycle.
The image processing apparatus according to any one of claims 1 to 9. The enlarged interlaced image is a moving image including a plurality of frames.
The pattern detection unit
A noise-enhanced image in which the combing noise is emphasized is created from the plurality of frames, and the noise-enhanced image is created.
A difference image showing the difference between each line of the noise-enhanced image is created.
A vertical profile representing the characteristics of the pixel values included in each line of the difference image is created.
The period of the striped pattern is calculated from the position of the peak of the power spectrum of the vertical profile.
The image processing apparatus according to claim 10. An image processing program that restores the original interlaced image from the enlarged interlaced image in which the original interlaced image is enlarged.
A step of detecting a striped pattern due to combing noise included in the enlarged interlaced image by image processing the enlarged interlaced image, and a step of detecting the striped pattern due to combing noise.
A step of calculating the reduction magnification from the enlarged interlaced image to the original interlaced image based on the striped pattern, and
An image processing program that causes a computer to perform a step of creating at least one of a restored image obtained by restoring the original interlaced image and a field image constituting the restored image based on the reduced magnification. This is an image processing method for restoring the original interlaced image from the enlarged interlaced image in which the original interlaced image is enlarged.
A step of detecting a striped pattern due to combing noise included in the enlarged interlaced image by image processing the enlarged interlaced image, and a step of detecting the striped pattern due to combing noise.
A step of calculating the reduction magnification from the enlarged interlaced image to the original interlaced image based on the striped pattern, and
An image processing method including a step of creating at least one of a restored image obtained by restoring the original interlaced image and a field image constituting the restored image based on the reduced magnification. An image processing program that restores the original interlaced image from an enlarged interlaced image magnified at a certain magnification by adding additional lines to the original interlaced image at predetermined intervals.
A step of detecting a striped pattern due to combing noise included in the enlarged interlaced image by image processing the enlarged interlaced image, and a step of detecting the striped pattern due to combing noise.
A step of calculating a parameter for specifying the position of the additional row included in the enlarged interlaced image based on the striped pattern, and a step of calculating the parameter.
By removing the additional row from the enlarged interlaced image based on the parameter for specifying the position of the additional row, at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image is created. An image processing program that lets your computer perform the steps you want to take. An image processing method for restoring an original interlaced image from an enlarged interlaced image enlarged at a certain magnification by adding additional lines to the original interlaced image at predetermined intervals.
A step of detecting a striped pattern due to combing noise included in the enlarged interlaced image by image processing the enlarged interlaced image, and a step of detecting the striped pattern due to combing noise.
A step of calculating a parameter for specifying the position of the additional row included in the enlarged interlaced image based on the striped pattern, and a step of calculating the parameter.
By removing the additional row from the enlarged interlaced image based on the parameter for specifying the position of the additional row, at least one of the restored image obtained by restoring the original interlaced image and the field image constituting the restored image is created. Image processing methods, including steps to be performed. An image processing device that restores the original interlaced image from an enlarged interlaced image magnified at a certain magnification by adding additional lines to the original interlaced image at predetermined intervals.
By performing image processing on the enlarged interlaced image, a detection unit that detects overlapping rows having the same or substantially the same pixel value included in the enlarged interlaced image, and a detection unit.
An image processing apparatus including a restored image obtained by restoring the original interlaced image and a restoration unit for creating at least one of two field images constituting the restored image by removing the overlapping lines from the enlarged interlaced image. ..

Images