JP7329964B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to image processing technology using optical flow.

In recent years, the importance of techniques for associating pixels between images has increased. The correspondence referred to here is the relationship between the pixels of the reference image that are considered to be the same as the pixels of the image of interest, and can be expressed by the coordinates of two points. When a stereo or multi-viewpoint image is input, the depth of the object can be calculated from the pixel correspondence relationship, so it can be applied to three-dimensional image processing. In addition, if images (moving images) captured in a temporally continuous manner are input and their corresponding relationships are expressed as relative coordinates, they become motion vectors. Obtaining a motion vector in this way is called motion detection. A set of motion vectors obtained by motion detection of a plurality of pixels is called an optical flow (in a narrow sense, information obtained by mapping motion vectors corresponding to pixels is called an optical flow). By using optical flow, it becomes possible to track moving objects, stabilize moving images, frame interpolation, and so on.

Gradient method and template matching are typical methods for obtaining optical flow. The gradient method calculates the optical flow from the direction and magnitude of spatiotemporal luminance change of pixels. The basic idea of the gradient method is to calculate a motion vector from the average spatio-temporal luminance change of pixels in a patch centered on a pixel of interest (hereinafter referred to as a region of interest). Lucas Kanade method (hereinafter referred to as LK method, Non-Patent Document 1) is known as a representative method.

In template matching, a plurality of patches (hereinafter referred to as reference regions) are set at various positions on the reference image for the region of interest, the correlation between the reference regions of the region of interest is calculated, and the relative position of the reference region with the highest correlation is calculated. to get the motion vector. This is called motion search. The optical flow is calculated by performing motion search while scanning the pixel of interest and calculating the motion vector of each pixel. In general template matching, SSD and SAD (Sum of Absolute Difference) are often used for correlation value calculation.

To visualize the optical flow, a motion vector is superimposed on the image as an arrow (Patent Document 1), or the HSV value is calculated with the direction of the motion vector of the pixel as the hue and the length as the saturation, and converted to RGB. There is also a method of generating, recording, and displaying an image (Non-Patent Document 2).

JP-A-2000-23167

B. D. Lucas and T. Kanade (1981), An iterative image registration technique with an application to stereo vision. Proceedings of Imaging Understanding Workshop, pages 121-130 Simon Baker, A Database and Evaluation Methodology for Optical Flow, In Proc. Eleventh IEEE International Conference on Computer Vision (ICCV 2007), Rio de Janeiro, Brazil, October 2007

It was not possible to generate a visualized image of the optical flow that can confirm the correlation between the outline of the optical flow and the shape of the subject.

In order to solve this problem, for example, the image processing apparatus of the present invention has the following configuration. i.e.
An image processing device that generates display image data including a subject,
a first acquisition means for acquiring first image data including a subject;
a second obtaining means for obtaining, as second image data, an optical flow representing a motion vector for each pixel of the first image data;
conversion means for converting the optical flow acquired by the second acquisition means into data representing components of a color image;
synthesizing means for synthesizing the first image data obtained by the first obtaining means and the components obtained by the converting means to generate third image data for display ;
The first obtaining means converts the color image data including the subject into color image data in which the color difference is weakened according to a preset value, and uses the color image data obtained by the conversion as the first image data. Acquired,
The synthesizing means generates synthetic image data from each component value of the first image data acquired by the first acquiring means and the hue and saturation obtained by the converting means. Color image data represented by RGB component values is generated as the third image data .

According to the present invention, it is possible to generate an optical flow visualized image that allows confirmation of the correlation between the outline of the optical flow and the shape of the subject.

1 is a diagram showing an example of a block configuration of an image processing apparatus according to an embodiment; FIG. 4 is a flowchart showing an image processing procedure according to the first embodiment; 8 is a flowchart showing an image processing procedure according to the second embodiment; 9 is a flowchart showing an image processing procedure according to the third embodiment; 13 is a flowchart showing an image processing procedure according to the fourth embodiment; 4A and 4B are diagrams for explaining the contents of image processing according to the first embodiment; FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

[First embodiment]
FIG. 1 is a block configuration diagram of an image processing apparatus that visualizes optical flow according to the first embodiment.

This apparatus has a bus 101 , a CPU 102 , a memory 103 , a non-volatile memory 104 , a display section 108 , an operation section 109 and an image acquisition section 110 . A bus 101 performs data transmission between modules. The CPU 102 is a processor that comprehensively controls each component and executes calculation processing and image processing. The memory 103 functions as a main memory and work area for the CPU 102 . A nonvolatile memory 104 is a memory that stores an image processing program, various data to be processed, parameters, and the like. The CPU 102 executes a program stored in the nonvolatile memory 104 using the memory 103 as a work area, reads out various data stored in the nonvolatile memory 104, and performs image processing, which will be described later. The nonvolatile memory 104 is typically an HDD (hard disk drive), but may be another storage medium. A display unit 108 displays various menus and images that are processing results. An operation unit 109 includes a keyboard, mouse, touch panel, and the like, and receives instruction input from the user. The display unit 108 and the operation unit 109 constitute a user interface. The image acquisition unit 110 acquires an image. When this image processing device is used as a camera, it corresponds to a sensor.

Assume that the nonvolatile memory 104 in the embodiment already stores an image file of one still image and an optical flow file corresponding to the still image (hereinafter referred to as a target image). A still image is, for example, an image captured in a continuous shooting mode of a digital camera. Also, the optical flow is a motion vector in units of pixels obtained from the still image and the temporally disjointed image. A motion vector is represented by horizontal and vertical components.

Image processing executed by the CPU 102 in this embodiment will be described below with reference to the flowchart of FIG. The program shown in FIG. 1 is stored in the nonvolatile memory 104, read out to the memory 103 when executed, and executed by the CPU 102. FIG.

At S<b>2000 , CPU 102 reads a still image of a subject from an image file stored in nonvolatile memory 104 . Then, when the read still image is color image data, the CPU 102 converts it into monochrome image data composed of only one component of lightness (or luminance).

In S<b>2010 , CPU 102 reads an optical flow from the corresponding optical flow file stored in nonvolatile memory 103 . The optical flow may be generated in advance or generated by calling an optical flow generation subroutine as the process of this step.

In S2020, CPU 102 converts the vector corresponding to the pixel of the optical flow into direction and length. This direction is, for example, a counterclockwise angle with the horizontal right direction set to 0 degree. Specifically, it is obtained by calculating arctan{y component value/x component value}. The length can be obtained by obtaining the square root of the sum of the squares of the x component value and y component value of the vector. The CPU 102 performs this process for all vectors of the optical flow.

At S2030, CPU 102 converts the direction and length of the vector obtained at S2020 into hue and saturation in the color space. This is performed for all pixels.

Let v=(vx, vy) be a vector in the optical flow, H be the hue, and S be the saturation. .

In the present embodiment, the hue and its calculation are assumed to be frequencies. However, if V is a zero vector or infinitely close to a zero vector, H is set to NaN (Not A Number). As a condition for judging that the vector is close to the zero vector, it is judged whether or not each of vx and vy is smaller than ε. Although ε is set to 0.01 in this embodiment, there is no limitation to this value. ε may be set to a value sufficiently close to 0, which is equal to or less than the pixel precision of the optical flow. The precision is 1 when performing a pixel-by-pixel motion search. Further, when the gradient method is used, it is possible to calculate to a fractional precision, but when the same subject is generally targeted, it is difficult to obtain a precision of 0.01 pixel or less. and The hue H can be defined as an angle of the hue circle, but the user may be allowed to freely set to which hue the reference angle 0 is assigned.

A coefficient that determines the strength of the saturation S is represented by r. r is determined either so that the length of the largest motion vector in the image does not eventually saturate, or to the range of motion lengths we wish to represent. Specifically, when the maximum vector length on the image is 100 pixels, r=1/100. Alternatively, if the motion length to be expressed ranges from 0 to 10 pixels, then r=1/10. At this time, the saturation of the finally visualized optical flow is saturated.

As a result of the above, the lightness, which is the component value of the monochrome image data obtained from the still image, and the hue and saturation obtained from the optical flow are obtained. In other words, the three components (or attributes) “brightness, hue, and saturation” of each pixel of the color image are aligned.

Therefore, in S2040, the CPU 102 converts the brightness, hue, and saturation into RGB values for each pixel to generate display image data for visualization. The conversion from [brightness, hue, saturation] to "R, G, B" can be performed, for example, by the calculations shown in the following equations (3) to (5).

Here, equations (3) to (5) are equations for converting the HSV values calculated by equations (1) and (2) shown above into RGB values. "mod" in the formula represents a remainder, and in this embodiment, the value after the decimal point is calculated. "clamp" represents a function that returns a value restricted within a closed interval between the first argument, the upper limit and the lower limit, with the second argument as the upper limit and the third argument as the lower limit.

The following formula (6) is a formula for generating the RGB pixel values (R', G', B') of the visualized image by multiplying the RGB values calculated by the above formula by the brightness L of the monochrome image. there is

Equation (6) can be said to represent an equation for superimposing optical flow on a monochrome image. Here, gain and offset are elements that determine the color tone of the visualized image, and in this embodiment, they are set to 1.0 and 0.0, respectively. do not have. The CPU 102 generates image data for display by executing the above formulas (3) to (6) for all pixels.

At S<b>2050 , CPU 102 outputs the display image data generated at S<b>2040 to display unit 108 . For the user, an image is displayed in which the optical flow is visualized and the relationship between the subject of the original image and the optical flow can be intuitively grasped.

As described above, according to the present embodiment, it is possible to generate an optical flow visualized image as if an image expressing motion is superimposed on a monochrome image using optical flow with color difference (hue, saturation). can be done. In this image, unlike images with superimposed arrows or images that visualize simple optical flow, the presence or absence of movement in the details of the subject, and the magnitude and direction of movement can be confirmed. In particular, since motion can be expressed with high density, it can be said that it is suitable for visualizing motion at subject boundaries and motion of small subjects. Conversely, when the optical flow calculation accuracy is low, it is also useful for confirming whether the general shape of the optical flow and the shape of the subject match.

Here, in order to facilitate understanding of the above processing, description will be made with reference to FIG.

Reference numeral 150 in FIG. 6 is an example of a monochrome image (brightness component) of a subject (an airliner in the illustrated case) acquired in S2000 of FIG. Reference numeral 151 denotes an optical flow read from the optical flow file acquired in S2010 of FIG. Reference numeral 152 denotes an image in which the hue and saturation obtained by conversion from the optical flow vector obtained in S2030 of FIG. 2 are visualized. Reference numeral 153 denotes a display image obtained by converting the lightness, hue, and saturation components obtained in S2040 of FIG. 2 into the RGB space. It should be noted that in the patent drawings, color images may be shown as drawings, and the illustrated images themselves may not be displayed.

In addition, although the visualization of the optical flow is represented by the HSV color system in this embodiment, it is not limited to this. ITU-R Recommendation BT. 709, the horizontal and vertical components of the optical flow may be mapped to the luminance, the color difference of the blue component, and the color difference of the red component, respectively, and then synthesized into a monochrome image.

In this embodiment, the processing of each step is performed for each image, but the present invention is not limited to this, and the processing of each step is performed for each pixel, and a series of flows is performed for all pixels. may

In this embodiment, gain and offset are set to 1.0 and 0.0, respectively, but are not limited to this. For example, they may be 0.5 and 0.5, respectively. This indicates that the luminance component of the visualized image does not necessarily have to match the input image as long as the shape of the subject in the image can be discriminated.

In the HSV color system, if the input monochrome image is dark, the generated visualized image will not be bright. By adjusting the gain and offset, the luminance as an HSV value will increase, and the visualized image will become more vivid when visualized. can get. In this embodiment, the direction of the motion vector of the optical flow is represented by hue, but only the speed of motion may be visualized. For this purpose, for example, formulas 3 to 6 can be calculated with H=0.

In the present embodiment, it is assumed that the CPU 102 performs conversion processing and optical flow generation calculated by Equations 1 to 6, but the present invention is not limited to this. For example, a configuration such as that shown in FIG. The conversion unit 105 in FIG. 1B performs calculations of Expressions 1 to 5 such as motion vector imaging and color conversion. The optical flow generation unit 106 generates an optical flow from the acquired image and the temporally preceding or following image. The image synthesizing unit 107 performs calculation processing of Expression 6. Data input/output is the same as the flow described in FIG. Even with such a configuration, a visualized image of optical flow can be generated.

In the present embodiment, a configuration for acquiring an image from the volatile memory 104 has been described, but the present invention is not limited to this. A configuration may be adopted in which a visualized image is displayed on the display unit 108 as a moving image.

[Second embodiment]
Next, a processing procedure in the second embodiment will be described. Unless otherwise specified, the configuration of the device is assumed to be the same as that of the first embodiment, and the description thereof will be omitted.

The flowchart of FIG. 3 shows the processing procedure executed by the CPU 102 in the second embodiment. The difference from FIG. 2 is that S3015 is added after S2010, and the rest is the same as in FIG.

In S3015, CPU 102 generates an image (edge image data) indicating the degree of edge from the monochrome image data. In this embodiment, the square root of the sum of the squares of the horizontal and vertical Sobel filter values is used. The obtained edge image data is an image represented by one component. Therefore, if this edge degree is regarded as lightness, synthetic image data can be generated in the same manner as in the first embodiment. In general, edge image data is assigned a larger value as the degree of edge is higher. Regarding the relationship between the edge degree and the brightness of the edge image data generated in this embodiment, the higher the edge degree, the darker the image data, and the lower the edge degree, the brighter the image data generated. Therefore, if the edge image data is 8 bits, the maximum value is 255, so in the embodiment, "255-edge degree" is generated as the edge image data. As a result, the higher the degree of edge, the more coordinated it becomes as a black line, and the non-edge portion becomes brighter, so that the hue and saturation obtained from the optical flow can be easily perceived as a color. The formula for obtaining the pixel value L' of the edge image from the monochrome image is given by the following formula '(7).

ここで、Ｌsobel_xとＬsobel_yは、着目画素のＳｏｂｅｌフィルタの結果とする。Ｋは、エッジを可視化する際の強度を示す。本実施形態では“２”とするが、ユーザが操作部１０９を介して、最終的な可視化画像の見やすさに応じて設定すればよく、この値に限定されるものではない。

Here, Lsobel _x and Lsobel _y are the result of the Sobel filter of the pixel of interest. K indicates strength in visualizing edges. Although "2" is set in the present embodiment, the user may set it via the operation unit 109 according to the visibility of the final visualized image, and the value is not limited to this value.

At S3040, the pixel values of the hue, saturation, and edge image are converted into RGB values to generate a visualized image. The processing is the same as in the explanation of S2040 in the first embodiment, except that the expression (6) is changed to the following expression (8).

According to the second embodiment, similarly to the first embodiment, it is possible to freeze a visualized image in which the outline of the optical flow matches the shape of the subject. In particular, in the case of a subject with gradation, in the first embodiment, it becomes difficult to determine whether the change in color on the visualized image is due to movement or due to brightness change of the subject itself. On the other hand, according to the second embodiment, the edge of the image is extracted, the edge component is regarded as a monochrome image of brightness, and the optical flow is superimposed thereon to visualize the two easily distinguishable. Images can be generated.

In the second embodiment, a monochrome image is acquired to generate an edge image, but the edge image may be directly input.

In the second embodiment, the Sobel filter is used to generate the edge image, but the present invention is not limited to this, and other edge extraction methods such as the Laplacian filter and the Canny method may be used. Alternatively, RGB values may be generated from an image subjected to edge enhancement processing. For example, if the pixel value of an edge-enhanced image filtered by a 3×3 kernel with 9 at the center and −1 at the others is L″, the RGB values of the visualized image are expressed by the following equation (9 ).

ここで、ｇａｉｎ２とｏｆｆｓｅｔ２は、画像の見え方を調整するパラメータである。本第２の実施形態では、ｇａｉｎ２＝０．４，ｏｆｆｓｅｔ２＝０．５とするが、これに限定されず、ユーザが適宜設定できるようにしても良い。ラプラシアンフィルタの結果、画素値が１を超えている場合、単純に１から０のレンジでクランプすると、強調したエッジが飽和処理により潰れてしまう。本第２の実施形態では、Ｏｆｆｓｅｔ２を加えても１を超えにくい十分小さい値をｇａｉｎ２に設定することにより、強調したエッジがｃｌａｍｐ関数による飽和処理で潰れてないようにしている。ｏｆｆｓｅｔ２を、０に比べれば十分に大きい０．５としたのは、第１の実施形態でも説明したとおり、ＲＧＢ値が暗くなりすぎ、オプティカルフローを重畳したときの鮮やかさが損なわれるのを防ぐためである。

Here, gain2 and offset2 are parameters for adjusting how the image looks. In the second embodiment, gain2=0.4 and offset2=0.5, but the present invention is not limited to this, and the user may set them as appropriate. If the pixel value exceeds 1 as a result of the Laplacian filter, simply clamping in the range from 1 to 0 will crush the emphasized edge due to saturation processing. In the second embodiment, gain2 is set to a sufficiently small value that does not easily exceed 1 even when Offset2 is added, so that the emphasized edge is not crushed by saturation processing using the clamp function. Offset2 is set to 0.5, which is sufficiently large compared to 0. As explained in the first embodiment, it prevents the RGB values from becoming too dark and impairing the vividness when optical flow is superimposed. It's for.

[Third embodiment]
Next, a processing procedure in the third embodiment will be described. Unless otherwise specified, the configuration of the device is assumed to be the same as that of the first embodiment, and the description thereof will be omitted.

The flowchart of FIG. 4 shows the processing procedure executed by the CPU 102 in the third embodiment. The difference from FIG. 2 is that S4015 and S4016 are added after S2010, and the rest is the same as FIG. 2, so the description is omitted.

In S4015, CPU 101 acquires a global motion vector (GMV). GMV is a two-dimensional vector representing the motion of the entire screen. Although the method of calculating GMV is not particularly limited, as an example, a method of calculating also from optical flow will be described. From all motion vectors in the optical flow, a histogram is generated that indicates the frequency of each pixel in horizontal and vertical components. The modes of the histograms of the horizontal component and the vertical component are set as the motion vectors of the horizontal component and the vertical component, respectively, and this is set as GMV.

In S4016, CPU 102 subtracts the GMV obtained above from each motion vector of the optical flow to obtain vector V after correction. In this embodiment, assuming that the motion vector at each pixel position in the optical flow is v', the corrected vector v is expressed by the following equation (10).

Thereafter, processing is performed as an optical flow having the motion vector v after correction.

In an image obtained by panning the camera so as to follow a moving subject, the background moves on the screen and the subject remains relatively still with respect to the angle of view. By subtracting the GMV from the motion vector and executing it for all pixels, it is possible to create an optical flow in which the background portion stops and the subject moves.

Optical flow is generally represented by a relative speed with respect to the screen. According to the third embodiment, by obtaining the GMV and subtracting the GMV from each motion vector of the optical flow, not the relative velocity but the absolute velocity can be obtained. In other words, according to the third embodiment, it is possible to generate an image that visualizes motion with absolute velocity.

In the third embodiment, the color difference and saturation are obtained from the difference vector obtained by subtracting the GMV from the vector represented by the pixel of the optical flow. , can also be applied to the fourth embodiment described below.

[Fourth embodiment]
Next, a processing procedure in the fourth embodiment will be described. Unless otherwise specified, the configuration of the device is assumed to be the same as that of the first embodiment, and the description thereof will be omitted.

The flowchart of FIG. 5 shows the processing procedure executed by the CPU 102 in the fourth embodiment. Compared to the first embodiment (FIG. 2), S2000 is changed to S5000, S2040 is changed to S5040, and S5015 is added after S2010. do.

In the first to third embodiments, a monochrome image represented by a single component or an edge image is used as an image indicating the presence of a subject. The difference is that a color image is used.

At S5010, CPU 102 inputs original color image data showing a subject. Note that the optical flow acquired in the next step S2010 corresponds to this original image.

In S5015, CPU 102 generates second color image data by weakening the color difference component of the original color image data. In this embodiment, the color difference components are weakened by the following equations (11), (12), and (13).

式（１１）は、画素のＲＧＢ値のＹＵＶ値への変換式である。また、式（１２）は、ＹＵＶ値のうち、色差成分であるＵ，Ｖをｄ（ｄ＞１）で割って、色差の強さを弱めている。本実施例では、ｄは１０とする。式（１３）は、ＹＵＶ値をさらにＲＧＢ値に変換する式であり、これをオリジナルのカラー画像データの全画素に対して行うことで、色差成分を弱めた第２のカラー画像が生成される。

Equation (11) is an equation for converting RGB values of a pixel into YUV values. In addition, Equation (12) weakens the strength of the color difference by dividing the color difference components U and V of the YUV values by d (d>1). In this embodiment, d is set to 10. Equation (13) is an equation for further converting the YUV values into RGB values. By performing this on all pixels of the original color image data, a second color image with weakened color difference components is generated. .

In S5040, CPU 102 applies the hue and saturation obtained from the optical flow to the pixel values of the second color image to generate a visualized image for display. Here, the RGB values are calculated using Equation (14) below instead of Equation (6) described in the first embodiment. (R', G', B') in Equation (14) are the pixel values of the visualized image for display. It should be noted that “o” in the formula indicates calculation of the Hadamard product, which is the product of each vector element.

The CPU 102 generates a visualized image composed of RGB values by performing these processes on all pixels.

According to the fourth embodiment, even if the image on which the optical flow is superimposed is not a monochrome image, it is possible to superimpose the optical flow and create a visualized image by weakening the color component. In the fourth embodiment, the YUV color system is used to weaken the color difference components, but the present invention is not limited to this. A configuration may be used in which RGB is converted to the HSV color system, the saturation is weakened, and then returned to RGB.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the claims are appended to make public the scope of the invention.

101 Bus 102 CPU 103 Memory 104 Non-volatile memory 105 Conversion unit 106 Optical flow generation unit 107 Image synthesis unit 107 Image synthesis unit

Claims (9)

An image processing device that generates display image data including a subject,
a first acquisition means for acquiring first image data including a subject;
a second obtaining means for obtaining, as second image data, an optical flow representing a motion vector for each pixel of the first image data;
conversion means for converting the optical flow acquired by the second acquisition means into data representing components of a color image;
synthesizing means for synthesizing the first image data obtained by the first obtaining means and the components obtained by the converting means to generate third image data for display ;
The first obtaining means converts the color image data including the subject into color image data in which the color difference is weakened according to a preset value, and uses the color image data obtained by the conversion as the first image data. Acquired,
The synthesizing means generates synthetic image data from each component value of the first image data acquired by the first acquiring means and the hue and saturation obtained by the converting means. generating color image data represented by RGB component values as the third image data;
An image processing apparatus characterized by: 2. The image processing apparatus according to claim 1, wherein said synthesizing means superimposes said first image data and said second image data. The first acquisition means converts color image data including the subject into monochrome image data composed only of brightness components, and acquires the monochrome image data obtained by the conversion as the first image data. 3. The image processing apparatus according to claim 1 or 2. The first acquisition means converts monochrome image data including the subject into edge image data representing edge strength, and acquires the edge image data obtained by the conversion as the first image data. 3. The image processing device according to claim 1 or 2. 5. The image data of claim 4, wherein the first image data acquired by the first acquisition means is image data having a smaller value as the degree of edge is higher and a larger value as the degree of edge is lower. The described image processing device. The conversion means converts the direction and magnitude of the vector represented by the optical flow into values indicating hue and saturation in a color space,
The synthesizing means generates synthesized image data from the pixel values of the first image data obtained by the first obtaining means and the hue and saturation obtained by the converting means. The image processing apparatus according to any one of claims 1 to 5, wherein a color image represented by RGB component values is generated as the third image data. said transforming means obtains histograms of horizontal and vertical components of all vectors of said optical flow, and extracts each most frequent component value as a component value of a global motion vector;
7. A differential vector obtained by subtracting the global motion vector from a vector corresponding to each pixel of the optical flow is converted into data representing components of the color image. 1. The image processing apparatus according to claim 1. A control method for an image processing device that generates display image data including a subject, comprising:
a first obtaining step in which the first obtaining means obtains first image data including the subject;
a second acquiring step of acquiring, as second image data, an optical flow representing a motion vector for each pixel of the first image data;
a conversion step in which the conversion means converts the optical flow acquired in the second acquisition step into data representing components of a color image;
a synthesizing step of synthesizing the first image data obtained in the first obtaining step and the components obtained in the converting step to generate third image data for display; have
In the first obtaining step, the color image data including the subject is converted into color image data in which the color difference is weakened according to a preset value, and the color image data obtained by the conversion is used as the first image data. Acquired,
In the synthesizing step, synthesized image data is generated from each component value of the first image data obtained in the first obtaining step and the hue and saturation obtained in the converting step, and from the synthesized image data generating color image data represented by RGB component values as the third image data;
A control method for an image processing apparatus, characterized by: A program that is read and executed by a computer to cause the computer to perform each step of the method according to claim 8 .

Images