JP5874466B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention provides an image processing apparatus, an image processing, and an image processing apparatus capable of reducing the load of frame interpolation processing and improving the image quality of a user's attention portion when simultaneously displaying a plurality of moving images in which frames are interpolated. The present invention relates to a method and a program.

  Conventionally, as a method of reproducing a moving image such as an animation, there is a method of generating moving image data by interpolating frame data between a plurality of key frame data and reproducing the moving image. (See Patent Document 1).

  A plurality of such moving images can be arranged in parallel on the same screen and displayed simultaneously.

JP 2002-074382 A

However, performing frame interpolation processing for each of a plurality of moving images places a heavy load on the image processing apparatus, and therefore, it is necessary to reduce the frame rate by that amount to generate interpolation frame data. A decrease in the data generation speed (frame rate) of the interpolated frame leads to a decrease in the moving image reproduction speed (frame rate), and a decrease in the moving image reproduction speed causes a decrease in image quality.
For this reason, it is demanded that at least the user's attention portion of the displayed image has a high image quality, but the conventional methods including Patent Document 1 do not sufficiently meet such a demand.

  The present invention has been made in view of such a situation, and when simultaneously displaying a plurality of moving images in which frames are interpolated, the load of frame interpolation processing is reduced without degrading the image quality of the user's attention portion. The purpose is to reduce.

In order to achieve the above object, an image processing apparatus according to one embodiment of the present invention includes a plurality of key frame data as display targets of a plurality of planes arranged in a three-dimensional space including a predetermined display area, Moving image generating means for generating moving image data composed of a plurality of frames of data interpolated between key frames, and the predetermined display area for the plurality of moving image data generated by the moving image generating means In a three-dimensional space in which the horizontal axis is the X axis, the vertical axis is the Y axis, and the vertical axis to the X and Y axes is the Z axis, the vertical direction in the three-dimensional space in it, based on the arrangement positions of the plurality of planes in the Z-axis, and the priority determining means for determining a priority based on the priority determined by the priority determining means, the moving image generation hands A frame rate setting means for individually setting a frame rate when each of the plurality of moving image data is generated, and the moving image generating means at each of the frame rates individually set by the frame rate setting means By combining the data of the plurality of moving images generated by the above, a combining unit that generates display image data in which the plurality of planes are arranged in the predetermined display area, and the data generated by the combining unit And display control means for controlling display of the display image.

  According to the present invention, when a plurality of moving images in which frames are interpolated are displayed at the same time, it is possible to reduce the load of frame interpolation processing and improve the image quality of the portion of interest of the user.

1 is a block diagram illustrating a hardware configuration of an image processing apparatus according to an embodiment of the present invention. It is a functional block diagram which shows the functional structure for performing a plain moving image synthetic | combination display process among the functional structures of the image processing apparatus of FIG. It is a figure which shows the positional relationship of the whole drawing area | region equivalent to the whole screen of the display part of the image processing apparatus of FIG. 2, and a plain moving image. It is a figure which shows an example of the key frame of a plain moving image. 3 is a timing chart illustrating an example of generation timing of each data of a first plane moving image to a third plane moving image when the image processing apparatus of FIG. 2 executes a plain moving image combining display process. FIG. 6 shows an example of a display image in which the first to third plane videos in FIG. 5 are arranged in that order. 3 is a flowchart for explaining the flow of a plain video composition display process executed by the image processing apparatus of FIG. 1 having the functional configuration of FIG. 2.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a hardware configuration of an image processing apparatus according to an embodiment of the present invention. The image processing apparatus 1 is configured as a personal computer, for example.

  The image processing apparatus 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an input unit 16, and an output unit. 17, a storage unit 18, a communication unit 19, and a drive 20.

The CPU 11 executes various processes according to a program recorded in the ROM 12 or a program loaded from the storage unit 18 to the RAM 13.
The RAM 13 appropriately stores data necessary for the CPU 11 to execute various processes.

  The CPU 11, ROM 12, and RAM 13 are connected to each other via a bus 14. An input / output interface 15 is also connected to the bus 14. An input unit 16, an output unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

The input unit 16 includes a keyboard, a mouse, and the like, and inputs various types of information according to user instruction operations.
The output unit 17 includes a display (display unit 51 in FIG. 2 described later), a speaker, and the like, and outputs images and sounds.
The storage unit 18 is composed of a hard disk, a DRAM (Dynamic Random Access Memory), or the like, and stores various image data.
The communication unit 29 controls communication performed with another device (not shown) via a network including the Internet.

  A removable medium 31 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately attached to the drive 20. The program read from the removable medium 31 by the drive 20 is installed in the storage unit 18 as necessary. The removable medium 31 can also store various data such as image data stored in the storage unit 18 in the same manner as the storage unit 18.

FIG. 2 is a functional block diagram showing a functional configuration for executing the plain video composition display processing among the functional configurations of such an image processing apparatus 1.
The plain moving image composition display processing refers to a series of processing until N (N is an integer value of 2 or more) plain moving images are arranged on the display screen and simultaneously displayed. The plain video will be described later.

When the plain video composition display processing is executed, in the CPU 11, the plain video generation units 41-1 to 41-N, the plain video priority determination unit 42, the frame rate setting units 43-1 to 43-N, The synthesizing unit 44 and the display control unit 45 function. Further, the display unit 51 functions in the output unit 17.
In the following, each of the N plane videos to be displayed in the plain video composite display process is uniquely assigned a number from 1 to N, and the plane video with the number k is referred to as “kth plane video”. "

The plane moving image generating unit 41-k generates k-th plane moving image data in units of unit images.
Here, the unit image refers to an image that is a processing unit of a moving image, and a field or a frame is an example. In the present embodiment, the unit image is a frame and will be described below.
The plain moving image is a moving image that is configured as, for example, a CG (Computer Graphics) animation and includes a character that performs a predetermined action such as dance as a moving object. This moving object is included and displayed on a predetermined plane.
The plane image on which the moving object is arranged in this way is hereinafter referred to as a “plane image”. This plain image is adopted as a frame (hereinafter, such a frame is referred to as a “plane frame”), and a moving image composed of a plurality of plane frames is a “plane moving image” in this specification.
Further, with reference to FIGS. 3 and 4, the plane frame and the plain moving image will be described in detail.

FIG. 3 is a diagram illustrating a positional relationship between the entire drawing area corresponding to the entire display screen and the plain moving image.
Of the plane frames constituting the plane video DP, more precisely, the plane frames constituting the plane video DP, the plane frame to be displayed can be arranged at an arbitrary position in the entire drawing area M corresponding to the entire screen of the display unit 51. ing.
In FIG. 3, in the entire drawing area M, the horizontal axis is the X axis, the vertical axis is the Y axis, and the direction perpendicular to the X and Y axes (the depth direction in the entire drawing area M) is the Z axis. A virtual three-dimensional space is constructed (defined). Although the origin may be arbitrary, in the present embodiment, it is assumed that it is set at the lower left corner of the entire drawing area M.
In the example of FIG. 3, the plain video DP is arranged so that the coordinates (Xp, Yp, Zp) are centered in the entire drawing region M having the horizontal length Wa and the vertical length Ha. Yes. That is, the plain video DP is located at an arbitrary position in the virtual three-dimensional space including not only the two-dimensional direction (X direction and Y direction) of the entire drawing area M but also the depth direction (Z direction) of the entire drawing area M. The position can be changed, and the position can be changed with time.
Accordingly, the position of the character in this plain video DP can also be defined by the coordinates (X, Y, Z) in this virtual three-dimensional space. In addition, the movement of the character includes the temporal transition of the coordinates (X, Y, Z) and rotation angles θpz, θpy, θpz (hereinafter, information on these rotation angles) with respect to the X, Y, and Z axes. Are collectively referred to as “rotation information”).
However, such movement (change) of the character is not defined in all the plane frames constituting the plain video DP, but is defined only in a so-called key frame. Based on these definitions, a plurality of plane frames are interpolated between the key frames.

FIG. 4 shows an example of a key frame of a plain moving image.
The movement of the character is defined by three-dimensional coordinates (X, Y, Z) such as a plurality of end points and joint points, rotation information, and the like in each key frame. In the example of FIG. 4, the head end point nh, neck joint point, waist joint point nw, left elbow joint point, left hand end point, right elbow joint point, right hand end point, left knee joint point, left foot joint point Characters (dancers) are defined by three-dimensional coordinates (X, Y, Z), rotation information, and the like for each of the end point nll, the right knee joint point, and the right foot end point nlr.

  Also, the plain video DP itself can freely move and rotate in the three-dimensional direction with respect to the entire drawing area M in FIG. 3 as described above, and information that enables such movement and rotation is also available. Defined in keyframe.

  Returning to FIG. 2, each of the plane moving image generation units 41-1 to 41-N generates data of each of the first plane moving image to the Nth plane moving image in units of plane frames, and the plane moving image priority determination unit 42 To provide.

The plain video priority determination unit 42 determines the priority for ensuring display quality (to be displayed with high quality) for each of the first plane video to the Nth plane video.
Each of the frame rate setting units 43-1 to 43-N sets the frame rate of each of the first plane moving image to the Nth plane moving image based on each priority determined by the plane moving image priority determining unit 42. . That is, the frame rate of the highest priority plane moving image among the first plane moving image to the Nth plane moving image is set to the highest, and thereafter, the frame rate of the plain moving image is set to sequentially decrease in order of priority.
Each of the plane moving image generation units 41-1 to 41-N has data of each of the first plane moving image to the Nth plane moving image at each frame rate set by each of the frame rate setting units 43-1 to 43-N. Are generated in units of plane frames and provided to the combining unit 44.
The synthesizing unit 44 synthesizes the data of each of the first plane video to the Nth plane video in units of plane frames, so that the entire display screen in which each of the first plane video to the Nth plane video is arranged in parallel (FIG. 3) (hereinafter referred to as “display image data”) is generated and supplied to the display control unit 45.
The display control unit 45 performs control to display the display image on the display unit 51 in units of frames.

  Hereinafter, the plain moving image composition display processing executed by the image processing apparatus 1 having the functional configuration of FIG. 2 will be described with reference to FIGS. 5 to 7 as appropriate.

  As described above, the frame rate can be individually set for each data of the first plane video to the Nth plane video. In this case, the display frame rate for the entire display image in which each of the first plane video to the Nth plane video is arranged in parallel is the least common multiple of each frame rate.

  In addition, when updating the frame of the display image, there may be a case where interpolated plane frame data (hereinafter referred to as “interpolated data”) is not generated for a low-frame-rate plain moving image. In such a case, the interpolating data generated last time is reused in the synthesizing unit 44 to generate frame data of the display image.

As described above, since the frame rate of each data of the first plane video to the N-th plane video can be set individually, the user operates the input unit 16 (FIG. 1), so that the frame rate of a specific plane video is obtained. It is possible to instruct the setting to increase the frame rate and decrease the frame rate of other plain moving images.
For example, when a display image in which a first plane video to an N-th plane video including each of N characters who dance is displayed on the display unit 51 is displayed, the user The setting can be instructed. That is, the user can instruct the setting of increasing the frame rate of the plain video including the main dancer by tailoring a specific character as the main dancer. On the other hand, the user can tailor other characters as a back dancer and instruct the setting to lower the frame rate of a plain video including the back dancer.
In this case, the plain video priority determination unit 42 determines the priority of each data of the first plane video to the N-th plane video based on the user's setting instruction. Each of the frame rate setting units 43-1 to 43-N sets the frame rate of each of the data of the first plane moving image to the Nth plane moving image based on each priority determined by the plane moving image priority determining unit 42. Set.
As a result, in the entire display image in which each of the first plane video to the N-th plane video is arranged in parallel, the plane video with a high degree of user's attention (high priority) is displayed with high image quality. The plain video is displayed with slightly reduced image quality. As a result, it is possible to reduce the load of the frame interpolation processing while maintaining high image quality for the user's attention portion in the display image.

However, depending on the action of the character, the plain video that the user is interested in, that is, the plain video that is desired to be displayed with high image quality, may change over time. In the first plane video to the Nth plane video including each of the N characters who dance, the standing positions of the N characters (for example, the depth direction (Z direction) and the position in the X axis direction) sequentially change, etc. This is the case when the main dancer changes sequentially.
In such a case, the plain video composition display processing is executed as follows.

First, the plane moving image priority determination unit 42 determines the highest priority among the data of the first to Nth plane moving images.
Specifically, in the present embodiment, among the first plane video to the Nth plane video, the highest priority is given based on the information of the plane frame data (interpolation data) to be displayed (target generated most recently). The plain video data is determined.

More specifically, the plain video priority determination unit 42 determines the highest priority plane video data according to the following first to fourth rules.
(Rule 1)
Among the data of the first plane video to the Nth plane video, the data placed at the forefront position, that is, the highest position in the vertical direction with respect to the entire drawing area M (the Z coordinate is the highest position) is given the highest priority. .
(Rule 2)
When there are a plurality of data corresponding to the first rule, the one near the center of the display screen (the entire drawing area M in the example of FIG. 3) in the X direction (horizontal direction) is given the highest priority.
(Rule 3)
When there are a plurality of data corresponding to the second rule, the data having the largest character size (occupied area in the plane) is given the highest priority.
(Rule 4)
When there are a plurality of data corresponding to the third rule, any one of them is given the highest priority.

Regarding the first rule, the plane moving image priority determination unit 42 uses the following formula (1) for each piece of plane frame data (interpolation data) to be displayed among the first to Nth plane moving images. Are calculated, and the one with the largest calculation result is determined as the data of the highest priority plain moving image.
[Equation 1]
Z-coordinate value of the plain frame (position in the depth direction) + Z-coordinate value of the character's hip joint point (1)
Specifically, for example, in the example of FIGS. 3 and 4, the one with the maximum calculation result of the following equation (2) is determined as the data of the highest priority plane moving image.
[Equation 2]
Zp + Zw × cos θpy (2)

Regarding the second rule, the plane video priority determination unit 42 has the highest priority according to each X coordinate of plane frame data (interpolation data) to be displayed among the first plane video to the Nth plane video. Determine the plain video data.
Specifically, in the example of FIGS. 3 and 4, for example, the calculation result of the following expression (3) is closest to half the length Wa (= Wa / 2) in the X direction of the entire drawing area M. It is determined as the highest priority plain video data.
[Equation 3]
Xp + (Xw−Wp / 2) × cos θpy (3)

Regarding the third rule, the plane moving image priority determination unit 42 selects the end point of the foot from the end point of the character included in the plane frame to be displayed among the first to N-th plane moving images (separate left and right). Lengths H1 and H2 are calculated, and the one having the maximum value among the lengths H1 and H2 is determined as the data of the highest priority plane moving image.
Specifically, for example, in the example of FIGS. 3 and 4, the largest of the lengths H1 and H2 calculated according to the following equation (4) is determined as the data with the highest priority plane moving image.
[Equation 4]
H1 = Yh-Ylr
H2 = Yh-Ylr
... (4)
In the third rule, since the determination is based on the character size, plane rotation is not considered. The lengths H1 and H2 described above are not accurate as an index for character size comparison, but considering that the third rule is an assistant to the first rule and the second rule, Such simplification is preferable because the determination process becomes simple.

Here, as a specific example, consider the case where N = 3, that is, the case where the data of the first plane video to the third plane video are combined to generate display image data.
It is assumed that interpolation data is generated when the frame rate of the highest priority plain moving image data is 20 fps and the frame rate of other plain moving image data is 5 fps. In this case, the display frame rate of the entire display image in which each of the first plane video to the third plane video is arranged is the least common multiple of 20 fps and 5 fps, that is, 20 fps.

FIG. 5 is a timing chart showing an example of the generation timing of each data of the first plane video to the third plane video in this case.
Specifically, timing charts for each of the first plane video DP1, the second plane video DP2, and the third plane video DP3 are shown in order from the top in FIG.
A band (large rectangle) indicating each of the first plane video DP1, the second plane video DP2, and the third plane video DP3 is divided into a plurality of squares (small rectangles). Is shown.

  FIG. 6 shows an example of a display image in which the first to third plane videos are arranged in that order.

FIG. 6A shows an example of a display image 50 ms before time t0 in FIG.
Assume that at the timing when the display image of FIG. 6A is displayed, the plane moving image priority determination unit 42 determines that the data of the first plane moving image has the highest priority.

  In this case, at time t0 after 50 ms, the frame rate setting unit 43-1 sets the frame rate of the first plane video D1 having the highest priority to 20 fps. On the other hand, each of the frame rate setting units 43-2 and 43-3 sets the frame rates of the second plane moving image D2 and the third plane moving image D3 to 5 fps, respectively.

The plane moving image generating unit 41-1 sequentially generates the data (interpolation data) of each plane frame constituting the first plane moving image D1 having the highest priority at a frame rate of 20 fps, that is, every 50 ms, and supplies it to the combining unit 44. .
On the other hand, each of the plane moving image generation units 41-2 and 41-3 generates data (interpolation data) of each plane frame constituting each of the second plane moving image D2 and the third plane moving image D3 at a frame rate of 5 fps. Sequentially generated every 200 ms and supplied to the synthesis unit 44.

  The synthesizing unit 44 generates display image data by synthesizing the data (interpolation data) of each plane frame of the first plane video D1, the second plane video D2, and the third plane video D3 every 50 ms. To do.

Specifically, as shown in FIG. 5, at time t0, the interpolation data of each of the first plane video D1, the second plane video D2, and the third plane video D3 is newly generated, and these new interpolations are performed. The data are combined to generate display image data.
At each time of 50 ms intervals up to time t1 thereafter (one scale on the time axis in FIG. 5), interpolation data of the first-priority first plane video D1 is newly generated at each time (every 50 ms) It is used for generating display image data by the combining unit 44. On the other hand, for the second plane video D2 and the third plane video D3, no new interpolation data is generated, so that the interpolation data generated at time t0 is repeatedly used for generation of display image data by the combining unit 44.
Then, only at the time t1 when 200 ms elapses from the time t0, not only the first-priority first plane video D1 but also the interpolation data of the second plane video D2 and the third plane video D3 are newly generated. These new interpolation data are combined in the combining unit 44 to generate display image data.

  That is, during the period from time t0 to time t1, the display image itself displayed on the entire display unit 51 under the control of the display control unit 45 is updated at 20 fps (that is, every 50 ms). However, in the display image, the portion of the display image that is actually updated every 50 ms is only the top-priority first plane video D1, and the time t0 for the other second plane video D2 and third plane video D3. The display is not updated until 200 ms elapses and time t1 is reached. That is, the display of the second plane video D2 and the third plane video D3 is updated every 200 ms.

During this time, the plane video priority determination unit 42 obtains the data of the highest priority plane video from the data (interpolation data) of each plane frame of the first plane video D1 to the third plane video D3 every 50 ms. Determine sequentially.
Each time 50 ms elapses after the plain video priority determination unit 42 sequentially determines the highest priority plane video data, each of the frame rate setting units 43-1 to 43-3 has a plane video priority 50 ms before. Based on each priority determined by the determination unit 42, the frame rate of each of the first plane video D1 to the third plane video D3 is set.

Specifically, in the example of FIG. 5, between the time t0 and the time t1, the plane video priority determination unit 42 determines that the first plane video D1 has the highest priority every 50 ms. The
Therefore, in the time zone T1 from 50 ms after the time t0 to the time tx (until 50 ms after the time t1), the frame rate setting unit 43-1 makes a frame of the first-priority first plane video D1 every 50 ms. The rate is set to 20 fps each time, and the frame rates of the second plane video D2 and the third plane video D3 are set to 5 fps each time by the frame rate setting units 43-2 and 43-3, respectively. .

FIG. 6B shows an example of a display image at the timing of time tx in FIG.
At the time tx when the display image of FIG. 6B is displayed, the situation has changed from when the display image of FIG. 6A is displayed. It is assumed that the data of the moving image D2 is determined to have the highest priority. That is, at the timing of time tx, the highest priority plane video is changed from the first plane video D1 to the second plane video D2.

  In this case, in the time zone T2 after time ty (50 ms after time tx), the frame rate of the second plane video D2 with the highest priority is set to 20 fps every 50 ms by the frame rate setting unit 43-2. In addition, each frame rate of the first plane moving image D1 and the third plane moving image D3 is set to 5 fps by the frame rate setting units 43-1 and 43-3, respectively.

Then, at each time of 50 ms interval after time tx (at 50 fps), the interpolation data of the second-priority moving image D2 with the highest priority is newly generated at each time, and the display image data of the combining unit 44 is displayed. Used for generation. On the other hand, since no new interpolation data is generated for the first plane video D1 and the third plane video D3, the interpolation data generated last time is used to generate display image data in the repetitive synthesis unit 44.
Then, only the second-priority second plane video D2 having the highest priority at the timing when 200 ms has elapsed from the time t1 when all the new interpolation data of the first plane video D1 to the third plane video D3 was previously generated, that is, the timing at the time t2. In addition, the interpolation data of each of the first plane video D1 and the third plane video D3 is also newly generated, and these new interpolation data are respectively combined by the combining unit 44 to generate display image data. The

  That is, in the time zone T2 after the time tx, the display image itself displayed on the entire display unit 51 under the control of the display control unit 45 is updated at 20 fps (that is, every 50 ms). However, in the display image, the portion of the display image that is actually updated every 50 ms is only the second plane video D2 with the highest priority, and the other first plane video D1 and third plane video D3 are 200 ms. The display is updated at each time t2, t3.

Further, with reference to FIG. 7, the flow of the plain video composition display process will be described below.
FIG. 7 is a flowchart for explaining an example of the flow of the plain video composition display process executed by the image processing apparatus 1 of FIG. 1 having the functional configuration of FIG.

  In step S <b> 1, the plain video priority determination unit 42 determines the priority of each of the data of the first plane video to the N-th plane video arranged in parallel in the display image.

  In step S2, each of the frame rate setting units 43-1 to 43-N sets the frame rate of each of the first plane video to the Nth plane video based on each priority determined in the process of step S1. To do.

In step S3, each of the plane moving image generation units 41-1 to 41-N performs the data (interpolation data) of each plane frame of the first plane moving image to the Nth plane moving image at each frame rate set in the process of step S2. ) Is generated.
However, since each frame rate is set individually, it may be different. In such a case, the plane frame data is generated every time for the plane moving image in which the highest frame rate is set. On the other hand, for plain video with a lower frame rate, new interpolation data is generated if the time corresponding to the frame rate has not elapsed since the previous interpolation data was generated. Instead, the previous interpolation data is used.

  In step S4, the synthesis unit 44 synthesizes the data (interpolation data) of each plane frame of the first plane video to the Nth plane video, so that each of the first plane video to the Nth plane video is arranged in parallel. Display image data, more precisely, display image frame data is generated.

  In step S5, the display control unit 45 causes the display unit 51 to display the frame of the display image.

In step S6, the plain video priority determination unit 42 and the like determine whether or not an instruction to end the process has been issued.
If the end of the process has not been instructed yet, it is determined as NO in step S5, and the process returns to step S1. That is, until the end of the process is instructed, the processes of steps S1 to S6 are repeatedly executed, and the frames of the display image displayed on the display unit 51 are sequentially updated.
Thereafter, when the end of the process is instructed, it is determined as YES in step S6, and the plain moving image composition display process is ended.

As described above, the image processing apparatus 1 includes the plain moving image generation units 41-1 to 41-N, the plain moving image priority determination unit 42, the frame rate setting units 43-1 to 43-N, and the combining unit 44. And a display control unit 45 (see FIG. 2).
The plane moving image generating units 41-1 to 41-N have a plurality of planes as display targets of a plurality of planes arranged in a three-dimensional space including a predetermined display area (the entire drawing area M in FIG. 3 in this embodiment). Data of the first to N-th plane moving images, each of which is composed of the data of the key frame (plane frame) and a plurality of plane frame data (interpolated data) to be interpolated between the key frames.
The plane video priority determination unit 42 prioritizes the display quality for each data of the first plane video to the Nth plane video based on the arrangement positions of a plurality of planes in the vertical direction with respect to a predetermined display area. To decide.
The frame rate setting units 43-1 to 43-N individually set the frame rate at which each data of the first plane moving image to the Nth plane moving image is generated based on the determined priority.
The synthesizing unit 44 synthesizes the data of the first plane video to the N-th plane video generated at each individually set frame rate, and more precisely synthesizes the data of each plane frame. Display image data in which a plurality of planes are arranged in a predetermined display area is generated.
The display control unit 45 controls display of the display image based on the data generated by the synthesis unit 44.

As described above, the frame rate at the time of generating plain video data (at the time of generating interpolation data) is not uniform for all data, and the higher the priority, the higher the frame rate. As a result, the processing load on the image processing apparatus 1 is reduced as compared with the case where all the plain video data is uniformly generated at a high frame rate.
The priority is set based on, for example, the first rule described above based on the arrangement positions of a plurality of planes in the vertical direction with respect to the predetermined display area. Here, the predetermined display area and the vertical direction correspond to the depth direction of the display screen. In addition, the user has a feature that, for example, a plane existing on the front surface (front side) is easier to focus on. Since the priority is set based on this feature, it is related to the degree of attention of the user. For this reason, the higher the priority, the higher the frame rate means that the higher the frame rate is, the more the image quality is improved.
Accordingly, it is possible to reduce the load of the frame interpolation process (interpolation data generation process) and improve the image quality of the user's attention part.

  In addition, this invention is not limited to the above-mentioned embodiment, The deformation | transformation in the range which can achieve the objective of this invention, improvement, etc. are included in this invention.

For example, in the above-described embodiment, as a priority determination method, a method of sequentially applying and determining in order of the first rule to the fourth rule using a set of the first rule to the fourth rule is employed. However, it is not particularly limited to this.
Any combination of arbitrary types of the first rule to the fourth rule may be used as one set, or any one of the first rule to the fourth rule may be used alone.
Furthermore, a rule different from the first rule to the fourth rule may be used. Further, in this case, only another rule may be used, or another rule and any combination of any kind of the first rule to the fourth rule may be used as one set.

For example, the synthesis unit 44
Generating display image data in which each of the plurality of planes is arranged in parallel in a predetermined direction of a predetermined display area (the entire drawing area M in the above-described embodiment);
The plane video priority determination unit 42 determines the priority based on the arrangement position of each of the plurality of planes in a predetermined direction with reference to the center position of the predetermined display area.
You may do it.
The user has a feature that it is easy to focus on the center position of a predetermined display area (for example, the entire display screen). Since the priority is set based on this feature, it is related to the degree of attention of the user. For this reason, the higher the priority, the higher the frame rate means that the higher the frame rate is, the more the image quality is improved.
Accordingly, it is possible to reduce the load of the frame interpolation process (interpolation data generation process) and improve the image quality of the user's attention part.
Here, if the X direction (FIG. 3) is adopted as the predetermined direction, it becomes a priority determination method according to the second rule described above.

Also, for example, the plain video generation units 41-1 to 41-N
As a display target for each of a plurality of planes, generate a plurality of moving image data including moving objects,
The plane video priority determination unit 42 further determines the priority based on the occupied area in the plane of the moving object.
You may do it.
The user has a feature that the larger the moving object in the plane, the easier it is to pay attention. Since the priority is set based on this feature, it is related to the degree of attention of the user. For this reason, the higher the priority, the higher the frame rate means that the higher the frame is, the higher the frame rate is, so that the higher the priority is, the higher the image quality is.
Accordingly, it is possible to reduce the load of the frame interpolation process (interpolation data generation process) and improve the image quality of the user's attention part.
Here, if a dancing character or the like is employed as the moving object, the priority determination method according to the third rule described above is used.

In the above-described embodiment, the image processing apparatus 1 to which the present invention is applied has been described using a personal computer as an example, but is not particularly limited thereto.
For example, the present invention can be applied to general electronic devices having a moving image display control function. Specifically, for example, the present invention is applicable to a digital camera, a notebook personal computer, a printer, a television receiver, a video camera, a portable navigation device, a smartphone, a mobile phone, a portable game machine, and the like.

The series of processes described above can be executed by hardware or can be executed by software.
In other words, the functional configuration of FIG. 2 is merely an example and is not particularly limited. That is, it is sufficient that the image processing apparatus 1 has a function capable of executing the above-described series of processing as a whole, and what functional blocks are used to realize this function is not particularly limited to the example of FIG. .
In addition, one functional block may be constituted by hardware alone, software alone, or a combination thereof.

When a series of processing is executed by software, a program constituting the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  The recording medium including such a program is not only constituted by the removable medium 31 of FIG. 1 distributed separately from the apparatus main body in order to provide the program to the user, but also in a state of being incorporated in the apparatus main body in advance. It is comprised with the recording medium etc. which are provided in this. The removable medium 31 is composed of, for example, a magnetic disk (including a floppy disk), an optical disk, a magneto-optical disk, or the like. The optical disk is composed of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk), or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. In addition, the recording medium provided to the user in a state of being incorporated in advance in the apparatus main body includes, for example, the ROM 12 in FIG. 1 in which the program is recorded, the hard disk included in the storage unit 18 in FIG.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in time series along the order, but is not necessarily performed in time series, either in parallel or individually. The process to be executed is also included.

  As mentioned above, although several embodiment of this invention was described, these embodiment is only an illustration and does not limit the technical scope of this invention. The present invention can take other various embodiments, and various modifications such as omission and replacement can be made without departing from the gist of the present invention. These embodiments and modifications thereof are included in the scope and gist of the invention described in this specification and the like, and are included in the invention described in the claims and the equivalents thereof.

[Appendix 1]
As a display target of each of a plurality of planes arranged in a three-dimensional space including a predetermined display area, a moving image composed of a plurality of key frame data and a plurality of frame data interpolated between the key frames is displayed. Video generation means for generating data,
Priority determining means for determining priority for the plurality of moving image data generated by the moving image generating means based on the arrangement positions of the plurality of planes perpendicular to the predetermined display area;
Frame rate setting means for individually setting a frame rate when each of the plurality of moving image data is generated by the moving image generation means based on the priority determined by the priority determination means;
The plurality of planes are arranged in the predetermined display area by combining the data of the plurality of moving images generated by the moving image generation unit at each of the frame rates individually set by the frame rate setting unit. Combining means for generating display image data,
Display control means for controlling the display of the display image based on the data generated by the combining means;
An image processing apparatus comprising:
[Appendix 2]
The synthesizing unit generates data of the display image in which each of the plurality of planes is arranged in parallel in a predetermined direction of the predetermined display area,
The priority determination means further determines the priority based on an arrangement position of each of the plurality of planes in the predetermined direction with reference to a center position of the predetermined display area.
The image processing apparatus according to appendix 1, wherein:
[Appendix 3]
The moving image generating means generates data of the plurality of moving images each including a moving object as a display target for each of the plurality of planes,
The priority determination means further determines the priority based on an occupied area of the moving object in the plane.
The image processing apparatus according to appendix 1 or 2, characterized in that:
[Appendix 4]
In an image processing method executed by an image processing apparatus that executes control of display of a moving image,
As a display target of each of a plurality of planes arranged in a three-dimensional space including a predetermined display area, a moving image composed of a plurality of key frame data and a plurality of frame data interpolated between the key frames is displayed. A video generation step for generating data respectively;
A priority determination step of determining a priority for the plurality of moving image data generated by the processing of the moving image generation step, based on the arrangement positions of the plurality of planes in a direction perpendicular to the predetermined display area;
Frame rate setting for individually setting a frame rate when each of the plurality of moving image data is generated by the processing of the moving image generation step based on the priority determined by the processing of the priority determination step Steps,
By combining the data of the plurality of moving images generated by the processing of the moving image generation step at each of the frame rates individually set by the processing of the frame rate setting step, the plurality of planes are displayed in the predetermined manner. A synthesizing step for generating display image data arranged in the area;
A display control step for controlling the display of the display image based on the data generated by the process of the combining step;
An image processing method comprising:
[Appendix 5]
A computer that controls the display of moving images,
As a display target of each of a plurality of planes arranged in a three-dimensional space including a predetermined display area, a moving image composed of a plurality of key frame data and a plurality of frame data interpolated between the key frames is displayed. Movie generation means for generating data,
Priority determining means for determining priority for the plurality of moving image data generated by the moving image generating means based on the arrangement positions of the plurality of planes perpendicular to the predetermined display area;
A frame rate setting unit for individually setting a frame rate when each of the plurality of moving image data is generated by the moving image generation unit based on the priority determined by the priority determination unit;
The plurality of planes are arranged in the predetermined display area by combining the data of the plurality of moving images generated by the moving image generation unit at each of the frame rates individually set by the frame rate setting unit. Means for generating display image data,
Display control means for controlling display of the display image based on the data generated by the combining means;
A program characterized by functioning as

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 11 ... CPU, 17 ... Output part, 18 ... Memory | storage part, 19 ... Communication part, 31 ... Removable media, 41-1 thru | or 41-N. ..Plain video generation unit, 42... Plain video priority determination unit, 43-1 to 43-N... Frame rate setting unit, 44. ..Display section

Claims (5)

As a display target of each of a plurality of planes arranged in a three-dimensional space including a predetermined display area, a moving image composed of a plurality of key frame data and a plurality of frame data interpolated between the key frames is displayed. Video generation means for generating data,
With respect to the plurality of moving image data generated by the moving image generating means , the horizontal axis with respect to the predetermined display area is the X axis, the vertical axis is the Y axis, and the X axis and the Y axis In a three-dimensional space having a vertical axis as a Z-axis , priority determining means for determining a priority based on the arrangement positions of the plurality of planes in the Z-axis that is the vertical direction in the three-dimensional space ;
Frame rate setting means for individually setting a frame rate when each of the plurality of moving image data is generated by the moving image generation means based on the priority determined by the priority determination means;
The plurality of planes are arranged in the predetermined display area by combining the data of the plurality of moving images generated by the moving image generation unit at each of the frame rates individually set by the frame rate setting unit. Combining means for generating display image data,
Display control means for controlling the display of the display image based on the data generated by the combining means;
An image processing apparatus comprising: The synthesizing unit generates data of the display image in which each of the plurality of planes is arranged in parallel in a predetermined direction of the predetermined display area,
The priority determination means further determines the priority based on an arrangement position of each of the plurality of planes in the predetermined direction with reference to a center position of the predetermined display area.
The image processing apparatus according to claim 1. The moving image generating means generates data of the plurality of moving images each including a moving object as a display target for each of the plurality of planes,
The priority determination means further determines the priority based on an occupied area of the moving object in the plane.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. In an image processing method executed by an image processing apparatus that executes control of display of a moving image,
As a display target of each of a plurality of planes arranged in a three-dimensional space including a predetermined display area, a moving image composed of a plurality of key frame data and a plurality of frame data interpolated between the key frames is displayed. A video generation step for generating data respectively;
With respect to the plurality of moving image data generated by the moving image generation step, the horizontal axis with respect to the predetermined display area is the X axis, the vertical axis is the Y axis, and the X axis and the Y axis A priority determining step for determining a priority based on the arrangement positions of the plurality of planes on the Z-axis that is the vertical direction in the three-dimensional space in a three-dimensional space having a vertical axis with respect to ,
Frame rate setting for individually setting a frame rate when each of the plurality of moving image data is generated by the processing of the moving image generation step based on the priority determined by the processing of the priority determination step Steps,
By combining the data of the plurality of moving images generated by the processing of the moving image generation step at each of the frame rates individually set by the processing of the frame rate setting step, the plurality of planes are displayed in the predetermined manner. A synthesizing step for generating display image data arranged in the area;
A display control step for controlling the display of the display image based on the data generated by the process of the combining step;
An image processing method comprising: A computer that controls the display of moving images,
As a display target of each of a plurality of planes arranged in a three-dimensional space including a predetermined display area, a moving image composed of a plurality of key frame data and a plurality of frame data interpolated between the key frames is displayed. Movie generation means for generating data,
With respect to the plurality of moving image data generated by the moving image generating means , the horizontal axis with respect to the predetermined display area is the X axis, the vertical axis is the Y axis, and the X axis and the Y axis In a three-dimensional space having a vertical axis as a Z-axis , priority determining means for determining a priority based on the arrangement positions of the plurality of planes in the Z-axis that is the vertical direction in the three-dimensional space ;
A frame rate setting unit for individually setting a frame rate when each of the plurality of moving image data is generated by the moving image generation unit based on the priority determined by the priority determination unit;
The plurality of planes are arranged in the predetermined display area by combining the data of the plurality of moving images generated by the moving image generation unit at each of the frame rates individually set by the frame rate setting unit. Means for generating display image data,
Display control means for controlling display of the display image based on the data generated by the combining means;
A program characterized by functioning as

Images