JP4891938B2 - Compression encoding preprocessing apparatus and compression encoding preprocessing program - Google Patents

Description

  The present invention relates to a compression encoding preprocessing apparatus and a compression encoding preprocessing program for processing an image signal before compression encoding.

Conventionally, as an apparatus for processing an image signal by paying attention to the movement of a viewer's viewpoint, an image relating to a region in the vicinity of the viewpoint position in an image displayed on a screen is provided with means for detecting the eye movement of the viewer It is known that the amount of code assigned to a signal is made larger than the amount of code assigned to an image signal related to another region, thereby suppressing image quality degradation felt by the user during decoding (see, for example, Patent Documents 1 and 2). ).
JP 7-44110 A Japanese Patent No. 3924794

  However, the conventional one makes the viewer view the moving image with the head fixed, and it is difficult to automatically measure the position of the gazing point when the viewer moves the head. It was a very low possibility. Further, in the conventional one, there is a possibility that a compression code with low versatility depending on the visual characteristics of the viewer in the target image may be obtained.

  The present invention has been made in order to solve the conventional problems, and a compression coding pre-processing apparatus capable of efficiently obtaining a highly versatile high compression code that does not depend on the visual characteristics of a viewer, and An object of the present invention is to provide a compression coding pre-processing program.

The compression coding pre-processing apparatus of the present invention is a compression coding pre-processing apparatus that processes an image signal before compression coding and outputs the processed image signal to the compression coding processing apparatus, and an image related to the input image signal A moving area detecting means for detecting a moving area with movement by a predetermined method; an optical flow information acquiring means for extracting a moving object from the detected moving area and acquiring optical flow information of the extracted moving object; Detecting an appearance object appearing in the screen from the outside of the screen on which the image is displayed or a foreground appearance object appearing in a state hidden in the image displayed on the screen based on the optical flow information, and detecting An appearing object that outputs an object area instruction signal indicating the position of the appearing object or the foreground appearing object An object having a moving speed equal to or higher than a moving speed threshold for generating a jumping eye movement with respect to a viewing distance indicating a size of the screen and a distance from a viewer viewing the screen to the screen; , A jumping eye movement region detection unit that outputs a jumping eye movement region signal indicating the detected position of the object, and the object region instruction signal or the jumping eye movement region when detecting based on the optical flow information And a signal band limiting unit that limits a spatial frequency band of the moving region of the image signal and outputs the signal to the compression coding processing apparatus during a period in which the signal is output .

  With this configuration, the compression coding pre-processing device of the present invention takes into account the movement of the gazing point due to the optical flow information and the jumping eye movement, and the period during which the viewer's visual perception of the moving object is reduced by the jumping eye movement. Since the signal band of the moving area including the moving object is limited, a highly versatile high-compression code that does not depend on the visual characteristics of the viewer can be efficiently obtained.

Furthermore, the compression coding pre-processing apparatus of the present invention further includes an image signal instruction means for inputting the object area instruction signal and the jumping eye movement area signal, and the image signal instruction means includes the compression encoding process. As an image signal output by the apparatus, an intra-frame encoded image signal that is encoded by intra-frame encoding, and a forward predictive encoding that is encoded using a frame image signal that is reproduced first during reproduction are encoded. Inter-frame forward prediction encoded image signal, or bi-directional prediction code encoded by bi-directional predictive encoding encoded using a frame image signal reproduced earlier and a frame image signal reproduced later The compressed encoding processing apparatus is instructed to select one of the encoded image signals .

  With this configuration, the compression coding pre-processing device of the present invention takes into account the gaze point movement due to the optical flow information and the jumping eye movement, the intra-frame coded image signal, the inter-frame forward prediction coded image signal, and Since either one of the bidirectional predictive encoded image signals is instructed to the compression encoding processing apparatus as an image signal at the time of compression encoding, a highly versatile high compression code that does not depend on the visual characteristics of the viewer can be efficiently obtained. be able to.

The compression coding pre-processing program of the present invention is a compression coding pre-processing for causing a computer to function as a compression coding pre-processing device that processes an image signal before compression coding and outputs the processed image signal to the compression coding processing device. A program, wherein the computer extracts a moving object from the moving area detected by moving area detecting means for detecting a moving area having movement from an image related to the input image signal by a predetermined method. Optical flow information acquisition means for acquiring optical flow information of the moving object, and an appearance object appearing in the screen from the outside of the screen on which the image is displayed or appearing from a state hidden in the image displayed on the screen A foreground appearance object to be detected based on the optical flow information, and the detected appearance object or Appearing object detection means for outputting an object area instruction signal indicating the position of the foreground appearing object, and the viewing distance indicating the size of the screen and the distance from the viewer viewing the screen to the screen A jumping eye movement that outputs a jumping eye movement region signal indicating a position of the detected object when an object having a movement speed equal to or higher than a movement speed threshold value that generates the jumping eye movement is detected based on the optical flow information. A signal band that limits the spatial frequency band of the moving area of the image signal and outputs the signal to the compression coding processing apparatus during a period in which the object area instruction signal or the jumping eye movement area signal is output; It has the structure which functions as a limiting means.

  With this configuration, the compression coding pre-processing program of the present invention takes into account the movement of the gazing point due to the optical flow information and the jumping eye movement, and the period during which the viewer's visual perception of the moving object is reduced by the jump eye movement Since the signal band of the moving area including the moving object is limited, a highly versatile high-compression code that does not depend on the visual characteristics of the viewer can be efficiently obtained.

Furthermore, in the compression coding pre-processing program of the present invention, the compression coding pre- processing apparatus further comprises an image signal instruction means for inputting the object area instruction signal and the jumping eye movement area signal, and the compression encoding pre-processing program, the computer, further as an image signal, wherein the compression coding processing apparatus outputs intraframe coded image signal to be encoded by intra-frame coding, is reproduced first at the time of reproduction forward predictive coding inter-frame forward predictive coded picture signal to be encoded by the reduction, or a frame image signal reproduced after the frame image signal reproduced first when the reproduction of coded using frame image signal that preparative said compression encoding processor whether to any of the bidirectional predictive coded picture signal to be encoded by the bidirectional predictive encoding for encoding using It has a structure to function as an image signal instructing means for instructing.

  With this configuration, the compression coding pre-processing program of the present invention takes into account optical flow information and gaze point movement due to jumping eye movements, intra-frame coded image signals, inter-frame forward prediction coded image signals, and Since either one of the bidirectional predictive encoded image signals is instructed to the compression encoding processing apparatus as an image signal at the time of compression encoding, a highly versatile high compression code that does not depend on the visual characteristics of the viewer can be efficiently obtained. be able to.

  The present invention provides a compression coding pre-processing apparatus and a compression coding pre-processing program having an effect of being able to efficiently obtain a highly versatile high-compression code that does not depend on the visual characteristics of a viewer. Is something that can be done.

  Before describing embodiments of the present invention, human visual characteristics will be described.

(1) Description of central vision and peripheral vision Visual vision includes central vision and peripheral vision. Since central vision has high visual acuity and high sensitivity regarding details of shape and spatial structure, information on texture, form, and color of the visual target is mainly processed. In peripheral vision, the visual acuity is lower than in central vision, but since the sensitivity of spatial coordinate perception and fluctuation perception is high, information on the position and motion of the target is processed.

(2) Explanation of slidable pursuit eye movement and jumping eye movement The movement of the moving object in the moving image can be captured by following the eye movement. As the follow-up by eye movement, mainly, a smooth pursuit eye movement (Smooth Pursuit) and a jumping eye movement (Saccade) are known. The sliding pursuit eye movement is a movement that moves the eyeball following the moving target, and functions for a slow target movement speed with a visual angle of several degrees / second or less. The jumping eye movement is a jumping movement that quickly moves the eyeball to a target position, and functions for a quick target movement that cannot be followed by the sliding pursuit eye movement.

(3) Explanation of follow-up by jumping eye movement When the target captured at the fovea of the retina moves at high speed, and the eyeball can no longer capture the target at the fovea, jumping eye movement occurs. Hereinafter, a description will be given with reference to FIG. FIG. 1 is a conceptual diagram showing a response of an eyeball during a jumping eye movement with respect to a motion target. As an initial state, it is assumed that the viewer is capturing the visual target at the fovea of the retina at time t1 shown in FIG. As the time progresses from time t1, the target moves and the target moves off the fovea, resulting in a reduction in visual acuity. The target moves to a position sufficiently away from the fovea.

  At this time, the time t3 when the jumping eye movement occurs is about 150 msec to 200 msec after the time when the eyeball can no longer catch the target in the fovea (approx. T1 for sufficiently fast movement). Further, from 50 msec before the jumping eye movement occurs until the moment (time t3) when the jumping eye movement occurs, the visual signal is suppressed and the motion index becomes invisible. The time when the target is not visible is t2. Subsequently, a period of about 50 msec to 100 msec from time t3 to t4 is a rising period in which the eyeball responds to the movement of the object, and the visual acuity of the viewer is considerably reduced. Therefore, it is considered that the period from time t2 to t4 shown in FIG. 1 is a period in which the moving target is hardly visible due to the jumping eye movement.

(4) Explanation of gaze FIG. 2 is a conceptual diagram showing eye movement during jumping eye movement with respect to a high-speed motion target. A solid line in the figure indicates a visual target position with respect to time displacement, and a broken line in the figure indicates a visual line position with respect to time displacement. As shown in FIG. 2, when following a high-speed motion target, the target is tracked while jumping the eyeball by jumping eye movement. When the target position and the line-of-sight position coincide for a moment, the viewer can capture the moving target with high visual acuity.

  When the target moves within the moving image frame, the movement of gazing at the moving image by eye movement is considered to be due to repetition of the following movements.

  (A) When the gazing point is captured in the initial state in the initial state, (a) When the target moves at high speed, if the eye cannot capture the target in the fovea, a jumping eye movement occurs and moves Attempts to capture the visual target with central vision. (B) When the visual target moves at low speed, it tries to capture the motion visual target with central vision by sliding pursuit eye movement. (B) In the initial state, when the gazing point is captured with peripheral vision, the object to be gazed with peripheral vision is noticed, and the details of the target are grasped with central vision after adjusting the central visual field with jumping eye movements. To do.

  As described above, with respect to a movement part that causes jumping eye movement in compression coding, a large band limitation or motion compensation with low accuracy / accuracy is required for a period from about 50 msec at which jumping eye movement occurs to about 100 msec after jumping eye movement. Even if it is applied, it can be said that deterioration (blurring, block distortion, etc.) is not conspicuous. On the other hand, in the subsequent period, the viewer may follow the fast movement while jumping eye movements, and may recognize deterioration at any point where the eye movement speed matches the movement speed in the moving image. It can be seen that it is preferable not to easily limit the bandwidth more than the loss of high frequency components due to imaging / display accumulation blur or to perform motion compensation with low accuracy and accuracy simply because it is a moving part.

  Next, the configuration of an embodiment of the compression coding pre-processing apparatus according to the present invention will be described. The compression coding pre-processing device in the present embodiment includes an image signal output device that outputs a moving image signal for displaying a moving image on a screen of a predetermined size, and a moving image using a predetermined image compression coding method. It is provided between a compression encoding processing apparatus that compresses and encodes an image signal.

In the following description, the compression coding processing device compresses and encodes a moving image signal based on the MPEG (Moving Picture Coding Experts Group) -2 method, but is not limited thereto. MPEG-4 and H.264 It can also be applied to those using a moving image compression standard such as H.264. Also, a moving image frame at time t is referred to as a moving image frame F (t), and a pixel value at a pixel position (i, j) of the moving image frame F (t) is referred to as F _{i, j} (t).

  As shown in FIG. 3, the compression coding pre-processing apparatus 10 in the present embodiment includes a frame memory 11 that stores an image signal, a clock signal generator 12 that generates a clock signal, and a scene that detects a scene change. A change detection unit 13 and a moving region detection unit 14 that detects a moving region including a moving image and outputs a moving region image signal 20 are provided.

  Further, the compression coding preprocessing apparatus 10 includes a motion vector detection unit 15 that detects a motion vector based on the motion region image signal 20, an appearance object detection unit 16 that detects an appearance object that appears on the screen, and a jump. A jumping eye movement region detecting unit 17 that detects a sexual eye movement region, a signal band limiting unit 18 that limits a signal band, and a picture switching unit 19 that switches pictures. Note that the compression coding pre-processing device 10 is configured by a computer, for example, and operates based on a predetermined program.

  The frame memory 11 includes frame memories 11a to 11d, and stores a moving image signal input from an image signal output device (not shown) for each frame based on a clock signal generated by the clock signal generator 12. . The image signal stored in the frame memory 11c will be described below as the image signal of the encoding target frame at time t.

  The clock signal generator 12 generates a clock signal (shown as a CLK signal) having a predetermined frequency, and the image signal stored in the frame memory 11 is shifted based on this clock signal.

  The scene change detection unit 13 detects a scene change (scene change) based on difference information between frames before and after the encoding target frame at time t. Specifically, the scene change detection unit 13 compares the image signal stored in the frame memory 11c with the image signal stored in the frame memory 11b to obtain difference information between the two, and the difference between the two is obtained. If there is, it is determined that there has been a scene change, and a clock control signal (CLK control signal and illustrated) for shifting the image signal stored in the frame memory 11 is output to the clock signal generator 12. ing.

  When there is no scene change, the moving area detecting unit 14 receives an image signal from the scene change detecting unit 13 and detects a moving area indicating a moving area in the input image signal. Specifically, for example, when the position of the frame F on the time axis is t, the moving region detection unit 14 includes an input image frame sequence F (t) including t ± q frames before and after that; −q ≦ t ≦ q On the other hand, the first-order one-dimensional discrete wavelet decomposition is performed in the time axis direction, and the high-frequency component is obtained to extract the image signal of the moving area image frame sequence from the original image frame sequence.

  In addition, the moving area detecting unit 14 outputs a moving area signal 20 indicating an image signal of the extracted moving area image frame sequence to the moving vector detecting unit 15. In FIG. 1, the moving area image signals 20 stored in the frame memories 11a to 11d when there is no scene change are shown as moving area image signals 20a to 20d, respectively. The moving area detection unit 14 constitutes a moving area detection unit according to the present invention.

  The motion vector detection unit 15 includes motion vector detection units 15a to 15c, extracts a signal of a moving object (moving object) from the motion area image signal 20, and detects a motion vector indicating the motion of the object. ing. Specifically, for example, the motion vector detection unit 15a performs n-order two-dimensional discrete wavelet decomposition in the spatial direction on the motion area image signal 20a and the motion area image signal 20b, The horizontal, vertical and diagonal high-frequency components are obtained, and the motion vector of the object between the motion region image signal 20a and the motion region image signal 20b is hierarchically detected.

  The motion vector detection unit 15 extracts and tracks an object using the detected motion vector between frames, and detects a signal including optical flow information which is motion field information of each point of the object on the screen. It outputs to the part 16 and the jumping eye movement area | region detection part 17. FIG. The motion vector detection unit 15 constitutes a moving object extraction unit and an optical flow information acquisition unit according to the present invention.

  The appearance object detection unit 16 detects the position of the appearance object based on the optical flow information, and outputs a signal indicating the position of the appearance object to the signal band limiting unit 18 and the picture switching unit 19 as an OBJ (OBJect) area instruction signal. It is like that. Here, an appearance object is an object (appearance object) that appears (frame-in) from the outside of the screen in the screen, or an object (foreground) that appears from an obstruction (occlusion) image displayed on the screen (foreground). This is a generic term for "appearing objects". In addition, the appearance object detection part 16 comprises the moving object extraction means which concerns on this invention.

  Based on the optical flow information, the jumping eye movement region detection unit 17 detects the movement position of an object having a movement speed that causes jumping eye movement, and outputs a signal indicating the position of the corresponding object S (Saccade ) The signal is output to the signal band limiting unit 18 and the picture switching unit 19 as an area instruction signal. Here, the jumping eye movement region detecting unit 17 compares the moving speed of the object indicated by the optical flow information with a moving speed threshold value determined in advance based on the size of the screen on which the image is displayed and the standard viewing distance. When the moving speed of the object is larger than the moving speed threshold, it is determined that the object is performing a jumping eye movement. The standard viewing distance refers to a viewing distance of 1 pixel per viewing angle of 1 minute at which a pixel structure is not perceived by a viewer with a visual acuity of 1.0.

  Hereinafter, the relationship between the size of the screen for displaying an image, the standard viewing distance, and the jumping eye movement will be described with reference to FIG.

  As described in the description of human visual characteristics, the sliding pursuit eye movement functions for a slow target movement speed with a visual angle of several degrees / second or less, and in the following description, it is 10 degrees / second. It is assumed that jumping eye movements occur with the above movement.

  FIG. 4 shows the number of pixels constituting each screen, the standard viewing distance, and the standard view in each video system of SHV (Super Hi-Vision), HDTV (Hi-Vision TV), and NTSC (National Television Standards Committee). It shows the angle of view at the viewing distance.

In each video format, when shooting a subject with the camera so that the shooting angle of view is the same, and viewing the moving image at the standard viewing distance, the movement speed between frames in which jumping eye movement occurs is as follows: become.
(1) In the case of SHV: 100 [degrees] / 10 [degrees / second] = 10 [seconds]
(2) HDTV: 30 [degrees] / 10 [degrees / second] = 3 [seconds]
(3) For NTSC: 10 [degrees] / 10 [degrees / second] = 1 [seconds]

  That is, it can be said that jumping eye movements occur in a fast movement across the screen in the above-described number of seconds in each video system. When shooting a subject with the same angle of view with a camera, due to the viewing distance, the SHV has a movement speed that is about 3.3 times that of HDTV and about 10 times that of NTSC, and it can be seen that jumping eye movements are more likely to occur. .

  The following is a standard video for high-definition system evaluation (hereinafter referred to as ITE standard video) created by the Institute of Television Engineers, which is currently used as a standard in HDTV, and a high-definition system issued by the same corporation. Refer to “Standard moving image for evaluation, Statistics measurement manual (hereinafter referred to as manual)”.

  In the above description, the inter-frame motion vector value of each moving image sequence of 60 sequences of ITE standard moving images is shown. According to the book, it is generally said that the fastest movement in a television image is a movement that crosses the screen in one second, and the average movement is a movement that crosses the screen in several seconds. For example, in HDTV, the fastest motion is a motion speed of 1920 [pixels] / (60 × 1) [field (1 second)] = 32 [pixels / field] between one field. As an average motion in HDTV, if the screen is crossed in 5 seconds, 1920 [pixel] / (60 × 5) [field] = 6.4 [pixel / field].

  As described above, when the movement speed at which the jumping eye movement is generated is 10 [degrees / second], the time for crossing the screen in HDTV is 3 seconds, and the movement speed at that time is 1920 [pixels] / (60 × 3 ) [Field (3 seconds)] = 10.7 [pixel / field].

  On the other hand, if the subject is photographed with the SHV camera at the same angle of view, 7680 [pixels] / (60 × 5) [ Frame (5 seconds)] = 25.6 [pixel / frame]. If the movement speed at which the jumping eye movement is generated is 10 [degrees / second], the time to cross the screen in SHV is 10 seconds, and the movement speed at that time is 7680 [pixels] / (60 × 10) [ Frame (10 seconds)] = 12.8 [pixel / frame].

  As mentioned above, the screen size differs depending on the video system, so the moving speed is changed so that the gaze point movement due to the jumping eye movement occurs with respect to the predetermined screen size and viewing distance. It is determined as a speed threshold. For example, the above-described 12.8 [pixel / frame] is used as the moving speed threshold in SHV. When an object having a movement speed equal to or greater than the movement speed threshold is detected, the jumping eye movement area detection unit 17 determines that the object is performing jumping eye movement, and outputs an S area instruction signal related to the object. It is designed to output. Here, the S area indicated by the S area instruction signal may be an area where the object is displayed, or may include an area around the object obtained by adding a predetermined number of pixels to the area where the object is displayed. Good.

  Returning to FIG. 3, the description of the configuration is continued. The signal band limiting unit 18 limits the spatial frequency band of the image signal based on the OBJ region instruction signal and the S region instruction signal, and removes a high frequency component having a predetermined frequency or more from the image signal. Yes. The signal band limiting unit 18 outputs an image signal including a limited spatial frequency band signal component to the compression coding processing apparatus. The signal band limiting unit 18 constitutes a signal band limiting unit according to the present invention.

Specifically, the signal band limiting unit 18 operates as described below.
First, the signal band limiting unit 18 is configured to perform the intra-S region in a period in which the S region instruction signal is output from the jumping eye movement region detection unit 17, that is, in a period in which the visual acuity of the viewer is reduced by the jumping eye movement. The spatial frequency band of the image signal is limited. If the jumping eye movement is repeated in a monotonous pattern, the signal band limiting unit 18 may, for example, perform the first spatial frequency because the viewer may be able to follow a fast movement while jumping eye movement. It is preferable to configure to perform only the band.

  Further, it is assumed that the signal band limiting unit 18 is a period in which the OBJ region instruction signal is output from the appearance object detection unit 16 and the object region that is framed in from the outside of the screen to the inside of the screen is viewed with peripheral vision. The spatial frequency band of the image signal in the vicinity of the object area to be framed in is limited from the time until the time when the central visual field is assumed to move to the object area by jumping eye movement.

  Similarly, the signal band limiting unit 18 is a period in which the OBJ region instruction signal is output from the appearance object detection unit 16, and from the time when it is assumed that the object region appearing in the foreground from the background is viewed with peripheral vision. The spatial frequency band of the image signal in the vicinity of the object region is limited in a period until the time when it is assumed that the central visual field is moved to the object region by the jumping eye movement.

  Further, when the OBJ area instruction signal and the S area instruction signal are not input, the signal band limiting unit 18 does not limit the spatial frequency band of the image signal, and directly compresses the image signal input from the frame memory 11. To output.

  The picture switching unit 19 outputs a picture switching control signal for switching the picture type used by the compression coding processing apparatus to the compression coding processing apparatus based on the OBJ area instruction signal and the S area instruction signal. . That is, in this embodiment, since the compression encoding processing apparatus uses the MPEG-2 system, the picture switching unit 19 converts an image signal of a frame to be encoded into an I picture (intra-frame encoded image). Signal), P picture (interframe forward predictive encoded image signal), and B picture (bidirectional predictive encoded image signal) are output.

  Specifically, the picture switching unit 19 is used when the gazing point movement due to the jumping eye movement is considered to occur, for example, when a large object moves on the screen or when panning is executed on the entire screen. A picture switching control signal for instructing to use compression coding with high compression efficiency, such as a B picture, is output to the compression coding processing apparatus during a period in which jumping eye movements occur and visual information is considered to be suppressed / decreased in visual acuity It is like that.

  On the other hand, the picture switching unit 19 moves the frame in the case of a long-time movement that may follow the moving object by the central vision by repeating a jumping eye movement even at a high speed movement or a movement close to the sliding pursuit eye movement. A picture switching control signal for instructing to use a P picture or an I picture with high inter-correlation and high motion vector detection accuracy is output to the compression coding processing apparatus. Note that the picture switching unit 19 constitutes an image signal instruction unit according to the present invention.

  Next, the operation of the compression coding pre-processing apparatus 10 in the present embodiment will be described with reference to FIG.

  An image signal input from an image signal output device (not shown) is stored in the frame memories 11a to 11d for each frame based on the clock signal.

  The scene change detection unit 13 compares the image signal stored in the frame memory 11c, which is the image signal of the encoding target frame at time t, with the image signal stored in the frame memory 11b, and obtains difference information between them. get. At this time, if there is a difference between the two, the scene change detection unit 13 determines that a scene change has occurred, and outputs a clock control signal to the clock signal generator 12. As a result, the image signal stored in the frame memory 11 is shifted. On the other hand, if there is no difference between the two, the scene change detection unit 13 determines that there is no scene change and outputs an image signal to the moving region detection unit 14.

  The moving area detection unit 14 receives an image signal from the scene change detection unit 13 and detects a moving area in the input image signal. The detection of the moving area by the moving area detector 14 will be specifically described with reference to FIG. Note that F (t + 3) shown in FIG. 5 will be described as a moving area detection target.

  The moving region detection unit 14 uses, for example, Daubechies secondary wavelets (support length = 4), as shown in the schematic diagrams of FIGS. 5A and 5B, F (t), F (t + 1), First-order one-dimensional discrete wavelet decomposition is performed on time-series data in the time axis direction of a moving image frame sequence composed of F (t + 2) and F (t + 3), and MA (t) that is a high-frequency component is calculated. That is, MA (t) includes a time-series change component for a support length of 4 frames.

  Similarly, MA (t + 2) is calculated from time series data in the time axis direction including F (t + 2), F (t + 3), F (t + 4), and F (t + 5).

  Then, MA (even) is calculated by performing AND calculation of MA (t) and MA (t + 2).

  Further, MA (t + 1) is similarly calculated from time-series data in the time axis direction including F (t + 1), F (t + 2), F (t + 3), and F (t + 4).

  Next, MA (t + 3) is similarly calculated from time-series data in the time axis direction including F (t + 3), F (t + 4), F (t + 5), and F (t + 6).

  Then, MA (odd) is calculated by performing AND calculation of MA (t + 1) and MA (t + 3).

  Next, an AND calculation of MA (even) and MA (odd) is performed, binarization processing is performed, Ma (t) is calculated, and AND calculation of F (t + 3) and Ma (t) is performed. Thus, a moving area image frame sequence M (t) for one frame is calculated.

Subsequently, as shown in FIG. 6, the motion vector detection unit 15 performs n-order two-dimensional discrete wavelet decomposition in the spatial direction on each frame of the motion region image frame sequence M (t), A frequency component M_LL ⁿ (t) and horizontal, vertical, and diagonal high frequency components M_LH ⁿ (t), M_HL ⁿ (t), and M_HH ⁿ (t) are obtained.

  In addition, the motion vector detection unit 15 uses the frame M (t) at time t in the motion area image frame sequence M (t) as a reference frame, and sets the frame M (t + 1) at time t + 1 as a reference frame, to frame M (t). And M (t + 1) are detected in a hierarchical manner. Here, the motion vector detection processing by the motion vector detection unit 15 will be described in detail with reference to FIG. FIG. 7 is a flowchart showing a motion vector detection process executed by the motion vector detection unit 15. It is assumed that the target moving image frame F (t) is in the time range of 0 ≦ t ≦ T.

  First, the time t is initialized to “0” (step S10), and the index n indicating the wavelet decomposition hierarchy is initialized to “1” (step S11).

Next, a block matching method is used between the low frequency component M_LL ⁿ (t) and the low frequency component M_LL ⁿ (t + 1) to correspond to the pixel position (i, j) of the low frequency component M_LL ⁿ (t). The motion vector V ⁿ _{i, j} (t) is detected (step S12).

Next, the pixel value at the pixel position (i, j) of the low frequency component M_LL ⁿ (t), the horizontal high frequency component M_LH ⁿ (t), the vertical high frequency component M_HL ⁿ (t), and the diagonal high frequency component M_HH ⁿ (t). Pixels at pixel positions shifted by V ⁿ _{i, j} (t) by M_LL ⁿ _{i, j} (t), M_LH ⁿ _{i, j} (t), M_HL ⁿ _{i, j} (t), and M_HH ⁿ _{i, j} (t) Replace values, motion correction low frequency component M′_LL ⁿ (t), motion correction horizontal high frequency component M′_LH ⁿ (t), motion correction vertical high frequency component M′_HL ⁿ (t) and motion correction diagonal high frequency component M′_HH ⁿ (t) is calculated (step S13).

Next, the motion correction low frequency component M′_LL ⁿ (t), the motion correction horizontal high frequency component M′_LH ⁿ (t), the motion correction vertical high frequency component M′_HL ⁿ (t), and the motion correction diagonal high frequency component M ′ _HH ⁿ (t) is reconstructed to the first floor, and the motion-corrected low-frequency component M′_LL ⁿ⁻¹ (t) is calculated (step S14).

Next, the low frequency component M_LL ⁿ⁻¹ (t) is replaced with the motion corrected low frequency component M′_LL ⁿ⁻¹ (t) (step S15).

  Next, it is determined whether or not the index n indicating the wavelet decomposition hierarchy has reached the maximum value N (step S16). If a negative determination is made, the index n is incremented (step S17), and the process of step S12 is performed. Return.

If it is determined in step S16 that the index n has reached the maximum value N, the motion vector V ⁰ _{i, j} (t) for the original motion area image M (t) is calculated from [Equation 1] (step 1). S18).

  Next, it is determined whether or not the time t has reached the time T (step S19). If a negative determination is made, the time t is incremented (step S20), and the process returns to step S11.

  If it is determined in step S19 that time t has reached time T, the motion vector detection process ends.

Subsequently, the motion vector detection unit 15 uses the motion vector V ⁰ _{i, j} (t) between frames at an arbitrary pixel position (k, m) on F (t ₀ ), and F (t); −q ≦ Track the motion at t ≦ q. From this time-series optical flow (motion field of each point on the screen) information, the motion vector detection unit 15 extracts and tracks an object.

Specifically, the motion vector detection unit 15 uses the motion vector V ⁰ _{i, j} (t ′) (1 ≦ i ≦ I, 1 ≦ j ≦ J, t ₀ −q ≦ t ′) to generate a moving image. pixel position of the frame F (t) (i, j ) to track the motion from time _{t 0} to the time axis negative direction over _{t 0-q} of. Further, using the motion vector V ⁰ _{i, j} (t ′) (1 ≦ i ≦ I, 1 ≦ j ≦ J, t ′ ≦ t _{0 + q} ), the pixel position (i, j) of the moving image frame F (t) is used. ) to track the motion from time t ₀ to the time axis plus direction over t _{0 + q} of.

Next, based on the tracking result of the pixel position (i, j) of the moving image frame F (t), the absolute _{value of} the moving vector V ⁰ _{i, j} (t ′) (t _0−q ≦ t ′ ≦ t _{0 + q} ) and time t _s the value is minimized, it determines the tracking position (k, m). In addition, since the motion direction of the same object between successive frames is considered to be substantially continuous, the motion vector detection unit 15 performs motion vector V ⁰ _{i, j} (t ′) (t _0−q ≦ t ′ ≦ 0), the motion vector value and the object position are tracked, and the time at which the object enters the frame is determined as the appearance object time t_in. The motion vector detection unit 15 calculates an inner product of the motion vectors V ⁰ _{i, j} (t) and V ⁰ _{i, j} (t−1), and calculates the inner product value obtained by the calculation and a predetermined threshold th. Compare This threshold value th is for determining the continuity of motion vector motion. If the inner product value is larger than the threshold value th, it indicates that the motion vector motion continuity is impaired. If the inner product value is larger than the threshold th, it is presumed that there has been a motion such as an object appearing in the foreground from the background, so the motion vector detection unit 15 determines the corresponding time as the appearance object time t_appear.

  Returning to FIG. 3, the description of the operation will be continued. When the motion vector detection unit 15 determines the appearance object time t_in or t_appear, the appearance object detection unit 16 determines that the appearance object has been detected, and sends the OBJ region instruction signal indicating the position of the appearance object to the signal band limiting unit 18 and The image is output to the picture switching unit 19.

  In parallel, the jumping eye movement region detection unit 17 is based on the optical flow information output from the motion vector detection unit 15 and the movement speed threshold, and the movement position of the object having a movement speed that causes jumping eye movement. When a corresponding object is detected, an S area instruction signal indicating the position of the object is output to the signal band limiting unit 18 and the picture switching unit 19.

  Next, when either the OBJ region instruction signal from the appearance object detection unit 16 or the S region instruction signal from the jumping eye movement region detection unit 17 is input, the signal band limiting unit 18 detects the detected object. The spatial frequency band of the image near the display position is limited, and an image signal from which a predetermined high-frequency component has been removed is output to the compression coding processing apparatus. On the other hand, when the OBJ region instruction signal and the S region instruction signal are not input, the signal band limiting unit 18 does not limit the spatial frequency band of the image signal, and directly compresses the image signal input from the frame memory 11. Output to.

  In parallel, when the picture switching unit 19 receives either the OBJ region instruction signal from the appearance object detection unit 16 or the S region instruction signal from the jumping eye movement region detection unit 17, the compression coding process is performed. In the compression encoding of the apparatus, a picture switching control signal for designating whether an image signal of a frame to be encoded at present is an I picture, a P picture, or a B picture is output. On the other hand, the picture switching unit 19 does not output the picture switching control signal when the OBJ area instruction signal and the S area instruction signal are not input.

  Note that by programming each process described in the explanation of the operation and causing the computer to operate according to the program, the program can cause the computer to function as the compression encoding pre-processing device 10. In this case, the computer has the configuration shown in FIG.

  As described above, according to the compression coding pre-processing apparatus 10 according to the present embodiment, the viewer of the moving object by the jumping eye movement is considered in consideration of the optical flow information and the gaze point movement by the jumping eye movement. Since the signal band of the moving area including the moving object is limited during the period when the visual perception of the image decreases, a highly versatile high-compression code that does not depend on the visual characteristics of the viewer can be efficiently obtained.

  In addition, according to the compression coding pre-processing device 10 in the present embodiment, the viewer views the region of the appearing object in the peripheral view in consideration of the optical flow information of the appearing object and gaze point movement due to the jumping eye movement. The signal band of the moving region including the appearing object is limited during the period from the time when it is assumed to be viewed in the period until it is assumed that the central visual field is moved to the area of the appearing object by the jumping eye movement. A highly versatile high-compression code that does not depend on the visual characteristics of the viewer can be obtained efficiently.

  In addition, according to the compression coding pre-processing apparatus 10 in the present embodiment, the viewer determines the area of the foreground appearance object in consideration of the optical flow information of the foreground appearance object and gaze point movement due to the jumping eye movement. A configuration that limits the signal band of the moving region including the foreground appearance object during a period from when it is assumed to be viewed in peripheral vision to when it is assumed that the central visual field is moved to the region of the foreground appearance object by jumping eye movement Therefore, a highly versatile high-compression code that does not depend on the visual characteristics of the viewer can be efficiently obtained.

  In addition, according to the compression coding pre-processing device 10 in the present embodiment, intra-frame coded image signals and inter-frame forward prediction coding are performed in consideration of optical flow information and gaze point movement due to jumping eye movements. Since either the image signal or the bi-directional predictive encoded image signal is instructed to the compression encoding processing apparatus as an image signal at the time of compression encoding, high versatility and high compression that does not depend on the visual characteristics of the viewer The code can be obtained efficiently.

  In particular, the compression coding pre-processing device 10 can obtain a great effect in a wide-field video system. Specifically, the time for the moving object in the screen to move is generally about 1 second on average, and the period for limiting the bandwidth of the moving area on the screen by jumping eye movement is about 150 to 250 msec, Since most of the moving area in the screen (generally about 60% to 80% in the frame) is followed by jumping eye movement in SHV, the effect of the present invention is very effective in SHV which is a wide field of view / ultra high definition video. It can be said that it is very expensive.

  As described above, the compression coding pre-processing apparatus and the compression coding pre-processing program according to the present invention can efficiently obtain a highly-compressed high-compression code that does not depend on the visual characteristics of the viewer. It is useful as a compression encoding pre-processing apparatus and a compression encoding pre-processing program for processing an image signal before compression encoding.

Conceptual diagram showing the response of the eyeball during jumping eye movements to a motion target Conceptual diagram showing eye movements for high-speed movement targets during jumping eye movements The block diagram which shows the structure in one Embodiment of the compression coding pre-processing apparatus which concerns on this invention The figure which shows the number of pixels which comprise a screen by each video system of SHV, HDTV, and NTSC, a standard viewing distance, and an angle of view in a standard viewing distance Explanatory drawing about the detection of the moving region by a moving region detection part in one Embodiment of the compression coding pre-processing apparatus which concerns on this invention FIG. 3 is a schematic diagram of n-th order two-dimensional discrete wavelet decomposition performed by the motion vector detection unit on the motion region image frame sequence M (t) in the embodiment of the compression coding pre-processing apparatus according to the present invention. The flowchart which shows the detection process of the motion vector which a motion vector detection part performs in one Embodiment of the compression coding pre-processing apparatus which concerns on this invention

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Compression encoding pre-processing apparatus 11 (11a-11d) Frame memory 12 Clock signal generator 13 Scene change detection part 14 Motion area detection part (motion area detection means)
15 (15a-15c) Motion vector detection unit (moving object extraction means, optical flow information acquisition means)
16 Appearance object detection part (moving object extraction means)
17 Jumping Eye Movement Region Detection Unit 18 Signal Band Limiting Unit (Signal Band Limiting Unit)
19 Picture switching unit (image signal instruction means)
20 (20a-20d) Moving region image signal

Claims (4)

A compression coding pre-processing device that processes an image signal before compression coding and outputs the processed image signal to a compression coding processing device,
A moving region detecting means for detecting a moving region having movement from an image related to the input image signal by a predetermined method;
An optical flow information acquisition means for extracting a moving object from the detected moving area and acquiring optical flow information of the extracted moving object;
An appearance object that appears in the screen from the outside of the screen on which the image is displayed or a foreground appearance object that appears from a state hidden in the image displayed on the screen is detected based on the optical flow information, and detected. An appearance object detection unit that outputs an object area instruction signal indicating a position of the appearance object or the foreground appearance object;
An object having a movement speed equal to or higher than a movement speed threshold for generating a jumping eye movement with respect to a viewing distance indicating a size of the screen and a distance from a viewer who views the screen to the screen. A jumping eye movement region detection unit that outputs a jumping eye movement region signal indicating the position of the detected object when detected based on information;
Signal band limiting means for limiting the spatial frequency band of the moving region of the image signal and outputting the same to the compression coding processing device during a period in which the object region instruction signal or the jumping eye movement region signal is output. A compression coding pre-processing apparatus characterized by the above. The compression coding pre-processing apparatus further includes an image signal instruction means for inputting the object area instruction signal and the jumping eye movement area signal,
The image signal instructing means encodes an image signal output from the compression encoding processing device using an intra-frame encoded image signal encoded by intra-frame encoding and a frame image signal previously reproduced during reproduction. Bi-directional encoding using an inter-frame forward predictive encoded image signal encoded by forward predictive encoding or a frame image signal reproduced earlier during reproduction and a frame image signal reproduced later 2. The compression coding pre-processing device according to claim 1, wherein the compression coding processing device is instructed to select one of bidirectional prediction coded image signals to be coded by predictive coding . A compression encoding preprocessing program that causes a computer to function as a compression encoding preprocessing device that processes an image signal before compression encoding and outputs the processed image signal to a compression encoding processing device,
The computer,
A moving region detecting means for detecting a moving region having movement from an image related to the input image signal by a predetermined method;
An optical flow information acquisition means for extracting a moving object from the detected moving area and acquiring optical flow information of the extracted moving object;
An appearance object that appears in the screen from the outside of the screen on which the image is displayed or a foreground appearance object that appears from a state hidden in the image displayed on the screen is detected based on the optical flow information, and detected. Appearance object detection means for outputting an object area instruction signal indicating the position of the appearance object or the foreground appearance object;
An object having a movement speed equal to or higher than a movement speed threshold for generating a jumping eye movement with respect to a viewing distance indicating a size of the screen and a distance from a viewer who views the screen to the screen. A jumping eye movement region detecting means for outputting a jumping eye movement region signal indicating the position of the detected object when detected based on information;
In the period in which the object area instruction signal or the jumping eye movement area signal is output, the spatial frequency band of the moving area of the image signal is limited to function as signal band limiting means for outputting to the compression coding processing apparatus. A compression encoding preprocessing program characterized by the above. The compression coding pre-processing apparatus further includes an image signal instruction means for inputting the object area instruction signal and the jumping eye movement area signal,
The compression encoding pre-processing program stores the computer,
Further, as an image signal output from the compression coding processing apparatus, forward prediction is performed by using an intra-frame encoded image signal encoded by intra-frame encoding, and a frame image signal previously reproduced during reproduction. Coded by bi-directional predictive coding, which is coded using an inter-frame forward-predicted encoded image signal encoded by encoding, or a frame image signal reproduced earlier during reproduction and a frame image signal reproduced later 4. The compression coding preprocessing program according to claim 3, wherein the compression coding preprocessing program is made to function as an image signal instruction means for instructing the compression coding processing device which of the two-way predictive coded image signals to be converted to. .

Images