JP4722147B2 - Image processing method, image processing apparatus, and image processing program - Google Patents

Description

  The present invention relates to a technique for extracting necessary minimum information from time-series image data.

  In recent years, TV conferences using IP cameras, mobile phones, and the like are popular. In particular, communication at a low bit rate is required for cost reduction. The same applies to remote monitoring using a WEB camera. When the bit rate is lowered, as shown in FIG. 7, there is a problem that block noise increases and not only the image quality is remarkably deteriorated but also important changes such as facial expressions are difficult to understand.

  According to Non-Patent Document 1, interpolation of facial expression change is shown by combining a facial expression model and image processing. However, since the artificial facial expression model is used, there is a problem that the obtained facial expression is unnatural.

  There is almost no problem as to how to transmit actual facial expression change information if sufficient bandwidth is secured. However, at a low bit rate, it is necessary to add image processing suitable for it, but a method that does not use an artificial model shown in Non-Patent Document 1 is essential.

In addition, in order to reduce the bit rate, not only facial expressions but also important important information is lost, and thus a contrivance is necessary.
Lee Yuan-chu, 1 other, "Realization of facial expression transformation based on sketch images and multi-scale edges", IEICE Transactions D-II, IEICE, May 1999, Vol. J82-D-II , No.5, p.900-908

  As shown in FIG. 7, there is a known problem that block noise such as blurring, ringing, and mosquito noise occurs as the bit rate is reduced. For this reason, the transmitted image becomes an image that is difficult to see due to compression. In addition, although the block noise is alleviated by applying a low-pass filter, there is a problem that important image feature amounts are blurred and situation changes such as facial expression changes are difficult to understand.

  In the case of a high bit rate, the image quality is improved, but there is a problem that transmission takes time because the transmission capacity is large.

  The present invention has been made in view of the above, and an object of the present invention is to detect an important feature amount by image processing in order to communicate a change in the situation at high speed even at a low bit rate.

In the image processing method according to the first aspect of the present invention, the time-series image is transmitted in order to communicate a change in status at a bit rate lower than a bit rate capable of securing a band for transmitting the entire image information amount of the time-series image data. An image processing method for detecting an important feature amount of interest from data, the step of inputting time-series image data by an input means, and storing the time-series image data in a storage means; and a change area detection means The time series image data is read from the storage means, a difference image is obtained by calculating a difference between the time series image data, the step of storing the difference image in the difference data storage means, and the time series weighting means, The difference images are read from the difference data storage means in chronological order, and the luminance of the pixels of the difference image is measured from the past to the present. A feature image is obtained by weighting and adding linearly or nonlinearly so as to increase a finding coefficient, and the time-series image data is read from the storage unit by a statistical selection unit, and pixels of the time-series image data are read out. Pixels that detect a variation in luminance, and whose luminance is equal to or greater than a predetermined value and continuously appear within a predetermined variation range for a predetermined time or more, and whose luminance variation range is larger than the predetermined variation range Alternatively, a pixel in which the time when the luminance exceeding a predetermined value continuously appears is shorter than the predetermined time is identified as background noise, and unnecessary information is removed by removing the background noise from the feature image. And obtaining a transmission target image with a reduced amount of information .

  In the present invention, the difference image obtained by the change area detecting means is weighted so as to be clearly bright from the past to the present, so that an afterimage of a change in image data, such as a change in facial expression, is obtained. It is possible to obtain a feature image that is effective. In addition, the statistical selection means identifies background noise based on statistical properties, so that only necessary information can be selected and the amount of information can be reduced, so that a change in situation can be communicated at high speed even at a low bit rate. It becomes possible.

  In the present invention, the background noise can be simply processed by identifying the background noise based on the luminance variation width and the luminance duration of the pixel. Real-time processing is possible.

The image processing apparatus according to the second aspect of the present invention provides the time-series image in order to communicate the status change at a bit rate lower than a bit rate capable of securing a band for transmitting the entire image information amount of the time-series image data. An image processing apparatus for detecting an important feature amount of interest from data, comprising: input means for inputting time-series image data; storage means for storing the input time-series image data; and The time-series image data is read out, and the change area detection means for obtaining a difference image by calculating the difference between the time-series image data, the difference data storage means for storing the difference image, and the difference data storage means in time series order from the difference data storage means Read out the difference image and linearly or nonlinearly increase the weighting coefficient from the past to the present with respect to the brightness of the pixel of the difference image. A time-series weighting unit that obtains a feature image by adding the data, and reads out the time-series image data from the storage unit, detects a change in luminance of a pixel of the time-series image data, and has a luminance greater than or equal to a predetermined value The pixel of interest is a pixel that appears continuously within a predetermined fluctuation range for a predetermined time or more, and a pixel whose luminance fluctuation range is larger than the predetermined fluctuation range, or a luminance of a predetermined value or more continues. Statistical selection that identifies a pixel whose appearance time is shorter than the predetermined time as background noise, and removes the background noise from the feature image to remove unnecessary information and obtain a transmission target image with a reduced amount of information And means.

  An image processing program according to a third aspect of the present invention causes a computer to execute each step in the image processing method.

  According to the present invention, since a situation change is communicated at high speed even at a low bit rate, an important feature amount can be detected by image processing.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to the present embodiment. The image processing apparatus shown in the figure includes a data input unit 100, a data storage unit 110, a change area detection unit 120, a difference data storage unit 130, a time series weighting unit 140, a statistical selection unit 150, and a display unit 160. The image processing apparatus may be configured by a computer including an arithmetic processing device, a storage device, a memory, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image processing apparatus, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or provided through a network. Details of each part will be described below.

  The data input unit 100 inputs time-series images through a camera (not shown) and stores the images in the data storage unit 110.

  The change area detection unit 120 reads the time series image stored in the data storage unit 110, and obtains a temporal change of the time series image by difference calculation. Then, the difference image obtained by the difference calculation is stored in the difference data storage unit 130.

  The time series weighting unit 140 reads the difference images stored in the difference data storage unit 130 in time series order, and weights the read difference images linearly or nonlinearly so that the gray value of the past difference image becomes lighter. I do.

  The statistical selection unit 150 selects the movement of the object of interest based on the magnitude of the movement and the statistical model in order to remove the extra movement of the background, that is, movement other than the object of interest.

  A display unit 160 such as a screen of a mobile phone or a personal computer terminal displays an image of a portion selected by the statistical selection unit 150 among the images weighted by the time series weighting unit 140. For the time series images from the present to the past, the afterimage effect is produced in the displayed image by changing the gray value by weighting.

  Next, processing of the change area detection unit 120 and the time series weighting unit 140 will be described. FIG. 2 shows a state in which a weighted difference image 210 is calculated from the time-series image 200 stored in the data storage unit 110.

  The data accumulating unit 110 accumulates time-series images as much as an afterimage of about 5 seconds remains from the past to the present. FIG. 2 shows a time-series image 200 at times t and t−1 in which a change in facial expression of a person including the background is captured.

  The change area detection unit 120 reads the time-series images 200 that are temporally adjacent from the data storage unit 110 and calculates a difference image 210 based on the time difference.

  The time-series weighting unit 140 performs weighted addition so that the obtained difference image is clearly brighter when it is closer to the current time and darker as the past, and obtains a feature image in which an afterimage effect is produced at a location where the image has changed.

  The weighting function may be linear or non-linear as indicated by reference numerals 220 and 221 in FIG.

The image gray value (luminance) is defined as Img (i, j, t). Img (i, j, t) is an image gray value at the pixel (i, j) in the image at time t. The number of pixels of one image is N · M (for example, the number of vertical pixels is N and the number of horizontal pixels is M). However, 0 ≦ i ≦ N, 0 ≦ j ≦ M, and i and j are integers. The time difference takes the absolute value after taking the difference between the temporally adjacent images 200. This result is called a difference image. The difference image is calculated by the following equation (1).

Each difference image is weighted as shown by the following equation (2) to obtain a feature image used for display.

  Here, s indicates weighting. F is for normalization so that the sum of the weighted difference images falls within the display gradation (for example, within 8 bits). T is the number of past images to be used, and is the number of remaining afterimages for a desired time. The weighting function is, for example, s [t] = 1−t / T or s [t] = cos (πt / 2T).

  Next, the process of the statistical selection unit 150 will be described. The statistical selection unit 150 identifies the background noise and the target of interest from the time series image and selects only the target of interest. Discrimination between background noise and the object of interest is performed using the statistical properties of both. It takes advantage of the fact that the background noise has a low probability of appearing at the same position with the same luminance (tone value) continuously, and that the object of interest continuously appears with the same luminance. Statistically, the background noise has a Poisson distribution, and the object of interest is a Gaussian distribution.

  For this reason, the statistical selection unit 150 reads the time-series image from the data storage unit 110, and targets a portion where the same region (pixel) continues for a certain period of time with a certain luminance or more. On the other hand, even if the luminance is above a certain level, the background noise is a portion where the luminance fluctuation range is large, or a portion where the continuous appearance time in the same region (pixel) is short. In addition, since the luminance of the object of interest also fluctuates within a certain range, 3 to 5 out of 8-bit gradations are set for each pixel as a luminance fluctuation range, and within a certain image area (radius 1 to 10). Whether or not the amount of change in (pixel) is an object of interest is identified.

  FIG. 3 shows a change in luminance value in a certain pixel. The pixel indicated by reference numeral 300 has little variation in luminance value, and has continued for a certain time or more, so that it is identified as an object of interest. On the other hand, the pixel denoted by reference numeral 310 is identified as background noise because the luminance value fluctuates up and down and the duration is short.

  FIG. 4 shows a state in which background noise is removed from an image 400 including background noise. As described above, when there is an excessive motion with a large luminance change in the background, it is emphasized in the feature image obtained by weighted addition of the difference image, but can be removed as background noise by the statistical selection unit 150.

  Next, effects of the image processing apparatus according to the present embodiment will be described. 5 and 6 show the evaluation results with and without applying this method. Using various time-series images such as facial expression changes and natural phenomenon changes, we conducted experiments to evaluate the average amount of information and the visibility regarding subjective difficulty. As a result, as shown in FIG. 5, when the amount of image information without processing is 1.0, the image information amount is relatively about 20% according to this method.

  As shown in FIG. 6, when the visibility was evaluated, when the original image that was not compressed was the best, the subjective evaluation was low because it was difficult to see without processing. On the other hand, according to the present method, there was almost no problem. Only a part of the meaningful movement is displayed, so it did not give the best results, but stayed within the valid range. From this, it was confirmed that this method is an image processing method optimum for low bit rate communication.

  Therefore, according to the present embodiment, the afterimage effect is obtained by calculating the difference image by the change region detection unit 120 and the time-series weighting unit 140 performing weighted addition so that the weighting coefficient increases from the past to the present. A feature image created can be obtained. In addition, since the statistical selection unit 150 selects a portion of interest in which the luminance variation of the pixel is gradual and appears as an object of interest, unnecessary information such as background noise can be removed and the amount of image information can be reduced. Even at a low bit rate, it becomes possible to communicate the change of the situation at high speed.

  It should be noted that the present invention can be widely applied not only to facial expression changes, but also to natural environment changes and motion changes.

It is a block diagram which shows the structure of the image processing apparatus in one embodiment. It is the schematic which shows the process which the said image processing apparatus calculates a difference image from a time-sequential image, and performs weighting. It is a figure which shows the example of a change of the luminance value of a pixel. It is a schematic diagram which shows a mode that background noise is removed. It is a graph which shows the difference in the amount of image information by the system of a conventional system and this Embodiment. It is a graph which shows the difference in visibility by the system of a conventional system and this embodiment. It is a figure which shows the block noise of the block image in MPEG etc.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Data input part 110 ... Data storage part 120 ... Change area detection part 130 ... Difference data storage part 140 ... Time series weighting part 150 ... Statistical selection part 160 ... Display part

Claims (3)

In order to communicate situation changes at a bit rate lower than the bit rate that can secure the bandwidth for transmitting the total amount of image information of the time-series image data, the important feature amount of interest is extracted from the time-series image data. An image processing method to detect,
Inputting time-series image data by the input means, and storing the time-series image data in the storage means;
A step of reading the time series image data from the storage means by a change area detection means, obtaining a difference image by calculating a difference between the time series image data, and storing the difference image in the difference data storage means;
The time series weighting means reads the difference images from the difference data storage means in chronological order, and linearly or nonlinearly adds the weighting coefficient to the luminance of the pixels of the difference image so that the weighting coefficient increases from the past to the present. Obtaining a feature image by:
The statistical selection means reads out the time-series image data from the storage means, detects a variation in luminance of the pixels of the time-series image data, and a luminance of a predetermined value or more is within a predetermined fluctuation range for a predetermined time The pixel that appears continuously is the target of interest, and the pixel whose luminance fluctuation range is larger than the predetermined fluctuation range, or the time when the luminance of a predetermined value or higher continues to appear is shorter than the predetermined time period. Identifying a pixel as background noise, removing unnecessary information by removing the background noise from the feature image, and obtaining a transmission target image with a reduced amount of information ;
An image processing method comprising: In order to communicate situation changes at a bit rate lower than the bit rate that can secure the bandwidth for transmitting the total amount of image information of the time-series image data, the important feature amount of interest is extracted from the time-series image data. An image processing device to detect,
An input means for inputting time-series image data;
Storage means for storing the input time-series image data;
A change area detecting unit that reads out the time-series image data from the storage unit and obtains a difference image by calculating a difference between the time-series image data; and
Difference data storage means for storing the difference image;
The difference image is read from the difference data storage means in time series order, and a characteristic image is obtained by performing weighted addition linearly or nonlinearly so that the weighting coefficient increases from the past to the present with respect to the luminance of the pixels of the difference image. Time series weighting means;
Pixels that read out the time-series image data from the storage means, detect a variation in luminance of the pixels of the time-series image data, and have a luminance of a predetermined value or more appearing continuously for a predetermined time within a predetermined variation range And a pixel whose luminance fluctuation range is larger than the predetermined fluctuation range, or a pixel whose luminance is continuously higher than a predetermined value is shorter than the predetermined time, is identified as background noise. And statistical selection means for obtaining a transmission target image in which unnecessary information is removed by removing the background noise from the feature image to reduce the amount of information ,
An image processing apparatus comprising:   An image processing program for causing a computer to execute each step in the image processing method according to claim 1.