JP5565737B2 - Video event detection apparatus and operation method thereof - Google Patents

Description

  The present invention relates to a video event detection apparatus and a method of operating the same for detecting a timing at which a line starts to be written and / or a timing at which a line is erased at a place reflected in a video file with high accuracy.

  Non-Patent Document 1 is a prior art document that detects an action of “writing on a board” based on the amount of change in characters and figures written on a blackboard in a video of a lecture.

  The method of Non-Patent Document 1 is as follows.

  First, a frame image is sampled from the video at regular intervals, and a frame image F (n) is obtained by binarizing the difference between the (n−1) th frame image F (n−1) and the nth frame image F (n). ) Is determined for each frame image, and the teacher area is detected.

  Next, in order to obtain a lecturer erased image G (n) in which the lecturer area is erased from the frame image F (n), pixels other than the lecturer area of the frame image F (n) are transferred to the lecturer erased image G (n). The pixels of the previous instructor erased image G (n−1) corresponding to the instructor area of the frame image F (n) are transferred to the instructor erased image G (n).

  Next, a character difference image is obtained by binarizing the difference between the instructor erased image G (n) and the instructor erased image G (n−1). That is, a character area that has changed in the frame image F (n) and the frame image F (n−1) is detected.

  Finally, the amount of change in the number of character pixels in the character difference image is obtained, and it is determined from the amount of change whether the writing is performed.

Yudai Shinogi, Hironobu Fujiyoshi: Automatic Generation of Lecture Video Considering Viewer's Attention from High Resolution Video, Journal of the Institute of Image Information and Television Engineers, Vol.62, No.2, pp.240-246, 2008.

  In the prior art, since a region that greatly changes in a lecture video is a lecturer region, a moving region obtained by a difference between frame images is detected as a lecturer region. However, although the moving area (instructor area) can be generally detected from the difference between the frame images, it is difficult to detect it accurately. In particular, when there is little movement of the lecturer, it is extremely difficult to detect the lecturer area. In addition, an instructor erased image is generated after the instructor area is detected, and a region that has changed in the instructor erased image is obtained from the difference between the instructor erased image and the previous instructor erased image, and this is used as the character difference image. It is susceptible to noise such as lighting fluctuations and blurred characters. Due to these problems, the amount of change in the number of character pixels may not be obtained correctly, and the problem is that detection accuracy such as the start time of a board is low.

  The present invention has been made in view of the above problems, and the object of the present invention is to detect with high accuracy the timing at which a line starts to be written and / or the timing at which a line is erased at a location shown in a video file. It is an object of the present invention to provide a video event detection apparatus and an operation method thereof.

  In order to solve the above-mentioned problem, the first aspect of the present invention is a video event detection apparatus for detecting a timing at which a line starts to be written and / or a timing at which a line is erased at a location shown in a video file, A line erasure processing unit that erases a line for each sampling from each frame image sampled in the playback order from the video file and generates a line erased image, and foreground of a place reflected in the frame image for each sampling of each frame image A foreground region detection unit that generates a foreground region image indicating a region, and for each sampling of each frame image, generates a difference image between the frame image and a line erased image generated from the frame image, Generate a line image that is an image including a difference image background region corresponding to a region other than the region indicated by the corresponding foreground region image An image generation unit; and a line image storage unit for storing each line image, wherein the line image generation unit is generated for each frame image using a frame image reproduced before the frame image. When the line image is in the line image storage unit, the line image is read out, and the line image foreground area corresponding to the area indicated by the foreground area image corresponding to the frame image to be reproduced later is extracted from the line image. The line image foreground region is included in the line image corresponding to the frame image to be reproduced later, and the video event detection device further calculates a dose indicating the amount of lines in each line image, Based on the change in the dose in the playback order of each frame image, the timing at which a line starts to be written at the location shown in the video file and / or the timing at which the line is erased It is a solutions with a video event detection device according to claim comprising an event detector for detecting a grayed.

  For example, in the first aspect of the present invention, the event detection unit, for each of a plurality of frame images that are sequentially played back, has a frame image that is played back immediately before the target frame image. The dose increment corresponding to the frame image reproduced immediately before the plurality of frame images is calculated in advance by calculating the dose increment obtained by subtracting the dose corresponding to the frame image reproduced immediately before from the dose corresponding to the image. The dose increment corresponding to a frame image other than the frame image to be reproduced later than the threshold value that is greater than the threshold value is equal to or less than the threshold value, and the reproduction timing of the frame image to be reproduced first and the highest If the time difference between the playback timing of the frame image to be played back later is larger than a preset threshold value, The reproduction timing of the generated frame image is set as the timing at which the line starts to be written, and the dose reduction obtained by subtracting the dose corresponding to the frame image reproduced immediately after the frame image from the dose corresponding to a certain frame image in advance. If it is larger than the set threshold value, the reproduction timing of the frame image reproduced immediately after that is set as the timing at which the line is erased.

  For example, in the first aspect of the present invention, the event detection unit calculates a cumulative dose obtained by accumulating the doses in the order of reproduction of the frame images, and increases the cumulative dose corresponding to a plurality of frame images that are sequentially reproduced. The cumulative dose corresponding to the frame image reproduced immediately before the frame image from the cumulative dose corresponding to the frame image reproduced first in the plurality of frame images, the rate being greater than a preset threshold value. The frame image reproduced first from the accumulated dose corresponding to the frame image reproduced immediately after the frame image reproduced first and the cumulative dose increment reduced is not more than a preset threshold value If the cumulative dose increment obtained by subtracting the cumulative dose corresponding to is larger than the threshold value, the frame image reproduced immediately after the frame image reproduced first Is set in advance as the timing at which the line starts to be written, and the cumulative dose decrement is obtained by subtracting the cumulative dose corresponding to a frame image reproduced immediately after the frame image from the cumulative dose corresponding to a frame image. If it is greater than the threshold, the playback timing of the frame image that is played back immediately after that is the timing at which the line is erased.

  According to a second aspect of the present invention, there is provided an operation method of a video event detection device for detecting a timing at which a line starts to be written and / or a timing at which a line is erased at a location reflected in a video file, An erasure processing unit erases a line for each sampling from each frame image sampled in the playback order from the video file to generate a line erasure image, and the foreground region detection unit of the video event detection device samples the frame image Each time, a foreground region image indicating a foreground region of a place reflected in the frame image is generated, and the line image generation unit of the video event detection device generates the frame image and the frame for each sampling of the frame image. A difference image of the line-erased image generated from the image is generated, and the difference image is compared with the corresponding foreground region image. A line image that is an image including a difference image background area corresponding to a region other than the region shown is generated, the line image is stored in a line image storage unit provided in the video event detection device, and the line image generation unit is For each frame image, when a line image generated using a frame image reproduced prior to the frame image is in the line image storage unit, the line image is read and reproduced from the line image later. The video event detection device extracts a line image foreground area corresponding to the area indicated by the foreground area image corresponding to the frame image to be reproduced, includes the line image foreground area in the line image corresponding to the frame image to be reproduced later, and The event detection unit calculates a dose indicating the amount of the line in each line image, and based on the change of the dose in the reproduction order of each frame image, the video The solutions have a method of operation of the image event detection device, characterized in that the timing and / or line a line at the location starts written reflected in Airu detects the timing to be erased.

  For example, in the second aspect of the present invention, the event detection unit, for each of a plurality of sequentially reproduced frame images, has a frame image to be reproduced when there is a frame image to be reproduced immediately before the target frame image. The dose increment corresponding to the frame image reproduced immediately before the plurality of frame images is calculated in advance by calculating the dose increment obtained by subtracting the dose corresponding to the frame image reproduced immediately before from the dose corresponding to the image. The dose increment corresponding to a frame image other than the frame image to be reproduced later than the threshold value that is greater than the threshold value is equal to or less than the threshold value, and the reproduction timing of the frame image to be reproduced first and the highest If the time difference between the playback timing of the frame image to be played back later is larger than a preset threshold value, The reproduction timing of the generated frame image is set as the timing at which the line starts to be written, and the dose reduction obtained by subtracting the dose corresponding to the frame image reproduced immediately after the frame image from the dose corresponding to a certain frame image in advance. If it is larger than the set threshold value, the reproduction timing of the frame image reproduced immediately after that is set as the timing at which the line is erased.

  For example, in the second aspect of the present invention, the event detection unit calculates a cumulative dose obtained by accumulating the doses in the order of reproduction of the frame images, and increases the cumulative dose corresponding to a plurality of frame images that are sequentially reproduced. The cumulative dose corresponding to the frame image reproduced immediately before the frame image from the cumulative dose corresponding to the frame image reproduced first in the plurality of frame images, the rate being greater than a preset threshold value. The frame image reproduced first from the accumulated dose corresponding to the frame image reproduced immediately after the frame image reproduced first and the cumulative dose increment reduced is not more than a preset threshold value If the cumulative dose increment obtained by subtracting the cumulative dose corresponding to is larger than the threshold value, the frame image reproduced immediately after the frame image reproduced first Is set in advance as the timing at which the line starts to be written, and the cumulative dose decrement is obtained by subtracting the cumulative dose corresponding to a frame image reproduced immediately after the frame image from the cumulative dose corresponding to a frame image. If it is greater than the threshold, the playback timing of the frame image that is played back immediately after that is the timing at which the line is erased.

  According to the present invention, it is possible to detect with high accuracy the timing at which a line starts to be written and / or the timing at which a line is erased at a location reflected in a video file.

It is a functional block diagram which shows the structure of the video event detection apparatus which concerns on this Embodiment. It is a flowchart which shows the operation | movement method of a video event detection apparatus. FIG. 3A is an example of a frame image, and FIG. 3B is an example of a line erased image. FIG. 4A is an example of a binarized image, and FIG. 4B is a diagram showing the position of the blackboard in the line erased image. It is a figure which shows an example of the mode of the production | generation of a line image. FIG. 5 is a diagram for use in explaining the first embodiment related to an event detection unit 16; FIG. 10 is a diagram for use in explaining the second embodiment related to the event detection unit 16.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a functional block diagram showing the configuration of the video event detection apparatus according to the present embodiment. FIG. 2 is a flowchart showing an operation method of the video event detection apparatus.

  In FIG. 1, a video event detection device 1 is a device that detects the timing at which a line starts to be written or the timing at which a line is erased at a location shown in a video file. The place is, for example, a classroom, and the lines are written on the blackboard using chalk as letters and figures, for example. The line is erased by, for example, a blackboard eraser.

  The video event detection apparatus 1 includes a video acquisition unit 11 that reads a video file from a storage device (not shown), a line deletion processing unit 12 that deletes a line from a frame image of the video file and generates a line deletion image, and a frame image. A foreground region detection unit 13 that generates a foreground region image indicating a foreground region at a predetermined location, and a line image generation unit 14 that generates an image called a line image using a frame image, a line erased image, and a foreground region image. , A line image storage unit 15 for storing line images, and a timing for calculating the dose indicating the amount of lines in each line image and starting to write the above line based on the change in dose in the reproduction order of each frame image And an event detector 16 for detecting the timing at which lines are deleted. For example, the event detection unit 16 sets an index to the video file as the detection result. The detection result may be output by another method, for example, displayed on a display device (not shown).

  The video acquisition unit 11 reads a video file from a storage device (not shown), samples the frame images from the video file in the order of playback, and uses the sampled frame images as a line erasure processing unit 12, a foreground area detection unit 13, and a line image generation unit. 14 for output. A sampling speed indicating which frame image is to be sampled is determined in advance, and the video acquisition unit 11 performs sampling based on the sampling speed.

  In step 101 of FIG. 2, the video acquisition unit 11 performs sampling and output of the nth frame image.

  Returning to FIG. 1, the line erasure processing unit 12 erases a line from the frame image every time the frame image is output, that is, every sampling, that is, generates an image after erasure (referred to as a line erased image). The erased image is output to the foreground area detection unit 13.

  In step 102 in FIG. 2, the line erasure processing unit 12 generates and outputs a line erasure image corresponding to the nth frame image that is a target in each frame image.

  Generation of a line erased image (line elimination) can be realized by, for example, a smoothing process based on morphology. FIG. 3A is an example of a frame image, and a teacher writes lines such as characters and figures on the blackboard. When the maximum value filter is executed after applying the minimum value filter to the frame image, a line-erased image shown in FIG. 3B is obtained. The line erased image is an image in which lines such as characters are erased. FIG. 3 shows the result when a character with high lightness is written on a blackboard with low lightness. When a character is written with a black pen on a whiteboard, the minimum value is applied after the maximum value filter is applied. You just need to run the filter.

  Returning to FIG. 1, the foreground area detection unit 13 generates a foreground area image indicating a foreground area at a location shown in the frame image for each output of the frame image and the line-erased image, that is, for each sampling, and erases the line. The image and the foreground area image are output to the line image generation unit 14.

  The foreground area image is a binarized image showing the contour of the lecturer, for example, as shown in FIG.

  Here, the foreground area detection unit 13 stores, for example, the position of the blackboard when the position of the blackboard is known in advance, as shown in FIG. Color), a color area not similar to the color is detected as a foreground area, and a foreground area image indicating the area is generated. The color of the blackboard may be stored in advance.

  Information indicating the color of the blackboard (color information) may be obtained and stored as follows. For example, a color frequency distribution at a plurality of locations on the blackboard is obtained, and a color distribution having a frequency equal to or higher than a predetermined value is used as color information. Alternatively, a normal distribution in the color space of the blackboard color or a mixed normal distribution is used as the color information. Clustering is performed for colors at a plurality of locations on the blackboard, and information indicating the color of the center of the cluster is used as color information.

  The similarity may be determined based on the distance between the stored color of the blackboard (color information) and the color of the target position (color information).

  If the blackboard area is expected to be larger than the foreground area, the blackboard area is automatically detected, or the blackboard area is manually set, and the color (or color distribution) that appears frequently throughout the area ) May be the blackboard color (color distribution).

  In the above description, a method of separation based on color information is described. However, a color space such as Lab or HSV including brightness may be used.

  In step 103 of FIG. 2, the foreground area detection unit 13 determines whether or not the nth frame image is the first sampled frame image, and if so (YES), the line erasure image is determined in advance. The color at the selected position (blackboard color) is detected (step 104), and a foreground area image is generated based on the color and output together with the line erased image (step 105). On the other hand, if the n-th frame image is not the first sampled frame image (step 103: NO), a foreground area image is generated based on the color detected and stored in step 104, and the foreground area image is generated. Together with the output (step 105).

  Returning to FIG. 1, the line image generation unit 14 generates a line image for each output of the frame image, the line erased image, and the foreground area image, that is, for each sampling of the frame image, and the line image is stored in the line image storage unit 15 and Output to the event detector 16. The line image storage unit 15 stores a line image.

  As shown in FIG. 5, specifically, the line image generation unit 14 firstly outputs the output frame image (the frame image F (n) in the figure) and the line erased image generated from the frame image F (n). A difference image of (output line erased image: line erased image M (n) in the figure) is generated.

  Next, the line image generation unit 14 generates a binary image obtained by binarizing the difference image (binarized image B (n) in the figure).

  Next, the line image generation unit 14 extracts a corresponding region (referred to as a difference image background region) other than the region indicated by the output foreground region image from the binarized image B (n).

  Next, if there is no frame image to be reproduced prior to the frame image, that is, if the frame image is a frame image sampled first, the line image generation unit 14 includes the difference image background region and the foreground region. A line image (line image C (n) in the figure) having no characters or the like in the region corresponding to the image is generated.

  Next, the line image generation unit 14 stores the line image C (n) in the line image storage unit 15.

  On the other hand, if there is a frame image to be reproduced prior to the frame image, that is, if the frame image is not the first sampled frame image, the line image generation unit 14 determines the previous sampled frame image, that is, first. A line image generated using the reproduced frame image (line image C (n−1) in the figure) is read from the line image storage unit 15. That is, the line image storage unit 15 outputs the line image C (n−1) to the line image generation unit 14 in response to a request from the line image generation unit 14.

  Next, the line image generation unit 14 generates an area (line) corresponding to an area indicated by the output foreground area image (the foreground area image corresponding to the frame image F (n)) from the line image C (n−1). The image foreground area is extracted.

  Next, the line image generation unit 14 generates a line image (line image C (n) in the figure) including the difference image background area and the line image foreground area. The line image C (n) is stored in the line image storage unit 15 and used as the line image C (n−1) in the generation of the next line image.

  When the frame image is a frame image sampled first, the line image includes a difference image background region that is a part of the binarized image B (n).

  On the other hand, when the frame image is not the first sampled frame image, the image composed of the difference image background region and the line image foreground region is a line image. The line image includes a difference image background area.

  That is, for each sampling of each frame image, the line image generation unit 14 generates a difference image between the frame image and a line erased image generated from the frame image, and indicates the corresponding foreground region image of the difference image. A line image that is an image including a corresponding difference image background region other than the region to be generated is generated. The line image generation unit 14 reads out the line image, particularly when the line image generated using the frame image reproduced before the frame image is in the line image storage unit 15. Therefore, the line image foreground area corresponding to the area indicated by the foreground area image corresponding to the frame image reproduced later is extracted, and the line image foreground area is included in the line image corresponding to the frame image reproduced later. is there.

  In step 106 of FIG. 2, the line image generation unit 14 generates the difference image and the binarized image B (n) in FIG. In step 107, the line image generation unit 14 generates and outputs the line image C (n) in FIG. 5, for example. These are the processes when the frame image is not the first sampled frame image. Otherwise, the illustration is omitted.

  Returning to FIG. 1, the event detection unit 16 calculates a dose indicating the amount of lines in each line image output for each sampling, and generates a video file based on the change in the dose in the reproduction order of each frame image. It detects when the line starts to be written at the reflected location and when the line is erased.

  In other words, the event detection unit 16 detects, for example, the timing at which a lesson in the classroom is started and characters and the like start to be written on the blackboard, and the timing at which the lesson is completed and characters and the like are erased from the blackboard.

  The event detection unit 16 may be configured to detect one timing.

  In step 108 of FIG. 2, the event detector 16 calculates the dose of the line image as described above. In step 109, the event detection unit 16 determines whether the reproduction timing of the corresponding frame image is the timing at which the line starts to be written based on the change in dose in the reproduction order of each frame image. If it is time to start writing (YES), in step 110, an index to that effect is set in the video file. If it is not the timing when the line starts to be written (NO), in step 111, the event detection unit 16 determines whether the reproduction timing of the corresponding frame image is the timing when the line is erased. If it is the timing at which the line is erased (YES), the event detection unit 16 sets an index to that effect in the video file in step 112. If the process of step 112 is completed, or if it is determined in step 111 that the reproduction timing of the corresponding frame image is not the timing for erasing the line (NO), the control returns to step 101 and the next The same processing is performed for the frame image and the like. Note that the processing of the flowchart in FIG. 2 ends when processing of the last sampled frame image or the like is completed.

  Here, an example of timing detection will be described.

  That is, when there is a frame image to be reproduced immediately before the target frame image for each of the plurality of frame images to be reproduced sequentially (target frame image), the event detection unit 16 relates to the target frame image. The dose increment is calculated by subtracting the dose corresponding to the frame image reproduced immediately before from the dose corresponding to. The event detection unit 16 corresponds to a frame image other than the frame image to be reproduced most recently, and the dose increment corresponding to the frame image to be reproduced most recently among the plurality of frame images is greater than a preset threshold value. A time difference between the reproduction timing of the frame image reproduced first and the reproduction timing of the frame image reproduced most recently is set in advance. If it is larger than the threshold, the reproduction timing of the frame image reproduced most recently is set as the timing at which a line starts to be written.

  Further, the event detection unit 16 determines that the dose decrement obtained by subtracting the dose corresponding to the frame image reproduced immediately after the frame image from the dose corresponding to a certain frame image is larger than a preset threshold value. The reproduction timing of the frame image reproduced immediately after is set as the timing at which the line is deleted.

  The dose is, for example, the number of pixels in the line area in the line image. The dose is, for example, the number of pixels that have changed between the line image for which the dose is obtained and the line image corresponding to the frame image reproduced immediately before the frame image corresponding to the line image. The presence or absence of a change is determined by comparing similarities using pixels in the vicinity of the target pixel.

  FIG. 6 is a diagram for explaining the first embodiment.

  The bar graph of FIG. 6 is a graph showing the dose increment and the dose decrement for each frame image. Although the dose decrement is a positive number, for the sake of convenience, a graph in the negative direction is used here.

  The threshold value Th1 is a preset threshold value for dose increment, and the threshold value Th2 is a preset threshold value for dose decrement.

  The video file includes, for example, a chapter CH1 that is a part up to the timing at which a line starts to be written, a chapter CH3 that is a part from when a line starts to be written until it is deleted, and a chapter that is a part after the line is deleted. It consists of CH3. However, it is assumed that the video file does not include the index ID2 indicating the start timing of the chapter CH2, that is, the former timing. Also, the video does not include the index ID3 indicating the start timing of the chapter CH3, that is, the latter timing.

  For the first frame image of chapter CH1 (n = 1), there is no frame image to be reproduced earlier, so the dose increment and dose decrement cannot be calculated and the graph is not displayed.

  In chapter CH1, the dose increment obtained by subtracting the dose corresponding to the frame image (n = 1) from the dose corresponding to the subsequent frame image (n = 2) is, for example, before the line starts to be written. It is smaller than the threshold value Th1. Similarly, for each frame image (n = 3, 4, 5), the dose increment is equal to or less than the threshold value Th1.

  However, for the first frame image (n = 6) of chapter CH2, the line starts to be written, so that the dose increment becomes larger than the threshold value Th1, for example.

  That is, for each frame image obtained by removing a frame image (n = 1) from a plurality of sequentially reproduced frame images (n = 1, 2, 3, 4, 5, 6), the dose corresponding to the frame image is used. A dose increment is calculated by subtracting the dose corresponding to the frame image reproduced immediately before the frame image.

  Further, the dose increment corresponding to the frame image (n = 6) reproduced most recently in the frame image (n = 1, 2, 3, 4, 5, 6) is larger than the threshold value Th1.

  The dose increment corresponding to the frame image (n = 2, 3, 4, 5) is equal to or less than the threshold value Th1.

  Further, here, if the time difference between the playback timing of the frame image (n = 1) and the playback timing of the most recently played frame image (n = 6) is greater than a preset threshold value T1. The event detection unit 16 sets the reproduction timing of the frame image (n = 6) reproduced most recently as the timing at which a line starts to be written, and sets, for example, the index ID2 to the video file.

  Now, for the frame image (n = 7, 8, 11, 12), for example, the dose increment becomes larger than the threshold Th1, but for the frame image (n = 7, 8, 12), Since the dose increment corresponding to the frame image is larger than the threshold value Th1, the reproduction timing of the frame image (n = 7, 8, 12) is not the timing at which the line starts to be written. For the frame image (n = 11), the dose increment corresponding to the immediately preceding frame image (n = 9, 10) is equal to or less than the threshold Th1, but the reproduction timing of the frame image (n = 9) and the frame image The time difference from the reproduction timing of (n = 11) is equal to or less than the threshold value T1. Therefore, the reproduction timing of the frame image (n = 11) is not the timing at which the line starts to be written.

  Thereafter, for the first frame image (n = 15) of chapter CH3, the line is deleted, so that the dose decrement is larger than the threshold value Th2, for example. Therefore, the event detection unit 16 sets the reproduction timing of the frame image (n = 15) as the timing at which the line is deleted, and sets, for example, the index ID3 to the video file.

  Next, another example of timing detection will be described.

  The event detection unit 16 first calculates a cumulative dose obtained by accumulating doses in the order of reproduction of each frame image.

  In addition, the event detection unit 16 has a cumulative dose increase rate corresponding to a plurality of sequentially reproduced frame images that is greater than a preset threshold value, and the frame that is reproduced first in the plurality of frame images. A frame to be reproduced first when the cumulative dose increment obtained by subtracting the cumulative dose corresponding to the frame image reproduced immediately before the frame image from the cumulative dose corresponding to the image is equal to or less than a preset threshold value. If the cumulative dose increment obtained by subtracting the cumulative dose corresponding to the frame image reproduced first from the cumulative dose corresponding to the frame image reproduced immediately after the image is larger than the threshold, the first reproduction is performed. The reproduction timing of the frame image reproduced immediately after the frame image to be reproduced is set as the timing at which the line starts to be written.

  Further, the event detection unit 16 has a cumulative dose decrement obtained by subtracting a cumulative dose corresponding to a frame image reproduced immediately after the frame image from a cumulative dose corresponding to a certain frame image, which is larger than a preset threshold value. Then, the reproduction timing of the frame image that is reproduced immediately after that is the timing at which the line is erased.

  For example, the dose is calculated in the same manner as in the first embodiment.

  FIG. 7 is a diagram for use in explaining the second embodiment, and this bar graph shows the cumulative dose for each frame image.

  As shown in FIG. 7A, for the first sampled frame image (n = 1), for example, before the line starts to be written, the accumulated dose is zero.

  The cumulative dose corresponding to the frame image (n = 2) is obtained by adding the dose corresponding to the frame image (n = 2) to the cumulative dose (0) corresponding to the frame image (n = 1). . For example, the cumulative dose increment (accumulated dose for frame image (n = 2) minus the cumulative dose (0) for frame image (n = 1)) is before the line begins to be written. It is below the threshold value Th3 for the dose increment. Therefore, also for the frame images (n = 3, 4, 5), the cumulative dose increment is similarly equal to or less than the threshold value Th3.

  The slope of the straight line 711 indicates the average increase rate of these accumulated doses, and is equal to or less than a threshold value for a preset increase rate.

  Therefore, the reproduction timing of each frame image is not the timing at which a line starts to be written.

  As shown in FIG. 7B, for the frame image (n = 6), for example, after the line starts to be written, the accumulated dose increases. For example, the cumulative dose increment for the frame image (n = 6) is greater than the threshold value Th3.

  Similarly, for the frame image (n = 7, 8), for example, after the line starts to be written, the cumulative dose increment becomes larger than the threshold value Th3.

  Further, the slope of the straight line 712 indicates the average increase rate of these accumulated doses, and is larger than the threshold value for the increase rate described above.

  Further, as described above, the cumulative dose increment for the frame image (n = 5) is less than or equal to the threshold Th3, and the cumulative dose increment for the frame image (n = 6) is greater than the threshold Th3.

  Therefore, the event detection unit 16 sets the playback timing of the frame image (n = 6) as the timing to start writing, and sets the index to the video file, for example, as in the first embodiment.

  As shown in FIG. 7C, for example, for the frame image (n = 15), since the line is deleted, the cumulative dose decrement (from the cumulative dose corresponding to the frame image (n = 14) to the frame The image (those obtained by subtracting the cumulative dose corresponding to n = 15) is, for example, larger than the threshold value Th4 for the cumulative dose decrement. Therefore, the event detection unit 16 sets the playback timing of the frame image (n = 15) as the timing at which the line is erased, and sets the index in the video file as in the first embodiment, for example.

  As described above, according to the present embodiment, the video event detection device 1 detects the timing at which a line starts to be written and / or the timing at which the line is erased at a location shown in the video file. A line erasure processing unit 12 that erases a line for each sampling from each frame image sampled in the playback order from the video file and generates a line erasure image, and is reflected in the frame image for each sampling of each frame image. Foreground region detection unit 13 that generates a foreground region image indicating a foreground region of a place, and for each sampling of each frame image, generates a difference image between the frame image and a line erased image generated from the frame image, An image that includes a difference image background area other than the area indicated by the corresponding foreground area image of the difference image. A line image generation unit 14 that generates a line image and a line image storage unit 15 that stores each line image. The line image generation unit 14 generates a frame that is reproduced before each frame image. When a line image generated using an image is stored in the line image storage unit 15, the line image is read out, and corresponds to the area indicated by the foreground area image corresponding to the frame image reproduced later from the line image. The line image foreground area is extracted, and the line image foreground area is included in the line image corresponding to the frame image to be reproduced later. The video event detection apparatus 1 further indicates the amount of lines in each line image. Calculate the dose and based on the change of the dose in the playback order of each frame image, the timing when the line starts to be written at the location shown in the video file and / or the line is erased It comprises an event detection unit 16 for detecting the timing.

  For example, when there is a frame image to be reproduced immediately before the target frame image for each of a plurality of frame images to be sequentially reproduced, the event detection unit 16 calculates the relevant dose from the dose corresponding to the target frame image. Calculate the dose increment by subtracting the dose corresponding to the frame image reproduced immediately before, the dose increment corresponding to the frame image reproduced most recently in the plurality of frame images is greater than a preset threshold value, and The dose increment corresponding to the frame image other than the frame image to be reproduced later is equal to or less than the threshold value, and the reproduction timing of the frame image reproduced first and the reproduction of the frame image reproduced most recently. If the time difference from the timing of the frame is greater than a preset threshold value, The raw timing is the timing when the line starts to be written, and a threshold value is set in which a dose decrement is obtained by subtracting the dose corresponding to a frame image reproduced immediately after the frame image from the dose corresponding to a certain frame image. If it is larger, the reproduction timing of the frame image reproduced immediately after that is the timing at which the line is erased, so the timing at which the line starts to be written and / or the line is erased at the location shown in the video file Can be detected with high accuracy.

  In addition, for example, the event detection unit 16 calculates a cumulative dose obtained by accumulating doses in the order of reproduction of each frame image, and an increase rate of the cumulative dose corresponding to a plurality of frame images to be sequentially reproduced is set in advance. A cumulative dose increment that is larger than the threshold value and is obtained by subtracting the cumulative dose corresponding to the frame image reproduced immediately before the frame image from the cumulative dose corresponding to the frame image reproduced first in the plurality of frame images in advance. The cumulative dose corresponding to the frame image reproduced first is subtracted from the cumulative dose corresponding to the frame image reproduced immediately after the earliest frame image that is not more than the set threshold value. If the cumulative dose increment is greater than the threshold value, the timing of reproduction of the frame image reproduced immediately after the frame image reproduced first is If the cumulative dose reduction obtained by subtracting the cumulative dose corresponding to a frame image reproduced immediately after the frame image from the cumulative dose corresponding to a certain frame image is greater than a preset threshold value Since the playback timing of the frame image that is played back immediately after that is the timing at which the line is erased, the timing at which the line starts to be written and / or the timing at which the line is erased at the location shown in the video file is highly accurate. Can be detected.

  A computer program for causing a computer to function as the video event detection apparatus 1 can be recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic disk, an optical disk, a magneto-optical disk, a magnetic tape, or a communication such as the Internet. It can be distributed widely through the network.

DESCRIPTION OF SYMBOLS 1 Image | video event detection apparatus 11 Image | video acquisition part 12 Line erasure process part 13 Foreground area | region detection part 14 Line image generation part 15 Line image memory | storage part 16 Event detection part

Claims (7)

A video event detection device for detecting when a line starts to be written and / or when a line is erased in a place reflected in a video file,
A line erasure processing unit that erases a line for each sampling from each frame image sampled in the reproduction order from the video file and generates a line erasure image;
A foreground region detection unit that generates a foreground region image indicating a foreground region of a place reflected in the frame image for each sampling of the frame image;
For each sampling of each frame image, a difference image between the frame image and a line erased image generated from the frame image is generated, and a difference image corresponding to a region other than the region indicated by the corresponding foreground region image of the difference image A line image generation unit that generates a line image that is an image including a background region;
A line image storage unit for storing each line image,
The line image generation unit
For each frame image, when a line image generated using a frame image reproduced prior to the frame image is in the line image storage unit, the line image is read and reproduced from the line image later. A line image foreground area corresponding to the area indicated by the foreground area image corresponding to the frame image to be reproduced, and including the line image foreground area in the line image corresponding to the frame image to be reproduced later,
The video event detection device further includes:
A dose indicating the amount of lines in each line image is calculated, and based on the change of the dose in the reproduction order of each frame image, the timing at which lines start to be written at the locations shown in the video file and / or lines A video event detection apparatus comprising: an event detection unit that detects a timing at which the video is deleted. The event detection unit
For each of a plurality of frame images that are sequentially played back, if there is a frame image that is played back immediately before the target frame image, a dose corresponding to the target frame image is changed to a frame image played back immediately before the target frame image. The dose increment obtained by subtracting the corresponding dose is calculated, and the dose increment corresponding to the frame image reproduced most recently in the plurality of frame images is larger than a preset threshold value and the frame image reproduced most recently The time difference between the reproduction timing of the frame image to be reproduced first and the reproduction timing of the frame image to be reproduced most recently is the dose increment corresponding to a frame image other than Is greater than a preset threshold, the line indicates the playback timing of the frame image to be played back last. The timing that is begin,
If the dose decrement obtained by subtracting the dose corresponding to a frame image reproduced immediately after the frame image from the dose corresponding to a certain frame image is larger than a preset threshold value, the frame image reproduced immediately after the frame image The video event detection device according to claim 1, wherein the reproduction timing is a timing at which the line is erased. The event detection unit
Calculate a cumulative dose by accumulating the dose in the order of reproduction of each frame image,
The rate of increase in cumulative dose corresponding to a plurality of frame images that are sequentially reproduced is greater than a preset threshold value, and the cumulative dose corresponding to the frame image that is reproduced first in the plurality of frame images A frame that is reproduced immediately after the earliest frame image that has a cumulative dose increment obtained by subtracting the cumulative dose corresponding to the frame image that is reproduced immediately before the frame image is equal to or less than a preset threshold value. If the cumulative dose increment obtained by subtracting the cumulative dose corresponding to the earliest frame image to be reproduced from the cumulative dose corresponding to the image is greater than the threshold value, it is reproduced immediately after the earliest frame image to be reproduced. The frame image playback timing is the timing when the line starts to be written,
If the cumulative dose deduction obtained by subtracting the cumulative dose corresponding to the frame image reproduced immediately after the frame image from the cumulative dose corresponding to a certain frame image is larger than a preset threshold value, it is reproduced immediately after that. The video event detection device according to claim 1, wherein the reproduction timing of the frame image is a timing at which the line is deleted. An operation method of a video event detection device for detecting a timing at which a line starts to be written and / or a timing at which a line is erased at a location reflected in a video file,
The line erasure processing unit of the video event detection device generates a line erasure image by erasing a line for each sampling from each frame image sampled in the playback order from the video file,
The foreground area detection unit of the video event detection device generates a foreground area image indicating a foreground area of a place reflected in the frame image for each sampling of the frame images,
The line image generation unit of the video event detection device generates a difference image between the frame image and a line erased image generated from the frame image for each sampling of the frame images, and a corresponding foreground of the difference image Generating a line image that is an image including a difference image background region corresponding to a region other than the region indicated by the region image, and storing the line image in a line image storage unit provided in the video event detection device;
For each frame image, the line image generation unit reads the line image when the line image generated using the frame image reproduced before the frame image is in the line image storage unit, A line image foreground area corresponding to the area indicated by the foreground area image corresponding to the frame image reproduced later is extracted from the line image, and the line image foreground area is extracted from the line image corresponding to the frame image reproduced later. Including
The event detection unit of the video event detection apparatus calculates a dose indicating the amount of the line in each line image, and is reflected in the video file based on the change in the dose in the reproduction order of the frame images. A method of operating a video event detection apparatus, comprising: detecting a timing at which a line starts to be written and / or a timing at which a line is erased. The event detection unit
For each of a plurality of frame images that are sequentially played back, if there is a frame image that is played back immediately before the target frame image, a dose corresponding to the target frame image is changed to a frame image played back immediately before the target frame image. The dose increment obtained by subtracting the corresponding dose is calculated, and the dose increment corresponding to the frame image reproduced most recently in the plurality of frame images is larger than a preset threshold value and the frame image reproduced most recently The time difference between the reproduction timing of the frame image to be reproduced first and the reproduction timing of the frame image to be reproduced most recently is the dose increment corresponding to a frame image other than Is greater than a preset threshold, the line indicates the playback timing of the frame image to be played back last. The timing that is begin,
If the dose decrement obtained by subtracting the dose corresponding to a frame image reproduced immediately after the frame image from the dose corresponding to a certain frame image is larger than a preset threshold value, the frame image reproduced immediately after the frame image The operation method of the video event detection apparatus according to claim 4, wherein the reproduction timing is a timing at which the line is erased. The event detection unit
Calculate a cumulative dose by accumulating the dose in the order of reproduction of each frame image,
The rate of increase in cumulative dose corresponding to a plurality of frame images that are sequentially reproduced is greater than a preset threshold value, and the cumulative dose corresponding to the frame image that is reproduced first in the plurality of frame images A frame that is reproduced immediately after the earliest frame image that has a cumulative dose increment obtained by subtracting the cumulative dose corresponding to the frame image that is reproduced immediately before the frame image is equal to or less than a preset threshold value. If the cumulative dose increment obtained by subtracting the cumulative dose corresponding to the earliest frame image to be reproduced from the cumulative dose corresponding to the image is greater than the threshold value, it is reproduced immediately after the earliest frame image to be reproduced. The frame image playback timing is the timing when the line starts to be written,
If the cumulative dose deduction obtained by subtracting the cumulative dose corresponding to the frame image reproduced immediately after the frame image from the cumulative dose corresponding to a certain frame image is larger than a preset threshold value, it is reproduced immediately after that. The operation method of the video event detection apparatus according to claim 4, wherein the reproduction timing of the frame image is a timing at which the line is deleted.   A computer program for causing a computer to function as the video event detection device according to claim 1.