JP7540862B2 - Image processing device and image processing system - Google Patents

Description

The present invention relates to an image processing device and an image processing system.

There is known an image processing device that expands digitally compressed imaging data acquired from an imaging device via a network, and detects the movement of a subject captured by the imaging device using the expanded imaging data (see, for example, Patent Document 1).

Patent No. 4325596

Conventional image processing devices constantly detect the movement of a subject based on the difference between adjacent frames, resulting in a large processing load. Image processing devices in surveillance camera systems must perform processes to expand acquired imaging data, detect the movement of the subject, and output data based on the acquired imaging data to another device such as a display device, so there is a demand to reduce the processing load of the process of detecting the movement of the subject.

Therefore, the present invention has been made in consideration of these points, and aims to provide an image processing device and an image processing system that can reduce the processing load of detecting the movement of a subject.

The image processing device according to the first aspect of the present invention has an image data acquisition unit that acquires compressed video data including I frame data, B frame data, and P frame data, an I frame extraction unit that extracts a plurality of the I frame data from the compressed video data, a motion detection unit that detects the motion of a subject included in the compressed video data based on the plurality of I frame data extracted by the I frame extraction unit, a compression unit that further compresses the compressed video data to generate recompressed data, and a data output unit that outputs the recompressed data and timing data in the recompressed data indicating the motion detection timing at which the motion detection unit detected the motion.

The data output unit may output the recompressed data associated with time data indicating a time, and the timing data indicating a time at which the motion detection unit detected the motion.

The data output unit may output feature data indicating the characteristics of the subject in which the motion was detected, in association with the image frame in which the motion was detected in the recompressed data.

The motion detection unit may detect the motion based on the plurality of I frame data separated by a time interval longer than a predetermined time interval.

The motion detection unit may execute the motion detection process when I-frame data equal to or greater than a threshold is included within a predetermined period, and may stop execution of the motion detection process when I-frame data less than the threshold is included within the predetermined period.

The image processing system according to the second aspect of the present invention includes a plurality of imaging devices and an image processing device that recompresses compressed video data received from the plurality of imaging devices. Each of the plurality of imaging devices has an image generation unit that generates captured image data, a first compression unit that generates the compressed video data by compressing the captured image data, and a transmission unit that transmits the compressed video data in association with imaging device identification information for identifying the imaging device. The image processing device has an image data acquisition unit that acquires compressed video data including I frame data, B frame data, and P frame data, an I frame extraction unit that extracts a plurality of I frame data from the compressed video data, a motion detection unit that detects the motion of a subject included in the compressed video data based on the plurality of I frame data extracted by the I frame extraction unit, a second compression unit that further compresses the compressed video data to generate recompressed data, and a data output unit that outputs the recompressed data and timing data in the recompressed data indicating the motion detection timing at which the motion detection unit detected the motion.

The present invention has the advantage of being able to reduce the processing load of detecting the movement of a subject.

1 is a diagram showing a configuration of an image processing system S according to an embodiment of the present invention. FIG. 1 is a diagram showing the configurations of an imaging device 1 and an image processing device 3. FIG. 11 is a flowchart showing the operation of a motion detection unit 35. 4 is a diagram showing the structure of data output by a data output unit 36. FIG.

[Configuration of image processing system S]
1 is a diagram showing a configuration of an image processing system S according to an embodiment of the present invention. The image processing system S has a plurality of imaging devices 1 (imaging devices 1a, 1b, 1c), a display device 2, and an image processing device 3. The image processing system S is, for example, a surveillance system in which the display device 2 displays images captured by the plurality of imaging devices 1.

The multiple imaging devices 1 and the image processing device 3 are connected by a network N1. The image processing device 3 and the display device 2 are connected by a network N2. The networks N1 and N2 are any network such as a local area network or the Internet. The networks N1 and N2 may be the same network.

The imaging device 1 is a camera capable of capturing moving images, such as a surveillance camera installed within a facility. The imaging device 1 generates compressed moving image data by compressing the captured moving images. The compressed moving image data is data encoded, for example, by MPEG (Moving Picture Experts Group), Motion JPEG (Joint Photographic Experts Group), or H.264. The imaging device 1 transmits the generated compressed moving image data to the image processing device 3 via the network N1.

The display device 2 is a device for displaying on a display the moving image data generated by expanding the compressed moving image data received from the image processing device 3. The display device 2 has, for example, a network video recorder, and can also store the moving image data acquired from the image processing device 3 via the network N2.

The image processing device 3 receives compressed video data from the multiple imaging devices 1 via the network N1. The image processing device 3 decompresses the received compressed video data and detects the movement of the subject based on the decompressed video data. The image processing device 3 generates recompressed data by recompressing the compressed video data received from the multiple imaging devices 1, and transmits the generated recompressed data to the display device 2 via the network N2. The recompression process is a process in which the compressed video data is decompressed and then compressed again, or a process in which the compressed video data is further compressed.

[Configuration of imaging device 1 and image processing device 3]
2 is a diagram showing the configurations of the imaging device 1 and the image processing device 3. The imaging device 1 has an imaging section 11, an image generation section 12, a first compression section 13, and a transmission section 14. The image processing device 3 has an image data acquisition section 31, an expansion section 32, a second compression section 33, an I frame extraction section 34, a motion detection section 35, and a data output section 36. The image data acquisition section 31, the expansion section 32, the second compression section 33, the I frame extraction section 34, the motion detection section 35, and the data output section 36 are realized by, for example, a processor, but may be realized by a circuit other than a processor.

The imaging unit 11 includes an imaging element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor, and a lens. The imaging unit 11 converts an optical signal input to the lens into an electrical signal, and then outputs the electrical signal to the image generation unit 12. The image generation unit 12 generates captured image data by converting the electrical signal acquired from the imaging unit 11 into digital data. The image generation unit 12 outputs the generated captured image data to the first compression unit 13.

The first compression unit 13 generates compressed video data by compressing the captured image data acquired from the image generation unit 12. The first compression unit 13 compresses the captured image data using, for example, MPEG. Specifically, the first compression unit 13 compresses the captured image data by converting the captured image data into an image frame (I frame) in which the image data is coded without using inter-frame prediction, or into an image frame (B frame or P frame) in which the image data is coded using inter-frame prediction. The first compression unit 13 outputs the generated compressed video data to the transmission unit 14.

The transmission unit 14 transmits the compressed video data acquired from the first compression unit 13 to the image processing device 3 in association with imaging device identification information for identifying the imaging device 1. By the transmission unit 14 associating the imaging device identification information for identifying the imaging device with the compressed video data, the image processing device 3 can identify the imaging device 1 that transmitted the compressed video data.

Next, the configuration of the image processing device 3 will be described. The image data acquisition unit 31 acquires compressed video data including I frame data, B frame data, and P frame data from the transmission unit 14 via the network N1. The image data acquisition unit 31 outputs the acquired compressed video data to the decompression unit 32 and the I frame extraction unit 34.

The decompression unit 32 generates decompressed image data, which is data obtained by decompressing the compressed video data by decompressing the compressed video data acquired from the image data acquisition unit 31. The decompression unit 32 outputs the decompressed image data to the second compression unit 33.

The second compression unit 33 further compresses the compressed video image data acquired from the imaging device 1 to generate recompressed data. Specifically, the second compression unit 33 generates recompressed data by compressing, for example, the decompressed image data acquired from the decompression unit 32. The second compression unit 33 may use a compression method different from that of the first compression unit 13. The second compression unit 33 outputs the generated recompressed data to the data output unit 36.

The second compression unit 33 generates recompressed data with a higher compression ratio than the compressed video data generated by the first compression unit 13. In this way, the second compression unit 33 generates recompressed data with a higher compression ratio than the compressed video data generated by the first compression unit 13, so that the image processing device 3 can transmit image data based on the captured image data generated by multiple imaging devices 1 to the display device 2 via the network N2.

The I frame extraction unit 34 extracts I frame data from the compressed video data acquired from the image data acquisition unit 31. The I frame extraction unit 34 extracts multiple I frame data included in the compressed video data. The I frame extraction unit 34 outputs the extracted multiple I frames to the motion detection unit 35. The multiple I frame data are, for example, one I frame data and the next I frame data following that I frame data. The multiple I frame data output to the motion detection unit 35 may be multiple I frame data in which other I frame data is included between them in the compressed video data.

The motion detection unit 35 detects the motion of a subject contained in the compressed video data generated by the first compression unit 13 based on the multiple I frame data extracted by the I frame extraction unit 34. The motion detection unit 35 detects motion based on multiple I frame data separated by a time longer than a predetermined time interval. The predetermined time interval here is, for example, one second or more. By the motion detection unit 35 detecting motion based on such multiple I frame data, the frequency of motion detection processing can be reduced, thereby suppressing the processing load on the image processing device 3.

When the motion detection unit 35 detects the motion of the subject, it outputs timing data indicating the time when the motion was detected to the data output unit 36. Furthermore, when the motion detection unit 35 detects the motion of the subject, it notifies the data output unit 36 of feature data indicating the features of the subject whose motion was detected. The feature data that the motion detection unit 35 notifies the data output unit 36 of is data indicating the shape, color, or size of the subject. For example, if the subject is a person wearing red clothes, the feature data is data indicating "a person wearing red clothes." The feature of the subject is notified to the data output unit 36 as, for example, character data representing the feature or coded data corresponding to the feature.

The data output unit 36 outputs the recompressed data acquired from the second compression unit 33 to the display device 2 via the network N2. The data output unit 36 also outputs timing data indicating the motion detection timing at which the motion detection unit 35 detected motion in the recompressed data to the display device 2 via the network N2. The data output unit 36 outputs the recompressed data and the timing data in association with each other. The data output unit 36 outputs, for example, the timing data including an image frame number, which is information for identifying an image frame in which motion was detected in the recompressed data.

The data output unit 36 may output the recompressed data associated with time data indicating the time, and timing data indicating the time at which the motion detection unit 35 detected the motion. The recompressed data associated with the time data is, for example, data in which time information is added to an image frame included in the recompressed data. In this way, the data output unit 36 outputs the recompressed data associated with the time data and the timing data indicating the time at which the motion was detected, so that the display device 2 can identify the image frame in the recompressed data that corresponds to the time data indicating the same time as the time indicated by the timing data.

Furthermore, the data output unit 36 may output feature data indicating the features of the subject whose movement has been detected by the motion detection unit 35 in association with the image frame in which the movement of the subject was detected in the recompressed data. For example, the data output unit 36 simultaneously outputs the feature data notified by the motion detection unit 35 and the image frame number in which the movement was detected. The data output unit 36 may output a range of numbers or a range of times corresponding to multiple image frames containing the subject whose movement has been detected in association with the feature data. By operating the data output unit 36 in this manner, the display device 2 can display a character string indicating the features of the subject while displaying an image in which the movement has been detected by the motion detection unit 35, making it easier for a person monitoring using the display device 2 to identify the subject whose movement has been detected.

[Operation of the motion detection unit 35]
3 is a flowchart showing the operation of the motion detection unit 35. The motion detection unit 35 acquires an I frame extracted from the compressed video data from the I frame extraction unit 34 (S11). The motion detection unit 35 determines whether a predetermined period has passed since the previous motion detection (S12).

When the motion detection unit 35 determines that the predetermined waiting time has not elapsed (NO in S12), it deletes the acquired I frame. When the motion detection unit 35 determines that the predetermined waiting time has elapsed (YES in S12), it starts calculating the number of I frames contained within the predetermined period (S13). The predetermined period is determined based on the minimum time required for a monitored object to pass through the imaging range of the imaging device 1, and is, for example, one second. When the motion detection unit 35 determines that the predetermined period contains I frame data equal to or greater than the threshold (YES in S14), it executes a motion detection process (S15). On the other hand, when the motion detection unit 35 determines that the predetermined period contains I frame data less than the threshold (NO in S14), it stops executing the motion detection process (S16).

When there is a large amount of change in the images contained in the compressed video data, the proportion of I frame data included in the compressed video data tends to be high. Therefore, by determining whether or not the motion detection unit 35 will perform motion detection processing based on the number of I frames within a specified period, the motion detection unit 35 can not perform motion detection processing when the probability of detecting the motion of the subject is low, and can perform motion detection processing when the probability of detecting the motion of the subject is high. Therefore, the load on the motion detection unit 35 can be reduced when the probability of detecting the motion of the subject is low.

When the motion detection unit 35 executes the motion detection process and detects the motion of the subject, it identifies the characteristics of the subject whose motion was detected (S17). Then, the motion detection unit 35 notifies the data output unit 36 of the time when the motion of the subject was detected and the identified characteristics of the subject (S18). If the motion detection unit 35 wishes to continue the motion detection process (NO in S19), it acquires an I frame again (S11). If the motion detection unit 35 does not wish to continue the motion detection process (YES in S19), it ends the process.

[Structure of Data Output by Data Output Unit 36]
4 is a diagram showing the structure of data output by the data output unit 36. The data P includes, for example, one or more frames of recompressed data. The header P1 is the header portion of the data P. The display device 2 identifies that it has received the data P by identifying the header P1. The time information P2 is timing data indicating the time at which the motion detection unit 35 detected the motion of the subject. The feature data P3 is data indicating the feature of the subject whose motion has been detected by the motion detection unit 35. The recompressed data P4 is the payload portion of the data P. The recompressed data P4 includes, for example, one or more frames of recompressed data.

The display device 2 records the data P output by the data output unit 36, for example, in a network video recorder that the display device 2 has. The display device 2 determines that, among the multiple image frames included in the recompressed data P4, motion has been detected in an image frame that corresponds to the same time as the time indicated by the timing data included in the time information P2. When information indicating the characteristics of the subject in which motion has been detected is included in the feature data P3, the display device 2 displays the characteristics indicated by the feature data by superimposing them on image frames of a predetermined period starting from the image frame corresponding to the time indicated by the timing data included in the time information P2. In this way, by the display device 2 displaying the characteristics of the subject by superimposing them on a moving image, it becomes easier for the user to recognize the subject whose motion has been detected by the motion detection unit 35.

When the display device 2 receives an operation from the user to check the time when movement was detected, it searches for data P whose time information P2 contains timing data, and expands and displays on the display the recompressed data P4 associated with the extracted data P. In addition, when the display device 2 receives an operation from the user to input features of the subject for which movement was detected, it searches for data P whose input features are included in feature data P3, and expands and displays on the display the recompressed data P4 associated with the extracted data P. By the display device 2 operating in this manner, the user can search for subjects that have moved and easily check images in which the subjects that have moved appear.

[Effects of the image processing device 3 according to this embodiment]
As described above, the image processing device 3 has an I-frame extraction unit 34 that extracts I-frames from compressed video data, and a motion detection unit 35 that detects the motion of a subject based on the extracted I-frames. The motion detection unit 35 does not constantly detect the motion of a subject based on the difference between adjacent frames, but detects the motion of a subject only when I-frame data included in the compressed video data is extracted, so that the processing load of the process of detecting the motion of a subject can be reduced.

Moreover, the motion detection unit 35 detects the motion of the subject based on the multiple I frame data extracted by the I frame extraction unit 34, and notifies the data output unit 36 of timing data indicating the time when the motion of the subject was detected and the characteristics of the detected subject. By the motion detection unit 35 operating in this manner, the image processing system S can display a moving image on the display device 2 so that the user can easily identify the subject detected by the motion detection unit 35.

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

Reference Signs List 1: Imaging device 2: Display device 3: Image processing device 11: Imaging section 12: Image generating section 13: First compression section 14: Transmission section 31: Image data acquisition section 32: Decompression section 33: Second compression section 34: I-frame extraction section 35: Motion detection section 36: Data output section

Claims (5)

an image data acquisition unit that acquires compressed video data including I frame data, B frame data, and P frame data;
an I frame extraction unit that extracts a plurality of I frame data from the compressed video data;
a motion detection unit that detects a motion of a subject included in the compressed video data based on the plurality of I frame data extracted by the I frame extraction unit;
a compression unit that further compresses the compressed video data to generate recompressed data;
a data output unit that outputs the recompressed data and timing data indicating a motion detection timing at which the motion detection unit detected the motion in the recompressed data;
having
the motion detection unit detects the motion based on the plurality of I frame data which include other I frame data therebetween and are spaced apart by a time longer than a predetermined time interval;
Image processing device. the data output unit outputs the recompressed data associated with time data indicating a time, and the timing data indicating a time at which the motion detection unit detected the motion.
The image processing device according to claim 1 . the data output unit outputs feature data indicating features of the subject from which the motion has been detected in association with the image frame from which the motion has been detected in the recompressed data.
3. The image processing device according to claim 1 or 2. the motion detection unit executes the motion detection process when the I frame data equal to or greater than a threshold is included within a predetermined period, and stops execution of the motion detection process when the I frame data less than the threshold is included within the predetermined period.
The image processing device according to claim 1 . a plurality of imaging devices; and an image processing device that recompresses compressed video data received from the plurality of imaging devices;
Each of the plurality of imaging devices includes:
an image generating unit that generates captured image data;
a first compression unit that generates the compressed video data by compressing the captured image data;
A transmission unit for transmitting the compressed video data;
having
The image processing device includes:
an image data acquisition unit that acquires compressed video data including I frame data, B frame data, and P frame data;
an I frame extraction unit that extracts a plurality of I frame data from the compressed video data;
a motion detection unit that detects a motion of a subject included in the compressed video data based on the plurality of I frame data extracted by the I frame extraction unit;
a second compression unit that further compresses the compressed video data to generate recompressed data;
a data output unit that outputs the recompressed data and timing data indicating a motion detection timing at which the motion detection unit detected the motion in the recompressed data;
having
the motion detection unit detects the motion based on the plurality of I frame data which include other I frame data therebetween and are spaced apart by a time longer than a predetermined time interval;
Image processing system.

Images