JP7754423B2 - Image processing device, method and program - Google Patents

Description

One aspect of the present invention relates to an image processing device, method, and program with video magnification (VM) processing functionality.

In the real world, minute changes in images of a subject can sometimes contain important messages. For example, differences in the way an athlete uses their muscles or joints can be recorded as minute changes in the image of the subject, and these minute changes can be used as one factor in judging the athlete's athletic performance. Furthermore, minute changes such as heartbeats, chest movement due to breathing, changes in the brightness of the color of the body surface, and vibrations from cranes or engines can be used as one factor in detecting abnormal conditions based on images. However, human vision has difficulty detecting minute changes in images.

As a result, VM processing technology has attracted attention. VM is a technology that visualizes minute changes in an image by detecting and highlighting these changes. For example, Non-Patent Documents 1, 2, and 3 describe technology that uses optical flow, the Euler method, or phase change to detect changes in the color or movement of a subject in an image, and then applies a time-series filter to detect minute changes in the subject in the image.

Ce Liu, Antonio Torralba, William T. Freeman, Fredo Durand, Edward H. Adelson. “Motion Magnification”. ACM Transactions on Graphics (2005). Yichao Zhang, Silvia L. Pintea, and Jan C. van Gemert. “Video Acceleration Magnification”. IEEE International Conference on Computer Vision and Pattern Recognition (2017). Shoichiro Takeda, Kazuki Okami, Dan Mikami, Megumi Isogai, Hideaki Kimata. “Jerk-Aware Video Acceleration Magnification”. IEEE International Conference on Computer Vision and Pattern Recognition (2018).

When analyzing the minute details of a subject based on VM-processed video, it is usually sufficient to analyze only the subject area. Therefore, a method can be considered in which the subject area is detected by subject tracking processing and VM processing is applied only to this subject area. However, this method leaves subject tracking noise in the video data after VM processing, resulting in degradation of the quality of the video data after VM processing.

The present invention has been made in light of the above circumstances, and aims to provide a technique that makes it possible to obtain high-quality VM-processed video data that is free of noise .

To solve the above problem, one aspect of the image processing device or method of the present invention acquires first video data containing an image of a subject in an image frame, detects minute changes in pixels from the acquired first video data, and generates second video data containing image components that have been enhanced by enhancing the detected minute changes in pixels. In parallel with the minute change enhancement process, the device also performs a process of detecting a limited area containing the image of the subject from the first video data as a subject area. Third video data corresponding to the detected subject area is then extracted from the second video data, and the extracted third video data is combined with the first video data to generate fourth video data, and the generated fourth video data is output.

In one aspect of the present invention, VM processing is applied to the entire region of a frame image of input video data or to a local region including a subject region, and the subject region is detected from the input video data separately from this VM processing, and VM-processed video data of the subject region is extracted from the video data after the VM processing based on information about the detected subject region, and the VM-processed video data of the extracted subject region is synthesized with the input video data. Therefore, for example, if a subject region is detected by subject tracking processing and VM processing is applied only to this subject region, noise due to the subject tracking processing may remain in the video data after the VM processing, but according to one aspect of the present invention , synthesized video data that does not include noise due to the subject region tracking processing can be obtained.

That is, according to one aspect of the present invention, it is possible to provide a technique that makes it possible to obtain high-quality VM-processed video data that does not contain noise .

FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the software configuration of the image processing apparatus according to the embodiment of the present invention. FIG. 3 is a flowchart showing an example of a processing procedure and processing contents of a series of image processing executed by the image processing apparatus shown in FIG. FIG. 4 is a flowchart showing an example of the processing procedure and processing contents of the VM processing in the image processing procedure shown in FIG. FIG. 5 is a flowchart showing an example of the processing procedure and processing contents of the subject tracking processing from the image processing procedure shown in FIG.

Embodiments of the present invention will be described below with reference to the drawings.

[One embodiment]
(Configuration example)
FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment of the present invention, and FIG. 2 is a block diagram showing an example of the software configuration of the image processing apparatus.

The image processing device VD is provided as one of the processing functions in an information processing device such as a server computer or a personal computer. A camera and input/output devices (not shown) are connected to the image processing device VD via, for example, a signal cable or a network.

The camera may be, for example, a high-resolution camera, and captures subjects such as a person's body during exercise, the movement of a living body such as the heart or chest, or machinery such as a crane or engine in operation, and outputs video data containing the image of the subject. In addition to the camera, an external storage device storing video data of the subject may also be connected to the image processing device VD. The subject to be captured may also be various natural objects, animals, works of art, scenery, etc.

The input/output device may be, for example, a monitor device, and is used to display VM-processed video data output from the image processing device VD. In addition to the input/output device, the image processing device VD may also be connected to an external storage device or an information processing device for management or analysis.

The image processing device VD has a control unit 1 that uses a hardware processor such as a central processing unit (CPU), and this control unit 1 is connected via a bus 5 to a storage unit having a program storage unit 2 and a data storage unit 3, and an input/output interface (hereinafter, interface will be abbreviated as I/F) unit 4.

The input/output I/F unit 4 has a communication interface function and transmits and receives input video data and VM-processed video data between the above-mentioned cameras and input/output devices, etc., via a signal cable or network.

The program storage unit 2 is configured, for example, by combining a non-volatile memory such as a solid-state drive (SSD) (storage medium) that can be written to and read from at any time, with a non-volatile memory such as read-only memory (ROM), and stores middleware such as an operating system (OS), as well as application programs required to execute various control processes according to one embodiment. Hereinafter, the OS and each application program will be collectively referred to as the "program."

The data storage unit 3 is, for example, a combination of non-volatile memory such as an SSD, which can be written to and read from at any time, and volatile memory such as RAM (Random Access Memory), and includes an input video storage unit 31, a VM processing video storage unit 32, a subject area detection information storage unit 33, and a composite video storage unit 34 as the main storage units required to implement one embodiment.

The input video storage unit 31 is used to store high-resolution input video data acquired from a camera or external storage device.

The VM processing video storage unit 32 is used to store input video data that has been VM processed.

The subject area detection information storage unit 33 is used to store information representing the subject area detected from the input video data before VM processing.

The composite image storage unit 34 is used to store composite image data in which only the subject area of the input image data is replaced with VM-processed image data.

The control unit 1 includes the processing functions necessary to implement one embodiment: an input video acquisition processing unit 11, a VM processing unit 12, a subject tracking processing unit 13, a video synthesis processing unit 14, and a synthesized video output processing unit 15. These processing units 11 to 15 are all realized by causing the hardware processor of the control unit 1 to execute application programs stored in the program storage unit 2.

Note that some or all of the above processing units 11 to 15 may be implemented using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

The input video acquisition processing unit 11 acquires input video data from a video source such as a camera or external storage device via the input/output I/F unit 4, and stores the acquired input video data in the input video storage unit 31.

The VM processing unit 12 reads input video data frame by frame from the input video storage unit 31 and performs VM processing on the image of each read frame. The VM processing unit 12 then stores the VM-processed input video data frame by frame in the VM processing video storage unit 32. An example of VM processing will be described in the operation example.

The subject tracking processing unit 13 performs subject tracking processing in parallel with the VM processing by the VM processing unit 12. It reads input video data frame by frame from the input video storage unit 31 and detects the subject area from each of the read frame images. The subject tracking processing unit 13 then stores information representing the subject area detected from each of the frame images in the subject area detection information storage unit 33 in frame order. An example of the subject area detection processing will be described in the operation example.

The video synthesis processing unit 14 extracts VM-processed video data of the subject area from the VM-processed video data stored in the VM-processed video storage unit 32 based on information representing the subject area stored in the subject area detection information storage unit 33, and synthesizes the VM-processed video data of the extracted subject area with the input video data stored in the input video storage unit 31 by matching the time position and coordinate position. The video synthesis processing unit 14 then stores the synthesized video data generated by the synthesis process in the synthesized video storage unit 34.

The composite video output processing unit 15 reads the composite video data from the composite video storage unit 34 and outputs the read composite video data from the input/output I/F unit 4 to an input/output device or information processing device (not shown).

(Example of operation)
Next, an example of the operation of the image processing device VD configured as above will be described.
FIG. 3 is a flowchart showing an example of the processing procedure and processing contents of the image processing operation executed by the control unit 1 of the image processing device VD.

(1) Acquisition of Input Video Data For example, when a target user starts exercising, a camera captures the target user's entire body or specific parts during the exercise, and the video data is output. The video data may consist of, for example, high-resolution color moving images, with each frame of the image containing brightness information and color information for each pixel. Note that the video data does not necessarily have to be high-resolution color images, and may also be monochrome images with normal resolution. Furthermore, the video data may be video data stored in advance in an external storage device other than a camera.

In response to this, in step S1, the control unit 1 of the image processing device VD, under the control of the input video acquisition processing unit 11, receives the video data output from the camera or external storage device as input video data via the input/output I/F unit 4. Then, the received input video data is stored in chronological order by frame in the input video storage unit 31.

(2) VM Processing Next, in step S2, the control unit 1 of the image processing device VD, under the control of the VM processing unit 12, reads the input video data from the input video storage unit 31 frame by frame, and performs VM processing on the image of each frame that has been read.

As an example of VM processing, we will explain the operation of the VM processing related to minute changes in movement, as described in Non-Patent Document 2. Figure 4 is a flowchart showing an example of the processing procedure and processing content.

For example, if each frame image of the input video data has pixel coordinates (x,y), frame time position t=1,...,T, and color space c∈C, then I^c In this case, the VM processing unit 12 performs VM processing on the entire image area of each frame image or on a local area (x, y) ∈ (X, Y) including the subject image as follows, and generates VM-processed video data I^^cGet (x,y,t).

That is, first, in step S21, the VM processing unit 12 extracts the Y color signal component ^Iy (x,y,t) from the frame image for each frame of the input video data ^Ic (x,y,t). Note that as a result of the extraction process of the Y color signal component ^Iy (x,y,t), ^Ii (x,y,t) and ^Iq (x,y,t) remain.

Next, in step S32, the VM processing unit 12 converts the Y color signal component I y _K (x,y,t) of the low-resolution video into an analytic signal representation divided into a plurality of frequency bands ω∈Ω and a plurality of directions θ∈Θ, using a group of filters called Complex Steerable _{Filters (hereinafter abbreviated as CFS) having filter characteristics ψ ω,θ} ⁽ x,y), as follows:

Here, R ^y _ω,θ (x,y,t) represents the analytic signal representation, A ^y _ω,θ (x,y,t) represents the amplitude signal representation, and φ ^y _ω,θ (x,y,t) represents the phase signal representation. Of these, the phase signal representation is known to represent local motion changes centered on the coordinate (x,y) in the video data. This is described in detail, for example, in Neal Wadhwa, Michael Rubinstein, Fredo Durand, William T. Freeman. "Phase-based Video Motion Processing". ACM Transactions on Graphics (2013).

Subsequently, in step S23, the VM processing unit 12 applies time-series filtering h(t;f t) to the phase signal representation φ ^y _ω,θ (x,y, _t ) using a predetermined arbitrary time frequency f _t , thereby generating a minute phase signal representation C^ ^y _ω,θ (x,y,t) at the time frequency f _t as follows, where * represents a convolution operation.

Subsequently, in step S24, the VM processing unit 12 amplifies the generated minute phase signal representation C^ ^y _ω,θ (x,y,t) using an arbitrary enhancement rate α, and then multiplies the generated phase signal representation φ ^y _ω,θ (x,y,t) by
As a result, we obtain the phase signal expression φ^ ^y _ω,θ (x,y,t) in which only the small phase signal expression is emphasized.

Next, in step S25, the VM processing unit 12 constructs an analytic signal representation to which VM processing has been applied, using the phase signal representation φ̂ ^y _ω,θ (x,y,t) in which only the minute phase signal representation is emphasized, as follows:

The VM processing unit 12 then uses the CSF to calculate the Y color signal image I^ after VM processing in step S26 using the above equation (5).^y Then, the VM processing unit 12 converts the inversely converted Y color signal image I^ into (x, y, t).^y (x, y, t) is the I remaining by the extraction process in step S21.ⁱ(x,y,t) ，I^q(x,y,t) and the VM processed image data I^^C Generate (x,y,t).

Finally, the VM processing unit 12 generates the generated VM-processed video data I^^C (x, y, t) is stored in the VM processing image storage unit 32. Note that the processing procedure and processing content of the VM processing are not limited to those described above, and any method can be applied.

(3) Subject Tracking Processing In parallel with the VM processing by the VM processing unit 12, the control unit 1 of the image processing device VD, under the control of the subject tracking processing unit 13, executes a process of tracking the image of the subject contained in the input video data in chronological order in step S3 as follows.

That is, the object tracking processing unit 13 reads input video data from the input video storage unit 31. Then, at each preset timing t, the object tracking processing unit 13 detects an object region (x, y, t) ∈ (X _obj , Y _obj , t) from the corresponding frame image of the input video data.

Figure 5 is a flowchart showing an example of the processing procedure and processing content of the subject detection process performed by the subject tracking processing unit 13.

As shown in FIG. 5, first in step S31, the subject tracking processing unit 13 detects a plurality (N) of joint points {(x _joint , y _joint , t)} ^N _i=1 of a subject, for example, a person, from the frame image at each timing t.

Next, in step S32, the subject tracking processing unit 13 extracts image areas from the frame images, each of which has a predetermined size and shape and is centered around the N detected joint points {(x _joint , y _joint , t)} ^N _i=1 . For example, a range (x, y, t) ∈ {N(x _joint , y _joint , t)} ^N _i=1 with a radius of M pixels, corresponding to the range of motion centered on a human joint, is extracted from the frame images. The extracted image area is then designated as the subject area (X _obj , Y _obj , t) ∈ {N(x _joint , y _joint , t)} ^N _i=1 .

The above-mentioned subject region detection process is described in detail in, for example, the following document.
Z. Cao, G. Hidalgo, T. Simon, S. Wei and Y. Sheikh, "OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields". IEEE Transactions on Pattern Analysis & Machine Intelligence, 2021.

In the above explanation, an example was given in which the range for extracting the subject region was set in advance to a range with a radius of M pixels. However, this is not limiting. For example, the joint length of the subject person could be calculated in pixel units from the input video data, and the value of M could be set based on the calculated joint length. In this way, it is possible to set an optimal radius M range that takes into account individual differences between subjects.

As a method for setting the subject region from the input video data, an image segmentation method or the like can be applied. The image segmentation method is described in detail in, for example, the following document.
Chuong Huynh, Anh Tuan Tran, Khoa Luu, and Minh Hoai. “Progressive Semantic Segmentation”, IEEE Transactions on Pattern Analysis & Machine Intelligence, 2021.

In addition, the subject area tracking method is not limited to the above method, and any method can be used depending on the type and shape of the subject. For example, if the subject is an object other than a human body, the length between the movable base points of this object is calculated. Furthermore, the range from which the subject area is extracted may be other shapes such as an oval or rectangle in addition to a circle.

Finally, the subject tracking processing unit 13 stores information representing the subject area detected as described above in the subject area detection information storage unit 33 in step S33.

(4) Video Composition Processing Next, in step S4, the control unit 1 of the image processing device VD, under the control of the video composition processing unit 14, executes processing to generate composite video data as follows.

That is, the video synthesis processing unit 14 first reads information representing the detected subject area from the subject area detection information storage unit 33, and then, based on this information representing the subject area, extracts VM-processed video data corresponding to the subject area from the VM-processed video data stored in the VM-processed video storage unit 32. The video synthesis processing unit 14 then synthesizes the VM-processed video data of the extracted subject area with the input video data stored in the input video storage unit 31, matching the time position and coordinate position.

The video synthesis processing unit 14 stores the video data obtained by the above synthesis process in the synthesized video storage unit 34. Thus, the above video synthesis process results in synthesized video data in which only the subject area of the input video data has been replaced with VM-processed video data.

When combining the VM-processed video data of the subject region with the input video data, pixel smoothing processing may be performed on the boundary between the VM-processed video data of the subject region and the input video data. For example, image processing techniques such as alpha blending may be used to perform image processing so that the boundary between the VM-processed video data of the subject region and the input video data is smoothly combined. In this way, the contours of the subject region that have been emphasized by VM processing are not lost, making it possible to generate more natural VM-combined video data.

(5) Output of composite video data Finally, in step S5, under the control of the composite video output processing unit 15, the control unit 1 of the image processing device VD reads out the composite video data in which only the subject area of the input video data has been replaced with VM-processed video data from the composite video memory unit 34, and outputs the read-out composite video data from the input/output I/F unit 4 to an input/output device not shown.

The input/output device then displays the composite video data on a monitor, for example. Therefore, by viewing the displayed VM-processed composite video, an administrator can clearly see even minute changes in the movements of the entire body or specific parts of the subject, for example, a target user exercising.

The composite video data may be transmitted to an external information processing device. In this case, the external information processing device can perform analysis processing, such as extracting features of minute changes in the subject by performing image processing on the composite video data, inputting the extracted features of minute changes into a learning model, and obtaining information representing the analysis results of the subject's movement features.

(effect)
As described above, in one embodiment, the image processing device VD performs VM processing on video data acquired from, for example, a camera or an external storage device, and in parallel with this, performs processing to detect a subject area from the input video data. Then, video data corresponding to the subject area is extracted from the VM-processed video data, and the VM-processed video data of the extracted subject area is synthesized with the input video data by matching the time position and coordinate position.

Therefore, according to one embodiment, the following effect is achieved. That is, if a subject area is detected by subject tracking processing and VM processing is applied only to the detected subject area, noise due to the subject tracking processing may remain in the video data after VM processing. In contrast, in one embodiment, VM processing is applied to the entire area of a frame image of input video data or a local area including the subject area, and the subject area is detected from the input video data separately from this VM processing. VM-processed video data of the subject area is extracted from the video data after VM processing based on information about the detected subject area, and the extracted VM-processed video data of the subject area is synthesized with the input video data. Therefore, high-quality synthesized video data that does not include noise due to the subject area tracking processing can be obtained.

Furthermore, in one embodiment, the subject area is tracked by detecting the positions of multiple joints of a person from input video data, and setting ranges with a predetermined radius M centered on each of the detected joint positions, so that the area surrounded by these ranges is determined to be the subject area. As a result, the subject area can be detected with simpler processing than when detecting the outline of the subject.

Furthermore, when setting the subject area, the length of the person's joints is detected from the input video data, and the radius M is set based on this joint length, making it possible to set an optimal radius M that takes into account individual differences between subjects.

Furthermore, when combining input video data with VM-processed video data of the subject area, image processing techniques such as alpha blending can be applied to smoothly combine the boundary between the subject area and the input video data, preventing the outline of the subject area that has been emphasized by VM processing from being lost, making it possible to generate natural VM-combined video data.

[Other embodiments]
(1) In one embodiment, the input video data acquisition process, VM process, subject area tracking process, video synthesis process, and synthesized video output process are performed in a single image processing device VD. However, the present invention is not limited to this. Part of the series of processes from the VM process, subject area tracking process, video synthesis process, super-resolution process, and video output process may be distributed among multiple information processing devices.

Furthermore, the image processing device does not need to be configured to exclusively perform a series of processes from acquiring input video data to outputting composite video data, but may also be configured to perform other processing functions, such as subject area analysis processing functions.

(2) In one embodiment, a series of processes from the acquisition of input video data to the output of composite video data is performed in real time. However, this is not limiting. For example, video data for the entire period or a portion of a person's exercise may be temporarily stored in the input video storage unit 31, and VM processing, subject area tracking processing, and video composition processing may be performed collectively on this stored video data.

Alternatively, all or part of the composite video data may be temporarily stored in the composite video storage unit 34, and in this state, when a request to acquire the composite video data is sent, for example, from an administrator's terminal device, the composite video data may be read from the composite video storage unit 34 and transmitted in bulk to an input/output device or an external information processing device, etc.

(3) In one embodiment, an example was described in which video data consisting of moving images was used to generate video data in which only the subject area was subjected to VM processing. However, video data in which only the subject area was subjected to VM processing may also be generated using image data consisting of still images.

(4) In addition, the functional configuration of the image processing device, the processing procedures and contents, the VM processing, the processing procedures and contents of the subject area tracking processing, and the video synthesis processing may be modified and implemented in various ways without departing from the spirit and scope of this invention.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations corresponding to the embodiments may be adopted as appropriate.

In short, this invention is not limited to the above-described embodiment, and can be embodied by modifying the components in the implementation stage without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiment. For example, some components may be omitted from all of the components shown in the embodiment. Furthermore, components from different embodiments may be combined as appropriate.

VD...image processing device 1...control unit 2...program storage unit 3...data storage unit 4...input/output I/F unit 5...bus 11...input video acquisition processing unit 12...VM processing unit 13...subject tracking processing unit 14...video synthesis processing unit 15...synthesized video output processing unit 31...input video storage unit 32...VM processed video storage unit 33...subject area detection information storage unit 34...synthesized video storage unit

Claims (6)

a first processing unit that acquires first video data including an image of a subject in an image frame;
a second processing unit that performs minute change enhancement processing to detect minute changes in pixels from the acquired first video data, and generate second video data that includes image components that have been subjected to enhancement processing on the detected minute changes in pixels in the first video data; and
a third processing unit that performs processing to detect a limited area including an image of the subject from the first video data as a subject area in parallel with the small change emphasis processing;
a fourth processing unit that extracts third video data corresponding to the detected object region from the second video data, and synthesizes the extracted third video data with the first video data by aligning the time position and coordinate position, thereby generating fourth video data;
a fifth processing unit that outputs the generated fourth video data. The image processing device of claim 1, wherein the third processing unit detects, from the first video data, an image area that is preset to include the range of motion of the subject as the subject area. The image processing device of claim 1, wherein the third processing unit calculates the length between the multiple movable base points of the subject in pixel units from the first video data, sets an image area including the movable range of the subject based on the calculated length between the movable base points, and detects the set image area from the first video data as the subject area. The image processing device of claim 1, wherein the fourth processing unit further performs processing to smooth the boundary between the third video data and the first video data when combining the third video data with the first video data. An image processing method executed by an information processing device,
a first process for acquiring first video data including an image of a subject in an image frame;
a second processing step of performing minute change enhancement processing to detect minute changes in pixels from the acquired first video data, and to generate second video data including image components that have been subjected to enhancement processing on the detected minute changes in pixels in the first video data;
a third processing step of detecting a limited area including an image of the subject from the first video data as a subject area in parallel with the small change emphasis processing;
a fourth processing step of extracting third image data corresponding to the detected object region from the second image data, and synthesizing the extracted third image data with the first image data by aligning the time position and coordinate position, thereby generating fourth image data;
and a fifth processing step of outputting the generated fourth video data. A program that causes a processor provided in the image processing device to execute processing performed by at least one of the first to fifth processing units provided in the image processing device described in any one of claims 1 to 4.