JP6870728B2 - Image processing device, image forming device and image processing method - Google Patents

Description

The present invention relates to an image processing technique for extracting still image data from moving image data, and more particularly to a technique that can be used for creating an album or the like.

In recent years, with the improvement of performance and image quality of video cameras and smartphones, still image data that can be extracted from moving image data has been generated.

Further, a technique has been proposed in which a desired still image is extracted from moving image data and an album of the extracted still image can be created. For example, Patent Document 1 provides a technique for automatically creating an album by detecting an image of a predetermined person in a moving image and selecting a representative part from a plurality of frames in which the image of the person is reflected. is suggesting.

Japanese Unexamined Patent Publication No. 2009-88687

By the way, conventionally, the moving image data is simply regarded as a set of a plurality of frame image data. Therefore, sufficient studies have not been conducted from the viewpoint of extracting still image data by utilizing the temporal continuity that is a characteristic of the moving image data.

The present disclosure has been made in view of such a situation, and an object of the present disclosure is to provide a technique for extracting still image data by utilizing the temporal continuity which is a characteristic of the moving image data.

The image processing apparatus according to one aspect of the present disclosure includes a still image data generation unit and a continuous still image data extraction unit. The still image data generation unit generates a plurality of still image data from the moving image data. The continuous still image data extraction unit is still image data that includes an image of the same subject and is temporally continuous at a predetermined cycle from the generated plurality of still image data, and is the same. The continuous still image data that satisfies the preset condition for the movement of the image of the subject is extracted.

The image forming apparatus according to another aspect of the present disclosure includes the image processing apparatus and an image forming unit for forming an image on a print medium.

An image processing method according to another aspect of the present disclosure includes generating a plurality of still image data from moving image data, including an image of the same subject from the generated plurality of still image data, and having a predetermined period. This includes extracting continuous still image data that is temporally continuous in time and satisfies the condition that the movement of the image of the same subject is preset.

According to the present disclosure, it is possible to provide a technique for realizing extraction of still image data utilizing the temporal continuity that is a characteristic of moving image data.

It is a block diagram which shows the functional structure of the image forming apparatus 100 which concerns on one Embodiment of this disclosure. It is a flowchart which shows the content of the still image acquisition processing which concerns on one Embodiment. It is a flowchart which shows the content of the person registration process which concerns on one Embodiment. It is a data flow diagram which shows the content of the frame image data generation processing which concerns on one Embodiment. It is explanatory drawing which shows the content of the plurality of frame image data including the image of the same person. It is explanatory drawing which shows the contents of two frame image data F1 and F2 including the image of the same person. It is explanatory drawing which shows the content of the frame image data which a subject is moving in a perspective direction.

Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The embodiment is a mode for carrying out the present disclosure.

As shown in FIG. 1, it is a block diagram which shows the functional structure of the image forming apparatus 100 which concerns on one Embodiment of this disclosure. The image forming apparatus 100 includes a control unit 110, an image forming unit 120, an operation display unit 130, a storage unit 140, and a communication interface unit 150. The communication interface unit 150 is also referred to as a communication I / F unit.

The control unit 110, the image forming unit 120, the operation display unit 130, the storage unit 140, and the communication interface unit 150 are examples of an image processing device, a printing processing device, an operation display device, a storage device, and a communication interface device, respectively.

The image forming apparatus 100 is connected to the smartphone 200 by short-range wireless communication via the communication interface unit 150. As a result, the image forming apparatus 100 can receive the moving image data generated by imaging with the smartphone 200.

In the present embodiment, the short-range wireless communication uses CLASS2 of BLUETOOTH (registered trademark). BLUETOOTH (registered trademark) CLASS2 is a communication with an output of 2.5 mW, and is a short-range wireless communication capable of communicating within a distance of about 10 m between the image forming apparatus 100 and the smartphone 200.

The control unit 110 includes storage means such as RAM and ROM, and a processor such as an MPU (Micro Processing Unit) or a CPU (Central Processing Unit). The processor is an example of a control means.

Further, the control unit 110 has a controller function related to interfaces such as various I / O, USB (universal serial bus), bus, and other hardware. The control unit 110 controls the entire image forming apparatus 100.

The control unit 110 further includes a frame extraction unit 111, a person collation unit 112, a speed detection unit 113, a high-speed movement section extraction unit 114, a still image output unit 115, and a cycle setting unit 116. The person matching unit 112 includes a person registration unit 112a.

The processor executes a program stored in the ROM or the like. As a result, the control unit 110 functions as a frame extraction unit 111, a person collation unit 112, a speed detection unit 113, a high-speed moving section extraction unit 114, a still image output unit 115, and a cycle setting unit 116.

The frame extraction unit 111, the person matching unit 112, the speed detection unit 113, the high-speed moving section extraction unit 114, the still image output unit 115, and the cycle setting unit 116 are the frame extraction device, the person matching device, the speed detection device, and the high-speed moving section, respectively. This is an example of an extraction device, a still image output device, and a cycle setting device.

The image forming unit 120 forms an image on paper or other sheet-shaped print medium. The operation display unit 130 functions as a touch panel including the operation unit and the display unit. The operation display unit 130 displays various menu screens as input screens on the display unit, and further receives user operation input through the operation unit.

The storage unit 140 is a storage device including a hard disk drive, a flash memory, or the like, which are non-temporary recording media.

The storage unit 140 stores control programs and data corresponding to the processes executed by the control unit 110, respectively. The storage unit 140 has a frame memory 141 for temporarily storing frame image data, a still image storage area 142, and a person registration data storage area 143. The storage area is a part of the data storage unit in the storage unit 140.

It is conceivable that the frame memory 141, the still image storage area 142, and the person registration data storage area 143 are a part of the data storage unit in one storage device. On the other hand, it is also conceivable that the still image storage area 142 and the person registration data storage area 143 are separate storage devices.

The control unit 110 can execute the still image acquisition process according to the embodiment. Reference numerals S10, S20, S30, ... In FIG. 2 are identification codes for a plurality of steps in the still image acquisition process.

As shown in FIG. 2, it is a flowchart which shows the content of the still image acquisition processing which concerns on one Embodiment. In step S10 of the still image acquisition process, the person collation unit 112 executes a person registration process that involves the use of the operation display unit 130 by the user. In the person registration process, the person registration unit 112a can register the information of the person to be detected in the still image to be extracted as one of the conditions for extracting the still image data from the moving image data. ..

S11, S12, S13, ... In FIG. 3 are identification codes for a plurality of steps in the still image acquisition process.

As shown in FIG. 3, it is a flowchart which shows the content of the person registration process which concerns on one Embodiment. In step S11 of the person registration process, the person registration unit 112a executes the moving image data acquisition process. In the moving image data acquisition process, the person registration unit 112a selects the moving image data MD according to the operation of the operation display unit 130 by the user. Further, the person registration unit 112a can take the moving image data MD into the image forming apparatus 100 through, for example, a wireless communication device (not shown) or a portable storage medium (not shown).

In step S12, the person matching unit 112 of the control unit 110 executes the person detection process. In the person detection process, the person matching unit 112 generates frame image data from the moving image data MD. Further, the person matching unit 112 extracts a person detection area, which is an image area having the characteristics of a person, from the still image represented by the frame image data file.

The person matching unit 112 can extract a person detection region by using machine learning such as SVM (Support Vector Machine) based on, for example, HOG (Histograms of Oriented Gradients) features.

In step S13, the person matching unit 112 executes the person classification process. In the person classification process, the person matching unit 112 classifies a person in the person detection area, for example, by determining which of the pre-registered family images the image of the person in the person detection area corresponds to. ..

For example, family information includes information on father A, mother B, son C, and daughter D. The person registration unit 112a registers the family information in the storage unit 140 in advance in response to an operation on the operation display unit 130 by the user.

The person matching unit 112 selects frame image data including a face image having a size larger than a preset image size. Further, the person collation unit 112 classifies the selected frame image data into a plurality of groups according to the classification result of the person classification process, and displays the selected frame image data on the operation display unit 130.

Further, the person collation unit 112 classifies whether each of the plurality of groups corresponds to the father A, the mother B, the son C, the daughter D, or another person according to the input operation to the operation display unit 130 by the user. Can be modified.

The user can perform an operation to correct the misrecognition that the group of the son C contains a still image of the father A, for example. As a result, the person matching unit 112 can improve the accuracy of machine learning.

The person matching unit 112 generates a database with the father A, the mother B, the son C, and the daughter D as records. HOG features of facial images of father A, mother B, son C and daughter D are registered in the database.

In step S14, the person matching unit 112 executes the clothing selection process. In the clothing selection process, the person matching unit 112 extracts the HOG feature amount of the clothing worn by each of the father A, the mother B, the son C, and the daughter D from the frame image data. As a result, the person matching unit 112 can identify a person by using the HOG features of the facial images of the father A, the mother B, the son C, and the daughter D, as well as the HOG features of the clothes image. This is because each person often wears the same clothes, and different people tend to wear different clothes.

In step S15, the person registration unit 112a of the person collation unit 112 executes the database registration process. In the database registration process, the person registration unit 112a stores the database of the father A, the mother B, the son C, and the daughter D in the person registration data storage area 143 of the storage unit 140.

In the database, father A, mother B, son C and daughter D are records. The database includes the HOG feature amount of each face image, the HOG feature amount of the clothing image, the machine learning data of the face image, the machine learning data of the clothing image, and the height and other user-inputtable attribute data.

Further, the person registration unit 112a can also register the face image and clothing image data of each person by using the still image data captured by the digital camera according to the operation by the user. The person registration unit 112a can use such image data to generate the HOG feature amount of the face image and the HOG feature amount of the clothing image and register them in the database. In this embodiment, it is assumed that the HOG feature amount is generated based on the YUV image data having a small calculation load in image recognition.

As shown in FIG. 2, in step S20, the user sets the still image extraction mode via the operation display unit 130. The still image extraction mode includes various modes such as a mode for extracting a close-up photo of a person's face and a mode for extracting a group photo, as well as a continuous photo mode in which a continuous photo with a lively feeling is extracted. There is. In the setting of the still image extraction mode, the control unit 110 causes the operation display unit 130 to display a screen (not shown) for accepting the selection of each mode.

For example, when the continuous photo mode is selected, the cycle setting unit 116 causes the operation display unit 130 to display an operation display screen (not shown) that accepts the cycle setting input. The cycle setting input is an input operation for setting a cycle which is a time interval of the frame image. The cycle is used to set the extraction cycle of the frame image as a still image. However, if no user input is made, 0.2 seconds shall be used as the initial setting.

In step S30, the person matching unit 112 selects a person through the operation of the operation display unit 130 by the user. In the selection of a person, the person matching unit 112 causes the operation display unit 130 to display a screen for accepting the selection of the father A, the mother B, the son C, and the daughter D. In this example, it is assumed that son C is selected.

In step S40, the frame extraction unit 111 executes the frame image generation process. The frame extraction unit 111 is an example of a device that functions as a still image data generation unit. In the frame image generation process, the frame extraction unit 111 generates frame image data having a predetermined period from the moving image data MD having a frame rate of, for example, 30 fps. The predetermined period is preset by, for example, the user. For example, it is conceivable that the predetermined period of the initial setting is 0.2 seconds.

FIG. 4 is a data flow diagram showing the contents of the frame image data generation process according to the embodiment. In FIG. 4, a data flow diagram is shown on the upper side, and GOP (Group of Pictures) is shown on the lower side. The data flow diagram shows the flow from the extraction of frame image data from the moving image data MD to the conversion and storage of the extracted data. The frame image data is configured as YUV image data. The frame image data generation process is a process of extracting a plurality of frame image data from the moving image data MD, and is executed by the frame extraction unit 111.

For frame image data generation processing, for example, MPEG-4 (ISO / IEC 14496) or H.M. The processing specified in 264 is included. In the frame image data generation process, the frame extraction unit 111 generates frame image data from an I frame (Intra-coded Frame), a P frame (Predicted Frame), and a B frame (Bi-directed Printed Frame).

An I-frame is a frame that is coded without motion compensation. I-frames are also called intra-frames or keyframes. The I frame constitutes a GOP together with the P frame (Predicted Frame) and the B frame (Bi-directed Printed Frame).

It is possible to generate frame image data by performing inter-frame processing on the P frame with the I frame. It is possible to generate frame image data by performing inter-frame processing on the B frame with the I frame, the P frame, and other B frames.

The moving image data is generated from a plurality of frame image data arranged in chronological order. The plurality of frame image data are often approximated between the frames before and after the time series. Inter-frame prediction is a process that utilizes such properties of moving image data. Inter-frame prediction is a process of predicting the current frame image from the previous frame image in chronological order.

Specifically, in the inter-frame prediction, the movement of each pixel block is estimated, and the difference between the pixel blocks after the movement is DCT-transformed and quantized. As a result, the compression rate in GOP units is increased. The P-frame is reproduced from the I-frame by using a motion vector. The motion vector is a motion vector of each pixel block.

The frame extraction unit 111 generates frame image data as YUV image data including brightness data and color difference data by performing an inverse discrete cosine transform (also called an inverse DCT transform) on an I frame. The inverse DCT transform is performed, for example, for each 8 × 8 pixel or 16 × 16 pixel block. The frame extraction unit 111 stores the reproduced frame image data in the frame memory 141.

The frame extraction unit 111 generates difference data by performing an inverse discrete cosine transform on the P frame and the B frame. The frame extraction unit 111 generates frame image data by executing inter-frame processing using the difference data and the motion vector. The motion vector is data generated when the motion image data MD is encoded. This process is performed by MPEG-4 or H.M. This is a normal decoding process specified in 264.

The frame extraction unit 111 executes a frame image data generation process based on the P frame and the B frame on the RAM (not shown) of the control unit 110. The frame extraction unit 111 stores the frame image data of a preset cycle in the frame memory 141. Specifically, when the frame rate of the moving image data MD is 30 fps, the period is set to 0.2 seconds. In this case, the frame extraction unit 111 stores the frame image data in the frame memory 141 every 6 frames. On the other hand, the frame extraction unit 111 discards other frame image data. As a result, the frame extraction unit 111 can reduce the excessive consumption of the frame memory 141.

When the frame image data is stored in the frame memory 141, the frame extraction unit 111 stores the motion vector used in the generation of the frame image data in association with each frame image data.

In step S50, the person matching unit 112 executes the person detection process. In the person detection process, the person matching unit 112 determines whether or not the person selected in step S30 is included in each of the plurality of frame image data stored in the frame memory 141.

For example, it is conceivable that the person selected in step S30 is son C. In this case, the person matching unit 112 attempts to detect the son C from the frame image data as YUV image data including the luminance data and the color difference data.

The person matching unit 112 can detect and identify a person by using, for example, a well-known OpenCV (Open Source Computer Vision Library). First, the person matching unit 112 detects a person from the frame image data. Further, the person matching unit 112 determines whether or not the face of the person detected by using the HOG feature amount of the face image of the son C is the face of the son C.

When the accuracy of determining whether or not the face is the face of the son C is low, the person matching unit 112 uses the HOG feature amount of the clothes image of the son C to determine whether or not the face of the person is the face of the son C. To judge. In particular, when the son C is photographed from the lateral direction and the size of the face image is small, the HOG feature amount of the clothing image can be used as an auxiliary. This is because, in general, when a continuous image with a lively feeling is taken, it is difficult to stably capture a face image in close-up.

In step S60, the speed detection unit 113 of the control unit 110 detects the moving speed of the person who is the subject. In this example, the person is son C. The speed detection unit 113 specifies a temporal section in which an image of the same subject is included in a plurality of frame image data that are continuous in time.

The speed detection unit 113 can estimate the speed of the image of the subject by using the motion vector of the pixel block constituting the image of the same subject, that is, the image region including the image of the same person. This is because the pixel block constituting the image region including the image of the son C has a motion vector corresponding to the translational movement speed of the son C.

In the example shown in FIG. 5, nine images of frame image data F1 to F9 in which the son C is photographed as the same person are shown. The frame image data F1 to F9 represent a series of dynamic continuous images in which the son C is running against the background of three stationary trees.

FIG. 6 is an explanatory diagram showing the contents of two frame image data F1 and F2 in which the same person is represented. In the two frame image data F1 and F2 in FIG. 6, the image of the son C is moving in translation to the right toward FIG. The translational movement amount of the image of the son C is the movement amount between the frames of the two frame image data F1 and F2. The translational movement speed can be calculated by dividing the translational movement amount by the cycle. However, in the present embodiment, the translational movement speed is estimated from the motion vector assuming 30 fps.

In step S70, the high-speed moving section extraction unit 114 of the control unit 110 determines whether or not the moving speed of the image of the son C is equal to or higher than a preset threshold value. The high-speed movement section extraction unit 114 advances the process to step S80 when the movement speed of the image of the son C is equal to or higher than the threshold value, and steps the process when the movement speed of the image of the son C is less than the threshold value. Proceed to S90.

In step S80, the high-speed moving section extraction unit 114 executes the frame image data grouping process. The frame image data grouping process is frame image data including the image of the son C, which is temporally continuous in a predetermined cycle before and after the two frame image data F1 and F2.

FIG. 5 shows an example in which the shooting of the son C is started with the frame image data F1 and the shooting of the son C is continued until the frame image data F9.

The high-speed moving section extraction unit 114 groups nine frame image data F1 to F9 and stores them in the still image storage area 142. As a result, the nine frame image data F1 to F9 can be managed and handled as one continuous image data file. In the still image storage area 142, nine frame image data F1 to F9 are DCT-converted and stored as JPEG still image data.

The high-speed moving section extraction unit 114 is an example of a continuous still image data extraction unit. The nine frame image data F1 to F9 are examples of continuous still image data.

Further, in the present embodiment, the state in which the moving speed of the image of the subject (son C) is equal to or higher than the threshold value may be momentary in one continuous image data file. That is, the state in which the moving speed of the image of the subject is equal to or higher than the threshold value may occur once in one continuous image data file. The moving speed of the image does not have to exceed the threshold value for the entire continuous image data file. However, it is also conceivable that the high-speed moving section extraction unit 114 determines that the moving speed exceeds the threshold value in all sections of one continuous image data file as a necessary condition.

The control unit 110 repeatedly executes the processes of steps S40 to S80 for the plurality of frame image data until the final frame image data becomes the processing target (step S90).

In step S100, the still image output unit 115 executes the frame image data output process. In the frame image data output process, the still image output unit 115 displays one continuous image data file on the operation display unit 130 as nine thumbnail images. Here, one continuous image data file includes nine grouped frame image data F1 to F9.

The control unit 110 selects an arbitrary frame image from the nine frame image data F1 to F9 as a processing target according to the operation of touching the thumbnail image by the user, and processes all the frame images at once. It can also be selected as a target.

The still image output unit 115 executes a process of forming an image on a print medium in the image forming unit 120 by controlling the image forming unit 120 based on the selected frame image data according to a user's instruction. Can be made to.

The printed matter output by the image forming unit 120 can be used for an album or the like as a continuous photograph having a dynamic feeling. On the other hand, the still image output unit 115 can also transmit a continuous image data file to the smartphone 200 or a personal computer (not shown) via the communication interface unit 150 in response to an instruction operation by the user.

As described above, according to the image forming apparatus 100 according to the present embodiment, taking advantage of the temporal continuity that is a characteristic of moving image data, a continuous image (continuous photograph) having a dynamic feeling is produced for a predetermined period of time. It is possible to extract a plurality of continuously continuous still image data (frame image data).

The present disclosure can be implemented not only in the above embodiment but also in the following modifications.

[Modification 1]
In the above embodiment, frame image data in which the translational movement speed of the image of a person as a subject exceeds a threshold value is extracted as a continuous image with a dynamic feeling. However, the dynamic continuous image is not limited to the image having a high translational movement speed.

Specifically, for example, as shown in FIG. 7, an image showing a scene in which a subject approaches in a perspective direction is also included in a continuous image having a dynamic feeling.

In the example shown in FIG. 7, the size of the region where the person as the detection target is detected changes from the size of the detection region Fr1 to the size of the detection region Fr2. In step S70, it is conceivable that the speed detection unit 113 makes a determination based on whether or not the rate of change in the image size of the subject is equal to or greater than a preset threshold value. The rate of change in the image size of the subject is the rate of reduction or enlargement per unit time.

In the P frame or the B frame, the motion vector associated with the pixel blocks constituting the image of the person as the subject has many components in the divergence direction when the person as the detection target approaches. On the other hand, the motion vector has many components in the convergence direction when the person to be detected moves away.

Translational movement, perspective movement, and combinations thereof can be detected by analyzing the translational component, the divergence component, and the convergence component of the motion vector. The speed detection unit 113 determines whether or not the image is a continuous image with a dynamic feeling based on at least one of a translation component, a divergence component, and a convergence component of a motion vector accompanying the movement speed of the image of a person as a subject. Can be done.

[Modification 2]
In the above embodiment, frame image data is extracted as a continuous image with a dynamic feeling based on the moving speed of an image of a person as a subject. However, the dynamic continuous image is not limited to the image having a high moving speed.

Specifically, for example, a person as a subject may be doing gymnastics. In this case, the image of the person as the subject is an image when the limbs of the person are moving or rotating without movement. In such a case, the control unit 110 may extract a continuous image including an image of a person whose limbs are moving or rotating as a continuous image with a dynamic feeling.

In this way, it is conceivable that the limbs of the person to be detected are moving or rotating. In this case, the motion vector associated with the pixel block constituting the image of the person as the subject may have a random direction and size. In this case, when the motion vector of the pixel block constituting the image of the person as the subject has a random direction and size, the speed detection unit 113 makes the continuous image corresponding to the motion vector a dynamic continuous image. Can be determined to be.

[Modification 3]
In the above embodiment, a motion vector is used as data used for determining whether or not the image is a continuous image with a dynamic feeling. However, the motion vector does not necessarily have to be used, and other data may be adopted.

Specifically, the speed detection unit 113 may be configured to detect the moving speed of the person by referring to the position or size of the person area existing in each frame image, for example. For example, the speed detection unit 113 calculates the difference in the position of the person area between the frame images adjacent in time as the movement distance, and if the distance is larger than a predetermined threshold value, the person in the moving image moves at high speed. It can be judged that it is. Here, the person area is an area including an image of a person.

The speed detection unit 113 further compares not only the position of the person area but also the size of the person area, and even when the difference in size is larger than a predetermined threshold value, the image of the person in the moving image is moving at high speed. You may judge that.

The method using the motion vector in the above embodiment has an advantage that the calculation load is small and the processing can be speeded up. On the other hand, the method in this modified example has an advantage that the degree of freedom in designing the process for determining whether or not the continuous image has a dynamic feeling is high.

Further, for example, one of the difference in the position of the person area and the difference in the size of the person area may be mainly used, and the other may be used in a complementary manner to combine the two. As described above, the speed detection unit 113 may determine whether or not the movement of the image of the same subject satisfies the preset condition.

[Modification example 4]
In the above embodiment, the present invention is applied to the image forming apparatus. However, the present invention can also be applied to devices that function as image processing devices, such as smartphones and personal computers.

Claims (4)

A still image data generator that generates multiple still image data from moving image data,
From the generated plurality of still image data, the still image data includes the image of the same subject and is continuous in time at a predetermined cycle, and the movement of the image of the same subject is preset. It is equipped with a continuous still image data extraction unit that extracts continuous still image data that satisfies the above conditions .
The still image data generation unit uses the motion vector included in the moving image data to reproduce an image of a plurality of pixel blocks to which the motion vector is associated, and generates the plurality of still image data. And
The continuous still image data extraction unit performs translational movement of the subject and translational movement of the subject based on a translation component, a divergence component, and a convergence component of the motion vector associated with at least one pixel block constituting the image of the subject. An image processing device that determines whether or not the movement of a combination of perspective movement satisfies a preset condition. The image processing apparatus according to claim 1, further
An operation display unit that displays the screen and accepts input from the user,
An image processing device including a cycle setting unit for displaying a screen for receiving a cycle setting input, which is a predetermined cycle setting, on the operation display unit. The image processing apparatus according to claim 1 and
An image forming apparatus including an image forming unit for forming an image on a print medium. Generating multiple still image data from moving image data and
From the generated plurality of still image data, the still image data includes the image of the same subject and is continuous in time at a predetermined cycle, and the movement of the image of the same subject is preset. Extracting continuous still image data that meets the above conditions
Only including,
The plurality of still image data are generated by reproducing an image of a plurality of pixel blocks to which the motion vector is associated by using the motion vector included in the motion image data.
When the continuous still image data is extracted, the subject's image is based on a translation component, a divergence component, and a convergence component of the motion vector associated with at least one pixel block constituting the image of the subject. An image processing method in which it is determined whether or not the movement of a combination of translational movement and perspective movement satisfies a preset condition.

Images