JP7192259B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

Techniques have been proposed for reliably detecting a fluctuation region from the entire screen and accurately detecting an object to be monitored. (For example, Patent Document 1)

JP 2011-123562 A

For example, when shooting a video of a scene where a person dances with the camera fixed indoors, and the camera's autofocus function is used to track the dancing person, who is the main subject, moving within the shooting range. Assume a case of continuous focusing.

In such a situation, as the in-focus position moves, even the background image portion is significantly out of focus, and is photographed in a blurry state called "out of focus." There is

As described above, when an image on the background side is shot in a blurred state, if image processing is required to separate the image of the main subject moving in the shot image afterward from the background, the background and the This causes a problem that the accuracy in determining the division from the main subject is lowered.

Even the technology described in the above patent document does not take into consideration the blurring of the background image caused by the function of such an autofocus function.

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and its object is to provide an image processing apparatus, an image processing method, and a program capable of accurately separating a moving object moving in an image from the background. to do.

One aspect of the present invention is
determining means for comparing a target image constituting a moving image with a reference image within a preset image position range to determine the reliability of the target image as a background image;
background image creating means for creating background image information using a plurality of target images determined to be reliable by the determining means;
with
The background image creating means recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets a weight for each area as a background image. ,
For division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.

According to the present invention, it is possible to accurately separate a moving object moving in an image from the background.

1 is a block diagram showing the functional configuration of an electronic circuit of a smart phone according to one embodiment of the present invention; FIG. 4 is a flowchart showing a series of processes from photographing the dance according to the embodiment to scoring based on the photographed result. FIG. 3 is a flowchart showing the processing contents of a subroutine relating to background template image creation of FIG. 2 according to the embodiment; FIG. FIG. 4 is a diagram showing an example of weighting settings according to the position of a main subject in an image according to the embodiment;

An embodiment in which the present invention is applied to a smart phone in which an application program for dance analysis is installed will be described in detail below with reference to the drawings.

FIG. 1 is a block diagram illustrating the functional configuration of an electronic circuit of the smartphone 10. As shown in FIG. In this smart phone 10 , 11 is a speaker that outputs audio, and 12 is a microphone that inputs audio, both of which are connected to an audio processing unit 13 .

When the smartphone 10 performs voice input/output, the voice processing unit 13 digitizes the voice input from the microphone 12, converts it into code data, performs predetermined data compression, and outputs it to the system bus SB. Digital music data is received via the system bus SB, decompressed, converted into an analog audio signal, and output to the speaker 11 for sound emission.

The processor 14 controls the overall operation of the smartphone 10 using a main memory 15 and a solid state drive (SSD) 16 connected via the system bus SB.

The main memory 15 is, for example, an SDRAM, and is a work memory that develops and holds an application program read from the SSD 16 when the processor 14 executes the program. The SSD 16 consists of a non-volatile memory, and stores and stores an OS (operating system), various application programs including a dance analysis program 16A, which will be described later, and various other fixed data and contents. is read out to the main memory 15 or the like by the processor 14 as appropriate.

During video recording, the processor 14 converts a series of image data obtained by shooting into a moving image file including data compression. The data is sequentially decoded and decompressed in the reverse order of recording, and reproduced and displayed.

The system bus SB further includes a display unit 17, a touch input unit 18, an imaging unit 19, a GPS receiving unit 20, an acceleration sensor 21, a gyro sensor 22, a geomagnetic sensor 23, an operation unit 24, a vibrator 25, and a wireless communication unit 26. , an NFC (Near Field Communication) unit 27 and an external device interface (I/F) 28 .

The display unit 17 is composed of, for example, a color liquid crystal panel with a backlight and a diagonal size of about 6 inches, and a drive circuit for them. A touch input unit 18 using a transparent electrode film is integrated with the display unit 17 . The touch input unit 18 digitizes a time-series coordinate signal corresponding to a user's touch operation including a multi-touch operation, and sends it to the processor 14 as a touch operation signal.

The imaging unit 19 includes an imaging unit provided on at least one of the front and back sides of the housing of the smartphone 10, and an image processing unit that compresses and files image data obtained by imaging by the imaging units.

The GPS receiver 20 calculates the three-dimensional coordinates (latitude, longitude, altitude) of the current position and the current time based on incoming radio waves from a plurality of GPS satellites (not shown) received by the GPS antenna 29. It is sent to the processor 14 via SB.

The acceleration sensor 21 detects acceleration in each of three axial directions orthogonal to each other, and can detect the orientation of the smartphone 10 from the direction of gravitational acceleration.

The gyro sensor 22 is composed of vibrating gyroscopes arranged in three mutually orthogonal axial directions. is analyzed, and even if the absolute current position cannot be recognized by the output from the GPS antenna 29 and the GPS receiving unit 20, the current position is updated by autonomous navigation in cooperation with the acceleration sensor 21 and the geomagnetic sensor 23. can be executed.

The geomagnetism sensor 23 detects the geomagnetism in each of three mutually orthogonal axial directions, and can detect the azimuth to which the smartphone 10 is directed at that time based on the direction of magnetic north.

The operation unit 24 is provided on the side of the housing of the smartphone 10, and is composed of several operation keys including a power key, and transmits key operation signals corresponding to those key operations to the processor 14 via the system bus SB. do.

The vibrator 25 is composed of a small motor and a weight eccentrically attached to the rotating shaft of the motor, and generates vibrations extending over the entire smart phone 10 by rotating the motor.

The wireless communication unit 26 uses antennas 30 to 32, for example, a 4th generation mobile phone, a wireless LAN system conforming to IEEE802.11a/11b/11g/11n, and a short-range wireless communication line such as Bluetooth (registered trademark). , and transmits and receives data to and from an external device by wireless communication.

The NFC unit 27 uses, for example, the FeliCa (registered trademark) standard and the MIFARE (registered trademark) standard to perform data transmission/reception and power supply operation with an external device in a short distance without contact.

The external device interface 28 enables connection or attachment of, for example, headphones, earphones, USB devices, and memory cards via the headphone jack 33, micro USB terminal 34, and memory card slot 35, respectively.

A memory card inserted in the memory card slot 35 can record moving image data files and the like used in a dance analysis program, which will be described later, and read out the recorded data files.

Next, the operation of this embodiment will be described.
FIG. 2 shows the flow of a series of processes from shooting a moving image of a dancing person to scoring the dance of the person in the moving image file obtained by the shooting.

At the beginning, the user installs the smartphone 10, preferably by fixing it using a tripod or the like, and then shoots a moving image of a dancing person (step S101).

Prior to this shooting, for example, at the bottom of the screen of the display unit 17,
"Set the angle of view so that the four corners of the background image are clearly visible."
The user may be urged to shoot an appropriate moving image by superimposing a character guide message such as .

This video recording may be performed in a normal camera mode without using the dance analysis program 16A, which is an application program installed in the smartphone 10, or from a device outside the smartphone 10. A moving image file that has already been photographed may be taken in and recorded in the memory card attached to the memory card slot 35 or the SSD 16 .

After that, if the dance analysis program 16A has not been started, the program is started again, and then the moving image file to be analyzed is selected.

For the selected moving image file, the processor 14 first executes a process of creating a background template image for cutting out and extracting a dancing person from the image (step S102).

FIG. 3 is a flow chart showing the processing contents of a subroutine for creating this background template image.

At the beginning, the processor 14 sets one frame (for example, the first frame) of the moving image to be shot this time as a reference initial background image. At this time, it is determined by image recognition processing whether the one frame is out of focus, and the frame that is not out of focus is set as a reference initial background image (step S201). ).

After the initial background image is selected in this way, the background template image is created using the individual frame images that make up the moving image file.

At the beginning of the process, the processor 14 determines whether or not the background template image has been created by determining whether or not the process for all frame images has been completed (step S202).

If it is determined that the process has not ended yet (No in step S202), the processor 14 selects the frame image that is not referred to in the creation of the background template image and that is located at the top in time series. , is set as an image to be decoded (hereinafter referred to as "target image") (step S203).

Next, the processor 14 generates an image obtained by averaging the images that have been integrated as the background template image up to that point (hereinafter referred to as the “previous target image”), and the peripheral image range of the target image set this time, for example, the four corners. 64 pixels long×64 pixels wide image blocks are verified for matching (step S204).

The previous target image is assumed to be an image based on accumulation from the initial background image first selected in step S201.

Based on the verification result, the processor 14 determines whether or not the difference is equal to or less than a preset threshold value (step S205).

Here, the matching of the image areas of the four corners of the previous target image and the target image set this time is verified because, in the case of dance, the dancers do not move to the positions of the four corners of the image and dance. If it is determined that the four corners do not match (No in step S205), for example, the background side of the frame image is greatly out of focus due to the tracking processing of the main subject in the image by the autofocus function, etc., resulting in an out-of-focus state. or the photographing range of the frame image may differ from the photographing range of the initial background image.

Therefore, the processor 14 assumes that the target image set at that time will not be used for creating the background template image, and returns to the processing from step S202 to set the next frame image.

If it is determined in step S205 that the difference between the four corners of the two target images is equal to or less than the preset threshold value and they match (Yes in step S205), then the processor 14 A weighting setting is performed for the background area of the screen according to the position (step S206).

Here, the reason why the background area of the screen is weighted according to the position of the main subject in the target image is that in the case of dance, the movements and choreography of the dancers are determined according to each dance, and the movements of the dancers are determined according to the sequence of dances. In some cases, the dancers stay near the center of the dance floor for a certain amount of time while dancing, and other times they dance while staying near the left or right end of the dance floor.

When creating a background template from such a dance video, the object is to prevent a dancer who has stayed at a specific place for a certain amount of time from being included in the background.

FIG. 4 is a diagram showing an example of weight setting according to the position of the main subject MO in the image. In this example, a horizontally long rectangular image is divided horizontally into four areas, the background area is equally divided, the total value of the entire screen is set to "200", and each area is weighted. indicates

FIG. 4(A)-1 shows a case where the main subject MO stays almost in the center of the image and is dancing. In this case, as shown in FIG. 4(A)-2, in the background area divided into four, weighting "70" is set for each of the two areas on the left and right ends, while weighting "70" is set for the two areas near the center. are each set with a weight of "30".

FIG. 4(B)-1 shows a case where the main subject MO stays almost on the left end side of the image and is dancing. In this case, as shown in FIG. 4(B)-2, the background area divided into four is sequentially weighted from the left end to "30", "40", "60", and "70". It is set so that

Further, FIG. 4(C)-1 shows a case where the main subject MO is positioned almost on the right end side of the image. In this case, as shown in FIG. 4(C)-2, weighting is performed sequentially from the right end of the background area divided into four, with "30", "40", "60", and "70". It is set so that

In FIG. 4, the horizontal rectangular image is divided horizontally into four areas, and the background area is equally divided. The user of the smartphone 10 himself/herself may be able to arbitrarily select one division pattern according to the movement of the main subject MO in the dance from among the plurality of division patterns.

In that case, the user himself/herself can appropriately select the division pattern in consideration of the movement of the main subject MO in the image, so that the accuracy of cutting out and extracting the main subject MO from the image later can be further improved. .

After weighting the background area of the screen according to the position of the main subject as shown in FIG. (step S207), the process returns to step S202 to set the next frame image.

By repeating the processes of steps S202 to S207 for the frame images constituting the moving image file, the background template image is created using only the frame images in which the background side of the image is clear.
Since this weighted background image generation method is a well-known technique, detailed description thereof will be omitted.

Then, after executing the processing for the last frame image that constitutes the moving image file, if it is determined in step S202 that the background template image has been created (Yes in step S202), the processor 14 executes the subroutine of FIG. , and returns to the main routine shown in FIG.

In FIG. 2, processing for cutting out and extracting the main subject MO of the dancer is again performed on all frame images based on the created background template image (step S103).

After that, the processor 14 executes scoring based on a preset scoring method based on the extracted main subject MO of the dancer, and after displaying the scoring result on the display unit 17 (step S104), Thus, the processing of FIG. 2 based on the dance analysis program 16A ends.

As described in detail above, according to this embodiment, it is possible to accurately separate a moving object moving in an image from the background.

In the above embodiment, as also shown in FIG. 4, the position of the main subject MO in the target image is recognized, and the target image is divided into a plurality of background areas based on the recognized position of the main subject. Since the weighting is set for each of the divided background areas, the background template image is created assuming that the background area where the main subject MO does not clearly exist is more weighted. It is possible to further improve the accuracy when cutting out and extracting the main subject MO from the image.

In the above embodiment, the case of applying to a smartphone in which an application program for dance analysis is installed has been explained, but the present invention is not limited to that, and an object moving in an image is used as a main subject. The MO can be similarly applied to other image processing apparatuses that cut out and extract from a background image.

Further, in the above-described embodiment, the frames constituting the moving image that are "out of focus" are not used for creating the background template image (steps S203 to S205 in FIG. 3), and the main subject in the target image is Although both of the weight setting of the background area of the screen according to the position of (step S206 in FIG. 3) are performed, only one of the processes may be performed.

Furthermore, in the above embodiment, the setting of the initial background image that serves as a reference (step S201 in FIG. 3) is based on one frame (for example, the first frame) of the moving image that is shot this time that is not in the “out of focus” state. However, if a file of the dance floor on which the dance will be performed this time already exists in the multiple recorded still images and/or videos, that image will be used as the reference initial background. It may be an image.

Further, in the above embodiment, the setting of the weighting of the background area of the screen according to the position of the main subject in step S206 of FIG. Weighting template data may be created in advance according to the position of the , and the weighting may be set based on this template data.

Furthermore, in the above-described embodiment, dance was explained as an example, but it is not limited to dance, and acting and performance such as figure skating, floor exercise in gymnastics, and cheerleading may be performed.

In addition, the device that implements the program for executing such processing is not limited to smartphones, and can be applied to mobile type personal computers, movie cameras that shoot moving images, and the like.

In addition, the present invention is not limited to the above-described embodiments, and can be variously modified in the implementation stage without departing from the scope of the invention. Moreover, each embodiment may be implemented in combination as much as possible, and in that case, the combined effect can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent elements are deleted from all the constituent elements shown in the embodiments, the problems described in the column of problems to be solved by the invention can be solved, and the effects described in the column of effects of the invention is obtained, the configuration from which this constituent element is deleted can be extracted as an invention.

The invention described in the original claims of the present application is appended below.
[Claim 1]
determining means for comparing a target image constituting a moving image with a reference image within a preset image position range to determine the reliability of the target image as a background image;
background image creating means for creating background image information using a plurality of target images determined to be reliable by the determining means;
An image processing device comprising:
[Claim 2]
The background image creating means recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets weighting for each area as the background image. 2. The image processing apparatus according to claim 1.
[Claim 3]
3. The image processing apparatus according to claim 2, wherein one of a plurality of division patterns prepared in advance can be arbitrarily selected for dividing the plurality of areas.
[Claim 4]
The determination means compares the target image constituting the moving image with the reference image within a preset image position range, and determines that the target image is reliable as a background image if the difference is equal to or less than a preset threshold. 4. The image processing apparatus according to any one of claims 1 to 3, wherein the image processing apparatus determines.
[Claim 5]
The background image creation means recognizes the position of the teacher from a video taken in advance as teacher data, and weights the background image based on template data weighted according to the position of the teacher created in advance. 5. The image processing apparatus according to any one of claims 1 to 4, wherein the image processing apparatus sets .
[Claim 6]
6. The image processing apparatus according to any one of claims 1 to 5, wherein said background image creation means creates background image information by a weighted background image generation method.
[Claim 7]
7. The image processing apparatus according to claim 1, wherein said reference image is one frame of said moving image.
[Claim 8]
8. The image processing apparatus according to any one of claims 1 to 7, wherein the reference image is an image that is not "out of focus".
[Claim 9]
9. The image processing apparatus according to any one of claims 1 to 8, wherein said preset image position range is a peripheral area of an image.
[Claim 10]
10. The image processing apparatus according to claim 9, wherein the preset image position range is four corner areas of the image.
[Claim 11]
a determination step of comparing a target image constituting a moving image with a reference image within a preset image position range to determine the reliability of the target image as a background image;
a background image creation step of creating background image information using a plurality of target images determined to be reliable in the determination step;
An image processing method comprising:
[Claim 12]
A program executed by a computer installed in a moving image handling device, wherein the computer is
a determination means for comparing a target image constituting a moving image with a reference image within a preset image position range to determine the reliability of the target image as a background image;
background image creating means for creating background image information using a plurality of target images determined to be reliable by the determining means;
A program that acts as a

10 ... smart phone,
11...Speaker,
12... Microphone,
13 ... audio processing unit,
14 processor,
15... main memory,
16 SSD (Solid State Drive),
16A ... dance analysis program,
17 ... display unit,
18 ... touch input section,
19... Imaging unit,
20 ... GPS receiver,
21 ... acceleration sensor,
22 ... gyro sensor,
23 ... geomagnetic sensor,
24 operation unit,
25... vibrator,
26... wireless communication unit,
27 NFC unit,
28 ... external device interface (I/F),
29 ... GPS antenna,
30 to 32...Antenna,
33... headphone jack,
34 ... micro USB terminal,
35... memory card slot,
MO ... main subject (in the image),
SB: system bus

Claims (19)

determining means for comparing a target image constituting a moving image with a reference image within a preset image position range to determine the reliability of the target image as a background image;
background image creating means for creating background image information using a plurality of target images determined to be reliable by the determining means;
with
The background image creating means recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets a weight for each area as a background image. ,
For the division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.
Image processing device. 2. The image processing apparatus according to claim 1, wherein the image position range is a partial range of the target image forming the moving image and the reference image. The determination means compares the target image constituting the moving image with the reference image within a preset image position range, and determines that the target image is reliable as a background image if the difference is equal to or less than a preset threshold. 3. The image processing apparatus according to claim 1, wherein the image processing apparatus determines. determining means for comparing a target image constituting a moving image with a reference image within a preset image position range to determine the reliability of the target image as a background image;
background image creating means for creating background image information using a plurality of target images determined to be reliable by the determining means;
with
The background image creation means recognizes the position of the teacher from a video taken in advance as teacher data, and weights the background image based on template data weighted according to the position of the teacher created in advance. to set the
Image processing device. The background image creation means creates background image information by a weighted background image generation method.
The image processing apparatus according to any one of claims 1 to 4 . 6. The image processing apparatus according to claim 1 , wherein said reference image is one frame of said moving image. 7. The image processing apparatus according to any one of claims 1 to 6 , wherein the reference image is an image that is not "out of focus". 8. The image processing apparatus according to any one of claims 1 to 7 , wherein said preset image position range is a peripheral area of an image. 9. The image processing apparatus according to claim 8 , wherein the preset image position range is four corner areas of the image. By setting any one of a plurality of frame images constituting a moving image as a reference image and comparing each of the plurality of frame images with the reference image in a predetermined image area, each of the plurality of frame images a determination means for determining whether or not there is reliability as a background image for
background image creation means for creating a background image using the plurality of frame images determined to be reliable when there are a plurality of frame images determined to be reliable by the determination means;
with
The background image creating means recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets a weight for each area as a background image. ,
For the division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.
Image processing device. By setting any one of a plurality of frame images constituting a moving image as a reference image and comparing each of the plurality of frame images with the reference image in a predetermined image area, each of the plurality of frame images a determination means for determining whether or not there is reliability as a background image for
background image creation means for creating a background image using the plurality of frame images determined to be reliable when there are a plurality of frame images determined to be reliable by the determination means;
with
The background image creation means recognizes the position of the teacher from a video taken in advance as teacher data, and weights the background image based on template data weighted according to the position of the teacher created in advance. to set the
Image processing device. a determination step of determining that the background image is reliable when the preset image position range of the target image constituting the moving image and the preset image position range of the reference image match ;
a background image creation step of creating background image information using a plurality of target images determined to be reliable in the determination step;
has
The background image creating step recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets weighting for each area as the background image. ,
For the division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.
Image processing method. a determination step of determining that the background image is reliable when the preset image position range of the target image constituting the moving image and the preset image position range of the reference image match;
a background image creation step of creating background image information using a plurality of target images determined to be reliable in the determination step;
has
In the background image creation step, the position of the teacher is recognized from a video that is taken as teacher data in advance, and weighted as a background image based on template data that is weighted according to the position of the teacher, which has been created in advance. to set the
Image processing method. By setting any one of a plurality of frame images constituting a moving image as a reference image and comparing each of the plurality of frame images with the reference image in a predetermined image area, each of the plurality of frame images a determination step of determining whether or not the background image is reliable for
a background image creating step of creating a background image using the plurality of frame images determined to be reliable when there are a plurality of frame images determined to be reliable in the determining step;
has
The background image creating step recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets weighting for each area as the background image. ,
For the division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.
Image processing method. By setting any one of a plurality of frame images constituting a moving image as a reference image and comparing each of the plurality of frame images with the reference image in a predetermined image area, each of the plurality of frame images a determination step of determining whether or not the background image is reliable for
a background image creating step of creating a background image using the plurality of frame images determined to be reliable when there are a plurality of frame images determined to be reliable in the determining step;
has
In the background image creation step, the position of the teacher is recognized from a video that is taken as teacher data in advance, and weighted as a background image based on template data that is weighted according to the position of the teacher, which has been created in advance. to set the
Image processing method. A program executed by a computer installed in a moving image handling device, wherein the computer is
Judgment means for comparing the target image that constitutes the moving image with a reference image within a preset image position range to judge the reliability of the target image as a background image; background image creating means for creating background image information using a plurality of target images;
function as
The background image creating means recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets a weight for each area as a background image. ,
For the division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.
program . A program executed by a computer installed in a moving image handling device, wherein the computer is
a determination means for comparing a target image constituting a moving image with a reference image within a preset image position range to determine reliability of the target image as a background image;
background image creating means for creating background image information using a plurality of target images determined to be reliable by the determining means;
function as
The background image creation means recognizes the position of the teacher from a video taken in advance as teacher data, and weights the background image based on template data weighted according to the position of the teacher created in advance. to set the
program. A program executed by a computer installed in a moving image handling device, wherein the computer is
By setting any one of a plurality of frame images constituting a moving image as a reference image and comparing each of the plurality of frame images with the reference image in a predetermined image area, each of the plurality of frame images Determination means for determining whether or not there is reliability as a background image for
background image creating means for creating a background image using a plurality of frame images determined to be reliable when there are a plurality of frame images determined to be reliable by the determining means;
function as
The background image creating means recognizes the position of the main subject in the target image, divides the target image into a plurality of areas based on the recognized position of the main subject, and sets a weight for each area as a background image. ,
For the division of the plurality of areas, one can be arbitrarily selected from a plurality of division patterns prepared in advance.
program. A program executed by a computer installed in a moving image handling device, wherein the computer is
By setting one of a plurality of frame images constituting a moving image as a reference image and comparing each of the plurality of frame images with the reference image in a predetermined image area, each of the plurality of frame images Determination means for determining whether or not there is reliability as a background image for
background image creation means for creating a background image using a plurality of frame images determined to be reliable when there are a plurality of frame images determined to be reliable by the determination means;
function as
The background image creation means recognizes the position of the teacher from a video taken in advance as teacher data, and weights the background image based on template data weighted according to the position of the teacher created in advance. to set the
program.

Images