JP7552155B2 - Image processing device, image processing method, and model generation device - Google Patents

Description

The present invention relates to a video processing device, a video processing method, and a model generation device.

An invention that generates highlight scenes has been known for some time (Patent Document 1). The invention described in Patent Document 1 generates a scene as a highlight scene if the volume of cheers is greater than a predetermined value.

JP 2012-147296 A

However, there are scenes where the players are not visible even when the cheers are loud, and such scenes are not necessarily suitable as highlight scenes.

The present invention has been made in consideration of the above problems, and its purpose is to provide a video processing device, a video processing method, and a model generation device that are capable of generating appropriate highlight scenes.

A video processing device according to one aspect of the present invention includes an extraction unit that extracts a reference scene including a characteristic action or a characteristic sound from a sports video, and a generation unit that generates, as a highlight scene, a scene including a first scene going back a first predetermined time to a second scene going forward a second predetermined time based on the reference scene extracted by the extraction unit.

The present invention makes it possible to generate appropriate highlight scenes.

FIG. 1 is a schematic diagram showing the configuration of a video processing device 10 and a model generating device 50 according to the present embodiment of the present invention. FIG. 2 is a diagram for explaining an example of a highlight scene extraction method. FIG. 3 is a diagram for explaining another example of the highlight scene extraction method. FIG. 4 is a flowchart illustrating an example of the operation of the video processing device 10 according to the present embodiment of the present invention.

Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, identical parts are given the same reference numerals and the description will be omitted.

(Example of configuration of video processing device)
The video processing device 10 is a device that extracts highlight scenes and generates a highlight video from the extracted highlight scenes. When there are multiple highlight scenes, the highlight video is a combination of the multiple highlight scenes. In this embodiment, the highlight scene refers to the highlight of a sport, typically a score scene. However, some sports do not have a score scene (e.g., sumo). In sumo, where there are no score scenes, a highlight scene corresponds to a scene where a wrestler executes a technique. In this embodiment, the video processing device 10 is described as a device different from the model generation device 50, but some or all of the functions of the model generation device 50 may be incorporated into the video processing device 10.

An example of the configuration of the video processing device 10 and the model generating device 50 will be described with reference to FIG. 1. First, the video processing device 10 will be described. As shown in FIG. 1, the video processing device 10 includes a storage device 11, a control unit 12, and an interface 13.

The storage device 11 is a device that records or saves the content acquired by the video acquisition device 20, and is typically a recorder. As an example, such a recorder is composed of a HDD (Hard Disk Drive) or an SSD (Solid State Drive). Note that cloud storage using cloud computing may be used as the storage device 11. Therefore, the video processing device 10 does not necessarily need to include the storage device 11.

The video acquisition device 20 is, as an example, a device that acquires a television program, and is typically a television. The video acquisition device 20 may also be a smartphone, tablet terminal, personal computer, or game machine that acquires video content through an Internet line. Television programs or video content include sports broadcasts, dramas, movies, documentaries, anime, news, and the like, and in this embodiment, the target of highlight scene extraction is sports broadcasts. Regarding the type of sport, any sport provided through television or an Internet line is the target of highlight scene extraction. Representative sports provided through television or an Internet line include baseball, basketball, American football, ice hockey, soccer, tennis, volleyball, table tennis, rugby, cricket, fencing, golf, and sumo. In this embodiment, sports broadcasts and sports content are collectively referred to as sports video. Therefore, sports video includes sports broadcasts acquired through television and sports content acquired through an Internet line. However, to simplify the explanation, the target of highlight scene extraction is a television program in the following. The functions of the video acquisition device 20 may be incorporated into the video processing device 10. In this case, the video processing device 10 can function as a video camera that is easy to carry. Video cameras that are easy to carry are sometimes called handy cameras. The sports footage mentioned above also includes sports footage shot by ordinary users with video cameras.

As an example, the interface 13 is implemented as hardware such as a network adapter, communication software, or a combination of these, and is configured to realize wired or wireless communication. The interface 13 also functions as an input unit and an output unit for transmitting and receiving data.

The interface 13 is used for communication with the user terminal 30 and with the model generating device 50 via the network 40. The user terminal 30 is a device operated by a user, and is typically a remote control for operating a recorder. However, the user terminal 30 is not limited to a remote control. The user terminal 30 may be a smartphone, a tablet terminal, a personal computer, or a game console. In this embodiment, the user terminal 30 is described as a remote control. The network 40 may be configured using either a wireless or wired system, or both systems, and the network 40 may include the Internet. In this embodiment, the video processing device 10 and the model generating device 50 are connected to the network 40 using a wireless communication system.

The user operates the remote control (user terminal 30) to select a live sports broadcast program from which they wish to extract highlight scenes. In other words, in this embodiment, it is assumed that the live sports broadcast program from which they wish to extract highlight scenes has been recorded in the storage device 11. When the user operates the remote control to select a live sports broadcast program from which they wish to extract highlight scenes, the signal transmitted from the remote control is output to the control unit 12 via the interface 13. The control unit 12 extracts highlight scenes from the live sports broadcast program selected by the user based on the received signal.

The control unit 12 is a general-purpose microcomputer equipped with a CPU (Central Processing Unit), a memory, an input/output unit, and the like. The CPU, memory, input/output unit, and the like are electrically connected via a bus (not shown). A computer program for functioning as the video processing device 10 is installed in the microcomputer. By executing the computer program, the microcomputer functions as multiple information processing circuits equipped in the video processing device 10. Note that here, an example is shown in which multiple information processing circuits equipped in the video processing device 10 are realized by software, but it is of course possible to configure the information processing circuits by preparing dedicated hardware for executing each information process shown below. In addition, the multiple information processing circuits may be configured by individual hardware. The control unit 12 is equipped with a video analysis unit 14 and a highlight extraction unit 17 as multiple information processing circuits. The video analysis unit 14 is further classified into a competition identification unit 15 and a feature detection unit 16.

When the sport identification unit 15 receives a signal transmitted from the user terminal 30, it identifies the type of sports broadcast program (sport) selected by the user. An example of the identification method will be described. When a sports broadcast program is recorded in the storage device 11, tag information indicating the type of the recorded sports broadcast program may be stored at the same time. In such a case, the sport identification unit 15 can identify the type of sports broadcast program by referring to the tag information. Alternatively, the sport identification unit 15 may analyze the video of the sports broadcast program to identify the type of sports broadcast program. For example, if lines are drawn on the court or ground, if a three-point line is detected, it can be identified as basketball, and if a center circle and a penalty box are detected, it can be identified as soccer. It can also be identified from the shape of the uniform, the shape of the ball used in the game, the number of players, etc. The sport identification unit 15 outputs the identified type of sports broadcast program to the feature detection unit 16.

The feature detection unit 16 analyzes the live sports program selected by the user and detects distinctive facial expressions, distinctive actions, and distinctive sounds. Well-known video recognition technology is used to detect distinctive facial expressions and distinctive actions. One example of video recognition technology is template matching using a model (template). Template matching is a technology that detects scenes with high similarity by changing the position and size of a model. As an index of similarity, SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), etc. are used. These are well-known technologies, so detailed explanations are omitted.

An example of a distinctive facial expression is a player's facial expression of joy. Examples of distinctive actions include a player's fist pump, a player raising his hand to respond to the audience, high-fiving other players, and applause by the players or the audience. In other words, distinctive actions are actions that show joy. Well-known voice recognition technology is used to detect distinctive sounds. Examples of distinctive sounds include specific phrases such as "Yay!" and "Come on" that are uttered when a player scores. Alternatively, distinctive sounds also include sounds with a volume above a certain level.

The models of characteristic facial expressions and characteristic movements are generated by the model generation device 50. As will be described in detail later, the model generation device 50 generates models of characteristic facial expressions and characteristic movements by using deep learning as an example.

The feature detection unit 16 analyzes a sports broadcast program by dividing it into predetermined time intervals. For example, if the sports broadcast program is about soccer, the first half of the match lasts 45 minutes, so the feature detection unit 16 divides the video into five-minute intervals and analyzes each part into nine parts. The feature detection unit 16 outputs characteristic facial expressions, characteristic actions, and characteristic sounds detected using template matching and voice recognition technology to the highlight extraction unit 17. Note that five minutes is just an example and can be changed as appropriate depending on the performance of the computer, etc.

The highlight extraction unit 17 (extraction unit, generation unit) extracts a scene (hereinafter sometimes referred to as a reference scene) containing a characteristic facial expression, characteristic action, and characteristic sound detected by the feature detection unit 16 as a highlight scene, as well as the scenes before and after it. The scenes before and after it are defined as scenes including a first scene that goes back a first predetermined time from the reference scene to a second scene that goes forward a second predetermined time from the reference scene. The highlight extraction unit 17 stores the extracted highlight scene as a highlight video in the storage device 11. Note that if multiple highlight scenes are extracted, the highlight extraction unit 17 connects the multiple highlight scenes to generate one highlight video.

When the highlight video is stored in the storage device 11, the video processing device 10 may inform the user via the interface 13 that the generation of the highlight video is complete. This allows the user to enjoy the highlight video.

Next, the model generating device 50 will be described. As shown in FIG. 1, the model generating device 50 includes a server 51, a control unit 52, a storage device 55, and an interface 56. The location where the model generating device 50 is installed is not particularly limited, but for example, the model generating device 50 is installed in a management center owned by the business operator that manages the model generating device 50.

Server 51 stores a large number of past live sports broadcasts and sports content. Server 51 also stores live sports broadcasts and sports content to be broadcast in the future. Note that any structure that functions as a storage medium does not have to be a server. Examples of structures that function as storage media include cassette tapes, CD-ROMs, DVD-ROMs, and BLU-RAY discs.

The control unit 52 is a general-purpose microcomputer, similar to the control unit 12. The control unit 52 has a video acquisition unit 53 and a model generation unit 54 as multiple information processing circuits.

The video acquisition unit 53 acquires previously stored sports broadcast programs and sports content from the server 51. The video acquisition unit 53 outputs the acquired sports broadcast programs and sports content to the model generation unit 54.

The model generation unit 54 generates a model by analyzing the sports broadcast program and sports content acquired from the video acquisition unit 53. The models referred to here are models of characteristic facial expressions and models of characteristic movements, as described above. These models are used in video recognition technology. Well-known AI technology is used to generate the model. Deep learning is used as an example of AI technology. In deep learning, neural networks are used to extract features (characteristic facial expressions, characteristic movements) from large amounts of video data (data related to sports broadcast programs and sports content). This makes it possible to extract the characteristics of movements unique to each sport, in addition to features such as facial expressions of joy and fist pumps that are common to many sports. The same is true for audio. In deep learning, audio models are generated by extracting features such as the strength, frequency, volume, and specific words of the input audio data. When sports videos are input to the neural network, the sports videos are classified by sport, and the classified videos are input to the neural network. The classified videos are learned by the neural network, and a model of characteristic facial expressions, a model of characteristic movements, and a model of characteristic voice are generated for each sport. The models generated in this manner are stored in the storage device 55. Note that, below, a model showing a distinctive facial expression generated by the model generation unit 54 may be referred to as a facial expression model, a model showing a distinctive action may be referred to as a action model, and a model showing a distinctive voice may be referred to as a voice model.

The model generating device 50 transmits the model to the video processing device 10 via the network 40. The transmitted model is stored in the storage device 11. The model stored in the storage device 11 is called up when extracting highlight scenes.

Next, an example of a highlight scene extraction method will be described with reference to Figures 2 and 3. In Figures 2 and 3, the sport is soccer.

The horizontal axis in Figure 2 represents the duration of a soccer match. In Figure 2, a goal is scored 30 minutes into the match, and at 30 minutes and 5 seconds, the player who scored pumps his fist in celebration, and the crowd starts cheering louder. In a soccer broadcast program that includes such scenes, let's say that the user wishes to extract a highlight scene. As mentioned above, the user's wishes are expressed by operating the remote control.

First, when the sport identification unit 15 receives a signal sent from the remote control, it identifies the type of sports broadcast program (sport) selected by the user. In this case, it is assumed that soccer has been identified. Next, the feature detection unit 16 refers to the storage device 11 to obtain facial expression models, movement models, and voice models related to soccer. The feature detection unit 16 divides the soccer broadcast program into predetermined time segments and begins its analysis. In this case, it is assumed that one part is from 25 minutes to 35 minutes. The feature detection unit 16 analyzes the 25-35 minute video to see if there are any scenes that match or are similar to the facial expression models, movement models, and voice models. As a result of the analysis, the feature detection unit 16 detects the players' happy facial expressions, players' fist pumps, and voices (cheering) with a volume above a predetermined volume in the 30 minute 5 second scene.

The highlight extraction unit 17 uses the scene at 30 minutes and 5 seconds as a reference and extracts scenes including a first scene going back a first predetermined time to a second scene going forward a second predetermined time. In the example shown in FIG. 2, the first predetermined time is 10 seconds, and the first scene is the scene at 29 minutes and 55 seconds. The second predetermined time is 5 seconds, and the second scene is the scene at 30 minutes and 10 seconds. In the example shown in FIG. 2, the highlight scene is 15 seconds.

When a soccer player scores a goal, subsequent scenes may include the player celebrating, the player pumping his fist in celebration, and loud cheers. According to this embodiment, the characteristics of the scene after the goal is detected, and a scene including the scenes before and after the goal is generated as a highlight scene. This makes it possible to generate a scene including the goal scene as a highlight scene without extracting the goal scene itself. Furthermore, since the highlight scene is generated taking into account the player's facial expressions and movements in addition to the sound, it is possible to generate a more appropriate highlight scene than with conventional technology.

The feature detection unit 16 extracts scenes in which the facial expression model, movement model, and voice model match or are similar without any time difference. However, a certain degree of time difference may be tolerated. The tolerated time difference is not particularly limited, but may be set to a range of 1 to 2 seconds, for example.

In the example shown in FIG. 2, the scenes before and after the reference scene are included, but this is not limiting. For example, as shown in FIG. 3, a first scene that is a first predetermined time (10 seconds) back from the reference scene may be extracted as a highlight scene. Even in this case, it is possible to extract a scene that includes a scoring scene as a highlight scene, without extracting the scoring scene itself.

Next, an example of the operation of the video processing device 10 will be described with reference to the flowchart in FIG.

In step S101, when the control unit 12 receives a signal transmitted from the user terminal 30 (YES in step S101), the process proceeds to step S103. When no signal is transmitted from the user terminal 30 (NO in step S101), the process waits.

In step S103, the sport identification unit 15 retrieves the sports broadcast program selected by the user from the storage device 11 and identifies the type of sports broadcast program. In this example, it is assumed that the type of sports broadcast program is identified as soccer. Processing proceeds to step S105, where the feature detection unit 16 refers to the storage device 11 to retrieve a facial expression model, a movement model, and a voice model related to soccer.

The process proceeds to step S107, where the feature detection unit 16 divides the soccer broadcast program into predetermined time segments and begins analyzing it. The feature detection unit 16 uses well-known video recognition and voice recognition technologies to analyze each divided part to see if there are any scenes that match or are similar to the facial expression model, movement model, or voice model.

The process proceeds to step S109, where when a scene that matches or is similar to the facial expression model, movement model, and voice model is detected, that scene is extracted as a reference scene. The highlight extraction unit 17 uses the reference scene as a reference and extracts scenes including a first scene that goes back a first predetermined time to a second scene that goes forward a second predetermined time. This results in the extraction of a highlight scene. The process proceeds to step S111, where the highlight extraction unit 17 stores the extracted highlight scene in the storage device 11 as a highlight video.

(Action and Effect)
As described above, the image processing device 10 according to the present embodiment provides the following advantageous effects.

The video processing device 10 extracts a reference scene including a characteristic action and a characteristic sound from a sports video, and generates a highlight scene including a first scene that goes back a first predetermined time based on the extracted reference scene to a second scene that goes forward a second predetermined time. According to this embodiment, the characteristics of a scene after a scoring scene are detected, and a highlight scene including scenes before and after the scene is generated. This makes it possible to generate a highlight scene including a scoring scene without extracting the scoring scene itself. In addition to the sound, the highlight scene is generated taking into account the player's facial expression and the player's action, making it possible to generate a more appropriate highlight scene than with conventional technology. Note that in the above embodiment, both a characteristic action and a characteristic sound are used, but this is not necessarily limited to this. The video processing device 10 may extract a reference scene including either a characteristic action or a characteristic sound from a sports video, and generate a highlight scene.

Each of the functions described in the above embodiments may be implemented by one or more processing circuits. Processing circuits include programmed processing devices, such as processors that include electrical circuitry. Processing circuits also include devices, such as application specific integrated circuits (ASICs) or circuit components, arranged to perform the described functions.

As described above, an embodiment of the present invention has been described, but the descriptions and drawings that form part of this disclosure should not be understood as limiting this invention. Various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art from this disclosure.

Other examples of scenes after a goal has been scored include, in basketball, the goal being broken after a dunk, in tennis, the ball boy picking up the ball (if it is a double fault, a point has been scored), and in volleyball, the players forming a huddle.

Highlight scenes are not limited to scoring scenes. In soccer, highlight scenes can also include offside scenes, scenes in which the referee issues a yellow or red card, or a player continues dribbling for a certain period of time. In such scenes that do not involve scoring, it is difficult for players to show expressions of joy, so two models are generated: a motion model and a voice model.

The sports from which highlight scenes are extracted may be limited. For example, only sports in which players compete for points while splitting into their own half and the opposing half may be the targets for highlight scene extraction. Sports in which players compete for points while splitting into their own half and the opposing half include soccer, basketball, American football, ice hockey, tennis, badminton, volleyball, table tennis, rugby, futsal, and the like. In these sports, as in soccer described above, this embodiment makes it possible to generate scenes including scoring scenes as highlight scenes without extracting the scoring scenes themselves.

Of course, even in these sports where players compete for points while splitting into their own half and the opposing half, highlight scenes are not limited to scoring scenes. In ice hockey, a highlight scene could be the scene where the acrylic panel in front of the spectators breaks. In tennis, badminton, and volleyball, a challenge scene could be a highlight scene. Highlight scenes can be generated by extracting these scenes and the scenes before and after them. A challenge scene refers to a scene where a so-called "challenge" is made, in which a player requests reconfirmation of a decision using video footage or other means when there is doubt about the decision.

The final moments of the game can also be a highlight.

The model generation device 50 may generate models of characteristic facial expressions, characteristic actions, and characteristic voices using only footage of when the match is interrupted in a live sports broadcast. For example, in soccer, footage of when the match is interrupted includes footage from when a goal is scored until the referee blows the whistle to start the match, or footage from when the referee blows the whistle due to a foul or other reason until the referee blows the whistle again to start the match. In other words, when a competitive match is interrupted, it may be defined as the period from the signal to stop the match to when the signal to start the match is given again. Also, a fight scene is a scene when the match is interrupted in all sports.

When analyzing video, in addition to detecting the degree of match with the facial expression model, movement model, and voice model, it is also possible to take into account whether the player in the center of the frame is a certain size or larger, whether the players around the player in the center of the frame are wearing the same uniform, whether they are on the field, etc.

In the above embodiment, highlight scenes of a live sports program selected by the user are extracted, but this is not limited to the above. Regardless of the user's instructions, if a live sports program is stored in the storage device 11, the control unit 12 may automatically extract highlight scenes of the stored live sports program.

10 Video processing device 11, 55 Storage device 12, 52 Control unit 13, 56 Interface 14 Video analysis unit 15 Sport identification unit 16 Feature detection unit 17 Highlight extraction unit 20 Video acquisition device 30 User terminal 40 Network 50 Model generation device 51 Server 53 Video acquisition unit 54 Model generation unit

Claims (5)

An extraction unit that extracts a reference scene including a characteristic action or a characteristic sound from a sports video;
a generation unit that generates, as a highlight scene, a scene including a first scene that is a first predetermined time period back to a second scene that is a second predetermined time period forward based on the reference scene extracted by the extraction unit,
the characteristic action or the characteristic sound is generated in advance as a model for each sport related to the sports video and stored in a storage device, and the generated model is a model when the match of the sport is interrupted,
The extraction unit extracts the reference scene using the model stored in the storage device.
13. A video processing device comprising: The video processing device according to claim 1, characterized in that the characteristic action is a joyful action when the match of the sport is interrupted or a action specific to the sport, and the characteristic sound is a sound containing a specific phrase or a sound having a volume equal to or greater than a predetermined volume . 3. The image processing device according to claim 2 , wherein the period during which the competition is interrupted refers to the period from a signal to interrupt the competition to a signal to restart the competition. Extracting reference scenes including characteristic actions or characteristic sounds from sports videos;
A video processing method for generating, as highlight scenes, scenes including a first scene that is a first predetermined time period back from the extracted reference scene to a second scene that is a second predetermined time period forward from the extracted reference scene,
the characteristic action or the characteristic sound is generated in advance as a model for each sport related to the sports video and stored in a storage device, and the generated model is a model when the match of the sport is interrupted,
extracting the reference scene using the model stored in the storage device;
23. A video processing method comprising: A model generation device for generating a model used by the video processing device according to any one of claims 1 to 3, comprising:
The model generation device is characterized by having a model generation unit that classifies the sports video by sport using deep learning, and generates, for each sport, a facial expression model that shows characteristic facial expressions , a movement model that shows characteristic movements, and a voice model that shows characteristic voices .

Images