JP7628528B2 - Content Processing Device - Google Patents

Description

The present invention relates to a content processing device that can easily provide content that focuses on one person in a group of people who are the subjects of a moving image.

Content such as concerts and plays have traditionally been distributed via networks either live or on-demand (see, for example, Non-Patent Document 1).

In recent years, the content being distributed has tended not to focus on the appearance of a particular idol or actor, but rather to allow fans to enjoy focusing on their favorite performers among the many idols, actors, and other performers.

In addition, recent advances in VR (Virtual Reality) technology have made it possible to create virtual spaces that combine avatars with real-life idols and actors, increasing the need for image processing that focuses on specific performers displayed within content.

Furthermore, a technique has been proposed for extracting multiple people from an image and tracking each of the extracted people (see, for example, Non-Patent Document 2).

https://livr.jp/app-banner https://www.programmersought.com/article/17005126187/

However, many idol groups are made up of a large number of members, and for example, several dozen members may belong to the same group. In such cases, creating content that focuses on just one idol among several dozen idols dancing vigorously on stage can be time-consuming and costly.

It is also difficult to automatically detect a single idol of interest within the content being played. For example, even if you try to identify each member of an idol group using facial image recognition, in the case of a large idol group, the number of pixels in the facial area for each person is significantly smaller, making it difficult to obtain a clear image. Furthermore, when factors such as changes in image quality due to stage lighting and changes in costume design are also taken into account, it is difficult to qualify each person's characteristics.

In addition, the members of an idol group are usually of the same age, sex and country, and there is little difference in characteristics such as skin tone and body type, making it difficult to automatically identify each individual.

One aspect of the present invention aims to realize a technology that makes it possible to easily provide content that focuses on one person in a group of people who are the subjects of a video image.

A content processing device according to one aspect of the present invention includes a content input receiving unit that receives input of content in which moving images are played together with sound, a section detection unit that detects a section from the input content in which sound similar to sound corresponding to a predetermined length of audio data is played, an object detection unit that detects an object from an image played immediately after the detected section, and a label assignment unit that assigns a label to the detected object.

In one aspect of the present invention, the label assignment unit may estimate a label related to a name of the object by a calculation using a model parameter obtained by machine learning.
Each aspect of the present invention may be realized by a computer. In this case, the program for realizing the system on a computer by causing the computer to operate as each part (software element) of the above-mentioned system, and the computer-readable recording medium on which the program is recorded, also fall within the scope of the present invention.

According to one aspect of the present invention, it is possible to realize a technology that makes it possible to easily provide content that focuses on one person in a group that is the subject of a video image.

1 is a block diagram showing an example of a functional configuration of a content processing device according to a first embodiment. FIG. 13 is a diagram showing an example of an image when a song A of an idol group is being performed. FIG. 1 shows an image of a scene from an idol group's performance. FIG. 11 is a diagram showing an example of a GUI that is displayed on a display of a content processing device when an object is designated. 13 is a diagram showing an example of an image that has been subjected to image processing by an object image processing unit; FIG. 13 is a diagram showing another example of an image that has been subjected to image processing by the object image processing unit. FIG. 11 is a flowchart illustrating an example of the flow of a content playback process. FIG. 11 is a block diagram showing an example of a functional configuration of a content processing device according to a second embodiment. FIG. 1 is a diagram illustrating an example of a virtual space. FIG. 1 is a diagram showing an example of the configuration of a computer that executes instructions of a program, which is software that realizes each function of a content processing device.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings.
First Embodiment
(Content processing device)
1 is a block diagram showing an example of the functional configuration of a content processing device 10 according to a first embodiment. As shown in the figure, the content processing device 10 has a content input receiving unit 11, a content playback unit 12, and an operation input receiving unit 13.

The content input receiving unit 11 receives input of content data. The content data is supplied via a network such as the Internet. The content processing device may also access a wide area wireless communication network such as a 5G communication system, and the content data may be supplied via the wide area wireless communication network.

As an example, the content is MP4 format data recorded from a concert, a play, etc. The content data includes moving images and audio, and may also include information such as subtitles as necessary. In other words, the content input to the content processing device is content in which moving images are played together with audio. For example, the audio may be a piece of music played at a concert.

Here, we will mainly use as an example a case where the content is MP4 format data filmed at an idol group concert. At an idol group concert, a group consisting of multiple idols (people) sing and dance along with the music being played. The singing and dancing along with the music is also called a performance.

The content playback unit 12 plays back the content whose input has been accepted by the content input acceptance unit 11. The played back content is displayed, for example, on the display 50. The display 50 is also equipped with a speaker, and the sound of the played back content is output from the speaker of the display 50.

The operation input reception unit 13 receives operation input from a user to the content processing device 10. The operation input reception unit 13 may be configured with a keyboard, a mouse, etc. Alternatively, the display 50 may be configured to include a touch sensor, and the operation input reception unit 13 may be configured by detecting operation input of a GUI (Graphical User Interface) displayed on the display 50.

As shown in FIG. 1, the content processing device 10 also has a section detection unit 31, an object detection unit 32, a label assignment unit 33, and an object image processing unit 34.

(Section detection unit)
The section detection unit 31 detects, from the input content, a section in which a sound similar to a sound corresponding to audio data of a given length is reproduced.

The audio data provided in advance here is audio data corresponding to audio of a duration sufficiently short compared to the overall time length of the input content. For example, if the input content is a concert by an idol group, audio data corresponding to parts of a song performed at the concert, such as the intro, interlude, and chorus, is provided in advance. However, the audio of the audio data is not limited to these, and may be any number of consecutive bars of sound in a song. The audio data may be stored, for example, in a memory unit of the content processing device 10 (not shown).

The audio may include instrumental sounds, singing, sound effects, etc.

The section detection unit 31, for example, calculates the similarity between the two audio signals by comparing the features of the audio signal of pre-given audio data with the audio playback signal of the content being played. During playback of the content, the section detection unit 31 continuously extracts audio signals of the same temporal length as the audio data, and detects sections where the similarity is equal to or greater than a threshold by comparing the features of the extracted audio signals with the features of the audio signal of the audio data. The section detected here will have the same temporal length as the audio data.

(Object detection unit)
The object detection unit 32 detects an object from an image played back immediately after the detected section.

The image played immediately after the section detected by the section detection unit 31 is an image played temporally after that section. For example, it may be one or more images from among the images 1 to 30 frames after the final frame of the multiple frames included in the video of that section. For example, if the input content is the content of an idol group concert, the image played immediately after the detected section will show several idols (people), and the images of these people will be detected as objects by the object detection unit 32.

As an example, object detection can be performed using the graph cut method. In the graph cut method, the boundary of the area that constitutes the foreground object image to be cut out is first calculated from the color distribution and pixel color gradient of two types of images: a foreground object image containing the object to be cut out, and a background image. The image is then cut out along the calculated boundary, and the foreground object image to be cut out is extracted.

(Labeling unit)
The label assignment unit 33 assigns a label to the object detected by the object detection unit 32 .

Here, the processing of the label assignment unit 33 will be described using an example of content for an idol group concert. In this case, the detected object is a person, and the label assignment unit 33 assigns a label related to the person's name. In other words, the person's name is assigned to the object as a label.

The label assignment unit 33 is configured, for example, by a convolutional neural network (CNN) and executes a process of estimating the labels of objects in an input image. At this time, the label assignment unit 33 estimates the labels of the objects using model parameters obtained by machine learning executed in advance.

In this machine learning, for example, data in which correct labels are assigned to objects detected from images captured in advance of an idol group concert is used as training data. In general, idol group concerts are often held frequently at the same venue. In this case, there may be a large amount of video data of the same idol group captured from the same viewpoint.

Also, when idol groups hold frequent concerts, they usually play the same songs at each concert. Idol groups often dance in a fixed formation and with fixed choreography for each song. So, for example, when song A is played, immediately after the interlude, a set number of people will stand in the front row and take the same pose.

Figure 2 shows an example of an image when a song A by an idol group is being played. The figure shows an image 101 displayed on the display 50, and image 101 is an image taken immediately after the interlude of song A. The figure shows three people, 111, 112, and 113, each of whom is posing with their right hand raised.

While a song is being played, each member of an idol group often moves vigorously, but there are times during the song when each member of the idol group becomes almost still. For example, the image in Figure 2 corresponds to such a scene. In an idol group performance where the dancers follow a set choreography, the relative positions of each member and the poses they will take in such a scene are known in advance, making it relatively easy to estimate the labels of objects in the image.

In FIG. 2, person 111, person 112, and person 113 are each detected as an object in the image. Data to which the names (first name, nickname, etc.) of person 111, person 112, and person 113 are assigned as labels is used as training data in machine learning of model parameters used by label assignment unit 33. If there is a large amount of video data of the same idol group shot from the same viewpoint, it is possible to generate a large amount of such training data. Machine learning using a large amount of training data also improves the accuracy of the estimation results of label assignment unit 33.

In the example shown in FIG. 2, the three people are all in the same pose, but if there is a scene in which each person stands still in a different pose, it will be easier to identify each person. Machine learning using images of such scenes can further improve the accuracy of the estimation results of the labeling unit 33.

(Object Tracking)
Furthermore, the object detection unit 32 tracks the object to which the label has been assigned within the video of the content that is played back temporally after the section detected by the section detection unit.

In object tracking, for example, the similarity between an object detected from an image in a certain frame and an object detected from an image one frame later is calculated. For example, the similarity of the objects is calculated by comparing features related to the color, shape, and pattern of the objects. Objects that have a similarity above a threshold are then identified as the same object, thereby tracking the objects.

By tracking objects in this way, once a person has been labeled, that same label will continue to be assigned to them in the video images displayed during content playback.

Note that, for example, object detection may be performed multiple times while the same piece of music is being played, thereby enabling the object to be tracked more accurately. For example, first audio data corresponding to the sound played in the intro and second audio data corresponding to the sound played in the interlude of the same piece of music may be provided in advance.

In this case, first audio data corresponding to a first audio, which is the audio of a section that is played earlier in time in the content, and second audio data corresponding to a second audio, which is the audio of a section that is played later in time in the content, are provided in advance. Then, the section detection unit 31 detects the first section in which an audio similar to the first audio is played, and the second section in which an audio similar to the second audio is played, respectively.

In this case, the labeling unit 33 estimates the name of the object detected by the object detection unit 32 from the image immediately after the first section and labels it, and re-labels the object detected by the object detection unit from the image immediately after the second section. The object detection unit 32 then tracks the object labeled in accordance with the first section and the object labeled in accordance with the second section.

For example, if an idol group has a large number of members, it is difficult to obtain an image that shows all of the members in one scene. FIG. 3 shows an image 131 displayed on the display 50, which is a scene from a performance by an idol group consisting of eight members. FIG. 3 corresponds, for example, to a scene immediately after the intro of a certain song.

In the scene shown in FIG. 3, all eight members are almost still and in the same pose, but person 142 is located behind person 141, and person 144 is located behind person 143. It is difficult to detect and track person 142 and person 144 as objects from the image of the scene shown in FIG. 3. In addition, when the labeling unit 33 estimates the names of the detected people in the scene shown in FIG. 3, it is difficult to correctly estimate the names of person 142 and person 144.

On the other hand, in idol group performances, there are often multiple scenes in which all members are almost completely still during the performance of a single song. Since the formation changes depending on the scene, it is possible to select scenes in which a person who is not visible in one scene is visible in another.

For example, if audio data corresponding to the sound played when an idol group changes formation in a song is provided in advance, the names of multiple people positioned in the front row of the stage can be estimated each time the formation changes. Also, each of the multiple people positioned in the front row of the stage can be tracked each time the formation changes.

That is, multiple pieces of audio data (first audio data, second audio data, ...) may be provided in advance, and labels may be assigned to the first scene, second scene, ... that is played immediately after the section corresponding to each piece of audio data. For example, if people are detected and labeled in each of the multiple scenes, it may be possible to track each person more accurately while the content is being played.

For example, if the same sound is played repeatedly in a piece of music, even if only one piece of sound data is provided, it is still possible to assign a label each time in the first scene, the second scene, etc.

(Object image processing section)
The object image processing unit 34 performs a predetermined image processing on an object designated by a user among the plurality of objects detected by the object detection unit 34. The object is designated based on, for example, a label attached to the object.

Figure 4 is a diagram showing an example of a GUI displayed on the display 50 of the content processing device 10 when an object is specified. In this example, "Idol group XXX 8th generation member list" is displayed. Here, "XXX" indicates the name of the idol group that will perform at the concert of the content being played. This idol group is made up of, for example, 20 members, and the member list is updated every time at least one member is replaced. The member list at the time the group was formed is the first generation, and thereafter, the member list is updated to the second generation, third generation, ... each time at least one member is replaced.

In the member table shown in FIG. 4, the leftmost column shows "Members," and lists the names of the 20 members who make up idol group XXX. Here, each person's name is indicated by "AAA," "BBB," "CCC," ... Note that in reality, each person's name is associated with an identification number, and each identification number corresponds to a label assigned by label assignment unit 33.

In the member table shown in Figure 4, the center column shows "Profile", which describes the profile of each member.

In the member table shown in FIG. 4, the rightmost column shows "Attention," and this column is where the user makes their designation. For example, the user inputs an operation for designating a person via the operation input receiving unit 13, and designates a person on whom the user has an interest in the member table shown in FIG. 4. In this example, the user has an interest in the member "CCC," and a star mark indicating that this person ("CCC") has been designated is displayed in the "Attention" column.

The object image processing unit 34 applies a predetermined image processing to an image of a person that the user focuses on, among the images of the content being played back. Figure 5 shows an image 161 that is displayed on the display 50 and is an example of an image that has been subjected to image processing by the object image processing unit 34.

Here, for example, it is assumed that the user has specified person 113 as the person of interest using the GUI described above with reference to FIG. 4. In the example of FIG. 5, a mark (in this example, a heart-shaped mark) 171 is displayed superimposed near person 113. As an example of image processing by object image processing unit 34, as shown in FIG. 5, a predetermined image (in this example, mark 171) is displayed superimposed within a predetermined range near the specified object.

Figure 6 is a diagram showing another example of an image that has been subjected to image processing by the object image processing unit 34. In the example of Figure 6, an image 191 in which a person 113 is displayed in an enlarged form is displayed on the display 50. In other words, only the person 113 specified by the user is displayed in an enlarged form on the display 50. As an example of image processing by the object image processing unit 34, the specified object is displayed in an enlarged form, as shown in Figure 6.

Furthermore, a mark such as that shown in FIG. 5 may be superimposed on an enlarged image such as that shown in FIG. 6.

The image processing by the object image processing unit 34 is continuously performed on the images of each frame constituting the moving image of the content being played. For example, when a mark 171 is superimposed as shown in FIG. 5, the mark 171 will be displayed near the person 113 dancing to the music at all times while the content is being played. Also, when an object is displayed in an enlarged manner as shown in FIG. 6, an image showing almost only the person 113 will be displayed on the display at all times while the content is being played.

Next, an example of the flow of content playback processing by the content processing device 10 will be described with reference to the flowchart in FIG. 7. This processing is executed after the content input receiving unit 11 receives input of content data. Here, an example will be described in which content data of a live-streamed concert by an idol group is input.

In step S101, the content playback unit 12 plays the input content.

In step S102, the section detection unit 31 calculates the similarity between the sound corresponding to the pre-given audio data and the sound of the content being played. As described above, audio data of parts of a piece of music played at a concert, such as an intro, interlude, chorus, etc., is given in advance. The section detection unit 31 calculates the similarity between the two audio signals, for example, by comparing the features of the audio signal of the pre-given audio data with the playback signal of the audio of the content being played.

In step S103, the section detection unit 31 determines whether a similar section has been detected. As described above, the section detection unit 31 continuously extracts audio signals of the same time length as the audio data during playback of the content, and compares the extracted audio signal features with the audio signal features of the audio data. Then, the section detection unit 31 detects, for example, a section in which the similarity is equal to or greater than a threshold value as a similar section.

In addition, when multiple pieces of audio data are provided in advance, the similarity of the audio signals is calculated by comparing each of the features of the audio signals of the multiple pieces of audio data with the audio signal of the audio of the content being played. In this way, for example, a section similar to the audio of the first audio data, or a section similar to the audio of the second audio data, etc. are detected as similar sections.

If it is determined in step S103 that a similar section has been detected, the process proceeds to step S104.

In step S104, the object detection unit 32 detects an object from the image played immediately after the detected section. At this time, for example, an image of one or more people appearing in the image played immediately after the detected section is detected as an object by the object detection unit 32.

In step S105, the label assignment unit 33 estimates the name of the object. At this time, the name of the person detected as the object in the processing of step S104 is estimated. As described above, the label assignment unit 33 is configured, for example, by a CNN, and executes processing to estimate the label of the object of the input image. At this time, the label assignment unit 33 estimates the name of the object using model parameters obtained by machine learning executed in advance.

In step S106, the labeling unit 33 assigns the name estimated in the process of step S105 as a label to the object detected in the process of step S104. In other words, the name of the person is assigned to the object as a label.

In step S107, the object detection unit 32 tracks the object to which the label was assigned in the process of step S106. At this time, for example, the similarity between an object detected from an image of a certain frame and an object detected from an image one frame later is calculated, and objects having a similarity equal to or greater than a threshold value are identified as the same object, thereby tracking the object.

In step S108, the object image processing unit 34 applies a predetermined image processing to the object specified by the user from among the multiple objects detected by the object detection unit 32.

As a result, for example, as described above with reference to FIG. 5, a mark is displayed near the specified object. Alternatively, for example, as described above with reference to FIG. 6, only the person specified by the user is enlarged and displayed. Furthermore, a mark such as that shown in FIG. 5 may be superimposed on the image enlarged as in FIG. 6.

The image processing by the object image processing unit 34 is performed continuously on each frame of images that make up the moving image of the content being played.

If no object is specified, step S108 is skipped.

If it is determined in step S103 that no similar section has been detected, steps S104 to S106 are skipped and step S107 is executed. If no label has yet been assigned, steps S107 and S108 cannot be executed, and so these steps are also skipped.

In step S109, the content playback unit 12 determines whether the content has been played to the end. If it is determined in step S109 that the content has not been played to the end, the process returns to step S102, and the section detection unit 31 compares the extracted audio signal feature quantity with the audio signal feature quantity of the audio data. The section detection unit 31 then detects sections whose similarity is equal to or greater than a threshold value as similar sections.

In this way, by repeatedly executing the processes of steps S102 to S109, a section corresponding to pre-given audio data is detected once or multiple times (step S103), and an object is detected from the image immediately following the detected section (step S104). Then, the name of the detected object is estimated (step S105), and a label is assigned (step S106). Once a label has been assigned to the object, it is tracked within the image of the content being played back (step S107), and image processing is performed on the designated object (step S108).

The processing of steps S104 to S106 is performed each time a section similar to the sound of pre-given audio data is detected, so that, for example, a person who could not be detected in the first scene can be detected in the second scene. Also, even if the estimation of the person's name is incorrect in the first scene, it is possible to correct the label to the correct one in the second scene. Furthermore, even if a different object (person) is mistakenly tracked after the first scene, it is possible to track the correct object (person) from the second scene onwards.

If it is determined in step S109 that the content has been played to the end, the content playback process ends.

In this way, the content playback process is carried out.

In the above explanation, the content is mainly MP4 format data filmed from an idol group concert, but the same processing may be performed on content filmed from a play, for example. A play usually features multiple actors, and if audio data corresponding to specific lines or sound effects is provided in advance, it also becomes easier to estimate the names of people, who are objects detected in the images of the content.

In other words, in a scene during a play, when a particular line or sound effect is spoken, the relative positions of each person and the pose they will be in are known in advance, making it relatively easy to estimate the labels of objects in the image.

(Effects of the First Embodiment)
As described above, according to this embodiment, it is possible to assign a label to an object detected in an image of a content to be played. At this time, the section detection unit 31 detects a section of an audio similar to the audio of pre-given audio data, so it becomes easier to predict in advance which object will be detected at which position in the image of the content to be played. It also becomes easier to predict in advance what pose a person, which is an object to be detected, will be in.

As a result, it is possible to estimate the label of an object based on features such as the position and pose of the person that is the object, making it possible to estimate the label with high accuracy even if the image quality of the content is low. Or, even if the lighting or costumes at a concert are changed, it is still possible to estimate the label with high accuracy.

Furthermore, since labels are estimated for objects that are in the same positional relationship and have the same pose as objects in the training data, there is a high possibility that labels can be estimated even when learning with a small amount of training data. Therefore, highly accurate label estimation is possible regardless of the degree of learning of the model parameters used by the label assignment unit 33.

In this way, by tracking objects with estimated labels, it becomes possible to apply image processing that focuses on a specified person. In this way, for example, fans can enjoy focusing on their favorite performers among many performers such as idols and actors.

Therefore, according to this embodiment, it is possible to easily provide content that focuses on one person in a group that is the subject of a video image.

Second Embodiment
8 is a block diagram showing an example of a functional configuration of a content processing apparatus 10 according to the second embodiment. Note that components having the same functions as those described in the first embodiment are denoted by the same reference numerals, and descriptions thereof will be omitted as appropriate.

The content processing device 10 shown in FIG. 8 includes a motion analysis unit 61 and an animation image generation unit 71. The rest of the configuration is the same as that of the content processing device 10 shown in FIG. 1.

(Motion Analysis Section)
The motion analysis unit 61 analyzes the image of the content to analyze the motion of an object (e.g., a person) detected in the image. The motion analysis unit 61 also identifies, for example, the motion of a specified object. For example, it identifies the motion of the person's arms, legs, head, etc. in a section having a predetermined time length in the content. Note that the specified object may be, for example, an object specified as a person of interest by the user's operation described above with reference to FIG. 4.

As an example, the motion analysis unit 61 identifies the joint positions of a person, which is an object detected in an image, and analyzes how each part of the person, such as the arms, legs, and head, is moving. Note that in the processing by the motion analysis unit 61, the joint positions of the person and/or the movement of each part may be estimated using model parameters obtained by prior machine learning, for example.

The analysis results of the motion analysis unit 61 are supplied to the animation image generation unit 71 as information representing the movement of the person.

(Animation image generation unit)
The animation image generator 71 generates an animation image based on the analysis result of the motion analyzer 61. Here, the generated animation image is, for example, an image of a moving character. As an example, the character has arms, legs, and a head similar to a human body shape, and moves its joints in the same way as a human.

The animation image generator 71 generates an animation image that moves a character based on the information representing the person's movement supplied from the motion analyzer 61. That is, an animation image is generated of a character that moves in the same way as the person's movement identified by the motion analyzer 61.

Note that in the example of FIG. 8, the animation image generating unit 71 is provided inside the content processing device 10, but the animation image generating unit 71 may be provided, for example, in a device different from the content processing device 10.

That is, the motion analysis unit 61 analyzes the motion of a person designated by the user from among the multiple people detected by the object detection unit 32, and the analysis result of the motion analysis unit 61 is supplied to the animation image generation unit 71, which generates an animation image that moves in the same manner as the designated person.

(Character display)
The animation of the moving character may be displayed, for example, on the display 50. In this way, for example, the user can imitate the movements of a person of interest in an idol group. For example, the user can dance along to a song with the idol that the user likes.

In addition, the animation of a moving character may be used, for example, as an avatar displayed in a virtual space. That is, the animation image generated by the animation image generating unit 71 may be an image of a user's avatar. FIG. 9 is a diagram showing an example of a virtual space. The virtual space 200 shown in FIG. 9 is modeled after an event venue. The figure also shows characters 211 to 216 used as avatars.

Each of the avatars is a character that represents each user in the virtual space 200, and is set to move within the virtual space 200 based on, for example, the user's operation. That is, each of the characters 211 to 216 is associated with a different user, and moves and moves within the virtual space 200 according to the user's operation.

For example, avatars of users who are fans of an idol group can gather on stage 200a in virtual space 200 and dance to the music. For example, each user can analyze the movements of a person in the idol group that catches their attention with motion analysis unit 61, and generate an animation image of the user's avatar character moving with animation image generation unit 71. By playing this animation image on stage 200a in virtual space 200, users can enjoy watching their avatars dancing to the same choreography as the idol group.

Furthermore, an image of an idol group's performance may be superimposed and displayed on the image of the virtual space 200.

(Effects of the Second Embodiment)
As described above, according to this embodiment, it is possible to analyze the movement of a specified object among objects detected from an image of a content, and generate an animation image that moves in the same way as the specified object. In this way, for example, the variety of ways in which fans can enjoy the content increases, focusing on their favorite performers among a large number of performers such as idols and actors.

Therefore, according to this embodiment, it is possible to easily provide content that focuses on one person in a group that is the subject of a video image.

<Other embodiments>
In the above-described embodiment, the content processing device 10 may be configured, for example, by a personal computer, a game machine, etc., or may be configured by a smartphone, etc. Alternatively, some functions of the content processing device 10 may be realized by a personal computer, etc., and other functions may be realized by a smartphone, etc.

In addition, in the above-described embodiment, the camera used to capture the content to be played back by the content processing device 10 may be selected arbitrarily.

On the other hand, for example, a stereoscopic camera may be used to capture content. In this way, an image with parallax can be captured, and the content played back by the content processing device 10 can be displayed in 3D. Note that content may be captured using two cameras to obtain an image with parallax.

For example, a 360-degree camera may be used to capture content. A 360-degree camera has a wide-angle lens, such as a fisheye lens, and can capture panoramic view images. In content generated from images captured using a 360-degree camera, it is possible to display images viewed from any viewpoint, for example.

In this way, for example, it is possible to play back an image from a viewpoint in which a person specified by the user stands out. Also, by shooting content using two 360-degree cameras, it is possible to display an image with parallax that focuses on a specified person.

<Example of software implementation>
The above-mentioned content processing device 10 is a program for causing a computer to function, and can be realized by a program for causing a computer to function as the content processing device 10. In this case, the content processing device 10 includes a computer having at least one control device (e.g., a processor) and at least one storage device (e.g., a memory) as hardware for executing the above-mentioned program. An example of such a computer is shown in FIG. 10.

The computer 500 includes at least one processor 501 and at least one memory 502. The memory 502 stores a program 520 for operating the computer 500 as a content processing device 10. In the computer 500, the processor 501 reads and executes this program 520 from the memory 502, thereby realizing each function of the content processing device 10.

The processor 501 may be, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a microcontroller, or a combination of these.

For example, the memory 502 may be a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

The computer 500 may further include a RAM (Random Access Memory) for expanding the program 520 during execution and for temporarily storing various data. The computer 500 may further include a communication interface for transmitting and receiving data to and from other devices. The computer 500 may further include an input/output interface for connecting input/output devices such as a keyboard, mouse, display, and printer.

The program 520 for operating the computer 500 as the content processing device 10 can be recorded on a non-transitory, tangible recording medium 530 that can be read by the computer 500. For example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit can be used as such a recording medium 530. The computer 500 can obtain the program 520 via such a recording medium 530.

The program 520 for operating the computer 500 as the content processing device 10 can be transmitted via a transmission medium. For example, a communication network or broadcast waves can be used as such a transmission medium. The computer 500 can also obtain the program 520 via such a transmission medium.

Furthermore, some or all of the functions of the content processing device 10 can be realized by logic circuits. For example, an integrated circuit in which logic circuits that function as each of the above control blocks are formed is also included in the scope of the present invention. In addition, it is also possible to realize the functions of each of the above control blocks by, for example, a quantum computer.

According to each aspect of the present invention described above, the above-mentioned effects can be achieved, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization and innovation."

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in the different embodiments.

〔summary〕
A content processing device according to aspect 1 of the present invention includes a content input receiving unit that receives input of content in which moving images are played together with audio, a section detection unit that detects a section from the input content in which audio similar to audio corresponding to audio data of a predetermined length that has been given in advance is played, an object detection unit that detects an object from an image that is played immediately after the detected section, and a label assignment unit that assigns a label to the detected object.

In the content processing device according to aspect 2 of the present invention, in the above aspect 1, the object is a person, and the label assignment unit assigns a label related to the name of the person.

In the content processing device according to aspect 3 of the present invention, in the above aspect 1 or 2, the label assignment unit estimates the label of the object by performing a calculation using model parameters obtained by machine learning.

A content processing device according to aspect 4 of the present invention is any one of aspects 1 to 3 above, in which the audio played together with the moving image is music.

In the content processing device according to aspect 5 of the present invention, in any one of aspects 1 to 4 above, the object detection unit further tracks the object to which the label has been assigned in a video of the content that is played back after the section detected by the section detection unit.

In the content processing device according to aspect 6 of the present invention, in the above aspect 5, first audio data corresponding to a first audio, which is an audio of a section played earlier in time in the content, and second audio data corresponding to a second audio, which is an audio of a section played later in time in the content, are provided in advance, the section detection unit detects a first section in which an audio similar to the first audio is played and a second section in which an audio similar to the second audio is played, and the object detection unit tracks an object to which a label is assigned corresponding to the first section and an object to which a label is assigned corresponding to the second section, respectively.

The content processing device according to aspect 7 of the present invention is the above-mentioned aspect 5 or aspect 6, further comprising an object image processing unit that performs a predetermined image processing on an object designated by a user from among the multiple objects detected by the object detection unit.

In the content processing device according to aspect 8 of the present invention, in the above aspect 7, the object image processing unit performs image processing to enlarge and display the specified object.

In the content processing device according to aspect 9 of the present invention, in the above aspect 7 or 8, the object image processing unit performs image processing to superimpose and display a predetermined image within a predetermined range in the vicinity of the specified object.

The content processing device according to aspect 10 of the present invention is any one of aspects 5 to 9 above, further comprising a motion analysis unit that analyzes the motion of the detected object.

In the content processing device according to aspect 11 of the present invention, in the above aspect 10, the object is a person, and the motion analysis unit analyzes the motion of a person designated by a user from among a plurality of people detected by the object detection unit, and the analysis result of the motion analysis unit is supplied to an animation image generation unit that generates an animation image that moves in the same manner as the designated person.

In a content processing device according to aspect 12 of the present invention, in the above aspect 11, the animation image is an image of the user's avatar.

The content processing method according to aspect 13 of the present invention includes the steps of accepting input of content in which moving images are played together with sound, detecting a section from the input content in which sound similar to sound corresponding to a predetermined length of audio data is played, detecting an object from an image played immediately after the detected section, and assigning a label to the detected object.

The program according to aspect 14 of the present invention causes a computer to function as a content processing device having an input receiving unit that receives input of content in which moving images are played together with audio, a section detection unit that detects a section from the input content in which audio similar to audio corresponding to audio data of a predetermined length is played, an object detection unit that detects an object from an image played immediately after the detected section, and a label assignment unit that assigns a label to the detected object.

REFERENCE SIGNS LIST 10 Content processing device 11 Content input receiving unit 12 Content playback unit 13 Operation input receiving unit 31 Section detection unit 32 Object detection unit 33 Labeling unit 34 Object image processing unit 50 Display 61 Motion analysis unit 71 Animation image generation unit

Claims (9)

an input receiving unit that receives an input of a content for playing back a moving image of a plurality of people together with audio;
a section detection unit that detects, from the input content, a section in which a sound similar to a sound corresponding to a given audio data of a predetermined length is reproduced;
an object detection unit that detects, as objects , images of a plurality of people appearing in an image played immediately after the detected section;
a label assignment unit that assigns a label corresponding to a name of the object to the object detected by the object detection unit by using model parameters obtained by machine learning executed in advance . The content processing device according to claim 1 , wherein the sound reproduced together with the moving image is a piece of music. The object detection unit further
The content processing device according to claim 1 , further comprising: a section for detecting the object to which the label has been assigned, the object being tracked and detected within a moving image of the content that is played back after the section detected by the section detection section. First audio data corresponding to a first sound, which is a sound of a section to be reproduced earlier in time in the content, and second audio data corresponding to a second sound, which is a sound of a section to be reproduced later in time in the content, are provided in advance;
the section detection unit detects a first section in which a sound similar to the first sound is reproduced and a second section in which a sound similar to the second sound is reproduced;
The content processing device according to claim 3 , wherein the labeling unit assigns a label to an object detected by the object detection unit from the image immediately after the first section, and assigns a label again to an object detected by the object detection unit from the image immediately after the second section. The content processing device according to claim 3 , further comprising an object image processing unit that performs predetermined image processing on an object designated by a user from among the plurality of objects detected by the object detection unit. The content processing device according to claim 5 , wherein the object image processing unit performs image processing for enlarging and displaying the specified object. The content processing device according to claim 5 , wherein the object image processing unit displays a predetermined image in a superimposed manner within a predetermined range near the specified object. The content processing device according to claim 3 , further comprising a motion analysis unit that analyzes the motion of the detected object. the motion analysis unit analyzes a motion of a person designated by a user among a plurality of people detected as objects by the object detection unit,
The content processing device according to claim 8 , wherein the analysis result of the analysis unit is supplied to an animation image generation unit that generates an animation image that moves in the same manner as the specified person.

Images