JP7640248B2 - Image generating device and image generating method - Google Patents

Description

The present invention relates to an image generating device and an image generating method.

Conventionally, a device that creates subtitles from video data is known. In this device, a face detection means detects facial features and face positions from the video data, and a voice recognition means detects voice features from the video data. Each detected feature is sent to a speaker identification means and compared with the features of the speaker registered in a voice/face correspondence data storage means, and as a result, the position of the speaker is identified. The voice of the identified speaker is converted to text by a voice recognition means. A speech bubble creation means creates a speech bubble based on the speaker's position and text data, and a video creation means combines the video data, audio data, and speech bubble data to create new video data (see, for example, Patent Document 1).

JP 2007-027990 A

However, the above-mentioned conventional techniques have the following problems. If the facial features or audio features detected from video data are not registered in the audio/face correspondence data storage means, there is a possibility that the speaker and audio will be incorrectly linked. As a result, the audio of a different speaker may be displayed in the speech bubble, and the viewer may erroneously recognize the correspondence between the speaker and audio.

One aspect of the present invention aims to enable viewers to properly recognize the association between objects and text data.

In order to solve the above problem, an image generating device according to one aspect of the present invention includes an acquisition unit that acquires image data representing an image and audio data representing audio accompanying the image, a specific processing execution unit that executes a specific processing to identify an object that is generating audio among one or more objects included in the image by inputting the image data and audio data acquired by the acquisition unit into a trained model, and a superimposition unit that superimposes text data corresponding to the audio data at a position within the image that corresponds to the result of the specific processing.

To solve the above problem, an image generating device according to another aspect of the present invention includes an acquisition unit that acquires image data representing an image and audio data representing audio accompanying the image, a specific processing execution unit that executes a specific processing to identify an object that is generating audio among one or more objects included in the image by inputting the image data and audio data acquired by the acquisition unit into a trained model, and a superimposition unit that generates an image on which information identifying the object is superimposed, the information having a display form corresponding to accuracy information indicating the accuracy of the identification processing.

In order to solve the above problems, an image generating method according to one aspect of the present invention includes an acquisition step of acquiring image data representing an image and audio data representing audio accompanying the image, a specific processing execution step of executing a specific processing step of identifying an object that is generating audio among one or more objects included in the image by inputting the image data and audio data acquired in the acquisition step into a trained model, and a superimposition step of superimposing text data corresponding to the audio data at a position within the image that corresponds to the result of the specific processing.

In order to solve the above problem, an image generating method according to another aspect of the present invention includes an acquisition step of acquiring image data representing an image and audio data representing audio accompanying the image, a specific processing execution step of executing a specific processing step of identifying an object that is generating audio among one or more objects included in the image by inputting the image data and audio data acquired in the acquisition step into a trained model, and a superimposition step of generating an image on which information identifying the object is superimposed, the information having a display form corresponding to accuracy information indicating the accuracy of the identification process.

According to one aspect of the present invention, the viewer can properly recognize the association between the object and the text data.

1 is a perspective view of a display device according to a first embodiment of the present invention. 2 is a block diagram showing a configuration of a main part of an image generating device included in the display device shown in FIG. 1 . 2 is a diagram showing an example of a display displayed on a display unit of the display device shown in FIG. 1 . 1. FIG. 4 is a diagram showing another example of display displayed on the display unit of the display device shown in FIG. 1. FIG. 4 is a diagram showing another example of display displayed on the display unit of the display device shown in FIG. 1. FIG. 4 is a diagram showing another example of display displayed on the display unit of the display device shown in FIG. 4 is a flowchart showing an image generating method executed by an image generating device included in the display device shown in FIG. 1 . FIG. 11 is a block diagram showing a configuration of a main part of an image generating device according to a second embodiment of the present invention. FIG. 9 is a diagram showing an example of a display displayed on the display unit shown in FIG. 8 . 9A and 9B are diagrams illustrating other examples of display displayed on the display unit illustrated in FIG. 8 . 9 is a flowchart showing an image generating method executed by the image generating device shown in FIG. 8 .

[Embodiment 1]
Hereinafter, an embodiment of the present invention will be described with reference to FIGS.

Figure 1 is a front perspective view of a display device 1 according to this embodiment. In this embodiment, the display device 1 is realized as a television receiver. As shown in Figure 1, the display device 1 includes at least an image generating device 10 and a display unit 40.

The configuration of the image generating device 10 included in the display device 1 shown in FIG. 1 will be described in detail below with reference to FIG. 2. FIG. 2 is a block diagram showing the main configuration of the image generating device 10.

As shown in FIG. 2, the image generating device 10 includes an acquisition unit 11, a voice/image recognition unit 12, a text data generating unit 13, a subtitle display position determining unit 14, and a superimposing unit 15. The acquisition unit 11 is an example of an "acquisition unit" in the claims. The voice/image recognition unit 12 and the subtitle display position determining unit 14 are an example of a "specific processing execution unit" in the claims. The superimposing unit 15 is an example of a "superimposing unit" in the claims. In FIG. 2, the reference numeral 20 indicates a voice recognition DB, the reference numeral 30 indicates an image recognition DB, and the reference numeral 40 indicates a display unit. The voice recognition DB 20 and the image recognition DB 30 may be (1) provided inside the display device 1, or (2) provided outside the display device 1 and configured to be able to communicate with the display device 1.

The acquisition unit 11 is configured to acquire image data representing an image and audio data representing audio accompanying the image. As an example, the acquisition unit 11 separates and extracts image data and audio data from data obtained by demodulating broadcast waves. As another example, the acquisition unit 11 separates and extracts image data and audio data from data obtained by decoding content data. Note that the image represented by the image data may be a still image or a moving image (video). Then, as shown in FIG. 2, the image data acquired by the acquisition unit 11 is output to each of the image recognition unit provided in the audio/image recognition unit 12 and the superimposition unit 15. Also, the audio data acquired by the acquisition unit 11 is output to each of the text data generation unit 13 and the audio recognition unit provided in the audio/image recognition unit 12.

The audio/image recognition unit 12 is configured to (1) identify the object that emitted the sound represented by the audio data acquired by the acquisition unit 11, (2a) identify each object included as a subject in the image represented by the image data acquired by the acquisition unit 11, and (2b) specify the position of each object on the screen. The audio/image recognition unit 12 outputs (1a) identification information of the object that emitted the sound represented by the audio data, (2a) identification information of each object included as a subject in the image represented by the image data, and (2b) position information of each object included as a subject in the image represented by the image data to the subtitle display position determination unit 14.

The voice/image recognition unit 12 can be composed of a voice recognition unit and an image recognition unit, for example, as shown in FIG. 2.

The voice recognition unit identifies the object that emitted the voice represented by the voice data acquired by the acquisition unit 11. To perform this identification (an example of the "second identification process" in the claims), the voice recognition unit uses, for example, a trained model (an example of the "second trained model" in the claims) stored in a voice recognition DB. This trained model is a trained model (for example, a neural network) that takes voice data as input and outputs identification information of the object that emitted the voice represented by the voice data. Here, the identification information of the object is information for distinguishing the object from other objects, for example, an identifier assigned to the object. This trained model can be constructed, for example, by machine learning using a combination of voice data and identification information of the object that emitted the voice represented by the voice data as training data.

For example, when audio data representing the voice of Prime Minister A is input to this trained model, identification information for Prime Minister A is output from this trained model. Also, when audio data representing the voice of President T is input to this trained model, identification information for President T is output from this trained model. Also, when audio data representing the sound of a dog barking is input to this trained model, identification information for the dog is output from this trained model. Also, when audio data representing the sound of an ambulance siren is input to this trained model, identification information for the ambulance is output from this trained model.

The image recognition unit identifies each object included as a subject in the image represented by the image data acquired by the acquisition unit 11, and specifies the position of each object on the screen. To perform this identification and specification (an example of the "first specification process" in the claims), the image recognition unit uses, for example, a trained model (an example of the "first trained model" in the claims) stored in an image recognition DB. This trained model is a trained model (e.g., a neural network) that receives image data as input and outputs identification information and position information of each object included as a subject in the image represented by the image data. Here, the position information of an object is information that represents the position of the object on the screen, for example, the coordinates of a representative point of the object. This trained model can be constructed, for example, by machine learning using a combination of image data and identification information and position information of objects included as subjects in the image represented by the image data as training data.

For example, when image data representing an image including Prime Minister A and President T as subjects is input to this trained model, the trained model outputs identification information and location information for Prime Minister A, as well as identification information and location information for President T. Also, for example, when image data representing an image including a dog and an ambulance as subjects is input to this trained model, the trained model outputs identification information and location information for the dog, as well as identification information and location information for the ambulance.

The text data generating unit 13 is configured to convert the voice represented by the voice data acquired by the acquiring unit 11 into text data. Methods for converting the voice represented by the voice data into text data are well known, so a description thereof will be omitted here. The text data generating unit 13 outputs the generated text data to the superimposing unit 15.

Based on the identification information and position information generated by the audio/image recognition unit 12, the subtitle display position determination unit 14 identifies an object that emits the sound represented by the audio data acquired by the acquisition unit 11, among the objects included as subjects in the image represented by the image data acquired by the acquisition unit 11. Here, the object identified by the subtitle display position determination unit 14 is an object whose identification information obtained by the image recognition unit matches the identification information obtained by the audio recognition unit. Then, based on the position information of the identified object, the subtitle display position determination unit 14 determines the position at which the text generated by the text data generation unit 13 is displayed as subtitles. For example, the subtitle display position determination unit 14 sets the position at which the text generated by the text data generation unit 13 is displayed as subtitles near the identified object. Alternatively, the subtitle display position determination unit 14 displays the position at which the text generated by the text data generation unit 13 is displayed as subtitles inside a speech bubble image displayed near the identified object.

The superimposition unit 15 is configured to superimpose the text data generated by the text data generation unit 13, which corresponds to the audio data acquired by the acquisition unit 11, on an image represented by the image data acquired by the acquisition unit 11. The superimposition unit 15 sets the position at which to superimpose the text data generated by the text data generation unit 13, to the position determined by the subtitle display position determination unit 14. The image with subtitles generated by the superimposition unit 15 is displayed on the display unit 40.

The display device 1 makes it possible to display subtitles obtained by converting audio into text at a location corresponding to the position of the object that emitted the audio. Therefore, even a viewer watching the display device 1, for example a viewer with hearing impairment, can properly recognize the association between the object and the subtitles.

The superimposing unit 15 may determine the position at which to superimpose the text data by referring to accuracy information indicating the accuracy of the identification process. Here, the identification process refers to a process including object identification in the voice recognition unit, object identification in the image recognition unit, and object identification in the subtitle display position determination unit 14.

For example, if the likelihood indicated by the likelihood information is higher than a predetermined level, the superimposition unit 15 superimposes the text data as subtitles near the object generating the sound. In other cases, the superimposition unit 15 superimposes the text data as subtitles in a predetermined area unrelated to the object generating the sound, such as the bottom edge, top edge, right edge, or left edge of the screen.

Alternatively, if the likelihood indicated by the likelihood information is higher than a predetermined level, the superimposition unit 15 superimposes a speech bubble image indicating that the sound is generated from the object near the object generating the sound, and superimposes the text data as subtitles inside the speech bubble image. In other cases, the superimposition unit 15 superimposes the text data as subtitles in a predetermined area unrelated to the object generating the sound, such as the bottom edge, top edge, right edge, or left edge of the screen.

With the above configuration, the overlay unit 15 can appropriately determine the position at which to overlay the text data.

Note that the "predetermined height" referred to here is not particularly limited and may be set appropriately as needed.

(Example of what appears on the display)
FIG. 3 is a diagram showing one display example displayed on the display unit 40 of the display device 1 shown in FIG. 1, and FIGS. 4 to 6 are diagrams showing other display examples.

As shown in FIG. 3, when the above-mentioned identification process identifies the object generating the voice as a human OB1, the superimposition unit 15 can superimpose text data TD1 corresponding to the voice (in this example, "Hello") on a speech bubble image near the OB1 generating the voice.

This allows a viewer watching the display unit 40, even a viewer with a hearing impairment, to properly recognize the association between the object (OB1 in this example) and the subtitle ("Hello" in this example).

On the other hand, as shown in FIG. 4, if the above-mentioned identification process identifies the object generating the sound as a human OB2, the superimposition unit 15 can superimpose text data TD2 corresponding to the sound (in this example, "Welcome") on the speech bubble image near the OB2 generating the sound. This also achieves the same effect as the example shown in FIG. 3.

If the identification process fails to identify the object generating the sound, the superimposition unit 15 may superimpose, as subtitles, text data corresponding to the audio data indicating the sound generated by the object in a predetermined area unrelated to the object generating the sound, such as the bottom edge of the screen, the top edge of the screen, the right edge of the screen, or the left edge of the screen.

(Modification of the first embodiment)
In the present embodiment, the object is described as a human being, but the object is not limited to a human being and may be any object that emits a sound. For example, the object may be a dog that emits a barking sound or an ambulance that emits a siren sound.

As shown in FIG. 5, when the above-mentioned identification process identifies the object generating the sound as a dog OB3, the superimposition unit 15 can superimpose text data TD3 corresponding to the sound (in this example, "woof woof") on the speech bubble image near the OB3 generating the sound. This also achieves the same effect as the example shown in FIG. 3.

On the other hand, as shown in FIG. 6, if the above-mentioned identification process identifies the object generating the sound as OB4, an ambulance, the superimposition unit 15 can superimpose text data TD4 corresponding to the sound (in this example, "Peep-peep-peep") on the speech bubble image near OB4 generating the sound. This also achieves the same effect as the example shown in FIG. 3.

In addition, in this embodiment, a configuration has been described in which only text data representing the sound emitted by an object is superimposed near the object, but the present invention is not limited to this. For example, the configuration may be as follows.

That is, the superimposition unit 15 may display specific information of an object near the object in addition to text data representing the sound emitted by the object. Here, the specific information of the object is information for identifying the object, such as the name of the object. Also, similar to the superimposition of text data, the superimposition of the specific information may be performed only when the likelihood indicated by the above-mentioned likelihood information is higher than a predetermined level.

In this case, although not specifically shown, for example, in Figures 3 to 6 above, specific information may be displayed at positions on the display unit 40 corresponding to the text data TD1 to TD4.

With the above configuration, a viewer watching the display unit 40, even a viewer with hearing impairment, can more appropriately recognize the association between objects and names.

The display method of this embodiment will be described below with reference to FIG. 7. FIG. 7 is a flowchart showing an image generation method executed by the image generation device 10 included in the display device 1 shown in FIG. 1.

As shown in FIG. 7, the image generation method includes an acquisition step S10, a specific processing execution step S11, and a superimposition step S12.

In the acquisition step S10, the image generating device 10 acquires image data representing an image input from outside and audio data representing audio accompanying the image.

Next, in the identification process execution step S11, the image generating device 10 inputs the image data and audio data acquired in the acquisition step S10 into a trained model to identify the object that is generating the audio among one or more objects contained in the image.

Finally, in the superimposition step S12, the image generating device 10 superimposes the text data corresponding to the audio data at a position within the image that corresponds to the result of the specific processing.

[Embodiment 2]
Other embodiments of the present invention will be described below. For ease of explanation, the same reference numerals are given to members having the same functions as those described in the above embodiment, and the description thereof will not be repeated.

Figure 8 is a block diagram showing the main configuration of the image generating device 10a according to this embodiment. Compared to the first embodiment, the only difference is that the image generating device 10a according to this embodiment does not include a text data generating unit. The following will focus on this difference.

In this embodiment, the image generating device 10a does not include a text data generating unit, and so does not generate text data. In this embodiment, the superimposing unit 15 generates an image on which information identifying the object is superimposed, the information having a display form corresponding to the accuracy information indicating the accuracy of the above-mentioned identification process.

(Example of what appears on the display)
FIG. 9 is a diagram showing one display example displayed on the display unit 40 shown in FIG. 8, and FIG. 10 is a diagram showing another display example.

In the example shown in FIG. 9, the image generating device 10a identifies two objects that are generating sounds through the identification process described above. The image generating device 10a then generates an image in which the names OB1 and OB2 of these two objects are superimposed in a display manner that indicates that the identification process is highly likely. Here, as a display manner that indicates that the identification process is highly likely, a display manner is adopted in which a rectangular speech bubble with a protrusion on the side closest to the object is displayed together with the object names OB1 and OB2.

This allows a viewer looking at the display unit 40, even a viewer with a hearing impairment, to properly recognize the objects, in this example, OB1 and OB2.

On the other hand, in the example shown in FIG. 10, the image generating device 10a identifies two objects that are generating sound through the identification process described above. The image generating device 10a then generates an image in which the names OB1 and OB2 of these two objects are superimposed in a display manner that indicates that the identification process is unlikely to be reliable. Here, as a display manner that indicates that the identification process is unlikely to be reliable, a display manner is adopted in which a rectangular speech bubble with no protrusions on the side closest to the object is displayed together with the object names OB1 and OB2.

This allows a viewer looking at the display unit 40, even a viewer with hearing impairment, to recognize that the likelihood of the objects, in this example OB1 and OB2, is low.

The display method of this embodiment will be described below with reference to FIG. 11. FIG. 11 is a flowchart showing the image generation method executed by the image generation device 10a shown in FIG. 8.

As shown in FIG. 11, the image generation method includes an acquisition step S20, a specific processing execution step S21, and a superimposition step S22.

As shown in FIG. 11, in this embodiment, the processing contents of steps S20 and S21 are similar to the processing contents of steps S10 and S11 in embodiment 1, respectively, and therefore the description thereof is omitted.

Then, in the superimposition step S22, the image generating device 10a generates an image on which information identifying the object is superimposed, the information having a display form corresponding to the accuracy information indicating the accuracy of the identification process.

〔summary〕
An image generating device (10) according to aspect 1 of the present invention includes an acquisition unit (11) that acquires image data representing an image and audio data representing audio accompanying the image, a specific processing execution unit (14) that executes a specific processing to identify an object that is generating audio among one or more objects contained in the image by inputting the image data and audio data acquired by the acquisition unit into a learned model, and a superimposition unit (15) that superimposes text data corresponding to the audio data at a position within the image that corresponds to the result of the identification processing.

The above configuration enables viewers watching the display device (1), even those with hearing impairments, to properly recognize the association between objects and text data (subtitles).

In the image generating device (10) according to aspect 2 of the present invention, in the above aspect 1, the superimposition unit may determine the position at which to superimpose the text data by referring to accuracy information indicating the accuracy of the specific process.

With the above configuration, the superimposition unit (15) can appropriately determine the position at which to superimpose the text data.

In the image generating device (10) according to aspect 3 of the present invention, in the above aspect 2, the identification process execution unit may execute a first identification process to acquire positions and identification information of one or more objects included in the image by inputting the image data into a first trained model, execute a second identification process to acquire identification information of the audio data by inputting the audio data into a second trained model, and identify the object generating the audio by referring to the results of the first identification process and the results of the second identification process.

With the above configuration, the identification process execution unit (14) can reliably identify the object that is generating the sound.

In the image generating device (10) according to aspect 4 of the present invention, in the above aspect 2 or 3, the superimposition unit may superimpose the text data near the object generating the sound when the likelihood indicated by the likelihood information is higher than a predetermined level.

In the image generating device (10) according to aspect 5 of the present invention, in the above aspect 4, the superimposition unit may superimpose a speech bubble image indicating that the sound is emitted from the object near the object generating the sound when the likelihood indicated by the likelihood information is higher than a predetermined level, and may superimpose the text data on the speech bubble image.

With the above configuration, the superimposition unit (15) can superimpose text data at a more appropriate position.

The image generating device (10) according to aspect 6 of the present invention is any one of aspects 2 to 5 above, and the superimposition unit may generate an image in which information identifying the object is superimposed near the object when the likelihood indicated by the likelihood information is higher than a predetermined height.

With the above configuration, viewers watching the display unit (40), even those with hearing impairments, can more appropriately recognize the association between objects and text data (subtitles).

The image generating device (10a) according to aspect 7 of the present invention includes an acquisition unit (11) that acquires image data representing an image and audio data representing audio accompanying the image, a specific processing execution unit (14) that executes a specific processing to identify an object that is generating audio among one or more objects included in the image by inputting the image data and audio data acquired by the acquisition unit into a trained model, and a superimposition unit (15) that generates an image on which information identifying the object is superimposed, the information having a display form corresponding to accuracy information indicating the accuracy of the identification processing.

The above configuration provides the same effects as in aspect 1 above.

The image generating method according to aspect 8 of the present invention includes an acquisition step of acquiring image data representing an image and audio data representing audio accompanying the image, a specific processing execution step of executing a specific processing step of identifying an object that is generating audio among one or more objects included in the image by inputting the image data and audio data acquired in the acquisition step into a trained model, and a superimposition step of superimposing text data corresponding to the audio data at a position within the image that corresponds to the result of the specific processing.

The above configuration provides the same effects as in aspect 1 above.

The image generating method according to aspect 9 of the present invention includes an acquisition step of acquiring image data representing an image and audio data representing audio accompanying the image, an identification process execution step of executing an identification process for identifying an object generating audio among one or more objects included in the image by inputting the image data and audio data acquired in the acquisition step into a trained model, and a superimposition step of generating an image on which information identifying the object is superimposed, the information having a display form corresponding to accuracy information indicating the accuracy of the identification process.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. The technical scope of the present invention also includes embodiments obtained by appropriately combining the technical means disclosed in the different embodiments. Furthermore, new technical features can be formed by combining the technical means disclosed in the respective embodiments.

REFERENCE SIGNS LIST 1 Display device 10, 10a Image generating device 11 Acquisition unit 12 Voice/image recognition unit 13 Text data generating unit 14 Specific process execution unit 15 Superimposition unit 20 Voice recognition DB
30 Image recognition database
40 Display section OB1-4 Object 1-4
TD1-4 Text data 1-4

Claims (8)

an acquisition unit that acquires image data representing an image and audio data representing audio accompanying the image;
A specific processing execution unit that executes a specific processing to identify an object generating a sound among one or more objects included in the image by inputting the image data and the voice data acquired by the acquisition unit into a trained model;
a superimposition unit that superimposes text data corresponding to the voice data at a position within the image according to a result of the identification process ,
The overlay unit determines a position at which to overlay the text data by referring to probability information indicating a probability in the specified process . The specific process execution unit is
A first identification process is executed to obtain positions and identification information of one or more objects included in the image by inputting the image data into a first trained model;
A second identification process is executed to obtain identification information of the voice data by inputting the voice data into a second trained model;
The image generating device according to claim 1 , further comprising: a processor for processing the image generated by the object generating the sound; The image generating device according to claim 1 , wherein the superimposing unit superimposes the text data near an object generating the sound when the likelihood indicated by the likelihood information is higher than a predetermined level. The overlapping portion is
If the likelihood indicated by the likelihood information is higher than a predetermined level, a balloon image indicating that the sound is emitted from the object is superimposed near the object emitting the sound;
The image generating device according to claim 3 , wherein the text data is superimposed on the speech balloon image. The image generating device according to any one of claims 1 to 4 , wherein the superimposition unit generates an image in which information identifying the object is superimposed near the object when the likelihood indicated by the likelihood information is higher than a predetermined height. an acquisition unit that acquires image data representing an image and audio data representing audio accompanying the image;
A specific processing execution unit that executes a specific processing to identify an object generating a sound among one or more objects included in the image by inputting the image data and the voice data acquired by the acquisition unit into a trained model;
a superimposing unit that generates an image on which information that identifies the object is superimposed, the information having a display form corresponding to certainty information that indicates the certainty of the identification process; and
An image generating device comprising: an acquiring step of acquiring image data representing an image and audio data representing audio accompanying the image;
A process for executing a process for identifying an object that is generating a sound among one or more objects included in the image by inputting the image data and the sound data acquired in the acquisition process into a trained model;
a superimposing step of superimposing text data corresponding to the voice data at a position within the image according to a result of the identification process ,
In the superimposing step, a position at which to superimpose the text data is determined by referring to accuracy information indicating accuracy in the specific process . an acquiring step of acquiring image data representing an image and audio data representing audio accompanying the image;
A process for executing a process for identifying an object that is generating a sound among one or more objects included in the image by inputting the image data and the sound data acquired in the acquisition process into a trained model;
a superimposing step of generating an image on which information for identifying the object is superimposed, the information having a display form corresponding to certainty information indicating the certainty of the identification process;
An image generating method comprising:

Images