JP7614877B2 - Content production device and program - Google Patents

Description

The present invention relates to a content production device and a program.

Attempts are being made to create content that does not contain human speech by using pieced together synthetic speech.

Patent Document 1 describes the configuration of a voice synthesis device. Specifically, it describes that the voice synthesis device includes "multiple synthetic sound creation means 300 that estimate the length of silence between synthetic sounds or generate silence using a predetermined method, and create a voice file that stores information on the length of silence between the synthetic sounds." With the above configuration, the voice synthesis device described in Patent Document 1 generates synthetic sound (broadcast program, etc.) that fits within a predetermined time length.

Patent document 1 also describes, in paragraph 0018, that the speech speed of the audio is converted to fit the program to the desired length.

JP 2016-009061 A

If the text is created in a fixed format, the speaking speed may have to be changed "significantly," so the text must be adjusted to maintain quality.

The present invention was developed based on the recognition of the above problems, and aims to provide a content production device and program that can produce content of a predetermined length that is natural and has high audio quality without relying on speech rate conversion.

[1] In order to solve the above problem, a content production device according to one aspect of the present invention includes a template storage unit that stores a content template having a plurality of variations of sentence templates for generating sentences, a data acquisition unit that acquires data, a variation generation unit that generates sentences by applying the data to the sentence templates for each of the plurality of variations included in the content template and determines the total length of the generated sentences, a search processing unit that searches for combinations of variations using an evaluation function that is an appropriately determined total of evaluation values for the variations under constraints regarding conditions regarding the length of pauses that are joints when connecting the sentences and conditions regarding the length of the sentences and the total length of the pauses, a selection unit that selects a combination of variations that satisfies the constraints based on the value of the evaluation function, and a pause adjustment unit that adjusts the length of the pauses so as to satisfy the conditions regarding the length of the pauses and the conditions regarding the length of the sentences and the total length of the pauses.

[2] In one aspect of the present invention, in the content production device described above, the content template is configured as a sequence of topics, and the topics are configured to include a plurality of the variations that can be selected mutually exclusively.

[3] In one aspect of the present invention, in the content production device described above, the conditions regarding the duration of the pauses include a condition regarding the duration of an inter-sentence pause, which is a pause at a division of a sentence included in the variation, and a condition regarding the duration of an inter-topic pause, which is a pause at a division of a topic.

[4] In one aspect of the present invention, in the content production device described above, the pause adjustment unit adjusts the duration of the inter-sentence pauses so that they are all the same, and adjusts the duration of the inter-topic pauses so that they are all the same.

[5] In one aspect of the present invention, in the content production device described above, the evaluation value for each variation is predefined as an attribute value of the variation included in the content template.

[6] Also, one aspect of the present invention is a program for causing a computer to function as a content production device, the program comprising: a template storage unit that stores a content template having a plurality of variations of sentence templates for generating sentences; a data acquisition unit that acquires data; a variation generation unit that generates sentences by applying the data to the sentence templates for each of the plurality of variations included in the content template and determines the total length of the generated sentences; a search processing unit that searches for combinations of variations using an appropriately determined total of evaluation values for each of the variations as an evaluation function under constraints regarding a condition regarding the length of a pause that is a joint when connecting the sentences and a condition regarding the length of the sentences and the total length of the pauses; a selection unit that selects a combination of variations that satisfies the constraints based on the value of the evaluation function; and a pause adjustment unit that adjusts the length of the pause so as to satisfy the condition regarding the length of the pause and the condition regarding the length of the sentences and the total length of the pauses.

According to the present invention, it is possible to create content of a desired length without relying on speech speed conversion technology (expansion and contraction of speech time).

1 is a block diagram showing a schematic functional configuration of a content production device according to an embodiment of the present invention. 2 is a schematic diagram showing an example of the configuration of weather forecast data used in the embodiment. FIG. 10 is a schematic diagram showing an example of the configuration of "weather forecast (today)" data used in the embodiment. FIG. 4 is a schematic diagram showing an example of the configuration of data of a content template stored in a template storage unit according to the embodiment. FIG. 10 is a schematic diagram showing an example of the configuration of topic template data according to the embodiment. FIG. 10 is a schematic diagram showing an example of the relationship between a sentence template included in a topic template in the embodiment and a generated sentence generated based on the sentence template. FIG. 13 is a schematic diagram showing another example of the relationship between a sentence template included in a topic template and a generated sentence generated based on the sentence template in the embodiment. FIG. 2 is a schematic diagram showing the interrelationships between topics and their variations generated by a variation generating unit 30 based on a content template in the embodiment. FIG. 11 is a schematic diagram showing conditions under which a search processing unit according to the embodiment performs search processing. FIG. 13 is a flowchart showing an overall processing procedure of the content production device according to the embodiment. 11 is a schematic diagram showing an example of a user interface screen displayed by the content production device according to the embodiment. FIG.

Next, an embodiment of the present invention will be described with reference to the drawings. The content production device 1 according to this embodiment automatically generates content using acquired data. The content generated by the content production device 1 can be transmitted over a wide area via a broadcast signal, or can be sent to a terminal device via the Internet or the like. The content production device 1 holds content template data in advance in order to create content. The content production device 1 creates content by applying the above data to the content template. The content is, for example, audio content. The content production device 1 automatically generates audio content by using a voice synthesis technique. The content production device 1 also performs a process of searching for many variations (candidates) of content in order to automatically generate content of a desired length. In this embodiment, a content production device 1 that automatically generates a weather forecast program (a program with only audio) will be described as an example.

FIG. 1 is a block diagram showing the schematic functional configuration of a content production device according to this embodiment. In the figure, 1 is a content production device. The content production device 1 is configured to include a data receiving unit 10, a template storage unit 20, a variation generating unit 30, a search processing unit 40, a selection unit 50, a pose adjustment unit 60, and an output unit 70. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Also, non-volatile storage means such as a magnetic hard disk drive or a solid state drive (SSD) may be used as necessary. Also, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program.

The data receiving unit 10 receives weather forecast data from an external device via a communication network (e.g., the Internet, etc.). The external device is, for example, a data server device operated by a weather forecasting agency. The weather forecast data received by the data receiving unit 10 includes information related to the weather forecast. An example of the configuration of the weather forecast data will be described later with reference to another drawing. The data receiving unit 10 passes the received weather forecast data to the variation generating unit 30. The data receiving unit 10 may be referred to as a "data acquiring unit." Instead of receiving data through communication, the data receiving unit 10 may acquire data by a method that does not rely on communication, such as reading it from a recording medium.

The template storage unit 20 stores data of templates for creating content. A template for creating content is called a content template. A content template has sentence templates for generating sentences consisting of one sentence or multiple sentences for multiple variations. As one form, the content template may be configured as a series of topics. A topic is configured to include one or multiple variations. In one topic, the variations can be selected mutually exclusive. In addition, an evaluation value for each variation may be provided in a predetermined form as an attribute value of the above-mentioned variation in the content template. This evaluation value is used as information for evaluation when searching for and selecting a combination of variations. The above-mentioned sentence templates stored in the template storage unit 20 may be text (a series of character codes) of a sentence in a natural language, sentences expressed in a synthesis instruction format, or both. The synthesis instruction format is a data format for synthesis instructions to a speech synthesizer, and includes data on the sequence of sounds and data on prosodic instructions. The specific form of the synthesis instruction format may differ depending on the speech synthesizer. The specific configuration of the template will be explained later with reference to another diagram.

The variation generation unit 30 generates sentences by applying the received data to the sentence template for each of the multiple variations included in the template. The variation generation unit 30 also determines the total duration of the generated sentences. An example of a method by which the variation generation unit 30 determines this total duration is as follows. That is, the variation generation unit 30 uses the function of a voice synthesizer to generate synthetic voice corresponding to each sentence in the generated sentence. The variation generation unit 30 also determines the total duration of the variations, which is the sum of the durations of the synthetic voices of each of the generated sentences. For example, the variation generation unit 30 generates synthetic voice, which inevitably determines the duration of the synthetic voice.

The search processing unit 40 performs the search process under constraints regarding conditions related to the duration of pauses, which are the joints when connecting synthetic voices, and conditions related to the total duration of the synthetic voice and the total duration of the pauses. The evaluation function used by the search processing unit 40 when performing the search process may be the total of the evaluation values of the variations to be selected. In other words, the search processing unit 40 searches for a combination of variations that will increase the value of the evaluation function. Note that the conditions related to the duration of pauses may include conditions related to the duration of inter-sentence pauses, which are pauses at the division of sentences included in the variation, and conditions related to the duration of inter-topic pauses, which are pauses at the division of topics. Specific examples of these conditions will be described with reference to FIG. 9. Details of the processing by the search processing unit 40 will also be described later.

The selection unit 50 selects one combination of variations that satisfies the time constraints based on the value of the evaluation function. For example, the selection unit 50 selects the combination with the best evaluation function value from among the combinations of variations that satisfy the above constraints. Alternatively, the selection unit 50 selects one combination that is determined to have a good evaluation function value (which does not necessarily have to be the best evaluation function value) depending on the search algorithm executed by the above search processing unit 40.

The pause adjustment unit 60 adjusts the duration of a pause so as to satisfy conditions related to the duration of the pause, and also to satisfy conditions related to the sum of the duration of the synthetic voice and the duration of the pause (i.e., the total duration of the content to be generated). A pause is a silent section in audio content. In this embodiment, the pause adjustment unit 60 adjusts both inter-sentence pauses inserted at sentence breaks and inter-topic pauses inserted at topic breaks. By the pause adjustment unit 60 adjusting the duration of the pause, the overall length of the content generated by the content production device 1 can be adjusted to satisfy given specified conditions.

As an example of its processing, the pause adjustment unit 60 may adjust the duration of inter-sentence pauses so that they are all the same. The pause adjustment unit 60 may also adjust the duration of inter-topic pauses so that they are all the same. Conversely, the duration of inter-sentence pauses does not have to be constant. The duration of inter-topic pauses does not have to be constant.

The output unit 70 outputs content (one piece of audio content in which multiple topics are linked) that is a combination of the variations selected by the selection unit 50. In the content output by the output unit 70, the length of the pauses (inter-sentence pauses, inter-topic pauses) has already been adjusted by the pause adjustment unit 60. Silence or a similar sound has already been inserted into the pause section. The content output by the output unit 70 is passed to equipment for broadcasting or equipment for internet distribution.

2 is a schematic diagram showing an example of the configuration of weather forecast data. As described above, the data receiving unit 10 receives this weather forecast data. As shown in the figure, the weather forecast data 100 is configured to include three data items: date, prefecture, and content. The date represents the date covered by this weather forecast data. In the example shown in the figure, the date is "January 25, 2020." The prefecture represents the name of the prefecture (region) covered by this weather forecast data.
In the illustrated example, the prefecture is "Kanagawa Prefecture." The content is a portion having the substance of the weather forecast data. The content has a sequence of a plurality of data chunks. In the illustrated example, the content has a sequence of data including "alert/advisory" data 101, "weather forecast (today)" data 102, "expected temperature (today)" data 103, and "probability of precipitation (today)" data 104. The "probability of precipitation (today)" data 104 may be followed by another data chunk. Here, the data is configured so that each of the "alert/advisory" data 101, "weather forecast (today)" data 102, "expected temperature (today)" data 103, and "probability of precipitation (today)" data 104 corresponds to a topic to be described later. A more specific configuration example of the "weather forecast (today)" data 102 among these will be described next.

3 is a schematic diagram showing an example of the configuration of the "weather forecast (today)" data 102. As shown in the figure, the "weather forecast (today)" data 102 is expressed as tabular data, and has the following items: region, wind direction, weather, time transition, and local weather. The region is the name of a region when a prefecture is divided into smaller units. The wind direction is the wind direction forecast.
Weather is a weather classification expressed by words such as sunny, cloudy, rain, snow, etc. Time transition is an expression that expresses the time transition of weather. For example, expressions of time transition often used in weather forecasts are "later cloudy", "occasional rain", "temporary snow", etc. Local weather is an expression that expresses the weather aspect that may differ depending on a more local location within the above-mentioned region. In the illustrated example, for the region "East", the wind direction is "West", the weather is "sunny", the time transition is "later cloudy", and there is no local weather. Also, for the region "West", the wind direction is "South", the weather is "sunny", the time transition is "later cloudy", and the local weather is "rain in places".

In the example shown in FIG. 3, the "Weather forecast (today)" data 102 is expressed as tabular data. However, any format may be used as the data expression format. As an example, the data shown in FIG. 3 may be expressed in an expression format using XML (extensible Markup Language). Data may also be expressed in other formats.

Figure 3 shows an example of the configuration of "Weather forecast (today)" data 102, but the "Warning/advisory" data 101, "Expected temperature (today)" data 103, "Chance of precipitation (today)" data 104, and other data can also be configured in an appropriate form as needed. In this way, the data receiving unit 10 acquires the information required to generate content from outside.

Figure 4 is a schematic diagram showing an example of the data configuration of a content template stored in the template storage unit 20. As shown in the figure, the content template is configured to include multiple topic templates. Furthermore, these multiple topic templates have an order. In other words, one content template is configured as a string of topic templates having an order. In the example shown in the figure, the content template includes topic templates such as "Greeting 1", "Topic: Warning/Caution", "Topic: Today's Weather", "Topic: Today's Forecasted Temperature", "Topic: Probability of Precipitation", and "Greeting 2". A more detailed configuration of the topic templates will be explained next.

5 is a schematic diagram showing an example of the configuration of data of one topic template. As shown in the figure, the data of the topic template is configured to include a required flag and variation data. The required flag is information indicating whether the topic is required in the content. A required topic must be included when generating the content. A non-required topic is allowed to be omitted, for example, for the purpose of adjusting the overall time length of the content. The creator of the content template may appropriately decide whether to set the required flag to true or false. A variation is data representing multiple forms of a sentence that the content production device 1 can generate for that topic. The variation data may be configured as data in a table format, for example, as shown in the figure. In the example shown in the figure, this table holds information on five types of variations, from number 1 to number 5. The number of variations is not limited to five and is arbitrary. This table has the following items: number, sentence template, evaluation value, generated sentence, synthetic voice of each sentence, and total duration. A sentence template is a template that serves as a basis for the variation generation unit 30 to generate sentences. Each sentence template has a portion (parameter) that can be replaced with weather forecast data. The evaluation value is a numerical value used as an evaluation when selecting the variation. For example, the evaluation value may be a fixed value that is predetermined for each variation. The generated sentence, synthetic voice, and duration items are blank at the template stage, and are filled in when the variation generation unit 30 generates an actual sentence. The generated sentence is a field that stores a sentence generated by the variation generation unit 30 based on the sentence template. The synthetic voice is a field that stores the synthetic voice of each sentence synthesized by the variation generation unit 30 based on the generated sentence. The synthetic voice data is stored, for example, as data of a series of sound pressure levels (or data obtained by encoding the data). The total duration is the sum of the lengths of time when the synthetic voice of each sentence is played back, and is expressed, for example, in seconds.

Note that within a content template, whether a topic is required or must be omitted may be specified so as to depend on whether other topics are adopted. For example, if topic A precedes topic B, it may be specified that "topic B is required if topic A exists." Or, for example, if topic A precedes topic B, it may be specified that "topic B cannot be adopted if topic A exists." In other words, it may be possible to introduce a relationship in which multiple topics are adopted as a set or multiple topics are adopted exclusively. Note that the manner of dependency between multiple topics within a content template is not limited to the examples given here, and any relationship may be introduced.

The above describes the data of the content template and the topic template for constructing the content template. In other words, audio content is generated by arranging multiple topics in a predetermined order according to the content template. However, non-essential topics may be omitted.
Furthermore, one topic can have multiple variations. Since the topic template has a sentence template for each variation, the variation generation unit 30 can generate sentences for each variation using the topic template. Furthermore, the variation generation unit 30 actually synthesizes voice based on the generated sentences, thereby determining the total duration of the synthesized voice. Furthermore, an evaluation value is given for each variation in the topic template. In other words, after the variation generation unit 30 generates multiple variations of each topic, it becomes possible to search for the optimal form for the entire content based on the duration and evaluation value. This search process will be described later.

An example was described above in which a fixed value was assigned in advance as the evaluation value for each variation. However, the evaluation value for each variation does not necessarily have to be fixed in advance. For example, the evaluation value may be variable depending on the content of the generated sentence generated based on the sentence template.

6 is a schematic diagram showing an example of the relationship between the sentence template included in the topic template described above and the generated sentence generated based on the sentence template. In the illustrated example, the sentence template is data representing that the sentence "This is the weather today in (prefecture)." may be followed by a sentence "In (region), the wind is (wind direction), (weather) (time transition) (local weather)." multiple times. In this example, this repetition is for each region within the prefecture. Each of the (prefecture), (region), (wind direction), (weather), (time transition), and (local weather) included in the sentence template is a parameter. The parameters can be replaced with actual data when converted into the generated sentence. The actual data that replaces the parameters is included in the weather forecast data acquired by the data receiving unit 10. In this example, the parameter (prefecture) is replaced with the actual prefecture name "Kanagawa Prefecture" (see FIG. 2). As a result, based on the template "This is the weather today in (prefecture).", the variation generating unit 30 generates the sentence "This is the weather today in Kanagawa Prefecture." In addition, the parameters (area), (wind direction), (weather), (time transition), and (local weather) are replaced with information in the "weather forecast (today)" data shown in FIG. 3. For example, for the eastern part of Kanagawa Prefecture, based on the template "In (area), wind will be (wind direction), (weather) (time transition) (local weather)," the variation generation unit 30 generates the sentence "In the eastern part, wind will be from the west, sunny, then cloudy." Also, for the western part of Kanagawa Prefecture, based on the template "In (area), wind will be (wind direction), (weather) (time transition) (local weather)," the variation generation unit 30 generates the sentence "In the western part, wind will be from the south, sunny, then cloudy, with rain in some places."

Figure 7 is a schematic diagram showing another example of the relationship between the sentence template included in the topic template described above and the generated sentence generated based on the sentence template. Note that Figures 6 and 7 are different variations included in the same topic, "Today's Weather". In the example of Figure 7, the sentence template is data expressing the expression "Today's weather in (prefecture) will be (weather) (time transition) throughout the prefecture." This sentence template in Figure 7 omits information on (wind direction) and (local weather) compared to the example sentence template shown in Figure 6. In other words, the information is omitted from the way of conveying information in Figure 6. In addition, this sentence template generates a sentence only when (weather) (time transition) is the same in all (areas) within (prefecture). In other cases, this variation is treated as not existing. Each of (prefecture), (weather), and (time transition) included in this sentence template is a parameter. As in the case of Figure 6, the parameters present in the sentence template are replaced with information from actual data (weather forecast data). The variation generation unit 30 generates the sentence "Today's weather in Kanagawa Prefecture will be sunny throughout the prefecture, then cloudy" by replacing parameters using actual data based on the sentence template in FIG. 7.

As described above, the two types of sentence templates shown in FIG. 6 and FIG. 7 belong to two variations of the same topic with different information details. The variation generation unit 30 generates a plurality of generated sentences as variations for one topic based on the same received data (weather forecast data in this case) by using a plurality of sentence templates. The variation generation unit 30 also generates voices by voice synthesis processing corresponding to each of the plurality of generated sentences. The total duration of the voices differs depending on the variation. In the examples shown in FIG. 6 and FIG. 7, the generated sentences created based on the sentence template in FIG. 6 are longer than the generated sentences created based on the sentence template in FIG. 7. In addition, for the synthetic voices generated corresponding to the respective generated sentences, the total duration of the synthetic voices corresponding to the sentence template in FIG. 6 is longer than the total duration of the synthetic voices corresponding to the sentence template in FIG. 7. In this way, the variation generation unit 30 generates synthetic voices having different total durations for one topic by using different sentence templates.

Figure 8 is a schematic diagram showing the interrelationships between the topics and their variations generated by the variation generation unit 30 based on the content template. In other words, Figure 8 shows a search space having multiple variations for each topic. As shown in the figure, the search space has a series of serial topics. Furthermore, each topic included in the series can have one or more parallel variations. In the example shown in Figure 8, "Greeting 1" has one variation. Furthermore, "Topic 1" has three parallel variations. Furthermore, "Topic 2" has three parallel variations, one of which is marked "None". This "None" indicates that the variation does not have a generated sentence and a synthetic voice. Furthermore, "Greeting 2" has one variation. Each of the normal variations other than the variation marked "None" has a generated sentence, a synthetic voice for each sentence, a total time length, and an evaluation value. The generated sentence is a sentence generated by the variation generation unit 30 based on a sentence template. The synthetic voice is a voice synthesized by the variation generation unit 30 based on each sentence of the generated sentence. The total duration is the sum of the durations from the beginning to the end of the synthetic speech of each sentence generated by the variation generation unit 30. The evaluation value is the evaluation value of that variation. The evaluation value is a numerical value that indicates the desirability of that variation being selected. As a general rule, the evaluation value is given to a variation that is more detailed and therefore has a greater total duration.

The variation generation unit 30 constructs a search space as shown in FIG. 8. In other words, the variation generation unit 30 generates generated sentences for each variation in the search space, generates synthetic voice corresponding to the generated sentences, and determines the total time length of the synthetic voice. The evaluation value of each variation is either predetermined or determined at the time when the generated sentences or synthetic voice are generated. In other words, the evaluation value of each variation is determined by the time when the variation generation unit 30 constructs this search space. In this way, in a state in which the specific values of the time length and evaluation value for each variation in the search space shown in FIG. 8 are determined, the search processing unit 40 can select a variation that maximizes the total evaluation value. Specifically, the search processing unit 40 can select one variation from each topic. Then, the search processing unit 40 finds a solution that maximizes the total evaluation value while satisfying a predetermined condition. The solution here is a combination of variations selected for each topic in the content. The variation generation unit 30 writes data representing the search space shown in FIG. 8 into a storage medium such as a memory, so that the search processing unit 40 can refer to it.

Figure 9 is a schematic diagram showing constraint conditions when the search processing unit 40 performs the search process. As shown in the figure, the conditions when performing the search process include the total content length, the inter-sentence pause lower limit, the inter-sentence pause upper limit, the inter-topic pause lower limit, and the inter-topic pause upper limit. These conditions are, for example, set in advance and written to a memory or the like that can be referenced by the search processing unit 40. The total content length is the total time length of the content generated by the content production device 1. The time length when the audio of all topics in the content to be produced is concatenated must not exceed the total content length. The inter-sentence pause lower limit is the lower limit of the time length of a pause between sentences. The inter-sentence pause upper limit is the upper limit of the time length of a pause between sentences. The inter-topic pause lower limit is the lower limit of a pause between topics. The inter-topic pause upper limit is the upper limit of a pause between topics. The unit of these setting values is all seconds.

Between sentences is the connection between each sentence in the generated topic. In other words, between sentences is the location where a "<sentence separator>" exists in the sentence template and generated sentence shown in Figures 6 and 7. Between topics is the location between topics in the content template. The location indicated by "<topic separator>" in Figure 8 is between topics.

In the illustrated example, the total content length is 240.000 seconds. The lower limit of the inter-sentence pause is 0.700 seconds. The upper limit of the inter-sentence pause is 1.500 seconds. The lower limit of the inter-topic pause is 1.000 seconds. The upper limit of the inter-topic pause is 3.000 seconds.

The search processing unit 40 searches for a combination of variations that maximizes the sum of the evaluation values of the selected variations while satisfying the set conditions. The inter-sentence pauses are set so that there is a difference between their lower and upper limits. The inter-topic pauses are also set so that there is a difference between their lower and upper limits. Therefore, the search processing unit 40 selects a variation on the premise that the duration of these pauses can be adjusted later. In other words, the search processing unit 40 concatenates all of the synthetic voices of the selected variations, and searches for a combination of variations that maximizes the sum of the evaluation values from among combinations in which the length of the result of setting the inter-sentence pauses and inter-topic pauses to their respective lower limits is equal to or less than the above content length, and the length of the result of setting the inter-sentence pauses and inter-topic pauses to their respective upper limits is equal to or more than the above content length. The fact that the inter-sentence pauses and inter-topic pauses are adjustable means that the duration of each of these pauses can be extended or shortened within the set conditions.

The above constraints on content length can be expressed mathematically as follows. That is, the content length is a seconds (fixed length), the number of topics included in the content is b, the total number of sentences is c, the total duration of the audio portion is d seconds, the lower limit of inter-sentence pauses is e seconds, the upper limit of inter-sentence pauses is f seconds, the lower limit of inter-topic pauses is g seconds, and the upper limit of inter-topic pauses is h seconds. The constraints in this case are expressed by the following formula (1).

d+(c-b)×e+(b-1)×g≦a≦d+(c-b)×f+(b-1)×h
...(1)

If we perform a search to satisfy the constraints in this formula, we can make the total content length exactly a seconds by appropriately adjusting the length of the inter-sentence pauses and the inter-topic pauses.

Instead of setting the content length to a fixed value as described above, an upper and lower limit may be set for the content length, and the search process may be performed under those constraints.

For optional topics, the search processing unit 40 may find a solution that does not select a variation that belongs to that topic. Optional topics are topics for which the required flag is set to "false" in the topic template shown in FIG. 5. Not selecting a variation for an optional topic is equivalent to selecting a variation for which no generated sentence or synthetic voice exists (for example, a variation marked "(none)" in FIG. 8).

The search processing unit 40 may use any search method (algorithm) to perform the search process. As an example, the search processing unit 40 may use an A ^* search algorithm with a predetermined length restriction. The A ^* search itself is an existing method. In this case, the search processing unit 40 performs a depth-wise search starting from the first topic of the content and prioritizing variations with high evaluation values. At the location of a certain topic during the search process, the search processing unit 40 holds the expected evaluation value of the remaining topics (the sum of the evaluation values when the variation with the maximum evaluation value is selected for each topic), the total maximum time length of the remaining topics (the sum of the time lengths when the variation with the maximum total time length is selected for each topic and the upper limit value is selected for each pause length), and the minimum time length (the sum of the time lengths when the variation with the minimum total time length is selected for each topic and the lower limit value is selected for each pause length). The search processing unit 40 then adds the maximum and minimum total durations of the remaining topics to the maximum (total duration when the upper limit value is selected for each pause length) and minimum (total duration when the lower limit value is selected for each pause length) values of the sum of durations up to that topic location, respectively, to estimate the maximum and minimum total durations of the content to be generated.The search processing unit 40 then extends the search process for the hypothesis with the maximum (sum of the sum of the evaluation values up to that topic location and the expected evaluation value of the remaining topics) among the hypotheses whose maximum and minimum total durations fall within a predetermined range.

However, the processing by the search processing unit 40 does not necessarily have to be based on the algorithm exemplified above. Regardless of which search method is used, the search processing unit 40 searches for a solution that maximizes the sum of the evaluation values within the search space while eliminating combinations of variations in which the total time length of the content does not satisfy the constraints.

The duration T of the entire content when the search processing unit 40 performs a search is calculated as follows. Once the combination of variations is determined, the length of the synthetic voice of those variations, and the number of inter-sentence pauses and inter-topic pauses when generating content using those variations are determined. The duration of one inter-sentence pause and the duration of one inter-topic pause are variable within the constraints shown in FIG. 9. The durations of all inter-sentence pauses are set to be equal to each other. The durations of all inter-topic pauses are also set to be equal to each other. At this time, the following values of A, B, and C (all durations) are as follows:
A: Total duration of synthetic speech of selected variations B: Duration of inter-sentence pauses × number of inter-sentence pauses (= total number of sentences - number of topics) C: Duration of inter-topic pauses × number of inter-topic pauses (= number of topics - 1) And the duration T of the entire content is expressed as T = (A + B + C), where B and C are variable and can each take a value between the maximum and minimum values. The search processing unit 40 searches for combinations of variations within the constraint that T satisfies the above-mentioned condition.

Figure 10 is a flowchart showing the overall processing procedure of the content production device 1. The processing procedure will be explained below with reference to this flowchart.

First, in step S1, the data receiving unit 10 receives data for content creation. The data for content creation is, for example, the above-mentioned weather forecast data 100 (see FIG. 2). The data receiving unit 10 passes the received data to the variation generating unit 30.

Next, in step S2, the variation generation unit 30 reads content template data from the template storage unit 20. An example of content template data is as shown in FIG. 4.

Next, in step S3, the variation generation unit 30 applies the data received in step S1 to the template loaded in step S2 to generate variations of the content. Here, the variation generation unit 30 generates all variations that may be used. Each variation includes information on the generated sentence, the synthesized voice of each sentence, and the total duration of the voice. In addition, an evaluation value is assigned to each variation.

Next, in step S4, the search processing unit 40 searches for the variations generated in step S3. The search processing has already been described, and the search processing unit 40 searches for a combination of variations that satisfies the conditions related to the length (time length) of the content and that results in a high evaluation value. The search processing unit 40 passes the results of the search processing to the selection unit 50. The results of the search processing include information on the combination of variations and information on the evaluation value when that combination is selected.

Next, in step S5, the selection unit 50 selects a combination of variations. Specifically, the selection unit 50 selects a combination of variations that will result in a high total evaluation value. As an example, the selection unit 50 selects a combination of variations that will result in the highest total evaluation value among combinations that satisfy the constraints on the length of the content. The selection unit 50 passes information about the selected combination of variations to the pose adjustment unit 60.

Next, in step S6, the pause adjustment unit 60 adjusts the length of the pauses. Specifically, the pause adjustment unit 60 adjusts the time length of one inter-sentence pause and one inter-topic pause. The pause adjustment unit 60 determines P _S and P _T that satisfy the following equation (2) based on the total length L of the content to be produced.

L _U +N _S・P _S +N _T・P _T =L... (2)

In formula (2), N _S and N _T are the number of inter-sentence pauses and the number of inter-topic pauses, respectively, when the selected combination of variations is adopted. Once the combination of variations is determined, the values of N _S and N _T are determined. P _S and P _T are the duration of one inter-sentence pause and one inter-topic pause, respectively. P _S and P _T are values to be determined by the pause adjustment unit 60. L _U is the total duration of the synthetic speech. Note that the pause adjustment unit 60 may strike an appropriate balance between the values of P _S and P _T as necessary.

Once the values of _PS and _PT are determined, the pause adjustment unit 60 inserts pauses (silent intervals) of lengths determined by _PS and _PT between sentences and topics, respectively, while linking all the generated synthetic voices together to create one piece of voice content as a whole. The pause adjustment unit 60 passes the created voice content to the output unit 70.

Next, in step S7, the output unit 70 outputs the created audio content to the outside. The audio content is distributed, for example, by broadcasting (radio or television) or Internet distribution.

Figure 11 is a schematic diagram showing an example of a screen of the user interface of the content production device 1. The user assumed by the content production device 1 is a content creator. The content production device 1 functions, for example, as a server device, and displays this screen on a display device of a user terminal (such as a PC) which is a client device. The example screen shown is for presenting to the user the composition of the audio content (weather information program) produced by the content production device 1.

On this screen, the character string "Content Production" is displayed as the screen title. In addition, the character string "2020/1/25 10:00 AM Weather Information" is displayed as the content name. This content name is set appropriately. In addition, it is displayed that the content length is "240 seconds". The content length is set in advance (see FIG. 9). In addition, the character string "2020/1/25 10:00:00" is displayed as the broadcast date and time. The broadcast date and time are set in advance. The content production device 1 may automatically output the generated audio content to the outside at an appropriate timing based on the broadcast date and time setting. In addition, the character string "Not completed" is displayed as the sending status. This sending status may be displayed based on information on the status of each content managed by a management unit (not shown) in the content production device 1, for example.

In addition, a column of information for each topic is displayed at the bottom of this screen. The screen may be scrollable up and down as appropriate. Topic information displayed includes the name of the topic, a name indicating the type of variation, and the character string of the generated sentence. The screen may also have a listen button and an edit button. When the user presses (clicks, etc.) the listen button, the user can listen to the synthesized speech for that topic. When the user presses the edit button, the user may be able to use the edit screen to modify (edit) the generated sentence or resynthesize speech based on the modified sentence.

Note that the user interface shown here is just an example, and the content production device 1 may be configured to display other information on the screen or allow the user to perform other operations.

According to this embodiment, the content production device 1 can automatically produce content by selecting and combining appropriate sentences from a plurality of variations based on a template. Also, according to this embodiment, the content production device 1 can automatically produce content such that the overall length (duration) of the content is a desired value. Also, according to this embodiment, the content production device 1 can automatically produce content in the form of announcements at a natural speed without performing speech speed conversion processing. Also, according to this embodiment, the content production device 1 selects variations based on evaluation values within the constraints of time (duration of the content), and as a result, can automatically produce content with a high evaluation value (i.e., more preferable content).

At least some of the functions of the content production device in the above-mentioned embodiment can be realized by a computer. In that case, a program for realizing the functions may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to realize the functions. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, a DVD-ROM, a USB memory, and a storage device such as a hard disk built into a computer system.
Furthermore, the term "computer-readable recording medium" may include a medium that temporarily and dynamically stores a program, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a medium that stores a program for a certain period of time, such as a volatile memory inside a computer system that serves as a server or client in such a case. The above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Although the embodiment has been described above, the present invention may be further embodied in the following modified examples.

In the example of content described in the embodiment, the topics include "Greeting 1" and "Greeting 2." Only one variation of each of these topics is included. For topics that do not have multiple variations like this, the content production device 1 may not automatically generate them (search for a solution for adjusting the time length). For example, the part that corresponds to the greeting may be produced using another method.

In the embodiment, voice synthesis of all variations is performed in advance to specify the duration of each sentence, but voice synthesis processing generally requires time and computational resources. Therefore, before actually performing voice synthesis processing, a process for accurately determining the duration of each sentence using a simpler process may be performed first. A process for accurately determining only the duration of a sentence without actually performing voice synthesis processing is possible using existing technology (for example, using a duration model that is common in DNN voice synthesis technology). In other words, instead of performing voice synthesis processing prior to search, only the duration may be obtained, and then a search process may be performed, and after the search process is completed, voice synthesis processing may be performed only for the sentence that was used as a solution.
Generally, the cost of processing to obtain only the duration of a voice is significantly smaller than the cost of processing to actually generate the voice waveform.

In the above embodiment, a configuration including voice synthesis processing has been described, but the content production device 1 may not include voice synthesis processing. After the search process is completed, spoken sentences may be output only for sentences that are to be used as solutions. These output spoken sentences may be subjected to separate voice synthesis processing.

In the above embodiment, an example of processing to accurately obtain the duration of each sentence has been described, but the duration of each sentence may include an error. In other words, in the search stage, the duration of each sentence is allowed to include a certain degree of error. This is because, in the search stage, if the duration of each sentence has a certain degree of accuracy, the error can be absorbed by length adjustment using pause adjustment units or speech rate conversion technology. In that case, the total content length shown in FIG. 9 may be a parameter expressed by a lower limit and an upper limit, like other parameters. Alternatively, the total content length may have an allowable error range. The allowable error range may be expressed in seconds (e.g., ±10 seconds) or as a ratio to the total content length (e.g., ±5%).

In the above embodiment, an example was described in which a fixed value was given as the evaluation value of the variation, but the value does not necessarily have to be fixed in advance. The evaluation value of the variation only needs to be determined when the search process is performed. For example, when searching, the evaluation value may be made variable depending on whether the sentences used up to that point include the same sentence or an equivalent sentence. This makes it possible to avoid repeating the same expression or the same information. The evaluation value may also be changed depending on the total value of the duration of the sentences used up to that point. This makes it possible to change the content depending on the remaining duration. In other words, even if the generated sentence is the same, the evaluation value may be changed depending on the timing at which it appears in the content or the context of the sentence.

In the above embodiment, the process is performed after the exact length of each sentence is determined in advance, but this length may be an approximate value. In that case, the actual length of the content created by this method may deviate from the target length depending on the difference between each approximate value and the exact value. If the deviation is small, it may be adjusted to the exact length using speech speed conversion technology or the like. It is known that naturalness is not lost even if speech speed conversion is adopted within a certain range (A. Nakamura et al. "A New Approach to Compensate Degeneration of Speech Intelligibility for Elderly Listeners", IEEE Transaction on Broadcasting, Vol.42, No3, 1996, etc.). For example, if the final length of the content is 240 seconds, it is possible to create sentences with an error allowable in the present invention of 12 seconds, and then adjust this error by speech speed conversion or the like so that it does not sound unnatural to the ear. Here, it is required to produce content within the range of allowable error. One example of a method for doing so is to check the average speech speed of the speech synthesizer of the device in advance and use this as a reference speech speed, and estimate the speech time from the number of characters using this value. Since the speech speed produced by speech synthesis is generally constant, it is expected that the speech time can be estimated within the range of allowable error. Depending on the use of the content, it may be required to speak a little faster or slower. In this case, the average speech speed of the speech synthesizer can be uniformly shifted to a desired value using speech speed conversion technology or the like, and this can be used as a new reference speech speed. As with the above method, it is possible to estimate the speech time with little error according to the reference speech speed. By managing the reference speech speed, content can be produced with an amount of information according to that speech speed.

In the embodiment, a process for automatically generating audio content for a weather forecast (weather information) has been described. The generated content may be something other than a weather forecast. For example, the content production device 1 may produce a news program in which the news is read out using a synthetic voice, a securities market program in which price information for each security issue is provided using a synthetic voice, a music program in which the songs to be played are introduced using a synthetic voice, or other content.

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment and includes designs that do not deviate from the gist of the present invention.

The present invention can be used, for example, in content production. However, the scope of use of the present invention is not limited to the examples given here.

1 Content production device 10 Data receiving unit (data acquisition unit)
20 Template storage unit 30 Variation generation unit 40 Search processing unit 50 Selection unit 60 Pose adjustment unit 70 Output unit

Claims (6)

a template storage unit that stores a content template having a plurality of variations of a sentence template for generating a sentence;
A data acquisition unit for acquiring data;
a variation generation unit that generates a sentence by applying the data to the sentence template for each of the plurality of variations included in the content template, and determines a duration of the synthetic speech that is a sum of durations of the synthetic speech corresponding to each sentence of the generated sentence;
a search processing unit that searches for combinations of variations using an appropriately determined total sum of evaluation values for each of the variations as an evaluation function under constraints related to conditions related to the duration of pauses that serve as joints when connecting the sentences and conditions related to the total duration of the synthetic voice and the pauses;
a selection unit that selects a combination of variations that satisfies the constraints based on a value of the evaluation function;
a pause adjustment unit that adjusts the duration of the pause so as to satisfy a condition regarding the duration of the pause and a condition regarding a total duration of the synthetic voice and the duration of the pause;
A content production device comprising: the content template is organized as a sequence of topics;
the topic is configured to include a plurality of the variations that may be selected mutually exclusively;
The content production device according to claim 1 . The condition regarding the duration of the pause includes a condition regarding the duration of an inter-sentence pause, which is a pause at a division of a sentence included in the variation, and a condition regarding the duration of an inter-topic pause, which is a pause at a division of a topic.
The content production device according to claim 2 . the pause adjustment unit adjusts the time lengths of the inter-sentence pauses so that they are all the same, and adjusts the time lengths of the inter-topic pauses so that they are all the same;
The content production device according to claim 3 . the evaluation value for each variation is predetermined as an attribute value of the variation included in the content template;
The content production device according to any one of claims 1 to 4. a template storage unit that stores a content template having a plurality of variations of a sentence template for generating a sentence;
A data acquisition unit for acquiring data;
a variation generation unit that generates a sentence by applying the data to the sentence template for each of the plurality of variations included in the content template, and determines a duration of the synthetic speech that is a sum of durations of the synthetic speech corresponding to each sentence of the generated sentence;
a search processing unit that searches for combinations of variations using a suitably determined total of evaluation values for each of the variations as an evaluation function under constraints related to conditions related to the duration of pauses that serve as joints when connecting the sentences and conditions related to the total duration of the synthetic voice and the pauses;
a selection unit that selects a combination of variations that satisfies the constraints based on a value of the evaluation function;
a pause adjustment unit that adjusts the duration of the pause so as to satisfy a condition regarding the duration of the pause and a condition regarding a total duration of the synthetic voice and the duration of the pause;
A program for causing a computer to function as a content production device comprising:

Images